Documentation Source Text

Check-in [ac3ef1f6aa]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Updates to the TH3 documentation.
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256:ac3ef1f6aa4506b005bf35b255d79ebb6fad449f32263e96fa62670e0495bd94
User & Date: drh 2018-05-19 14:21:54
Context
2018-05-22
15:28
Update information on read-only WAL-mode databases to conform with the new capabilities added in version 3.22.0. check-in: ba027764e3 user: drh tags: trunk
2018-05-19
14:21
Updates to the TH3 documentation. check-in: ac3ef1f6aa user: drh tags: trunk
2018-05-18
18:02
Merge fixes from the 3.23 branch. check-in: 45753832a8 user: drh tags: trunk
Changes
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Changes to pages/th3.in.

    64     64   [https://en.wikipedia.org/wiki/DO-178B|DO-178B].
    65     65   
    66     66   <p>The first code for TH3 was laid down on 2008-09-25.
    67     67   An intense effort over the next 10 months resulted in TH3 achieving
    68     68   100% MC/DC on 2009-07-25.  The TH3 code continues to be improved and
    69     69   expanded.
    70     70   
    71         -<p>As of 2016-08-29, the TH3 source tree consists 
    72         -and over 500,000 lines of source code in 1582 separate files.
           71  +<p>As of 2018-05-19, the TH3 source tree consists 
           72  +and well over 500,000 lines of source code in 1709 separate files.
    73     73   
    74     74   <h1>Operation</h1>
    75     75   
    76     76   <p>TH3 is a test program generator.  The output of TH3 is a program
    77     77   implemented in C-code and intended to be
    78     78   linked against the SQLite library under test.  The generated test
    79     79   program is compiled and run on the target platform in order to verify
    80     80   correct operation of SQLite on that platform.</p>
    81     81   
    82     82   <p>The inputs to TH3 are test modules written in C or SQL and
    83     83   small configuration
    84     84   files that determine how to initialize SQLite.  The
    85         -TH3 package includes over 1,300 test
    86         -modules and more than 40 configurations (as of 2016-08-29).
           85  +TH3 package includes  1,444 test
           86  +modules and more than 47 configurations (as of 2018-05-19).
    87     87   New modules and configurations
    88     88   can be added to customize TH3 for specialized applications.
    89     89   Each time TH3 is run, it reads
    90     90   a subset of the available test modules and configuration files to generate
    91     91   a custom C program that performs all of the specified tests under all
    92     92   specified configurations.  A complete test of SQLite normally involves running
    93     93   TH3 multiple times to generate multiple test programs covering different
................................................................................
    94     94   aspects of SQLite's operation, then linking all test programs against
    95     95   a common SQLite library and running them separately on the target platform.
    96     96   </p>
    97     97   
    98     98   <p>There are no arbitrary limits in TH3.  One could generate a
    99     99   single test program that contained all test modules and all configuration files.
   100    100   However, such a test program might be too large to deploy on embedded
   101         -platforms.  (As of 2016-08-29, a full-up TH3 test is over 820,000 lines and
   102         -55MB of C code.)  TH3 provides the ability to break the library of test
          101  +platforms.  (As of 2018-05-19, a full-up TH3 test is over 850,000 lines and
          102  +58MB of C code.)  TH3 provides the ability to break the library of test
   103    103   modules up into smaller, more easily digested pieces.</p>
   104    104   
   105    105   <p>Each individual test module might contain dozens, hundreds, or thousands
   106    106   of separate tests.  The test modules can be written in C or as scripts of
   107    107   SQL or a mixture of the two.  About two-thirds of the existing test modules are
   108         -written SQL with the remainder either in pure C or a combination of C and SQL.
          108  +written in pure SQL with the remainder either in pure C or a combination 
          109  +of C and SQL.
   109    110   </p>
   110    111   
   111    112   <p>Each test module file contains a header which describes the circumstances
   112    113   under which the test is valid.  For a particular configuration, only those
   113    114   modules that are compatible with the configuration are run.  </p>
   114    115   
   115    116   <h1>Generating A Test Program</h1>
................................................................................
   141    142   In a working installation, one would normally want
   142    143   to specify optimization parameters and compile-time switches on the
   143    144   compiler command line.</p>
   144    145   
   145    146   <p>For testing on embedded systems, the mkth3.tcl script and the compiler
   146    147   steps shown above are performed on an ordinary workstation using
   147    148   a cross-compiler, then the resulting test program is
   148         -transfer onto the device to be run.</p>
          149  +transferred onto the device to be run.</p>
   149    150   
   150    151   <p>Once the test program is generated, it is run with no arguments to
   151    152   perform the tests.  Progress information as well as error diagnostics
   152    153   appear on standard output.  (Alternative output arrangements can be made
   153    154   using a compile-time option for embedded devices that lack a standard
   154    155   output channel.) The program returns zero if there are no
   155    156   errors and non-zero if any problems were detected.</p>
................................................................................
   253    254   TCL script used to automate TH3 testing on workstations.  Multitest.tcl
   254    255   automatically compiles SQLite, then
   255    256   runs ./th3make repeatedly with a variety of alignments, and captures
   256    257   the output in a succinct summary screen.  A typical multitest.tcl run
   257    258   generates output that looks like this:
   258    259   
   259    260   <codeblock>
          261  +./multitest.tcl -q --jobs 3
          262  +start-time: 2018-05-19 03:17:12 UTC
   260    263   file mkdir sqlite3bld
   261    264   cd sqlite3bld
   262         -exec sh /home/drh/sqlite/sqlite/configure
          265  +exec sh /ramdisk/sqlite/configure
   263    266   file copy -force config.h ../config.h
   264    267   exec make clean sqlite3.c
   265    268   file rename sqlite3.c ../sqlite3.c
   266         -aa4f0f90c9c77424943e026a2ecee4a6c7f9e0d3  ../sqlite3.c
   267    269   file rename sqlite3.h ../sqlite3.h
   268    270   exec make clean sqlite3.c OPTS=-DSQLITE_ENABLE_UPDATE_DELETE_LIMIT=1
   269    271   file rename sqlite3.c ../sqlite3udl.c
   270         -0d3bbc92c433f940253bb2c7c19de7783133929d  ../sqlite3udl.c
   271    272   exec make clean sqlite3.c OPTS=-DSQLITE_SMALL_STACK=1
   272    273   file rename sqlite3.c ../sqlite3ss.c
   273         -fcf6963e94096324461076d3b9e9dc1888e066e1  ../sqlite3ss.c
   274    274   cd ..
   275    275   *******************************************************************************
   276         -t03: demo.rc................................................... Ok   (00:00:05)
   277         -t01: quick.rc.................................................. Ok   (00:03:44)
   278         -t02: cov.rc.................................................... Ok   (00:03:58)
   279         -t04: quick.rc extensions.rc -D_HAVE_SQLITE_CONFIG_H............ Ok   (00:05:25)
   280         -t08: vfs-cov.rc................................................ Ok   (00:03:35)
   281         -t05: cov.rc -DSQLITE_ENABLE_STAT4 -D_HAVE_SQLITE_CONFIG_H...... Ok   (00:04:40)
   282         -t09: quick.rc -DSQLITE_TEST_REALLOC_STRESS -funsigned-char..... Ok   (00:04:25)
   283         -t10: quick.rc -DSQLITE_THREADSAFE=0 -DLONGDOUBLE_TYPE=double... Ok   (00:03:00)
   284         -t11: quick.rc sqlite3ss.c -DSQLITE_MAX_ATTACHED=125............ Ok   (00:03:45)
   285         -t14: quick.rc -DSQLITE_TRACE_SIZE_LIMIT=15 cov1/main16.test.... Ok   (00:00:17)
   286         -t12: quick.rc -DSQLITE_RUNTIME_BYTEORDER -fsigned-char......... Ok   (00:03:46)
   287         -t13: quick.rc -DSQLITE_DIRECT_OVERFLOW_READ.................... Ok   (00:03:42)
   288         -t16: test.rc alignment2.rc sqlite3udl.c........................ Ok   (00:35:25)
   289         -t15: test.rc alignment1.rc..................................... Ok   (00:48:20)
   290         -t19: valgrind1.rc -O3 extensions.rc............................ Ok   (00:33:14)
   291         -t17: memdebug1.rc extensions.rc................................ Ok   (01:23:44)
   292         -t18: memdebug2.rc extensions.rc................................ Ok   (01:23:13)
   293         -t20: valgrind2.rc -O3 extensions.rc............................ Ok   (00:40:21)
   294         -t21: test-ex1.rc............................................... Ok   (00:40:37)
   295         -t23: test-ex3.rc............................................... Ok   (00:29:04)
   296         -t22: test-ex2.rc............................................... Ok   (00:53:28)
   297         -t24: test-ex4.rc............................................... Ok   (00:47:59)
   298         -t26: test.rc alignment4.rc -m32 CC=clang....................... Ok   (00:37:38)
   299         -t25: test.rc alignment3.rc sqlite3udl.c........................ Ok   (01:04:01)
   300         -t27: test.rc alignment5.rc extensions.rc....................... Ok   (00:44:58)
   301         -t28: test.rc alignment6.rc..................................... Ok   (00:28:49)
   302         -t32: fast.rc alignment1.rc extensions.rc -m32.................. Ok   (00:30:05)
   303         -t29: test.rc alignment7.rc..................................... Ok   (00:34:17)
   304         -t33: fast.rc alignment2.rc sqlite3udl.c........................ Ok   (00:13:51)
   305         -t35: fast.rc alignment4.rc..................................... Ok   (00:11:08)
   306         -t36: fast.rc alignment5.rc..................................... Ok   (00:12:31)
   307         -t37: fast.rc alignment6.rc..................................... Ok   (00:11:15)
   308         -t34: fast.rc alignment3.rc sqlite3udl.c........................ Ok   (00:23:05)
   309         -t38: fast.rc alignment7.rc..................................... Ok   (00:12:26)
   310         -t39: fast.rc -fsanitize=undefined.............................. Ok   (00:24:15)
          276  +t01: cov.rc.................................................... Ok   (00:03:42)
          277  +t02: cov.rc ++STAT4 ++DESERIALIZE -D_HAVE_SQLITE_CONFIG_H...... Ok   (00:04:45)
          278  +t03: vfs-cov.rc................................................ Ok   (00:03:59)
          279  +t04: demo.rc................................................... Ok   (00:00:05)
          280  +t07: test.rc ../th3private/*.test.............................. Ok   (00:00:21)
          281  +t08: test.rc ../th3private/*.test ++STAT4...................... Ok   (00:01:41)
          282  +t05: quick.rc.................................................. Ok   (00:04:26)
          283  +t09: quick.rc ~TEST_REALLOC_STRESS -funsigned-char............. Ok   (00:05:39)
          284  +t10: quick.rc ~THREADSAFE=0 -DLONGDOUBLE_TYPE=double........... Ok   (00:03:24)
          285  +t06: quick.rc extensions.rc -D_HAVE_SQLITE_CONFIG_H............ Ok   (00:09:03)
          286  +t11: quick.rc sqlite3ss.c ~MAX_ATTACHED=125.................... Ok   (00:04:39)
          287  +t12: quick.rc ~BYTEORDER=0 ++RTREE............................. Ok   (00:07:28)
          288  +t13: quick.rc ~DISABLE_INTRINSIC ++RTREE....................... Ok   (00:07:31)
          289  +t16: quick.rc ~TRACE_SIZE_LIMIT=15 cov1/main16.test............ Ok   (00:00:22)
          290  +t14: quick.rc ~DIRECT_OVERFLOW_READ -fsigned-char.............. Ok   (00:04:35)
          291  +t15: quick.rc ~UNTESTABLE ~EXTRA_IFNULLROW..................... Ok   (00:01:44)
          292  +t17: quick.rc ~MAX_MMAP_SIZE=0................................. Ok   (00:04:46)
          293  +t18: quick.rc ++NULL_TRIM ++OFFSET_SQL_FUNC.................... Ok   (00:04:47)
          294  +t19: quick.rc ++BATCH_ATOMIC_WRITE ++DESERIALIZE............... Ok   (00:05:41)
          295  +t20: lean1.rc quick.rc......................................... Ok   (00:03:09)
          296  +t22: test.rc alignment2.rc sqlite3udl.c........................ Ok   (00:44:22)
          297  +t21: test.rc alignment1.rc..................................... Ok   (01:02:32)
          298  +t23: memdebug1.rc extensions.rc................................ Ok   (01:49:58)
          299  +t25: valgrind1.rc -O3 extensions.rc............................ Ok   (00:56:08)
          300  +t24: memdebug2.rc extensions.rc................................ Ok   (01:43:34)
          301  +t27: test-ex1.rc............................................... Ok   (00:45:00)
          302  +t26: valgrind2.rc -O3 extensions.rc............................ Ok   (01:02:52)
          303  +t29: test-ex3.rc............................................... Ok   (00:31:48)
          304  +t28: test-ex2.rc............................................... Ok   (01:12:03)
          305  +t30: test-ex4.rc............................................... Ok   (01:09:47)
          306  +t32: test.rc alignment4.rc -m32 CC=clang....................... Ok   (00:48:31)
          307  +t31: test.rc alignment3.rc sqlite3udl.c........................ Ok   (01:22:29)
          308  +t34: test.rc alignment6.rc..................................... Ok   (00:35:31)
          309  +t33: test.rc alignment5.rc extensions.rc....................... Ok   (00:59:33)
          310  +t35: test.rc alignment7.rc..................................... Ok   (00:44:10)
          311  +t40: fast.rc alignment2.rc sqlite3udl.c........................ Ok   (00:15:46)
          312  +t39: fast.rc alignment1.rc extensions.rc -m32.................. Ok   (00:33:19)
          313  +t36: test.rc ~MUTATION_TEST.................................... Ok   (00:35:45)
          314  +t42: fast.rc alignment4.rc..................................... Ok   (00:13:03)
          315  +t43: fast.rc alignment5.rc..................................... Ok   (00:13:32)
          316  +t44: fast.rc alignment6.rc..................................... Ok   (00:11:41)
          317  +t41: fast.rc alignment3.rc sqlite3udl.c........................ Ok   (00:26:31)
          318  +t45: fast.rc alignment7.rc..................................... Ok   (00:12:57)
          319  +t46: fast.rc -fsanitize=undefined.............................. Ok   (00:38:18)
   311    320   *******************************************************************************
   312         -0 failures on 35 th3makes and 171555634 tests in (05:08:31) 3 cores on bella
   313         -SQLite 3.14.1 2016-08-11 13:08:14 34aed3a318a413fd180604365546c1f530d1c60c
          321  +0 failures on 44 th3makes and 198583082 tests in (07:16:01) 3 cores on bella
          322  +SQLite 3.24.0 2018-05-18 17:58:33 c6071ac99cfa4b6272ac4d739fc61a85acb544f6c1c2a
   314    323   </codeblock>
   315    324   
   316    325   <p>As can be seen above, a single run
   317    326   of multitest.tcl invokes th3make dozens of times and takes between 12 and 24
   318    327   CPU hours.  The middle section of the output shows the arguments to each
   319    328   individual th3make run and the result and elapse time for that th3make.
   320    329   All build products and output for the separate th3make runs are
   321    330   captures in subdirectories for post-test analysis.
   322    331   The two-line summary at the bottom shows the total number of errors and tests
   323    332   over all th3make runs and the total elapse time, together with the 
   324    333   [SQLITE_SOURCE_ID] information for the version of SQLite that was
   325    334   tested.  This summary information is recorded in the
   326         -<a href="https://www.sqlite.org/checklists/3140000/index#c6">release
          335  +<a href="https://www.sqlite.org/checklists">release
   327    336   checklist</a> during final testing.
          337  +
          338  +<p>Abbreviations are applied in the multitest.tcl output so that
          339  +each th3make invocation will fit on a single 80-column output line.
          340  +The initial "th3make" verb is omitted.
          341  +"~" is shorthand for "-DSQLITE_" and "++" is stands for
          342  +"-DSQLITE_ENABLE".  Hence, multitest.tcl output line
          343  +
          344  +<codeblock>
          345  +quick.rc ~DISABLE_INTRINSIC ++RTREE
          346  +</codeblock>
          347  +
          348  +<p>Really means
          349  +
          350  +<codeblock>
          351  +th3make quick.rc -DSQLITE_DISABLE_INTRINSIC -DSQLITE_ENABLE_RTREE
          352  +</codeblock>
   328    353   
   329    354   <h1>Test Coverage</h1>
   330    355   
   331    356   <p>Using one particular subset of the available TH3 test modules (the "cov1"
   332    357   tests) SQLite obtained 
   333    358   [test coverage | 100% branch test coverage] and 100% [MC/DC] as measured
   334    359   by [http://gcc.gnu.org/onlinedocs/gcc/Gcov.html | gcov]