Documentation Source Text

Check-in [8bc0546bb6]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Updates to TH3 documentation.
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 8bc0546bb60a6f8a07b794809d6d374b8a0b3c68
User & Date: drh 2016-08-29 11:56:55
Context
2016-08-29
16:07
Improvements to the opcode.html page. Minor tweaks to legacy c_interface.html and queryplanner-ng.html check-in: 5baf5a5191 user: drh tags: trunk
15:01
Merge latest trunk changes into experimental branch. check-in: b39f7479c0 user: dan tags: experimental
11:56
Updates to TH3 documentation. check-in: 8bc0546bb6 user: drh tags: trunk
2016-08-27
23:54
Use &lt; instead of < in the r-tree documentation. check-in: a55027b452 user: drh tags: trunk
Changes
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Changes to pages/testing.in.

   169    169   
   170    170   <li><p>
   171    171   The <b>[TH3]</b> test harness is a set of proprietary tests, written in
   172    172   C that provide 100% branch test coverage (and 100% MC/DC test coverage) to
   173    173   the core SQLite library.  The TH3 tests are designed to run
   174    174   on embedded and specialized platforms that would not easily support
   175    175   TCL or other workstation services.  TH3 tests use only the published 
   176         -SQLite interfaces.  TH3 is free to [SQLite Consortium] members 
   177         -and is available by license to others.  TH3 consists of about
          176  +SQLite interfaces. TH3 consists of about
   178    177   <tcl>MB {$stat(th3NByte)}</tcl> MB or <tcl>KB {$stat(th3SLOC)}</tcl> KSLOC
   179    178   of C code implementing <tcl>N {$stat(th3NTest)}</tcl> distinct test cases.
   180    179   TH3 tests are heavily parameterized, though, so a full-coverage test runs
   181    180   about <tcl>MB {$stat(th3NECov)}</tcl> million different test
   182    181   instances.  The cases that provide 100% branch test coverage constitute
   183    182   a subset of the total TH3 test suite.  A soak test
   184    183   prior to release does hundreds of millions of tests.

Changes to pages/th3.in.

     1         -<title>SQLite TH3</title>
            1  +<title>TH3</title>
     2      2   <tcl>hd_keywords {TH3}</tcl>
     3      3   
     4         -<h1 align="center">TH3: SQLite Test Harness #3</h1>
            4  +<fancy_format>
     5      5   
     6         -<h2>1.0 Overview</h2>
            6  +<h1>Overview</h1>
     7      7   
     8      8   <p>SQLite Test Harness #3 (hereafter "TH3") is one of
     9      9   [three test harnesses] used for testing SQLite.
    10         -TH3 is designed to meet the following objectives:</p>
           10  +TH3 meets the following objectives:</p>
    11     11   
    12     12   <ul>
    13     13   <li><p> TH3 is able to run on embedded platforms that lack the support
    14     14        infrastructure of workstations.</p></li>
    15     15   
    16     16   <li><p> TH3 tests SQLite in an as-deployed configuration using only
    17     17        published and documented interfaces.
    18     18        In other words, TH3 tests the compiled object code, not
    19     19        the source code, thus verifying that no problems were introduced
    20         -     by compiler bugs.  "Fly what you test and test what you fly."</p></li>
           20  +     by compiler bugs.  "Test what you fly and fly what you test."</p></li>
    21     21   
    22     22   <li><p> TH3 checks SQLite's response to out-of-memory errors, disk I/O
    23     23        errors, and power loss during transaction commit. </p></li>
    24     24   
    25     25   <li><p> TH3 exercises SQLite in a variety of run-time configurations
    26     26        (UTF8 vs UTF16, different pages sizes, varying journal modes, etc.)
    27     27        </p></li>
    28     28   
    29         -<li><p> TH3 achieves 100% branch test coverage (and 100% MC/DC)
           29  +<li><p> TH3 achieves 100% branch test coverage (and 100% 
           30  +    [https://en.wikipedia.org/wiki/Modified_condition/decision_coverage|MC/DC])
    30     31       over the SQLite core.
    31         -    (Test coverage of the operating-system specific [VFSes] and extensions
    32         -    such as FTS and RTREE is less than 100%). </p></li>
           32  +    (Test coverage of extensions such as FTS and RTREE is less than 100%).
           33  +     </p></li>
    33     34   </ul>
    34     35   
    35     36   <p>TH3 was originally written for validation testing only, but has
    36     37   subsequently been used for development testing and debugging
    37     38   as well, and has proven very helpful in those roles.  A full-coverage
    38     39   test takes less than five minutes on a workstation and hence
    39     40   serves as a fast regression test during day-to-day maintenance
    40     41   of the SQLite code base.</p>
    41     42   
    42         -<h2>2.0 Operation</h2>
           43  +<h2>History</h2>
           44  +
           45  +<p>TH3 originated from an effort to test SQLite on 
           46  +[https://en.wikipedia.org/wiki/Symbian|SymbianOS].
           47  +Prior to TH3, all SQLite tests were run using the
           48  +[http://www.tcl.tk/|TCL] script language, but TCL would not (easily)
           49  +compile on SymbianOS which made testing difficult.  The first attempt
           50  +to remedy this problem was the "TH1" (Test Harness #1) scripting 
           51  +language - a reimplementation of parts of the TCL language in a 
           52  +more portable form that would compile and run on SymbianOS, and 
           53  +that was sufficient to run the SQLite tests.  TH1
           54  +did not survive as a standard testing tool for SQLite,
           55  +but it did find continued service as a
           56  +scripting language used to customize the 
           57  +[http://www.fossil-scm.org/|Fossil] version control system.
           58  +There was also a "Test Harness #2", all traces of which have been
           59  +lost.  TH3 was the third attempt.
           60  +
           61  +<p>At about that same time, some avionics manufacturers were
           62  +expressing interest in SQLite, which prompted the SQLite developers
           63  +to design TH3 to support the rigorous testing standards of
           64  +[https://en.wikipedia.org/wiki/DO-178B|DO-178B].
           65  +
           66  +<p>The first code for TH3 was laid down on 2008-09-25.
           67  +An intense effort over the next 10 months resulted in TH3 achieving
           68  +100% MC/DC on 2009-07-25.  The TH3 code continues to be improved and
           69  +expanded.
           70  +
           71  +<p>As of 2016-08-29, the TH3 source tree consists 
           72  +and over 500,000 lines of source code in 1582 separate files.
           73  +
           74  +<h1>Operation</h1>
    43     75   
    44     76   <p>TH3 is a test program generator.  The output of TH3 is a program
    45     77   implemented in C-code and intended to be
    46     78   linked against the SQLite library under test.  The generated test
    47     79   program is compiled and run on the target platform in order to verify
    48     80   correct operation of SQLite on that platform.</p>
    49     81   
    50     82   <p>The inputs to TH3 are test modules written in C or SQL and
    51     83   small configuration
    52     84   files that determine how to initialize SQLite.  The
    53         -TH3 package includes over eleven hundred test
    54         -modules and more than three dozen configuration files.
           85  +TH3 package includes over 1,300 test
           86  +modules and more than 40 configurations (as of 2016-08-29).
    55     87   New modules and configurations
    56     88   can be added to customize TH3 for specialized applications.
    57     89   Each time TH3 is run, it reads
    58     90   a subset of the available test modules and configuration files to generate
    59     91   a custom C program that performs all of the specified tests under all
    60     92   specified configurations.  A complete test of SQLite normally involves running
    61     93   TH3 multiple times to generate multiple test programs covering different
    62     94   aspects of SQLite's operation, then linking all test programs against
    63     95   a common SQLite library and running them separately on the target platform.
    64         -SQLite will be found to work if all test programs pass.</p>
           96  +</p>
    65     97   
    66     98   <p>There are no arbitrary limits in TH3.  One could generate a
    67     99   single test program that contained all test modules and all configuration files.
    68    100   However, such a test program might be too large to deploy on embedded
    69         -platforms.  Hence, TH3 provides the ability to break the library of test
          101  +platforms.  (As of 2016-08-29, a full-up TH3 test is over 820,000 lines and
          102  +55MB of C code.)  TH3 provides the ability to break the library of test
    70    103   modules up into smaller, more easily digested pieces.</p>
    71    104   
    72    105   <p>Each individual test module might contain dozens, hundreds, or thousands
    73    106   of separate tests.  The test modules can be written in C or as scripts of
    74    107   SQL or a mixture of the two.  About two-thirds of the existing test modules are
    75    108   written SQL with the remainder either in pure C or a combination of C and SQL.
    76    109   </p>
    77    110   
    78    111   <p>Each test module file contains a header which describes the circumstances
    79    112   under which the test is valid.  For a particular configuration, only those
    80    113   modules that are compatible with the configuration are run.  </p>
    81    114   
    82         -<h2>3.0 Generating A Test Program</h2>
          115  +<h1>Generating A Test Program</h1>
    83    116   
    84    117   <p>The TH3 program generator is a TCL script named "<tt>mkth3.tcl</tt>".
    85    118   To generate a test program, one has merely to run this script and supply
    86    119   the names of files containing test modules and configurations on the
    87    120   command line.  Test modules are files that use the "<tt>.test</tt>" suffix
    88    121   and configurations are files that use the "<tt>.cfg</tt>" suffix.  A
    89    122   typical invocation of mkth3.tcl might look something like the following:</p>
................................................................................
   171    204   configurations and "pager08", "build33", "orderby01", etc. are test modules.
   172    205   Compile-time and run-time options are available to increase or decrease
   173    206   the amount of output.
   174    207   The output can be increased by showing each test case within each
   175    208   test module.  The output can be decreased
   176    209   by degrees: omitting test modules starts and stops,
   177    210   omitting configuration starts and stops, and finally by omitting all output.
   178         -<h3>3.1 Test Automation Scripts</h3>
          211  +
          212  +<h2>Test Automation Scripts</h2>
   179    213   
   180    214   <p>TH3 comes with additional TCL scripts that help automate the testing
   181    215   process on workstations.  The "th3make" script automatically runs "mkth3.tcl"
   182    216   and "gcc" and then runs the resulting test program and checks the results.
   183    217   Arguments to th3make include all of the "*.test" test modules and 
   184    218   "*.cfg" configurations that are to be included in the test.  Additional
   185    219   options to th3make can cause the test program to be compiled using different
................................................................................
   235    269   file rename sqlite3.c ../sqlite3udl.c
   236    270   0d3bbc92c433f940253bb2c7c19de7783133929d  ../sqlite3udl.c
   237    271   exec make clean sqlite3.c OPTS=-DSQLITE_SMALL_STACK=1
   238    272   file rename sqlite3.c ../sqlite3ss.c
   239    273   fcf6963e94096324461076d3b9e9dc1888e066e1  ../sqlite3ss.c
   240    274   cd ..
   241    275   *******************************************************************************
   242         -t01: quick.rc.................................................. Ok   (00:04:00)
   243         -t02: cov.rc.................................................... Ok   (00:04:40)
   244         -t03: quick.rc extensions.rc -D_HAVE_SQLITE_CONFIG_H............ Ok   (00:05:22)
   245         -t04: cov.rc -DSQLITE_ENABLE_STAT4 -D_HAVE_SQLITE_CONFIG_H...... Ok   (00:05:20)
   246         -t05: test.rc ../th3private/*.test.............................. Ok   (00:00:17)
   247         -t06: test.rc ../th3private/*.test -DSQLITE_ENABLE_STAT4........ Ok   (00:00:43)
   248         -t07: quick.rc -DSQLITE_TEST_REALLOC_STRESS -funsigned-char..... Ok   (00:04:56)
   249         -t08: quick.rc -DSQLITE_THREADSAFE=0 -fsigned-char.............. Ok   (00:03:12)
   250         -t09: quick.rc sqlite3ss.c -DSQLITE_MAX_ATTACHED=125............ Ok   (00:04:04)
   251         -t10: quick.rc -DSQLITE_RUNTIME_BYTEORDER....................... Ok   (00:03:58)
   252         -t11: quick.rc -DSQLITE_DIRECT_OVERFLOW_READ.................... Ok   (00:04:01)
   253         -t12: fast.rc................................................... Ok   (00:14:19)
   254         -t13: fast.rc alignment1.rc -m32................................ Ok   (00:20:51)
   255         -t14: fast.rc alignment2.rc sqlite3udl.c........................ Ok   (00:16:06)
   256         -t15: fast.rc alignment4.rc..................................... Ok   (00:12:55)
   257         -t16: fast.rc alignment5.rc..................................... Ok   (00:14:58)
   258         -t17: fast.rc alignment6.rc..................................... Ok   (00:14:31)
   259         -t18: fast.rc alignment7.rc..................................... Ok   (00:16:06)
   260         -t19: fast.rc alignment8.rc sqlite3udl.c........................ Ok   (00:24:09)
   261         -t20: test.rc alignment1.rc..................................... Ok   (00:49:27)
   262         -t21: test.rc alignment2.rc sqlite3udl.c........................ Ok   (00:38:43)
   263         -t22: test.rc alignment4.rc -m32 CC=clang....................... Ok   (00:39:49)
   264         -t23: test.rc alignment5.rc..................................... Ok   (00:36:33)
   265         -t24: test.rc alignment6.rc..................................... Ok   (00:33:53)
   266         -t25: test.rc alignment7.rc..................................... Ok   (00:42:16)
   267         -t26: test.rc alignment8.rc sqlite3udl.c........................ Ok   (01:05:22)
   268         -t27: memdebug.rc extensions.rc................................. Ok   (01:35:56)
   269         -t28: fast.rc -fsanitize=undefined.............................. Ok   (00:15:09)
   270         -t29: min.rc -O3 -valgrind...................................... Ok   (01:26:10)
   271         -t30: min.rc -O3 -valgrind extensions.rc........................ Ok   (01:47:12)
   272         -t31: test-ex.rc................................................ Ok   (03:20:18)
          276  +t03: demo.rc................................................... Ok   (00:00:05)
          277  +t01: quick.rc.................................................. Ok   (00:03:44)
          278  +t02: cov.rc.................................................... Ok   (00:03:58)
          279  +t04: quick.rc extensions.rc -D_HAVE_SQLITE_CONFIG_H............ Ok   (00:05:25)
          280  +t08: vfs-cov.rc................................................ Ok   (00:03:35)
          281  +t05: cov.rc -DSQLITE_ENABLE_STAT4 -D_HAVE_SQLITE_CONFIG_H...... Ok   (00:04:40)
          282  +t09: quick.rc -DSQLITE_TEST_REALLOC_STRESS -funsigned-char..... Ok   (00:04:25)
          283  +t10: quick.rc -DSQLITE_THREADSAFE=0 -DLONGDOUBLE_TYPE=double... Ok   (00:03:00)
          284  +t11: quick.rc sqlite3ss.c -DSQLITE_MAX_ATTACHED=125............ Ok   (00:03:45)
          285  +t14: quick.rc -DSQLITE_TRACE_SIZE_LIMIT=15 cov1/main16.test.... Ok   (00:00:17)
          286  +t12: quick.rc -DSQLITE_RUNTIME_BYTEORDER -fsigned-char......... Ok   (00:03:46)
          287  +t13: quick.rc -DSQLITE_DIRECT_OVERFLOW_READ.................... Ok   (00:03:42)
          288  +t16: test.rc alignment2.rc sqlite3udl.c........................ Ok   (00:35:25)
          289  +t15: test.rc alignment1.rc..................................... Ok   (00:48:20)
          290  +t19: valgrind1.rc -O3 extensions.rc............................ Ok   (00:33:14)
          291  +t17: memdebug1.rc extensions.rc................................ Ok   (01:23:44)
          292  +t18: memdebug2.rc extensions.rc................................ Ok   (01:23:13)
          293  +t20: valgrind2.rc -O3 extensions.rc............................ Ok   (00:40:21)
          294  +t21: test-ex1.rc............................................... Ok   (00:40:37)
          295  +t23: test-ex3.rc............................................... Ok   (00:29:04)
          296  +t22: test-ex2.rc............................................... Ok   (00:53:28)
          297  +t24: test-ex4.rc............................................... Ok   (00:47:59)
          298  +t26: test.rc alignment4.rc -m32 CC=clang....................... Ok   (00:37:38)
          299  +t25: test.rc alignment3.rc sqlite3udl.c........................ Ok   (01:04:01)
          300  +t27: test.rc alignment5.rc extensions.rc....................... Ok   (00:44:58)
          301  +t28: test.rc alignment6.rc..................................... Ok   (00:28:49)
          302  +t32: fast.rc alignment1.rc extensions.rc -m32.................. Ok   (00:30:05)
          303  +t29: test.rc alignment7.rc..................................... Ok   (00:34:17)
          304  +t33: fast.rc alignment2.rc sqlite3udl.c........................ Ok   (00:13:51)
          305  +t35: fast.rc alignment4.rc..................................... Ok   (00:11:08)
          306  +t36: fast.rc alignment5.rc..................................... Ok   (00:12:31)
          307  +t37: fast.rc alignment6.rc..................................... Ok   (00:11:15)
          308  +t34: fast.rc alignment3.rc sqlite3udl.c........................ Ok   (00:23:05)
          309  +t38: fast.rc alignment7.rc..................................... Ok   (00:12:26)
          310  +t39: fast.rc -fsanitize=undefined.............................. Ok   (00:24:15)
   273    311   *******************************************************************************
   274         -0 failures on 31 th3make runs and 166721387 tests in (16:25:29)
   275         -SQLite 3.8.10 2015-05-05 18:52:54 04afa3febee32854fbb09ef8d4ffffd432119716
          312  +0 failures on 35 th3makes and 171555634 tests in (05:08:31) 3 cores on bella
          313  +SQLite 3.14.1 2016-08-11 13:08:14 34aed3a318a413fd180604365546c1f530d1c60c
   276    314   </pre></blockquote>
   277    315   
   278    316   <p>As can be seen above, a single run
   279         -of multitest.tcl invokes th3make dozens times and takes between 12 and 24
   280         -hours.  The middle section of the output shows the arguments to each individual
   281         -th3make run and the result and elapse time for that th3make.
          317  +of multitest.tcl invokes th3make dozens or times and takes between 12 and 24
          318  +CPU hours.  The middle section of the output shows the arguments to each
          319  +individual th3make run and the result and elapse time for that th3make.
   282    320   All build products and output for the separate th3make runs are
   283    321   captures in subdirectories for post-test analysis.
   284    322   The two-line summary at the bottom shows the total number of errors and tests
   285    323   over all th3make runs and the total elapse time, together with the 
   286    324   [SQLITE_SOURCE_ID] information for the version of SQLite that was
   287    325   tested.  This summary information is recorded in the
   288         -<a href="https://www.sqlite.org/checklists/3081000/index#c6">release
   289         -checklist</a> during testing.
          326  +<a href="https://www.sqlite.org/checklists/3140000/index#c6">release
          327  +checklist</a> during final testing.
   290    328   
   291         -<h2>4.0 Test Coverage</h2>
          329  +<h1>Test Coverage</h1>
   292    330   
   293    331   <p>Using one particular subset of the available TH3 test modules (the "cov1"
   294    332   tests) SQLite obtained 
   295    333   [test coverage | 100% branch test coverage] and 100% [MC/DC] as measured
   296    334   by [http://gcc.gnu.org/onlinedocs/gcc/Gcov.html | gcov]
   297    335   on Linux x86 and x86_64 hardware.  All releases of SQLite since
   298    336   [version 3.6.17] (circa 2009-08-10) have been tested to this standard. 
................................................................................
   300    338   are committed to maintaining 100% branch coverage and MC/DC for all 
   301    339   future releases of SQLite.</p>
   302    340   
   303    341   <p>The cov1 test set used to obtain 100% branch test coverage are only a
   304    342   subset of the tests currently implemented using TH3.  New test modules are
   305    343   added on a regular basis.</p>
   306    344   
   307         -<h2>5.0 TH3 License</h2>
          345  +<h1>TH3 License</h1>
   308    346   
   309    347   <p>SQLite itself is in the <a href="copyright.html">public domain</a> and
   310    348   can be used for any purpose.  But TH3 is proprietary and requires a license.
   311    349   </p>
   312    350   
   313    351   <p>Even though open-source users do not have direct access to TH3, all
   314    352   users of SQLite benefit from TH3 indirectly since each version of SQLite is
   315    353   validated running TH3 on multiple platforms (Linux, Windows, WinRT, Mac,
   316    354   OpenBSD) prior to release.  So anyone using an official release
   317    355   of SQLite can deploy their application with the confidence of knowing that
   318    356   it has been tested using TH3.  They simply cannot rerun those tests
   319    357   themselves without purchasing a TH3 license.</p>