Documentation Source Text

Check-in [7d4f8c9ada]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Increased information about TH3.
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 7d4f8c9ada8291fa44e3345f3639a0aeb92606fc
User & Date: drh 2015-05-15 14:48:20
Context
2015-05-26
12:26
Cherrypick several recent minor changes from trunk. check-in: 0f4cfe6cd0 user: drh tags: branch-3.8.10
2015-05-15
14:53
Link from the ATTACH documentation page to the DETACH page. Increased information about TH3. These are cherry-picks from trunk. check-in: 643e08dfb8 user: drh tags: branch-3.8.10
14:51
Link from the ATTACH documentation page to the DETACH page. check-in: b3cbd71993 user: drh tags: trunk
14:48
Increased information about TH3. check-in: 7d4f8c9ada user: drh tags: trunk
2015-05-11
19:12
Update fts3.html with a description of the matchinfo 'b' option. check-in: 2dad241898 user: dan tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to pages/th3.in.

31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
...
117
118
119
120
121
122
123
























































124
125
126
127
128
129













































































































130
131
132
133
134
135
136
    (Test coverage of the operating-system specific [VFSes] and extensions
    such as FTS and RTREE is less than 100%). </p></li>
</ul>

<p>TH3 was originally written for validation testing only, but has
subsequently been used for development testing and debugging
as well, and has proven very helpful in those roles.  A full-coverage
test takes less than three minutes on a workstation and hence
serves as a fast regression test during day-to-day maintenance
of the SQLite code base.</p>

<h2>2.0 Operation</h2>

<p>TH3 is a test program generator.  The output of TH3 is a program
written in ANSI-C and intended to be
linked against the SQLite library under test.  The generated test
program is compiled and run on the target platform in order to verify
correct operation of SQLite on that platform.</p>

<p>The inputs to TH3 are test modules written in C or SQL and
small configuration
files that determine how to initialize SQLite.  The
TH3 package includes over one thousand test
modules and several dozen configuration files.
New modules and configurations
can be added to customize TH3 for specialized applications.
Each time TH3 is run, it reads
a subset of the available test modules and configuration files to generate
a custom C program that performs all of the specified tests under all
configurations.  A complete test of SQLite normally involves running
TH3 multiple times to generate multiple test programs covering different
aspects of SQLite's operation, then linking all test programs against
a common SQLite library and running them separately on the target platform.
SQLite will be found to work if all test programs pass.</p>

<p>There are no arbitrary limits in TH3.  One could generate a
single test program that contained all test modules and configuration files.
However, such a test program might be too large to deploy on embedded
platforms.  Hence, TH3 provides the ability to break the library of test
modules up into smaller, more easily digested pieces.</p>

<p>Each individual test module might contain dozens, hundreds, or thousands
of separate tests.  The test modules can be written in C or as scripts of
SQL or a mixture of the two.  About two-thirds of the existing test modules are
................................................................................
<p>Once the test program is generated, it is run with no arguments to
perform the tests.  Progress information as well as error diagnostics
appear on standard output.  (Alternative output arrangements can be made
using a compile-time option for embedded devices that lack a standard
output channel.) The program returns zero if there are no
errors and non-zero if any problems were detected.</p>

























































<p>TH3 comes with additional TCL scripts (for example:
"th3make" and "fulltest.tcl")
which help to automate the process of running mkth3.tcl, compiling th3.c and
sqlite3.c, running the resulting test programs, and analyzing the results.
These other scripts are for convenience only and are not required in order
to make use of TH3.














































































































<h2>4.0 Test Coverage</h2>

<p>Using one particular subset of the available TH3 test modules (the "cov1"
tests) SQLite obtained 
[test coverage | 100% branch test coverage] and 100% [MC/DC] as measured
by [http://gcc.gnu.org/onlinedocs/gcc/Gcov.html | gcov]







|






|







|
|





|






|







 







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|
<
<
<
<
<
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
...
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180





181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
    (Test coverage of the operating-system specific [VFSes] and extensions
    such as FTS and RTREE is less than 100%). </p></li>
</ul>

<p>TH3 was originally written for validation testing only, but has
subsequently been used for development testing and debugging
as well, and has proven very helpful in those roles.  A full-coverage
test takes less than five minutes on a workstation and hence
serves as a fast regression test during day-to-day maintenance
of the SQLite code base.</p>

<h2>2.0 Operation</h2>

<p>TH3 is a test program generator.  The output of TH3 is a program
implemented in C-code and intended to be
linked against the SQLite library under test.  The generated test
program is compiled and run on the target platform in order to verify
correct operation of SQLite on that platform.</p>

<p>The inputs to TH3 are test modules written in C or SQL and
small configuration
files that determine how to initialize SQLite.  The
TH3 package includes over eleven hundred test
modules and more than three dozen configuration files.
New modules and configurations
can be added to customize TH3 for specialized applications.
Each time TH3 is run, it reads
a subset of the available test modules and configuration files to generate
a custom C program that performs all of the specified tests under all
specified configurations.  A complete test of SQLite normally involves running
TH3 multiple times to generate multiple test programs covering different
aspects of SQLite's operation, then linking all test programs against
a common SQLite library and running them separately on the target platform.
SQLite will be found to work if all test programs pass.</p>

<p>There are no arbitrary limits in TH3.  One could generate a
single test program that contained all test modules and all configuration files.
However, such a test program might be too large to deploy on embedded
platforms.  Hence, TH3 provides the ability to break the library of test
modules up into smaller, more easily digested pieces.</p>

<p>Each individual test module might contain dozens, hundreds, or thousands
of separate tests.  The test modules can be written in C or as scripts of
SQL or a mixture of the two.  About two-thirds of the existing test modules are
................................................................................
<p>Once the test program is generated, it is run with no arguments to
perform the tests.  Progress information as well as error diagnostics
appear on standard output.  (Alternative output arrangements can be made
using a compile-time option for embedded devices that lack a standard
output channel.) The program returns zero if there are no
errors and non-zero if any problems were detected.</p>

<p>Typical output from a single TH3 test program run looks like this:

<blockquote><pre>
With SQLite 3.8.11 2015-05-15 04:13:15 56ef98a04765c34c1c2f3ed7a6f03a732f3b886e
-DSQLITE_COVERAGE_TEST
-DSQLITE_NO_SYNC
-DSQLITE_SYSTEM_MALLOC
-DSQLITE_THREADSAFE=1
Config-begin c1.
Begin c1.pager08
End c1.pager08
Begin c1.build33
End c1.build33
Begin c1.orderby01
End c1.orderby01
<i>... 15014 lines of output omitted ....</i>
Begin 64k.syscall01
End 64k.syscall01
Begin 64k.build01
End 64k.build01
Begin 64k.auth01
End 64k.auth01
Config-end 64k. TH3 memory used: 6373738
Config-begin wal1.
Begin wal1.wal37
End wal1.wal37
Config-end wal1. TH3 memory used: 100961
All 226 VDBE coverage points reached
th3: 0 errors out of 1442264 tests in 213.741 seconds. 64-bit little-endian
th3: SQLite 3.8.11 2015-05-15 04:13:15 56ef98a04765c34c1c2f3ed7a6f03a732f3b886e
</pre></blockquote>

<p>The output begins with a report of the [SQLITE_SOURCE_ID]
(cross-checked again [sqlite3_sourceid()]) for the
SQLite under test and the compile-time options used as reported
by [sqlite3_compileoption_get()].  The output concludes with a summary
of the test results and a repeat of the [SQLITE_SOURCE_ID].  If any
errors are detected, additional lines detail the problem.  The error
reporting lines always begin with a single space character so that they
can be quickly extracted from large output files using:

<blockquote><pre>
grep "&#94; "
</pre></blockquote>

<p>The default output shows the beginning and end of each configuration
and test module combination.  In the example above "c1" and "64k" are
configurations and "pager08", "build33", "orderby01", etc. are test modules.
Compile-time and run-time options are available to increase or decrease
the amount of output.
The output can be increased by showing each test case within each
test module.  The output can be decreased
by degrees: omitting test modules starts and stops,
omitting configuration starts and stops, and finally by omitting all output.
<h3>3.1 Test Automation Scripts</h3>

<p>TH3 comes with additional TCL scripts that help automate the testing





process on workstations.  The "th3make" script automatically runs "mkth3.tcl"
and "gcc" and then runs the resulting test program and checks the results.
Arguments to th3make include all of the "*.test" test modules and 
"*.cfg" configurations that are to be included in the test.  Additional
options to th3make can cause the test program to be compiled using different
compilers (GCC, Clang, MSVC), to use different output verbosity levels, to
run the test program under valgrind, to check the output for coverage using
gcov, and so forth.  The th3make script also accepts "*.rc" filenames as
arguments.  These *.rc files are just collections of other arguments that
are commonly used together for a single purpose.  For example, the "quick.rc"
file contains a set of eight arguments to th3make that run a fast (3-minute)
full-coverage test.  This allows the operator to type "./th3make quick.rc" as
a short-cut to typing out all of the required command-line options.  The
following are a few of the more than 40 available *.rc files:

<ul>
<li><b>alignment</b><i>N</i><b>.rc</b> - 
    These files contain -D options to the compiler that are used by
    various notable downstreams.
<li><b>cov.rc</b> - Options for measuring test coverage
<li><b>extensions.rc</b> - Options to enable [FTS4], [R-Trees],
    and [SQLITE_STAT4|STAT4].
<li><b>fast.rc</b> - Run mosts tests, including those not needed for
    coverage, skipping only soak tests, using delivery compiler options
    (ex: -O3)
<li><b>memdebug.rc</b> - like test.rc but also enable
    [SQLITE_MEMDEBUG|-DSQLITE_MEMDEBUG].
<li><b>min.rc</b> - Run the minimum set of tests needed for 100% coverage.
<li><b>quick.rc</b> - Run all tests required for 100% coverage tests
    using -Os and [SQLITE_DEBUG|-DSQLITE_DEBUG].
<li><b>test.rc</b> - Run the same tests as in fast.rc but without
    compiler optimization and enabling options like
    [SQLITE_DEBUG|-DSQLITE_DEBUG] and
    -DSQLITE_ENABLE_EXPENSIVE_ASSERT.
<li><b>test-ex.rc</b> - long-running soak tests.
</ul>

<p>The TH3 repository also includes the "multitest.tcl" script, another
TCL script used to automate TH3 testing on workstations.  Multitest.tcl
automatically compiles SQLite, then
runs ./th3make repeatedly with a variety of alignments, and captures
the output in a succinct summary screen.  A typical multitest.tcl run
generates output that looks like this:

<blockquote><pre>
file mkdir sqlite3bld
cd sqlite3bld
exec sh /home/drh/sqlite/sqlite/configure
file copy -force config.h ../config.h
exec make clean sqlite3.c
file rename sqlite3.c ../sqlite3.c
aa4f0f90c9c77424943e026a2ecee4a6c7f9e0d3  ../sqlite3.c
file rename sqlite3.h ../sqlite3.h
exec make clean sqlite3.c OPTS=-DSQLITE_ENABLE_UPDATE_DELETE_LIMIT=1
file rename sqlite3.c ../sqlite3udl.c
0d3bbc92c433f940253bb2c7c19de7783133929d  ../sqlite3udl.c
exec make clean sqlite3.c OPTS=-DSQLITE_SMALL_STACK=1
file rename sqlite3.c ../sqlite3ss.c
fcf6963e94096324461076d3b9e9dc1888e066e1  ../sqlite3ss.c
cd ..
*******************************************************************************
t01: quick.rc.................................................. Ok   (00:04:00)
t02: cov.rc.................................................... Ok   (00:04:40)
t03: quick.rc extensions.rc -D_HAVE_SQLITE_CONFIG_H............ Ok   (00:05:22)
t04: cov.rc -DSQLITE_ENABLE_STAT4 -D_HAVE_SQLITE_CONFIG_H...... Ok   (00:05:20)
t05: test.rc ../th3private/*.test.............................. Ok   (00:00:17)
t06: test.rc ../th3private/*.test -DSQLITE_ENABLE_STAT4........ Ok   (00:00:43)
t07: quick.rc -DSQLITE_TEST_REALLOC_STRESS -funsigned-char..... Ok   (00:04:56)
t08: quick.rc -DSQLITE_THREADSAFE=0 -fsigned-char.............. Ok   (00:03:12)
t09: quick.rc sqlite3ss.c -DSQLITE_MAX_ATTACHED=125............ Ok   (00:04:04)
t10: quick.rc -DSQLITE_RUNTIME_BYTEORDER....................... Ok   (00:03:58)
t11: quick.rc -DSQLITE_DIRECT_OVERFLOW_READ.................... Ok   (00:04:01)
t12: fast.rc................................................... Ok   (00:14:19)
t13: fast.rc alignment1.rc -m32................................ Ok   (00:20:51)
t14: fast.rc alignment2.rc sqlite3udl.c........................ Ok   (00:16:06)
t15: fast.rc alignment4.rc..................................... Ok   (00:12:55)
t16: fast.rc alignment5.rc..................................... Ok   (00:14:58)
t17: fast.rc alignment6.rc..................................... Ok   (00:14:31)
t18: fast.rc alignment7.rc..................................... Ok   (00:16:06)
t19: fast.rc alignment8.rc sqlite3udl.c........................ Ok   (00:24:09)
t20: test.rc alignment1.rc..................................... Ok   (00:49:27)
t21: test.rc alignment2.rc sqlite3udl.c........................ Ok   (00:38:43)
t22: test.rc alignment4.rc -m32 CC=clang....................... Ok   (00:39:49)
t23: test.rc alignment5.rc..................................... Ok   (00:36:33)
t24: test.rc alignment6.rc..................................... Ok   (00:33:53)
t25: test.rc alignment7.rc..................................... Ok   (00:42:16)
t26: test.rc alignment8.rc sqlite3udl.c........................ Ok   (01:05:22)
t27: memdebug.rc extensions.rc................................. Ok   (01:35:56)
t28: fast.rc -fsanitize=undefined.............................. Ok   (00:15:09)
t29: min.rc -O3 -valgrind...................................... Ok   (01:26:10)
t30: min.rc -O3 -valgrind extensions.rc........................ Ok   (01:47:12)
t31: test-ex.rc................................................ Ok   (03:20:18)
*******************************************************************************
0 failures on 31 th3make runs and 166721387 tests in (16:25:29)
SQLite 3.8.10 2015-05-05 18:52:54 04afa3febee32854fbb09ef8d4ffffd432119716
</pre></blockquote>

<p>As can be seen above, a single run
of multitest.tcl invokes th3make dozens times and takes between 12 and 24
hours.  The middle section of the output shows the arguments to each individual
th3make run and the result and elapse time for that th3make.
All build products and output for the separate th3make runs are
captures in subdirectories for post-test analysis.
The two-line summary at the bottom shows the total number of errors and tests
over all th3make runs and the total elapse time, together with the 
[SQLITE_SOURCE_ID] information for the version of SQLite that was
tested.  This summary information is recorded in the
<a href="https://www.sqlite.org/checklists/3081000/index#c6">release
checklist</a> during testing.

<h2>4.0 Test Coverage</h2>

<p>Using one particular subset of the available TH3 test modules (the "cov1"
tests) SQLite obtained 
[test coverage | 100% branch test coverage] and 100% [MC/DC] as measured
by [http://gcc.gnu.org/onlinedocs/gcc/Gcov.html | gcov]