Documentation Source Text

Check-in [15837d1afc]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Update the testing.html document and the release date of 3.6.23.
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 15837d1afc802724beeaadbf0a53b12dcebbbaf2
User & Date: drh 2010-03-09 19:35:32
Context
2010-03-10
23:17
Fix typo in change log for 3.6.23. check-in: 128ffc00f3 user: drh tags: trunk
2010-03-09
19:35
Update the testing.html document and the release date of 3.6.23. check-in: 15837d1afc user: drh tags: trunk
02:22
Move the proposed release date for 3.6.23 to 2010-03-10. check-in: 612e4dfe61 user: drh tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to pages/changes.in.

38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
      http://www.sqlite.org/src/timeline</a>.</p>
    }
    hd_close_aux
    hd_enable_main 1
  }
}

chng {2010 March 10 (3.6.23)} {
<li> Added the [secure_delete pragma].
<li> Added the [sqlite3_compileoption_used()] and
     [sqlite3_compileoption_get()] interfaces as well as the
     [compile_options pragma] and the [sqlite_compileoption_used()] and
     [sqlite_compileoption_get()] SQL functions.
<li> Added the [sqlite3_log()] interface together with the
     [SQLITE_CONFIG_LOG] verb to [sqlite3_config()].  The ".log" command







|







38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
      http://www.sqlite.org/src/timeline</a>.</p>
    }
    hd_close_aux
    hd_enable_main 1
  }
}

chng {2010 March 09 (3.6.23)} {
<li> Added the [secure_delete pragma].
<li> Added the [sqlite3_compileoption_used()] and
     [sqlite3_compileoption_get()] interfaces as well as the
     [compile_options pragma] and the [sqlite_compileoption_used()] and
     [sqlite_compileoption_get()] SQL functions.
<li> Added the [sqlite3_log()] interface together with the
     [SQLITE_CONFIG_LOG] verb to [sqlite3_config()].  The ".log" command

Changes to pages/testing.in.

7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

46


47
48
49
50
51
52
53
..
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
...
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175



176
177
178
179
180
181
182
183
...
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
# facilitate that, all the size values are defined by variables here
# which are then used as needed through the document.
#
# NOTE:  Also update the version number in the text!!!
#

# sloc sqlite3.c
set stat(coreSLOC)    65745  ;# Non-comment lines of amalgamation code 
# sloc test*.c
set stat(tclcSLOC)    14572  ;# Non-comment lines of test C code
# ls test*.c tclsqlite.c | wc
set stat(tclcNfile)      32  ;# Number of files of TCL C testcode + tclsqlite.c
# ls -l test*.c tclsqlite.c | awk '{sum+=$5}END{print sum}'
set stat(tclcNByte)  700287  ;# Number of bytes of TCL C testcode + tclsqlite.c
# sloc *.test *.tcl
set stat(tclsSLOC)   194071  ;# Non-comment lines of TCL test script
# ls *.test *.tcl | wc
set stat(tclsNFile)     511  ;# Number of files of TCL test script
# ls -l *.test *.tcl | awk '{sum+=$5}END{print sum}'
set stat(tclsNByte) 8298364  ;# Number of bytes of TCL test script
# grep do_test *.test | wc
set stat(tclNTest)    24866  ;# Number of test cases in the TCL test suite
set stat(tclNEval)  2151154  ;# Number of test case evaluations
set stat(nSqlFuzz)   107457  ;# Number of SQL fuzz tests
set stat(vqNEval)     42102  ;# Number of test evaluations for veryquick.test
set stat(vqStmtCov)   97.23  ;# veryquick statement coverage
set stat(vqBrCov)     92.57  ;# veryquick branch coverage
set stat(allStmtCov)  99.50  ;# all.test statement coverage
set stat(allBrCov)    97.41  ;# all.test condition/decision coverage
# tclsh mkth3.tcl cfg/*.cfg */*.test >th3.c; sloc th3.c
set stat(th3SLOC)    340773  ;# Non-comment lines in full th3.c
# ls -l th3.c
set stat(th3NByte) 23689140  ;# Number of bytes in full th3.c
# grep th3testBegin */*.test
# grep th3oomBegin */*.test
# grep th3ioerrBegin */*.test
# grep '^--testcase' */*.test
set stat(th3NTest)    12287  ;# Number of test cases
# from output of a full test run.

set stat(th3NEval)  1355348  ;# Number of test case evaluations


set stat(th3StmtCov) 100.00  ;# TH3 statement coverage
set stat(th3BrCov)   100.00  ;# TH3 branch coverage
# wc `fossil ls | awk '/\.test$/{print $2}'`
set stat(sltsSLOC)  44858977 ;# Non-comment lines of SLT test script
# ls -l `fossil ls | awk '/\.test$/{print $2}'` | awk '{sum+=$5}END{print sum}'
set stat(sltsNByte) 1116748159 ;# Bytes of SLT test script
# fossil ls | awk '/.test$/{print $2}' | wc
................................................................................
<h1 align="center">How SQLite Is Tested</h1>

<h2>1.0 Introduction</h2>

<p>The reliability and robustness of SQLite is achieved in part
by thorough and careful testing.</p>

<p>As of [version 3.6.19] (all statistics in the report are against that
release of SQLite),
the SQLite library consists of approximately
<tcl>KB {$stat(coreSLOC)}</tcl> KSLOC of C code.
(KSLOC means thousands of "Source Lines Of Code" or, in other words,
lines of code excluding blank lines and comments.)
By comparison, the project has
<tcl>
................................................................................
from the others.
</p>

<ol>
<li><p>
<tcl>hd_fragment tcl {TCL test suite}</tcl>
The <b>TCL Tests</b> are the oldest set of tests for SQLite.  
The TCL tests are contained in the same source tree as the
SQLite core and like the SQLite core are in the public domain.  The
TCL tests are the primary tests used during development.
The TCL tests are written using the 
[http://www.tcl.tk/ | TCL scripting language].
The TCL test harness itself consists of <tcl>KB {$stat(tclcSLOC)}</tcl> KSLOC 
of C code used to create the TCL interface.  The test scripts are contained
in <tcl>N {$stat(tclsNFile)}</tcl> files totaling 
<tcl>MiB {$stat(tclsNByte)}</tcl>MB in size.  There are
<tcl>N {$stat(tclNTest)}</tcl> distinct test cases, but many of the test
cases are parameterized and run multiple times (with different parameters)
so that on a full test run, about <tcl>MB {$stat(tclNEval)}</tcl> million 
separate tests are performed and provide 
<tcl>hd_puts $stat(allStmtCov)</tcl>% statement and
<tcl>hd_puts $stat(allBrCov)</tcl>% branch test coverage.
</p>
</li>

<li><p>
The <b>[TH3]</b> test harness is a set of proprietary tests, written in
C that provide 100% branch test coverage (and 100% MC/DC test coverage) to
the core SQLite library.  The TH3 tests are designed to run
on embedded and specialized platforms that would not easily support
TCL or other workstation services.  TH3 tests use only the published 
SQLite interfaces.  TH3 is free to [SQLite Consortium] members 
and is available by license to others.  TH3 consists of about
<tcl>MB {$stat(th3NByte)}</tcl> MB or <tcl>KB {$stat(th3SLOC)}</tcl> KSLOC
of C code implementing <tcl>N {$stat(th3NTest)}</tcl> distinct test cases.
TH3 tests are heavily parameterized, though, so a full test runs
about <tcl>MB {$stat(th3NEval)}</tcl> million different test



instances.  Additional information on TH3 is
[TH3 | available separately].</p></li>

<li><p>
<tcl>hd_fragment slt {SLT} {SQL Logic Tests}</tcl>
The <a href="http://www.sqlite.org/sqllogictest"><b>SQL Logic Test</b></a>
or SLT test harness is used to run huge numbers
of SQL statements against both SQLite and several other SQL database engines
................................................................................
<p>All of the tests above must run successfully, on multiple platforms
and under multiple compile-time configurations,
before each release of SQLite.</p>

<p>Prior to each check-in to the SQLite source tree, developers
typically run a subset (called "veryquick") of the Tcl tests
consisting of about 
<tcl>KB {$stat(vqNEval)}</tcl> thousand test cases
and covering
<tcl>hd_puts $stat(vqStmtCov)</tcl>% of the core SQLite source code statements.
The veryquick tests covers everything except the anomaly, fuzz, and 
soak tests.  The idea behind the veryquick tests are that they are
sufficient to catch most errors, but also run in only a few minutes
instead of a few hours.</p>

<tcl>hd_fragment anomoly</tcl>
<h2>3.0 Anomaly Testing</h2>








|

|



|

|

|

|

|
|

|
|
|
|
|

|

|




|

>
|
>
>







 







|







 







|











|
<
<













|
|
>
>
>
|







 







|
<
<
|







7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
..
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
...
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161


162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
...
195
196
197
198
199
200
201
202


203
204
205
206
207
208
209
210
# facilitate that, all the size values are defined by variables here
# which are then used as needed through the document.
#
# NOTE:  Also update the version number in the text!!!
#

# sloc sqlite3.c
set stat(coreSLOC)    67215  ;# Non-comment lines of amalgamation code 
# sloc test*.c
set stat(tclcSLOC)    14691  ;# Non-comment lines of test C code
# ls test*.c tclsqlite.c | wc
set stat(tclcNfile)      32  ;# Number of files of TCL C testcode + tclsqlite.c
# ls -l test*.c tclsqlite.c | awk '{sum+=$5}END{print sum}'
set stat(tclcNByte)  714533  ;# Number of bytes of TCL C testcode + tclsqlite.c
# sloc *.test *.tcl
set stat(tclsSLOC)   200444  ;# Non-comment lines of TCL test script
# ls *.test *.tcl | wc
set stat(tclsNFile)     536  ;# Number of files of TCL test script
# ls -l *.test *.tcl | awk '{sum+=$5}END{print sum}'
set stat(tclsNByte) 8780683  ;# Number of bytes of TCL test script
# grep do_test *.test | wc
set stat(tclNTest)    25400  ;# Number of test cases in the TCL test suite
set stat(tclNEval)  2204181  ;# Number of test case evaluations
set stat(nSqlFuzz)   107457  ;# Number of SQL fuzz tests
set stat(vqNEval)     98214  ;# Number of test evaluations for veryquick.test
#  set stat(vqStmtCov)   97.23  ;# veryquick statement coverage
#  set stat(vqBrCov)     92.57  ;# veryquick branch coverage
#  set stat(allStmtCov)  99.50  ;# all.test statement coverage
#  set stat(allBrCov)    97.41  ;# all.test condition/decision coverage
# tclsh mkth3.tcl cfg/*.cfg */*.test >th3.c; sloc th3.c
set stat(th3SLOC)    602888  ;# Non-comment lines in full th3.c
# ls -l th3.c
set stat(th3NByte) 45648069  ;# Number of bytes in full th3.c
# grep th3testBegin */*.test
# grep th3oomBegin */*.test
# grep th3ioerrBegin */*.test
# grep '^--testcase' */*.test
set stat(th3NTest)    29644  ;# Number of test cases
# from output of a full test run.
set stat(th3NECov)   1502643  ;# Number of test case evals for coverage
set stat(th3NETest)  1504866  ;# Number of test case evaluations
set stat(th3NEExt) 589175483  ;# Number of test case evals extended
set stat(th3NERel) 2500000000 ;# Number of test case evals release
set stat(th3StmtCov) 100.00  ;# TH3 statement coverage
set stat(th3BrCov)   100.00  ;# TH3 branch coverage
# wc `fossil ls | awk '/\.test$/{print $2}'`
set stat(sltsSLOC)  44858977 ;# Non-comment lines of SLT test script
# ls -l `fossil ls | awk '/\.test$/{print $2}'` | awk '{sum+=$5}END{print sum}'
set stat(sltsNByte) 1116748159 ;# Bytes of SLT test script
# fossil ls | awk '/.test$/{print $2}' | wc
................................................................................
<h1 align="center">How SQLite Is Tested</h1>

<h2>1.0 Introduction</h2>

<p>The reliability and robustness of SQLite is achieved in part
by thorough and careful testing.</p>

<p>As of [version 3.6.23] (all statistics in the report are against that
release of SQLite),
the SQLite library consists of approximately
<tcl>KB {$stat(coreSLOC)}</tcl> KSLOC of C code.
(KSLOC means thousands of "Source Lines Of Code" or, in other words,
lines of code excluding blank lines and comments.)
By comparison, the project has
<tcl>
................................................................................
from the others.
</p>

<ol>
<li><p>
<tcl>hd_fragment tcl {TCL test suite}</tcl>
The <b>TCL Tests</b> are the oldest set of tests for SQLite.  
They are contained in the same source tree as the
SQLite core and like the SQLite core are in the public domain.  The
TCL tests are the primary tests used during development.
The TCL tests are written using the 
[http://www.tcl.tk/ | TCL scripting language].
The TCL test harness itself consists of <tcl>KB {$stat(tclcSLOC)}</tcl> KSLOC 
of C code used to create the TCL interface.  The test scripts are contained
in <tcl>N {$stat(tclsNFile)}</tcl> files totaling 
<tcl>MiB {$stat(tclsNByte)}</tcl>MB in size.  There are
<tcl>N {$stat(tclNTest)}</tcl> distinct test cases, but many of the test
cases are parameterized and run multiple times (with different parameters)
so that on a full test run, about <tcl>MB {$stat(tclNEval)}</tcl> million 
separate tests are performed.


</p>
</li>

<li><p>
The <b>[TH3]</b> test harness is a set of proprietary tests, written in
C that provide 100% branch test coverage (and 100% MC/DC test coverage) to
the core SQLite library.  The TH3 tests are designed to run
on embedded and specialized platforms that would not easily support
TCL or other workstation services.  TH3 tests use only the published 
SQLite interfaces.  TH3 is free to [SQLite Consortium] members 
and is available by license to others.  TH3 consists of about
<tcl>MB {$stat(th3NByte)}</tcl> MB or <tcl>KB {$stat(th3SLOC)}</tcl> KSLOC
of C code implementing <tcl>N {$stat(th3NTest)}</tcl> distinct test cases.
TH3 tests are heavily parameterized, though, so a full-coverage test runs
about <tcl>MB {$stat(th3NECov)}</tcl> million different test
instances.  The cases that provide 100% branch test coverage constitute
a subset of the total TH3 test suite.  A soak test
prior to release does in excess of <tcl>GB {$stat(th3NERel)}</tcl> billion
tests.  Additional information on TH3 is
[TH3 | available separately].</p></li>

<li><p>
<tcl>hd_fragment slt {SLT} {SQL Logic Tests}</tcl>
The <a href="http://www.sqlite.org/sqllogictest"><b>SQL Logic Test</b></a>
or SLT test harness is used to run huge numbers
of SQL statements against both SQLite and several other SQL database engines
................................................................................
<p>All of the tests above must run successfully, on multiple platforms
and under multiple compile-time configurations,
before each release of SQLite.</p>

<p>Prior to each check-in to the SQLite source tree, developers
typically run a subset (called "veryquick") of the Tcl tests
consisting of about 
<tcl>KB {$stat(vqNEval)}</tcl> thousand test cases.


The veryquick tests include everything except the anomaly, fuzz, and 
soak tests.  The idea behind the veryquick tests are that they are
sufficient to catch most errors, but also run in only a few minutes
instead of a few hours.</p>

<tcl>hd_fragment anomoly</tcl>
<h2>3.0 Anomaly Testing</h2>