Documentation Source Text

Check-in [cd844328a2]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Update statistics in the testing.html document.
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: cd844328a29351592b126c316355c870b423b210
User & Date: drh 2009-02-17 18:10:11
Context
2009-02-18
18:36
Move C-API requirements text out of sqlite.h.in and into docsrc. check-in: cb8f091c8f user: drh tags: trunk
2009-02-17
18:10
Update statistics in the testing.html document. check-in: cd844328a2 user: drh tags: trunk
06:47
Remove unused label "loaddb_out" from example code in backup.in. check-in: 4801fe3258 user: dan tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to pages/testing.in.

5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
..
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
...
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
...
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374

375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
...
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
...
429
430
431
432
433
434
435

436
437
438
439
440
441
442
443
# that change frequently.  We want the document to be up-to-date.  To
# facilitate that, all the size values are defined by variables here
# which are then used as needed through the document.
#
# NOTE:  Also update the version number in the text!!!
#

set stat(coreSLOC)    61885  ;# Non-comment lines of amalgamation code 
set stat(tclcSLOC)    15489  ;# Non-comment lines of test C code
set stat(tclcNfile)      30  ;# Number of files of TCL C testcode + tclsqlite.c
set stat(tclcNByte)  735632  ;# Number of bytes of TCL C testcode + tclsqlite.c
set stat(tclsSLOC)   183913  ;# Non-comment lines of TCL test script
set stat(tclsNFile)     449  ;# Number of files of TCL test script
set stat(tclsNByte) 7856401  ;# Number of bytes of TCL test script
set stat(tclNTest)    23602  ;# Number of test cases in the TCL test suite
set stat(tclNEval)  1057281  ;# Number of test case evaluations
set stat(nSqlFuzz)    35034  ;# Number of SQL fuzz tests
set stat(vqNEval)     39609  ;# Number of test evaluations for veryquick.test
set stat(vqStmtCov)   95.85  ;# veryquick statement coverage

set stat(allStmtCov)  99.38  ;# all.test statement coverage
set stat(allCDCov)    94.97  ;# all.test condition/decision coverage
set stat(th3SLOC)    141815  ;# Non-comment lines of TH3 script + mkth3.tcl
set stat(th3NByte)  9626908  ;# Number of bytes of TH3 test script + mkth3.tcl
set stat(th3NTest)     2137  ;# Number of test cases
set stat(th3NEval)  8030189  ;# Number of test case evaluations
set stat(sltsSLOC)    23277414 ;# Non-comment lines of SLT test script
set stat(sltsNByte) 1015905899 ;# Bytes of SLT test script
set stat(sltsNFile)        546 ;# Files of SLT test script
set stat(sltcSLOC)        1293 ;# Non-comment lines of SLT C code
set stat(sltNTest)     5331384 ;# Number of test cases in SLT
set stat(nAssert)         2450 ;# Number of assert statements


set stat(totalSLOC) [expr {$stat(tclcSLOC)+$stat(tclsSLOC)+
                           $stat(th3SLOC)+$stat(sltcSLOC)+$stat(sltsSLOC)}]

proc GB {expr} {
  set n [uplevel #0 expr $expr]
................................................................................
<h1 align="center">How SQLite Is Tested</h1>

<h2>1.0 Introduction</h2>

<p>The reliability and robustness of SQLite is achieved in large part
by thorough and careful testing.</p>

<p>As of [version 3.6.10] (all statistics in the report are against that
release of SQLite),
the SQLite library consists of approximately
<tcl>KB {$stat(coreSLOC)}</tcl> KSLOC of C code.
(KSLOC means thousands of "Source Lines Of Code" or, in other words,
lines of code excluding blank lines and comments.)
By comparison, the project has
<tcl>
................................................................................
introduced database corruption.</p>

<h3>3.2 Crash Testing</h3>

<p>Crash testing seeks to demonstrate that an SQLite database will not
go corrupt if the application or operating system crashes or if there
is a power failure in the middle of a database update.  A separate
reported called
<a href="atomiccommit.html">Atomic Commit in SQLite</a> describes the
defensive measure SQLite takes to prevent database corruption following
a crash.  Crash tests strive to verify that those defensive measures
are working correctly.</p>

<p>It is impractical to do crash testing using real power failures, of
course, and so crash testing is done in simulation.  An alternative
................................................................................
harnesses are zealous to enforce this.</p>

<h2>7.0 Test Coverage</h2>

<p>The <a href="http://gcc.gnu.org/onlinedocs/gcc/Gcov.html">gcov</a>
utility is used to measure the "test coverage" of the SQLite test suite.
SQLite strives for but does not yet obtain 100% test coverage.  A major
goal of the SQLite project is to obtain 100% condition/decision coverage
during 2009.</P.

<p>Test coverage can be measured in several ways.  "Statement coverage"
measures (as a percentage of the whole) how many lines of code are
exercised by the test cases. The TCL test suite obtains
<tcl>hd_puts $stat(allStmtCov)</tcl>% statement coverage on the SQLite
core.  (The SQLite core, in this case, excludes the operating-system
dependent [sqlite3_vfs | VFS] backends.)
"Condition/Decision" or "C/D" coverage measures (again, as a percentage of the
whole) how many machine-code branch instructions are taken at least
once in both directions.  The TCL test suite obtains
<tcl>hd_puts $stat(allCDCov)</tcl>% C/D coverage.</p>

<p>To illustrate the difference between statement coverage and
condition/decision coverage, consider the following hypothetical
line of C code:</p>

<blockquote><pre>
if( a>b && c!=25 ){ d++; }
</pre></blockquote>

<p>Such a line of C code might generate a dozen separate machine code
instructions.  If any one of those instructions is ever evaluated, then
we say that the statement has been tested.  So, for example, it might

be the case that the condition is always false and the "d" variable is
never incremented.  Even so, statement coverage counts this line of
code as having been tested.</p>

<p>C/D coverage is more strict.  With C/D coverage, each test and
each subblock within the statement is considered separately.  In order
to achieve 100% C/D coverage in the example above, there must be at
least three test cases:</p>

<p><ul>
<li> a<=b
<li> a>b && c==25
<li> a>b && c!=25
</ul></p>

<p>C/D test coverage is normally less than statement coverage since
a C program will typically contain some defensive tests which in practice
are always true or always false.  
For testing purposes, the SQLite source code defines
macros called ALWAYS() and NEVER().   The ALWAYS() macro
surrounds conditions
which are expected to always evaluated to true and NEVER() surrounds
conditions that are always evaluate to false.  These macros serve as
................................................................................
#define ALWAYS(X)  ((X)?1:assert(0),0)
#define NEVER(X)   ((X)?assert(0),1:0)
</pre></blockquote>

<p>When measuring test coverage, these macros are defined to be constant
truth values so that they do not generate assembly language branch
instructions, and hence do not come into play when calculating the
C/D coverage level:</p>

<blockquote><pre>
#define ALWAYS(X)  (1)
#define NEVER(X)   (0)
</pre></blockquote>

<p>Another macro used in conjuction with test coverage measurement is
................................................................................
macro is a no-op:</p>

<blockquote><pre>
#define testcase(X)
</pre></blockquote>

<p>But in a coverage measuring build, the testcase() macro generates code

that evaluates the condition in its argument.  Then during analysis, a check
is made to insure tests exists that evaluate the conditional to both true
and false.  Testcase() macros are used, for example, to help verify that
boundary values are tested.  For example:</p>

<blockquote><pre>
testcase( a==b );
testcase( a==b+1 );







|
|
|
|
|
|
|
|
|
|
|
|
>
|
|
|
|
|
|
|
|
|
|
|
|







 







|







 







|







 







|








|


|


|









>
|



|

|








|







 







|







 







>
|







5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
..
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
...
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
...
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
...
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
...
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
# that change frequently.  We want the document to be up-to-date.  To
# facilitate that, all the size values are defined by variables here
# which are then used as needed through the document.
#
# NOTE:  Also update the version number in the text!!!
#

set stat(coreSLOC)    62210  ;# Non-comment lines of amalgamation code 
set stat(tclcSLOC)    15710  ;# Non-comment lines of test C code
set stat(tclcNfile)      31  ;# Number of files of TCL C testcode + tclsqlite.c
set stat(tclcNByte)  747687  ;# Number of bytes of TCL C testcode + tclsqlite.c
set stat(tclsSLOC)   185697  ;# Non-comment lines of TCL test script
set stat(tclsNFile)     456  ;# Number of files of TCL test script
set stat(tclsNByte) 7924321  ;# Number of bytes of TCL test script
set stat(tclNTest)    23813  ;# Number of test cases in the TCL test suite
set stat(tclNEval)  1052056  ;# Number of test case evaluations
set stat(nSqlFuzz)   105234  ;# Number of SQL fuzz tests
set stat(vqNEval)     40637  ;# Number of test evaluations for veryquick.test
set stat(vqStmtCov)   96.94  ;# veryquick statement coverage
set stat(vqBrCov)     89.80  ;# veryquick branch coverage
set stat(allStmtCov)  99.37  ;# all.test statement coverage
set stat(allBrCov)    95.16  ;# all.test condition/decision coverage
set stat(th3SLOC)     34050  ;# Non-comment lines of TH3 script + mkth3.tcl
set stat(th3NByte)  2398737  ;# Number of bytes of TH3 test script + mkth3.tcl
set stat(th3NTest)     7014  ;# Number of test cases
set stat(th3NEval)  2251379  ;# Number of test case evaluations
set stat(sltsSLOC)    44332060 ;# Non-comment lines of SLT test script
set stat(sltsNByte) 1098986501 ;# Bytes of SLT test script
set stat(sltsNFile)        594 ;# Files of SLT test script
set stat(sltcSLOC)        1124 ;# Non-comment lines of SLT C code
set stat(sltNTest)     5830161 ;# Number of test cases in SLT
set stat(nAssert)         2442 ;# Number of assert statements


set stat(totalSLOC) [expr {$stat(tclcSLOC)+$stat(tclsSLOC)+
                           $stat(th3SLOC)+$stat(sltcSLOC)+$stat(sltsSLOC)}]

proc GB {expr} {
  set n [uplevel #0 expr $expr]
................................................................................
<h1 align="center">How SQLite Is Tested</h1>

<h2>1.0 Introduction</h2>

<p>The reliability and robustness of SQLite is achieved in large part
by thorough and careful testing.</p>

<p>As of [version 3.6.11] (all statistics in the report are against that
release of SQLite),
the SQLite library consists of approximately
<tcl>KB {$stat(coreSLOC)}</tcl> KSLOC of C code.
(KSLOC means thousands of "Source Lines Of Code" or, in other words,
lines of code excluding blank lines and comments.)
By comparison, the project has
<tcl>
................................................................................
introduced database corruption.</p>

<h3>3.2 Crash Testing</h3>

<p>Crash testing seeks to demonstrate that an SQLite database will not
go corrupt if the application or operating system crashes or if there
is a power failure in the middle of a database update.  A separate
white-paper titled
<a href="atomiccommit.html">Atomic Commit in SQLite</a> describes the
defensive measure SQLite takes to prevent database corruption following
a crash.  Crash tests strive to verify that those defensive measures
are working correctly.</p>

<p>It is impractical to do crash testing using real power failures, of
course, and so crash testing is done in simulation.  An alternative
................................................................................
harnesses are zealous to enforce this.</p>

<h2>7.0 Test Coverage</h2>

<p>The <a href="http://gcc.gnu.org/onlinedocs/gcc/Gcov.html">gcov</a>
utility is used to measure the "test coverage" of the SQLite test suite.
SQLite strives for but does not yet obtain 100% test coverage.  A major
goal of the SQLite project is to obtain 100% branch coverage
during 2009.</P.

<p>Test coverage can be measured in several ways.  "Statement coverage"
measures (as a percentage of the whole) how many lines of code are
exercised by the test cases. The TCL test suite obtains
<tcl>hd_puts $stat(allStmtCov)</tcl>% statement coverage on the SQLite
core.  (The SQLite core, in this case, excludes the operating-system
dependent [sqlite3_vfs | VFS] backends.)
"Branch" coverage measures (again, as a percentage of the
whole) how many machine-code branch instructions are taken at least
once in both directions.  The TCL test suite obtains
<tcl>hd_puts $stat(allBrCov)</tcl>% branch coverage.</p>

<p>To illustrate the difference between statement coverage and
branch coverage, consider the following hypothetical
line of C code:</p>

<blockquote><pre>
if( a>b && c!=25 ){ d++; }
</pre></blockquote>

<p>Such a line of C code might generate a dozen separate machine code
instructions.  If any one of those instructions is ever evaluated, then
we say that the statement has been tested.  So, for example, it might
be the case that the conditional expression is
always false and the "d" variable is
never incremented.  Even so, statement coverage counts this line of
code as having been tested.</p>

<p>Branch coverage is more strict.  With branch coverage, each test and
each subblock within the statement is considered separately.  In order
to achieve 100% branch coverage in the example above, there must be at
least three test cases:</p>

<p><ul>
<li> a<=b
<li> a>b && c==25
<li> a>b && c!=25
</ul></p>

<p>Branch test coverage is normally less than statement coverage since
a C program will typically contain some defensive tests which in practice
are always true or always false.  
For testing purposes, the SQLite source code defines
macros called ALWAYS() and NEVER().   The ALWAYS() macro
surrounds conditions
which are expected to always evaluated to true and NEVER() surrounds
conditions that are always evaluate to false.  These macros serve as
................................................................................
#define ALWAYS(X)  ((X)?1:assert(0),0)
#define NEVER(X)   ((X)?assert(0),1:0)
</pre></blockquote>

<p>When measuring test coverage, these macros are defined to be constant
truth values so that they do not generate assembly language branch
instructions, and hence do not come into play when calculating the
branch coverage level:</p>

<blockquote><pre>
#define ALWAYS(X)  (1)
#define NEVER(X)   (0)
</pre></blockquote>

<p>Another macro used in conjuction with test coverage measurement is
................................................................................
macro is a no-op:</p>

<blockquote><pre>
#define testcase(X)
</pre></blockquote>

<p>But in a coverage measuring build, the testcase() macro generates code
that evaluates the conditional expression in its argument.  
Then during analysis, a check
is made to insure tests exists that evaluate the conditional to both true
and false.  Testcase() macros are used, for example, to help verify that
boundary values are tested.  For example:</p>

<blockquote><pre>
testcase( a==b );
testcase( a==b+1 );