Documentation Source Text

<title>How SQLite Is Tested</title>

<tcl>
# This document contains many size statistics about SQLite, statistics
# that change frequently.  We want the document to be up-to-date.  To
# facilitate that, all the size values are defined by variables here
# which are then used as needed through the document.
#
# NOTE:  Also update the version number in the text!!!
#

set stat(coreSLOC)    62210  ;# Non-comment lines of amalgamation code 
set stat(tclcSLOC)    15710  ;# Non-comment lines of test C code
set stat(tclcNfile)      31  ;# Number of files of TCL C testcode + tclsqlite.c
set stat(tclcNByte)  747687  ;# Number of bytes of TCL C testcode + tclsqlite.c
set stat(tclsSLOC)   185697  ;# Non-comment lines of TCL test script
set stat(tclsNFile)     456  ;# Number of files of TCL test script
set stat(tclsNByte) 7924321  ;# Number of bytes of TCL test script
set stat(tclNTest)    23813  ;# Number of test cases in the TCL test suite
set stat(tclNEval)  1052056  ;# Number of test case evaluations
set stat(nSqlFuzz)   105234  ;# Number of SQL fuzz tests
set stat(vqNEval)     40637  ;# Number of test evaluations for veryquick.test
set stat(vqStmtCov)   96.94  ;# veryquick statement coverage
set stat(vqBrCov)     89.80  ;# veryquick branch coverage
set stat(allStmtCov)  99.37  ;# all.test statement coverage
set stat(allBrCov)    95.16  ;# all.test condition/decision coverage
set stat(th3SLOC)     34050  ;# Non-comment lines of TH3 script + mkth3.tcl
set stat(th3NByte)  2398737  ;# Number of bytes of TH3 test script + mkth3.tcl
set stat(th3NTest)     7014  ;# Number of test cases
set stat(th3NEval)  2251379  ;# Number of test case evaluations
set stat(sltsSLOC)    44332060 ;# Non-comment lines of SLT test script
set stat(sltsNByte) 1098986501 ;# Bytes of SLT test script
set stat(sltsNFile)        594 ;# Files of SLT test script
set stat(sltcSLOC)        1124 ;# Non-comment lines of SLT C code
set stat(sltNTest)     5830161 ;# Number of test cases in SLT
set stat(nAssert)         2442 ;# Number of assert statements


set stat(totalSLOC) [expr {$stat(tclcSLOC)+$stat(tclsSLOC)+
                           $stat(th3SLOC)+$stat(sltcSLOC)+$stat(sltsSLOC)}]

proc GB {expr} {
  set n [uplevel #0 expr $expr]

<h1 align="center">How SQLite Is Tested</h1>

<h2>1.0 Introduction</h2>

<p>The reliability and robustness of SQLite is achieved in large part
by thorough and careful testing.</p>

<p>As of [version 3.6.11] (all statistics in the report are against that
release of SQLite),
the SQLite library consists of approximately
<tcl>KB {$stat(coreSLOC)}</tcl> KSLOC of C code.
(KSLOC means thousands of "Source Lines Of Code" or, in other words,
lines of code excluding blank lines and comments.)
By comparison, the project has
<tcl>

introduced database corruption.</p>

<h3>3.2 Crash Testing</h3>

<p>Crash testing seeks to demonstrate that an SQLite database will not
go corrupt if the application or operating system crashes or if there
is a power failure in the middle of a database update.  A separate
white-paper titled
<a href="atomiccommit.html">Atomic Commit in SQLite</a> describes the
defensive measure SQLite takes to prevent database corruption following
a crash.  Crash tests strive to verify that those defensive measures
are working correctly.</p>

<p>It is impractical to do crash testing using real power failures, of
course, and so crash testing is done in simulation.  An alternative

harnesses are zealous to enforce this.</p>

<h2>7.0 Test Coverage</h2>

<p>The <a href="http://gcc.gnu.org/onlinedocs/gcc/Gcov.html">gcov</a>
utility is used to measure the "test coverage" of the SQLite test suite.
SQLite strives for but does not yet obtain 100% test coverage.  A major
goal of the SQLite project is to obtain 100% branch coverage
during 2009.</P.

<p>Test coverage can be measured in several ways.  "Statement coverage"
measures (as a percentage of the whole) how many lines of code are
exercised by the test cases. The TCL test suite obtains
<tcl>hd_puts $stat(allStmtCov)</tcl>% statement coverage on the SQLite
core.  (The SQLite core, in this case, excludes the operating-system
dependent [sqlite3_vfs | VFS] backends.)
"Branch" coverage measures (again, as a percentage of the
whole) how many machine-code branch instructions are taken at least
once in both directions.  The TCL test suite obtains
<tcl>hd_puts $stat(allBrCov)</tcl>% branch coverage.</p>

<p>To illustrate the difference between statement coverage and
branch coverage, consider the following hypothetical
line of C code:</p>

<blockquote><pre>
if( a>b && c!=25 ){ d++; }
</pre></blockquote>

<p>Such a line of C code might generate a dozen separate machine code
instructions.  If any one of those instructions is ever evaluated, then
we say that the statement has been tested.  So, for example, it might
be the case that the conditional expression is
always false and the "d" variable is
never incremented.  Even so, statement coverage counts this line of
code as having been tested.</p>

<p>Branch coverage is more strict.  With branch coverage, each test and
each subblock within the statement is considered separately.  In order
to achieve 100% branch coverage in the example above, there must be at
least three test cases:</p>

<p><ul>
<li> a<=b
<li> a>b && c==25
<li> a>b && c!=25
</ul></p>

<p>Branch test coverage is normally less than statement coverage since
a C program will typically contain some defensive tests which in practice
are always true or always false.  
For testing purposes, the SQLite source code defines
macros called ALWAYS() and NEVER().   The ALWAYS() macro
surrounds conditions
which are expected to always evaluated to true and NEVER() surrounds
conditions that are always evaluate to false.  These macros serve as

#define ALWAYS(X)  ((X)?1:assert(0),0)
#define NEVER(X)   ((X)?assert(0),1:0)
</pre></blockquote>

<p>When measuring test coverage, these macros are defined to be constant
truth values so that they do not generate assembly language branch
instructions, and hence do not come into play when calculating the
branch coverage level:</p>

<blockquote><pre>
#define ALWAYS(X)  (1)
#define NEVER(X)   (0)
</pre></blockquote>

<p>Another macro used in conjuction with test coverage measurement is
the <tt>testcase()</tt> macro.  The argument is a condition for which
we want test cases that evaluate to both true and false.
In non-coverage builds (that is to so, in release builds) the testcase()
macro is a no-op:</p>

<blockquote><pre>
#define testcase(X)
</pre></blockquote>

<p>But in a coverage measuring build, the testcase() macro generates code
that evaluates the conditional expression in its argument.  
Then during analysis, a check
is made to insure tests exists that evaluate the conditional to both true
and false.  Testcase() macros are used, for example, to help verify that
boundary values are tested.  For example:</p>

<blockquote><pre>
testcase( a==b );
testcase( a==b+1 );