Documentation Source Text

set stat(sltsNFile)        610 ;# Files of SLT test script
# sloc md5.c slt_*.c sqllogictest.c
set stat(sltcSLOC)        1307 ;# Non-comment lines of SLT C code
# grep '^query' `fossil ls | awk '/\.test$/{print $2}'` | wc
set stat(sltNTest)     7195024 ;# Number of test cases in SLT
# grep 'assert(' sqlite3.c | wc
set stat(nAssert)         2717 ;# Number of assert statements
# grep 'testcase(' sqlite3.c | grep -v define | wc
set stat(nTestcase)        535 ;# Number of testcase statements

set stat(totalSLOC) [expr {$stat(tclcSLOC)+$stat(tclsSLOC)+
                           $stat(th3SLOC)+$stat(sltcSLOC)+$stat(sltsSLOC)}]

proc GB {expr} {
  set n [uplevel #0 expr $expr]
  hd_puts [format %.2f [expr {$n/(1000.0*1000.0*1000.0)}]]

not possible to write cross-platform tests for those modules.  Extensions
such as FTS3 and RTree are also excluded from the analysis.</p>

<h3>7.1 Statement versus branch coverage</h3>

<p>There are many ways to measure test coverage.  The most popular
metric is "statement coverage".  When you hear someone say that their
program as "XX% test coverage" without further explanation, they usually
mean statement coverage.  Statement coverage measures what percentage
of lines of code are executed at least once by the test suite.</p>

<p>Branch coverage is more rigorous than statement coverage.  Branch
coverage measure the number of machine-code branch instructions that
are evaluated at least once on both directions.</p>

macros that precede the if statement verify that both cases are tested:</p>

<blockquote><pre>
testcase( mask & SQLITE_OPEN_MAIN_DB );
testcase( mask & SQLITE_OPEN_TEMP_DB );
if( (mask & (SQLITE_OPEN_MAIN_DB|SQLITE_OPEN_TEMP_DB))!=0 ){ ... }
</pre></blockquote>

<p>The SQLite source code contains <tcl>N {$stat(nTestcase)}</tcl>
uses of the testcase() macro.</p>

<tcl>hd_fragment {mcdc} {MC/DC}</tcl>
<h3>7.4 Branch coverage versus MC/DC</h3>

<p>Two methods of measuring test coverage were described above:
"statement" and "branch" coverage.  There are many other test coverage
metrics besides these two.  Another popular metric is "Modified
Condition/Decision Coverage" or MC/DC.  
<a href="http://en.wikipedia.org/wiki/Modified_Condition/Decision_Coverage">
Wikipedia</a> defines MC/DC as follows:</p>

<ul>
<li> Each decision tries every possible outcome.
<li> Each condition in a decision takes on every possible outcome.
<li> Each entry and exit point is invoked.
<li> Each condition in a decision is shown to independently
     affect the outcome of the decision.
</ul>

<p>In the C programming language 
where <b><tt>&amp;&amp;</tt></b> and <b><tt>||</tt></b>
are "short-circuit" operators, MC/DC and branch coverage are very nearly
the same thing.  The primary difference is in boolean vector tests.
One can test for any of several bits in bit-vector and still obtain
100% branch test coverage even though the second element of MC/DC - the
requirement that each condition in a decision take on every possible outcome -
might not be satisfied.</p>

check for correctness.  Static analysis consists mostly of making
sure SQLite compiles without warnings, even when all warnings are
enabled.  SQLite is developed primarily using GCC and it does
compile without warnings on GCC using the -Wall and -Wextra flags.
There are occasional reports of warnings coming from VC++, however.</p>

<p>Static analysis has not proven to be helpful in finding
bugs in SQLite.  We cannot call to mind a single problem in SQLite that
was detected by static analysis that was not first seen by one
of the other testing methods described above.  On the other hand,
we have on occasion introduced new bugs in our efforts to get SQLite
to compile without warnings.</p>

<p>Our experience, then, is that static analysis is counter-productive
to quality.  In other words, focusing on static analysis (being