Documentation Source Text

Check-in [3b5097e255]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Tweaks to the testing documentation.
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 3b5097e2555ca1006e165176251bb6d99e3baab4
User & Date: drh 2009-08-26 13:13:57
Context
2009-08-31
13:47
Query planner documentation update. check-in: d8db9f9b5c user: drh tags: trunk
2009-08-26
13:13
Tweaks to the testing documentation. check-in: 3b5097e255 user: drh tags: trunk
02:00
Updates, clarifications, and typo fixes in the SQL language documentation. check-in: 7b2b285803 user: drh tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to pages/testing.in.

54
55
56
57
58
59
60


61
62
63
64
65
66
67
...
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
...
536
537
538
539
540
541
542



543
544
545
546
547
548
549
...
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
...
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
set stat(sltsNFile)        610 ;# Files of SLT test script
# sloc md5.c slt_*.c sqllogictest.c
set stat(sltcSLOC)        1307 ;# Non-comment lines of SLT C code
# grep '^query' `fossil ls | awk '/\.test$/{print $2}'` | wc
set stat(sltNTest)     7195024 ;# Number of test cases in SLT
# grep 'assert(' sqlite3.c | wc
set stat(nAssert)         2717 ;# Number of assert statements



set stat(totalSLOC) [expr {$stat(tclcSLOC)+$stat(tclsSLOC)+
                           $stat(th3SLOC)+$stat(sltcSLOC)+$stat(sltsSLOC)}]

proc GB {expr} {
  set n [uplevel #0 expr $expr]
  hd_puts [format %.2f [expr {$n/(1000.0*1000.0*1000.0)}]]
................................................................................
not possible to write cross-platform tests for those modules.  Extensions
such as FTS3 and RTree are also excluded from the analysis.</p>

<h3>7.1 Statement versus branch coverage</h3>

<p>There are many ways to measure test coverage.  The most popular
metric is "statement coverage".  When you hear someone say that their
program as "XX% test coverage" without further qualification, they usually
mean statement coverage.  Statement coverage measures what percentage
of lines of code are executed at least once by the test suite.</p>

<p>Branch coverage is more rigorous than statement coverage.  Branch
coverage measure the number of machine-code branch instructions that
are evaluated at least once on both directions.</p>

................................................................................
macros that precede the if statement verify that both cases are tested:</p>

<blockquote><pre>
testcase( mask & SQLITE_OPEN_MAIN_DB );
testcase( mask & SQLITE_OPEN_TEMP_DB );
if( (mask & (SQLITE_OPEN_MAIN_DB|SQLITE_OPEN_TEMP_DB))!=0 ){ ... }
</pre></blockquote>




<tcl>hd_fragment {mcdc} {MC/DC}</tcl>
<h3>7.4 Branch coverage versus MC/DC</h3>

<p>Two methods of measuring test coverage were described above:
"statement" and "branch" coverage.  There are many other test coverage
metrics besides these two.  Another popular metric is "Modified
................................................................................
<li> Each condition in a decision takes on every possible outcome.
<li> Each entry and exit point is invoked.
<li> Each condition in a decision is shown to independently
     affect the outcome of the decision.
</ul>

<p>In the C programming language 
where the <tt>&amp;&amp;</tt> and <tt>||</tt> operator
are "short-circuit" operators, MC/DC and branch coverage are very nearly
the same thing.  The primary difference is in boolean vector tests.
One can test for any of several bits in bit-vector and still obtain
100% branch test coverage even though the second element of MC/DC - the
requirement that each condition in a decision take on every possible outcome -
might not be satisfied.</p>

................................................................................
check for correctness.  Static analysis consists mostly of making
sure SQLite compiles without warnings, even when all warnings are
enabled.  SQLite is developed primarily using GCC and it does
compile without warnings on GCC using the -Wall and -Wextra flags.
There are occasional reports of warnings coming from VC++, however.</p>

<p>Static analysis has not proven to be helpful in finding
bugs.  We cannot call to mind a single problem in SQLite that
was detected by static analysis that was not first seen by one
of the other testing methods described above.  On the other hand,
we have on occasion introduced new bugs in our efforts to get SQLite
to compile without warnings.</p>

<p>Our experience, then, is that static analysis is counter-productive
to quality.  In other words, focusing on static analysis (being







>
>







 







|







 







>
>
>







 







|







 







|







54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
...
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
...
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
...
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
...
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
set stat(sltsNFile)        610 ;# Files of SLT test script
# sloc md5.c slt_*.c sqllogictest.c
set stat(sltcSLOC)        1307 ;# Non-comment lines of SLT C code
# grep '^query' `fossil ls | awk '/\.test$/{print $2}'` | wc
set stat(sltNTest)     7195024 ;# Number of test cases in SLT
# grep 'assert(' sqlite3.c | wc
set stat(nAssert)         2717 ;# Number of assert statements
# grep 'testcase(' sqlite3.c | grep -v define | wc
set stat(nTestcase)        535 ;# Number of testcase statements

set stat(totalSLOC) [expr {$stat(tclcSLOC)+$stat(tclsSLOC)+
                           $stat(th3SLOC)+$stat(sltcSLOC)+$stat(sltsSLOC)}]

proc GB {expr} {
  set n [uplevel #0 expr $expr]
  hd_puts [format %.2f [expr {$n/(1000.0*1000.0*1000.0)}]]
................................................................................
not possible to write cross-platform tests for those modules.  Extensions
such as FTS3 and RTree are also excluded from the analysis.</p>

<h3>7.1 Statement versus branch coverage</h3>

<p>There are many ways to measure test coverage.  The most popular
metric is "statement coverage".  When you hear someone say that their
program as "XX% test coverage" without further explanation, they usually
mean statement coverage.  Statement coverage measures what percentage
of lines of code are executed at least once by the test suite.</p>

<p>Branch coverage is more rigorous than statement coverage.  Branch
coverage measure the number of machine-code branch instructions that
are evaluated at least once on both directions.</p>

................................................................................
macros that precede the if statement verify that both cases are tested:</p>

<blockquote><pre>
testcase( mask & SQLITE_OPEN_MAIN_DB );
testcase( mask & SQLITE_OPEN_TEMP_DB );
if( (mask & (SQLITE_OPEN_MAIN_DB|SQLITE_OPEN_TEMP_DB))!=0 ){ ... }
</pre></blockquote>

<p>The SQLite source code contains <tcl>N {$stat(nTestcase)}</tcl>
uses of the testcase() macro.</p>

<tcl>hd_fragment {mcdc} {MC/DC}</tcl>
<h3>7.4 Branch coverage versus MC/DC</h3>

<p>Two methods of measuring test coverage were described above:
"statement" and "branch" coverage.  There are many other test coverage
metrics besides these two.  Another popular metric is "Modified
................................................................................
<li> Each condition in a decision takes on every possible outcome.
<li> Each entry and exit point is invoked.
<li> Each condition in a decision is shown to independently
     affect the outcome of the decision.
</ul>

<p>In the C programming language 
where <b><tt>&amp;&amp;</tt></b> and <b><tt>||</tt></b>
are "short-circuit" operators, MC/DC and branch coverage are very nearly
the same thing.  The primary difference is in boolean vector tests.
One can test for any of several bits in bit-vector and still obtain
100% branch test coverage even though the second element of MC/DC - the
requirement that each condition in a decision take on every possible outcome -
might not be satisfied.</p>

................................................................................
check for correctness.  Static analysis consists mostly of making
sure SQLite compiles without warnings, even when all warnings are
enabled.  SQLite is developed primarily using GCC and it does
compile without warnings on GCC using the -Wall and -Wextra flags.
There are occasional reports of warnings coming from VC++, however.</p>

<p>Static analysis has not proven to be helpful in finding
bugs in SQLite.  We cannot call to mind a single problem in SQLite that
was detected by static analysis that was not first seen by one
of the other testing methods described above.  On the other hand,
we have on occasion introduced new bugs in our efforts to get SQLite
to compile without warnings.</p>

<p>Our experience, then, is that static analysis is counter-productive
to quality.  In other words, focusing on static analysis (being