Documentation Source Text

Check-in [3b5097e255]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Tweaks to the testing documentation.
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 3b5097e2555ca1006e165176251bb6d99e3baab4
User & Date: drh 2009-08-26 13:13:57
Context
2009-08-31
13:47
Query planner documentation update. check-in: d8db9f9b5c user: drh tags: trunk
2009-08-26
13:13
Tweaks to the testing documentation. check-in: 3b5097e255 user: drh tags: trunk
02:00
Updates, clarifications, and typo fixes in the SQL language documentation. check-in: 7b2b285803 user: drh tags: trunk
Changes
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Changes to pages/testing.in.

    54     54   set stat(sltsNFile)        610 ;# Files of SLT test script
    55     55   # sloc md5.c slt_*.c sqllogictest.c
    56     56   set stat(sltcSLOC)        1307 ;# Non-comment lines of SLT C code
    57     57   # grep '^query' `fossil ls | awk '/\.test$/{print $2}'` | wc
    58     58   set stat(sltNTest)     7195024 ;# Number of test cases in SLT
    59     59   # grep 'assert(' sqlite3.c | wc
    60     60   set stat(nAssert)         2717 ;# Number of assert statements
           61  +# grep 'testcase(' sqlite3.c | grep -v define | wc
           62  +set stat(nTestcase)        535 ;# Number of testcase statements
    61     63   
    62     64   set stat(totalSLOC) [expr {$stat(tclcSLOC)+$stat(tclsSLOC)+
    63     65                              $stat(th3SLOC)+$stat(sltcSLOC)+$stat(sltsSLOC)}]
    64     66   
    65     67   proc GB {expr} {
    66     68     set n [uplevel #0 expr $expr]
    67     69     hd_puts [format %.2f [expr {$n/(1000.0*1000.0*1000.0)}]]
................................................................................
   393    395   not possible to write cross-platform tests for those modules.  Extensions
   394    396   such as FTS3 and RTree are also excluded from the analysis.</p>
   395    397   
   396    398   <h3>7.1 Statement versus branch coverage</h3>
   397    399   
   398    400   <p>There are many ways to measure test coverage.  The most popular
   399    401   metric is "statement coverage".  When you hear someone say that their
   400         -program as "XX% test coverage" without further qualification, they usually
          402  +program as "XX% test coverage" without further explanation, they usually
   401    403   mean statement coverage.  Statement coverage measures what percentage
   402    404   of lines of code are executed at least once by the test suite.</p>
   403    405   
   404    406   <p>Branch coverage is more rigorous than statement coverage.  Branch
   405    407   coverage measure the number of machine-code branch instructions that
   406    408   are evaluated at least once on both directions.</p>
   407    409   
................................................................................
   536    538   macros that precede the if statement verify that both cases are tested:</p>
   537    539   
   538    540   <blockquote><pre>
   539    541   testcase( mask & SQLITE_OPEN_MAIN_DB );
   540    542   testcase( mask & SQLITE_OPEN_TEMP_DB );
   541    543   if( (mask & (SQLITE_OPEN_MAIN_DB|SQLITE_OPEN_TEMP_DB))!=0 ){ ... }
   542    544   </pre></blockquote>
          545  +
          546  +<p>The SQLite source code contains <tcl>N {$stat(nTestcase)}</tcl>
          547  +uses of the testcase() macro.</p>
   543    548   
   544    549   <tcl>hd_fragment {mcdc} {MC/DC}</tcl>
   545    550   <h3>7.4 Branch coverage versus MC/DC</h3>
   546    551   
   547    552   <p>Two methods of measuring test coverage were described above:
   548    553   "statement" and "branch" coverage.  There are many other test coverage
   549    554   metrics besides these two.  Another popular metric is "Modified
................................................................................
   556    561   <li> Each condition in a decision takes on every possible outcome.
   557    562   <li> Each entry and exit point is invoked.
   558    563   <li> Each condition in a decision is shown to independently
   559    564        affect the outcome of the decision.
   560    565   </ul>
   561    566   
   562    567   <p>In the C programming language 
   563         -where the <tt>&amp;&amp;</tt> and <tt>||</tt> operator
          568  +where <b><tt>&amp;&amp;</tt></b> and <b><tt>||</tt></b>
   564    569   are "short-circuit" operators, MC/DC and branch coverage are very nearly
   565    570   the same thing.  The primary difference is in boolean vector tests.
   566    571   One can test for any of several bits in bit-vector and still obtain
   567    572   100% branch test coverage even though the second element of MC/DC - the
   568    573   requirement that each condition in a decision take on every possible outcome -
   569    574   might not be satisfied.</p>
   570    575   
................................................................................
   678    683   check for correctness.  Static analysis consists mostly of making
   679    684   sure SQLite compiles without warnings, even when all warnings are
   680    685   enabled.  SQLite is developed primarily using GCC and it does
   681    686   compile without warnings on GCC using the -Wall and -Wextra flags.
   682    687   There are occasional reports of warnings coming from VC++, however.</p>
   683    688   
   684    689   <p>Static analysis has not proven to be helpful in finding
   685         -bugs.  We cannot call to mind a single problem in SQLite that
          690  +bugs in SQLite.  We cannot call to mind a single problem in SQLite that
   686    691   was detected by static analysis that was not first seen by one
   687    692   of the other testing methods described above.  On the other hand,
   688    693   we have on occasion introduced new bugs in our efforts to get SQLite
   689    694   to compile without warnings.</p>
   690    695   
   691    696   <p>Our experience, then, is that static analysis is counter-productive
   692    697   to quality.  In other words, focusing on static analysis (being