Documentation Source Text

set stat(sltNTest)     7195342 ;# Number of test cases in SLT
# grep 'errors out of' all-out.txt | awk '{x+=$5}END{print x}'
set stat(sltNRun)     11879758 ;# Number of tests run in SLT
# grep 'assert(' sqlite3.c | wc
set stat(nAssert)         5835 ;# Number of assert statements
# grep 'testcase(' sqlite3.c | grep -v define | wc
set stat(nTestcase)       1096 ;# Number of testcase statements
# ls seeds | wc
set stat(nfuzzseed)        231 ;# Number of dbsqlfuzz seeds
# ls cases | wc   ;# run after -merge=1
set stat(nfuzzpath)      46214 ;# Number of discovered paths

set stat(totalSLOC) [expr {$stat(tclcSLOC)+$stat(tclsSLOC)+
                           $stat(th3SLOC)+$stat(sltcSLOC)+$stat(sltsSLOC)}]

proc GB {expr} {
  set n [uplevel #0 expr $expr]
  hd_puts [format %.2f [expr {$n/(1000.0*1000.0*1000.0)}]]

</tcl>
test code and test scripts - 
<tcl>KB {$stat(totalSLOC)}</tcl> KSLOC.</p>

<h2>Executive Summary</h2>

<ul>
<li> Four independently developed test harnesses
<li> 100% branch test coverage in an as-deployed configuration
<li> Millions and millions of test cases
<li> Out-of-memory tests
<li> I/O error tests
<li> Crash and power loss tests
<li> Fuzz tests
<li> Boundary value tests
<li> Disabled optimization tests
<li> Regression tests
<li> Malformed database tests
<li> Extensive use of assert() and run-time checks
<li> Valgrind analysis
<li> Undefined behavior checks
<li> Checklists
</ul>

<tcl>hd_fragment {harnesses} {test harness} {three test harnesses}</tcl>
<h1>Test Harnesses</h1>

<p>There are four independent test harnesses used for testing the 
core SQLite library.
Each test harness is designed, maintained, and managed separately
from the others.
</p>

<ol>
<li><p>

or SLT test harness is used to run huge numbers
of SQL statements against both SQLite and several other SQL database engines
and verify that they all get the same answers.  SLT currently compares
SQLite against PostgreSQL, MySQL, Microsoft SQL Server, and Oracle 10g.
SLT runs <tcl>MB {$stat(sltNTest)}</tcl> million queries comprising
<tcl>GB {$stat(sltsNByte)}</tcl>GB of test data.
</p></li>

<li><p>
The <a href="#dbsqlfuzz"><b>dbsqlfuzz</b></a> engine is a
proprietary fuzz tester.  Other [fuzz testing|fuzzers for SQLite] 
mutate either the SQL inputs or the database file.  Dbsqlfuzz mutates
both the SQL and the database file at the same time, and is thus able
to reach new error states.  Dbsqlfuzz is built using the
[http://llvm.org/docs/LibFuzzer.html|libFuzzer] framework of LLVM
with a custom mutator.  Starting from 
<tcl>hd_puts $stat(nfuzzseed)</tcl> seed files, the dbsqlfuzz fuzzer
has examined many tens of millions of mutations, resulting in 
<tcl>hd_puts $stat(nfuzzpath)</tcl> distinct test cases that form
the basis of each new fuzzing session.  Dbsqlfuzz helps ensure
that SQLite is robust against attack via malicious SQL or database
inputs.
</ol>

<p>In addition to the four main test harnesses, there several other
small programs that implement specialized tests.
<ol>
<li value="5">The "speedtest1.c" program 
estimates the performance of SQLite under a typical workload.  
<li>The "mptester.c" program is a stress test for multiple processes 
concurrently reading and writing a single database.
<li>The "threadtest3.c" program is a stress test for multiple threads using
SQLite simultaneously.  
<li>The "fuzzershell.c" program is used to
run some <a href="#fuzztesting">fuzz tests</a>.

text to be run against that database. Because it mutates both the input
database and the input SQL at the same time, dbsqlfuzz has been able to
find some obscure faults in SQLite that were missed by prior fuzzers that
mutated only SQL inputs or only the database file.
The SQLite developers usually leave an instance or two of dbsqlfuzz
running on the latest trunk code of SQLite whenever they are away from
the office for an extended period, such as overnight.</p>

<p>The dbsqlfuzz fuzzer has been so successful at hardening the
SQLite code base against malicious attack that it is now considered
one of the four primary test harnesses for SQLite.

<tcl>hd_fragment 3pfuzz {3rd-party fuzzers}</tcl>
<h3>Other third-party fuzzers</h3>

<p>SQLite seems to be a popular target for third-parties to fuzz.
The developers hear about many attempts to fuzz SQLite
and they do occasionally get bug reports found by independent