Documentation Source Text

Check-in [34e2a86993]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Add discussion of the AFL fuzzer to the testing document.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 34e2a86993e2733524738e9428b5d79713c66100
User & Date: drh 2015-04-25 18:17:41.728
Context
2015-04-27
16:42
Update the description of the content= option in fts5.html. (check-in: 47550c82de user: dan tags: trunk)
2015-04-25
18:17
Add discussion of the AFL fuzzer to the testing document. (check-in: 34e2a86993 user: drh tags: trunk)
2015-04-24
20:09
Update fts5.in with the UNINDEXED option. (check-in: 0742089777 user: dan tags: trunk)
Changes
Unified Diff Ignore Whitespace Patch
Changes to pages/changes.in.
20
21
22
23
24
25
26


27
28
29
30
31
32
33

chng {2015-06-00 (3.8.10)} {
<li>Added the sqldiff.exe utility program for computing the differences between two
    SQLite database files.
<li>Performance optimizations for sorting and "PRAGMA integrity_check"
<li>Fix many obscure problems discovered while SQL fuzzing
<li>Identify all methods for important objects in the interface documentation.


}

chng {2015-04-08 (3.8.9)} {
<li>Add VxWorks-7 as an officially supported and tested platform.
<li>Added the [sqlite3_status64()] interface.
<li>Fix memory size tracking so that it works even if SQLite uses more
    than 2GiB of memory.







>
>







20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

chng {2015-06-00 (3.8.10)} {
<li>Added the sqldiff.exe utility program for computing the differences between two
    SQLite database files.
<li>Performance optimizations for sorting and "PRAGMA integrity_check"
<li>Fix many obscure problems discovered while SQL fuzzing
<li>Identify all methods for important objects in the interface documentation.
<li>Made the [American Fuzzy Lop fuzzer]
    a standard part of SQLite's [testing|testing strategy].
}

chng {2015-04-08 (3.8.9)} {
<li>Add VxWorks-7 as an officially supported and tested platform.
<li>Added the [sqlite3_status64()] interface.
<li>Fix memory size tracking so that it works even if SQLite uses more
    than 2GiB of memory.
Changes to pages/cintro.in.
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216

  <p>Think of each SQL statement as a small computer program.  The purpose
  of [sqlite3_prepare()] is to compile that program into object code.
  The [prepared statement] is the object code.  The [sqlite3_step()] interface
  then runs the object code to get a result.

  <p>New applications should always invoke [sqlite3_prepare_v2()] instead
  of [sqlit3_prepare()].  The older [sqlite3_prepare()] is retained for
  backwards compatibility.  But [sqlite3_prepare_v2()] provides a much
  better interface.</p>
</td>

<tr><td valign="top" align="right">[sqlite3_step()]</td>
<td valign="top">
  This routine is used to evaluate a [prepared statement] that has been







|







202
203
204
205
206
207
208
209
210
211
212
213
214
215
216

  <p>Think of each SQL statement as a small computer program.  The purpose
  of [sqlite3_prepare()] is to compile that program into object code.
  The [prepared statement] is the object code.  The [sqlite3_step()] interface
  then runs the object code to get a result.

  <p>New applications should always invoke [sqlite3_prepare_v2()] instead
  of [sqlite3_prepare()].  The older [sqlite3_prepare()] is retained for
  backwards compatibility.  But [sqlite3_prepare_v2()] provides a much
  better interface.</p>
</td>

<tr><td valign="top" align="right">[sqlite3_step()]</td>
<td valign="top">
  This routine is used to evaluate a [prepared statement] that has been
Changes to pages/testing.in.
191
192
193
194
195
196
197
198
199


200
201
202
203
204




205
206
207
208
209
210
211
and verify that they all get the same answers.  SLT currently compares
SQLite against PostgreSQL, MySQL, Microsoft SQL Server, and Oracle 10g.
SLT runs <tcl>MB {$stat(sltNTest)}</tcl> million queries comprising
<tcl>GB {$stat(sltsNByte)}</tcl>GB of test data.
</p></li>
</ol>

<p>In addition to the three major test suites, there are a few separate
programs that implement specialized tests.  The "speedtest1.c" program 


estimates the performance of SQLite under a typical workload.  
The "mptester.c" program is a stress test for multiple processes 
concurrently reading and writing a single database.
And the "threadtest3.c" program is a stress test for multiple threads using
SQLite simultaneously.</p>





<p>All of the tests above must run successfully, on multiple platforms
and under multiple compile-time configurations,
before each release of SQLite.</p>

<p>Prior to each check-in to the SQLite source tree, developers
typically run a subset (called "veryquick") of the Tcl tests







|
|
>
>

|

|
|
>
>
>
>







191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
and verify that they all get the same answers.  SLT currently compares
SQLite against PostgreSQL, MySQL, Microsoft SQL Server, and Oracle 10g.
SLT runs <tcl>MB {$stat(sltNTest)}</tcl> million queries comprising
<tcl>GB {$stat(sltsNByte)}</tcl>GB of test data.
</p></li>
</ol>

<p>In addition to the three major test harnesses, there several other
small programs that implement specialized tests.
<ol>
<li value="4">The "speedtest1.c" program 
estimates the performance of SQLite under a typical workload.  
<li>The "mptester.c" program is a stress test for multiple processes 
concurrently reading and writing a single database.
<li>The "threadtest3.c" program is a stress test for multiple threads using
SQLite simultaneously.  
<li>The "fuzzershell.c" program is used to
run some <a href='#fuzztesting'>fuzz tests</a>.
</ol>
</p>

<p>All of the tests above must run successfully, on multiple platforms
and under multiple compile-time configurations,
before each release of SQLite.</p>

<p>Prior to each check-in to the SQLite source tree, developers
typically run a subset (called "veryquick") of the Tcl tests
360
361
362
363
364
365
366






























367
368
369
370
371
372
373
resulting prepared statement is run to make sure it gives a reasonable
result.</p>

<p>The SQL fuzz generator tests are part of the TCL test suite.
During a full test run, about <tcl>KB {$stat(nSqlFuzz)}</tcl> 
thousand fuzz SQL statements are
generated and tested.</p>































<h3>4.2 Malformed Database Files</h3>

<p>There are numerous test cases that verify that SQLite is able to
deal with malformed database files.
These tests first build a well-formed database file, then add
corruption by changing one or more bytes in the file by some means







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
resulting prepared statement is run to make sure it gives a reasonable
result.</p>

<p>The SQL fuzz generator tests are part of the TCL test suite.
During a full test run, about <tcl>KB {$stat(nSqlFuzz)}</tcl> 
thousand fuzz SQL statements are
generated and tested.</p>

<tcl>hd_fragment aflfuzz {American Fuzzy Lop fuzzer}</tcl>
<h4>4.1.1 SQL Fuzz Using The American Fuzzy Lop Fuzzer</h4>

<p>The <a href="http://lcamtuf.coredump.cx/afl/">American Fuzzy Lop</a>
or "AFL" fuzzer is a recent (circa 2014) innovation from Michal Zalewski.
Unlike most other fuzzers that blindly generate random inputs, the AFL
fuzzer instruments the program being tested (by editing the assembly-language
output from the C compiler) and uses that instrumentation to detect when
a mutated input causes the program to do something different - to follow
a new control path or loop a different number of times.  Inputs that provoke
new behavior are retained and further mutated.  In this way, AFL is able
to "discover" new behaviors of the program under test, including behaviors
that were never envisioned by the designers.

<p>AFL has proven remarkably adept at finding arcane bugs in SQLite.
Most of the problems found have been assert() statements where the conditional
could be false under obscure circumstances.  But AFL has also found
a fair number of crash bugs in SQLite, and even a few cases where SQLite 
computed incorrect results.

<p>Because of its past success, AFL became a standard part of the testing
strategy for SQLite beginning with [version 3.8.10].  There is at least one
instance of AFL running against SQLite continuously, 24/7/365, testing new
randomly mutated inputs against SQLite at a rate of a few hundred to a few
thousand per second.  Billions of inputs have been tried, but AFL's 
instrumentation has narrowed that down to less than 20,000 test cases that
cover all distinct behaviors.  These distinct test cases are periodically
captured and added to the [TCL test suite] where they are rerun automatically
during routine testing.

<h3>4.2 Malformed Database Files</h3>

<p>There are numerous test cases that verify that SQLite is able to
deal with malformed database files.
These tests first build a well-formed database file, then add
corruption by changing one or more bytes in the file by some means