Documentation Source Text

Check-in [34e2a86993]

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Comment:Add discussion of the AFL fuzzer to the testing document.
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 34e2a86993e2733524738e9428b5d79713c66100
User & Date: drh 2015-04-25 18:17:41
Update the description of the content= option in fts5.html. check-in: 47550c82de user: dan tags: trunk
Add discussion of the AFL fuzzer to the testing document. check-in: 34e2a86993 user: drh tags: trunk
Update with the UNINDEXED option. check-in: 0742089777 user: dan tags: trunk
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Changes to pages/

    20     20   
    21     21   chng {2015-06-00 (3.8.10)} {
    22     22   <li>Added the sqldiff.exe utility program for computing the differences between two
    23     23       SQLite database files.
    24     24   <li>Performance optimizations for sorting and "PRAGMA integrity_check"
    25     25   <li>Fix many obscure problems discovered while SQL fuzzing
    26     26   <li>Identify all methods for important objects in the interface documentation.
           27  +<li>Made the [American Fuzzy Lop fuzzer]
           28  +    a standard part of SQLite's [testing|testing strategy].
    27     29   }
    28     30   
    29     31   chng {2015-04-08 (3.8.9)} {
    30     32   <li>Add VxWorks-7 as an officially supported and tested platform.
    31     33   <li>Added the [sqlite3_status64()] interface.
    32     34   <li>Fix memory size tracking so that it works even if SQLite uses more
    33     35       than 2GiB of memory.

Changes to pages/

   202    202   
   203    203     <p>Think of each SQL statement as a small computer program.  The purpose
   204    204     of [sqlite3_prepare()] is to compile that program into object code.
   205    205     The [prepared statement] is the object code.  The [sqlite3_step()] interface
   206    206     then runs the object code to get a result.
   207    207   
   208    208     <p>New applications should always invoke [sqlite3_prepare_v2()] instead
   209         -  of [sqlit3_prepare()].  The older [sqlite3_prepare()] is retained for
          209  +  of [sqlite3_prepare()].  The older [sqlite3_prepare()] is retained for
   210    210     backwards compatibility.  But [sqlite3_prepare_v2()] provides a much
   211    211     better interface.</p>
   212    212   </td>
   213    213   
   214    214   <tr><td valign="top" align="right">[sqlite3_step()]</td>
   215    215   <td valign="top">
   216    216     This routine is used to evaluate a [prepared statement] that has been

Changes to pages/

   191    191   and verify that they all get the same answers.  SLT currently compares
   192    192   SQLite against PostgreSQL, MySQL, Microsoft SQL Server, and Oracle 10g.
   193    193   SLT runs <tcl>MB {$stat(sltNTest)}</tcl> million queries comprising
   194    194   <tcl>GB {$stat(sltsNByte)}</tcl>GB of test data.
   195    195   </p></li>
   196    196   </ol>
   197    197   
   198         -<p>In addition to the three major test suites, there are a few separate
   199         -programs that implement specialized tests.  The "speedtest1.c" program 
          198  +<p>In addition to the three major test harnesses, there several other
          199  +small programs that implement specialized tests.
          200  +<ol>
          201  +<li value="4">The "speedtest1.c" program 
   200    202   estimates the performance of SQLite under a typical workload.  
   201         -The "mptester.c" program is a stress test for multiple processes 
          203  +<li>The "mptester.c" program is a stress test for multiple processes 
   202    204   concurrently reading and writing a single database.
   203         -And the "threadtest3.c" program is a stress test for multiple threads using
   204         -SQLite simultaneously.</p>
          205  +<li>The "threadtest3.c" program is a stress test for multiple threads using
          206  +SQLite simultaneously.  
          207  +<li>The "fuzzershell.c" program is used to
          208  +run some <a href='#fuzztesting'>fuzz tests</a>.
          209  +</ol>
          210  +</p>
   205    211   
   206    212   <p>All of the tests above must run successfully, on multiple platforms
   207    213   and under multiple compile-time configurations,
   208    214   before each release of SQLite.</p>
   209    215   
   210    216   <p>Prior to each check-in to the SQLite source tree, developers
   211    217   typically run a subset (called "veryquick") of the Tcl tests
   360    366   resulting prepared statement is run to make sure it gives a reasonable
   361    367   result.</p>
   362    368   
   363    369   <p>The SQL fuzz generator tests are part of the TCL test suite.
   364    370   During a full test run, about <tcl>KB {$stat(nSqlFuzz)}</tcl> 
   365    371   thousand fuzz SQL statements are
   366    372   generated and tested.</p>
          373  +
          374  +<tcl>hd_fragment aflfuzz {American Fuzzy Lop fuzzer}</tcl>
          375  +<h4>4.1.1 SQL Fuzz Using The American Fuzzy Lop Fuzzer</h4>
          376  +
          377  +<p>The <a href="">American Fuzzy Lop</a>
          378  +or "AFL" fuzzer is a recent (circa 2014) innovation from Michal Zalewski.
          379  +Unlike most other fuzzers that blindly generate random inputs, the AFL
          380  +fuzzer instruments the program being tested (by editing the assembly-language
          381  +output from the C compiler) and uses that instrumentation to detect when
          382  +a mutated input causes the program to do something different - to follow
          383  +a new control path or loop a different number of times.  Inputs that provoke
          384  +new behavior are retained and further mutated.  In this way, AFL is able
          385  +to "discover" new behaviors of the program under test, including behaviors
          386  +that were never envisioned by the designers.
          387  +
          388  +<p>AFL has proven remarkably adept at finding arcane bugs in SQLite.
          389  +Most of the problems found have been assert() statements where the conditional
          390  +could be false under obscure circumstances.  But AFL has also found
          391  +a fair number of crash bugs in SQLite, and even a few cases where SQLite 
          392  +computed incorrect results.
          393  +
          394  +<p>Because of its past success, AFL became a standard part of the testing
          395  +strategy for SQLite beginning with [version 3.8.10].  There is at least one
          396  +instance of AFL running against SQLite continuously, 24/7/365, testing new
          397  +randomly mutated inputs against SQLite at a rate of a few hundred to a few
          398  +thousand per second.  Billions of inputs have been tried, but AFL's 
          399  +instrumentation has narrowed that down to less than 20,000 test cases that
          400  +cover all distinct behaviors.  These distinct test cases are periodically
          401  +captured and added to the [TCL test suite] where they are rerun automatically
          402  +during routine testing.
   367    403   
   368    404   <h3>4.2 Malformed Database Files</h3>
   369    405   
   370    406   <p>There are numerous test cases that verify that SQLite is able to
   371    407   deal with malformed database files.
   372    408   These tests first build a well-formed database file, then add
   373    409   corruption by changing one or more bytes in the file by some means