Documentation Source Text

<li> Fuzz tests
<li> Boundary value tests
<li> Disabled optimization tests
<li> Regression tests
<li> Malformed database tests
<li> Extensive use of assert() and run-time checks
<li> Valgrind analysis

</ul>

<tcl>hd_fragment {harnesses} {test harness} {three test harnesses}</tcl>
<h2>2.0 Test Harnesses</h2>

<p>There are three independent test harnesses used for testing the 
core SQLite library.
................................................................................
checking to make sure that nothing is written into the database
file which has not first been written and synced to the rollback journal.
If any discrepancies are found, an assertion fault is raised.</p>

<p>The journal tests are an additional double-check over and above
the crash tests to make sure that SQLite transactions will be atomic
across system crashes and power failures.</p>
























<tcl>hd_fragment disopttest</tcl>
<h2>9.0 Disabled Optimization Tests</h2>

<p>The [sqlite3_test_control]([SQLITE_TESTCTRL_OPTIMIZATIONS], ...) interface
allows selected SQL statement optimizations to be disabled at run-time.
SQLite should always generate exactly the same answer with optimizations
................................................................................
computation by counting the number of disk accesses, sort operations, 
full-scan steps, or other processing steps that occur during queries.
Those test cases will appear to fail when optimizations are disabled.
But the majority of test cases simply check that the correct answer
was obtained, and all of those cases can be run successfully with and
without the optimizations, in order to show that the optimizations do not
cause malfunctions.</p>


<tcl>hd_fragment staticanalysis</tcl>
<h2>10.0 Static Analysis</h2>

<p>Static analysis means analyzing code at or before compile-time to
check for correctness.  Static analysis consists mostly of making
sure SQLite compiles without warnings, even when all warnings are


enabled.  SQLite is developed primarily using GCC and it does
compile without warnings on GCC using the -Wall and -Wextra flags.
There are occasional reports of warnings coming from VC++, however.</p>







<p>Static analysis has not proven to be helpful in finding
bugs in SQLite.  We cannot call to mind a single problem in SQLite that
was detected by static analysis that was not first seen by one
of the other testing methods described above.  On the other hand,
we have on occasion introduced new bugs in our efforts to get SQLite
to compile without warnings.</p>

<p>Our experience, then, is that static analysis is counter-productive
to quality.  In other words, focusing on static analysis (being
concerned with compiler warnings) actually reduces the quality of the
code.  Nevertheless, we developers have capitulated to pressure from
users and actively work to eliminate compiler warnings.  We are
willing to do this because the other tests described above do an
excellent job of finding the bugs that are often introduced when
removing compiler warnings, so that product quality is probably not
decreased as a result.</p>

<tcl>hd_fragment summary</tcl>
<h2>11.0 Summary</h2>

<p>SQLite is open source.  This gives many people the idea that
it is not well tested as commercial software and is perhaps unreliable.
But that impression is false.  

<li> Fuzz tests
<li> Boundary value tests
<li> Disabled optimization tests
<li> Regression tests
<li> Malformed database tests
<li> Extensive use of assert() and run-time checks
<li> Valgrind analysis
<li> Signed-integer overflow checks
</ul>

<tcl>hd_fragment {harnesses} {test harness} {three test harnesses}</tcl>
<h2>2.0 Test Harnesses</h2>

<p>There are three independent test harnesses used for testing the 
core SQLite library.
................................................................................
checking to make sure that nothing is written into the database
file which has not first been written and synced to the rollback journal.
If any discrepancies are found, an assertion fault is raised.</p>

<p>The journal tests are an additional double-check over and above
the crash tests to make sure that SQLite transactions will be atomic
across system crashes and power failures.</p>

<tcl>hd_fragment intoverflow</tcl>
<h3>8.6 Signed-Integer Overflow Checks</h3>

<p>The various C language standards say that the signed-integer overflow
behavior is undefined.  In other words, when you add a value to a signed
integer such that the result is too large to fit in that integer, the
value does not necessarily wrap around to a negative number, as most
programmers expect.  It might do that.  But it might do something completely
different.  See, for example,
<a href="http://thiemonagel.de/2010/01/signed-integer-overflow/">here</a>
and <a href="http://blog.regehr.org/archives/482">here</a>.  Even the
same compiler might do something different with signed integer overflow
in different places in the code or at different optimizations settings.</p>

<p>SQLite never overflows a signed integer.
To verify this, the test suites are run at least once when compiled with
the -ftrapv option to GCC.  The -ftrapv option causes GCC to generate code
that will panic() on a signed integer overflow.  In addition, there are
many test cases the strive to provoke integer overflows
using boundary value calculations such as
 "<b>SELECT -1*(-9223372036854775808);</b>".


<tcl>hd_fragment disopttest</tcl>
<h2>9.0 Disabled Optimization Tests</h2>

<p>The [sqlite3_test_control]([SQLITE_TESTCTRL_OPTIMIZATIONS], ...) interface
allows selected SQL statement optimizations to be disabled at run-time.
SQLite should always generate exactly the same answer with optimizations
................................................................................
computation by counting the number of disk accesses, sort operations, 
full-scan steps, or other processing steps that occur during queries.
Those test cases will appear to fail when optimizations are disabled.
But the majority of test cases simply check that the correct answer
was obtained, and all of those cases can be run successfully with and
without the optimizations, in order to show that the optimizations do not
cause malfunctions.</p>


<tcl>hd_fragment staticanalysis</tcl>
<h2>10.0 Static Analysis</h2>

<p>Static analysis means analyzing code at or before compile-time to
check for correctness.  Static analysis includes looking at compiler
warning messages and running the code through more in-depth
analysis engines such as the
[http://clang-analyzer.llvm.org/ | Clang Static Analyzer].
SQLite is developed primarily using GCC and it does
compile without warnings on GCC using the -Wall and -Wextra flags.

The SQLite source code is also run through Clang from time to time 
as well, though most of the warnings there are false-positives.
VC++ often will generate a number of warnings from SQLite source code,
but the experience of SQLite developers is that VC++ warnings are of
lower quality and can be safely ignored.  Users are encouraged not
to stress over VC++ warnings.</p>

<p>Static analysis has not proven to be helpful in finding
bugs in SQLite.  We cannot call to mind a single problem in SQLite that
was detected by static analysis that was not first seen by one
of the other testing methods described above.  On the other hand,
we have on occasion introduced new bugs in our efforts to get SQLite
to compile without warnings.</p>

<p>Our experience, then, is that static analysis is not helpful
to maintaining high code quality and it is therefore deemphasized
in SQLite development and maintenance.</p>







<tcl>hd_fragment summary</tcl>
<h2>11.0 Summary</h2>

<p>SQLite is open source.  This gives many people the idea that
it is not well tested as commercial software and is perhaps unreliable.
But that impression is false.