Documentation Source Text

Check-in [3793e03ba1]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Updates to the testing page: talk of signed-integer overflow and verifying that it is not used.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 3793e03ba1929edae9d5bab57fdccf4933d85e8e
User & Date: drh 2011-03-06 03:32:08.112
Context
2011-03-07
13:40
Fix typos in Appendix A of fts3.html. (check-in: b3c69199b5 user: dan tags: trunk)
2011-03-06
03:32
Updates to the testing page: talk of signed-integer overflow and verifying that it is not used. (check-in: 3793e03ba1 user: drh tags: trunk)
2011-02-23
21:27
Make the opcode formatter resistent to comment text that looks like hyperlinks but is really an array index. (check-in: 479052b445 user: drh tags: trunk)
Changes
Unified Diff Ignore Whitespace Patch
Changes to pages/testing.in.
128
129
130
131
132
133
134

135
136
137
138
139
140
141
<li> Fuzz tests
<li> Boundary value tests
<li> Disabled optimization tests
<li> Regression tests
<li> Malformed database tests
<li> Extensive use of assert() and run-time checks
<li> Valgrind analysis

</ul>

<tcl>hd_fragment {harnesses} {test harness} {three test harnesses}</tcl>
<h2>2.0 Test Harnesses</h2>

<p>There are three independent test harnesses used for testing the 
core SQLite library.







>







128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
<li> Fuzz tests
<li> Boundary value tests
<li> Disabled optimization tests
<li> Regression tests
<li> Malformed database tests
<li> Extensive use of assert() and run-time checks
<li> Valgrind analysis
<li> Signed-integer overflow checks
</ul>

<tcl>hd_fragment {harnesses} {test harness} {three test harnesses}</tcl>
<h2>2.0 Test Harnesses</h2>

<p>There are three independent test harnesses used for testing the 
core SQLite library.
717
718
719
720
721
722
723























724
725
726
727
728
729
730
checking to make sure that nothing is written into the database
file which has not first been written and synced to the rollback journal.
If any discrepancies are found, an assertion fault is raised.</p>

<p>The journal tests are an additional double-check over and above
the crash tests to make sure that SQLite transactions will be atomic
across system crashes and power failures.</p>
























<tcl>hd_fragment disopttest</tcl>
<h2>9.0 Disabled Optimization Tests</h2>

<p>The [sqlite3_test_control]([SQLITE_TESTCTRL_OPTIMIZATIONS], ...) interface
allows selected SQL statement optimizations to be disabled at run-time.
SQLite should always generate exactly the same answer with optimizations







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
checking to make sure that nothing is written into the database
file which has not first been written and synced to the rollback journal.
If any discrepancies are found, an assertion fault is raised.</p>

<p>The journal tests are an additional double-check over and above
the crash tests to make sure that SQLite transactions will be atomic
across system crashes and power failures.</p>

<tcl>hd_fragment intoverflow</tcl>
<h3>8.6 Signed-Integer Overflow Checks</h3>

<p>The various C language standards say that the signed-integer overflow
behavior is undefined.  In other words, when you add a value to a signed
integer such that the result is too large to fit in that integer, the
value does not necessarily wrap around to a negative number, as most
programmers expect.  It might do that.  But it might do something completely
different.  See, for example,
<a href="http://thiemonagel.de/2010/01/signed-integer-overflow/">here</a>
and <a href="http://blog.regehr.org/archives/482">here</a>.  Even the
same compiler might do something different with signed integer overflow
in different places in the code or at different optimizations settings.</p>

<p>SQLite never overflows a signed integer.
To verify this, the test suites are run at least once when compiled with
the -ftrapv option to GCC.  The -ftrapv option causes GCC to generate code
that will panic() on a signed integer overflow.  In addition, there are
many test cases the strive to provoke integer overflows
using boundary value calculations such as
 "<b>SELECT -1*(-9223372036854775808);</b>".


<tcl>hd_fragment disopttest</tcl>
<h2>9.0 Disabled Optimization Tests</h2>

<p>The [sqlite3_test_control]([SQLITE_TESTCTRL_OPTIMIZATIONS], ...) interface
allows selected SQL statement optimizations to be disabled at run-time.
SQLite should always generate exactly the same answer with optimizations
742
743
744
745
746
747
748

749
750
751
752
753
754
755


756
757





758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
computation by counting the number of disk accesses, sort operations, 
full-scan steps, or other processing steps that occur during queries.
Those test cases will appear to fail when optimizations are disabled.
But the majority of test cases simply check that the correct answer
was obtained, and all of those cases can be run successfully with and
without the optimizations, in order to show that the optimizations do not
cause malfunctions.</p>


<tcl>hd_fragment staticanalysis</tcl>
<h2>10.0 Static Analysis</h2>

<p>Static analysis means analyzing code at or before compile-time to
check for correctness.  Static analysis consists mostly of making
sure SQLite compiles without warnings, even when all warnings are


enabled.  SQLite is developed primarily using GCC and it does
compile without warnings on GCC using the -Wall and -Wextra flags.





There are occasional reports of warnings coming from VC++, however.</p>

<p>Static analysis has not proven to be helpful in finding
bugs in SQLite.  We cannot call to mind a single problem in SQLite that
was detected by static analysis that was not first seen by one
of the other testing methods described above.  On the other hand,
we have on occasion introduced new bugs in our efforts to get SQLite
to compile without warnings.</p>

<p>Our experience, then, is that static analysis is counter-productive
to quality.  In other words, focusing on static analysis (being
concerned with compiler warnings) actually reduces the quality of the
code.  Nevertheless, we developers have capitulated to pressure from
users and actively work to eliminate compiler warnings.  We are
willing to do this because the other tests described above do an
excellent job of finding the bugs that are often introduced when
removing compiler warnings, so that product quality is probably not
decreased as a result.</p>

<tcl>hd_fragment summary</tcl>
<h2>11.0 Summary</h2>

<p>SQLite is open source.  This gives many people the idea that
it is not well tested as commercial software and is perhaps unreliable.
But that impression is false.  







>





|
|
>
>
|

>
>
>
>
>
|








|
<
|
|
<
<
<
<
<







766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799

800
801





802
803
804
805
806
807
808
computation by counting the number of disk accesses, sort operations, 
full-scan steps, or other processing steps that occur during queries.
Those test cases will appear to fail when optimizations are disabled.
But the majority of test cases simply check that the correct answer
was obtained, and all of those cases can be run successfully with and
without the optimizations, in order to show that the optimizations do not
cause malfunctions.</p>


<tcl>hd_fragment staticanalysis</tcl>
<h2>10.0 Static Analysis</h2>

<p>Static analysis means analyzing code at or before compile-time to
check for correctness.  Static analysis includes looking at compiler
warning messages and running the code through more in-depth
analysis engines such as the
[http://clang-analyzer.llvm.org/ | Clang Static Analyzer].
SQLite is developed primarily using GCC and it does
compile without warnings on GCC using the -Wall and -Wextra flags.
The SQLite source code is also run through Clang from time to time 
as well, though most of the warnings there are false-positives.
VC++ often will generate a number of warnings from SQLite source code,
but the experience of SQLite developers is that VC++ warnings are of
lower quality and can be safely ignored.  Users are encouraged not
to stress over VC++ warnings.</p>

<p>Static analysis has not proven to be helpful in finding
bugs in SQLite.  We cannot call to mind a single problem in SQLite that
was detected by static analysis that was not first seen by one
of the other testing methods described above.  On the other hand,
we have on occasion introduced new bugs in our efforts to get SQLite
to compile without warnings.</p>

<p>Our experience, then, is that static analysis is not helpful

to maintaining high code quality and it is therefore deemphasized
in SQLite development and maintenance.</p>






<tcl>hd_fragment summary</tcl>
<h2>11.0 Summary</h2>

<p>SQLite is open source.  This gives many people the idea that
it is not well tested as commercial software and is perhaps unreliable.
But that impression is false.