Documentation Source Text

Check-in [b7925a9372]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Updates to the "How SQLite Is Tested" document.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: b7925a93723fa5507c105b940e133cceffd74153
User & Date: drh 2014-08-11 23:56:01.512
Context
2014-08-12
02:49
Updates to the "How SQLite Is Tested" document. (check-in: 7aa17891ac user: drh tags: trunk)
2014-08-11
23:56
Updates to the "How SQLite Is Tested" document. (check-in: b7925a9372 user: drh tags: trunk)
21:17
Fix requirements marks in the file format document. Improvements to PRAGMA documentation. (check-in: e09448dc9d user: drh tags: trunk)
Changes
Side-by-Side Diff Ignore Whitespace Patch
Changes to pages/testing.in.
127
128
129
130
131
132
133
134

135
136
137
138
139
140
141
127
128
129
130
131
132
133

134
135
136
137
138
139
140
141







-
+







<li> Fuzz tests
<li> Boundary value tests
<li> Disabled optimization tests
<li> Regression tests
<li> Malformed database tests
<li> Extensive use of assert() and run-time checks
<li> Valgrind analysis
<li> Signed-integer overflow checks
<li> Undefined behavior checks
<li> Checklists
</ul>

<tcl>hd_fragment {harnesses} {test harness} {three test harnesses}</tcl>
<h2>2.0 Test Harnesses</h2>

<p>There are three independent test harnesses used for testing the 
720
721
722
723
724
725
726
727

728
729
730
731
732
733
734
735


























736
737
738

739
740
741
742




743
744







745
746


747
748
749
750
751
752
753
754
720
721
722
723
724
725
726

727
728







729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754



755
756



757
758
759
760


761
762
763
764
765
766
767


768
769

770
771
772
773
774
775
776







-
+

-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
-
-
-
+

-
-
-
+
+
+
+
-
-
+
+
+
+
+
+
+
-
-
+
+
-







If any discrepancies are found, an assertion fault is raised.</p>

<p>The journal tests are an additional double-check over and above
the crash tests to make sure that SQLite transactions will be atomic
across system crashes and power failures.</p>

<tcl>hd_fragment intoverflow</tcl>
<h3>8.6 Signed-Integer Overflow Checks</h3>
<h3>8.6 Undefined Behavior Checks</h3>

<p>The various C language standards say that the signed-integer overflow
behavior is undefined.  In other words, when you add a value to a signed
integer such that the result is too large to fit in that integer, the
value does not necessarily wrap around to a negative number, as most
programmers expect.  It might do that.  But it might do something completely
different.  See, for example,
<a href="http://thiemonagel.de/2010/01/signed-integer-overflow/">here</a>
<p>In in the C programming language, it is very easy to write code that
has "undefined" or "implementation defined" behavior.
That means that the code might work during development, but then give
a different answer on a different system, or when recompiled using different
compiler options.  
Examples of undefined and implementation-defined behavior in
ANSI C include:
<ul>
<li>Signed integer overflow.  (Signed integer overflow does <u>not</u>
necessarily wrap around, as most people expect.)
<li>Shifting an N-bit integer by more than N bits.
<li>Shifting by a negative amount.
<li>Shifting a negative number.
<li>Using the memcpy() function on overlapping buffers.
<li>The order of evaluation of function arguments.
<li>Whether or not "char" variables are signed or unsigned.
<li>And so forth....
</ul>

<p>Since undefined and implementation-defined behavior is non-portable
and can easily lead to incorrect answers, SQLite works very hard to avoid it.
For example,
when adding two integer column values together as part of an SQL statement,
SQLite does not simple add them together using the C-language "+" operator.
Instead, it first checks to make sure the
addition will not overflow, and if it will, it does the addition using
and <a href="http://blog.regehr.org/archives/482">here</a>.  Even the
same compiler might do something different with signed integer overflow
in different places in the code or at different optimizations settings.</p>
floating point instead.

<p>SQLite never overflows a signed integer.
To verify this, the test suites are run at least once when compiled with
the -ftrapv option to GCC.  The -ftrapv option causes GCC to generate code
<p>To help ensure that SQLite does not make use of undefined or
implementation defined behavior, the test suites are rerun using
instrumented builds that try to detect undefined behavior.  For example,
test suites are run using the "-ftrapv" option of GCC.  And they
that will panic() on a signed integer overflow.  In addition, there are
many test cases the strive to provoke integer overflows
are run again using the "-fsanitize=undefined" option on Clang.  And
again using the "/RTC1" option in MSVC.  Then the test suites are rerun
using options like "-funsigned-char" and "-fsigned-char" to make sure
that implementation differences do not matter either.  Tests are then repeated
on 32-bit and 64-bit systems and on big-endian and little-endian systems,
using a variety of CPU architectures.
Furthermore, the test suites are augmented with many test cases that are
using boundary value calculations such as
 "<b>SELECT -1*(-9223372036854775808);</b>".
deliberately designed to provoke undefined behavior.  For example:
"<b>SELECT -1*(-9223372036854775808);</b>".


<tcl>hd_fragment disopttest</tcl>
<h2>9.0 Disabled Optimization Tests</h2>

<p>The [sqlite3_test_control]([SQLITE_TESTCTRL_OPTIMIZATIONS], ...) interface
allows selected SQL statement optimizations to be disabled at run-time.
SQLite should always generate exactly the same answer with optimizations