Documentation Source Text

Check-in [80df0299e7]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Fix typos and improve phrasing in testing.in.
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256: 80df0299e780e23f869d8661fe98ad32bc8ed82f775a47beabfae9d383d3f53d
User & Date: drh 2020-02-21 15:27:03
Context
2020-02-22
12:53
Fix a typo in the when-to-use page. check-in: 556d3bb6c4 user: drh tags: trunk
2020-02-21
15:27
Fix typos and improve phrasing in testing.in. check-in: 80df0299e7 user: drh tags: trunk
2020-02-20
04:01
Workaround for older web browsers that may not always set the 'location.origin' DOM property. check-in: 5f54ffe04b user: mistachkin tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to pages/testing.in.

224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
...
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
...
409
410
411
412
413
414
415







416
417
418
419
420
421
422
...
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
...
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
...
532
533
534
535
536
537
538



539
540
541
542
543
544
545
...
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
consisting of about 
<tcl>KB {$stat(vqNEval)}</tcl> thousand test cases.
The veryquick tests include most tests other than the anomaly, fuzz, and 
soak tests.  The idea behind the veryquick tests are that they are
sufficient to catch most errors, but also run in only a few minutes
instead of a few hours.</p>

<tcl>hd_fragment anomoly</tcl>
<h1>Anomaly Testing</h1>

<p>Anomaly tests are tests designed to verify the correct behavior
of SQLite when something goes wrong.  It is (relatively) easy to build
an SQL database engine that behaves correctly on well-formed inputs
on a fully functional computer.  It is more difficult to build a system
that responds sanely to invalid inputs and continues to function following
................................................................................
Most of the findings have been assert() statements where the conditional
was false under obscure circumstances.  But AFL has also found
a fair number of crash bugs in SQLite, and even a few cases where SQLite 
computed incorrect results.

<p>Because of its past success, AFL became a standard part of the testing
strategy for SQLite beginning with [version 3.8.10] ([dateof:3.8.10]) until
it was superceded by better fuzzers in [version 3.29.0] ([dateof:3.29.0]).

<tcl>hd_fragment ossfuzz {OSS Fuzz}</tcl>
<h3>Google OSS Fuzz</h3>

<p>Beginning in 2016, a team of engineers at Google started the
[https://github.com/google/oss-fuzz|OSS Fuzz] project.  
OSS Fuzz uses a AFL-style guided fuzzer running on Google's infrastructure.
................................................................................
problems.  When a fix is checked in, the fuzzer automatically detects this
and emails a confirmation to the developers.

<p>SQLite is one of many open-source projects that OSS Fuzz tests. The
[https://www.sqlite.org/src/file/test/ossfuzz.c|test/ossfuzz.c] source file
in the SQLite repository is SQLite's interface to OSS fuzz.








<tcl>hd_fragment dbsqlfuzz {dbsqlfuzz}</tcl>
<h3>The dbsqlfuzz fuzzer</h3>

<p>Beginning in late 2018, SQLite has been fuzzed using a new proprietary
fuzzer called "dbsqlfuzz".  Dbsqlfuzz is built using the 
[http://llvm.org/docs/LibFuzzer.html|libFuzzer] framework of LLVM.

................................................................................
at the same time.  Dbsqlfuzz uses a custom
[https://github.com/google/fuzzer-test-suite/blob/master/tutorial/structure-aware-fuzzing.md|Structure-Aware Mutator]
on a specialized input file that defines both an input database and SQL
text to be run against that database. Because it mutates both the input
database and the input SQL at the same time, dbsqlfuzz has been able to
find some obscure faults in SQLite that were missed by prior fuzzers that
mutated only SQL inputs or only the database file.

<p>The SQLite developers usually leave an instance or two of dbsqlfuzz
running on the latest trunk code of SQLite whenever they are away from
the office for an extended period, such as overnight.  The dbsqlfuzz
fuzzer has proven so effective that OSSFuzz has ceased to find new
problems in SQLite, except for cases when a new bug is introduced
and OSSFuzz finds it overnight before dbsqlfuzz has had a chance to
run.  (Examples:
[https://www.sqlite.org/src/timeline?y=ci&c=c422afb507dc8757|&#91;1&#93;]
[https://www.sqlite.org/src/timeline?y=ci&c=0a2eb949f8a759e5|&#91;2&#93;]
[https://www.sqlite.org/src/timeline?y=ci&c=62f2235adf796c72|&#91;3&#93;])

<tcl>hd_fragment 3pfuzz {3rd-party fuzzers}</tcl>
<h3>Other third-party fuzzers</h3>

<p>SQLite seems to be a popular target for third-parties to fuzz.
The developers hear about many attempts to fuzz SQLite
and they do occasionally get bug reports found by independent
................................................................................
well during fuzz testing will tend to have (much) less than 
100% MC/DC.
This is because MC/DC testing discourages [defensive code] with
unreachable branches, but without defensive code, a fuzzer is
more likely to find a path that causes problems.  MC/DC testing
seems to work well for building code that is robust during typical
normal use, whereas fuzz testing is good for building code that is
robust against malecious attack.

<p>Of course, users would prefer code that is both robust in normal
use and resistent to malcious attack.  The SQLite developers are
dedicated to providing that.  The purpose of this section is merely
to point out that doing both at the same time is hard.

<p>For much of its history SQLite has been focused on 100% MC/DC testing.
Resistence to fuzzing attacks only became a concern with the introduction
of AFL in 2014.  For a while there, fuzzers were finding many problems
in SQLite.  In more recent years, the testing strategy of SQLite has
evolved to place more emphasis on fuzz testing.  We still maintain
100% MC/DC of the core SQLite code, but most testing CPU cycles are 
now devoted to fuzzing.

<p>While fuzz testing and 100% MC/DC testing are in tension, they
................................................................................
no effect on the integrity of the database.
The interesting cases are when bytes of the file that
define database structure get changed.  The malformed database tests
verify that SQLite finds the file format errors and reports them
using the [SQLITE_CORRUPT] return code without overflowing
buffers, dereferencing NULL pointers, or performing other
unwholesome actions.</p>




<h2>Boundary Value Tests</h2>

<p>SQLite defines certain [limits] on its operation, such as the
maximum number of columns in a table, the maximum length of an 
SQL statement, or the maximum value of an integer.  The TCL and TH3 test
suites both contains numerous tests that push SQLite right to the edge
................................................................................
asserts are so numerous and are in such performance critical places, that
the database engine runs about three times slower when asserts are enabled.
Hence, the default (production) build of SQLite disables asserts.  
Assert statements are only enabled when SQLite is compiled with the
SQLITE_DEBUG preprocessor macro defined.</p>

<p>See the [The Use Of assert In SQLite|Use Of assert in SQLite] document
for additional information about how SQLite uses assert().</p>b

<tcl>hd_fragment valgrind</tcl>
<h2>Valgrind</h2>

<p>[http://valgrind.org/ | Valgrind] is perhaps the most amazing
and useful developer tool in the world.  Valgrind is a simulator - it simulates
an x86 running a Linux binary.  (Ports of Valgrind for platforms other







|







 







|







 







>
>
>
>
>
>
>







 







<
|

|
<
<
<
<
<
<
<







 







|


|




|







 







>
>
>







 







|







224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
...
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
...
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
...
431
432
433
434
435
436
437

438
439
440







441
442
443
444
445
446
447
...
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
...
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
...
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
consisting of about 
<tcl>KB {$stat(vqNEval)}</tcl> thousand test cases.
The veryquick tests include most tests other than the anomaly, fuzz, and 
soak tests.  The idea behind the veryquick tests are that they are
sufficient to catch most errors, but also run in only a few minutes
instead of a few hours.</p>

<tcl>hd_fragment anomaly</tcl>
<h1>Anomaly Testing</h1>

<p>Anomaly tests are tests designed to verify the correct behavior
of SQLite when something goes wrong.  It is (relatively) easy to build
an SQL database engine that behaves correctly on well-formed inputs
on a fully functional computer.  It is more difficult to build a system
that responds sanely to invalid inputs and continues to function following
................................................................................
Most of the findings have been assert() statements where the conditional
was false under obscure circumstances.  But AFL has also found
a fair number of crash bugs in SQLite, and even a few cases where SQLite 
computed incorrect results.

<p>Because of its past success, AFL became a standard part of the testing
strategy for SQLite beginning with [version 3.8.10] ([dateof:3.8.10]) until
it was superseded by better fuzzers in [version 3.29.0] ([dateof:3.29.0]).

<tcl>hd_fragment ossfuzz {OSS Fuzz}</tcl>
<h3>Google OSS Fuzz</h3>

<p>Beginning in 2016, a team of engineers at Google started the
[https://github.com/google/oss-fuzz|OSS Fuzz] project.  
OSS Fuzz uses a AFL-style guided fuzzer running on Google's infrastructure.
................................................................................
problems.  When a fix is checked in, the fuzzer automatically detects this
and emails a confirmation to the developers.

<p>SQLite is one of many open-source projects that OSS Fuzz tests. The
[https://www.sqlite.org/src/file/test/ossfuzz.c|test/ossfuzz.c] source file
in the SQLite repository is SQLite's interface to OSS fuzz.

<p>OSS Fuzz no longer finds historical bugs in SQLite.  But it is still
running and does occasionally find issues in new development check-ins.
Examples:
[https://www.sqlite.org/src/timeline?y=ci&c=c422afb507dc8757|&#91;1&#93;]
[https://www.sqlite.org/src/timeline?y=ci&c=0a2eb949f8a759e5|&#91;2&#93;]
[https://www.sqlite.org/src/timeline?y=ci&c=62f2235adf796c72|&#91;3&#93;].

<tcl>hd_fragment dbsqlfuzz {dbsqlfuzz}</tcl>
<h3>The dbsqlfuzz fuzzer</h3>

<p>Beginning in late 2018, SQLite has been fuzzed using a new proprietary
fuzzer called "dbsqlfuzz".  Dbsqlfuzz is built using the 
[http://llvm.org/docs/LibFuzzer.html|libFuzzer] framework of LLVM.

................................................................................
at the same time.  Dbsqlfuzz uses a custom
[https://github.com/google/fuzzer-test-suite/blob/master/tutorial/structure-aware-fuzzing.md|Structure-Aware Mutator]
on a specialized input file that defines both an input database and SQL
text to be run against that database. Because it mutates both the input
database and the input SQL at the same time, dbsqlfuzz has been able to
find some obscure faults in SQLite that were missed by prior fuzzers that
mutated only SQL inputs or only the database file.

The SQLite developers usually leave an instance or two of dbsqlfuzz
running on the latest trunk code of SQLite whenever they are away from
the office for an extended period, such as overnight.</p>








<tcl>hd_fragment 3pfuzz {3rd-party fuzzers}</tcl>
<h3>Other third-party fuzzers</h3>

<p>SQLite seems to be a popular target for third-parties to fuzz.
The developers hear about many attempts to fuzz SQLite
and they do occasionally get bug reports found by independent
................................................................................
well during fuzz testing will tend to have (much) less than 
100% MC/DC.
This is because MC/DC testing discourages [defensive code] with
unreachable branches, but without defensive code, a fuzzer is
more likely to find a path that causes problems.  MC/DC testing
seems to work well for building code that is robust during typical
normal use, whereas fuzz testing is good for building code that is
robust against malicious attack.

<p>Of course, users would prefer code that is both robust in normal
use and resistant to malicious attack.  The SQLite developers are
dedicated to providing that.  The purpose of this section is merely
to point out that doing both at the same time is hard.

<p>For much of its history SQLite has been focused on 100% MC/DC testing.
Resistance to fuzzing attacks only became a concern with the introduction
of AFL in 2014.  For a while there, fuzzers were finding many problems
in SQLite.  In more recent years, the testing strategy of SQLite has
evolved to place more emphasis on fuzz testing.  We still maintain
100% MC/DC of the core SQLite code, but most testing CPU cycles are 
now devoted to fuzzing.

<p>While fuzz testing and 100% MC/DC testing are in tension, they
................................................................................
no effect on the integrity of the database.
The interesting cases are when bytes of the file that
define database structure get changed.  The malformed database tests
verify that SQLite finds the file format errors and reports them
using the [SQLITE_CORRUPT] return code without overflowing
buffers, dereferencing NULL pointers, or performing other
unwholesome actions.</p>

<p>The [dbsqlfuzz] fuzzer also does an excellent job of verifying
that SQLite responds sanely to malformed database files.</p>

<h2>Boundary Value Tests</h2>

<p>SQLite defines certain [limits] on its operation, such as the
maximum number of columns in a table, the maximum length of an 
SQL statement, or the maximum value of an integer.  The TCL and TH3 test
suites both contains numerous tests that push SQLite right to the edge
................................................................................
asserts are so numerous and are in such performance critical places, that
the database engine runs about three times slower when asserts are enabled.
Hence, the default (production) build of SQLite disables asserts.  
Assert statements are only enabled when SQLite is compiled with the
SQLITE_DEBUG preprocessor macro defined.</p>

<p>See the [The Use Of assert In SQLite|Use Of assert in SQLite] document
for additional information about how SQLite uses assert().</p>

<tcl>hd_fragment valgrind</tcl>
<h2>Valgrind</h2>

<p>[http://valgrind.org/ | Valgrind] is perhaps the most amazing
and useful developer tool in the world.  Valgrind is a simulator - it simulates
an x86 running a Linux binary.  (Ports of Valgrind for platforms other