Documentation Source Text

Check-in [0b4aaced39]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Improvements to the testing.html document.
Timelines: family | ancestors | descendants | both | branch-3.8.10
Files: files | file ages | folders
SHA1: 0b4aaced399fa68adcc0ee7ef55cdb0fd822251d
User & Date: drh 2015-05-15 15:56:00
Context
2015-05-19
14:05
Fix a typo on the testing.html page. check-in: db6bfdf9aa user: drh tags: branch-3.8.10
2015-05-15
15:56
Improvements to the testing.html document. check-in: 0b4aaced39 user: drh tags: branch-3.8.10
14:53
Link from the ATTACH documentation page to the DETACH page. Increased information about TH3. These are cherry-picks from trunk. check-in: 643e08dfb8 user: drh tags: branch-3.8.10
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to pages/testing.in.

668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708










709
710
711
712
713
714
715
"-g -fprofile-arcs -ftest-coverage" and then the test program is run.
Then "gcov -b" is run to generate a coverage report.
The coverage report is verbose and inconvenient to read, 
so the gcov-generated report is processed using
some simple scripts to put it into a more human-friendly format.
This entire process is automated using scripts, of course.

<p>Note that running SQLite using gcov is not a test of SQLite &mdash;
it is a test of the test suite.  The gcov run does not test SQLite because
the -fprofile-args and -ftest-coverage options cause the compiler to 
generate different code.  
The gcov run merely verifies that the test suite provides 100% branch test
coverage.  The gcov run is a test of the test - a meta-test.

<p>After gcov has been run to verify 100% branch test coverage,
then the test program is recompiled using delivery compiler options
(without the special -fprofile-arcs and -ftest-coverage options)
and the test program is rerun.
This second run is the actual test of SQLite.

<p>It is important to verify that both the gcov test run 
and the second real test run both give the same output.  Any
differences in output indicate either the use of undefined or
indeterminate behavior in the SQLite code (and hence a bug), 
or a bug in the compiler.
Note that SQLite has, over the previous decade, encountered bugs
in each of GCC, Clang, and MSVC.  Compiler bugs, while rare, do happen,
which is why it is so important to test the code in an as-delivered
configuration.

<tcl>hd_fragment thoughts1</tcl>
<h3>7.6 Experience with full test coverage</h3>

<p>The developers of SQLite have found that full coverage testing is an
extremely productive method for preventing the introduction of new bugs
as the system evolves.  Because every single branch
instruction in SQLite core code is covered by test cases, the developers
can be confident that changes they make in one part of the code
do not have unintended consequences in other parts of the code.
It would be extremely difficult maintain the quality of SQLite without such
assurances.</p>











<tcl>hd_fragment dynamicanalysis</tcl>
<h2>8.0 Dynamic Analysis</h2>

<p>Dynamic analysis refers to internal and external checks on the
SQLite code which are performed while the code is live and running.
Dynamic analysis has proven to be a great help in maintaining the







|












|













|
|



|
|
>
>
>
>
>
>
>
>
>
>







668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
"-g -fprofile-arcs -ftest-coverage" and then the test program is run.
Then "gcov -b" is run to generate a coverage report.
The coverage report is verbose and inconvenient to read, 
so the gcov-generated report is processed using
some simple scripts to put it into a more human-friendly format.
This entire process is automated using scripts, of course.

<p>Note that running SQLite with gcov is not a test of SQLite &mdash;
it is a test of the test suite.  The gcov run does not test SQLite because
the -fprofile-args and -ftest-coverage options cause the compiler to 
generate different code.  
The gcov run merely verifies that the test suite provides 100% branch test
coverage.  The gcov run is a test of the test - a meta-test.

<p>After gcov has been run to verify 100% branch test coverage,
then the test program is recompiled using delivery compiler options
(without the special -fprofile-arcs and -ftest-coverage options)
and the test program is rerun.
This second run is the actual test of SQLite.

<p>It is important to verify that the gcov test run 
and the second real test run both give the same output.  Any
differences in output indicate either the use of undefined or
indeterminate behavior in the SQLite code (and hence a bug), 
or a bug in the compiler.
Note that SQLite has, over the previous decade, encountered bugs
in each of GCC, Clang, and MSVC.  Compiler bugs, while rare, do happen,
which is why it is so important to test the code in an as-delivered
configuration.

<tcl>hd_fragment thoughts1</tcl>
<h3>7.6 Experience with full test coverage</h3>

<p>The developers of SQLite have found that full coverage testing is an
extremely effective method for locating and preventing bugs.
Because every single branch
instruction in SQLite core code is covered by test cases, the developers
can be confident that changes they make in one part of the code
do not have unintended consequences in other parts of the code.
The many new features and performance improvements that have been
added to SQLite in recent years would not have been possible without
the availability full-coverage testing.</p>

<p>Maintaining 100% MC/DC is laborious and time-consuming.
The level of effort needed to maintain full-coverage testing
is probably not cost effective for a typical application.
However, we think that full-coverage testing is justified for a
[most widely deployed|very widely deployed] infrastructure library
like SQLite, and especially for a database library which by its very
nature "remembers" past mistakes.


<tcl>hd_fragment dynamicanalysis</tcl>
<h2>8.0 Dynamic Analysis</h2>

<p>Dynamic analysis refers to internal and external checks on the
SQLite code which are performed while the code is live and running.
Dynamic analysis has proven to be a great help in maintaining the