Documentation Source Text

Check-in [41a06717b5]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Fix typos in the testing.html document.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 41a06717b5ff5f0345c9b57151c02c7f43ee0256
User & Date: drh 2009-08-24 23:05:28.000
Context
2009-08-24
23:31
Fix typos in malloc.html. (check-in: 292c7157b4 user: drh tags: trunk)
23:05
Fix typos in the testing.html document. (check-in: 41a06717b5 user: drh tags: trunk)
14:50
Begin updating the change history in preparation for 3.6.18. Modify the download page to reference the new fossil repositories. Link in the www2.sqlite.org backup website. (check-in: 9f9705ea83 user: drh tags: trunk)
Changes
Unified Diff Ignore Whitespace Patch
Changes to pages/testing.in.
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
completed successfully or else were completely rolled back.  The
[integrity_check] [PRAGMA] is used to make sure no database corruption
occurs.</p>

<p>The TH3 test harness needs to run on embedded systems that do not
necessarily have the ability to spawn child processes, so it uses
an in-memory VFS to simulate crashes.  The in-memory VFS can be rigged
to make snapshot of the entire filesystem after a set number of I/O
operations.  Crash tests run in a loop.  On each iteration of the loop,
the point at which a snapshot is made is advanced until the SQLite
operations being tested run to completion without ever hitting a
snapshot.  Within the loop, after the SQLite operation under test has
completed, the filesystem is reverted to the snapshot and random file
damage is introduced that is characteristic of the kinds of damage
one expects to see following a power loss.  Then the database is opened







|







279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
completed successfully or else were completely rolled back.  The
[integrity_check] [PRAGMA] is used to make sure no database corruption
occurs.</p>

<p>The TH3 test harness needs to run on embedded systems that do not
necessarily have the ability to spawn child processes, so it uses
an in-memory VFS to simulate crashes.  The in-memory VFS can be rigged
to make a snapshot of the entire filesystem after a set number of I/O
operations.  Crash tests run in a loop.  On each iteration of the loop,
the point at which a snapshot is made is advanced until the SQLite
operations being tested run to completion without ever hitting a
snapshot.  Within the loop, after the SQLite operation under test has
completed, the filesystem is reverted to the snapshot and random file
damage is introduced that is characteristic of the kinds of damage
one expects to see following a power loss.  Then the database is opened
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
programming dilemma:  Does one remove defensive code in order to obtain
100% branch coverage?</p>

<p>In SQLite, the answer to the previous question is "no".
For testing purposes, the SQLite source code defines
macros called ALWAYS() and NEVER().   The ALWAYS() macro
surrounds conditions
which are expected to always evaluated to true and NEVER() surrounds
conditions that are always evaluate to false.  These macros serve as
comments to indicate that the conditions are defensive code.
For standard builds, these macros are pass-throughs:</p>

<blockquote><pre>
#define ALWAYS(X)  (X)
#define NEVER(X)   (X)
</pre></blockquote>







|
|







447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
programming dilemma:  Does one remove defensive code in order to obtain
100% branch coverage?</p>

<p>In SQLite, the answer to the previous question is "no".
For testing purposes, the SQLite source code defines
macros called ALWAYS() and NEVER().   The ALWAYS() macro
surrounds conditions
which are expected to always evaluate as true and NEVER() surrounds
conditions that are always evaluated to false.  These macros serve as
comments to indicate that the conditions are defensive code.
For standard builds, these macros are pass-throughs:</p>

<blockquote><pre>
#define ALWAYS(X)  (X)
#define NEVER(X)   (X)
</pre></blockquote>
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591

<p>The developers of SQLite have found that full coverage testing is an
extremely productive method for preventing the introduction of new bugs
as the system evolves.  Because every single branch
instruction in SQLite core code is covered by test cases, the developers
can be confident that changes they make in one part of the code
do not have unintended consequences in other parts of the code.
It would extremely difficult maintain the quality of SQLite without such
assurances.</p>

<h2>8.0 Dynamic Analysis</h2>

<p>Dynamic analysis refers to internal and external checks on the
SQLite code which are performed while the code is live and running.
Dynamic analysis has proven to be a great help in maintaining the







|







577
578
579
580
581
582
583
584
585
586
587
588
589
590
591

<p>The developers of SQLite have found that full coverage testing is an
extremely productive method for preventing the introduction of new bugs
as the system evolves.  Because every single branch
instruction in SQLite core code is covered by test cases, the developers
can be confident that changes they make in one part of the code
do not have unintended consequences in other parts of the code.
It would be extremely difficult maintain the quality of SQLite without such
assurances.</p>

<h2>8.0 Dynamic Analysis</h2>

<p>Dynamic analysis refers to internal and external checks on the
SQLite code which are performed while the code is live and running.
Dynamic analysis has proven to be a great help in maintaining the
627
628
629
630
631
632
633
634

635
636
637
638
639
640
641
642
643
<p>Because it is a simulator, running a binary in valgrind is slower than 
running it on native hardware.  So it is impractical to run the full
SQLite test suite through valgrind.  However, the veryquick tests and
a subset of the TH3 tests are run through valgrind prior to every release.</p>

<h3>8.3 Memsys2</h3>

<p>SQLite contains a pluggable memory allocation subsystem.

The default implementation uses system malloc() and free(). 
However, if SQLite is compiled with SQLITE_MEMDEBUG, an alternative
memory allocation wrapper ([memsys2])
is inserted that looks for memory allocation
errors at run-time.  The memsys2 wrapper checks for memory leaks, of
course, but also looks for buffer overruns, uses of uninitialized memory,
and attempts to use memory after it has been freed.  These same checks
are also done by valgrind (and, indeed, valgrind does them better)
but memsys2 has the advantage of being much faster than valgrind, which







|
>

|







627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
<p>Because it is a simulator, running a binary in valgrind is slower than 
running it on native hardware.  So it is impractical to run the full
SQLite test suite through valgrind.  However, the veryquick tests and
a subset of the TH3 tests are run through valgrind prior to every release.</p>

<h3>8.3 Memsys2</h3>

<p>SQLite contains a pluggable
[memory allocation | memory allocation subsystem].
The default implementation uses system malloc() and free(). 
However, if SQLite is compiled with [SQLITE_MEMDEBUG], an alternative
memory allocation wrapper ([memsys2])
is inserted that looks for memory allocation
errors at run-time.  The memsys2 wrapper checks for memory leaks, of
course, but also looks for buffer overruns, uses of uninitialized memory,
and attempts to use memory after it has been freed.  These same checks
are also done by valgrind (and, indeed, valgrind does them better)
but memsys2 has the advantage of being much faster than valgrind, which
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
are atomic across system crashes and power failures is to write
all changes into the rollback journal file prior to changing the
database.  The TCL test harness contains an alternative
[sqlite3_vfs | Virtual File System ] implementation that helps to
verify this is occurring correctly.  The "journal-test VFS" monitors
all disk I/O traffic between the database file and rollback journal,
checking to make sure that nothing is written into the database
file which has not first by written and synced to the rollback journal.
If any discrepancies are found, an assertion fault is raised.</p>

<p>The journal tests are an additional double-check over and above
the crash tests to make sure that SQLite transactions will be atomic
across system crashes and power failures.</p>

<h2>9.0 Static Analysis</h2>







|







661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
are atomic across system crashes and power failures is to write
all changes into the rollback journal file prior to changing the
database.  The TCL test harness contains an alternative
[sqlite3_vfs | Virtual File System ] implementation that helps to
verify this is occurring correctly.  The "journal-test VFS" monitors
all disk I/O traffic between the database file and rollback journal,
checking to make sure that nothing is written into the database
file which has not first been written and synced to the rollback journal.
If any discrepancies are found, an assertion fault is raised.</p>

<p>The journal tests are an additional double-check over and above
the crash tests to make sure that SQLite transactions will be atomic
across system crashes and power failures.</p>

<h2>9.0 Static Analysis</h2>
Changes to pages/th3.in.
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
linked against the SQLite library under test.  The generated test
program is compiled and run on the target platform in order to verify
correct operation of SQLite on that platform.</p>

<p>The inputs to TH3 are test modules written in C or SQL and
small configuration
files that determine how to initialize SQLite.  The
TH3 package eventually will include hundreds of test
modules and dozens of configuration files.  New modules and configurations
can be added to customize TH3 for specialized applications.
Each time TH3 is run, it reads
a subset of the available test modules and configuration files to generate
a custom C program the performs all of the specified tests under all
configurations.  A complete test of SQLite normally involves running
TH3 multiple times to generate multiple test programs covering different
aspects of SQLite's operation, then linking all test programs against
a common SQLite library and running them separately on the target platform.
SQLite will be found to work if all test programs pass.</p>

<p>There are no arbitrary limits in TH3.  One could generate a
single test program that contained all test modules and configuration files.
However, such a test program might be too large to deploy on embedded
platforms.  Hence, TH3 provides the ability to break the library of test
modules up into smaller, more easily digested pieces.</p>

<p>Each individual test module might contain dozens, hundreds, or thousands
of separate tests.  The test modules can be written in C or as scripts of
SQL.
Test modules done SQL are very easy to write and
the test modules written in C are not as nearly as cumbersome 
to write as one might suppose.  The TH3
system provides high-level interfaces that simplify test writing.
A typical C-language test case in TH3 might contains slightly more syntax that 
the corresponding TCL-script test, but the difference is not that great.
The test modules written as SQL contain special comments that define
the boundaries and operation of each test case and the expected results.</p>







|




















|







40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
linked against the SQLite library under test.  The generated test
program is compiled and run on the target platform in order to verify
correct operation of SQLite on that platform.</p>

<p>The inputs to TH3 are test modules written in C or SQL and
small configuration
files that determine how to initialize SQLite.  The
TH3 package includes hundreds of test
modules and dozens of configuration files.  New modules and configurations
can be added to customize TH3 for specialized applications.
Each time TH3 is run, it reads
a subset of the available test modules and configuration files to generate
a custom C program the performs all of the specified tests under all
configurations.  A complete test of SQLite normally involves running
TH3 multiple times to generate multiple test programs covering different
aspects of SQLite's operation, then linking all test programs against
a common SQLite library and running them separately on the target platform.
SQLite will be found to work if all test programs pass.</p>

<p>There are no arbitrary limits in TH3.  One could generate a
single test program that contained all test modules and configuration files.
However, such a test program might be too large to deploy on embedded
platforms.  Hence, TH3 provides the ability to break the library of test
modules up into smaller, more easily digested pieces.</p>

<p>Each individual test module might contain dozens, hundreds, or thousands
of separate tests.  The test modules can be written in C or as scripts of
SQL.
Test modules done in SQL are very easy to write and
the test modules written in C are not as nearly as cumbersome 
to write as one might suppose.  The TH3
system provides high-level interfaces that simplify test writing.
A typical C-language test case in TH3 might contains slightly more syntax that 
the corresponding TCL-script test, but the difference is not that great.
The test modules written as SQL contain special comments that define
the boundaries and operation of each test case and the expected results.</p>
137
138
139
140
141
142
143
144
145
146
147
148
a license to access and use TH3.</p>

<p>Licensees of TH3 are given read access to the software configuration
management system used to manage TH3 and so can download the latest version
of TH3 (or any historical version) whenever they like.</p>

<p>Even though open-source users do not have direct access to TH3, all
users of SQLite benefit from TH3 indirectly since version of SQLite is
validated by TH3 prior to release.  So anyone using an official release
of SQLite can deploy their application with the confidence of knowing that
it has been tested using TH3.  They simply cannot rerun those tests
themselves without purchasing a TH3 license.</p>







|




137
138
139
140
141
142
143
144
145
146
147
148
a license to access and use TH3.</p>

<p>Licensees of TH3 are given read access to the software configuration
management system used to manage TH3 and so can download the latest version
of TH3 (or any historical version) whenever they like.</p>

<p>Even though open-source users do not have direct access to TH3, all
users of SQLite benefit from TH3 indirectly since each version of SQLite is
validated by TH3 prior to release.  So anyone using an official release
of SQLite can deploy their application with the confidence of knowing that
it has been tested using TH3.  They simply cannot rerun those tests
themselves without purchasing a TH3 license.</p>