Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
Comment: | Fix typos in the testing document. Change "insure" to "ensure" throughout the website. Ticket [6afbaac77a52917fc]. |
---|---|
Downloads: | Tarball | ZIP archive |
Timelines: | family | ancestors | descendants | both | trunk |
Files: | files | file ages | folders |
SHA1: |
8c1658c1e2cc9add8c9003a60b544344 |
User & Date: | drh 2010-05-08 14:05:04.000 |
Context
2010-05-24
| ||
22:39 | Tweaks to the wal.html document. (check-in: 34cbc3f991 user: drh tags: trunk) | |
2010-05-08
| ||
14:05 | Fix typos in the testing document. Change "insure" to "ensure" throughout the website. Ticket [6afbaac77a52917fc]. (check-in: 8c1658c1e2 user: drh tags: trunk) | |
2010-05-07
| ||
16:18 | Fleshing out the WAL documentation. (check-in: a7c7d3a520 user: drh tags: trunk) | |
Changes
Changes to pages/34to35.in.
︙ | ︙ | |||
400 401 402 403 404 405 406 | PARAGRAPH { The xDelete method is used delete a file. The name of the file is given in the second parameter. The filename will be in UTF-8. The VFS must convert the filename into whatever character representation the underlying operating system expects. If the syncDir parameter is true, then the xDelete method should not return until the change to the directory contents for the directory containing the | | | 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 | PARAGRAPH { The xDelete method is used delete a file. The name of the file is given in the second parameter. The filename will be in UTF-8. The VFS must convert the filename into whatever character representation the underlying operating system expects. If the syncDir parameter is true, then the xDelete method should not return until the change to the directory contents for the directory containing the deleted file have been synced to disk in order to ensure that the file does not "reappear" if a power failure occurs soon after. } PARAGRAPH { The xAccess method is used to check for access permissions on a file. The filename will be UTF-8 encoded. The flags argument will be [SQLITE_ACCESS_EXISTS] to check for the existence of the file, |
︙ | ︙ |
Changes to pages/35to36.in.
︙ | ︙ | |||
201 202 203 204 205 206 207 | NOT IN operators to conform to the standard and to give the same results as other SQL database engines. } INDENTED { <strong>Key Point:</strong> The change to the way NULL values are handled by the IN and NOT IN operators is technically a bug fix, not a design | | | 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 | NOT IN operators to conform to the standard and to give the same results as other SQL database engines. } INDENTED { <strong>Key Point:</strong> The change to the way NULL values are handled by the IN and NOT IN operators is technically a bug fix, not a design change. However, maintainers should check to ensure that applications do not depend on the older, buggy behavior prior to upgrading to version 3.6.0. } HEADING 2 {Changes To Column Naming Rules} PARAGRAPH { |
︙ | ︙ |
Changes to pages/atomiccommit.in.
︙ | ︙ | |||
376 377 378 379 380 381 382 | <br clear="both"> <a name="section_3_10"></a> <h3>3.10 Flushing Changes To Mass Storage</h3> <img src="images/ac/commit-9.gif" align="right" hspace="15"> <p>Another flush must occur to make sure that all the database changes are written into nonvolatile storage. | | | 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 | <br clear="both"> <a name="section_3_10"></a> <h3>3.10 Flushing Changes To Mass Storage</h3> <img src="images/ac/commit-9.gif" align="right" hspace="15"> <p>Another flush must occur to make sure that all the database changes are written into nonvolatile storage. This is a critical step to ensure that the database will survive a power loss without damage. However, because of the inherent slowness of writing to disk or flash memory, this step together with the rollback journal file flush in section 3.7 above takes up most the time required to complete a transaction commit in SQLite.</p> <br clear="both"> |
︙ | ︙ |
Changes to pages/faq.in.
︙ | ︙ | |||
415 416 417 418 419 420 421 | <p>During testing, the SQLite library is compiled with special instrumentation that allows the test scripts to simulate a wide variety of failures in order to verify that SQLite recovers correctly. Memory allocation is carefully tracked and no memory leaks occur, even following memory allocation failures. A custom VFS layer is used to simulate operating system crashes and power | | | 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 | <p>During testing, the SQLite library is compiled with special instrumentation that allows the test scripts to simulate a wide variety of failures in order to verify that SQLite recovers correctly. Memory allocation is carefully tracked and no memory leaks occur, even following memory allocation failures. A custom VFS layer is used to simulate operating system crashes and power failures in order to ensure that transactions are atomic across these events. A mechanism for deliberately injecting I/O errors shows that SQLite is resilient to such malfunctions. (As an experiment, try inducing these kinds of errors on other SQL database engines and see what happens!)</p> <p>We also run SQLite using [http://valgrind.org | valgrind] on Linux and verify that it detects no problems.</p> |
︙ | ︙ |
Changes to pages/malloc.in.
︙ | ︙ | |||
107 108 109 110 111 112 113 | <a name="testing"></a> <h2>2.0 Testing</h2> <p>Over 75% of the code in the SQLite source tree is devoted purely to [testing | testing and verification]. Reliability is important to SQLite. | | | 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 | <a name="testing"></a> <h2>2.0 Testing</h2> <p>Over 75% of the code in the SQLite source tree is devoted purely to [testing | testing and verification]. Reliability is important to SQLite. Among the tasks of the test infrastructure is to ensure that SQLite does not misuse dynamically allocated memory, that SQLite does not leak memory, and that SQLite responds correctly to a dynamic memory allocation failure.</p> <p>The test infrastructure verifies that SQLite does not misuse dynamically allocated memory by using a specially instrumented memory allocator. The instrumented memory allocator is enabled |
︙ | ︙ | |||
129 130 131 132 133 134 135 | <li><p><b>Bounds checking.</b> The instrumented memory allocator places sentinel values at both ends of each memory allocation to verify that nothing within SQLite writes outside the bounds of the allocation.</p></li> <li><p><b>Use of memory after freeing.</b> When each block of memory is freed, every byte is overwritten with a | | | | | 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 | <li><p><b>Bounds checking.</b> The instrumented memory allocator places sentinel values at both ends of each memory allocation to verify that nothing within SQLite writes outside the bounds of the allocation.</p></li> <li><p><b>Use of memory after freeing.</b> When each block of memory is freed, every byte is overwritten with a nonsense bit pattern. This helps to ensure that no memory is ever used after having been freed.</p></li> <li><p><b>Freeing memory not obtained from malloc.</b> Each memory allocation from the instrumented memory allocator contains sentinels used to verify that every allocation freed came from prior malloc.</p></li> <li><p><b>Uninitialized memory.</b> The instrumented memory allocator initializes each memory allocation to a nonsense bit pattern to help ensure that the user makes no assumptions about the content of allocation memory.</p></li> </ul> <p>Regardless of whether or not the instrumented memory allocator is used, SQLite keeps track of how much memory is currently checked out. There are hundreds of test scripts used for testing SQLite. At the end of each script, all objects are destroyed and a test is made to ensure that all memory has been freed. This is how memory leaks are detected. Notice that memory leak detection is in force at all times, during test builds and during production builds. Whenever one of the developers runs any individual test script, memory leak detection is active. Hence memory leaks that do arise during development are quickly detected and fixed.</p> <a name="oomtesting"></a> |
︙ | ︙ | |||
243 244 245 246 247 248 249 | the current test case. These latter two features are useful for tracking down the source of memory leaks detected by the test suite.</p> <p>The heavy wrapper that is used when [SQLITE_MEMDEBUG] is set also makes sure each new allocation is filled with nonsense data prior to returning the allocation to the caller. And as soon as an allocation is free, it is again filled with nonsense data. These two actions help | | | 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 | the current test case. These latter two features are useful for tracking down the source of memory leaks detected by the test suite.</p> <p>The heavy wrapper that is used when [SQLITE_MEMDEBUG] is set also makes sure each new allocation is filled with nonsense data prior to returning the allocation to the caller. And as soon as an allocation is free, it is again filled with nonsense data. These two actions help to ensure that the SQLite core does not make assumptions about the state of newly allocated memory and that memory allocations are not used after they have been freed.</p> <p>The heavy wrapper employed by [SQLITE_MEMDEBUG] is intended for use only during testing, analysis, and debugging of SQLite. The heavy wrapper has a significant performance and memory overhead and probably should not be used in production.</p> |
︙ | ︙ |
Changes to pages/testing.in.
︙ | ︙ | |||
208 209 210 211 212 213 214 | <tcl>hd_fragment anomoly</tcl> <h2>3.0 Anomaly Testing</h2> <p>Anomaly tests are tests designed to verify the correct behavior of SQLite when something goes wrong. It is (relatively) easy to build an SQL database engine that behaves correctly on well-formed inputs on a fully functional computer. It is more difficult to build a system | | | 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 | <tcl>hd_fragment anomoly</tcl> <h2>3.0 Anomaly Testing</h2> <p>Anomaly tests are tests designed to verify the correct behavior of SQLite when something goes wrong. It is (relatively) easy to build an SQL database engine that behaves correctly on well-formed inputs on a fully functional computer. It is more difficult to build a system that responds sanely to invalid inputs and continues to function following system malfunctions. The anomaly tests are designed to verify the latter behavior.</p> <tcl>hd_fragment oomtesting</tcl> <h3>3.1 Out-Of-Memory Testing</h3> <p>SQLite, like all SQL database engines, makes extensive use of |
︙ | ︙ | |||
324 325 326 327 328 329 330 | The interior of the loop is repeated multiple times for each snapshot with different random damage each time.</p> <tcl>hd_fragment multifail</tcl> <h3>3.4 Compound failure tests</h3> <p>The test suites for SQLite also explore the result of stacking | | | | 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 | The interior of the loop is repeated multiple times for each snapshot with different random damage each time.</p> <tcl>hd_fragment multifail</tcl> <h3>3.4 Compound failure tests</h3> <p>The test suites for SQLite also explore the result of stacking multiple failures. For example, tests are run to ensure correct behavior when an I/O error or OOM fault occurs while trying to recover from a prior crash. <tcl>hd_fragment fuzztesting</tcl> <h2>4.0 Fuzz Testing</h2> <p>[http://en.wikipedia.org/wiki/Fuzz_testing | Fuzz testing] seeks to establish that SQLite responds correctly to invalid, out-of-range, or malformed inputs.</p> <h3>4.1 SQL Fuzz</h3> <p>SQL fuzz testing consists of creating syntactically correct yet wildly nonsensical SQL statements and feeding them to SQLite to see what it will do with them. Usually some kind of error is returned |
︙ | ︙ | |||
388 389 390 391 392 393 394 | <tcl>hd_fragment regressiontesting</tcl> <h2>5.0 Regression Testing</h2> <p>Whenever a bug is reported against SQLite, that bug is not considered fixed until new test cases have been added to the TCL test suite which would exhibit the bug in an unpatched version of SQLite. Over the years, this has resulted in thousands and thousands of new tests being added | | | | | 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 | <tcl>hd_fragment regressiontesting</tcl> <h2>5.0 Regression Testing</h2> <p>Whenever a bug is reported against SQLite, that bug is not considered fixed until new test cases have been added to the TCL test suite which would exhibit the bug in an unpatched version of SQLite. Over the years, this has resulted in thousands and thousands of new tests being added to the TCL test suite. These regression tests ensure that bugs that have been fixed in the past are not reintroduced into future versions of SQLite.</p> <tcl>hd_fragment leakcheck</tcl> <h2>6.0 Automatic Resource Leak Detection</h2> <p>Resource leak occurs when system resources are allocated and never freed. The most troublesome resource leaks in many applications are memory leaks - when memory is allocated using malloc() but never released using free(). But other kinds of resources can also be leaked: file descriptors, threads, mutexes, etc.</p> <p>Both the TCL and TH3 test harnesses automatically track system resources and report resource leaks on <u>every</u> test run. No special configuration or setup is required. The test harnesses are especially vigilant with regard to memory leaks. If a change causes a memory leak, the test harnesses will recognize this quickly. SQLite is designed to never leak memory, even after an exception such as an OOM error or disk I/O error. The test harnesses are zealous to enforce this.</p> |
︙ | ︙ | |||
433 434 435 436 437 438 439 | <p>There are many ways to measure test coverage. The most popular metric is "statement coverage". When you hear someone say that their program as "XX% test coverage" without further explanation, they usually mean statement coverage. Statement coverage measures what percentage of lines of code are executed at least once by the test suite.</p> <p>Branch coverage is more rigorous than statement coverage. Branch | | | 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 | <p>There are many ways to measure test coverage. The most popular metric is "statement coverage". When you hear someone say that their program as "XX% test coverage" without further explanation, they usually mean statement coverage. Statement coverage measures what percentage of lines of code are executed at least once by the test suite.</p> <p>Branch coverage is more rigorous than statement coverage. Branch coverage measures the number of machine-code branch instructions that are evaluated at least once on both directions.</p> <p>To illustrate the difference between statement coverage and branch coverage, consider the following hypothetical line of C code:</p> <blockquote><pre> |
︙ | ︙ | |||
524 525 526 527 528 529 530 | <tcl>hd_fragment {testcase} {testcase macros}</tcl> <h3>7.3 Forcing coverage of boundary values and boolean vector tests</h3> <p>Another macro used in conjunction with test coverage measurement is the <tt>testcase()</tt> macro. The argument is a condition for which we want test cases that evaluate to both true and false. | | | | | | | | | 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 | <tcl>hd_fragment {testcase} {testcase macros}</tcl> <h3>7.3 Forcing coverage of boundary values and boolean vector tests</h3> <p>Another macro used in conjunction with test coverage measurement is the <tt>testcase()</tt> macro. The argument is a condition for which we want test cases that evaluate to both true and false. In non-coverage builds (that is to say, in release builds) the <tt>testcase()</tt> macro is a no-op:</p> <blockquote><pre> #define testcase(X) </pre></blockquote> <p>But in a coverage measuring build, the <tt>testcase()</tt> macro generates code that evaluates the conditional expression in its argument. Then during analysis, a check is made to ensure tests exist that evaluate the conditional to both true and false. <tt>Testcase()</tt> macros are used, for example, to help verify that boundary values are tested. For example:</p> <blockquote><pre> testcase( a==b ); testcase( a==b+1 ); if( a>b && c!=25 ){ d++; } </pre></blockquote> |
︙ | ︙ | |||
561 562 563 564 565 566 567 | /* ... */ break; } /* ... */ } </pre></blockquote> | | | | > | | 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 | /* ... */ break; } /* ... */ } </pre></blockquote> <p>For bitmask tests, <tt>testcase()</tt> macros are used to verify that every bit of the bitmask affects the test. For example, in the following block of code, the condition is true if the mask contains either of two bits indicating either a MAIN_DB or a TEMP_DB is being opened. The <tt>testcase()</tt> macros that precede the if statement verify that both cases are tested:</p> <blockquote><pre> testcase( mask & SQLITE_OPEN_MAIN_DB ); testcase( mask & SQLITE_OPEN_TEMP_DB ); if( (mask & (SQLITE_OPEN_MAIN_DB|SQLITE_OPEN_TEMP_DB))!=0 ){ ... } </pre></blockquote> <p>The SQLite source code contains <tcl>N {$stat(nTestcase)}</tcl> uses of the <tt>testcase()</tt> macro.</p> <tcl>hd_fragment {mcdc} *MC/DC {MC/DC testing}</tcl> <h3>7.4 Branch coverage versus MC/DC</h3> <p>Two methods of measuring test coverage were described above: "statement" and "branch" coverage. There are many other test coverage metrics besides these two. Another popular metric is "Modified |
︙ | ︙ | |||
654 655 656 657 658 659 660 | <h3>8.2 Valgrind</h3> <p>[http://valgrind.org/ | Valgrind] is perhaps the most amazing and useful developer tool in the world. Valgrind is a simulator - it simulates an x86 running a linux binary. (Ports of valgrind for platforms other than linux are in development, but as of this writing, valgrind only works reliably on linux, which in the opinion of the SQLite developers | | | 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 | <h3>8.2 Valgrind</h3> <p>[http://valgrind.org/ | Valgrind] is perhaps the most amazing and useful developer tool in the world. Valgrind is a simulator - it simulates an x86 running a linux binary. (Ports of valgrind for platforms other than linux are in development, but as of this writing, valgrind only works reliably on linux, which in the opinion of the SQLite developers means that linux should be the preferred platform for all software development.) As valgrind runs a linux binary, it looks for all kinds of interesting errors such as array overruns, reading from uninitialized memory, stack overflows, memory leaks, and so forth. Valgrind finds problems that can easily slip through all of the other tests run against SQLite. And, when valgrind does find an error, it can dump the developer directly into a symbolic debugger at the exact point where the error occur, to facilitate a quick fix.</p> |
︙ | ︙ | |||
699 700 701 702 703 704 705 | in SQLite to verify mutexes are held and released at all the right moments, in order to double-check that SQLite does work correctly in multi-threaded applications.</p> <tcl>hd_fragment journaltest</tcl> <h3>8.5 Journal Tests</h3> | | | 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 | in SQLite to verify mutexes are held and released at all the right moments, in order to double-check that SQLite does work correctly in multi-threaded applications.</p> <tcl>hd_fragment journaltest</tcl> <h3>8.5 Journal Tests</h3> <p>One of the things that SQLite does to ensure that transactions are atomic across system crashes and power failures is to write all changes into the rollback journal file prior to changing the database. The TCL test harness contains an alternative [sqlite3_vfs | Virtual File System ] implementation that helps to verify this is occurring correctly. The "journal-test VFS" monitors all disk I/O traffic between the database file and rollback journal, checking to make sure that nothing is written into the database |
︙ | ︙ | |||
744 745 746 747 748 749 750 | excellent job of finding the bugs that are often introduced when removing compiler warnings, so that product quality is probably not decreased as a result.</p> <tcl>hd_fragment summary</tcl> <h2>10.0 Summary</h2> | | | 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 | excellent job of finding the bugs that are often introduced when removing compiler warnings, so that product quality is probably not decreased as a result.</p> <tcl>hd_fragment summary</tcl> <h2>10.0 Summary</h2> <p>SQLite is open source. This gives many people the idea that it is not well tested as commercial software and is perhaps unreliable. But that impression is false. SQLite has exhibited very high reliability in the field and a very low defect rate, especially considering how rapidly it is evolving. The quality of SQLite is achieved in part by careful code design and implementation. But extensive testing also plays a vital role in maintaining and improving the quality of SQLite. This document has summarized the testing procedures that every release of SQLite undergoes with the hopes of inspiring the reader to understand that SQLite is suitable for use in mission-critical applications.</p> |