Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
| SHA1 Hash: | 15837d1afc802724beeaadbf0a53b12dcebbbaf2 |
|---|---|
| Date: | 2010-03-09 19:35:32 |
| User: | drh |
| Comment: | Update the testing.html document and the release date of 3.6.23. |
Tags And Properties
- branch=trunk inherited from [b2e03e19ab]
- sym-trunk inherited from [b2e03e19ab]
Changes
Changes to pages/changes.in
38 http://www.sqlite.org/src/timeline</a>.</p> 38 http://www.sqlite.org/src/timeline</a>.</p> 39 } 39 } 40 hd_close_aux 40 hd_close_aux 41 hd_enable_main 1 41 hd_enable_main 1 42 } 42 } 43 } 43 } 44 44 45 chng {2010 March 10 (3.6.23)} { | 45 chng {2010 March 09 (3.6.23)} { 46 <li> Added the [secure_delete pragma]. 46 <li> Added the [secure_delete pragma]. 47 <li> Added the [sqlite3_compileoption_used()] and 47 <li> Added the [sqlite3_compileoption_used()] and 48 [sqlite3_compileoption_get()] interfaces as well as the 48 [sqlite3_compileoption_get()] interfaces as well as the 49 [compile_options pragma] and the [sqlite_compileoption_used()] and 49 [compile_options pragma] and the [sqlite_compileoption_used()] and 50 [sqlite_compileoption_get()] SQL functions. 50 [sqlite_compileoption_get()] SQL functions. 51 <li> Added the [sqlite3_log()] interface together with the 51 <li> Added the [sqlite3_log()] interface together with the 52 [SQLITE_CONFIG_LOG] verb to [sqlite3_config()]. The ".log" command 52 [SQLITE_CONFIG_LOG] verb to [sqlite3_config()]. The ".log" command
Changes to pages/testing.in
7 # facilitate that, all the size values are defined by variables here 7 # facilitate that, all the size values are defined by variables here 8 # which are then used as needed through the document. 8 # which are then used as needed through the document. 9 # 9 # 10 # NOTE: Also update the version number in the text!!! 10 # NOTE: Also update the version number in the text!!! 11 # 11 # 12 12 13 # sloc sqlite3.c 13 # sloc sqlite3.c 14 set stat(coreSLOC) 65745 ;# Non-comment lines of amalgamation code | 14 set stat(coreSLOC) 67215 ;# Non-comment lines of amalgamation code 15 # sloc test*.c 15 # sloc test*.c 16 set stat(tclcSLOC) 14572 ;# Non-comment lines of test C code | 16 set stat(tclcSLOC) 14691 ;# Non-comment lines of test C code 17 # ls test*.c tclsqlite.c | wc 17 # ls test*.c tclsqlite.c | wc 18 set stat(tclcNfile) 32 ;# Number of files of TCL C testcode + tclsqlite.c 18 set stat(tclcNfile) 32 ;# Number of files of TCL C testcode + tclsqlite.c 19 # ls -l test*.c tclsqlite.c | awk '{sum+=$5}END{print sum}' 19 # ls -l test*.c tclsqlite.c | awk '{sum+=$5}END{print sum}' 20 set stat(tclcNByte) 700287 ;# Number of bytes of TCL C testcode + tclsqlite.c | 20 set stat(tclcNByte) 714533 ;# Number of bytes of TCL C testcode + tclsqlite.c 21 # sloc *.test *.tcl 21 # sloc *.test *.tcl 22 set stat(tclsSLOC) 194071 ;# Non-comment lines of TCL test script | 22 set stat(tclsSLOC) 200444 ;# Non-comment lines of TCL test script 23 # ls *.test *.tcl | wc 23 # ls *.test *.tcl | wc 24 set stat(tclsNFile) 511 ;# Number of files of TCL test script | 24 set stat(tclsNFile) 536 ;# Number of files of TCL test script 25 # ls -l *.test *.tcl | awk '{sum+=$5}END{print sum}' 25 # ls -l *.test *.tcl | awk '{sum+=$5}END{print sum}' 26 set stat(tclsNByte) 8298364 ;# Number of bytes of TCL test script | 26 set stat(tclsNByte) 8780683 ;# Number of bytes of TCL test script 27 # grep do_test *.test | wc 27 # grep do_test *.test | wc 28 set stat(tclNTest) 24866 ;# Number of test cases in the TCL test suite | 28 set stat(tclNTest) 25400 ;# Number of test cases in the TCL test suite 29 set stat(tclNEval) 2151154 ;# Number of test case evaluations | 29 set stat(tclNEval) 2204181 ;# Number of test case evaluations 30 set stat(nSqlFuzz) 107457 ;# Number of SQL fuzz tests 30 set stat(nSqlFuzz) 107457 ;# Number of SQL fuzz tests 31 set stat(vqNEval) 42102 ;# Number of test evaluations for veryquick.test | 31 set stat(vqNEval) 98214 ;# Number of test evaluations for veryquick.test 32 set stat(vqStmtCov) 97.23 ;# veryquick statement coverage | 32 # set stat(vqStmtCov) 97.23 ;# veryquick statement coverage 33 set stat(vqBrCov) 92.57 ;# veryquick branch coverage | 33 # set stat(vqBrCov) 92.57 ;# veryquick branch coverage 34 set stat(allStmtCov) 99.50 ;# all.test statement coverage | 34 # set stat(allStmtCov) 99.50 ;# all.test statement coverage 35 set stat(allBrCov) 97.41 ;# all.test condition/decision coverage | 35 # set stat(allBrCov) 97.41 ;# all.test condition/decision coverage 36 # tclsh mkth3.tcl cfg/*.cfg */*.test >th3.c; sloc th3.c 36 # tclsh mkth3.tcl cfg/*.cfg */*.test >th3.c; sloc th3.c 37 set stat(th3SLOC) 340773 ;# Non-comment lines in full th3.c | 37 set stat(th3SLOC) 602888 ;# Non-comment lines in full th3.c 38 # ls -l th3.c 38 # ls -l th3.c 39 set stat(th3NByte) 23689140 ;# Number of bytes in full th3.c | 39 set stat(th3NByte) 45648069 ;# Number of bytes in full th3.c 40 # grep th3testBegin */*.test 40 # grep th3testBegin */*.test 41 # grep th3oomBegin */*.test 41 # grep th3oomBegin */*.test 42 # grep th3ioerrBegin */*.test 42 # grep th3ioerrBegin */*.test 43 # grep '^--testcase' */*.test 43 # grep '^--testcase' */*.test 44 set stat(th3NTest) 12287 ;# Number of test cases | 44 set stat(th3NTest) 29644 ;# Number of test cases 45 # from output of a full test run. 45 # from output of a full test run. > 46 set stat(th3NECov) 1502643 ;# Number of test case evals for coverage 46 set stat(th3NEval) 1355348 ;# Number of test case evaluations | 47 set stat(th3NETest) 1504866 ;# Number of test case evaluations > 48 set stat(th3NEExt) 589175483 ;# Number of test case evals extended > 49 set stat(th3NERel) 2500000000 ;# Number of test case evals release 47 set stat(th3StmtCov) 100.00 ;# TH3 statement coverage 50 set stat(th3StmtCov) 100.00 ;# TH3 statement coverage 48 set stat(th3BrCov) 100.00 ;# TH3 branch coverage 51 set stat(th3BrCov) 100.00 ;# TH3 branch coverage 49 # wc `fossil ls | awk '/\.test$/{print $2}'` 52 # wc `fossil ls | awk '/\.test$/{print $2}'` 50 set stat(sltsSLOC) 44858977 ;# Non-comment lines of SLT test script 53 set stat(sltsSLOC) 44858977 ;# Non-comment lines of SLT test script 51 # ls -l `fossil ls | awk '/\.test$/{print $2}'` | awk '{sum+=$5}END{print sum}' 54 # ls -l `fossil ls | awk '/\.test$/{print $2}'` | awk '{sum+=$5}END{print sum}' 52 set stat(sltsNByte) 1116748159 ;# Bytes of SLT test script 55 set stat(sltsNByte) 1116748159 ;# Bytes of SLT test script 53 # fossil ls | awk '/.test$/{print $2}' | wc 56 # fossil ls | awk '/.test$/{print $2}' | wc ................................................................................................................................................................................ 96 <h1 align="center">How SQLite Is Tested</h1> 99 <h1 align="center">How SQLite Is Tested</h1> 97 100 98 <h2>1.0 Introduction</h2> 101 <h2>1.0 Introduction</h2> 99 102 100 <p>The reliability and robustness of SQLite is achieved in part 103 <p>The reliability and robustness of SQLite is achieved in part 101 by thorough and careful testing.</p> 104 by thorough and careful testing.</p> 102 105 103 <p>As of [version 3.6.19] (all statistics in the report are against that | 106 <p>As of [version 3.6.23] (all statistics in the report are against that 104 release of SQLite), 107 release of SQLite), 105 the SQLite library consists of approximately 108 the SQLite library consists of approximately 106 <tcl>KB {$stat(coreSLOC)}</tcl> KSLOC of C code. 109 <tcl>KB {$stat(coreSLOC)}</tcl> KSLOC of C code. 107 (KSLOC means thousands of "Source Lines Of Code" or, in other words, 110 (KSLOC means thousands of "Source Lines Of Code" or, in other words, 108 lines of code excluding blank lines and comments.) 111 lines of code excluding blank lines and comments.) 109 By comparison, the project has 112 By comparison, the project has 110 <tcl> 113 <tcl> ................................................................................................................................................................................ 139 from the others. 142 from the others. 140 </p> 143 </p> 141 144 142 <ol> 145 <ol> 143 <li><p> 146 <li><p> 144 <tcl>hd_fragment tcl {TCL test suite}</tcl> 147 <tcl>hd_fragment tcl {TCL test suite}</tcl> 145 The <b>TCL Tests</b> are the oldest set of tests for SQLite. 148 The <b>TCL Tests</b> are the oldest set of tests for SQLite. 146 The TCL tests are contained in the same source tree as the | 149 They are contained in the same source tree as the 147 SQLite core and like the SQLite core are in the public domain. The 150 SQLite core and like the SQLite core are in the public domain. The 148 TCL tests are the primary tests used during development. 151 TCL tests are the primary tests used during development. 149 The TCL tests are written using the 152 The TCL tests are written using the 150 [http://www.tcl.tk/ | TCL scripting language]. 153 [http://www.tcl.tk/ | TCL scripting language]. 151 The TCL test harness itself consists of <tcl>KB {$stat(tclcSLOC)}</tcl> KSLOC 154 The TCL test harness itself consists of <tcl>KB {$stat(tclcSLOC)}</tcl> KSLOC 152 of C code used to create the TCL interface. The test scripts are contained 155 of C code used to create the TCL interface. The test scripts are contained 153 in <tcl>N {$stat(tclsNFile)}</tcl> files totaling 156 in <tcl>N {$stat(tclsNFile)}</tcl> files totaling 154 <tcl>MiB {$stat(tclsNByte)}</tcl>MB in size. There are 157 <tcl>MiB {$stat(tclsNByte)}</tcl>MB in size. There are 155 <tcl>N {$stat(tclNTest)}</tcl> distinct test cases, but many of the test 158 <tcl>N {$stat(tclNTest)}</tcl> distinct test cases, but many of the test 156 cases are parameterized and run multiple times (with different parameters) 159 cases are parameterized and run multiple times (with different parameters) 157 so that on a full test run, about <tcl>MB {$stat(tclNEval)}</tcl> million 160 so that on a full test run, about <tcl>MB {$stat(tclNEval)}</tcl> million 158 separate tests are performed and provide | 161 separate tests are performed. 159 <tcl>hd_puts $stat(allStmtCov)</tcl>% statement and < 160 <tcl>hd_puts $stat(allBrCov)</tcl>% branch test coverage. < 161 </p> 162 </p> 162 </li> 163 </li> 163 164 164 <li><p> 165 <li><p> 165 The <b>[TH3]</b> test harness is a set of proprietary tests, written in 166 The <b>[TH3]</b> test harness is a set of proprietary tests, written in 166 C that provide 100% branch test coverage (and 100% MC/DC test coverage) to 167 C that provide 100% branch test coverage (and 100% MC/DC test coverage) to 167 the core SQLite library. The TH3 tests are designed to run 168 the core SQLite library. The TH3 tests are designed to run 168 on embedded and specialized platforms that would not easily support 169 on embedded and specialized platforms that would not easily support 169 TCL or other workstation services. TH3 tests use only the published 170 TCL or other workstation services. TH3 tests use only the published 170 SQLite interfaces. TH3 is free to [SQLite Consortium] members 171 SQLite interfaces. TH3 is free to [SQLite Consortium] members 171 and is available by license to others. TH3 consists of about 172 and is available by license to others. TH3 consists of about 172 <tcl>MB {$stat(th3NByte)}</tcl> MB or <tcl>KB {$stat(th3SLOC)}</tcl> KSLOC 173 <tcl>MB {$stat(th3NByte)}</tcl> MB or <tcl>KB {$stat(th3SLOC)}</tcl> KSLOC 173 of C code implementing <tcl>N {$stat(th3NTest)}</tcl> distinct test cases. 174 of C code implementing <tcl>N {$stat(th3NTest)}</tcl> distinct test cases. 174 TH3 tests are heavily parameterized, though, so a full test runs | 175 TH3 tests are heavily parameterized, though, so a full-coverage test runs 175 about <tcl>MB {$stat(th3NEval)}</tcl> million different test | 176 about <tcl>MB {$stat(th3NECov)}</tcl> million different test > 177 instances. The cases that provide 100% branch test coverage constitute > 178 a subset of the total TH3 test suite. A soak test > 179 prior to release does in excess of <tcl>GB {$stat(th3NERel)}</tcl> billion 176 instances. Additional information on TH3 is | 180 tests. Additional information on TH3 is 177 [TH3 | available separately].</p></li> 181 [TH3 | available separately].</p></li> 178 182 179 <li><p> 183 <li><p> 180 <tcl>hd_fragment slt {SLT} {SQL Logic Tests}</tcl> 184 <tcl>hd_fragment slt {SLT} {SQL Logic Tests}</tcl> 181 The <a href="http://www.sqlite.org/sqllogictest"><b>SQL Logic Test</b></a> 185 The <a href="http://www.sqlite.org/sqllogictest"><b>SQL Logic Test</b></a> 182 or SLT test harness is used to run huge numbers 186 or SLT test harness is used to run huge numbers 183 of SQL statements against both SQLite and several other SQL database engines 187 of SQL statements against both SQLite and several other SQL database engines ................................................................................................................................................................................ 191 <p>All of the tests above must run successfully, on multiple platforms 195 <p>All of the tests above must run successfully, on multiple platforms 192 and under multiple compile-time configurations, 196 and under multiple compile-time configurations, 193 before each release of SQLite.</p> 197 before each release of SQLite.</p> 194 198 195 <p>Prior to each check-in to the SQLite source tree, developers 199 <p>Prior to each check-in to the SQLite source tree, developers 196 typically run a subset (called "veryquick") of the Tcl tests 200 typically run a subset (called "veryquick") of the Tcl tests 197 consisting of about 201 consisting of about 198 <tcl>KB {$stat(vqNEval)}</tcl> thousand test cases | 202 <tcl>KB {$stat(vqNEval)}</tcl> thousand test cases. 199 and covering < 200 <tcl>hd_puts $stat(vqStmtCov)</tcl>% of the core SQLite source code statements. < 201 The veryquick tests covers everything except the anomaly, fuzz, and | 203 The veryquick tests include everything except the anomaly, fuzz, and 202 soak tests. The idea behind the veryquick tests are that they are 204 soak tests. The idea behind the veryquick tests are that they are 203 sufficient to catch most errors, but also run in only a few minutes 205 sufficient to catch most errors, but also run in only a few minutes 204 instead of a few hours.</p> 206 instead of a few hours.</p> 205 207 206 <tcl>hd_fragment anomoly</tcl> 208 <tcl>hd_fragment anomoly</tcl> 207 <h2>3.0 Anomaly Testing</h2> 209 <h2>3.0 Anomaly Testing</h2> 208 210