Documentation Source Text

Check-in [6fc727b510]
Login
Home Timeline Branches Tickets Wiki More...

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Update test metrics on the testing.html page.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256: 6fc727b5107887544e7d6692fd00e1aa996d70353beef08ec0db611d06efa43a
User & Date: drh 2017-07-26 15:06:51
Context
2017-07-26
17:25
Add a news entry. Fix the release date. Fix typos. (check-in: 9759ba8053 user: drh tags: trunk)
15:06
Update test metrics on the testing.html page. (check-in: 6fc727b510 user: drh tags: trunk)
2017-07-25
17:03
Change log typo fixes. (check-in: 0e82b7e2dd user: drh tags: trunk)
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to pages/testing.in. 

9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

47
48
49
50
51
52
53
54
55
56
..
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
...
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
...
165
166
167
168
169
170
171
172

173
174
175
176
177
178
179

# facilitate that, all the size values are defined by variables here
# which are then used as needed through the document.
#
# NOTE:  Also update the version number in the text!!!
#

# sloc sqlite3.c
set stat(coreSLOC)   122905  ;# Non-comment lines of amalgamation code 
# sloc test*.c
set stat(tclcSLOC)    25283  ;# Non-comment lines of test C code
# ls test*.c tclsqlite.c | wc
set stat(tclcNfile)      46  ;# Number of files of TCL C testcode + tclsqlite.c
# ls -l test*.c tclsqlite.c | awk '{sum+=$5}END{print sum}'
set stat(tclcNByte) 1191124  ;# Number of bytes of TCL C testcode + tclsqlite.c
# sloc `find . -name '*.test' -print` test/*.tcl
set stat(tclsSLOC)   306579  ;# Non-comment lines of TCL test script
# ls `find . -name '*.test' -print` test/*.tcl | wc
set stat(tclsNFile)    1080  ;# Number of files of TCL test script
# wc `find . -name '*.test' -print` test/*.tcl
set stat(tclsNByte) 13345897 ;# Number of bytes of TCL test script
# cat `find . -name '*.test' -print` | egrep 'do[_a-z]*_test' | wc
set stat(tclNTest)    39035  ;# Number of test cases in the TCL test suite
set stat(tclNEval)  2486283  ;# Number of test case evaluations (fulltest)
# grep fuzz device_one/test-out.txt | wc
set stat(nSqlFuzz)   105254  ;# Number of SQL fuzz tests
set stat(vqNEval)    135483  ;# Number of test evaluations for veryquick.test
#  set stat(vqStmtCov)   97.23  ;# veryquick statement coverage
#  set stat(vqBrCov)     92.57  ;# veryquick branch coverage
#  set stat(allStmtCov)  99.50  ;# all.test statement coverage
#  set stat(allBrCov)    97.41  ;# all.test condition/decision coverage
# tclsh mkth3.tcl cfg/*.cfg */*.test >th3.c; sloc th3.c
set stat(th3SLOC)    773290  ;# Non-comment lines in full th3.c
# ls -l th3.c
set stat(th3NByte) 56633680  ;# Number of bytes in full th3.c
# grep th3testCheck */*.test
# grep '^--result' */*.test
# grep '^--glob' */*.test
# grep '^--notglob' */*.test

set stat(th3NTest)      41661  ;# Number of test cases
# from output of a min.rc test run.
set stat(th3NECov)    1603083  ;# Number of test case evals for coverage
#set stat(th3NETest)  7247055  ;# Number of test case evaluations
#set stat(th3NEExt) 589175483  ;# Number of test case evals extended
#set stat(th3NERel) 2500000000 ;# Number of test case evals release
set stat(th3StmtCov) 100.00  ;# TH3 statement coverage
set stat(th3BrCov)   100.00  ;# TH3 branch coverage
# wc `find . -name '*.test'` | awk '{x+=$1}END{print x}'
set stat(sltsSLOC)  90489494 ;# Non-comment lines of SLT test script
................................................................................
# sloc md5.c slt_*.c sqllogictest.c
set stat(sltcSLOC)        1404 ;# Non-comment lines of SLT C code
# grep '^query' `fossil ls | awk '/\.test$/{print $2}'` | wc
set stat(sltNTest)     7195342 ;# Number of test cases in SLT
# grep 'errors out of' all-out.txt | awk '{x+=$5}END{print x}'
set stat(sltNRun)     11879758 ;# Number of tests run in SLT
# grep 'assert(' sqlite3.c | wc
set stat(nAssert)         5191 ;# Number of assert statements
# grep 'testcase(' sqlite3.c | grep -v define | wc
set stat(nTestcase)        844 ;# Number of testcase statements

set stat(totalSLOC) [expr {$stat(tclcSLOC)+$stat(tclsSLOC)+
                           $stat(th3SLOC)+$stat(sltcSLOC)+$stat(sltsSLOC)}]

proc GB {expr} {
  set n [uplevel #0 expr $expr]
  hd_puts [format %.2f [expr {$n/(1000.0*1000.0*1000.0)}]]
................................................................................
</tcl>

<h1>Introduction</h1>

<p>The reliability and robustness of SQLite is achieved in part
by thorough and careful testing.</p>

<p>As of [version 3.16.2] ([dateof:3.16.2]),
the SQLite library consists of approximately
<tcl>KB {$stat(coreSLOC)}</tcl> KSLOC of C code.
(KSLOC means thousands of "Source Lines Of Code" or, in other words,
lines of code excluding blank lines and comments.)
By comparison, the project has
<tcl>
hd_puts "[expr {int($stat(totalSLOC)/$stat(coreSLOC))}] times as much"
................................................................................
so that on a full test run millions of
separate tests are performed.
</p>
</li>

<li><p>
The <b>[TH3]</b> test harness is a set of proprietary tests, written in
C that provide 100% branch test coverage (and 100% MC/DC test coverage) to

the core SQLite library.  The TH3 tests are designed to run
on embedded and specialized platforms that would not easily support
TCL or other workstation services.  TH3 tests use only the published 
SQLite interfaces. TH3 consists of about
<tcl>MB {$stat(th3NByte)}</tcl> MB or <tcl>KB {$stat(th3SLOC)}</tcl> KSLOC
of C code implementing <tcl>N {$stat(th3NTest)}</tcl> distinct test cases.
TH3 tests are heavily parameterized, though, so a full-coverage test runs








|

|



|

|

|

|

|
|

|
|





|

|
|
|
|
|
>
|

|







 







|

|







 







|







 







|
>








9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
..
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
...
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
...
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181

# facilitate that, all the size values are defined by variables here
# which are then used as needed through the document.
#
# NOTE:  Also update the version number in the text!!!
#

# sloc sqlite3.c
set stat(coreSLOC)   125375  ;# Non-comment lines of amalgamation code 
# sloc test*.c
set stat(tclcSLOC)    25300  ;# Non-comment lines of test C code
# ls test*.c tclsqlite.c | wc
set stat(tclcNfile)      46  ;# Number of files of TCL C testcode + tclsqlite.c
# ls -l test*.c tclsqlite.c | awk '{sum+=$5}END{print sum}'
set stat(tclcNByte) 1193625  ;# Number of bytes of TCL C testcode + tclsqlite.c
# sloc `find . -name '*.test' -print` test/*.tcl
set stat(tclsSLOC)   317456  ;# Non-comment lines of TCL test script
# ls `find . -name '*.test' -print` test/*.tcl | wc
set stat(tclsNFile)    1135  ;# Number of files of TCL test script
# wc `find . -name '*.test' -print` test/*.tcl
set stat(tclsNByte) 13984314 ;# Number of bytes of TCL test script
# cat `find . -name '*.test' -print` | egrep 'do[_a-z]*_test' | wc
set stat(tclNTest)    39747  ;# Number of test cases in the TCL test suite
set stat(tclNEval)  2504837  ;# Number of test case evaluations (fulltest)
# grep fuzz device_one/test-out.txt | wc
set stat(nSqlFuzz)   111268  ;# Number of SQL fuzz tests
set stat(vqNEval)    142597  ;# Number of test evaluations for veryquick.test
#  set stat(vqStmtCov)   97.23  ;# veryquick statement coverage
#  set stat(vqBrCov)     92.57  ;# veryquick branch coverage
#  set stat(allStmtCov)  99.50  ;# all.test statement coverage
#  set stat(allBrCov)    97.41  ;# all.test condition/decision coverage
# tclsh mkth3.tcl cfg/*.cfg */*.test >th3.c; sloc th3.c
set stat(th3SLOC)    782314  ;# Non-comment lines in full th3.c
# ls -l th3.c
set stat(th3NByte) 57259210  ;# Number of bytes in full th3.c
# grep th3testCheck */*.test |wc
# grep '^--result' */*.test | wc
# grep '^--glob' */*.test | wc
# grep '^--notglob' */*.test | wc
# grep '^--eqp' */*.test | wc
set stat(th3NTest)      42213  ;# Number of test cases
# from output of a min.rc test run.
set stat(th3NECov)    1726070  ;# Number of test case evals for coverage
#set stat(th3NETest)  7247055  ;# Number of test case evaluations
#set stat(th3NEExt) 589175483  ;# Number of test case evals extended
#set stat(th3NERel) 2500000000 ;# Number of test case evals release
set stat(th3StmtCov) 100.00  ;# TH3 statement coverage
set stat(th3BrCov)   100.00  ;# TH3 branch coverage
# wc `find . -name '*.test'` | awk '{x+=$1}END{print x}'
set stat(sltsSLOC)  90489494 ;# Non-comment lines of SLT test script
................................................................................
# sloc md5.c slt_*.c sqllogictest.c
set stat(sltcSLOC)        1404 ;# Non-comment lines of SLT C code
# grep '^query' `fossil ls | awk '/\.test$/{print $2}'` | wc
set stat(sltNTest)     7195342 ;# Number of test cases in SLT
# grep 'errors out of' all-out.txt | awk '{x+=$5}END{print x}'
set stat(sltNRun)     11879758 ;# Number of tests run in SLT
# grep 'assert(' sqlite3.c | wc
set stat(nAssert)         5285 ;# Number of assert statements
# grep 'testcase(' sqlite3.c | grep -v define | wc
set stat(nTestcase)        846 ;# Number of testcase statements

set stat(totalSLOC) [expr {$stat(tclcSLOC)+$stat(tclsSLOC)+
                           $stat(th3SLOC)+$stat(sltcSLOC)+$stat(sltsSLOC)}]

proc GB {expr} {
  set n [uplevel #0 expr $expr]
  hd_puts [format %.2f [expr {$n/(1000.0*1000.0*1000.0)}]]
................................................................................
</tcl>

<h1>Introduction</h1>

<p>The reliability and robustness of SQLite is achieved in part
by thorough and careful testing.</p>

<p>As of [version 3.20.0] ([dateof:3.20.0]),
the SQLite library consists of approximately
<tcl>KB {$stat(coreSLOC)}</tcl> KSLOC of C code.
(KSLOC means thousands of "Source Lines Of Code" or, in other words,
lines of code excluding blank lines and comments.)
By comparison, the project has
<tcl>
hd_puts "[expr {int($stat(totalSLOC)/$stat(coreSLOC))}] times as much"
................................................................................
so that on a full test run millions of
separate tests are performed.
</p>
</li>

<li><p>
The <b>[TH3]</b> test harness is a set of proprietary tests, written in
C that provide 100% branch test coverage 
(and [MC/DC|100% MC/DC test coverage]) to
the core SQLite library.  The TH3 tests are designed to run
on embedded and specialized platforms that would not easily support
TCL or other workstation services.  TH3 tests use only the published 
SQLite interfaces. TH3 consists of about
<tcl>MB {$stat(th3NByte)}</tcl> MB or <tcl>KB {$stat(th3SLOC)}</tcl> KSLOC
of C code implementing <tcl>N {$stat(th3NTest)}</tcl> distinct test cases.
TH3 tests are heavily parameterized, though, so a full-coverage test runs

This page was generated in about 0.008s by Fossil 2.13 [6c2b5bafb6] 2020-10-20 23:40:13