Documentation Source Text

     frequent changes in and out of WAL mode and
     VACUUM that could (in theory) cause database corruption.
<li> Enhance the [sqlite3_trace()] mechanism so that nested SQL statements
     such as might be generated by virtual tables are shown but are shown
     in comments and without parameter expansion.  This 
     greatly improves tracing output when using the FTS3/4 and/or RTREE
     virtual tables.







}

chng {2010 December 08 (3.7.4)} {
<li> Added the [sqlite3_blob_reopen()] interface to allow an existing
     [sqlite3_blob] object to be rebound to a new row.
<li> Use the new [sqlite3_blob_reopen()] interface to improve the performance
     of FTS.

     frequent changes in and out of WAL mode and
     VACUUM that could (in theory) cause database corruption.
<li> Enhance the [sqlite3_trace()] mechanism so that nested SQL statements
     such as might be generated by virtual tables are shown but are shown
     in comments and without parameter expansion.  This 
     greatly improves tracing output when using the FTS3/4 and/or RTREE
     virtual tables.
<li> Change the xFileControl() methods on all built-in VFSes to return
     [SQLITE_NOTFOUND] instead of [SQLITE_ERROR] for an unrecognized
     operation code.
<li> The SQLite core invokes the [SQLITE_FCNTL_SYNC_OMITTED] 
     [sqlite3_file_control | file control]
     to the VFS in place of a call to xSync if the database has
     [PRAGMA synchronous] set to OFF.
}

chng {2010 December 08 (3.7.4)} {
<li> Added the [sqlite3_blob_reopen()] interface to allow an existing
     [sqlite3_blob] object to be rebound to a new row.
<li> Use the new [sqlite3_blob_reopen()] interface to improve the performance
     of FTS.

    <a href="http://www.mozilla.com/">\
    <img src="images/foreignlogos/mozilla.gif" alt="mozilla.com" border="0">\
    </a></td><td>\
    <a href="http://www.mozilla.com/">Mozilla</a> - Working to preserve \
    choice and innovation on the internet.</td></tr>'

  sponsor[1] = '<tr><td align="center">\
    <a href="http://www.symbian.com/">\
    <img src="images/foreignlogos/symbian.gif" alt="symbian.com" border="0">\
    </a></td><td>\
    <a href="http://www.symbian.com/">Symbian</a> - \
    The market-leading open operating system for advanced \
    data-enabled smartphones.</td></tr>'

  sponsor[2] = '<tr><td align="center">\
    <a href="http://www.bloomberg.com/">\
   <img src="images/foreignlogos/bloomberg.gif" alt="bloomberg.com" border="0">\
    </a></td><td>\
    <a href="http://www.bloomberg.com/">Bloomberg</a> - \
    A world leader in financial-information technology.</td></tr>'

  sponsor[3] = '<tr><td align="center">\
    <a href="http://www.adobe.com/">\
    <img src="images/foreignlogos/adobe-logo.gif" alt="adobe.com" border="0">\
    </a></td><td>\
    <a href="http://www.adobe.com/">Adobe</a> revolutionizes how the \
    world engages with ideas and information - anywhere, anytime, and \
    through any medium.</td></tr>'

  sponsor[4] = '<tr><td align="center">\
    <a href="http://www.oracle.com/us/technologies/embedded/">\
    <img src="images/foreignlogos/oracle.gif" alt="oracle.com" border="0">\
    </a></td><td>\
    <a href="http://www.oracle.com/us/technologies/embedded/">Oracle</a> -\
    Software.  Hardware.  Complete.</td></tr>'

  count = 5
  while( count>0 ){
    i = Math.floor(Math.random()*5)
    if( sponsor[i]!=null ){
      document.write(sponsor[i])
      sponsor[i] = null
      count--
    }

    <a href="http://www.mozilla.com/">\
    <img src="images/foreignlogos/mozilla.gif" alt="mozilla.com" border="0">\
    </a></td><td>\
    <a href="http://www.mozilla.com/">Mozilla</a> - Working to preserve \
    choice and innovation on the internet.</td></tr>'

  sponsor[1] = '<tr><td align="center">\








    <a href="http://www.bloomberg.com/">\
   <img src="images/foreignlogos/bloomberg.gif" alt="bloomberg.com" border="0">\
    </a></td><td>\
    <a href="http://www.bloomberg.com/">Bloomberg</a> - \
    A world leader in financial-information technology.</td></tr>'

  sponsor[2] = '<tr><td align="center">\
    <a href="http://www.adobe.com/">\
    <img src="images/foreignlogos/adobe-logo.gif" alt="adobe.com" border="0">\
    </a></td><td>\
    <a href="http://www.adobe.com/">Adobe</a> revolutionizes how the \
    world engages with ideas and information - anywhere, anytime, and \
    through any medium.</td></tr>'

  sponsor[3] = '<tr><td align="center">\
    <a href="http://www.oracle.com/us/technologies/embedded/">\
    <img src="images/foreignlogos/oracle.gif" alt="oracle.com" border="0">\
    </a></td><td>\
    <a href="http://www.oracle.com/us/technologies/embedded/">Oracle</a> -\
    Software.  Hardware.  Complete.</td></tr>'

  count = 4
  while( count>0 ){
    i = Math.floor(Math.random()*5)
    if( sponsor[i]!=null ){
      document.write(sponsor[i])
      sponsor[i] = null
      count--
    }

  hd_puts "<h3>$date - $title</h3>"
  regsub -all "\n( *\n)+" $text "</p>\n\n<p>" txt
  regsub -all {[Tt]icket #(\d+)} $txt \
      {<a href="http://www.sqlite.org/cvstrac/tktview?tn=\1">\0</a>} txt
  hd_resolve "<p>$txt</p>"
  hd_puts "<hr width=\"50%\">"
}

























newsitem {2010-December-08} {Version 3.7.4} {
  SQLite [version 3.7.4] is a regularly scheduled bi-monthly maintenance
  release of SQLite.  Upgrading from [version 3.7.2] and [version 3.7.3]
  is optional.  Upgrading from all other SQLite releases is recommended.

  This release features [full-text search] enhancements.  The older 

  hd_puts "<h3>$date - $title</h3>"
  regsub -all "\n( *\n)+" $text "</p>\n\n<p>" txt
  regsub -all {[Tt]icket #(\d+)} $txt \
      {<a href="http://www.sqlite.org/cvstrac/tktview?tn=\1">\0</a>} txt
  hd_resolve "<p>$txt</p>"
  hd_puts "<hr width=\"50%\">"
}

newsitem {2011-February-02} {Version 3.7.5} {
  SQLite [version 3.7.5] is a regularly scheduled bi-monthly maintenance
  release of SQLite.  Due to the discovery and fix of 
  [http://www.sqlite.org/src/tktview?name=5d863f876e | an obscure bug]
  that could cause database corruption, upgrading from all prior
  releases of SQLite is recommended.  This bug was found during code
  review and has not been observed in the wild.

  This release adds new [SQLITE_DBSTATUS_LOOKASIDE_HIT | opcodes] for the
  [sqlite3_db_status()] interface that allow more precise measurement of
  how the [lookaside memory allocator] is performing, which can be useful
  for tuning in applications with very tight memory constraints.

  The [sqlite3_vsnprintf()] interface was added.  This routine is simply
  a varargs version of the long-standing [sqlite3_snprintf()] interface.

  The output from [sqlite3_trace()] interface has been enhanced to work
  better (and faster) in systems that use recursive extensions such as
  [FTS3] or [RTREE].

  Testing with Valgrind shows that this release of SQLite is about 1%
  or 2% faster than the previous release for most operations.
}

newsitem {2010-December-08} {Version 3.7.4} {
  SQLite [version 3.7.4] is a regularly scheduled bi-monthly maintenance
  release of SQLite.  Upgrading from [version 3.7.2] and [version 3.7.3]
  is optional.  Upgrading from all other SQLite releases is recommended.

  This release features [full-text search] enhancements.  The older 

# facilitate that, all the size values are defined by variables here
# which are then used as needed through the document.
#
# NOTE:  Also update the version number in the text!!!
#

# sloc sqlite3.c
set stat(coreSLOC)    70630  ;# Non-comment lines of amalgamation code 
# sloc test*.c
set stat(tclcSLOC)    16848  ;# Non-comment lines of test C code
# ls test*.c tclsqlite.c | wc
set stat(tclcNfile)      35  ;# Number of files of TCL C testcode + tclsqlite.c
# ls -l test*.c tclsqlite.c | awk '{sum+=$5}END{print sum}'
set stat(tclcNByte)  807357  ;# Number of bytes of TCL C testcode + tclsqlite.c
# sloc *.test *.tcl
set stat(tclsSLOC)   209609  ;# Non-comment lines of TCL test script
# ls *.test *.tcl | wc
set stat(tclsNFile)     571  ;# Number of files of TCL test script
# ls -l *.test *.tcl | awk '{sum+=$5}END{print sum}'
set stat(tclsNByte) 9159527  ;# Number of bytes of TCL test script
# grep do_test *.test | wc; grep do_execsql_test *.test | wc
set stat(tclNTest)    26439  ;# Number of test cases in the TCL test suite
set stat(tclNEval)  2204181  ;# Number of test case evaluations
set stat(nSqlFuzz)   107457  ;# Number of SQL fuzz tests
set stat(vqNEval)     98214  ;# Number of test evaluations for veryquick.test
#  set stat(vqStmtCov)   97.23  ;# veryquick statement coverage
#  set stat(vqBrCov)     92.57  ;# veryquick branch coverage
#  set stat(allStmtCov)  99.50  ;# all.test statement coverage
#  set stat(allBrCov)    97.41  ;# all.test condition/decision coverage
# tclsh mkth3.tcl cfg/*.cfg */*.test >th3.c; sloc th3.c
set stat(th3SLOC)    628369  ;# Non-comment lines in full th3.c
# ls -l th3.c
set stat(th3NByte) 47475372  ;# Number of bytes in full th3.c
# grep th3testBegin */*.test
# grep th3oomBegin */*.test
# grep th3ioerrBegin */*.test
# grep '^--testcase' */*.test
set stat(th3NTest)    30641  ;# Number of test cases
# from output of a full test run.
set stat(th3NECov)   1871933  ;# Number of test case evals for coverage
#set stat(th3NETest)  1504866  ;# Number of test case evaluations
#set stat(th3NEExt) 589175483  ;# Number of test case evals extended
#set stat(th3NERel) 2500000000 ;# Number of test case evals release
set stat(th3StmtCov) 100.00  ;# TH3 statement coverage
set stat(th3BrCov)   100.00  ;# TH3 branch coverage
# wc `fossil ls | awk '/\.test$/{print $2}'`
set stat(sltsSLOC)  44860433 ;# Non-comment lines of SLT test script
# ls -l `fossil ls | awk '/\.test$/{print $2}'` | awk '{sum+=$5}END{print sum}'
set stat(sltsNByte) 1116768564 ;# Bytes of SLT test script
# fossil ls | awk '/.test$/{print $2}' | wc
set stat(sltsNFile)        612 ;# Files of SLT test script
# sloc md5.c slt_*.c sqllogictest.c
set stat(sltcSLOC)        1307 ;# Non-comment lines of SLT C code
# grep '^query' `fossil ls | awk '/\.test$/{print $2}'` | wc
set stat(sltNTest)     7195255 ;# Number of test cases in SLT
# grep 'assert(' sqlite3.c | wc
set stat(nAssert)         2957 ;# Number of assert statements
# grep 'testcase(' sqlite3.c | grep -v define | wc
set stat(nTestcase)        626 ;# Number of testcase statements

set stat(totalSLOC) [expr {$stat(tclcSLOC)+$stat(tclsSLOC)+
                           $stat(th3SLOC)+$stat(sltcSLOC)+$stat(sltsSLOC)}]

proc GB {expr} {
  set n [uplevel #0 expr $expr]
  hd_puts [format %.2f [expr {$n/(1000.0*1000.0*1000.0)}]]
................................................................................
<h1 align="center">How SQLite Is Tested</h1>

<h2>1.0 Introduction</h2>

<p>The reliability and robustness of SQLite is achieved in part
by thorough and careful testing.</p>

<p>As of [version 3.7.0] (all statistics in the report are against that
release of SQLite),
the SQLite library consists of approximately
<tcl>KB {$stat(coreSLOC)}</tcl> KSLOC of C code.
(KSLOC means thousands of "Source Lines Of Code" or, in other words,
lines of code excluding blank lines and comments.)
By comparison, the project has
<tcl>
................................................................................
stack overflows, memory leaks, and so forth.  Valgrind finds problems
that can easily slip through all of the other tests run against SQLite.
And, when Valgrind does find an error, it can dump the developer directly
into a symbolic debugger at the exact point where the error occur, to
facilitate a quick fix.</p>

<p>Because it is a simulator, running a binary in Valgrind is slower than 


running it on native hardware.  So it is impractical to run the full
SQLite test suite through Valgrind.  However, the veryquick tests and
a subset of the TH3 tests are run through Valgrind prior to every release.</p>


<tcl>hd_fragment memtesting</tcl>
<h3>8.3 Memsys2</h3>

<p>SQLite contains a pluggable
[memory allocation | memory allocation subsystem].
The default implementation uses system malloc() and free(). 
................................................................................
enabled and with optimizations disabled; the answer simply arrives quicker
with the optimizations turned on.  So in a production environment, one always
leaves the optimizations turned on (the default setting).</p>

<p>One verification technique used on SQLite is to run an entire test suite
twice, once with optimizations left on and a second time with optimizations
turned off, and verify that the same output is obtained both times.  This
verifies that the optimizations do not introduce errors.</p>

<p>Not all test cases can be handled this way.  Some test cases check
to verify that the optimizations really are reducing the amount of
computation by counting the number of disk accesses, sort operations, 
full-scan steps, or other processing steps that occur during queries.
Those test cases will appear to fail when optimizations are disabled.
But the majority of test cases simply look to see if the correct answer
was obtained, and all of those cases can be run successfully with and
without the optimizations, in order to show that the optimizations do not
cause malfunctions.</p>

<tcl>hd_fragment staticanalysis</tcl>
<h2>10.0 Static Analysis</h2>

# facilitate that, all the size values are defined by variables here
# which are then used as needed through the document.
#
# NOTE:  Also update the version number in the text!!!
#

# sloc sqlite3.c
set stat(coreSLOC)    73036  ;# Non-comment lines of amalgamation code 
# sloc test*.c
set stat(tclcSLOC)    18912  ;# Non-comment lines of test C code
# ls test*.c tclsqlite.c | wc
set stat(tclcNfile)      39  ;# Number of files of TCL C testcode + tclsqlite.c
# ls -l test*.c tclsqlite.c | awk '{sum+=$5}END{print sum}'
set stat(tclcNByte)  907877  ;# Number of bytes of TCL C testcode + tclsqlite.c
# sloc *.test *.tcl
set stat(tclsSLOC)   221967  ;# Non-comment lines of TCL test script
# ls *.test *.tcl | wc
set stat(tclsNFile)     617  ;# Number of files of TCL test script
# ls -l *.test *.tcl | awk '{sum+=$5}END{print sum}'
set stat(tclsNByte) 9756716  ;# Number of bytes of TCL test script
# grep do_test *.test | wc; grep do_execsql_test *.test | wc
set stat(tclNTest)    27664  ;# Number of test cases in the TCL test suite
set stat(tclNEval)  1435414  ;# Number of test case evaluations
set stat(nSqlFuzz)   103884  ;# Number of SQL fuzz tests
set stat(vqNEval)    134753  ;# Number of test evaluations for veryquick.test
#  set stat(vqStmtCov)   97.23  ;# veryquick statement coverage
#  set stat(vqBrCov)     92.57  ;# veryquick branch coverage
#  set stat(allStmtCov)  99.50  ;# all.test statement coverage
#  set stat(allBrCov)    97.41  ;# all.test condition/decision coverage
# tclsh mkth3.tcl cfg/*.cfg */*.test >th3.c; sloc th3.c
set stat(th3SLOC)    646817  ;# Non-comment lines in full th3.c
# ls -l th3.c
set stat(th3NByte) 48692553  ;# Number of bytes in full th3.c
# grep th3testBegin */*.test
# grep th3oomBegin */*.test
# grep th3ioerrBegin */*.test
# grep '^--testcase' */*.test
set stat(th3NTest)      32688  ;# Number of test cases
# from output of a full test run.
set stat(th3NECov)     735905  ;# Number of test case evals for coverage
#set stat(th3NETest)  1504866  ;# Number of test case evaluations
#set stat(th3NEExt) 589175483  ;# Number of test case evals extended
#set stat(th3NERel) 2500000000 ;# Number of test case evals release
set stat(th3StmtCov) 100.00  ;# TH3 statement coverage
set stat(th3BrCov)   100.00  ;# TH3 branch coverage
# wc `find . -name '*.test'` | awk '{x+=$1}END{print x}'
set stat(sltsSLOC)  90489494 ;# Non-comment lines of SLT test script
# ls -l `find . -name '*.test'` | awk '{sum+=$5}END{print sum}'
set stat(sltsNByte) 1120693457 ;# Bytes of SLT test script
# find . -name '*.test' | wc
set stat(sltsNFile)        629 ;# Files of SLT test script
# sloc md5.c slt_*.c sqllogictest.c
set stat(sltcSLOC)        1403 ;# Non-comment lines of SLT C code
# grep '^query' `fossil ls | awk '/\.test$/{print $2}'` | wc
set stat(sltNTest)     7200478 ;# Number of test cases in SLT
# grep 'assert(' sqlite3.c | wc
set stat(nAssert)         3242 ;# Number of assert statements
# grep 'testcase(' sqlite3.c | grep -v define | wc
set stat(nTestcase)        648 ;# Number of testcase statements

set stat(totalSLOC) [expr {$stat(tclcSLOC)+$stat(tclsSLOC)+
                           $stat(th3SLOC)+$stat(sltcSLOC)+$stat(sltsSLOC)}]

proc GB {expr} {
  set n [uplevel #0 expr $expr]
  hd_puts [format %.2f [expr {$n/(1000.0*1000.0*1000.0)}]]
................................................................................
<h1 align="center">How SQLite Is Tested</h1>

<h2>1.0 Introduction</h2>

<p>The reliability and robustness of SQLite is achieved in part
by thorough and careful testing.</p>

<p>As of [version 3.7.5] (all statistics in the report are against that
release of SQLite),
the SQLite library consists of approximately
<tcl>KB {$stat(coreSLOC)}</tcl> KSLOC of C code.
(KSLOC means thousands of "Source Lines Of Code" or, in other words,
lines of code excluding blank lines and comments.)
By comparison, the project has
<tcl>
................................................................................
stack overflows, memory leaks, and so forth.  Valgrind finds problems
that can easily slip through all of the other tests run against SQLite.
And, when Valgrind does find an error, it can dump the developer directly
into a symbolic debugger at the exact point where the error occur, to
facilitate a quick fix.</p>

<p>Because it is a simulator, running a binary in Valgrind is slower than 
running it on native hardware.  (To a first approximation, an application
running in Valgrind on a workstation will perform about the same as it
would running natively on a smartphone.)  So it is impractical to run the full
SQLite test suite through Valgrind.  However, the veryquick tests and
the coverage of the TH3 tests are run through Valgrind prior to every
release.</p>

<tcl>hd_fragment memtesting</tcl>
<h3>8.3 Memsys2</h3>

<p>SQLite contains a pluggable
[memory allocation | memory allocation subsystem].
The default implementation uses system malloc() and free(). 
................................................................................
enabled and with optimizations disabled; the answer simply arrives quicker
with the optimizations turned on.  So in a production environment, one always
leaves the optimizations turned on (the default setting).</p>

<p>One verification technique used on SQLite is to run an entire test suite
twice, once with optimizations left on and a second time with optimizations
turned off, and verify that the same output is obtained both times.  This
shows that the optimizations do not introduce errors.</p>

<p>Not all test cases can be handled this way.  Some test cases check
to verify that the optimizations really are reducing the amount of
computation by counting the number of disk accesses, sort operations, 
full-scan steps, or other processing steps that occur during queries.
Those test cases will appear to fail when optimizations are disabled.
But the majority of test cases simply check that the correct answer
was obtained, and all of those cases can be run successfully with and
without the optimizations, in order to show that the optimizations do not
cause malfunctions.</p>

<tcl>hd_fragment staticanalysis</tcl>
<h2>10.0 Static Analysis</h2>

<h2>1.0 Overview</h2>

<p>SQLite Test Harness #3 (hereafter "TH3") is one of
[three test harnesses] used for testing SQLite.
TH3 is designed to meet the following objectives:</p>

<ul>
<li><p> TH3 will work on embedded platforms that lack the support
     infrastructure of workstations.</p></li>

<li><p> TH3 will test SQLite in an as-deployed configuration, without the
     need to enable special test hooks in the core SQLite library. </p></li>

<li><p> TH3 will test SQLite's response to out-of-memory errors, disk I/O
     errors, and power loss during transaction commit. </p></li>

<li><p> TH3 will test SQLite in a variety of configurations (UTF8 vs UTF16,
     different pages sizes, varying journal modes, etc.) </p></li>

<li><p> TH3 will achieve 100% branch test coverage. </p></li>
</ul>

<p>TH3 is intended for validation testing.
Though it could be used for testing during development and debugging,
TH3 is not designed for that purpose.
The original TCL-based test suite for SQLite will continue to serve
as the primary platform for development and debug testing.
The reason that TH3 exists is that it meets other objectives that are
difficult to achieve with the TCL-based tests.</p>






<h2>2.0 Operation</h2>

<p>TH3 is a test program generator.  The output of TH3 is a program
written in ANSI-C and intended to be
linked against the SQLite library under test.  The generated test
program is compiled and run on the target platform in order to verify

<h2>1.0 Overview</h2>

<p>SQLite Test Harness #3 (hereafter "TH3") is one of
[three test harnesses] used for testing SQLite.
TH3 is designed to meet the following objectives:</p>

<ul>
<li><p> TH3 runs on embedded platforms that lack the support
     infrastructure of workstations.</p></li>

<li><p> TH3 tests SQLite in an as-deployed configuration using only
     published and documented interfaces.
     In other words, TH3 tests the compiled object code, not
     the source code, thus verifying that no problems were introduced
     by compiler bugs.</p></li>

<li><p> TH3 checks SQLite's response to out-of-memory errors, disk I/O
     errors, and power loss during transaction commit. </p></li>

<li><p> TH3 exercises SQLite in a variety of run-time configurations
     (UTF8 vs UTF16, different pages sizes, varying journal modes, etc.)
     </p></li>

<li><p> TH3 achieves 100% branch test coverage over SQLite core.
    (Test coverage of the operating-system specific VFSes and extensions
    such as FTS and RTREE is less than 100%). </p></li>
</ul>

<p>TH3 was originally written for validation testing only, but has
subsequently been used for development testing and debugging
as well, and has proven very helpful in those roles.  A full-coverage
test run for TH3 takes less than 10 minutes on a workstation and hence
serves as a fast but effect regression test during day-to-day maintenance
of the SQLite code base.</p>

<h2>2.0 Operation</h2>

<p>TH3 is a test program generator.  The output of TH3 is a program
written in ANSI-C and intended to be
linked against the SQLite library under test.  The generated test
program is compiled and run on the target platform in order to verify