Documentation Source Text

data preservation during a power loss.  Powersafe overwrite
is a boolean property: either the storage system has it or it does not.

<p>We say that a system has the powersafe overwrite property if the following
statement is true:

<blockquote>
  <b>Writing a range of bytes in a file at the application level will not
  change any bytes outside of that range, even if the write occurs
  just before a crash or power failure.</b>
</blockquote>

<p>The powersafe overwrite property says nothing about the state of the
bytes that were written.  Those bytes might contain their old values,
their new values, random values, or some combination of these.  The powersafe
overwrite property merely states that writes cannot change bytes outside
................................................................................
<p>In other words, powersafe overwrite means that there is no "collateral
damage" when a power loss occurs while writing.  Only those bytes actually
being written might be damaged.

<p>In practical terms, what the powersafe write property means is that when
the disk controller detects an impending power loss, it finishes writing
whatever sector it is working on prior to parking the heads.  It means that
individual sector writes run to completion once started, even if
there is a power loss.

<p>Consider what would happen if disk sector writes are interrupted
by a power loss.  If an application writes two or three bytes in the middle
of some file, the operating system will implement this by first reading
the entire sector containing those bytes, making the change to the
sector in memory, then writing the entire sector back to the disk.  If a power
loss occurs during the writeback and the sector was not completely written,
then on the next read after reboot, error correcting codes at the end 
of the sector will probably detect irrepariable damage and the disk 
controller will read out the sector as all zeros or all ones.  Thus
values will have changed outside of the range of the two or three bytes 
that were written at the application level - a violation of the powersafe
overwrite property.

<h2>SQLite Assumptions About Powersafe Overwrite</h2>

................................................................................
This had the effect of increasing write overhead on
many databases.  For a database with a [PRAGMA page_size] of 1024
(a very common choice) making a change to a single page in the database
now requires SQLite to backup three other adjacent pages to the rollback
journal, whereas formerly it only had to backup the one page that was
changing.  In [WAL mode], each transaction had to be padded out to the
next 4096-byte boundary in the WAL file, rather than the next 512-byte
boundary, resulting in thousands of extra bytes needing to be written
per transaction.

<p>The extra write overhead prompted a reexamination of assumptions about
powersafe overwrite.  With modern disk drives, the capacity has become
so large and the data density so great that a single sector is very
small and writing a single sector takes very little time.  We know that
disk drives can detect an impending power loss and continue
to operate for some small amount of time on residual energy because those
drives are able to park their heads before spinning down.  And
so if an impending power loss is detectable by the disk controller, it
seems reasonable that the controller will finish writing
whatever sector it is current working on when the imminent power loss 
is first detected, prior to parking the heads, as long as finishing
work on the sector does not take too long, which it should not with
small and dense sectors.  Hence it seems reasonable
to assume powersafe overwrite for modern disks.  Indeed, BerkeleyDB has
made this assumption for decades, we are told.  Caution is advised
though. As Roger Binns noted on the SQLite develpers mailing list:
"'poorly written' should be the main assumption about drive firmware."

<h2>Changes In SQLite Version 3.7.10</h2>

data preservation during a power loss.  Powersafe overwrite
is a boolean property: either the storage system has it or it does not.

<p>We say that a system has the powersafe overwrite property if the following
statement is true:

<blockquote>
  <b>When an application writes a range of bytes in a file, no
  bytes outside of that range will change, even if the write occurs
  just before a crash or power failure.</b>
</blockquote>

<p>The powersafe overwrite property says nothing about the state of the
bytes that were written.  Those bytes might contain their old values,
their new values, random values, or some combination of these.  The powersafe
overwrite property merely states that writes cannot change bytes outside
................................................................................
<p>In other words, powersafe overwrite means that there is no "collateral
damage" when a power loss occurs while writing.  Only those bytes actually
being written might be damaged.

<p>In practical terms, what the powersafe write property means is that when
the disk controller detects an impending power loss, it finishes writing
whatever sector it is working on prior to parking the heads.  It means that
individual sector writes will complete once started, even if
there is a power loss.

<p>Consider what would happen if disk sector writes are interrupted
by a power loss.  If an application writes two or three bytes in the middle
of some file, the operating system will implement this by first reading
the entire sector containing those bytes, making the change to the
sector in memory, then writing the entire sector back to the disk.  If a power
loss occurs during the writeback and the sector was not completely written,
then on the next read after reboot, error correcting codes
in the sector will probably detect irrepariable damage and the disk 
controller will read out the sector as all zeros or all ones.  Thus
values will have changed outside of the range of the two or three bytes 
that were written at the application level - a violation of the powersafe
overwrite property.

<h2>SQLite Assumptions About Powersafe Overwrite</h2>

................................................................................
This had the effect of increasing write overhead on
many databases.  For a database with a [PRAGMA page_size] of 1024
(a very common choice) making a change to a single page in the database
now requires SQLite to backup three other adjacent pages to the rollback
journal, whereas formerly it only had to backup the one page that was
changing.  In [WAL mode], each transaction had to be padded out to the
next 4096-byte boundary in the WAL file, rather than the next 512-byte
boundary, resulting in thousands of extra bytes being written
per transaction.

<p>The extra write overhead prompted a reexamination of assumptions about
powersafe overwrite.  With modern disk drives, the capacity has become
so large and the data density so great that a single sector is very
small and writing a single sector takes very little time.  We know that
disk drives can detect an impending power loss and continue
to operate for some small amount of time on residual energy because those
drives are able to park their heads before spinning down.  And
so if an impending power loss is detectable by the disk controller, it
seems reasonable that the controller will finish writing
whatever sector it is current working on when the imminent power loss 
is first detected, prior to parking the heads, as long as doing so
does not take too long, which it should not with
small and dense sectors.  Hence it seems reasonable
to assume powersafe overwrite for modern disks.  Indeed, BerkeleyDB has
made this assumption for decades, we are told.  Caution is advised
though. As Roger Binns noted on the SQLite develpers mailing list:
"'poorly written' should be the main assumption about drive firmware."

<h2>Changes In SQLite Version 3.7.10</h2>

<h1 align="center">How SQLite Is Tested</h1>

<h2>1.0 Introduction</h2>

<p>The reliability and robustness of SQLite is achieved in part
by thorough and careful testing.</p>

<p>As of [version 3.7.8] (all statistics in the report are against that
release of SQLite),
the SQLite library consists of approximately
<tcl>KB {$stat(coreSLOC)}</tcl> KSLOC of C code.
(KSLOC means thousands of "Source Lines Of Code" or, in other words,
lines of code excluding blank lines and comments.)
By comparison, the project has
<tcl>
hd_puts "[expr {int($stat(totalSLOC)/$stat(coreSLOC))}] times as much"
................................................................................
<h2>10.0 Static Analysis</h2>

<p>Static analysis means analyzing code at or before compile-time to
check for correctness.  Static analysis includes looking at compiler
warning messages and running the code through more in-depth
analysis engines such as the
[http://clang-analyzer.llvm.org/ | Clang Static Analyzer].
SQLite is developed primarily using GCC and it does
compile without warnings on GCC using the -Wall and -Wextra flags.
The SQLite source code is also run through Clang from time to time 
as well, though most of the warnings there are false-positives.

VC++ often will generate a number of warnings from SQLite source code,
but the experience of SQLite developers is that VC++ warnings are of
lower quality and can be safely ignored.  Users are encouraged not
to stress over VC++ warnings.</p>

<p>Static analysis has not proven to be helpful in finding
bugs in SQLite.  We cannot call to mind a single problem in SQLite that


was detected by static analysis that was not first seen by one
of the other testing methods described above.  On the other hand,
we have on occasion introduced new bugs in our efforts to get SQLite
to compile without warnings.</p>

<p>Our experience, then, is that static analysis is not helpful
to maintaining high code quality and it is therefore deemphasized
in SQLite development and maintenance.</p>

<tcl>hd_fragment summary</tcl>
<h2>11.0 Summary</h2>

<p>SQLite is open source.  This gives many people the idea that

<h1 align="center">How SQLite Is Tested</h1>

<h2>1.0 Introduction</h2>

<p>The reliability and robustness of SQLite is achieved in part
by thorough and careful testing.</p>

<p>As of [version 3.7.8],

the SQLite library consists of approximately
<tcl>KB {$stat(coreSLOC)}</tcl> KSLOC of C code.
(KSLOC means thousands of "Source Lines Of Code" or, in other words,
lines of code excluding blank lines and comments.)
By comparison, the project has
<tcl>
hd_puts "[expr {int($stat(totalSLOC)/$stat(coreSLOC))}] times as much"
................................................................................
<h2>10.0 Static Analysis</h2>

<p>Static analysis means analyzing code at or before compile-time to
check for correctness.  Static analysis includes looking at compiler
warning messages and running the code through more in-depth
analysis engines such as the
[http://clang-analyzer.llvm.org/ | Clang Static Analyzer].

SQLite compiles without warnings on GCC using the -Wall and -Wextra flags.


And the Clang Static Analyzer finds nothing wrong with SQLite either.
VC++ often will generate a number of warnings from SQLite source code,
but the experience of SQLite developers is that VC++ warnings are of
lower quality and can be safely ignored.  Users are encouraged not
to stress over VC++ warnings.</p>

<p>Static analysis has not proven to be especially helpful in finding
bugs in SQLite.  Static analysis has found a few bugs in SQLite, but
those are the exceptions.  More bugs have been
introduced into SQLite while trying to get it to compile without 
warnings than have been found by static analysis.




<p>Our experience, then, is that static analysis is not especially helpful
to maintaining high code quality and it is therefore deemphasized
in SQLite development and maintenance.</p>

<tcl>hd_fragment summary</tcl>
<h2>11.0 Summary</h2>

<p>SQLite is open source.  This gives many people the idea that