Documentation Source Text

Check-in [7c6ef04be8]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Additional notes on how to corrupt an SQLite database file.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 7c6ef04be878ce39b8305ee5fb4920e900800dd1
User & Date: drh 2013-10-02 13:34:01.790
Context
2013-10-04
17:27
Add new bullets to the change log for 3.8.1. (check-in: dc132e270c user: drh tags: trunk)
2013-10-02
13:34
Additional notes on how to corrupt an SQLite database file. (check-in: 7c6ef04be8 user: drh tags: trunk)
2013-09-30
12:10
Fix to the releaselog/current.html generator. (check-in: e7d9c2101a user: drh tags: trunk)
Changes
Unified Diff Ignore Whitespace Patch
Changes to pages/howtocorrupt.in.
30
31
32
33
34
35
36













37
38
39
40
41
42
43
Later, some other thread continued to write log information into the
old file descriptor, not realizing that the log file had been closed
already.  But because the file descriptor had been reopened by SQLite,
the information that was intended to go into the log file ended up
overwriting parts of the SQLite database, leading to corruption of the
database.</p>














<h3>1.2 Backup or restore while a transaction is active</h3>

<p>Systems that run automatic backups in the background might try to
make a backup copy of an SQLite database file while it is in the middle
of a transaction.  The backup copy then might contain some old and some
new content, and thus be corrupt.</p>








>
>
>
>
>
>
>
>
>
>
>
>
>







30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
Later, some other thread continued to write log information into the
old file descriptor, not realizing that the log file had been closed
already.  But because the file descriptor had been reopened by SQLite,
the information that was intended to go into the log file ended up
overwriting parts of the SQLite database, leading to corruption of the
database.</p>

<p>One example of this occurred circa 2013-08-30 on the canonical repository
for the <a href="http://www.fossil-scm.org/">Fossil DVCS</a>.  In that event,
file descriptor 2 (standard error) was being erroneously closed prior to 
[sqlite3_open_v2()] so that the file descriptor used for the
repository database file was 2.  Later, another application 
bug caused an assert() statement to emit
an error message by invoking write(2,...).  But since file descriptor 2 was 
now connected to a database file, the error message
overwrote part of the database.  To guard against this kind of problem,
SQLite [version 3.8.1] and later refuse to use low-numbered file descriptors
for database files. 
(See [SQLITE_MINIMUM_FILE_DESCRIPTOR] for additional information.)</p>

<h3>1.2 Backup or restore while a transaction is active</h3>

<p>Systems that run automatic backups in the background might try to
make a backup copy of an SQLite database file while it is in the middle
of a transaction.  The backup copy then might contain some old and some
new content, and thus be corrupt.</p>

62
63
64
65
66
67
68

69
70
71
72
73
74
75
<p>SQLite must see the journal files in order to recover from a crash
or power failure.  If the journal files are moved, deleted, or renamed
after a crash or power failure, then automatic recovery will not work
and the database may go corrupt.</p>

<p>Another manifestation of this problem is
[database corruption caused by inconsistent use of 8+3 filenames].</p>


<h2>2.0 File locking problems</h2>

<p>SQLite uses file locks on the database file, and on the 
[write-ahead log] or [WAL] file, to coordinate access between concurrent
processes.  Without coordination, two threads or processes might try
to make incompatible changes to a database file at the same time,







>







75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
<p>SQLite must see the journal files in order to recover from a crash
or power failure.  If the journal files are moved, deleted, or renamed
after a crash or power failure, then automatic recovery will not work
and the database may go corrupt.</p>

<p>Another manifestation of this problem is
[database corruption caused by inconsistent use of 8+3 filenames].</p>


<h2>2.0 File locking problems</h2>

<p>SQLite uses file locks on the database file, and on the 
[write-ahead log] or [WAL] file, to coordinate access between concurrent
processes.  Without coordination, two threads or processes might try
to make incompatible changes to a database file at the same time,
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
if one process crashes, the other process will be unable to recover the
transaction in progress because it will be looking in the wrong place
for the appropriate journal.</p>

<p>Beginning with SQLite [version 3.7.17], the unix OS interface will
send SQLITE_WARNING messages to the [error log] if a database file has 
multiple hard links.  As of this writing, SQLite still does not yet detect 
and warn about the use of database files through soft links.</p>


<h2>3.0 Failure to sync</h2>

<p>In order to guarantee that database files are always consistent, SQLite
will occasionally ask the operating system to flush all pending writes to
persistent storage then wait for that flush to complete.  This is 







|







206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
if one process crashes, the other process will be unable to recover the
transaction in progress because it will be looking in the wrong place
for the appropriate journal.</p>

<p>Beginning with SQLite [version 3.7.17], the unix OS interface will
send SQLITE_WARNING messages to the [error log] if a database file has 
multiple hard links.  As of this writing, SQLite still does not yet detect 
or warn about the use of database files through soft links.</p>


<h2>3.0 Failure to sync</h2>

<p>In order to guarantee that database files are always consistent, SQLite
will occasionally ask the operating system to flush all pending writes to
persistent storage then wait for that flush to complete.  This is 
349
350
351
352
353
354
355















356
357
358
359
360
361
362
the time you read this.</p>

<p>When running on QNX, it is recommended that [memory-mapped I/O] never
be used.  Furthermore, to use [WAL mode], it is recommended that applications
employ the [locking_mode | exclusive locking mode] in order to 
use [WAL without shared memory].
















<h2>7.0 Bugs in SQLite</h2>

<p>SQLite is [testing | very carefully tested] to help ensure that it is
as bug-free as possible.  Among the many tests that are carried out for
every SQLite version are tests that simulate power failures, I/O errors,
and out-of-memory (OOM) errors and verify that no database corrupt occurs
during any of these events.  SQLite is also field-proven with approximately







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
the time you read this.</p>

<p>When running on QNX, it is recommended that [memory-mapped I/O] never
be used.  Furthermore, to use [WAL mode], it is recommended that applications
employ the [locking_mode | exclusive locking mode] in order to 
use [WAL without shared memory].


<tcl>hd_fragment fscorruption {filesystem corruption}</tcl>
<h3>6.3 Filesystem Corruption</h3>

<p>Since SQLite databases are ordinary disk files, any malfunction in the
filesystem can corrupt the database.  Filesystems in modern operating systems
are very reliable, but errors do still occur.  For example, on 2013-10-01
the SQLite database that holds the
<a href="http://wiki.tcl.tk/">Wiki for Tcl/Tk</a> went corrupt a few days
after an erroneous attempt to increase the size of an EXT3 partition caused
the filesystem to begin malfunctioning.  In that event, the filesystem
eventually became so badly corrupted that the machine was unusable and
had to be rebuilt.  But the earliest symptom of trouble was the corrupted
SQLite database.</p>

<h2>7.0 Bugs in SQLite</h2>

<p>SQLite is [testing | very carefully tested] to help ensure that it is
as bug-free as possible.  Among the many tests that are carried out for
every SQLite version are tests that simulate power failures, I/O errors,
and out-of-memory (OOM) errors and verify that no database corrupt occurs
during any of these events.  SQLite is also field-proven with approximately