Documentation Source Text

Check-in [0992cf9ad0]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Update the compile-time options document and footprint size bounds. Updates to the file format document.
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 0992cf9ad0f531dc3239bf521025fcc8046898a3
User & Date: drh 2010-07-07 02:03:16
Context
2010-07-08
18:12
Update PRAGMA journal_mode documentation. Fix typos in the wal.html document. check-in: b1e171c029 user: drh tags: trunk
2010-07-07
02:03
Update the compile-time options document and footprint size bounds. Updates to the file format document. check-in: 0992cf9ad0 user: drh tags: trunk
2010-07-06
01:27
Adding first draft of WAL file format. check-in: 05824b6e6a user: drh tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to pages/compile.in.

575
576
577
578
579
580
581






582
583
584
585
586
587
588
...
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611





612
613
614
615
616
617
618
...
644
645
646
647
648
649
650
651

652
653
654
655
656
657
658
...
706
707
708
709
710
711
712




713
714
715
716
717
718
719
...
835
836
837
838
839
840
841




842
843
844
845
846
847
848
  that employ the BETWEEN operator.
}

COMPILE_OPTION {SQLITE_OMIT_BLOB_LITERAL} {
  When this option is defined, it is not possible to specify a blob in
  an SQL statement using the X'ABCD' syntax.
}







COMPILE_OPTION {SQLITE_OMIT_BUILTIN_TEST} {
  A standard SQLite build includes a small amount of logic controlled
  by the [sqlite3_test_control()] interface that is used to exercise
  parts of the SQLite core that are difficult to control and measure using
  the standard API.  This option omits that built-in test logic.
}
................................................................................

COMPILE_OPTION {SQLITE_OMIT_CHECK} {
  This option causes SQLite to omit support for CHECK constraints.
  The parser will still accept CHECK constraints in SQL statements,
  they will just not be enforced.
}

COMPILE_OPTION {SQLITE_OMIT_COMPLETE} {
  This option causes the [sqlite3_complete()] and [sqlite3_complete16()]
  interfaces to be omitted.
}

COMPILE_OPTION {SQLITE_OMIT_COMPILEOPTION_DIAGS} {
  This option is used to omit the compile-time option diagnostics available
  in SQLite, including the [sqlite3_compileoption_used()] and
  [sqlite3_compileoption_get()] C/C++ functions, the
  [sqlite_compileoption_used()] and [sqlite_compileoption_get()] SQL functions,
  and the [compile_options pragma].
}






COMPILE_OPTION {SQLITE_OMIT_COMPOUND_SELECT} {
  This option is used to omit the compound [SELECT] functionality. 
  [SELECT] statements that use the 
  UNION, UNION ALL, INTERSECT or EXCEPT compound SELECT operators will 
  cause a parse error.
}
................................................................................
  [sqlite3_global_recover()],
  [sqlite3_thread_cleanup()] and
  [sqlite3_memory_alarm()] interfaces.
}

COMPILE_OPTION {SQLITE_OMIT_DISKIO} {
  This option omits all support for writing to the disk and forces
  databases to exist in memory only.

}

COMPILE_OPTION {SQLITE_OMIT_EXPLAIN} {
  Defining this option causes the [EXPLAIN] command to be omitted from the
  library. Attempting to execute an [EXPLAIN] statement will cause a parse
  error.
}
................................................................................

COMPILE_OPTION {SQLITE_OMIT_LOCALTIME} {
  This option omits the "localtime" modifier from the date and time
  functions.  This option is sometimes useful when trying to compile
  the date and time functions on a platform that does not support the
  concept of local time.
}





COMPILE_OPTION {SQLITE_OMIT_MEMORYDB} {
  When this is defined, the library does not respect the special database
  name ":memory:" (normally used to create an [in-memory database]). If 
  ":memory:" is passed to [sqlite3_open()], [sqlite3_open16()], or
  [sqlite3_open_v2()], a file with this name will be 
  opened or created.
................................................................................
  for which the schema contains VIEW objects. 
}

COMPILE_OPTION {SQLITE_OMIT_VIRTUALTABLE} {
  This option omits support for the [sqlite3_vtab | Virtual Table]
  mechanism in SQLite.
}





COMPILE_OPTION {SQLITE_OMIT_WSD} {
  This options builds a version of the SQLite library that contains no
  Writable Static Data (WSD).  WSD is global variables and/or static
  variables.  Some platforms do not support WSD, and this option is necessary
  in order for SQLite to work those platforms.  








>
>
>
>
>
>







 







<
<
<
<
<







>
>
>
>
>







 







|
>







 







>
>
>
>







 







>
>
>
>







575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
...
599
600
601
602
603
604
605





606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
...
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
...
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
...
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
  that employ the BETWEEN operator.
}

COMPILE_OPTION {SQLITE_OMIT_BLOB_LITERAL} {
  When this option is defined, it is not possible to specify a blob in
  an SQL statement using the X'ABCD' syntax.
}

COMPILE_OPTION {SQLITE_OMIT_BTREECOUNT} {
  When this option is defined, an optimization that accelerates counting
  all entires in a table (in other words, an optimization that helps
  "SELECT count(*) FROM table" run faster) is omitted.
}

COMPILE_OPTION {SQLITE_OMIT_BUILTIN_TEST} {
  A standard SQLite build includes a small amount of logic controlled
  by the [sqlite3_test_control()] interface that is used to exercise
  parts of the SQLite core that are difficult to control and measure using
  the standard API.  This option omits that built-in test logic.
}
................................................................................

COMPILE_OPTION {SQLITE_OMIT_CHECK} {
  This option causes SQLite to omit support for CHECK constraints.
  The parser will still accept CHECK constraints in SQL statements,
  they will just not be enforced.
}






COMPILE_OPTION {SQLITE_OMIT_COMPILEOPTION_DIAGS} {
  This option is used to omit the compile-time option diagnostics available
  in SQLite, including the [sqlite3_compileoption_used()] and
  [sqlite3_compileoption_get()] C/C++ functions, the
  [sqlite_compileoption_used()] and [sqlite_compileoption_get()] SQL functions,
  and the [compile_options pragma].
}

COMPILE_OPTION {SQLITE_OMIT_COMPLETE} {
  This option causes the [sqlite3_complete()] and [sqlite3_complete16()]
  interfaces to be omitted.
}

COMPILE_OPTION {SQLITE_OMIT_COMPOUND_SELECT} {
  This option is used to omit the compound [SELECT] functionality. 
  [SELECT] statements that use the 
  UNION, UNION ALL, INTERSECT or EXCEPT compound SELECT operators will 
  cause a parse error.
}
................................................................................
  [sqlite3_global_recover()],
  [sqlite3_thread_cleanup()] and
  [sqlite3_memory_alarm()] interfaces.
}

COMPILE_OPTION {SQLITE_OMIT_DISKIO} {
  This option omits all support for writing to the disk and forces
  databases to exist in memory only.  This option has not been 
  maintained and probably does not work with newer versions of SQLite.
}

COMPILE_OPTION {SQLITE_OMIT_EXPLAIN} {
  Defining this option causes the [EXPLAIN] command to be omitted from the
  library. Attempting to execute an [EXPLAIN] statement will cause a parse
  error.
}
................................................................................

COMPILE_OPTION {SQLITE_OMIT_LOCALTIME} {
  This option omits the "localtime" modifier from the date and time
  functions.  This option is sometimes useful when trying to compile
  the date and time functions on a platform that does not support the
  concept of local time.
}

COMPILE_OPTION {SQLITE_OMIT_LOOKASIDE} {
  This option omits the [lookaside memory allocator].
}

COMPILE_OPTION {SQLITE_OMIT_MEMORYDB} {
  When this is defined, the library does not respect the special database
  name ":memory:" (normally used to create an [in-memory database]). If 
  ":memory:" is passed to [sqlite3_open()], [sqlite3_open16()], or
  [sqlite3_open_v2()], a file with this name will be 
  opened or created.
................................................................................
  for which the schema contains VIEW objects. 
}

COMPILE_OPTION {SQLITE_OMIT_VIRTUALTABLE} {
  This option omits support for the [sqlite3_vtab | Virtual Table]
  mechanism in SQLite.
}

COMPILE_OPTION {SQLITE_OMIT_WAL} {
  This option omits the "[write-ahead log]" (a.k.a. "[WAL]") capability.
}

COMPILE_OPTION {SQLITE_OMIT_WSD} {
  This options builds a version of the SQLite library that contains no
  Writable Static Data (WSD).  WSD is global variables and/or static
  variables.  Some platforms do not support WSD, and this option is necessary
  in order for SQLite to work those platforms.  

Changes to pages/features.in.

12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
    (<a href="omitted.html">Features not supported</a>)</li>
<li>A complete database is stored in a 
    <a href="onefile.html">single cross-platform disk file</a>.</li>
<li>Supports terabyte-sized databases and gigabyte-sized strings
    and blobs.  (See <a href="limits.html">limits.html</a>.)
<li>Small code footprint: 
    <a href="http://www.sqlite.org/cvstrac/wiki?p=SizeOfSqlite">
    less than 300KiB</a> fully configured or less
    than 180KiB with optional features omitted.</li>
<li><a href="speed.html">Faster</a> than popular client/server database
    engines for most common operations.</li>
<li>Simple, easy to use <a href="c3ref/intro.html">API</a>.</li>
<li>Written in ANSI-C.  <a href="tclsqlite.html">TCL bindings</a> included.
    Bindings for dozens of other languages 
    <a href="http://www.sqlite.org/cvstrac/wiki?p=SqliteWrappers">
    available separately.</a></li>







|
|







12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
    (<a href="omitted.html">Features not supported</a>)</li>
<li>A complete database is stored in a 
    <a href="onefile.html">single cross-platform disk file</a>.</li>
<li>Supports terabyte-sized databases and gigabyte-sized strings
    and blobs.  (See <a href="limits.html">limits.html</a>.)
<li>Small code footprint: 
    <a href="http://www.sqlite.org/cvstrac/wiki?p=SizeOfSqlite">
    less than 325KiB</a> fully configured or less
    than 190KiB with optional features omitted.</li>
<li><a href="speed.html">Faster</a> than popular client/server database
    engines for most common operations.</li>
<li>Simple, easy to use <a href="c3ref/intro.html">API</a>.</li>
<li>Written in ANSI-C.  <a href="tclsqlite.html">TCL bindings</a> included.
    Bindings for dozens of other languages 
    <a href="http://www.sqlite.org/cvstrac/wiki?p=SqliteWrappers">
    available separately.</a></li>

Changes to pages/fileformat2.in.

1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
....
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328


1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341




1342
1343
1344
1345
1346
1347
1348
magic number in the first 4 bytes of the WAL header is 0x377f0683 and
the integers are little-endian the magic number is 0x377f0682.
The checksum values are always stored in the frame header in a
big-endian format regardless of which byte order is used to compute
the checksum.</p>

<p>The checksum algorithm only works for content which is a multiple of
8 bytes in length.  In other words, if the puts are x(0) through x(N)
then N must be odd.
The checksum algorithm is as follows:

<blockquote><pre> 
s0 = s1 = 0
for i from 0 to n-1 step 2:
   s0 += x(i) + s1;
................................................................................
frame or are followed by a commit frame, then page P is read from
the database file.</p>

<p>To start a read transaction, the reader records the index of the last
valid frame in the WAL.  The reader uses this recorded "mxFrame" value
for all subsequent read operations.  New transactions can be appended
to the WAL, but as long as the reader uses its original mxFrame value
and ignores the newly appended content, it will see a consistent snapshot
of the database from a single point in time.  This technique allows
multiple concurrent readers to view different versions of the database
content simultaneously.</p>

<p>The reader algorithm in the previous paragraphs works correctly, but 
because frames for page P can appear anywhere within the WAL, the
reader has to scan the entire WAL looking for page P frames.  If the
WAL is large (multiple megabytes is typical) that scan can be slow,
and read performance suffers.  To overcome this problem, a separate
data structure called the wal-index is maintained to expedite the
search for frames of a particular page.</p>

<tcl>hd_fragment walindexformat {WAL-index format}</tcl>
<h3>4.5 WAL-Index Format</h3>

<p>Conceptually, the wal-index is shared memory, though the current
VFS implementations use a mmapped file for the wal-index.  Because


the wal-index is shared memory, SQLite does not support 
[PRAGMA journal_mode | journal_mode=WAL] 
on a network filesystem when clients are on different machines.
All users of the database must be able to share the same memory.</p>

<p>The purpose of the wal-index is to answer this question quickly:</p>

<blockquote><i>
Given a page number P and a maximum WAL frame index M,
return the largest WAL frame index for page P that does not exceed M, 
or return NULL if there are no frames for page P that do not exceed M.
</i></blockquote>






<p>The wal-index is transient.  After a crash, the wal-index is
reconstructed from the original WAL file.  The VFS is required
to either truncate or zero the header of the wal-index when the last
connection to it closes.  Because the wal-index is transient, it can
use an architecture-specific format; it does not have to be cross-platform.
Hence, unlike the database and WAL file formats which store all values







|







 







|
|
|
|













|
>
>













>
>
>
>







1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
....
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
magic number in the first 4 bytes of the WAL header is 0x377f0683 and
the integers are little-endian the magic number is 0x377f0682.
The checksum values are always stored in the frame header in a
big-endian format regardless of which byte order is used to compute
the checksum.</p>

<p>The checksum algorithm only works for content which is a multiple of
8 bytes in length.  In other words, if the inputs are x(0) through x(N)
then N must be odd.
The checksum algorithm is as follows:

<blockquote><pre> 
s0 = s1 = 0
for i from 0 to n-1 step 2:
   s0 += x(i) + s1;
................................................................................
frame or are followed by a commit frame, then page P is read from
the database file.</p>

<p>To start a read transaction, the reader records the index of the last
valid frame in the WAL.  The reader uses this recorded "mxFrame" value
for all subsequent read operations.  New transactions can be appended
to the WAL, but as long as the reader uses its original mxFrame value
and ignores subsequently appended content, the reader will see a 
consistent snapshot of the database from a single point in time.  
This technique allows multiple concurrent readers to view different 
versions of the database content simultaneously.</p>

<p>The reader algorithm in the previous paragraphs works correctly, but 
because frames for page P can appear anywhere within the WAL, the
reader has to scan the entire WAL looking for page P frames.  If the
WAL is large (multiple megabytes is typical) that scan can be slow,
and read performance suffers.  To overcome this problem, a separate
data structure called the wal-index is maintained to expedite the
search for frames of a particular page.</p>

<tcl>hd_fragment walindexformat {WAL-index format}</tcl>
<h3>4.5 WAL-Index Format</h3>

<p>Conceptually, the wal-index is shared memory, though the current
VFS implementations use a mmapped file for the wal-index.  The mmapped
file is in the same directory as the database and has the same name
as the database with a "<tt>-shm</tt>" suffix appended.  Because
the wal-index is shared memory, SQLite does not support 
[PRAGMA journal_mode | journal_mode=WAL] 
on a network filesystem when clients are on different machines.
All users of the database must be able to share the same memory.</p>

<p>The purpose of the wal-index is to answer this question quickly:</p>

<blockquote><i>
Given a page number P and a maximum WAL frame index M,
return the largest WAL frame index for page P that does not exceed M, 
or return NULL if there are no frames for page P that do not exceed M.
</i></blockquote>

<p>The <i>M</i> value in the previous paragraph is the "mxFrame" value
defined in [WAL read algorithm | section 4.4] that is read at the start 
of a transaction and which defines the maximum frame from the WAL that 
the reader will use.</p>

<p>The wal-index is transient.  After a crash, the wal-index is
reconstructed from the original WAL file.  The VFS is required
to either truncate or zero the header of the wal-index when the last
connection to it closes.  Because the wal-index is transient, it can
use an architecture-specific format; it does not have to be cross-platform.
Hence, unlike the database and WAL file formats which store all values