Documentation Source Text

      [Glossary "Database image" {
        A serialized blob of data representing an SQLite database. The
        contents of a database file are usually a valid database image.
      }]
      [Glossary "Database file" {<span class=todo>This.</span>}]
      [Glossary "Journal file" {<span class=todo>This.</span>}]
      [Glossary "Page size" {<span class=todo>This.</span>}]
      [Glossary "Sector size" {<span class=todo>This.</span>}]

      [Glossary "Journal Section" {<span class=todo>This.</span>}]
      [Glossary "Journal Header" {<span class=todo>This.</span>}]
      [Glossary "Journal Record" {<span class=todo>This.</span>}]
      [Glossary "Master Journal Pointer" {<span class=todo>This.</span>}]

    </table>

[h1 "SQLite Database Files" sqlite_database_files]
 
  <p>
    The bulk of this document, section <cite>database_file_format</cite>,

[h2 "Journal File Formats" journal_file_formats]

[h3 "Journal File Details" journal_file_format]

    <p>
      This section describes the format used by an SQLite journal file.

    <p>
      A journal file consists of one or more journal sections, optionally
      followed by a master journal pointer field.

    <p>
      A journal file consists of one or more <i>journal headers</i>, zero
      or more <i>journal records</i> and optionally a <i>master journal
      pointer</i>. Each journal file always begins with a
      <i>journal header</i>, followed by zero or more <i>journal records</i>.
      Following this may be a second <i>journal header</i> followed by a
      second set of zero or more <i>journal records</i> and so on. There

    <p>
      All <i>journal headers</i> are positioned in the file so that they 
      start at a <i>sector size</i> aligned offset. To achieve this, unused
      space may be left between the start of the second and subsequent
      <i>journal headers</i> and the end of the <i>journal records</i>
      associated with the previous header.

    <p>
      The following requirement defines a "well-formed journal header". This concept
      is used in the following sections. A well-formed journal header is defined as
      a blob of 28 bytes for which the journal magic field is set correctly and for
      which both the page size and sector size fields are set to power of two values
      greater than 512. Because there are no restrictions on the values that may be
      stored in the record count, checksum initializer or database page count fields,
      they do not enter into the definition of a well-formed journal header.

      [fileformat_import_requirement2 H35090]
      [fileformat_import_requirement2 H35180]
      [fileformat_import_requirement2 H35190]
      [fileformat_import_requirement2 H35200]

  [h4 "Journal Record Format" journal_record_format]

    <p>
      Each <i>journal record</i> contains the original data for a database page
      modified by the <i>write transaction</i>. If a rollback is required, then
      this data may be used to restore the contents of the database page to the
      state it was in before the <i>write transaction</i> was started.

      [Tr]<td style="white-space: nowrap">4 + <i>page-size</i><td>4<td>
                        This field contains a checksum value, calculated based
                        on the contents of the journaled database page data
                        (the previous field) and the values stored in the
                        <i>checksum initializer</i> field of the preceding
                        <i>journal header</i>.
    </table>

    <p>
      The checksum value stored in each journal record is calculated based
      on the contents of the page data field of the record and the value
      stored in the checksum initializer field of the journal header that
      occurs immediately before the journal record. The checksum initializer
      field is interpreted as a 32-bit unsigned integer. To this value is
      added the value stored in every 200th byte of the page data field,
      interpreted as an 8-bit unsigned integer, beginning with the byte
      at offset (page-size % 200). The sum is accumulated in a 32-bit 
      unsigned integer. Overflow is handled by wrapping around to zero.

      <div style="padding: 0 1ex; float:right">
      <div style="padding: 0 1ex; border:1px solid black">
      Example Checksum Calculation:
      <pre>
  Sum of values:
       0xFFFFFFE1 + 
       0x00000023 +
       0x00000032 +
       0x0000009E +
       0x00000062 +
       0x0000001F
      -----------
      0x100000155

  Truncated to 32-bits: 
       0x00000155</pre>
      </div></div>

    <p>
      For example, if the page-size is 1024 bytes, then the offsets within
      the page of the bytes added to the checksum initializer value are
      24, 224, 424, 624 and 824 (the first byte of the page is offset 0, the
      last byte is offset 1023). If the values of the bytes at these offsets
      are 0x23, 0x32, 0x9E, 0x62 and 0x1F, and the value of the checksum
      initializer field is 0xFFFFFFE1, then the value stored in the checksum
      field of the journal record is 0x00000155.


    <p>
      The set of <i>journal records</i> that follow a <i>journal header</i>
      in a journal file are packed tightly together. There are no
      alignment requirements for <i>journal records</i> as there are for
      <i>journal headers</i>.

      [fileformat_import_requirement2 H35100]
      [fileformat_import_requirement2 H35110]
      [fileformat_import_requirement2 H35120]


  [h4 "Master Journal Pointer" master_journal_ptr]

    <p>
      To support <i>atomic</i> transactions that modify more than one 
      database file, SQLite sometimes includes a <i>master journal pointer</i>
      record in a journal file. A <i>master journal pointer</i>

      [Tr]<td style="white-space: nowrap">12 + <i>name-length</i><td><i>8<td>
               Finally, the <b>journal magic</b> field always contains a
               well-known 8-byte string value; the same value stored in the
               first 8 bytes of a <i>journal header</i>. The well-known
               sequence of bytes is:
                 <pre>0xd9 0xd5 0x05 0xf9 0x20 0xa1 0x63 0xd7</pre>
    </table>

      [fileformat_import_requirement2 H35140]
      [fileformat_import_requirement2 H35150]
      [fileformat_import_requirement2 H35160]
      [fileformat_import_requirement2 H35170]

[h3 "Master-Journal File Details" masterjournal_file_format]

  <p>
    A <i>master-journal file</i> contains the full paths to two or more
    <i>journal files</i>, each encoded using UTF-8 encoding and terminated
    by a single nul character (byte value 0x00). There is no padding 

    with a database image that consists of 3 pages, also of page-size bytes
    each. The contents of the initial database image pages are A, B, C and
    D respectively. The final database image content is A, E and C. As 
    depicted, the file-system contains the initial database image, ABCD.
    However, if the journal file were to be somehow invalidated, then the 
    file-system would contain the final database image, AEC.

    [Figure filesystem2.gif figure_filesystem2 "Interim file-system state used to atomically overwrite database image ABCD with AEC"]

  <p class=todo>
    The exception for free-list leaves.


  [h3 "Multiple Database Transactions" multi_db_transactions]

  <p class=todo>
    Deleting the master-journal is used as the atomic operation.


[h1 "SQLite Interoperabilty Requirements" locking_protocol]

  <p class=todo>
    This section will describe the things that an SQLite compatible database
    client has to do in order to safely operate on a database at the same
    time as regular SQLite clients. Specifically, implementing the the 
    locking protocol and updating the change-counter in the database image
    header each time the database is modified.

[h1 References]

  <table id="refs" style="width:auto; margin: 1em 5ex">
    <tr><td style="width:5ex" id="ref_comer_btree">\[1\]<td>
     Douglas Comer, <u>Ubiquitous B-Tree</u>, ACM Computing Surveys (CSUR),
     v.11 n.2, pages 121-137, June 1979.
    <tr><td style="width:5ex" id="ref_knuth_btree">\[2\]<td>

<i>journal record</i> in the <i>journal file</i> shall be read from the
corresponding journal record.

HLR H35080
The contents of all <i>database image</i> pages for which there is no valid
<i>journal record</i> shall be read from the database file.


HLR H35090
A buffer of 28 bytes shall be considered a well-formed journal 
header if it is not excluded by requirements H35180, H35190 or H35200.

HLR H35180
A buffer of 28 bytes shall only be considered a well-formed journal
header if the first eight bytes of the buffer contain the values 0xd9, 
0xd5, 0x05, 0xf9, 0x20, 0xa1, 0x63, and 0xd7, respectively.

HLR H35190
A buffer of 28 bytes shall only be considered a well-formed journal
header if the value stored in the sector size field (the 4-byte big-endian 
unsigned integer at offset 20 of the buffer) contains a value that
is an integer power of two greater than 512.

HLR H35200
A buffer of 28 bytes shall only be considered a well-formed journal
header if the value stored in the page size field (the 4-byte big-endian 
unsigned integer at offset 24 of the buffer) contains a value that
is an integer power of two greater than 512.




HLR H35100
A buffer of (8 + page size) bytes shall be considered a well-formed journal 
record if it is not excluded by requirements H35110 or H35120.

HLR H35110
A journal record shall only be considered to be well-formed if the page number
field contains a value other than zero and the locking-page number, calculated
using the page size found in the first journal header of the journal file that
contains the journal record.

HLR H35120
A journal record shall only be considered to be well-formed if the checksum 
field contains a value equal to the sum of the value stored in the 
checksum-initializer field of the journal header that precedes the record
and the value stored in every 200th byte of the page data field, interpreted
as an 8-bit unsigned integer), starting with byte offset (page-size % 200) and
ending with the byte at byte offset (page-size - 200).

HLR H35130
A buffer shall be considered to contain a well-formed master journal pointer 
record if it is not excluded from this category by requirements H35140,
H35150, H35160 or H35170.

HLR H35140
A buffer shall only be considered to be a well-formed master journal pointer
if the final eight bytes of the buffer contain the values 0xd9, 0xd5, 0x05, 
0xf9, 0x20, 0xa1, 0x63, and 0xd7, respectively.

HLR H35150
A buffer shall only be considered to be a well-formed master journal pointer
if the size of the buffer in bytes is equal to the value stored as a 4-byte 
big-endian unsigned integer starting 16 bytes before the end of the buffer.

HLR H35160
A buffer shall only be considered to be a well-formed master journal pointer
if the first four bytes of the buffer, interpreted as a big-endian unsigned
integer, contain the page number of the locking page (the value
(1 + 2<sup>30</sup> / page-size), where page-size is the value stored in
the page-size field of the first journal header of the journal file).

HLR H35170
A buffer shall only be considered to be a well-formed master journal pointer
if the value stored as a 4-byte big-endian integer starting 12 bytes before
the end of the buffer is equal to the sum of all bytes, each interpreted
as an 8-bit unsigned integer, starting at offset 4 of the buffer and continuing
until offset (buffer-size - 16) (the 17th last byte of the buffer).