Documentation Source Text

      using an SQLite compatible system failure recovery mechanism to prevent
      database corruption or data loss are found in section 
      <cite>file_system_usage</cite>. Finally, rules and requirements for software 
      systems required to read and write live SQLite databases within a system that
      includes real SQLite clients that may also read and write to databases
      are presented in section <cite>locking_protocol</cite>.



      

  [h2 "Glossary"]
    <table id=glossary>
      <tr><td>Auto-vacuum last root-page<td>
        A page number stored as 32-bit integer at byte offset 52 of the
        database file header (see section <cite>file_header</cite>). In
        an auto-vacuum database, this is the numerically largest 
        <i>root-page</i> number in the database. Additionally, all pages that
................................................................................
      journal section in the file is depicted as containing <i>M</i> journal
      records.

    [Figure journal_format.gif figure_journal_format "Journal File Format"]

    <p>
      The following requirements define a well-formed journal section. This concept
      is used in section <cite>reading_from_files</cite>. Note that a journal 
      section that is not strictly speaking a well-formed journal section often
      contains important data. See section <cite>reading_from_files</cite> for details.

        [fileformat_import_requirement2 H35210]
        [fileformat_import_requirement2 H35220]
        [fileformat_import_requirement2 H35230]
        [fileformat_import_requirement2 H35240]










    [h4 "Journal Header Format" journal_header_format]

    <p>
      A <i>journal header</i> is <i>sector-size</i> bytes in size, where <i>
      sector-size</i> is the value returned by the xSectorSize method of
      the file handle opened on the database file. Only the first 28 bytes
      of the <i>journal header</i> are used, the remainder may contain garbage







      data. The first 28 bytes of each <i>journal header</i> consists of an 
      eight byte block set to a well-known value, followed by five big-endian 
      32-bit unsigned integer fields.
     
    [Figure journal_header.gif figure_journal_header "Journal Header Format"]

    <p>
      Figure <cite>figure_journal_header</cite> graphically depicts the layout
      of a <i>journal header</i>. The individual fields are described in
      the following table. The offsets in the 'byte offset' column of the
................................................................................
                         the size of each <i>journal header</i>.
      [Tr]<td>24<td>4<td>The <b>page size</b> field contains the database page
                         size used by the corresponding database file
                         when the journal file was created, in bytes.
    </table>

    <p>
      All <i>journal headers</i> are positioned in the file so that they 
      start at a <i>sector size</i> aligned offset. To achieve this, unused
      space may be left between the start of the second and subsequent
      <i>journal headers</i> and the end of the <i>journal records</i>
      associated with the previous header.

    <p>
      The following requirement defines a "well-formed journal header". This concept
      is used in the following sections. A well-formed journal header is defined as
      a blob of 28 bytes for which the journal magic field is set correctly and for
      which both the page size and sector size fields are set to power of two values
      greater than 512. Because there are no restrictions on the values that may be

      stored in the record count, checksum initializer or database page count fields,
      they do not enter into the definition of a well-formed journal header.

      [fileformat_import_requirement2 H35090]
      [fileformat_import_requirement2 H35180]
      [fileformat_import_requirement2 H35190]
      [fileformat_import_requirement2 H35200]

  [h4 "Journal Record Format" journal_record_format]

    <p>
      Each <i>journal record</i> contains the original data for a database page
      modified by the <i>write transaction</i>. If a rollback is required, then
      this data may be used to restore the contents of the database page to the
      state it was in before the <i>write transaction</i> was started.

    [Figure journal_record.gif figure_journal_record "Journal Record Format"]

    <p>
      A <i>journal record</i>, depicted graphically by figure
      <cite>figure_journal_record</cite>, contains three fields, as described
      in the following table. Byte offsets are relative to the start of the
................................................................................
      are 0x23, 0x32, 0x9E, 0x62 and 0x1F, and the value of the checksum
      initializer field is 0xFFFFFFE1, then the value stored in the checksum
      field of the journal record is 0x00000155.


    <p>
      The set of <i>journal records</i> that follow a <i>journal header</i>
      in a journal file are packed tightly together. There are no
      alignment requirements for <i>journal records</i> as there are for
      <i>journal headers</i>.

      [fileformat_import_requirement2 H35100]
      [fileformat_import_requirement2 H35110]
      [fileformat_import_requirement2 H35120]


  [h4 "Master Journal Pointer" master_journal_ptr]

    <p>
      To support <i>atomic</i> transactions that modify more than one 
      database file, SQLite sometimes includes a <i>master journal pointer</i>
      record in a journal file. A <i>master journal pointer</i>
      contains the name of a <i>master journal-file</i> along with a 
      check-sum and some well-known values that allow the 
      <i>master journal pointer</i> to be recognized as such when
      the journal file is read during a rollback operation.

    <p>
      As is the case for a <i>journal header</i>, the start of a <i>master
      journal pointer</i> is always positioned at a <i>sector size</i> 
      aligned offset. If the <i>journal record</i> or <i>journal header</i>
      that appears immediately before the <i>master journal pointer</i> does
      not end at an aligned offset, then unused space is left between the
      end of the <i>journal record</i> or <i>journal header</i> and the start

      of the <i>master journal pointer</i>.


    [Figure master_journal_ptr.gif figure_master_journal_ptr "Master Journal Pointer Format"]

    <p>
      A <i>master journal pointer</i>, depicted graphically by figure
      <cite>figure_master_journal_ptr</cite>, contains five fields, as 
      described in the following table. Byte offsets are relative to the 
................................................................................
  <p>
    However, if there is a valid journal file corresponding to the 
    database file present within the file-system then the situation 
    is more complicated. The file-system is considered to contain a valid 
    journal file if each of the following conditions are met:

  <ul>
    <li><p> A journal file is present in the file system, and
    <li><p> the journal file either does not end with a well-formed 
         <i>master-journal pointer</i> (see section 
         <cite>master_journal_ptr</cite>) or the <i>master-journal file</i> 
         referred to by the <i>master-journal pointer</i> is present in
         the file-system, and
    <li><p> the first 28 bytes of the journal file contain a 
         well-formed <i>journal header</i> (see section
         <cite>journal_header_format</cite>).  
  </ul>

  <p>
    If the file system contains a valid journal file, then the
    <i>page-size</i> used by and the number of pages in the <i>database
................................................................................
    corresponding page-number, then the current content of the database 
    image page is the blob of data stored in the page data field of the
    journal record. If the journal file does not contain a valid journal
    record for a page, then the current content of the database image page
    is the blob of data currently stored in the corresponding region of
    the database file.

  <p class=todo> About "valid" journal records


















  <p>
    Figure <cite>figure_filesystem1</cite> illustrates two distinct ways
    to store a database image within the file system. In this example, the
    database image consists of 4 pages of <i>page-size</i> bytes each. The
    content of each of the 4 pages is designated A, B, C and D, respectively.
    Representation 1 uses only the database file. In this case the entire
................................................................................
    [fileformat_import_requirement2 H35020]

  <p>
    If there is a valid journal file within the file-system, the 
    following requirements govern how a reader should determine the set
    of valid <i>journal records</i> that it contains.

  <p class=todo>Requirements for determining the validity of records.




  <p>
    The following requirements dictate the way in which database
    <i>page-size</i> and the number of pages in the database image
    should be determined by the reader.

    [fileformat_import_requirement2 H35030]

      using an SQLite compatible system failure recovery mechanism to prevent
      database corruption or data loss are found in section 
      <cite>file_system_usage</cite>. Finally, rules and requirements for software 
      systems required to read and write live SQLite databases within a system that
      includes real SQLite clients that may also read and write to databases
      are presented in section <cite>locking_protocol</cite>.





  [h2 "Glossary"]
    <table id=glossary>
      <tr><td>Auto-vacuum last root-page<td>
        A page number stored as 32-bit integer at byte offset 52 of the
        database file header (see section <cite>file_header</cite>). In
        an auto-vacuum database, this is the numerically largest 
        <i>root-page</i> number in the database. Additionally, all pages that
................................................................................
      journal section in the file is depicted as containing <i>M</i> journal
      records.

    [Figure journal_format.gif figure_journal_format "Journal File Format"]

    <p>
      The following requirements define a well-formed journal section. This concept
      is used in section <cite>reading_from_files</cite>. 



        [fileformat_import_requirement2 H35210]
        [fileformat_import_requirement2 H35220]
        [fileformat_import_requirement2 H35230]
        [fileformat_import_requirement2 H35240]

    <p>
      Note that a journal section that is not strictly speaking a well-formed
      journal section often contains important data. For example, many journal 
      files created by SQLite that consist of a single journal section and no
      master journal pointer contain a journal section that is not well-formed
      according to requirement H35240. See section <cite>reading_from_files</cite> 
      for details on when well-formedness is an important property of journal
      sections and when it is not.

    [h4 "Journal Header Format" journal_header_format]

    <p>




      A journal header is sector-size bytes in size, where sector-size is the 
      value stored as a 32-bit big-endian unsigned integer at byte offset 20 of
      the first journal header that occurs in the journal file. The sector-size
      must be an integer power of two greater than or equal to 512. The
      sector-size is chosen by the process that creates the journal file based
      on the considerations described in section <cite>writing_to_files</cite>.
      Only the first 28 bytes of the journal header are used, the remainder may
      contain garbage data. The first 28 bytes of each <i>journal header</i>
      consists of an eight byte block set to a well-known value, followed by
      five big-endian 32-bit unsigned integer fields.
     
    [Figure journal_header.gif figure_journal_header "Journal Header Format"]

    <p>
      Figure <cite>figure_journal_header</cite> graphically depicts the layout
      of a <i>journal header</i>. The individual fields are described in
      the following table. The offsets in the 'byte offset' column of the
................................................................................
                         the size of each <i>journal header</i>.
      [Tr]<td>24<td>4<td>The <b>page size</b> field contains the database page
                         size used by the corresponding database file
                         when the journal file was created, in bytes.
    </table>

    <p>
      Because a journal header always occurs at the start of a journal 
      section, and because the size of each journal section is always a
      multiple of sector-size bytes, journal headers are always positioned
      in the file such that they start at a sector-size aligned offset.


    <p>
      The following requirements define a "well-formed journal header". This
      concept is used in the following sections. A well-formed journal header
      is defined as a blob of 28 bytes for which the journal magic field is set
      correctly and for which both the page size and sector size fields are set
      to power of two values greater than 512. Because there are no
      restrictions on the values that may be stored in the record count,
      checksum initializer or database page count fields, they do not enter
      into the definition of a well-formed journal header.

      [fileformat_import_requirement2 H35090]
      [fileformat_import_requirement2 H35180]
      [fileformat_import_requirement2 H35190]
      [fileformat_import_requirement2 H35200]

  [h4 "Journal Record Format" journal_record_format]

    <p>
    Each <i>journal record</i> contains the data for a single database page,
    a page number identifying the page, and a checksum value used to help
    detect journal file corruption.


    [Figure journal_record.gif figure_journal_record "Journal Record Format"]

    <p>
      A <i>journal record</i>, depicted graphically by figure
      <cite>figure_journal_record</cite>, contains three fields, as described
      in the following table. Byte offsets are relative to the start of the
................................................................................
      are 0x23, 0x32, 0x9E, 0x62 and 0x1F, and the value of the checksum
      initializer field is 0xFFFFFFE1, then the value stored in the checksum
      field of the journal record is 0x00000155.


    <p>
      The set of <i>journal records</i> that follow a <i>journal header</i>
      in a journal file are packed tightly together. There are no alignment 
      requirements for <i>journal records</i>.


      [fileformat_import_requirement2 H35100]
      [fileformat_import_requirement2 H35110]
      [fileformat_import_requirement2 H35120]


  [h4 "Master Journal Pointer" master_journal_ptr]

    <p>
      If present, a master journal pointer occurs at the end of a journal file.
      There may or may not be unused space between the end of the final journal 
      section and the start of the master journal pointer.





    <p>



      A <i>master journal pointer</i> contains the full path of a 


      <i>master journal-file</i> along with a check-sum and some well-known values 
      that allow the <i>master journal pointer</i> to be unambiguously distinguished 
      from a journal record or journal header.

    [Figure master_journal_ptr.gif figure_master_journal_ptr "Master Journal Pointer Format"]

    <p>
      A <i>master journal pointer</i>, depicted graphically by figure
      <cite>figure_master_journal_ptr</cite>, contains five fields, as 
      described in the following table. Byte offsets are relative to the 
................................................................................
  <p>
    However, if there is a valid journal file corresponding to the 
    database file present within the file-system then the situation 
    is more complicated. The file-system is considered to contain a valid 
    journal file if each of the following conditions are met:

  <ul>
    <li> A journal file is present in the file system, and
    <li> the journal file either does not end with a well-formed 
         <i>master-journal pointer</i> (see section 
         <cite>master_journal_ptr</cite>) or the <i>master-journal file</i> 
         referred to by the <i>master-journal pointer</i> is present in
         the file-system, and
    <li> the first 28 bytes of the journal file contain a 
         well-formed <i>journal header</i> (see section
         <cite>journal_header_format</cite>).  
  </ul>

  <p>
    If the file system contains a valid journal file, then the
    <i>page-size</i> used by and the number of pages in the <i>database
................................................................................
    corresponding page-number, then the current content of the database 
    image page is the blob of data stored in the page data field of the
    journal record. If the journal file does not contain a valid journal
    record for a page, then the current content of the database image page
    is the blob of data currently stored in the corresponding region of
    the database file.

  <p>
    Not all journal records within a journal file are valid. A journal 
    record is said to be valid if:

  <ul>
    <li> The journal file that contains the journal record is a valid 
         journal file, and
    <li> all journal sections that occur before the journal section 
         containing the journal record are well-formed, and
    <li> the journal section that contains the journal record begins with 
         a well-formed journal header, and
    <li> the journal record itself and all journal records that occur before
         it in the same journal section are well-formed.
  </ul>

  <p>
    Note that it is not necessary for a journal record to be part of a
    well-formed journal section to be considered valid.

  <p>
    Figure <cite>figure_filesystem1</cite> illustrates two distinct ways
    to store a database image within the file system. In this example, the
    database image consists of 4 pages of <i>page-size</i> bytes each. The
    content of each of the 4 pages is designated A, B, C and D, respectively.
    Representation 1 uses only the database file. In this case the entire
................................................................................
    [fileformat_import_requirement2 H35020]

  <p>
    If there is a valid journal file within the file-system, the 
    following requirements govern how a reader should determine the set
    of valid <i>journal records</i> that it contains.

    [fileformat_import_requirement2 H35250]
    [fileformat_import_requirement2 H35260]
    [fileformat_import_requirement2 H35270]
    [fileformat_import_requirement2 H35280]

  <p>
    The following requirements dictate the way in which database
    <i>page-size</i> and the number of pages in the database image
    should be determined by the reader.

    [fileformat_import_requirement2 H35030]

count fields of the journal section's journal header.

HLR H35240
A buffer shall only be considered to contain a well-formed journal section 
if it is an integer multiple of sector-size bytes in size, where sector-size 
is the value stored in the sector size field of the journal section's journal
header.

count fields of the journal section's journal header.

HLR H35240
A buffer shall only be considered to contain a well-formed journal section 
if it is an integer multiple of sector-size bytes in size, where sector-size 
is the value stored in the sector size field of the journal section's journal
header.

HLR H35250
A journal record found within a valid journal file shall be considered a valid
journal record if it is not excluded from this category by requirement H35260,
H35270 or H35280.

HLR H35260
A journal record shall only be considered a valid journal record if it and any
other journal records that occur before it within the same journal section are
well-formed.

HLR H35270
A journal record shall only be considered a valid journal record if the journal
section to which it belongs begins with a well-formed journal header.

HLR H35280
A journal record shall only be considered a valid journal record if all journal
sections that occur before the journal section containing the journal record
are well-formed journal sections.