Documentation Source Text

Check-in [491737c7cf]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Modifications to fileformat.html.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 491737c7cf62892944eae07f67ac78b205b491b4
User & Date: dan 2009-04-13 14:24:28.000
References
2009-04-13
15:07
Merge [ebd923dab6] and [491737c7cf]. (check-in: 8f18472bac user: dan tags: trunk)
Context
2009-04-13
15:07
Merge [ebd923dab6] and [491737c7cf]. (check-in: 8f18472bac user: dan tags: trunk)
14:24
Modifications to fileformat.html. (check-in: 491737c7cf user: dan tags: trunk)
09:49
fileformat.html updates. (check-in: 46434fb185 user: dan tags: trunk)
Changes
Unified Diff Ignore Whitespace Patch
Changes to pages/fileformat.in.
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
      using an SQLite compatible system failure recovery mechanism to prevent
      database corruption or data loss are found in section 
      <cite>file_system_usage</cite>. Finally, rules and requirements for software 
      systems required to read and write live SQLite databases within a system that
      includes real SQLite clients that may also read and write to databases
      are presented in section <cite>locking_protocol</cite>.



      

  [h2 "Glossary"]
    <table id=glossary>
      <tr><td>Auto-vacuum last root-page<td>
        A page number stored as 32-bit integer at byte offset 52 of the
        database file header (see section <cite>file_header</cite>). In
        an auto-vacuum database, this is the numerically largest 
        <i>root-page</i> number in the database. Additionally, all pages that







<
<
<
<







211
212
213
214
215
216
217




218
219
220
221
222
223
224
      using an SQLite compatible system failure recovery mechanism to prevent
      database corruption or data loss are found in section 
      <cite>file_system_usage</cite>. Finally, rules and requirements for software 
      systems required to read and write live SQLite databases within a system that
      includes real SQLite clients that may also read and write to databases
      are presented in section <cite>locking_protocol</cite>.





  [h2 "Glossary"]
    <table id=glossary>
      <tr><td>Auto-vacuum last root-page<td>
        A page number stored as 32-bit integer at byte offset 52 of the
        database file header (see section <cite>file_header</cite>). In
        an auto-vacuum database, this is the numerically largest 
        <i>root-page</i> number in the database. Additionally, all pages that
2241
2242
2243
2244
2245
2246
2247
2248
2249
2250
2251
2252
2253
2254
2255
2256









2257
2258
2259
2260

2261


2262
2263
2264
2265
2266
2267
2268
2269
2270
2271
2272
2273
      journal section in the file is depicted as containing <i>M</i> journal
      records.

    [Figure journal_format.gif figure_journal_format "Journal File Format"]

    <p>
      The following requirements define a well-formed journal section. This concept
      is used in section <cite>reading_from_files</cite>. Note that a journal 
      section that is not strictly speaking a well-formed journal section often
      contains important data. See section <cite>reading_from_files</cite> for details.

        [fileformat_import_requirement2 H35210]
        [fileformat_import_requirement2 H35220]
        [fileformat_import_requirement2 H35230]
        [fileformat_import_requirement2 H35240]










    [h4 "Journal Header Format" journal_header_format]

    <p>
      A <i>journal header</i> is <i>sector-size</i> bytes in size, where <i>

      sector-size</i> is the value returned by the xSectorSize method of


      the file handle opened on the database file. Only the first 28 bytes
      of the <i>journal header</i> are used, the remainder may contain garbage
      data. The first 28 bytes of each <i>journal header</i> consists of an 
      eight byte block set to a well-known value, followed by five big-endian 
      32-bit unsigned integer fields.
     
    [Figure journal_header.gif figure_journal_header "Journal Header Format"]

    <p>
      Figure <cite>figure_journal_header</cite> graphically depicts the layout
      of a <i>journal header</i>. The individual fields are described in
      the following table. The offsets in the 'byte offset' column of the







|
<
<






>
>
>
>
>
>
>
>
>



|
>
|
>
>
|
|
|
|
|







2237
2238
2239
2240
2241
2242
2243
2244


2245
2246
2247
2248
2249
2250
2251
2252
2253
2254
2255
2256
2257
2258
2259
2260
2261
2262
2263
2264
2265
2266
2267
2268
2269
2270
2271
2272
2273
2274
2275
2276
2277
2278
2279
      journal section in the file is depicted as containing <i>M</i> journal
      records.

    [Figure journal_format.gif figure_journal_format "Journal File Format"]

    <p>
      The following requirements define a well-formed journal section. This concept
      is used in section <cite>reading_from_files</cite>. 



        [fileformat_import_requirement2 H35210]
        [fileformat_import_requirement2 H35220]
        [fileformat_import_requirement2 H35230]
        [fileformat_import_requirement2 H35240]

    <p>
      Note that a journal section that is not strictly speaking a well-formed
      journal section often contains important data. For example, many journal 
      files created by SQLite that consist of a single journal section and no
      master journal pointer contain a journal section that is not well-formed
      according to requirement H35240. See section <cite>reading_from_files</cite> 
      for details on when well-formedness is an important property of journal
      sections and when it is not.

    [h4 "Journal Header Format" journal_header_format]

    <p>
      A journal header is sector-size bytes in size, where sector-size is the 
      value stored as a 32-bit big-endian unsigned integer at byte offset 20 of
      the first journal header that occurs in the journal file. The sector-size
      must be an integer power of two greater than or equal to 512. The
      sector-size is chosen by the process that creates the journal file based
      on the considerations described in section <cite>writing_to_files</cite>.
      Only the first 28 bytes of the journal header are used, the remainder may
      contain garbage data. The first 28 bytes of each <i>journal header</i>
      consists of an eight byte block set to a well-known value, followed by
      five big-endian 32-bit unsigned integer fields.
     
    [Figure journal_header.gif figure_journal_header "Journal Header Format"]

    <p>
      Figure <cite>figure_journal_header</cite> graphically depicts the layout
      of a <i>journal header</i>. The individual fields are described in
      the following table. The offsets in the 'byte offset' column of the
2298
2299
2300
2301
2302
2303
2304
2305
2306
2307
2308
2309
2310
2311
2312
2313
2314
2315
2316

2317
2318
2319
2320
2321
2322
2323
2324
2325
2326
2327
2328
2329
2330
2331
2332
2333
2334
2335
2336
2337
2338
                         the size of each <i>journal header</i>.
      [Tr]<td>24<td>4<td>The <b>page size</b> field contains the database page
                         size used by the corresponding database file
                         when the journal file was created, in bytes.
    </table>

    <p>
      All <i>journal headers</i> are positioned in the file so that they 
      start at a <i>sector size</i> aligned offset. To achieve this, unused
      space may be left between the start of the second and subsequent
      <i>journal headers</i> and the end of the <i>journal records</i>
      associated with the previous header.

    <p>
      The following requirement defines a "well-formed journal header". This concept
      is used in the following sections. A well-formed journal header is defined as
      a blob of 28 bytes for which the journal magic field is set correctly and for
      which both the page size and sector size fields are set to power of two values
      greater than 512. Because there are no restrictions on the values that may be

      stored in the record count, checksum initializer or database page count fields,
      they do not enter into the definition of a well-formed journal header.

      [fileformat_import_requirement2 H35090]
      [fileformat_import_requirement2 H35180]
      [fileformat_import_requirement2 H35190]
      [fileformat_import_requirement2 H35200]

  [h4 "Journal Record Format" journal_record_format]

    <p>
      Each <i>journal record</i> contains the original data for a database page
      modified by the <i>write transaction</i>. If a rollback is required, then
      this data may be used to restore the contents of the database page to the
      state it was in before the <i>write transaction</i> was started.

    [Figure journal_record.gif figure_journal_record "Journal Record Format"]

    <p>
      A <i>journal record</i>, depicted graphically by figure
      <cite>figure_journal_record</cite>, contains three fields, as described
      in the following table. Byte offsets are relative to the start of the







|
|
<
|
|


|
|
|
|
|
>
|
|









|
|
|
<







2304
2305
2306
2307
2308
2309
2310
2311
2312

2313
2314
2315
2316
2317
2318
2319
2320
2321
2322
2323
2324
2325
2326
2327
2328
2329
2330
2331
2332
2333
2334
2335
2336

2337
2338
2339
2340
2341
2342
2343
                         the size of each <i>journal header</i>.
      [Tr]<td>24<td>4<td>The <b>page size</b> field contains the database page
                         size used by the corresponding database file
                         when the journal file was created, in bytes.
    </table>

    <p>
      Because a journal header always occurs at the start of a journal 
      section, and because the size of each journal section is always a

      multiple of sector-size bytes, journal headers are always positioned
      in the file such that they start at a sector-size aligned offset.

    <p>
      The following requirements define a "well-formed journal header". This
      concept is used in the following sections. A well-formed journal header
      is defined as a blob of 28 bytes for which the journal magic field is set
      correctly and for which both the page size and sector size fields are set
      to power of two values greater than 512. Because there are no
      restrictions on the values that may be stored in the record count,
      checksum initializer or database page count fields, they do not enter
      into the definition of a well-formed journal header.

      [fileformat_import_requirement2 H35090]
      [fileformat_import_requirement2 H35180]
      [fileformat_import_requirement2 H35190]
      [fileformat_import_requirement2 H35200]

  [h4 "Journal Record Format" journal_record_format]

    <p>
    Each <i>journal record</i> contains the data for a single database page,
    a page number identifying the page, and a checksum value used to help
    detect journal file corruption.


    [Figure journal_record.gif figure_journal_record "Journal Record Format"]

    <p>
      A <i>journal record</i>, depicted graphically by figure
      <cite>figure_journal_record</cite>, contains three fields, as described
      in the following table. Byte offsets are relative to the start of the
2392
2393
2394
2395
2396
2397
2398
2399
2400
2401
2402
2403
2404
2405
2406
2407
2408
2409
2410
2411
2412
2413
2414
2415
2416
2417
2418
2419
2420
2421
2422
2423
2424
2425
2426

2427
2428
2429
2430
2431
2432
2433
      are 0x23, 0x32, 0x9E, 0x62 and 0x1F, and the value of the checksum
      initializer field is 0xFFFFFFE1, then the value stored in the checksum
      field of the journal record is 0x00000155.


    <p>
      The set of <i>journal records</i> that follow a <i>journal header</i>
      in a journal file are packed tightly together. There are no
      alignment requirements for <i>journal records</i> as there are for
      <i>journal headers</i>.

      [fileformat_import_requirement2 H35100]
      [fileformat_import_requirement2 H35110]
      [fileformat_import_requirement2 H35120]


  [h4 "Master Journal Pointer" master_journal_ptr]

    <p>
      To support <i>atomic</i> transactions that modify more than one 
      database file, SQLite sometimes includes a <i>master journal pointer</i>
      record in a journal file. A <i>master journal pointer</i>
      contains the name of a <i>master journal-file</i> along with a 
      check-sum and some well-known values that allow the 
      <i>master journal pointer</i> to be recognized as such when
      the journal file is read during a rollback operation.

    <p>
      As is the case for a <i>journal header</i>, the start of a <i>master
      journal pointer</i> is always positioned at a <i>sector size</i> 
      aligned offset. If the <i>journal record</i> or <i>journal header</i>
      that appears immediately before the <i>master journal pointer</i> does
      not end at an aligned offset, then unused space is left between the
      end of the <i>journal record</i> or <i>journal header</i> and the start
      of the <i>master journal pointer</i>.


    [Figure master_journal_ptr.gif figure_master_journal_ptr "Master Journal Pointer Format"]

    <p>
      A <i>master journal pointer</i>, depicted graphically by figure
      <cite>figure_master_journal_ptr</cite>, contains five fields, as 
      described in the following table. Byte offsets are relative to the 







|
|
<









<
<
|
|
<
|
<


<
<
<
|
<
|
|
>







2397
2398
2399
2400
2401
2402
2403
2404
2405

2406
2407
2408
2409
2410
2411
2412
2413
2414


2415
2416

2417

2418
2419



2420

2421
2422
2423
2424
2425
2426
2427
2428
2429
2430
      are 0x23, 0x32, 0x9E, 0x62 and 0x1F, and the value of the checksum
      initializer field is 0xFFFFFFE1, then the value stored in the checksum
      field of the journal record is 0x00000155.


    <p>
      The set of <i>journal records</i> that follow a <i>journal header</i>
      in a journal file are packed tightly together. There are no alignment 
      requirements for <i>journal records</i>.


      [fileformat_import_requirement2 H35100]
      [fileformat_import_requirement2 H35110]
      [fileformat_import_requirement2 H35120]


  [h4 "Master Journal Pointer" master_journal_ptr]

    <p>


      If present, a master journal pointer occurs at the end of a journal file.
      There may or may not be unused space between the end of the final journal 

      section and the start of the master journal pointer.


    <p>



      A <i>master journal pointer</i> contains the full path of a 

      <i>master journal-file</i> along with a check-sum and some well-known values 
      that allow the <i>master journal pointer</i> to be unambiguously distinguished 
      from a journal record or journal header.

    [Figure master_journal_ptr.gif figure_master_journal_ptr "Master Journal Pointer Format"]

    <p>
      A <i>master journal pointer</i>, depicted graphically by figure
      <cite>figure_master_journal_ptr</cite>, contains five fields, as 
      described in the following table. Byte offsets are relative to the 
2513
2514
2515
2516
2517
2518
2519
2520
2521
2522
2523
2524
2525
2526
2527
2528
2529
2530
2531
2532
2533
  <p>
    However, if there is a valid journal file corresponding to the 
    database file present within the file-system then the situation 
    is more complicated. The file-system is considered to contain a valid 
    journal file if each of the following conditions are met:

  <ul>
    <li><p> A journal file is present in the file system, and
    <li><p> the journal file either does not end with a well-formed 
         <i>master-journal pointer</i> (see section 
         <cite>master_journal_ptr</cite>) or the <i>master-journal file</i> 
         referred to by the <i>master-journal pointer</i> is present in
         the file-system, and
    <li><p> the first 28 bytes of the journal file contain a 
         well-formed <i>journal header</i> (see section
         <cite>journal_header_format</cite>).  
  </ul>

  <p>
    If the file system contains a valid journal file, then the
    <i>page-size</i> used by and the number of pages in the <i>database







|
|




|







2510
2511
2512
2513
2514
2515
2516
2517
2518
2519
2520
2521
2522
2523
2524
2525
2526
2527
2528
2529
2530
  <p>
    However, if there is a valid journal file corresponding to the 
    database file present within the file-system then the situation 
    is more complicated. The file-system is considered to contain a valid 
    journal file if each of the following conditions are met:

  <ul>
    <li> A journal file is present in the file system, and
    <li> the journal file either does not end with a well-formed 
         <i>master-journal pointer</i> (see section 
         <cite>master_journal_ptr</cite>) or the <i>master-journal file</i> 
         referred to by the <i>master-journal pointer</i> is present in
         the file-system, and
    <li> the first 28 bytes of the journal file contain a 
         well-formed <i>journal header</i> (see section
         <cite>journal_header_format</cite>).  
  </ul>

  <p>
    If the file system contains a valid journal file, then the
    <i>page-size</i> used by and the number of pages in the <i>database
2546
2547
2548
2549
2550
2551
2552



2553














2554
2555
2556
2557
2558
2559
2560
    corresponding page-number, then the current content of the database 
    image page is the blob of data stored in the page data field of the
    journal record. If the journal file does not contain a valid journal
    record for a page, then the current content of the database image page
    is the blob of data currently stored in the corresponding region of
    the database file.




  <p class=todo> About "valid" journal records















  <p>
    Figure <cite>figure_filesystem1</cite> illustrates two distinct ways
    to store a database image within the file system. In this example, the
    database image consists of 4 pages of <i>page-size</i> bytes each. The
    content of each of the 4 pages is designated A, B, C and D, respectively.
    Representation 1 uses only the database file. In this case the entire







>
>
>
|
>
>
>
>
>
>
>
>
>
>
>
>
>
>







2543
2544
2545
2546
2547
2548
2549
2550
2551
2552
2553
2554
2555
2556
2557
2558
2559
2560
2561
2562
2563
2564
2565
2566
2567
2568
2569
2570
2571
2572
2573
2574
    corresponding page-number, then the current content of the database 
    image page is the blob of data stored in the page data field of the
    journal record. If the journal file does not contain a valid journal
    record for a page, then the current content of the database image page
    is the blob of data currently stored in the corresponding region of
    the database file.

  <p>
    Not all journal records within a journal file are valid. A journal 
    record is said to be valid if:

  <ul>
    <li> The journal file that contains the journal record is a valid 
         journal file, and
    <li> all journal sections that occur before the journal section 
         containing the journal record are well-formed, and
    <li> the journal section that contains the journal record begins with 
         a well-formed journal header, and
    <li> the journal record itself and all journal records that occur before
         it in the same journal section are well-formed.
  </ul>

  <p>
    Note that it is not necessary for a journal record to be part of a
    well-formed journal section to be considered valid.

  <p>
    Figure <cite>figure_filesystem1</cite> illustrates two distinct ways
    to store a database image within the file system. In this example, the
    database image consists of 4 pages of <i>page-size</i> bytes each. The
    content of each of the 4 pages is designated A, B, C and D, respectively.
    Representation 1 uses only the database file. In this case the entire
2585
2586
2587
2588
2589
2590
2591
2592



2593
2594
2595
2596
2597
2598
2599
    [fileformat_import_requirement2 H35020]

  <p>
    If there is a valid journal file within the file-system, the 
    following requirements govern how a reader should determine the set
    of valid <i>journal records</i> that it contains.

  <p class=todo>Requirements for determining the validity of records.




  <p>
    The following requirements dictate the way in which database
    <i>page-size</i> and the number of pages in the database image
    should be determined by the reader.

    [fileformat_import_requirement2 H35030]







|
>
>
>







2599
2600
2601
2602
2603
2604
2605
2606
2607
2608
2609
2610
2611
2612
2613
2614
2615
2616
    [fileformat_import_requirement2 H35020]

  <p>
    If there is a valid journal file within the file-system, the 
    following requirements govern how a reader should determine the set
    of valid <i>journal records</i> that it contains.

    [fileformat_import_requirement2 H35250]
    [fileformat_import_requirement2 H35260]
    [fileformat_import_requirement2 H35270]
    [fileformat_import_requirement2 H35280]

  <p>
    The following requirements dictate the way in which database
    <i>page-size</i> and the number of pages in the database image
    should be determined by the reader.

    [fileformat_import_requirement2 H35030]
Changes to req/hlr30000.txt.
997
998
999
1000
1001
1002
1003




















1004
count fields of the journal section's journal header.

HLR H35240
A buffer shall only be considered to contain a well-formed journal section 
if it is an integer multiple of sector-size bytes in size, where sector-size 
is the value stored in the sector size field of the journal section's journal
header.




























>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
count fields of the journal section's journal header.

HLR H35240
A buffer shall only be considered to contain a well-formed journal section 
if it is an integer multiple of sector-size bytes in size, where sector-size 
is the value stored in the sector size field of the journal section's journal
header.

HLR H35250
A journal record found within a valid journal file shall be considered a valid
journal record if it is not excluded from this category by requirement H35260,
H35270 or H35280.

HLR H35260
A journal record shall only be considered a valid journal record if it and any
other journal records that occur before it within the same journal section are
well-formed.

HLR H35270
A journal record shall only be considered a valid journal record if the journal
section to which it belongs begins with a well-formed journal header.

HLR H35280
A journal record shall only be considered a valid journal record if all journal
sections that occur before the journal section containing the journal record
are well-formed journal sections.