Documentation Source Text

Check-in [7f154daae5]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Update of fileformat.html.
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 7f154daae5e5da78740afe22f04757bef787fa81
User & Date: dan 2009-04-10 18:18:15
Context
2009-04-11
06:09
Update diagrams in fileformat.html. check-in: 7540fed845 user: dan tags: trunk
2009-04-10
18:18
Update of fileformat.html. check-in: 7f154daae5 user: dan tags: trunk
2009-04-09
14:55
Begin adding the details of how databases are safely updated to fileformat.html. check-in: aece6e4e5f user: dan tags: trunk
Changes
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Changes to pages/fileformat.in.

   370    370   
   371    371         [Glossary "Database image" {
   372    372           A serialized blob of data representing an SQLite database. The
   373    373           contents of a database file are usually a valid database image.
   374    374         }]
   375    375         [Glossary "Database file" {<span class=todo>This.</span>}]
   376    376         [Glossary "Journal file" {<span class=todo>This.</span>}]
          377  +      [Glossary "Page size" {<span class=todo>This.</span>}]
          378  +      [Glossary "Sector size" {<span class=todo>This.</span>}]
          379  +
          380  +      [Glossary "Journal Section" {<span class=todo>This.</span>}]
          381  +      [Glossary "Journal Header" {<span class=todo>This.</span>}]
          382  +      [Glossary "Journal Record" {<span class=todo>This.</span>}]
          383  +      [Glossary "Master Journal Pointer" {<span class=todo>This.</span>}]
   377    384   
   378    385       </table>
   379    386   
   380    387   [h1 "SQLite Database Files" sqlite_database_files]
   381    388    
   382    389     <p>
   383    390       The bulk of this document, section <cite>database_file_format</cite>,
................................................................................
  2147   2154   [h2 "Journal File Formats" journal_file_formats]
  2148   2155   
  2149   2156   [h3 "Journal File Details" journal_file_format]
  2150   2157   
  2151   2158       <p>
  2152   2159         This section describes the format used by an SQLite journal file.
  2153   2160   
         2161  +    <p>
         2162  +      A journal file consists of one or more journal sections, optionally
         2163  +      followed by a master journal pointer field.
         2164  +
  2154   2165       <p>
  2155   2166         A journal file consists of one or more <i>journal headers</i>, zero
  2156   2167         or more <i>journal records</i> and optionally a <i>master journal
  2157   2168         pointer</i>. Each journal file always begins with a
  2158   2169         <i>journal header</i>, followed by zero or more <i>journal records</i>.
  2159   2170         Following this may be a second <i>journal header</i> followed by a
  2160   2171         second set of zero or more <i>journal records</i> and so on. There
................................................................................
  2214   2225       <p>
  2215   2226         All <i>journal headers</i> are positioned in the file so that they 
  2216   2227         start at a <i>sector size</i> aligned offset. To achieve this, unused
  2217   2228         space may be left between the start of the second and subsequent
  2218   2229         <i>journal headers</i> and the end of the <i>journal records</i>
  2219   2230         associated with the previous header.
  2220   2231   
         2232  +    <p>
         2233  +      The following requirement defines a "well-formed journal header". This concept
         2234  +      is used in the following sections. A well-formed journal header is defined as
         2235  +      a blob of 28 bytes for which the journal magic field is set correctly and for
         2236  +      which both the page size and sector size fields are set to power of two values
         2237  +      greater than 512. Because there are no restrictions on the values that may be
         2238  +      stored in the record count, checksum initializer or database page count fields,
         2239  +      they do not enter into the definition of a well-formed journal header.
         2240  +
         2241  +      [fileformat_import_requirement2 H35090]
         2242  +      [fileformat_import_requirement2 H35180]
         2243  +      [fileformat_import_requirement2 H35190]
         2244  +      [fileformat_import_requirement2 H35200]
         2245  +
  2221   2246     [h4 "Journal Record Format" journal_record_format]
  2222   2247   
  2223   2248       <p>
  2224   2249         Each <i>journal record</i> contains the original data for a database page
  2225   2250         modified by the <i>write transaction</i>. If a rollback is required, then
  2226   2251         this data may be used to restore the contents of the database page to the
  2227   2252         state it was in before the <i>write transaction</i> was started.
................................................................................
  2246   2271         [Tr]<td style="white-space: nowrap">4 + <i>page-size</i><td>4<td>
  2247   2272                           This field contains a checksum value, calculated based
  2248   2273                           on the contents of the journaled database page data
  2249   2274                           (the previous field) and the values stored in the
  2250   2275                           <i>checksum initializer</i> field of the preceding
  2251   2276                           <i>journal header</i>.
  2252   2277       </table>
         2278  +
         2279  +    <p>
         2280  +      The checksum value stored in each journal record is calculated based
         2281  +      on the contents of the page data field of the record and the value
         2282  +      stored in the checksum initializer field of the journal header that
         2283  +      occurs immediately before the journal record. The checksum initializer
         2284  +      field is interpreted as a 32-bit unsigned integer. To this value is
         2285  +      added the value stored in every 200th byte of the page data field,
         2286  +      interpreted as an 8-bit unsigned integer, beginning with the byte
         2287  +      at offset (page-size % 200). The sum is accumulated in a 32-bit 
         2288  +      unsigned integer. Overflow is handled by wrapping around to zero.
         2289  +
         2290  +      <div style="padding: 0 1ex; float:right">
         2291  +      <div style="padding: 0 1ex; border:1px solid black">
         2292  +      Example Checksum Calculation:
         2293  +      <pre>
         2294  +  Sum of values:
         2295  +       0xFFFFFFE1 + 
         2296  +       0x00000023 +
         2297  +       0x00000032 +
         2298  +       0x0000009E +
         2299  +       0x00000062 +
         2300  +       0x0000001F
         2301  +      -----------
         2302  +      0x100000155
         2303  +
         2304  +  Truncated to 32-bits: 
         2305  +       0x00000155</pre>
         2306  +      </div></div>
         2307  +
         2308  +    <p>
         2309  +      For example, if the page-size is 1024 bytes, then the offsets within
         2310  +      the page of the bytes added to the checksum initializer value are
         2311  +      24, 224, 424, 624 and 824 (the first byte of the page is offset 0, the
         2312  +      last byte is offset 1023). If the values of the bytes at these offsets
         2313  +      are 0x23, 0x32, 0x9E, 0x62 and 0x1F, and the value of the checksum
         2314  +      initializer field is 0xFFFFFFE1, then the value stored in the checksum
         2315  +      field of the journal record is 0x00000155.
         2316  +
  2253   2317   
  2254   2318       <p>
  2255   2319         The set of <i>journal records</i> that follow a <i>journal header</i>
  2256   2320         in a journal file are packed tightly together. There are no
  2257   2321         alignment requirements for <i>journal records</i> as there are for
  2258   2322         <i>journal headers</i>.
         2323  +
         2324  +      [fileformat_import_requirement2 H35100]
         2325  +      [fileformat_import_requirement2 H35110]
         2326  +      [fileformat_import_requirement2 H35120]
         2327  +
  2259   2328   
  2260   2329     [h4 "Master Journal Pointer" master_journal_ptr]
  2261   2330   
  2262   2331       <p>
  2263   2332         To support <i>atomic</i> transactions that modify more than one 
  2264   2333         database file, SQLite sometimes includes a <i>master journal pointer</i>
  2265   2334         record in a journal file. A <i>master journal pointer</i>
................................................................................
  2313   2382         [Tr]<td style="white-space: nowrap">12 + <i>name-length</i><td><i>8<td>
  2314   2383                  Finally, the <b>journal magic</b> field always contains a
  2315   2384                  well-known 8-byte string value; the same value stored in the
  2316   2385                  first 8 bytes of a <i>journal header</i>. The well-known
  2317   2386                  sequence of bytes is:
  2318   2387                    <pre>0xd9 0xd5 0x05 0xf9 0x20 0xa1 0x63 0xd7</pre>
  2319   2388       </table>
         2389  +
         2390  +      [fileformat_import_requirement2 H35140]
         2391  +      [fileformat_import_requirement2 H35150]
         2392  +      [fileformat_import_requirement2 H35160]
         2393  +      [fileformat_import_requirement2 H35170]
  2320   2394   
  2321   2395   [h3 "Master-Journal File Details" masterjournal_file_format]
  2322   2396   
  2323   2397     <p>
  2324   2398       A <i>master-journal file</i> contains the full paths to two or more
  2325   2399       <i>journal files</i>, each encoded using UTF-8 encoding and terminated
  2326   2400       by a single nul character (byte value 0x00). There is no padding 
................................................................................
  2556   2630       with a database image that consists of 3 pages, also of page-size bytes
  2557   2631       each. The contents of the initial database image pages are A, B, C and
  2558   2632       D respectively. The final database image content is A, E and C. As 
  2559   2633       depicted, the file-system contains the initial database image, ABCD.
  2560   2634       However, if the journal file were to be somehow invalidated, then the 
  2561   2635       file-system would contain the final database image, AEC.
  2562   2636   
  2563         -    [Figure filesystem2.gif figure_filesystem2 "Interim file-system state"]
         2637  +    [Figure filesystem2.gif figure_filesystem2 "Interim file-system state used to atomically overwrite database image ABCD with AEC"]
         2638  +
         2639  +  <p class=todo>
         2640  +    The exception for free-list leaves.
  2564   2641   
  2565   2642   
  2566   2643     [h3 "Multiple Database Transactions" multi_db_transactions]
  2567   2644   
         2645  +  <p class=todo>
         2646  +    Deleting the master-journal is used as the atomic operation.
         2647  +
  2568   2648   
  2569   2649   [h1 "SQLite Interoperabilty Requirements" locking_protocol]
  2570   2650   
         2651  +  <p class=todo>
         2652  +    This section will describe the things that an SQLite compatible database
         2653  +    client has to do in order to safely operate on a database at the same
         2654  +    time as regular SQLite clients. Specifically, implementing the the 
         2655  +    locking protocol and updating the change-counter in the database image
         2656  +    header each time the database is modified.
         2657  +
  2571   2658   [h1 References]
  2572   2659   
  2573   2660     <table id="refs" style="width:auto; margin: 1em 5ex">
  2574   2661       <tr><td style="width:5ex" id="ref_comer_btree">\[1\]<td>
  2575   2662        Douglas Comer, <u>Ubiquitous B-Tree</u>, ACM Computing Surveys (CSUR),
  2576   2663        v.11 n.2, pages 121-137, June 1979.
  2577   2664       <tr><td style="width:5ex" id="ref_knuth_btree">\[2\]<td>

Changes to req/hlr30000.txt.

   902    902   <i>journal record</i> in the <i>journal file</i> shall be read from the
   903    903   corresponding journal record.
   904    904   
   905    905   HLR H35080
   906    906   The contents of all <i>database image</i> pages for which there is no valid
   907    907   <i>journal record</i> shall be read from the database file.
   908    908   
          909  +
          910  +HLR H35090
          911  +A buffer of 28 bytes shall be considered a well-formed journal 
          912  +header if it is not excluded by requirements H35180, H35190 or H35200.
          913  +
          914  +HLR H35180
          915  +A buffer of 28 bytes shall only be considered a well-formed journal
          916  +header if the first eight bytes of the buffer contain the values 0xd9, 
          917  +0xd5, 0x05, 0xf9, 0x20, 0xa1, 0x63, and 0xd7, respectively.
          918  +
          919  +HLR H35190
          920  +A buffer of 28 bytes shall only be considered a well-formed journal
          921  +header if the value stored in the sector size field (the 4-byte big-endian 
          922  +unsigned integer at offset 20 of the buffer) contains a value that
          923  +is an integer power of two greater than 512.
          924  +
          925  +HLR H35200
          926  +A buffer of 28 bytes shall only be considered a well-formed journal
          927  +header if the value stored in the page size field (the 4-byte big-endian 
          928  +unsigned integer at offset 24 of the buffer) contains a value that
          929  +is an integer power of two greater than 512.
          930  +
          931  +
          932  +
          933  +
          934  +HLR H35100
          935  +A buffer of (8 + page size) bytes shall be considered a well-formed journal 
          936  +record if it is not excluded by requirements H35110 or H35120.
          937  +
          938  +HLR H35110
          939  +A journal record shall only be considered to be well-formed if the page number
          940  +field contains a value other than zero and the locking-page number, calculated
          941  +using the page size found in the first journal header of the journal file that
          942  +contains the journal record.
          943  +
          944  +HLR H35120
          945  +A journal record shall only be considered to be well-formed if the checksum 
          946  +field contains a value equal to the sum of the value stored in the 
          947  +checksum-initializer field of the journal header that precedes the record
          948  +and the value stored in every 200th byte of the page data field, interpreted
          949  +as an 8-bit unsigned integer), starting with byte offset (page-size % 200) and
          950  +ending with the byte at byte offset (page-size - 200).
          951  +
          952  +HLR H35130
          953  +A buffer shall be considered to contain a well-formed master journal pointer 
          954  +record if it is not excluded from this category by requirements H35140,
          955  +H35150, H35160 or H35170.
          956  +
          957  +HLR H35140
          958  +A buffer shall only be considered to be a well-formed master journal pointer
          959  +if the final eight bytes of the buffer contain the values 0xd9, 0xd5, 0x05, 
          960  +0xf9, 0x20, 0xa1, 0x63, and 0xd7, respectively.
          961  +
          962  +HLR H35150
          963  +A buffer shall only be considered to be a well-formed master journal pointer
          964  +if the size of the buffer in bytes is equal to the value stored as a 4-byte 
          965  +big-endian unsigned integer starting 16 bytes before the end of the buffer.
          966  +
          967  +HLR H35160
          968  +A buffer shall only be considered to be a well-formed master journal pointer
          969  +if the first four bytes of the buffer, interpreted as a big-endian unsigned
          970  +integer, contain the page number of the locking page (the value
          971  +(1 + 2<sup>30</sup> / page-size), where page-size is the value stored in
          972  +the page-size field of the first journal header of the journal file).
          973  +
          974  +HLR H35170
          975  +A buffer shall only be considered to be a well-formed master journal pointer
          976  +if the value stored as a 4-byte big-endian integer starting 12 bytes before
          977  +the end of the buffer is equal to the sum of all bytes, each interpreted
          978  +as an 8-bit unsigned integer, starting at offset 4 of the buffer and continuing
          979  +until offset (buffer-size - 16) (the 17th last byte of the buffer).
          980  +
   909    981