Documentation Source Text

Check-in [92809645e2]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Add a "document structure" section to fileio.in.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 92809645e24bcd127780b8c3ecfe2e298a498d9d
User & Date: dan 2008-07-31 10:57:44.000
Context
2008-07-31
16:11
Further incremental improvements to fileio.in. (check-in: 89222751c6 user: dan tags: trunk)
10:57
Add a "document structure" section to fileio.in. (check-in: 92809645e2 user: dan tags: trunk)
2008-07-30
18:30
Add assumptions for "safe-append" and "atomic-write" VFS implementations. (check-in: 1276228968 user: dan tags: trunk)
Changes
Unified Diff Ignore Whitespace Patch
Changes to pages/fileio.in.
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
  <b>Javascript is required for some features of this document, including 
     table of contents, figure numbering and internal references (section
     numbers and hyper-links.
  </b>
</div>
<!-- End of standard rt docs header -->

<h1>Document Overview</h1>

  <p>
    SQLite stores an entire database within a single file, the format of
    which is described in the <i>SQLite File Database File Format</i> 
    document <cite>ff_sqlitert_requirements</cite>. Each database file is
    stored within a file system, presumably provided by the host operating
    system. Instead of interfacing with the operating system directly, 







|







58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
  <b>Javascript is required for some features of this document, including 
     table of contents, figure numbering and internal references (section
     numbers and hyper-links.
  </b>
</div>
<!-- End of standard rt docs header -->

<h1>Overview</h1>

  <p>
    SQLite stores an entire database within a single file, the format of
    which is described in the <i>SQLite File Database File Format</i> 
    document <cite>ff_sqlitert_requirements</cite>. Each database file is
    stored within a file system, presumably provided by the host operating
    system. Instead of interfacing with the operating system directly, 
138
139
140
141
142
143
144
145

146













147






148



149



150












151
152
153
154
155
156
157
158
159
160
161
162
163
164
    database file is being updated are presented in section 
    <cite>fs_characteristics</cite>.

  <p>
    This document does not specify the details of the interface that must
    be implemented by the VFS adaptor component, that is left to
    <cite>capi_sqlitert_requirements</cite>.
    






















  <h2>Document Organization</h2>



    <p class=todo>



      Add this section describing what is to come.













  <h2>Glossary</h2>
    <p class=todo>
      After this document is ready, make the vocabulary consistent and
      then add a glossary here.

<h1>VFS Adaptor Related Assumptions</h1>

  <h2>Performance Related Assumptions</h2>

    ASSUMPTION A21010
      It is assumed that writing a series of sequential blocks of data to 
      a file in order is faster than writing the same blocks in an arbitrary
      order.







|
>

>
>
>
>
>
>
>
>
>
>
>
>
>

>
>
>
>
>
>
|
>
>
>
|
>
>
>
|
>
>
>
>
>
>
>
>
>
>
>
>






|







138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
    database file is being updated are presented in section 
    <cite>fs_characteristics</cite>.

  <p>
    This document does not specify the details of the interface that must
    be implemented by the VFS adaptor component, that is left to
    <cite>capi_sqlitert_requirements</cite>.

  <h2>Document Structure</h2>

    <p>
      Section <cite>vfs_assumptions</cite> of this document describes the
      various assumptions made about the system to which the VFS adaptor
      component provides access. The basic capabilities and functions 
      required from the VFS implementation are presented along with the
      description of the VFS interface in 
      <cite>capi_sqlitert_requirements</cite>. Section
      <cite>vfs_assumptions</cite> compliments this by describing in more
      detail the assumptions made about VFS implementations on which the
      algorithms presented in this document depend. Some of these assumptions
      relate to performance issues, but most concern the expected state of
      the file-system following a failure that occurs midway through 
      modifying a database file.

    <p>
      Section <cite>database_connections</cite> introduces the concept of
      a <i>database connection</i>, a combination of a file-handle and
      in-memory cache used to access a database file. It also describes the
      VFS operations required when a new <i>database connection</i> is
      created (opened), and when one is destroyed (closed).

    <p>
      Section <cite>reading_data</cite> describes the steps required to
      open a <i>read transaction</i> and read data from a database file.

    <p>
      Section <cite>writing_data</cite> describes the steps required to
      open a <i>write transaction </i> and write data to a database file.

    <p>
      Section <cite>rollback</cite> describes the way in which aborted
      <i>write transactions</i> may be rolled back (reverted), either as
      a result of an explicit user directive or because an application,
      operating system or power failure occured while SQLite was midway
      through updating a database file.

    <p>
      Section <cite>page_cache_algorithms</cite> describes some of the
      algorithms used to determine exactly which portions of the database
      file are cached by a <i>page cache</i>, and the effect that they
      have on the quantity and nature of the required VFS operations.

  <h2>Glossary</h2>
    <p class=todo>
      After this document is ready, make the vocabulary consistent and
      then add a glossary here.

<h1 id=vfs_assumptions>VFS Adaptor Related Assumptions</h1>

  <h2>Performance Related Assumptions</h2>

    ASSUMPTION A21010
      It is assumed that writing a series of sequential blocks of data to 
      a file in order is faster than writing the same blocks in an arbitrary
      order.
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542

    <p class=todo>
      Write an explanation as to how the file-system properties influence
      the model used to predict file damage after a catastrophy.
 -->
 

<h1>Database Connections</h1>

  <p>
    Within this document, the term <i>database connection</i> has a slightly
    different meaning from that which one might assume. The handles returned
    by the <code>sqlite3_open()</code> and <code>sqlite3_open16()</code>
    APIs (<span class=todo>reference</span>) are referred to as <i>database
    handles</i>.  A <i>database connection</i> is a connection to a single







|







566
567
568
569
570
571
572
573
574
575
576
577
578
579
580

    <p class=todo>
      Write an explanation as to how the file-system properties influence
      the model used to predict file damage after a catastrophy.
 -->
 

<h1 id=database_connections>Database Connections</h1>

  <p>
    Within this document, the term <i>database connection</i> has a slightly
    different meaning from that which one might assume. The handles returned
    by the <code>sqlite3_open()</code> and <code>sqlite3_open16()</code>
    APIs (<span class=todo>reference</span>) are referred to as <i>database
    handles</i>.  A <i>database connection</i> is a connection to a single
559
560
561
562
563
564
565











566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583

584
585
586
587
588
589
590
    write operations can be batched for greater efficiency. Figure 
    <cite>figure_db_connection</cite> illustrates a system containing two
    database connections, each to a separate database file. The leftmost of
    the two depicted <i>database connections</i> is shared between two 
    <i>database handles</i>. The connection illustrated towards the right of
    the diagram is used by a single <i>database handle</i>.












  <p>
    A database connection is always in one of the following states:

  <ol>
    <li><i>Unlocked state</i> (no transaction).
    <li><i>Shared lock state</i> (read-only transaction).
    <li><i>Reserved lock state</i> (read/write transaction).
    <li><i>Pending lock state</i> (read/write transaction).
    <li><i>Exclusive lock state</i> (read/write transaction).
  </ol>

  <p> Obviously, each state corresponds to the type of lock held on the
      database file. In some cases, various actions apart from simply obtaining
      the file-system lock must take place when a <i>database connection</i>
      transitions from one state to another.
 
  <p class=todo>
    Maybe a state diagram will be possible...


  <h2 id=open_new_connection>Opening a New Connection</h2>

    <p>
      Opening a new database connection is a two-step process:

    <ol>







>
>
>
>
>
>
>
>
>
>
>


















>







597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
    write operations can be batched for greater efficiency. Figure 
    <cite>figure_db_connection</cite> illustrates a system containing two
    database connections, each to a separate database file. The leftmost of
    the two depicted <i>database connections</i> is shared between two 
    <i>database handles</i>. The connection illustrated towards the right of
    the diagram is used by a single <i>database handle</i>.

  <p>
    It may at first seem odd to mention the <i>page cache</i>, primarily
    an implementation detail, in this document. However, it is necessary to 
    acknowledge and describe the <i>page cache</i> in order to provide a
    more complete explanation of the nature and quantity of IO performed
    by SQLite. Further description of the <i>page cache</i> is provided in 
    section <cite>page_cache_descripton</cite>.
    
    

<!--
  <p>
    A database connection is always in one of the following states:

  <ol>
    <li><i>Unlocked state</i> (no transaction).
    <li><i>Shared lock state</i> (read-only transaction).
    <li><i>Reserved lock state</i> (read/write transaction).
    <li><i>Pending lock state</i> (read/write transaction).
    <li><i>Exclusive lock state</i> (read/write transaction).
  </ol>

  <p> Obviously, each state corresponds to the type of lock held on the
      database file. In some cases, various actions apart from simply obtaining
      the file-system lock must take place when a <i>database connection</i>
      transitions from one state to another.
 
  <p class=todo>
    Maybe a state diagram will be possible...
 -->

  <h2 id=open_new_connection>Opening a New Connection</h2>

    <p>
      Opening a new database connection is a two-step process:

    <ol>
628
629
630
631
632
633
634

635

636


637


638
639
640
641



642
643
644
645
646
647
648

    REQ H21009
      If the <i>database file header</i> cannot be read from a newly opened 
      database file (because the file is less than 100 bytes in size), the 
      connections <i>expected page-size</i> shall be set to the compile time
      value of the SQLITE_DEFAULT_PAGESIZE option.




  <h2>Closing a Connection</h2>


    <p class=todo>


      Close file handles. Discard pager cache.
    </ol>

  <h2>The Page Cache</h2>



 
<h1 id=reading_data>Reading Data</h1>
  <p>
    In order to return data from the database to the user, for example as
    the results of a SELECT query, SQLite must at some point read data
    from the database file. Usually, data is read from the database file in 
    aligned blocks of <i>page-size</i> bytes. The exception is when the







>

>
|
>
>
|
>
>
|
<

|
>
>
>







678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694

695
696
697
698
699
700
701
702
703
704
705
706

    REQ H21009
      If the <i>database file header</i> cannot be read from a newly opened 
      database file (because the file is less than 100 bytes in size), the 
      connections <i>expected page-size</i> shall be set to the compile time
      value of the SQLITE_DEFAULT_PAGESIZE option.

  <h2>Closing a Connection</h2>

    <p>
      Closing a database connection is a simple matter. The open VFS 
      file-handle is closed and in-memory <i>page cache</i> related resources
      are released. 

    REQ H21040
      When a <i>database connection</i> is closed, SQLite shall close the 
      associated file handle at the VFS level.


  <h2 id=page_cache_descripton>The Page Cache</h2>

    <p class=todo>
      Description of the page cache is.
 
<h1 id=reading_data>Reading Data</h1>
  <p>
    In order to return data from the database to the user, for example as
    the results of a SELECT query, SQLite must at some point read data
    from the database file. Usually, data is read from the database file in 
    aligned blocks of <i>page-size</i> bytes. The exception is when the