Documentation Source Text

Check-in [61fb0b3465]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Updated VFS documentation.
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256: 61fb0b3465ea3a467c354e24b269bc331eeada945a21e3e516611fa9b8133fe3
User & Date: drh 2017-11-08 15:43:30
Context
2017-11-13
13:11
Typos in the new walindex.html document. check-in: 59536a697b user: drh tags: trunk
2017-11-08
15:43
Updated VFS documentation. check-in: 61fb0b3465 user: drh tags: trunk
2017-11-04
20:26
Continuing work on the WAL file format document. check-in: 0d151f472f user: drh tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to pages/vfs.in.

1
2
3
4
5

6
7
8
9
10
11
12
13



14
15

16
17
18
19
20
21
22
..
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
..
64
65
66
67
68
69
70


71
72
73
74

75
76
77
78
79
80
81
82
..
99
100
101
102
103
104
105





















106
107
108
109
110
111
112
113
...
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
...
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
...
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
...
297
298
299
300
301
302
303
304
305
<title>The OS Backend (VFS) To SQLite</title>
<tcl>hd_keywords VFS VFSes {OS backend}</tcl>

<h1 align="center">
The SQLite OS Interface or "VFS"

</h1>

<p>
This article describes the SQLite OS portability layer or "VFS" - the
module at the bottom of the SQLite implementation stack
that provides portability across operating systems.
</p>




<img src="images/vfs1.gif" align="right" hspace="10">
<h2>1.0 The VFS In Relation To The Rest Of SQLite</h2>


<p>
The internal organization of the SQLite library can be viewed as the
stack of modules shown to the right.
The Tokenizer, Parser, and Code Generator components are used to
process SQL statements and convert them into executable programs 
in a virtual machine language or byte code.
................................................................................
in SQLite needs to communicate with the operating
system, they invoke methods in the VFS.  The VFS then invokes the
operating-specific code needed to satisfy the request.
Hence, porting SQLite to a new
operating system is simply a matter of writing a new OS interface layer
or "VFS".</p>

<h2>2.0 Multiple VFSes</h2>

<p>
The standard SQLite source tree contains built-in VFSes for unix
and windows.  Alternative VFSes can be
added at start-time or run-time using the
[sqlite3_vfs_register()] interface.
</p>
................................................................................
Separate [database connections] within the same process can be using
different VFSes at the same time.   For that matter, if a single
database connection has multiple database files open using
the [ATTACH] command, then each attached database might be using a
different VFS.
</p>



<p>
Unix builds come with multiple VFSes built-in.  The default VFS
for unix is called "unix" and is the VFS used in an overwhelming
majority of applications.  Other VFSes that can be found in unix

may include:
</p>

<ol>
<li><p><b>unix-dotfile</b> - uses dot-file locking rather than
          POSIX advisory locks.
<li><p><b>unix-excl</b> - obtains and holds an exclusive lock on
          database files, preventing other processes from accessing the
................................................................................
resulting in database corruption.  The "unix-none" VFS in particular
does no locking at all and will easily result in database corruption if
used by two or more database connections at the same time.
Programmers are encouraged to use only "unix" or "unix-excl" unless
there is a compelling reason to do otherwise.
</p>






















<h2>2.1 Specifying Which VFS To Use</h2>

<p>
There is always one VFS which is the default VFS.  On unix systems,
the "unix" VFS comes up as the default and on windows it is "win32".
If no other actions are taken, new database connections will make use
of the default VFS.
</p>
................................................................................
<p>
The VFS specified by a URI has the highest priority.  After that comes
a VFS specified as the fourth argument to [sqlite3_open_v2()].  The
default VFS is used if no VFS is specified otherwise.
</p>

<tcl>hd_fragment shim {VFS shims} {shims}</tcl>
<h2>2.2 VFS Shims</h2>

<p>
From the point of view of the uppers layers of the SQLite stack, each
open database file uses exactly one VFS.
But in practice, a particular VFS might
just be a thin wrapper around another VFS that does the real work.
We call a wrapper VFS a "shim".
................................................................................
(implemented in the 
[http://www.sqlite.org/src/doc/trunk/src/test_vfstrace.c | test_vfstrace.c]
source file) that writes a message associated with each VFS method call
into a log file, then passes control off to another VFS to do the actual
work.
</p>

<h2>2.3 Other Example VFSes</h2>

<p>
The following are other VFS implementations available in the public
SQLite source tree:
</p>

<ul>
................................................................................
<p>
There are other VFS implementations both in the core SQLite source code
library and in available extensions.  The list above is not meant to be
exhaustive but merely representative of the kinds of features that can
be realized using the VFS interface.
</p>

<h2>3.0 VFS Implementations</h2>

<p>
A new VFS is implemented by subclassing three objects:
</p>

<ul>
<li>[sqlite3_vfs]
................................................................................
a call to [sqlite3_vfs_register()].  The VFS implementation also
provides subclasses for [sqlite3_file] and [sqlite3_io_methods] but
those objects are not registered directly with SQLite.  Instead, the
[sqlite3_file] object is returned from the xOpen method of
[sqlite3_vfs] and the [sqlite3_file] object points to an instance
of the [sqlite3_io_methods] object.
</p>

<h2>To Be Continued...</h2>
|


|
<
>
|







>
>
>

<
>







 







|







 







>
>

|
|
<
>
|







 







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|







 







|







 







|







 







|







 







<
<
1
2
3
4

5
6
7
8
9
10
11
12
13
14
15
16
17

18
19
20
21
22
23
24
25
..
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
..
67
68
69
70
71
72
73
74
75
76
77
78

79
80
81
82
83
84
85
86
87
...
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
...
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
...
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
...
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
...
323
324
325
326
327
328
329


<title>The SQLite OS Interface or "VFS"</title>
<tcl>hd_keywords VFS VFSes {OS backend}</tcl>

<table_of_contents>


<h1>Introduction</h1>

<p>
This article describes the SQLite OS portability layer or "VFS" - the
module at the bottom of the SQLite implementation stack
that provides portability across operating systems.
</p>

<h1>The VFS In Relation To The Rest Of SQLite</h1>

<div>
<img src="images/vfs1.gif" align="right" hspace="10">

</div>

<p>
The internal organization of the SQLite library can be viewed as the
stack of modules shown to the right.
The Tokenizer, Parser, and Code Generator components are used to
process SQL statements and convert them into executable programs 
in a virtual machine language or byte code.
................................................................................
in SQLite needs to communicate with the operating
system, they invoke methods in the VFS.  The VFS then invokes the
operating-specific code needed to satisfy the request.
Hence, porting SQLite to a new
operating system is simply a matter of writing a new OS interface layer
or "VFS".</p>

<h1>Multiple VFSes</h1>

<p>
The standard SQLite source tree contains built-in VFSes for unix
and windows.  Alternative VFSes can be
added at start-time or run-time using the
[sqlite3_vfs_register()] interface.
</p>
................................................................................
Separate [database connections] within the same process can be using
different VFSes at the same time.   For that matter, if a single
database connection has multiple database files open using
the [ATTACH] command, then each attached database might be using a
different VFS.
</p>

<h2>Standard Unix VFSes</h2>

<p>
Unix builds come with multiple built-in VFSes.  The default VFS
for unix is called "unix" and is used in most applications.

Other VFSes that might be found in unix (depending on compile-time
options) include:
</p>

<ol>
<li><p><b>unix-dotfile</b> - uses dot-file locking rather than
          POSIX advisory locks.
<li><p><b>unix-excl</b> - obtains and holds an exclusive lock on
          database files, preventing other processes from accessing the
................................................................................
resulting in database corruption.  The "unix-none" VFS in particular
does no locking at all and will easily result in database corruption if
used by two or more database connections at the same time.
Programmers are encouraged to use only "unix" or "unix-excl" unless
there is a compelling reason to do otherwise.
</p>

<h2>Standard Windows VFSes</h2>


<p>
Windows builds also come with multiple built-in VFSes.  The default
Windows VFS is called "win32" and is used in most applications.
Other VFSes that might be found on windows builds include:
</p>

<ol>
<li><p><b>win32-longpath</b> - like "win32" except that pathnames can
          be up to 65534 bytes in length, whereas pathnames max out at
          1040 bytes in "win32".
<li><p><b>win32-none</b> - all file locking operations are no-ops.
<li><p><b>win32-longpath-none</b> - combination of "win32-longpath"
          and "win32-none" - long pathnames are supported and all lock
          operations are no-ops.
</ol>

<p>As with unix, most of the code for the various Windows VFSes is shared.

<h2>Specifying Which VFS To Use</h2>

<p>
There is always one VFS which is the default VFS.  On unix systems,
the "unix" VFS comes up as the default and on windows it is "win32".
If no other actions are taken, new database connections will make use
of the default VFS.
</p>
................................................................................
<p>
The VFS specified by a URI has the highest priority.  After that comes
a VFS specified as the fourth argument to [sqlite3_open_v2()].  The
default VFS is used if no VFS is specified otherwise.
</p>

<tcl>hd_fragment shim {VFS shims} {shims}</tcl>
<h2>VFS Shims</h2>

<p>
From the point of view of the uppers layers of the SQLite stack, each
open database file uses exactly one VFS.
But in practice, a particular VFS might
just be a thin wrapper around another VFS that does the real work.
We call a wrapper VFS a "shim".
................................................................................
(implemented in the 
[http://www.sqlite.org/src/doc/trunk/src/test_vfstrace.c | test_vfstrace.c]
source file) that writes a message associated with each VFS method call
into a log file, then passes control off to another VFS to do the actual
work.
</p>

<h2>Other Example VFSes</h2>

<p>
The following are other VFS implementations available in the public
SQLite source tree:
</p>

<ul>
................................................................................
<p>
There are other VFS implementations both in the core SQLite source code
library and in available extensions.  The list above is not meant to be
exhaustive but merely representative of the kinds of features that can
be realized using the VFS interface.
</p>

<h1>VFS Implementations</h1>

<p>
A new VFS is implemented by subclassing three objects:
</p>

<ul>
<li>[sqlite3_vfs]
................................................................................
a call to [sqlite3_vfs_register()].  The VFS implementation also
provides subclasses for [sqlite3_file] and [sqlite3_io_methods] but
those objects are not registered directly with SQLite.  Instead, the
[sqlite3_file] object is returned from the xOpen method of
[sqlite3_vfs] and the [sqlite3_file] object points to an instance
of the [sqlite3_io_methods] object.
</p>


Changes to pages/walformat.in.

304
305
306
307
308
309
310










311
312
313
314
315
316
And since there are guaranteed to be unused slots in aHash, that means
the loop that computes X is guaranteed to terminate.  The expected size
of X is less than 2.  The worst case is that X will be the same as the
number of entries in aPgno, in which case the algorithm runs at about
the same speed as a linear scan of aPgno.  But that worst case performance
is exceedingly rare.  Usually, the size of X will be small and the use
of the aHash array allows one to compute FindFrame(P,M) much faster.











<p>Note that each 32768-byte unit of the shm file has its own aHash and
aPgno arrays.  The aHash array for a single unit is only helpful in finding
aPgno entries in that same unit.  The overall FindFrame(P,M) function
needs to do hash lookups beginning with the latest unit and working
backwards to the oldest unit until it finds an answer.







>
>
>
>
>
>
>
>
>
>






304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
And since there are guaranteed to be unused slots in aHash, that means
the loop that computes X is guaranteed to terminate.  The expected size
of X is less than 2.  The worst case is that X will be the same as the
number of entries in aPgno, in which case the algorithm runs at about
the same speed as a linear scan of aPgno.  But that worst case performance
is exceedingly rare.  Usually, the size of X will be small and the use
of the aHash array allows one to compute FindFrame(P,M) much faster.

<p>Here is an alternative way of describing the hash look-up algorithm:
Start with h = (P * 383)%8192 and look at aHash[h] and subsequent entries,
wrapping around to zero when h reaches 8192, until finding an entry with
aHash[h]==0.  All aPgno entries having a page number of P will have an
index that is one of the aHash[h] values thusly computed.
But not all the computed aHash[h] values will
meet the matching criteria, so you must check them independently.  The
speed advantage comes about because normally this set of h values is
very small.

<p>Note that each 32768-byte unit of the shm file has its own aHash and
aPgno arrays.  The aHash array for a single unit is only helpful in finding
aPgno entries in that same unit.  The overall FindFrame(P,M) function
needs to do hash lookups beginning with the latest unit and working
backwards to the oldest unit until it finds an answer.