Documentation Source Text

Check-in [224d684b8b]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Fix typos in new documents.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 224d684b8bd2d44031b4a8fe96c68a007bb5c170
User & Date: drh 2013-04-16 13:57:56.615
Context
2013-04-17
12:58
Add a documentation page describing how to write run-time loadable extensions. (check-in: 6da9f893bc user: drh tags: trunk)
2013-04-16
13:57
Fix typos in new documents. (check-in: 224d684b8b user: drh tags: trunk)
13:09
Added the mmap.html and errlog.html documents. Updates to the "How to Corrupt" document. Update the change log for 3.7.17. (check-in: ccb921fd36 user: drh tags: trunk)
Changes
Unified Diff Ignore Whitespace Patch
Changes to pages/changes.in.
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
<li>Report rollback recovery in the [error log] as SQLITE_NOTICE_RECOVER_ROLLBACK.
    Change the error log code for WAL recover from 
    SQLITE_OK to SQLITE_NOTICE_RECOVER_WAL.
<li>Report the risky uses of [unlinked database files] and 
   [database filename aliasing] as SQLITE_WARNING messages in the [error log].
}

chng {2012-04-12 (3.7.16.2)} {
<li>Fix a bug (present since version 3.7.13) that could result in database corruption
    on windows if two or more processes try to access the same database file at the
    same time and immediately after third process crashed in the middle of committing
    to that same file.  See ticket 
    [http://www.sqlite.org/src/info/7ff3120e4f | 7ff3120e4f] for further
    information.








|







47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
<li>Report rollback recovery in the [error log] as SQLITE_NOTICE_RECOVER_ROLLBACK.
    Change the error log code for WAL recover from 
    SQLITE_OK to SQLITE_NOTICE_RECOVER_WAL.
<li>Report the risky uses of [unlinked database files] and 
   [database filename aliasing] as SQLITE_WARNING messages in the [error log].
}

chng {2013-04-12 (3.7.16.2)} {
<li>Fix a bug (present since version 3.7.13) that could result in database corruption
    on windows if two or more processes try to access the same database file at the
    same time and immediately after third process crashed in the middle of committing
    to that same file.  See ticket 
    [http://www.sqlite.org/src/info/7ff3120e4f | 7ff3120e4f] for further
    information.

Changes to pages/errlog.in.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

68
69
70

71
72
73

74
75
76
77
78
79
80
81
82
83
84
85
86
<title>The Error And Warning Log</title>
<tcl>hd_keywords {errlog} {error log}</tcl>

<h1 align="center">The Error And Warning Log</h1>

<p>SQLite can be configured to invoke a callback function containing
an error code and a terse error message whenever anomalies occur.
The mechanism is very helpful in tracking obscure problems that
occur rarely and in the field.  Application developers are encouraged
to take advantage of the error logging facility of SQLite in their
products, as it presents hardly any overhead at all, but can be a
huge aid for debugging.</p>

<h2>Setting Up The Error Logging Callback</h2>

<p>There can only be a single error logging callback per process.
The error logging callback is registered at start-time using C-code
similar to the following:

<blockquote><pre>
[sqlite3_config]([SQLITE_CONFIG_LOG], errLogCallback, pData);
</pre></blockquote>

<p>The error logger callback function might look something like this:</p>

<blockquote><pre>
void errorLogCallback(void *pArg, int iErrCode, const char *zMsg){
  fprintf(stderr, "(%d) %s\n", iErrCode, zMsg);
}
</pre></blockquote>

<p>The example above illustrates the signature of the error logger callback,
at least.  In an embedded application, one usually does not print
messages on stderr.  Instead, one might store the messages in a
preallocated circular buffer where they can be accessed when diagnostic
information is needed during debugging.  Or perhaps the messages can be
sent to [http://en.wikipedia.org/wiki/Syslog | Syslog].  Somehow, the
messages need to be stored where they are accessible to developers debugging
issues, not displayed to end users.</p>

<p>Do not misunderstand: There is nothing technically wrong with displaying 
the error logger messages to end users.  The messages do not contain
sensitive or private information that must be protected from unauthorized
viewing.  Rather the messages are technical in nature and are not useful
or meaningful to the typical end user.  These messages are intended for
database geeks.  Display them accordingly.</p>

<h2>Interface Details</h2>

<p>The third argument to the [sqlite3_config]([SQLITE_CONFIG_LOG],...) call
(the "pData" argument in the example above) is a pointer to arbitrary
data.  SQLite passes this pointer through to the first argument of the
error logger callback.  The pointer can be used to pass application-specific 
setup or state information, if desired.  Or it can simply be a NULL 
pointer which is ignored by the callback.</p>

<p>The second argument to the error logger callback is an integer
[extended error code].  The third argument to the error logger is the
text of the error message.  The error message text is stored in a fixed-length
stack buffer in the calling function and so will only be valid for the
duration of the error logger callback function.  The error logger should
make a copy of this message into persistent storage if retention of the
message is needed.</p>

<p>The error logger callback should be treated like a signal handler.
The application should do something with the error, quickly, then return
as soon as possible.  No other SQLite APIs should be invoked, directly or

indirectly, from the error logger.  In particular, the error logger callback
is invoked when a memory allocation fails, so it is generally a bad idea
to try to allocate memory inside the error logger.</p>


<p>Applications can use the [sqlite3_log(E,F,..)] API to send new messages
to the log, if desired.</p>


<h2>Variety of Error Messages</h2>

<p>The error messages that might be sent to the error logger and their
exact format is subject to changes from one release to the next.  So
applications should not depend on any particular error message text or
error codes.  Things do not change capriciously, but they do sometimes
changes.</p>

<p>The following is a partial list of the kinds of messages that might
appear in the error logger callback.</p>

<ul>







|


|




















|
|




|
|





|
|



|
|














|

>
|

|
>


|
>





|







1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
<title>The Error And Warning Log</title>
<tcl>hd_keywords {errlog} {error log}</tcl>

<h1 align="center">The Error And Warning Log</h1>

<p>SQLite can be configured to invoke a callback function containing
an error code and a terse error message whenever anomalies occur.
This mechanism is very helpful in tracking obscure problems that
occur rarely and in the field.  Application developers are encouraged
to take advantage of the error logging facility of SQLite in their
products, as it is very low CPU and memory cost but can be a
huge aid for debugging.</p>

<h2>Setting Up The Error Logging Callback</h2>

<p>There can only be a single error logging callback per process.
The error logging callback is registered at start-time using C-code
similar to the following:

<blockquote><pre>
[sqlite3_config]([SQLITE_CONFIG_LOG], errLogCallback, pData);
</pre></blockquote>

<p>The error logger callback function might look something like this:</p>

<blockquote><pre>
void errorLogCallback(void *pArg, int iErrCode, const char *zMsg){
  fprintf(stderr, "(%d) %s\n", iErrCode, zMsg);
}
</pre></blockquote>

<p>The example above illustrates the signature of the error logger callback.
However, in an embedded application, one usually does not print
messages on stderr.  Instead, one might store the messages in a
preallocated circular buffer where they can be accessed when diagnostic
information is needed during debugging.  Or perhaps the messages can be
sent to [http://en.wikipedia.org/wiki/Syslog | Syslog].  Somehow, the
messages need to be stored where they are accessible to developers,
not displayed to end users.</p>

<p>Do not misunderstand: There is nothing technically wrong with displaying 
the error logger messages to end users.  The messages do not contain
sensitive or private information that must be protected from unauthorized
viewing.  Rather the messages are technical in nature and are not useful
or meaningful to the typical end user.  The messages coming from the
error logger are intended for database geeks.  Display them accordingly.</p>

<h2>Interface Details</h2>

<p>The third argument to the [sqlite3_config]([SQLITE_CONFIG_LOG],...) 
interface (the "pData" argument in the example above) is a pointer to arbitrary
data.  SQLite passes this pointer through to the first argument of the
error logger callback.  The pointer can be used to pass application-specific 
setup or state information, if desired.  Or it can simply be a NULL 
pointer which is ignored by the callback.</p>

<p>The second argument to the error logger callback is an integer
[extended error code].  The third argument to the error logger is the
text of the error message.  The error message text is stored in a fixed-length
stack buffer in the calling function and so will only be valid for the
duration of the error logger callback function.  The error logger should
make a copy of this message into persistent storage if retention of the
message is needed.</p>

<p>The error logger callback should be treated like a signal handler.
The application should save off or otherwise process the error, then return
as soon as possible.  No other SQLite APIs should be invoked, directly or
indirectly, from the error logger.  SQLite is <u>not</u> reentrant through
the error logger callback.  In particular, the error logger callback
is invoked when a memory allocation fails, so it is generally a bad idea
to try to allocate memory inside the error logger.  Do not even think
about trying to store the error message in another SQLite database.</p>

<p>Applications can use the [sqlite3_log(E,F,..)] API to send new messages
to the log, if desired, but this is discouraged.  The [sqlite3_log()]
interface is intended for use by extensions only, not by applications.</p>

<h2>Variety of Error Messages</h2>

<p>The error messages that might be sent to the error logger and their
exact format is subject to changes from one release to the next.  So
applications should not depend on any particular error message text formats or
error codes.  Things do not change capriciously, but they do sometimes
changes.</p>

<p>The following is a partial list of the kinds of messages that might
appear in the error logger callback.</p>

<ul>
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133


134
135
136
137

138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
SQLITE_WARNING messages are logged when database files are renamed or
aliased in ways that can lead to database corruption.
(See [unlink corruption | 1] and [database filename aliasing | 2] for
additional information.)
</p>

<li><p>
Out of memory (OOM) errors conditions generate error logging events
with the SQLITE_NOMEM error code and a message that says how many bytes
of memory were requested by the failed allocation.
</p>

<li><p>I/O errors in the OS-interface generate error logging events.
The message to these events gives the line number in the source code where
the error originated and the filename associated with the event when
there is a corresponding file. </p>

<li><p>When database corruption is detected, an SQLITE_CORRUPT error
logger callback is invoked.  As with I/O errors, the error message text
contains the line number in the original source code where the error
was first detected.</p>

<li><p>
An error logger callbacked is invoked on SQLITE_MISUSE errors.


</ul>

<p>SQLite strives to keep error logger traffic low and only send messages
to the error logger when there really is something wrong.  Applications

might take this further and deliberately ignore certain classes of error
messages that they do not care about.  For example, an application that
makes frequent database schema changes might want to ignore all
SQLITE_SCHEMA errors.</p>

<h2>Summary</h2>

<p>The use of the error logger callback is highly recommended.
The debugging information that the error logger provides has proven
very useful in tracking down obscure problems that occurs with applications
after they get into the field.  The error logger callback has also 
proven useful in catching errors occasional errors that the application
misses because of inconsistent checking of API return codes.
Developers are encouraged to implement an error logger callback early
in the development cycle in order to spot unexpected behavior quickly,
and to leave the error logger callback turned on through deployment.
If the error logger never finds a problem, then no harm is done.  
But failure to set up an appropriate error logger might leave the
developres wishing they had after mysterious and unreproducible
bug reports start coming back from the field.</p>







|















|
>
>




>
|
















|
<
|
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161

162
SQLITE_WARNING messages are logged when database files are renamed or
aliased in ways that can lead to database corruption.
(See [unlink corruption | 1] and [database filename aliasing | 2] for
additional information.)
</p>

<li><p>
Out of memory (OOM) error conditions generate error logging events
with the SQLITE_NOMEM error code and a message that says how many bytes
of memory were requested by the failed allocation.
</p>

<li><p>I/O errors in the OS-interface generate error logging events.
The message to these events gives the line number in the source code where
the error originated and the filename associated with the event when
there is a corresponding file. </p>

<li><p>When database corruption is detected, an SQLITE_CORRUPT error
logger callback is invoked.  As with I/O errors, the error message text
contains the line number in the original source code where the error
was first detected.</p>

<li><p>
An error logger callback is invoked on SQLITE_MISUSE errors.
This is useful in detecting application design issues when return codes
are not consistently checked in the application code.
</ul>

<p>SQLite strives to keep error logger traffic low and only send messages
to the error logger when there really is something wrong.  Applications
might further cull the error message traffic 
by deliberately ignore certain classes of error
messages that they do not care about.  For example, an application that
makes frequent database schema changes might want to ignore all
SQLITE_SCHEMA errors.</p>

<h2>Summary</h2>

<p>The use of the error logger callback is highly recommended.
The debugging information that the error logger provides has proven
very useful in tracking down obscure problems that occurs with applications
after they get into the field.  The error logger callback has also 
proven useful in catching errors occasional errors that the application
misses because of inconsistent checking of API return codes.
Developers are encouraged to implement an error logger callback early
in the development cycle in order to spot unexpected behavior quickly,
and to leave the error logger callback turned on through deployment.
If the error logger never finds a problem, then no harm is done.  
But failure to set up an appropriate error logger might compromise

diagnostic capabilities later on.</p>
Changes to pages/mmap.in.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
<title>Memory-Mapped I/O</title>
<tcl>hd_keywords {mmap} {memory-mapped I/O}</tcl>

<h1 align="center">Memory-Mapped I/O</h1>

<p>The default mechanism by which SQLite accesses and updates database disk
files is the xRead() and xWrite() methods of the
[sqlite3_io_methods] VFS object.  These methods are typically implemented as
"read()" and "write()" system calls which cause the operating system
to copy disk content between the kernel buffer cache and user space.</p>

<p>Beginning with [version 3.7.17], SQLite now also has the option of 
accessing disk content directly using memory-mapped I/O and the new
xFetch() and xUnfetch() methods on [sqlite3_io_methods].</p>

<p>There are advantages and disadvantages to using memory-mapped I/O:
Advantages include:</p>

<ol>
<li><p>Many operations, especially I/O intensive operations, can be much
    faster since content does need to be copied between kernel space
    and user space.  In some cases, performance can nearly double.</p>

<li><p>The SQLite library may need less RAM since it is shares pages with
    the operating-system page cache and does not always need its own copy of
    working pages.</p>
</ol>

<p>But there are also disadvantages:</p>

<ol>











|



|







|







1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
<title>Memory-Mapped I/O</title>
<tcl>hd_keywords {mmap} {memory-mapped I/O}</tcl>

<h1 align="center">Memory-Mapped I/O</h1>

<p>The default mechanism by which SQLite accesses and updates database disk
files is the xRead() and xWrite() methods of the
[sqlite3_io_methods] VFS object.  These methods are typically implemented as
"read()" and "write()" system calls which cause the operating system
to copy disk content between the kernel buffer cache and user space.</p>

<p>Beginning with [version 3.7.17], SQLite has the option of 
accessing disk content directly using memory-mapped I/O and the new
xFetch() and xUnfetch() methods on [sqlite3_io_methods].</p>

<p>There are advantages and disadvantages to using memory-mapped I/O.
Advantages include:</p>

<ol>
<li><p>Many operations, especially I/O intensive operations, can be much
    faster since content does need to be copied between kernel space
    and user space.  In some cases, performance can nearly double.</p>

<li><p>The SQLite library may need less RAM since it shares pages with
    the operating-system page cache and does not always need its own copy of
    working pages.</p>
</ol>

<p>But there are also disadvantages:</p>

<ol>
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108













109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
into the newly allocated heap memory.  This involves (at a minimum)
a copy of the entire page.</p>

<p>But if SQLite wants to access a page of the databse file and
memory mapped I/O is enabled, it first calls the xFetch() method.
The xFetch() method asks the operating system to return a pointer to
the requested page, if possible.  If the requested page has been or
can be mapped into the applications address space, then xFetch returns
a pointer to that page for SQLite to use without having to copy anything.
Skipping the copy step is what makes memory mapped I/O faster.</p>

<p>SQLite does not assume that the xFetch() method will work.  If
a call to xFetch() returns a NULL pointer (indicating that the requested
pages is not currently mapped into the applications address space) then
SQLite silently falls back to using xRead().  An error is only reported
if xRead() also fails.</p>

<p>When updating the database file, SQLite always makes a copy of the
page content into heap memory before modifying the page.  This is necessary
since the changes are not suppose to be visible to other processes until
after the transaction commits and so the changes must occur in private space.
After all needed changes are completed, xWrite() is used to move the content
back into the database file.  The current xWrite() implementions for both
unix and windows check to see if section of the file being written is 
mapped into the applications address space, and if it is the write operation
is implemented using memcpy() rather than invoking a "write()" system call,
but that is just an implementation detail.  A memory copy occurs either way.
So the use of memory mapped I/O does not significantly change the performance
of database changes.  Memory mapped I/O is mostly a benefit for queries.</p>

<h2>Configuring Memory-Mapped I/O</h2>

<p>The "mmap_size" is the maximum number of bytes of the database file that
SQLite will try to map into the process address space at one time.  The
"mmap_size" applies separately to each database file, so the total amount
of process address space that could potentially be used is the "mmap_size"
times the number of open database files.</p>














<p>If mmap_size is set to N then all current implementation map the first
N bytes of the database file and use legacy xRead() calls for any content
beyond N bytes.  If the database file is smaller then N, then the entire
file is mapped.  In the future, new OS interfaces could, in theory, map
regions of the file other than the first N bytes, but no such OS interface
implementation currently exists.</p>

<p>The mmap_size is set separately for each database file using the
"[PRAGMA mmap_size]" statement.  The usual default mmap_size is zero,
meaning that memory mapped I/O is disabled by default.  However, the
default mmap_size can be increased either at compile-time using
the [SQLITE_DEFAULT_MMAP_SIZE] macro or at start-time using the
[sqlite3_config]([SQLITE_CONFIG_MMAP_SIZE],...) interface.</p>

<p>SQLite also maintains a hard upper bound on the mmap_size.  Attempts
to increase the mmap_size above this hard upper bound (using
[PRAGMA mmap_size] will automatically reduce the mmap_size to the hard
upper bound.  If the hard upper bound is zero, then memory mapped I/O
is impossible.  The hard upper bound can be set at compile-time using
the [SQLITE_MAX_MMAP_SIZE] macro.  If [SQLITE_MAX_MMAP_SIZE] is set to
zero, then the code used to implement memory mapped I/O is omitted from
the build.  The hard upper bound is automatically set to zero on certain
platforms (ex: OpenBSD) where memory mapped I/O does not work due to the
lack of a unified buffer cache.</p>







|





|






|













|
|


>
>
>
>
>
>
>
>
>
>
>
>
>
|

|

|











|







71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
into the newly allocated heap memory.  This involves (at a minimum)
a copy of the entire page.</p>

<p>But if SQLite wants to access a page of the databse file and
memory mapped I/O is enabled, it first calls the xFetch() method.
The xFetch() method asks the operating system to return a pointer to
the requested page, if possible.  If the requested page has been or
can be mapped into the application address space, then xFetch returns
a pointer to that page for SQLite to use without having to copy anything.
Skipping the copy step is what makes memory mapped I/O faster.</p>

<p>SQLite does not assume that the xFetch() method will work.  If
a call to xFetch() returns a NULL pointer (indicating that the requested
page is not currently mapped into the applications address space) then
SQLite silently falls back to using xRead().  An error is only reported
if xRead() also fails.</p>

<p>When updating the database file, SQLite always makes a copy of the
page content into heap memory before modifying the page.  This is necessary
since the changes are not suppose to be visible to other processes until
after the transaction commits and so the changes must occur in private memory.
After all needed changes are completed, xWrite() is used to move the content
back into the database file.  The current xWrite() implementions for both
unix and windows check to see if section of the file being written is 
mapped into the applications address space, and if it is the write operation
is implemented using memcpy() rather than invoking a "write()" system call,
but that is just an implementation detail.  A memory copy occurs either way.
So the use of memory mapped I/O does not significantly change the performance
of database changes.  Memory mapped I/O is mostly a benefit for queries.</p>

<h2>Configuring Memory-Mapped I/O</h2>

<p>The "mmap_size" is the maximum number of bytes of the database file that
SQLite will try to map into the process address space at one time.  The
mmap_size applies separately to each database file, so the total amount
of process address space that could potentially be used is the mmap_size
times the number of open database files.</p>

<p>To activate memory-mapped I/O, an application an set the mmap_size to some
large value.  For example:</p>

<blockquote><pre>
PRAGMA mmap_size=268435456;
</pre></blockquote>

<p>To disable memory-mapped I/O, simply set the mmap_size to zero:</p>

<blockquote><pre>
PRAGMA mmap_size=0;
</pre></blockquote>

<p>If mmap_size is set to N then all current implementations map the first
N bytes of the database file and use legacy xRead() calls for any content
beyond N bytes.  If the database file is smaller then N bytes, then the entire
file is mapped.  In the future, new OS interfaces could, in theory, map
regions of the file other than the first N bytes, but no such 
implementation currently exists.</p>

<p>The mmap_size is set separately for each database file using the
"[PRAGMA mmap_size]" statement.  The usual default mmap_size is zero,
meaning that memory mapped I/O is disabled by default.  However, the
default mmap_size can be increased either at compile-time using
the [SQLITE_DEFAULT_MMAP_SIZE] macro or at start-time using the
[sqlite3_config]([SQLITE_CONFIG_MMAP_SIZE],...) interface.</p>

<p>SQLite also maintains a hard upper bound on the mmap_size.  Attempts
to increase the mmap_size above this hard upper bound (using
[PRAGMA mmap_size]) will automatically cap the mmap_size at the hard
upper bound.  If the hard upper bound is zero, then memory mapped I/O
is impossible.  The hard upper bound can be set at compile-time using
the [SQLITE_MAX_MMAP_SIZE] macro.  If [SQLITE_MAX_MMAP_SIZE] is set to
zero, then the code used to implement memory mapped I/O is omitted from
the build.  The hard upper bound is automatically set to zero on certain
platforms (ex: OpenBSD) where memory mapped I/O does not work due to the
lack of a unified buffer cache.</p>