Documentation Source Text

Check-in [5c5e6f523a]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Additional information about the SQLITE_IOERR_SHMSIZE error code.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256: 5c5e6f523a184b44fb40780c823418274988243ec0fd53ac465d29ebbb14fcd2
User & Date: drh 2019-04-17 15:17:39.993
Context
2019-04-17
19:17
Remove unmatched close parenthesis from fileformat.html. (check-in: 41027bf4a9 user: dan tags: trunk)
15:17
Additional information about the SQLITE_IOERR_SHMSIZE error code. (check-in: 5c5e6f523a user: drh tags: trunk)
2019-04-16
19:52
Version 3.28.0 (check-in: 8084bf3aaf user: drh tags: trunk, release, version-3.28.0)
Changes
Unified Diff Ignore Whitespace Patch
Changes to pages/pragma.in.
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
    <a href="#pragma_auto_vacuum">auto_vacuum=incremental</a> mode
    or if there are no pages on the freelist.  ^If there are fewer than
    <i>N</i> pages on the freelist, or if <i>N</i> is less than 1, or
    if the "(<i>N</i>)" argument is omitted, then the entire
    freelist is cleared.</p>
}

Pragma journal_mode {
    <p>^(<b>PRAGMA DB.journal_mode;
        <br>PRAGMA DB.journal_mode
              = <i>DELETE | TRUNCATE | PERSIST | MEMORY | WAL | OFF</i></b></p>

    <p>This pragma queries or sets the journal mode for databases
    associated with the current [database connection].</p>)^








|







649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
    <a href="#pragma_auto_vacuum">auto_vacuum=incremental</a> mode
    or if there are no pages on the freelist.  ^If there are fewer than
    <i>N</i> pages on the freelist, or if <i>N</i> is less than 1, or
    if the "(<i>N</i>)" argument is omitted, then the entire
    freelist is cleared.</p>
}

Pragma {journal_mode {persistent journal mode}} {
    <p>^(<b>PRAGMA DB.journal_mode;
        <br>PRAGMA DB.journal_mode
              = <i>DELETE | TRUNCATE | PERSIST | MEMORY | WAL | OFF</i></b></p>

    <p>This pragma queries or sets the journal mode for databases
    associated with the current [database connection].</p>)^

726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741

Pragma journal_size_limit {
    <p><b>
    PRAGMA DB.journal_size_limit<br>
    PRAGMA DB.journal_size_limit = </b><i>N</i> <b>;</b>

  <p>^If a database connection is operating in
  [locking_mode | exclusive locking mode] or in
  [journal_mode | persistent journal mode] 
  (PRAGMA journal_mode=persist) then
  after committing a transaction the [rollback journal] file may remain in
  the file-system. This increases performance for subsequent transactions
  since overwriting an existing file is faster than append to a file,
  but it also consumes
  file-system space. After a large transaction (e.g. a [VACUUM]),
  the rollback journal file may consume a very large amount of space.







|
|







726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741

Pragma journal_size_limit {
    <p><b>
    PRAGMA DB.journal_size_limit<br>
    PRAGMA DB.journal_size_limit = </b><i>N</i> <b>;</b>

  <p>^If a database connection is operating in
  [exclusive locking mode] or in
  [persistent journal mode] 
  (PRAGMA journal_mode=persist) then
  after committing a transaction the [rollback journal] file may remain in
  the file-system. This increases performance for subsequent transactions
  since overwriting an existing file is faster than append to a file,
  but it also consumes
  file-system space. After a large transaction (e.g. a [VACUUM]),
  the rollback journal file may consume a very large amount of space.
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
    <p>^The legacy_file_format pragma is initialized to OFF when an existing
    database in the newer file format is first opened.</p>

    <p>^The default file format is set by the
    [SQLITE_DEFAULT_FILE_FORMAT] compile-time option.</p>
}

Pragma locking_mode {
    <p>^(<b>PRAGMA DB.locking_mode;
    <br>PRAGMA DB.locking_mode
                = <i>NORMAL | EXCLUSIVE</i></b>)^</p>
    <p>^This pragma sets or queries the database connection locking-mode. 
    ^The locking-mode is either NORMAL or EXCLUSIVE.

    <p>^In NORMAL locking-mode (the default unless overridden at compile-time







|







819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
    <p>^The legacy_file_format pragma is initialized to OFF when an existing
    database in the newer file format is first opened.</p>

    <p>^The default file format is set by the
    [SQLITE_DEFAULT_FILE_FORMAT] compile-time option.</p>
}

Pragma {locking_mode {exclusive locking mode} {EXCLUSIVE locking mode}} {
    <p>^(<b>PRAGMA DB.locking_mode;
    <br>PRAGMA DB.locking_mode
                = <i>NORMAL | EXCLUSIVE</i></b>)^</p>
    <p>^This pragma sets or queries the database connection locking-mode. 
    ^The locking-mode is either NORMAL or EXCLUSIVE.

    <p>^In NORMAL locking-mode (the default unless overridden at compile-time
1847
1848
1849
1850
1851
1852
1853
1854

1855
1856
1857
1858
1859
1860
1861
1862
1863
1864
1865
1866
<li>These pragmas are only available in builds using non-standard
compile-time options.
<li>These pragmas are used for testing SQLite and are not recommended
for use in application programs.</ol></p>
<tcl>
foreach prag [lsort [array names PragmaBody]] {
  hd_fragment pragma_$prag


  hd_puts "<h _id=pragma_$prag style=\"display:none\"> PRAGMA "
  hd_puts [join $PragmaKeys($prag) ", "]
  hd_puts "</h>"

  foreach x $PragmaKeys($prag) {
    hd_keywords *$x "PRAGMA $x" "$x pragma"
  }
  hd_puts "<hr>"
  hd_resolve $PragmaBody($prag)
}
</tcl>
<hr>







|
>
|
<
<
<
<
|
<





1847
1848
1849
1850
1851
1852
1853
1854
1855
1856




1857

1858
1859
1860
1861
1862
<li>These pragmas are only available in builds using non-standard
compile-time options.
<li>These pragmas are used for testing SQLite and are not recommended
for use in application programs.</ol></p>
<tcl>
foreach prag [lsort [array names PragmaBody]] {
  hd_fragment pragma_$prag
  set keys $PragmaKeys($prag)
  set x [lindex $keys 0]
  hd_puts "<h _id=pragma_$prag style=\"display:none\"> PRAGMA $x</h>"




  hd_keywords *$x "PRAGMA $x" "$x pragma" {*}[lrange $keys 1 end]

  hd_puts "<hr>"
  hd_resolve $PragmaBody($prag)
}
</tcl>
<hr>
Changes to pages/rescode.in.
487
488
489
490
491
492
493
494


495
496
497
498
499
500
501
  within the xShmMap method on the [sqlite3_io_methods] object
  while trying to open a new shared memory segment.
}
RESCODE SQLITE_IOERR_SHMSIZE           {SQLITE_IOERR | (19<<8)} {
  The SQLITE_IOERR_SHMSIZE error code is an [ext-v-prim|extended error code]
  for [SQLITE_IOERR] indicating an I/O error
  within the xShmMap method on the [sqlite3_io_methods] object
  while trying to resize an existing shared memory segment.


}
RESCODE SQLITE_IOERR_SHMLOCK           {SQLITE_IOERR | (20<<8)} {
  The SQLITE_IOERR_SHMLOCK error code is no longer used.
}
RESCODE SQLITE_IOERR_SHMMAP            {SQLITE_IOERR | (21<<8)} {
  The SQLITE_IOERR_SHMMAP error code is an [ext-v-prim|extended error code]
  for [SQLITE_IOERR] indicating an I/O error







|
>
>







487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
  within the xShmMap method on the [sqlite3_io_methods] object
  while trying to open a new shared memory segment.
}
RESCODE SQLITE_IOERR_SHMSIZE           {SQLITE_IOERR | (19<<8)} {
  The SQLITE_IOERR_SHMSIZE error code is an [ext-v-prim|extended error code]
  for [SQLITE_IOERR] indicating an I/O error
  within the xShmMap method on the [sqlite3_io_methods] object
  while trying to enlarge a ["shm" file] as part of
  [WAL mode] transaction processing.  This error may indicate that
  the underlying filesystem volume is out of space.
}
RESCODE SQLITE_IOERR_SHMLOCK           {SQLITE_IOERR | (20<<8)} {
  The SQLITE_IOERR_SHMLOCK error code is no longer used.
}
RESCODE SQLITE_IOERR_SHMMAP            {SQLITE_IOERR | (21<<8)} {
  The SQLITE_IOERR_SHMMAP error code is an [ext-v-prim|extended error code]
  for [SQLITE_IOERR] indicating an I/O error
Changes to pages/walformat.in.
1
2
3
4
5
6
7
8
9
<title>WAL-mode File Format</title>
<tcl>hd_keywords {wal-index} {WAL-index format} {WAL-index File Format}</tcl>

<table_of_contents>

<p>This document describes low-level details on how [WAL mode] is
implemented on unix and windows.

<p>The separate [file format] description provides details on the

|







1
2
3
4
5
6
7
8
9
<title>WAL-mode File Format</title>
<tcl>hd_keywords {WAL-mode File Format}</tcl>

<table_of_contents>

<p>This document describes low-level details on how [WAL mode] is
implemented on unix and windows.

<p>The separate [file format] description provides details on the
23
24
25
26
27
28
29


30
31
32
33
34
35
36
37
38

39
40
41
42
43
44
45
46
47
48


49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67







68
69

70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98

99
100
101
102
103
104
105
by three separate files:

<ol>
<li> The main database file with an arbitrary name "X".
<li> The write-ahead log file, usually named "X-wal".
<li> The wal-index file, usually named "X-shm".
</ol>



<p>The format of the main database file is as described in the
[file format] document.  The [file format version numbers] at offsets
18 and 19 into the main database must both be 2 to indicate that the
database is in WAL mode.  The main database may have an arbitrary
name allowed by the underlying filesystem.  No special file suffixes
are required, though ".db", ".sqlite", and ".sqlite3" seem to be
popular choices.


<p>The write-ahead log or "wal" file is a roll-forward journal
that records transactions that have been committed but not yet applied
to the main database.  Details on the format of the wal file are
describe in the [WAL format] subsection of the main [file format]
document.  The wal file is named by appending the four characters
"-wal" to the end of the name of the main database file.  Except
on 8+3 filesystems, such names are not allowed, and in that case
the file suffix is changed to ".WAL".  But as 8+3 filesystems are
increasingly rare, that exceptional case can usually be ignored.



<p>The wal-index file or "shm" file is not actually used as a file.
Rather, individual database clients mmap the shm file and use it
as shared memory for coordinating access to the database and as a cache
for quickly locating frame within the wal file.  The name
of the shm file is the main database file name with the four characters
"-shm" appended.  Or, for 8+3 filesystems, the shm file is the main
database file with the suffix changed to ".SHM".

<p>The shm does not contain any database content and is not required
to recover the database following a crash.  For that reason, the first
client to connect to a quiescent database will normally truncate the
shm file if it exists.  Since the content of the shm file does not need
to be preserved across a crash, the shm file is never fsync()-ed to disk.
In fact, if there were a mechanism by which SQLite could tell the 
operating system to never persist the shm file to disk but always hold
it in cache memory, SQLite would use that mechanism to avoid any
unnecessary disk I/O associated with the shm file.  However, no such
mechanism exists in standard posix.








<p>When a WAL mode database is in active use, all three of the above
files exist.


<p>If the last client using the database shuts down cleanly by
calling [sqlite3_close()], then a [checkpoint] is run automatically
in order to transfer all information from the wal file
over into the main database, and both the shm file
and the wal file are unlinked.  Thus, when the database is not in
use by any client, it is usually the case that only the main
database file exists on disk.
However, if the last client did not call [sqlite3_close()] before it
shut down, or if the last client to disconnect was a read-only client,
then the final cleanup operation does not occur and the
shm and wal files may still exist on disk even when the database is
not in use.

<h2>Variations</h2>

<p>When [PRAGMA locking_mode=EXCLUSIVE] is set, only a
single client is allowed to have the database open at one time.  Since
only a single client can use the database, the shm file is omitted.
The single client uses a buffer in heap memory as a substitute for the
memory-mapped shm file.

<p>If a read/write client invokes
[sqlite3_file_control]([SQLITE_FCNTL_PERSIST_WAL]) prior to shutdown,
then at shutdown a checkpoint is still run, but the shm file and wal
file are not deleted.
This allows subsequent read-only clients to connect to and read the
database.


<h1>The WAL-Index File Format</h1>

<p>
The WAL-index or "shm" file is used to coordinate access to the database
by multiple clients, and as a cache to help clients quickly locate frames
within the wal file.








>
>









>










>
>



















>
>
>
>
>
>
>

|
>
















|
|











>







23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
by three separate files:

<ol>
<li> The main database file with an arbitrary name "X".
<li> The write-ahead log file, usually named "X-wal".
<li> The wal-index file, usually named "X-shm".
</ol>

<h2>The Main Database File</h2>

<p>The format of the main database file is as described in the
[file format] document.  The [file format version numbers] at offsets
18 and 19 into the main database must both be 2 to indicate that the
database is in WAL mode.  The main database may have an arbitrary
name allowed by the underlying filesystem.  No special file suffixes
are required, though ".db", ".sqlite", and ".sqlite3" seem to be
popular choices.

<h2>The Write-Ahead-Log or "-wal" File</h2>
<p>The write-ahead log or "wal" file is a roll-forward journal
that records transactions that have been committed but not yet applied
to the main database.  Details on the format of the wal file are
describe in the [WAL format] subsection of the main [file format]
document.  The wal file is named by appending the four characters
"-wal" to the end of the name of the main database file.  Except
on 8+3 filesystems, such names are not allowed, and in that case
the file suffix is changed to ".WAL".  But as 8+3 filesystems are
increasingly rare, that exceptional case can usually be ignored.

<tcl>hd_fragment shm  {wal-index} {"shm" file}</tcl>
<h2>The Wal-Index or "-shm" file</h2>
<p>The wal-index file or "shm" file is not actually used as a file.
Rather, individual database clients mmap the shm file and use it
as shared memory for coordinating access to the database and as a cache
for quickly locating frame within the wal file.  The name
of the shm file is the main database file name with the four characters
"-shm" appended.  Or, for 8+3 filesystems, the shm file is the main
database file with the suffix changed to ".SHM".

<p>The shm does not contain any database content and is not required
to recover the database following a crash.  For that reason, the first
client to connect to a quiescent database will normally truncate the
shm file if it exists.  Since the content of the shm file does not need
to be preserved across a crash, the shm file is never fsync()-ed to disk.
In fact, if there were a mechanism by which SQLite could tell the 
operating system to never persist the shm file to disk but always hold
it in cache memory, SQLite would use that mechanism to avoid any
unnecessary disk I/O associated with the shm file.  However, no such
mechanism exists in standard posix.

<p>Because the shm is only used to coordinate access between concurrent
clients, the shm file is omitted if [exclusive locking mode]
is set, as an optimization.  When [exclusive locking mode] is set,
SQLite uses heap memory in place of the memory-mapped shm file.

<h2>File Lifecycles</h2>

<p>When a WAL mode database is in active use, all three of the above
files usually exist.  Except, the Wal-Index file is omitted if
[exclusive locking mode] is set.

<p>If the last client using the database shuts down cleanly by
calling [sqlite3_close()], then a [checkpoint] is run automatically
in order to transfer all information from the wal file
over into the main database, and both the shm file
and the wal file are unlinked.  Thus, when the database is not in
use by any client, it is usually the case that only the main
database file exists on disk.
However, if the last client did not call [sqlite3_close()] before it
shut down, or if the last client to disconnect was a read-only client,
then the final cleanup operation does not occur and the
shm and wal files may still exist on disk even when the database is
not in use.

<h2>Variations</h2>

<p>When [PRAGMA locking_mode=EXCLUSIVE] (exclusive locking mode) is set,
only a single client is allowed to have the database open at one time.  Since
only a single client can use the database, the shm file is omitted.
The single client uses a buffer in heap memory as a substitute for the
memory-mapped shm file.

<p>If a read/write client invokes
[sqlite3_file_control]([SQLITE_FCNTL_PERSIST_WAL]) prior to shutdown,
then at shutdown a checkpoint is still run, but the shm file and wal
file are not deleted.
This allows subsequent read-only clients to connect to and read the
database.

<tcl>hd_fragment walidxfmt {WAL-index format} {WAL-index File Format}</tcl>
<h1>The WAL-Index File Format</h1>

<p>
The WAL-index or "shm" file is used to coordinate access to the database
by multiple clients, and as a cache to help clients quickly locate frames
within the wal file.