Documentation Source Text

<title>SQLite Changes From Version 3.5.9 To 3.6.0</title>
<tcl>
proc CODE {text} {
  hd_puts "<blockquote><pre>"
  hd_puts $text
  hd_puts "</pre></blockquote>"
}
proc PARAGRAPH {text} {
  hd_resolve <p>$text</p>\n
}
set level(0) 0
set level(1) 0
proc HEADING {n name {tag {}}} {
  if {$tag!=""} {
    hd_fragment $tag
  }
  global level
  incr level($n)
  for {set i [expr {$n+1}]} {$i<10} {incr i} {
    set level($i) 0
  }
  if {$n==0} {
    set num {}
  } elseif {$n==1} {
    set num $level(1).0
  } else {
    set num $level(1)
    for {set i 2} {$i<=$n} {incr i} {
      append num .$level($i)
    }
  }
  incr n 1
  hd_puts "<h$n>$num $name</h$n>"
}

hd_keywords 35to36
HEADING 0 {Moving From SQLite 3.5.9 to 3.6.0}

PARAGRAPH {
  SQLite version 3.5.0 introduced a new OS interface layer that
  provided an abstraction of the underlying operating system.
  This was an important innovation and has proven to be helpful
  in porting and maintaining SQLite.
  However, the developers have discovered some minor flaws in the
  original "virtual file system" design introduced in version 3.5.0
  and so SQLite 3.6.0 includes some small yet incompatible changes
  to address these flaws.  The purpose of these document is to detail
  the incompatible changes so that system developers will know what
  alterations they need to make in order to move from SQLite version 3.5
  to version 3.6.
}

PARAGRAPH {
  <b>Important Note:</b>  The interface changes in SQLite version 3.6.0
  only effect the rare applications that make use of the 
  [sqlite3_vfs | virtual file system] interface in SQLite.  The
  changes introduced by SQLite version 3.6.0 will no impact on the
  vast majority of SQLite applications that use the built-in interfaces
  to unix, windows, and os/2.
}

HEADING 1 {Overview Of Changes}

PARAGRAPH {
  A quick enumeration of the changes in SQLite version 3.6.0
  is provide here.  Subsequent sections will describe these
  changes in more detail.
}
PARAGRAPH {
  <ol>
  <li><p>The signature of the xAccess method on the [sqlite3_vfs] object has
      been modified to
      return an [error code] and to store its output into an integer pointed
      to by a parameter, rather than returning the output directly.
      This change allows the xAccess() method to report failures.</p></li>
  <li><p>A new error code, [SQLITE_IOERR_ACCESS], was added for cases where
      the xAccess method of [sqlite3_vfs] fails due to an I/O error. </p></li>
  <li><p>The xGetTempname method has been removed from [sqlite3_vfs].
      In its place, the xOpen method is enhanced to open a temporary file
      of its own invention when the filename parameter is NULL.</p></li>
  <li><p>The signature of the xCheckReservedLock method on [sqlite3_io_methods]
      has been modifed so that it returns an [error code] and stores its
      boolean result into an integer pointed to by a parameter.</p></li>

  <li><p><i>TBD: local time method</i></p></li>

  <li><p>Ports of SQLite to new operation systems (operating systems other
      than the unix, windows, and os/2 ports provided as part of the core)
      must provide implementations of two new functions [sqlite3_initialize()]
      and [sqlite3_shutdown()].</p></li>

  </ol>
}
</tcl>

<title>The SQLite Amalgamation</title>

<h2>The SQLite Amalgamation</h2>

<p>The core SQLite library consists of about 74 files of C code
(as of version 3.5.5) in the core with 9 additional files
in the FTS3 extension.
Most of these are "source" files in the sense that they are stored 
in the configuration management system and are maintained directly. 
But 6 of the core C files are generated automatically during the 
compilation process.  Of the 83 code files, 64 are C code and 
19 are C header files.</p>

<p>The standard makefiles for SQLite have a target for building
an object we call the "amalgamation".  
The amalgamation is a single C code file, named "sqlite3.c",
that contains all C code 
for the core SQLite library and the FTS3 extension.
This file contains about 73K lines of code 
(51K if you omit blank lines and comments) and is nearly 2.7MiB 
in size.</p>

<p>The amalgamation contains everything you need to integrate SQLite 
into a larger project.  Just copy the amalgamation into your source 
directory and compile it along with the other C code files in your project.
You may also want to make use of the "sqlite3.h" header file that 
defines the programming API for SQLite. 

<title>The SQLite Amalgamation</title>

<h2>The SQLite Amalgamation</h2>

<p>The core SQLite library consists of about 76 files of C code
(as of version 3.6.0) in the core with 11 additional files
in the FTS3 and RTREE extensions.
Most of these are "source" files in the sense that they are stored 
in the configuration management system and are maintained directly. 
But 6 of the core C files are generated automatically during the 
compilation process.  Of the 87 code files, 66 are C code and 
21 are C header files.</p>

<p>The standard makefiles for SQLite have a target for building
an object we call the "amalgamation".  
The amalgamation is a single C code file, named "sqlite3.c",
that contains all C code 
for the core SQLite library and the FTS3 and RTREE extensions.
This file contains about 91K lines of code 
(55K if you omit blank lines and comments) and is over 3 megabytes
in size.</p>

<p>The amalgamation contains everything you need to integrate SQLite 
into a larger project.  Just copy the amalgamation into your source 
directory and compile it along with the other C code files in your project.
You may also want to make use of the "sqlite3.h" header file that 
defines the programming API for SQLite. 

<title>Custom Builds Of SQLite 3.6</title>

<h1 align="center">
Custom Builds Of SQLite Version 3.6<br>
or<br>
Porting SQLite 3.6 To New Operating Systems Or Unusual Platforms
</h1>

<h2>1.0 Introduction</h2>

<p>For most applications, the recommended method for building
SQLite is to use <a href="amalgamation.html">the amalgamation</a> code
file, <b>sqlite3.c</b>, and its corresponding header file
<b>sqlite3.h</b>.  The sqlite3.c code file should compile and
run on any unix, windows, OS/2, or Mac OSX system without any changes
or special compiler options.  Most applications can simply include
the sqlite3.c file together with the other C code files that make
up the application, compile them all together, and have working
and well configured version of SQLite.</p>

<blockquote><i>Most applications work great with SQLite in its
default configuration and with no special compile-time configuration.
Most developers should be able to completely ignore this document
and simply build SQLite from
<a href="amalgamation.html">the amalgamation</a> without any
special knowledge and without taking any special actions.</i></blockquote>

<p>However, highly tuned and specialized
applications may want or need to replace some of
SQLite's built-in system interface with alternative implementations
more suitable for the needs of the application.  SQLite is designed
to be easily reconfigured at compile-time to meet the specific
needs of individual projects.  Among the compile-time configuration
options for SQLite are these:</p>

<ul>
<li><p> Replace the built-in mutex subsystem with an alternative
        implementation.  </p></li>

<li><p> Completely disable all mutexing for use in single-threaded
        applications. </p></li>

<li><p> Reconfigure the memory allocation subsystem to use a memory
        allocator other the malloc() implementation from the standard
        library. </p></li>

<li><p> Realign the memory allocation subsystem so that it never calls
        malloc() at all but instead satisfies all memory requests using
        a fixed-size memory buffer assigned to SQLite at startup. </p></li>

<li><p> Replace the interface to the file system with an alternative
        design.  In other words, override all of the system calls that
        SQLite makes in order to talk to the disk with a completely different
        set of system calls. </p></li>

<li><p> Override other operating system interfaces such as calls to obtain
        Zulu or local time. </p></li>
</ul>

<p>Generally speaking, there are three separate subsystems within
SQLite that can be modified or overridden at compile-time.  The
mutex subsystem is used to serialize access to SQLite resources that
are shared among threads.  The memory allocation subsystem is used
to allocate memory required by SQLite objects and for the database
cache.  Finally, the [sqlite3_vfs | Virtual File System] subsystem is
used to provide a portable interface between SQLite and the underlying
operating system and especially the file system.  We call these three
subsystems the "interface" subsystems of SQLite.</p>

<p>We emphasis that most applications are well-served by the 
built-in default implementations of the SQLite interface subsystems.
Developers are encouraged to use the
default built-in implementations whenever possible
and to build SQLite without any special compile-time options or parameters.
However, some highly specialized applications may benefit from
substituting or modifying one or more of these built-in SQLite
interface subsystems.
Or, if SQLite is used on an operating system other than
unix, windows, or OS/2 (recognizing that Mac OSX is unix) then none
of the interface subsystems that come built into SQLite will work
and the application will need to provide alternative implementations
suitable for the target platform.</p>

<h2>2.0 Configuring Or Replacing The Mutex Subsystem</h2>

<p>In a multithreaded environment, SQLite uses mutexes to serialize
access to shared resources.
The mutex subsystem is only required for applications that access
SQLite from multiple threads.  For single-threaded applications, or
applications which only call SQLite from a single thread, the mutex
subsystem can be completely disabled by recompiling with the following
option:</p>

<blockquote><pre>
-DSQLITE_THREADSAFE=0
</pre></blockquote>

<p>Mutexes are cheap but they are not free, so performance will be better
when mutexes are completely disabled.  The resulting library footprint
will also be a little smaller.  Disability the mutexes as compile-time
is a recommended optimization for applications were it makes sense.</p>

<p>When using SQLite as a shared library, an application can test to see
whether or not mutexes have been disabled using the
[sqlite3_threadsafe()] API.  Applications that link against SQLite at
run-time and use SQLite from multiple threads should probably check this
API to make sure they did not accidently get linked against a version of
the SQLite library that has its mutexes disabled.  Single-threaded
applications will, of course, work correctly regardless of whether or
not SQLite is configured to be threadsafe, though they will be a little
bit faster when using versions of SQLite with mutexes disabled.</p>

<p>SQLite mutexes can also be disabled at run-time using the
the [sqlite3_config()] interface.  To completely disable all mutexing,
the application can invoke:</p>

<blockquote><pre>
sqlite3_config(SQLITE_CONFIG_SINGLETHREAD);
</pre></blockquote>

<p>Disabling mutexes at run-time is not as effective as disabling them
at compile-time since SQLite still must do a test of a boolean variable
to see if mutexes are enabled or disabled at each point where a mutex
might be required.  But there is still a performance advantage for
disabling mutexes at run-time.</p>

<p>For multi-threaded applications that are careful about how they
manage threads, SQLite supports an alternative run-time configuration
that is half way between not using any mutexes and the default situation
of mutexing everything in sight.  This in-the-middle mutex alignment can
be established as follows:</p>

<blockquote><pre>
sqlite3_config(SQLITE_CONFIG_MULTITHREAD);
sqlite3_config(SQLITE_CONFIG_MEMSTATS, 0);
</pre></blockquote>

<p>There are two separate configuration changes here which can
be used either togethr or separately. The
[SQLITE_CONFIG_MULTITHREAD] setting disables the mutexes that
serialize access to [database connection] objects and 
[prepared statement] objects.  With this setting, the application
is free to use SQLite from multiple threads, but it must make sure
than no two threads try to access the same [database connection]
or any [prepared statements] associated with the same 
[database connection] at the same time.  Two threads can use SQLite
at the same time, but they must use separate [database connections].
The second [SQLITE_CONFIG_MEMSTATS] setting disables the mechanism
in SQLite that tracks the total size of all outstanding memory
allocation requests.  This omits the need to mutex each call
to [sqlite3_malloc()] and [sqlite3_free()], which saves a huge
number of mutex operations.  But a consequence of disabling the
memory statistics mechanism is that the 
[sqlite3_memory_used()], [sqlite3_memory_highwater()], and
[sqlite3_soft_heap_limit()] interfaces cease to work.
</p>

<p>SQLite uses pthreads for its mutex implementation on unix and
SQLite requires a recursive mutex.  Most modern pthread implementations
support recursive mutexes, but not all do.  For systems that do not
support recursive mutexes, it is recommended that applications operate
in single-threaded mode only.  If this is not possible, SQLite provides
an alternative recursive mutex implementation built on top of the
standard "fast" mutexes of pthreads.  This alternative
implementation should work correctly as long as pthread_equal() is
atomic and the processor has a coherient data cache.  The alternative
recursive mutex implementation is enabled by the following
compiler command-line switch:</p>

<blockquote><pre>
-DSQLITE_HOMEGROWN_RECURSIVE_MUTEX=1
</pre></blockquote>

<p>When porting SQLite to a new operating system, it is usually necessary
to completely replace the built-in mutex subsystem with an alternative
built around the mutex primitives of the new operating system.  This
is accomplished by compiling SQLite with the following option:</p>

<blockquote><pre>
-DSQLITE_MUTEX_APPDEF=1
</pre></blockquote>

<p>When SQLite is compiled with the SQLITE_MUTEX_APPDEF=1 option, it
completely omits the implementation of its 
[sqlite3_mutex_alloc | mutex primitive functions].  But the SQLite
library still attempts to call these functions where necessary, so the
application must itself implement the
[sqlite3_mutex_alloc | mutex primitive functions] and link them together
with SQLite.</p>

<h2>3.0 Configuring Or Replacing The Memory Allocation Subsystem</h2>

<p>By default, SQLite obtains the memory it needs for objects and
cache from the malloc()/free() implementation of the standard library.
There is also on-going work with experimental memory allocators that
satisfy all memory requests from a single fixed memory buffer handed
to SQLite at application start.  Additional information on these
experimental memory allocators will be provided in a future revision
of this document.</p>

<p>SQLite supports the ability of an application to specify an alternative
memory allocator at run-time using the [sqlite3_config()] interface.
For example:</p>

<blockquote><pre>
sqlite3_config(SQLITE_CONFIG_MALLOC, dlmalloc, dlfree, dlrealloc, dlmalloc_usable_size);
</pre></blockquote>

<p>The [SQLITE_CONFIG_MALLOC] setting to [sqlite3_config()] passes four
function pointers into SQLite.  The first three functions work exactly
like malloc(), free(), and realloc(), respectively, from the standard
C library.  The fourth function pointer must be for a routine that
returns the size of a memory allocation given a pointer to that allocation.
In the default memory allocator implementation for SQLite, these fourth
"memsize" function is implemented by prepending an 8-byte size integer
to the beginning of every allocation.  The memsize function is not a
standard part of must memory alloction libraries and so must be implemented
in this way.  However, Doug Lea's dlmalloc implementation, as shown in the
example above, does provide an implementation of memsize which, if used,
reduces the size overhead and execution time of every memory allocation
and deallocation.</p>

<p><i>TBD: Talk about alternative zero-malloc implementations and how to
select them at compile-time.</i></p>

<p><i>TBD: Talk about how to disable any built-in memory allocator so that
an application is required to register the memory allocator at
startup.</i></p>

<h2>4.0 Adding New Virtual File Systems</h2>

<p>Since [version 3.5.0], SQLite has supported an interface called the
[sqlite3_vfs | virtual file system] or "VFS".
This object is somewhat misnamed since it
is really an interface to whole underlying operating system,
just the filesystem.</p>

<p> One of the interesting features
of the VFS interface is that SQLite can support multiple VFSes at the
same time.  Each [database connection] has to choose a single VFS for its
use when the connection is first opened using [sqlite3_open_v2()].
But if a process contains multiple [database connections] each can choose
a different VFS.  VFSes can be added at run-time using the
[sqlite3_vfs_register()] interface.</p>

<p>The default builds for SQLite on unix, windows, and OS/2 include 
a VFS appropriate for the target platform.  SQLite builds for other
operating systems do not contain a VFS by default, but the application
can register one or more at run-time.</p>

<h2>5.0 Porting SQLite To A New Operating System</h2>

<p>In order to port SQLite to a new operating system - an operating
system not supported by default - the application
must provide...</p>

<ul>
<li> a working mutex subsystem (but only if it is multithreaded), </li>
<li> a working memory allocation subsystem (assuming it lacks malloc()
in its standard library), and</li>
<li> a working VFS implementation.</li>
</ul>

<p>All of these things can be provided in a single auxiliary C code file
and then linked with the stock "sqlite3.c" code file to generate a working
SQLite build for the target operating system.  In addition to the
alternative mutex and memory allocation subsystems and the new VFS,
the auxiliary C code file should contain implementations for the
following two routines:</p>

<ul>
<li> [sqlite3_initialize()] </li>
<li> [sqlite3_shutdown()] </li>
</ul>

<p>The "sqlite3.c" code file contains default implementations of a VFS
and of the [sqlite3_initialize()] and [sqlite3_shutdown()] functions that
are appropriate for unix, windows, and os/2.
To prevent one of these default components from being loaded when sqlite3.c
is compiled, it is necessary to add the following compile-time
option:</p>

<blockquote><pre>
-DSQLITE_OS_OTHER=1
</pre></blockquote>

<p>

<p>The SQLite core will call [sqlite3_initialize()] early.  The auxiliary
C code file can contain an implementation of sqlite3_initialize() that
registers an appropriate VFS and also perhaps initializes an alternative
mutex system (if mutexes are required) or does any memory allocation
subsystem initialization that is required.
The SQLite core never calls [sqlite3_shutdown()] but it is part of the
official SQLite API and is not otherwise provided when compiled with
-DSQLITE_OS_OTHER=1, so the auxiliary C code file should probably provide
it for completeness.</p>


<p>