Documentation Source Text

Artifact [d293d4abb9]

Artifact d293d4abb98ca6e33f82c0c4a202f4ab89424c75:

<title>Custom Builds Of SQLite</title>
<tcl>hd_keywords {porting SQLite} {custom builds}</tcl>

<h1 align="center">
Custom Builds Of SQLite<br>
Porting SQLite To New Operating Systems

<h2>1.0 Introduction</h2>

<p>For most applications, the recommended method for building
SQLite is to use <a href="amalgamation.html">the amalgamation</a> code
file, <b>sqlite3.c</b>, and its corresponding header file
<b>sqlite3.h</b>.  The sqlite3.c code file should compile and
run on any unix, Windows system without any changes
or special compiler options.  Most applications can simply include
the sqlite3.c file together with the other C code files that make
up the application, compile them all together, and have working
and well configured version of SQLite.</p>

<blockquote><i>Most applications work great with SQLite in its
default configuration and with no special compile-time configuration.
Most developers should be able to completely ignore this document
and simply build SQLite from
<a href="amalgamation.html">the amalgamation</a> without any
special knowledge and without taking any special actions.</i></blockquote>

<p>However, highly tuned and specialized
applications may want or need to replace some of
SQLite's built-in system interfaces with alternative implementations
more suitable for the needs of the application.  SQLite is designed
to be easily reconfigured at compile-time to meet the specific
needs of individual projects.  Among the compile-time configuration
options for SQLite are these:</p>

<li><p> Replace the built-in mutex subsystem with an alternative
        implementation.  </p></li>

<li><p> Completely disable all mutexing for use in single-threaded
        applications. </p></li>

<li><p> Reconfigure the memory allocation subsystem to use a memory
        allocator other the malloc() implementation from the standard
        library. </p></li>

<li><p> Realign the memory allocation subsystem so that it never calls
        malloc() at all but instead satisfies all memory requests using
        a fixed-size memory buffer assigned to SQLite at startup. </p></li>

<li><p> Replace the interface to the file system with an alternative
        design.  In other words, override all of the system calls that
        SQLite makes in order to talk to the disk with a completely different
        set of system calls. </p></li>

<li><p> Override other operating system interfaces such as calls to obtain
        Zulu or local time. </p></li>

<p>Generally speaking, there are three separate subsystems within
SQLite that can be modified or overridden at compile-time.  The
mutex subsystem is used to serialize access to SQLite resources that
are shared among threads.  The memory allocation subsystem is used
to allocate memory required by SQLite objects and for the database
cache.  Finally, the [sqlite3_vfs | Virtual File System] subsystem is
used to provide a portable interface between SQLite and the underlying
operating system and especially the file system.  We call these three
subsystems the "interface" subsystems of SQLite.</p>

<p>We emphasis that most applications are well-served by the 
built-in default implementations of the SQLite interface subsystems.
Developers are encouraged to use the
default built-in implementations whenever possible
and to build SQLite without any special compile-time options or parameters.
However, some highly specialized applications may benefit from
substituting or modifying one or more of these built-in SQLite
interface subsystems.
Or, if SQLite is used on an operating system other than
Unix (Linux or Mac OS X), Windows (Win32 or WinCE), or OS/2 then none
of the interface subsystems that come built into SQLite will work
and the application will need to provide alternative implementations
suitable for the target platform.</p>

<h2>2.0 Configuring Or Replacing The Mutex Subsystem</h2>

<p>In a multithreaded environment, SQLite uses mutexes to serialize
access to shared resources.
The mutex subsystem is only required for applications that access
SQLite from multiple threads.  For single-threaded applications, or
applications which only call SQLite from a single thread, the mutex
subsystem can be completely disabled by recompiling with the following


<p>Mutexes are cheap but they are not free, so performance will be better
when mutexes are completely disabled.  The resulting library footprint
will also be a little smaller.  Disabling the mutexes at compile-time
is a recommended optimization for applications where it makes sense.</p>

<p>When using SQLite as a shared library, an application can test to see
whether or not mutexes have been disabled using the
[sqlite3_threadsafe()] API.  Applications that link against SQLite at
run-time and use SQLite from multiple threads should probably check this
API to make sure they did not accidentally get linked against a version of
the SQLite library that has its mutexes disabled.  Single-threaded
applications will, of course, work correctly regardless of whether or
not SQLite is configured to be threadsafe, though they will be a little
bit faster when using versions of SQLite with mutexes disabled.</p>

<p>SQLite mutexes can also be disabled at run-time using the
[sqlite3_config()] interface.  To completely disable all mutexing,
the application can invoke:</p>


<p>Disabling mutexes at run-time is not as effective as disabling them
at compile-time since SQLite still must do a test of a boolean variable
to see if mutexes are enabled or disabled at each point where a mutex
might be required.  But there is still a performance advantage for
disabling mutexes at run-time.</p>

<p>For multi-threaded applications that are careful about how they
manage threads, SQLite supports an alternative run-time configuration
that is half way between not using any mutexes and the default situation
of mutexing everything in sight.  This in-the-middle mutex alignment can
be established as follows:</p>

sqlite3_config(SQLITE_CONFIG_MEMSTATUS, 0);

<p>There are two separate configuration changes here which can
be used either togethr or separately. The
[SQLITE_CONFIG_MULTITHREAD] setting disables the mutexes that
serialize access to [database connection] objects and 
[prepared statement] objects.  With this setting, the application
is free to use SQLite from multiple threads, but it must make sure
than no two threads try to access the same [database connection]
or any [prepared statements] associated with the same 
[database connection] at the same time.  Two threads can use SQLite
at the same time, but they must use separate [database connections].
The second [SQLITE_CONFIG_MEMSTATUS] setting disables the mechanism
in SQLite that tracks the total size of all outstanding memory
allocation requests.  This omits the need to mutex each call
to [sqlite3_malloc()] and [sqlite3_free()], which saves a huge
number of mutex operations.  But a consequence of disabling the
memory statistics mechanism is that the 
[sqlite3_memory_used()], [sqlite3_memory_highwater()], and
[sqlite3_soft_heap_limit64()] interfaces cease to work.

<p>SQLite uses pthreads for its mutex implementation on Unix and
SQLite requires a recursive mutex.  Most modern pthread implementations
support recursive mutexes, but not all do.  For systems that do not
support recursive mutexes, it is recommended that applications operate
in single-threaded mode only.  If this is not possible, SQLite provides
an alternative recursive mutex implementation built on top of the
standard "fast" mutexes of pthreads.  This alternative
implementation should work correctly as long as pthread_equal() is
atomic and the processor has a coherent data cache.  The alternative
recursive mutex implementation is enabled by the following
compiler command-line switch:</p>


<p>When porting SQLite to a new operating system, it is usually necessary
to completely replace the built-in mutex subsystem with an alternative
built around the mutex primitives of the new operating system.  This
is accomplished by compiling SQLite with the following option:</p>


<p>When SQLite is compiled with the SQLITE_MUTEX_APPDEF=1 option, it
completely omits the implementation of its 
[sqlite3_mutex_alloc | mutex primitive functions].  But the SQLite
library still attempts to call these functions where necessary, so the
application must itself implement the
[sqlite3_mutex_alloc | mutex primitive functions] and link them together
with SQLite.</p>

<h2>3.0 Configuring Or Replacing The Memory Allocation Subsystem</h2>

<p>By default, SQLite obtains the memory it needs for objects and
cache from the malloc()/free() implementation of the standard library.
There is also on-going work with experimental memory allocators that
satisfy all memory requests from a single fixed memory buffer handed
to SQLite at application start.  Additional information on these
experimental memory allocators will be provided in a future revision
of this document.</p>

<p>SQLite supports the ability of an application to specify an alternative
memory allocator at run-time by filling in an instance of the
[sqlite3_mem_methods] object with pointers to the routines of the
alternative implementation then registering the new alternative
implementation using the [sqlite3_config()] interface.
For example:</p>

sqlite3_config(SQLITE_CONFIG_MALLOC, &my_malloc_implementation);

<p>SQLite makes a copy of the content of the [sqlite3_mem_methods] object
so the object can be modified after the [sqlite3_config()] call returns.</p>

<h2>4.0 Adding New Virtual File Systems</h2>

<p>Since [version 3.5.0] ([dateof:3.5.0]), 
SQLite has supported an interface called the
[sqlite3_vfs | virtual file system] or "VFS".
This object is somewhat misnamed since it
is really an interface to the whole underlying operating system, not
just the filesystem.</p>

<p> One of the interesting features
of the VFS interface is that SQLite can support multiple VFSes at the
same time.  Each [database connection] has to choose a single VFS for its
use when the connection is first opened using [sqlite3_open_v2()].
But if a process contains multiple [database connections] each can choose
a different VFS.  VFSes can be added at run-time using the
[sqlite3_vfs_register()] interface.</p>

<p>The default builds for SQLite on Unix, Windows, and OS/2 include 
a VFS appropriate for the target platform.  SQLite builds for other
operating systems do not contain a VFS by default, but the application
can register one or more at run-time.</p>

<h2>5.0 Porting SQLite To A New Operating System</h2>

<p>In order to port SQLite to a new operating system - an operating
system not supported by default - the application
must provide...</p>

<li> a working mutex subsystem (but only if it is multithreaded), </li>
<li> a working memory allocation subsystem (assuming it lacks malloc()
in its standard library), and</li>
<li> a working VFS implementation.</li>

<p>All of these things can be provided in a single auxiliary C code file
and then linked with the stock "sqlite3.c" code file to generate a working
SQLite build for the target operating system.  In addition to the
alternative mutex and memory allocation subsystems and the new VFS,
the auxiliary C code file should contain implementations for the
following two routines:</p>

<li> [sqlite3_os_init()] </li>
<li> [sqlite3_os_end()] </li>

<p>The "sqlite3.c" code file contains default implementations of a VFS
and of the [sqlite3_initialize()] and [sqlite3_shutdown()] functions that
are appropriate for Unix, Windows, and OS/2.
To prevent one of these default components from being loaded when sqlite3.c
is compiled, it is necessary to add the following compile-time


<p>The SQLite core will call [sqlite3_initialize()] early.  The auxiliary
C code file can contain an implementation of sqlite3_initialize() that
registers an appropriate VFS and also perhaps initializes an alternative
mutex system (if mutexes are required) or does any memory allocation
subsystem initialization that is required.
The SQLite core never calls [sqlite3_shutdown()] but it is part of the
official SQLite API and is not otherwise provided when compiled with
-DSQLITE_OS_OTHER=1, so the auxiliary C code file should probably provide
it for completeness.</p>