Documentation Source Text

  global nChng aChng xrefChng
  set aChng($nChng) [list $date $desc $options]
  set xrefChng($date) $nChng
  incr nChng
}

chng {2016-07-00 (3.14.0)} {
<li>Added support for [WITHOUT ROWID virtual tables].
<li>Improved the query planner so that the [OR optimization] can
    be used on [virtual tables] even if one or more of the disjuncts
    use the [LIKE], [GLOB], [REGEXP], [MATCH] operators.
<li>Added the csv.c loadable extensions, for reading CSV files.
<li>Enabled [persistent loadable extensions] using the new
    [SQLITE_OK_LOAD_PERMANENTLY] return code from the extension
    entry point.
<li>Add the vfsstat.c loadable extension - a VFS shim that measures I/O
    together with an [eponymous virtual table] that provides access to the measurements.
<li>Improved algorithm for running queries with both an ORDER BY and a LIMIT where

<title>The Virtual Table Mechanism Of SQLite</title>


<tcl>hd_keywords {virtual table} {virtual tables}</tcl>

<table_of_contents>

<h1>Introduction</h1>

<p>A virtual table is an object that is registered with an open SQLite
[database connection]. From the perspective of an SQL statement,
the virtual table object looks like any other table or view. 
But behind the scenes, queries and updates on a virtual table
invoke callback methods of the virtual table object instead of
reading and writing on the database file.

<p>The virtual table mechanism allows an application to publish
interfaces that are accessible from SQL statements as if they were
tables. SQL statements can do almost anything to a
virtual table that they can do to a real table, with the following
exceptions:

     (the [dbstat virtual table])
<li> Read and/or write the content of a comma-separated value (CSV)
     file
<li> Access the filesystem of the host computer as if it were a database table
<li> Enabling SQL manipulation of data in statistics packages like R
</ul>


<h2>Usage</h2>

<p>A virtual table is created using a [CREATE VIRTUAL TABLE] statement.

<tcl>
RecursiveBubbleDiagram create-virtual-table-stmt
</tcl>

arguments.

<p>Once a virtual table has been created, it can be used like any other 
table with the exceptions noted above and imposed by specific virtual
table implementations. A virtual table is destroyed using the ordinary
[DROP TABLE] syntax.

<h3>Temporary virtual tables</h3>

<p>There is no "CREATE TEMP VIRTUAL TABLE" statement.  To create a
temporary virtual table, add the "temp" schema
before the virtual table name.

<blockcuqote><pre>
   CREATE VIRTUAL TABLE <b>temp.</b>tablename USING module(arg1, ...);
</pre></blockquote>

<tcl>hd_fragment epovtab {eponymous virtual tables}  \
        {eponymous virtual table}</tcl>
<h3>Eponymous virtual tables</h3>

<p>Some virtual tables exist automatically in the "main" schema of
every database connection in which their
module is registered, even without a [CREATE VIRTUAL TABLE] statement.
Such virtual tables are called "eponymous virtual tables".
To use an eponymous virtual table, simply use the 
module name as if it were a table.

The [xCreate] method is called when a virtual table is first created
using the [CREATE VIRTUAL TABLE] statement.  The [xConnect] method whenever
a database connection attaches to or reparses a schema. When these two methods
are the same, that indicates that the virtual table has no persistent
state that needs to be created and destroyed.

<tcl>hd_fragment epoonlyvtab {eponymous-only virtual table}</tcl>
<h3>Eponymous-only virtual tables</h3>
<p>If the [xCreate] method is NULL, then
[CREATE VIRTUAL TABLE] statements are prohibited for that virtual table,
and the virtual table is an "eponymous-only virtual table".
Eponymous-only virtual tables are useful as 
[table-valued functions].

<p>
Note that SQLite versions prior to 3.9.0 did not check the xCreate method
for NULL before invoking it.  So if an eponymous-only virtual table is
registered with SQLite version 3.8.11.1 or earlier and a [CREATE VIRTUAL TABLE]
command is attempted against that virtual table module, a jump to a NULL
pointer will occur, resulting in a crash.

<h2>Implementation</h2>

<p>Several new C-level objects are used by the virtual table implementation:

<blockquote><pre>
  typedef struct sqlite3_vtab sqlite3_vtab;
  typedef struct sqlite3_index_info sqlite3_index_info;
  typedef struct sqlite3_vtab_cursor sqlite3_vtab_cursor;

definition might be extended with additional methods and in that case 
the iVersion value will be increased.

<p>The rest of the module structure consists of methods used to implement
various features of the virtual table. Details on what each of these 
methods do are provided in the sequel.

<h2>Virtual Tables And Shared Cache</h2>

<p>Prior to SQLite [version 3.6.17], the virtual table mechanism assumes 
that each [database connection] kept
its own copy of the database schema. Hence, the virtual table mechanism
could not be used in a database that has [shared cache mode] enabled. 
The [sqlite3_create_module()] interface would return an error if 
[shared cache mode] is enabled.  That restriction was relaxed
beginning with SQLite [version 3.6.17].

<h2>Creating New Virtual Table Implementations</h2>

<p>Follow these steps to create your own virtual table:

<p>
<ol>
<li> Write all necessary methods.
<li> Create an instance of the [sqlite3_module] structure containing pointers

(for testing purposes). You might use one of those as a guide. Locate 
these test virtual table implementations by searching 
for "sqlite3_create_module".

<p>You might also want to implement your new virtual table as a 
[sqlite3_load_extension | loadable extension].

<h1>Virtual Table Methods</h1>

<tcl>hd_fragment xcreate {sqlite3_module.xCreate} {xCreate}</tcl>
<h2>The xCreate Method</h2>

<blockquote><pre>
  int (*xCreate)(sqlite3 *db, void *pAux,
               int argc, char **argv,
               sqlite3_vtab **ppVTab,
               char **pzErr);
</pre></blockquote>

<p>The xCreate method is called to create a new instance of a virtual table 
in response to a [CREATE VIRTUAL TABLE] statement.
If the xCreate method is the same pointer as the [xConnect] method, then the
virtual table is an [eponymous virtual table].
If the xCreate method is omitted (if it is a NULL pointer) then the virtual 
table is an [eponymous-only virtual table].


<p>The db parameter is a pointer to the SQLite [database connection] that 
is executing the [CREATE VIRTUAL TABLE] statement. 
The pAux argument is the copy of the client data pointer that was the 
fourth argument to the [sqlite3_create_module()] or
[sqlite3_create_module_v2()] call that registered the 
[sqlite3_module | virtual table module]. 
The argv parameter is an array of argc pointers to null terminated strings. 
The first string, argv[0], is the name of the module being invoked.   The

<p>
If the xCreate method is the exact same pointer as the [xConnect] method,
that indicates that the virtual table does not need to initialize backing
store.  Such a virtual table can be used as an [eponymous virtual table]
or as a named virtual table using [CREATE VIRTUAL TABLE] or both.

<tcl>hd_fragment hiddencol {hidden column} {hidden columns}</tcl>
<h3>Hidden columns in virtual tables</h3>
<p>If a column datatype contains the special keyword "HIDDEN"
(in any combination of upper and lower case letters) then that keyword
it is omitted from the column datatype name and the column is marked 
as a hidden column internally. 
A hidden column differs from a normal column in three respects:

<p>

<p>An example use of hidden columns can be seen in the [FTS3] virtual 
table implementation, where every FTS virtual table
contains an [FTS hidden column] that is used to pass information from the
virtual table into [FTS auxiliary functions] and to the [FTS MATCH] operator.

<tcl>hd_fragment tabfunc2 {table-valued functions} \
         {table-valued function}</tcl>
<h3>Table-valued functions</h3>

<p>A [virtual table] that contains [hidden columns] can be used like
a table-valued function in the FROM clause of a [SELECT] statement.
The arguments to the table-valued function become constraints on 
the HIDDEN columns of the virtual table.

<p>For example, the "generate_series" extension (located in the

<p>Arguments on the virtual table name are matched to [hidden columns]
in order.  The number of arguments can be less than the
number of hidden columns, in which case the latter hidden columns are
unconstrained.  However, an error results if there are more arguments
than there are hidden columns in the virtual table.

<tcl>hd_fragment worid {WITHOUT ROWID virtual tables}</tcl>
<h3> WITHOUT ROWID Virtual Tables </h3>

<p>Beginning with SQLite [version 3.14.0], the CREATE TABLE statement that
is passed into [sqlite3_declare_vtab()] may contain a [WITHOUT ROWID] clause.
This is useful for cases where the rows of dataset being represented by the 
virtual table cannot easily be mapped into unique integers.  A CREATE TABLE
statement that includes WITHOUT ROWID must define one or more columns a
the PRIMARY KEY.  Every column of the PRIMARY KEY must individually be
NOT NULL and all columns for each row must be collectively unique.

<p>Note that SQLite does not enforce the PRIMARY KEY for a WITHOUT ROWID
virtual table.  Enforcement is the responsibility of the underlying
virtual table implementation.  But SQLite does assume that the PRIMARY KEY
constraint is valid - that the identified columns really are UNIQUE and
NOT NULL - and it uses that assumption to optimize queries against the
virtual table.

<p>The rowid column is not accessible on a
WITHOUT ROWID virtual table (of course).  Furthermore, since the
[xUpdate] method depends on having a valid rowid, the [xUpdate] method 
must be NULL for a WITHOUT ROWID virtual table.  That in turn means that
WITHOUT ROWID virtual tables must be read-only.


<tcl>############################################################# xConnect
hd_fragment xconnect {sqlite3_module.xConnect} {xConnect}</tcl>
<h2>The xConnect Method</h2>

<blockquote><pre>
  int (*xConnect)(sqlite3*, void *pAux,
               int argc, char **argv,
               sqlite3_vtab **ppVTab,
               char **pzErr);
</pre></blockquote>

<p>The xConnect method is required for every virtual table implementation, 
though the [xCreate] and xConnect pointers of the [sqlite3_module] object
may point to the same function if the virtual table does not need to
initialize backing store.

<tcl>############################################################ xBestIndex
hd_fragment xbestindex {sqlite3_module.xBestIndex} {xBestIndex}</tcl>
<h2>The xBestIndex Method</h2>

<p>SQLite uses the xBestIndex method of a virtual table module to determine
the best way to access the virtual table. 
The xBestIndex method has a prototype like this:

<blockquote><pre>
  int (*xBestIndex)(sqlite3_vtab *pVTab, sqlite3_index_info*);

<p>Note that xBestIndex will always be called before [xFilter], since
the idxNum and idxStr outputs from xBestIndex are required inputs to
xFilter.  However, there is no guarantee that xFilter will be called
following a successful xBestIndex.  

<p>The xBestIndex method is required for every virtual table implementation.

<h3>Inputs</h3>

<p>The main thing that the SQLite core is trying to communicate to 
the virtual table is the constraints that are available to limit 
the number of rows that need to be searched. The aConstraint[] array 
contains one entry for each constraint. There will be exactly 
nConstraint entries in that array.

means that the first column is used.  The second lowest bit corresponds
to the second column.  And so forth.  If the most significant bit of
colUsed is set, that means that one or more columns other than the 
first 63 columns are used.  If column usage information is needed by the
[xFilter] method, then the required bits must be encoded into either
the idxNum or idxStr output fields.

<h3>Outputs</h3>

<p>Given all of the information above, the job of the xBestIndex 
method it to figure out the best way to search the virtual table.

<p>The xBestIndex method fills the idxNum and idxStr fields with 
information that communicates an indexing strategy to the [xFilter] 
method. The information in idxNum and idxStr is arbitrary as far 

<p>By default, the SQLite core double checks all constraints on 
each row of the virtual table that it receives. If such a check 
is redundant, the xBestFilter method can suppress that double-check by 
setting aConstraintUsage[].omit.

<tcl>hd_fragment xdisconnect {sqlite3_module.xDisconnect} {xDisconnect}</tcl>
<h2>The xDisconnect Method</h2>

<blockquote><pre>
  int (*xDisconnect)(sqlite3_vtab *pVTab);
</pre></blockquote>

<p>This method releases a connection to a virtual table. 
Only the [sqlite3_vtab] object is destroyed.

<p>The xDisconnect method is required for every virtual table implementation,
though it is acceptable for the xDisconnect and [xDestroy] methods to be
the same function if that makes sense for the particular virtual table.

<tcl>########################################################## xDestroy
hd_fragment {sqlite3_module.xDestroy} {xDestroy}</tcl>
<h2>The xDestroy Method</h2>

<blockquote><pre>
  int (*xDestroy)(sqlite3_vtab *pVTab);
</pre></blockquote>

<p>This method releases a connection to a virtual table, just like 
the [xDisconnect] method, and it also destroys the underlying 
table implementation. This method undoes the work of [xCreate].

<p>The [xDisconnect] method is called whenever a database connection
that uses a virtual table is closed. The xDestroy method is only 
called when a [DROP TABLE] statement is executed against the virtual table.

<p>The xDestroy method is required for every virtual table implementation,
though it is acceptable for the [xDisconnect] and xDestroy methods to be
the same function if that makes sense for the particular virtual table.

<tcl>########################################################## xOpen
hd_fragment xopen {sqlite3_module.xOpen}</tcl>
<h2>The xOpen Method</h2>

<blockquote><pre>
  int (*xOpen)(sqlite3_vtab *pVTab, sqlite3_vtab_cursor **ppCursor);
</pre></blockquote>

<p>The xOpen method creates a new cursor used for accessing (read and/or
writing) a virtual table.  A successful invocation of this method 

The SQLite core will invoke the [xFilter] method
on the cursor prior to any attempt to position or read from the cursor.

<p>The xOpen method is required for every virtual table implementation.

<tcl>############################################################### xClose
hd_fragment xclose {sqlite3_module.xClose}</tcl>
<h2>The xClose Method</h2>

<blockquote><pre>
  int (*xClose)(sqlite3_vtab_cursor*);
</pre></blockquote>

<p>The xClose method closes a cursor previously opened by 
[sqlite3_module.xOpen | xOpen]. 
The SQLite core will always call xClose once for each cursor opened 
using xOpen.

<p>This method must release all resources allocated by the
corresponding xOpen call. The routine will not be called again even if it
returns an error.  The SQLite core will not use the
[sqlite3_vtab_cursor] again after it has been closed.

<p>The xClose method is required for every virtual table implementation.

<tcl>############################################################## xEof
hd_fragment xeof {sqlite3_module.xEof} {xEof}</tcl>
<h2>The xEof Method</h2>

<blockquote><pre>
  int (*xEof)(sqlite3_vtab_cursor*);
</pre></blockquote>

<p>The xEof method must return false (zero) if the specified cursor 
currently points to a valid row of data, or true (non-zero) otherwise. 
This method is called by the SQL engine immediately after each 
[xFilter] and [xNext] invocation.

<p>The xEof method is required for every virtual table implementation.

<tcl>############################################################## xFilter
hd_fragment xfilter {sqlite3_module.xFilter} {xFilter}</tcl>
<h2>The xFilter Method</h2>

<blockquote><pre>
  int (*xFilter)(sqlite3_vtab_cursor*, int idxNum, const char *idxStr,
                int argc, sqlite3_value **argv);
</pre></blockquote>

<p>This method begins a search of a virtual table. 

<p>This method must return [SQLITE_OK] if successful, or an sqlite 
[error code] if an error occurs.

<p>The xFilter method is required for every virtual table implementation.

<tcl>############################################################### xNext
hd_fragment xnext {sqlite3_module.xNext} {xNext}</tcl>
<h2>The xNext Method</h2>

<blockquote><pre>
  int (*xNext)(sqlite3_vtab_cursor*);
</pre></blockquote>

<p>The xNext method advances a [sqlite3_vtab_cursor | virtual table cursor]
to the next row of a result set initiated by [xFilter]. 
If the cursor is already pointing at the last row when this 
routine is called, then the cursor no longer points to valid 
data and a subsequent call to the [xEof] method must return true (non-zero). 
If the cursor is successfully advanced to another row of content, then
subsequent calls to [xEof] must return false (zero).

<p>This method must return [SQLITE_OK] if successful, or an sqlite 
[error code] if an error occurs.

<p>The xNext method is required for every virtual table implementation.

<tcl>############################################################## xColumn
hd_fragment xcolumn {sqlite3_module.xColumn} {xColumn}</tcl>
<h2>The xColumn Method</h2>

<blockquote><pre>
  int (*xColumn)(sqlite3_vtab_cursor*, sqlite3_context*, int N);
</pre></blockquote>

<p>The SQLite core invokes this method in order to find the value for 
the N-th column of the current row. N is zero-based so the first column 

methods to set the error message text, then return an appropriate
[error code].  The xColumn method must return [SQLITE_OK] on success.

<p>The xColumn method is required for every virtual table implementation.

<tcl>############################################################# xRowid
hd_fragment xrowid {sqlite3_module.xRowid} {xRowid}</tcl>
<h2>The xRowid Method</h2>

<blockquote><pre>
  int (*xRowid)(sqlite3_vtab_cursor *pCur, sqlite_int64 *pRowid);
</pre></blockquote>

<p>A successful invocation of this method will cause *pRowid to be
filled with the [rowid] of row that the
[sqlite3_vtab_cursor | virtual table cursor] pCur is currently pointing at.
This method returns [SQLITE_OK] on success.
It returns an appropriate [error code] on failure.</p>

<p>The xRowid method is required for every virtual table implementation.

<tcl>############################################################# xUpdate
hd_fragment xupdate {sqlite3_module.xUpdate} {xUpdate}</tcl>
<h2>The xUpdate Method</h2>

<blockquote><pre>
  int (*xUpdate)(
    sqlite3_vtab *pVTab,
    int argc,
    sqlite3_value **argv,
    sqlite_int64 *pRowid

<p>The xUpdate method is optional.
If the xUpdate pointer in the [sqlite3_module] for a virtual table
is a NULL pointer, then the virtual table is read-only.


<tcl>########################################################## xFindFunction
hd_fragment xfindfunction {sqlite3_module.xFindFunction} {xFindFunction}</tcl>
<h2>The xFindFunction Method</h2>

<blockquote><pre>
  int (*xFindFunction)(
    sqlite3_vtab *pVtab,
    int nArg,
    const char *zName,
    void (**pxFunc)(sqlite3_context*,int,sqlite3_value**),

first argument.

<p>The function pointer returned by this routine must be valid for
the lifetime of the [sqlite3_vtab] object given in the first parameter.

<tcl>############################################################ xBegin
hd_fragment xBegin {sqlite3_module.xBegin} {xBegin}</tcl>
<h2>The xBegin Method</h2>

<blockquote><pre>
  int (*xBegin)(sqlite3_vtab *pVTab);
</pre></blockquote>

<p>This method begins a transaction on a virtual table.
This is method is optional.  The xBegin pointer of [sqlite3_module]
may be NULL.

<p>This method is always followed by one call to either the
[xCommit] or [xRollback] method.  Virtual table transactions do
not nest, so the xBegin method will not be invoked more than once
on a single virtual table
without an intervening call to either [xCommit] or [xRollback].
Multiple calls to other methods can and likely will occur in between
the xBegin and the corresponding [xCommit] or [xRollback].

<tcl>############################################################ xSync
hd_fragment xsync {sqlite3_module.xSync}</tcl>
<h2>The xSync Method</h2>

<blockquote><pre>
  int (*xSync)(sqlite3_vtab *pVTab);
</pre></blockquote>


<p>This method signals the start of a two-phase commit on a virtual
table.
This is method is optional.  The xSync pointer of [sqlite3_module]
may be NULL.

<p>This method is only invoked after call to the [xBegin] method and
prior to an [xCommit] or [xRollback].  In order to implement two-phase
commit, the xSync method on all virtual tables is invoked prior to
invoking the [xCommit] method on any virtual table.  If any of the 
xSync methods fail, the entire transaction is rolled back.

<tcl>########################################################### xCommit
hd_fragment xcommit {sqlite3_module.xCommit} {xCommit}</tcl>
<h2>The xCommit Method</h2>

<blockquote><pre>
  int (*xCommit)(sqlite3_vtab *pVTab);
</pre></blockquote>

<p>This method causes a virtual table transaction to commit.
This is method is optional.  The xCommit pointer of [sqlite3_module]
may be NULL.

<p>A call to this method always follows a prior call to [xBegin] and
[sqlite3_module.xSync|xSync].


<tcl>############################################################## xRollback
hd_fragment xrollback {sqlite3_module.xRollback} {xRollback}</tcl>
<h2>The xRollback Method</h2>

<blockquote><pre>
  int (*xRollback)(sqlite3_vtab *pVTab);
</pre></blockquote>

<p>This method causes a virtual table transaction to rollback.
This is method is optional.  The xRollback pointer of [sqlite3_module]
may be NULL.

<p>A call to this method always follows a prior call to [xBegin].


<tcl>############################################################# xRename
hd_fragment xrename {sqlite3_module.xRename} {xRename}</tcl>
<h2>The xRename Method</h2>

<blockquote><pre>
  int (*xRename)(sqlite3_vtab *pVtab, const char *zNew);
</pre></blockquote>

<p>This method provides notification that the virtual table implementation
that the virtual table will be given a new name. 
If this method returns [SQLITE_OK] then SQLite renames the table.
If this method returns an [error code] then the renaming is prevented.

<p>The xRename method is required for every virtual table implementation.

<tcl>############################################################# xSavepoint
hd_fragment xsavepoint {sqlite3_module.xSavepoint} {xSavepoint}\
  xRelease xRollbackTo</tcl>
<h2>The xSavepoint, xRelease, and xRollbackTo Methods</h2>

<blockquote><pre>
  int (*xSavepoint)(sqlite3_vtab *pVtab, int);
  int (*xRelease)(sqlite3_vtab *pVtab, int);
  int (*xRollbackTo)(sqlite3_vtab *pVtab, int);
</pre></blockquote>