Documentation Source Text

Check-in [01b8f2aa54]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Documentation on WITHOUT ROWID virtual tables and the LIKE OR EQUAL optimization for virtual tables.
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 01b8f2aa541c007fcd64bee55c15356fad116cce
User & Date: drh 2016-06-03 22:40:03
Context
2016-06-04
17:25
Fix an error in fts5.html. check-in: 627d913e58 user: dan tags: trunk
2016-06-03
22:40
Documentation on WITHOUT ROWID virtual tables and the LIKE OR EQUAL optimization for virtual tables. check-in: 01b8f2aa54 user: drh tags: trunk
15:19
Convert the loadext.html page to the pretty-format. check-in: 899d627a5a user: drh tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to pages/changes.in.

18
19
20
21
22
23
24




25
26
27
28
29
30
31
  global nChng aChng xrefChng
  set aChng($nChng) [list $date $desc $options]
  set xrefChng($date) $nChng
  incr nChng
}

chng {2016-07-00 (3.14.0)} {




<li>Added the csv.c loadable extensions, for reading CSV files.
<li>Enabled [persistent loadable extensions] using the new
    [SQLITE_OK_LOAD_PERMANENTLY] return code from the extension
    entry point.
<li>Add the vfsstat.c loadable extension - a VFS shim that measures I/O
    together with an [eponymous virtual table] that provides access to the measurements.
<li>Improved algorithm for running queries with both an ORDER BY and a LIMIT where







>
>
>
>







18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
  global nChng aChng xrefChng
  set aChng($nChng) [list $date $desc $options]
  set xrefChng($date) $nChng
  incr nChng
}

chng {2016-07-00 (3.14.0)} {
<li>Added support for [WITHOUT ROWID virtual tables].
<li>Improved the query planner so that the [OR optimization] can
    be used on [virtual tables] even if one or more of the disjuncts
    use the [LIKE], [GLOB], [REGEXP], [MATCH] operators.
<li>Added the csv.c loadable extensions, for reading CSV files.
<li>Enabled [persistent loadable extensions] using the new
    [SQLITE_OK_LOAD_PERMANENTLY] return code from the extension
    entry point.
<li>Add the vfsstat.c loadable extension - a VFS shim that measures I/O
    together with an [eponymous virtual table] that provides access to the measurements.
<li>Improved algorithm for running queries with both an ORDER BY and a LIMIT where

Changes to pages/vtab.in.

1
2
3
4
5


6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
..
49
50
51
52
53
54
55

56
57
58
59
60
61
62
63
..
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
...
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
...
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
...
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354






355
356
357
358
359
360
361
362
...
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
...
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
...
498
499
500
501
502
503
504


505






















506
507
508
509
510
511
512
513
514
515
...
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
...
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
...
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
...
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
...
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
...
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
...
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
...
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
...
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
...
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
...
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
....
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
....
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
....
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
....
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
....
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
....
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
....
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
....
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
....
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
<title>The Virtual Table Mechanism Of SQLite</title>

<h1 align="center">The Virtual Table Mechanism Of SQLite</h1>
<tcl>hd_keywords {virtual table} {virtual tables}</tcl>



<h2>1.0 Introduction</h2>

<p>A virtual table is an object that is registered with an open SQLite
[database connection]. From the perspective of an SQL statement,
the virtual table object looks like any other table or view. 
But behind the scenes, queries and updates on a virtual table
invoke callback methods of the virtual table object instead of
reading and writing to the database file.

<p>The virtual table mechanism allows an application to publish
interfaces that are accessible from SQL statements as if they were
tables. SQL statements can do almost anything to a
virtual table that they can do to a real table, with the following
exceptions:

................................................................................
     (the [dbstat virtual table])
<li> Read and/or write the content of a comma-separated value (CSV)
     file
<li> Access the filesystem of the host computer as if it were a database table
<li> Enabling SQL manipulation of data in statistics packages like R
</ul>


<h3>1.1 Usage</h3>

<p>A virtual table is created using a [CREATE VIRTUAL TABLE] statement.

<tcl>
RecursiveBubbleDiagram create-virtual-table-stmt
</tcl>

................................................................................
arguments.

<p>Once a virtual table has been created, it can be used like any other 
table with the exceptions noted above and imposed by specific virtual
table implementations. A virtual table is destroyed using the ordinary
[DROP TABLE] syntax.

<h4>1.1.1 Temporary virtual tables</h4>

<p>There is no "CREATE TEMP VIRTUAL TABLE" statement.  To create a
temporary virtual table, add the "temp" schema
before the virtual table name.

<blockcuqote><pre>
   CREATE VIRTUAL TABLE <b>temp.</b>tablename USING module(arg1, ...);
</pre></blockquote>

<tcl>hd_fragment epovtab {eponymous virtual tables}  \
        {eponymous virtual table}</tcl>
<h4>1.1.2 Eponymous virtual tables</h4>

<p>Some virtual tables exist automatically in the "main" schema of
every database connection in which their
module is registered, even without a [CREATE VIRTUAL TABLE] statement.
Such virtual tables are called "eponymous virtual tables".
To use an eponymous virtual table, simply use the 
module name as if it were a table.
................................................................................
The [xCreate] method is called when a virtual table is first created
using the [CREATE VIRTUAL TABLE] statement.  The [xConnect] method whenever
a database connection attaches to or reparses a schema. When these two methods
are the same, that indicates that the virtual table has no persistent
state that needs to be created and destroyed.

<tcl>hd_fragment epoonlyvtab {eponymous-only virtual table}</tcl>
<h5>1.1.2.1 Eponymous-only virtual tables</h5>
<p>If the [xCreate] method is NULL, then
[CREATE VIRTUAL TABLE] statements are prohibited for that virtual table,
and the virtual table is an "eponymous-only virtual table".
Eponymous-only virtual tables are useful as 
[table-valued functions].

<p>
Note that SQLite versions prior to 3.9.0 did not check the xCreate method
for NULL before invoking it.  So if an eponymous-only virtual table is
registered with SQLite version 3.8.11.1 or earlier and a [CREATE VIRTUAL TABLE]
command is attempted against that virtual table module, a jump to a NULL
pointer will occur, resulting in a crash.

<h3>1.2 Implementation</h3>

<p>Several new C-level objects are used by the virtual table implementation:

<blockquote><pre>
  typedef struct sqlite3_vtab sqlite3_vtab;
  typedef struct sqlite3_index_info sqlite3_index_info;
  typedef struct sqlite3_vtab_cursor sqlite3_vtab_cursor;
................................................................................
definition might be extended with additional methods and in that case 
the iVersion value will be increased.

<p>The rest of the module structure consists of methods used to implement
various features of the virtual table. Details on what each of these 
methods do are provided in the sequel.

<h3>1.3 Virtual Tables And Shared Cache</h3>

<p>Prior to SQLite [version 3.6.17], the virtual table mechanism assumes 
that each [database connection] kept
its own copy of the database schema. Hence, the virtual table mechanism
could not be used in a database that has [shared cache mode] enabled. 
The [sqlite3_create_module()] interface would return an error if 
[shared cache mode] is enabled.  That restriction was relaxed
beginning with SQLite [version 3.6.17].

<h3>1.4 Creating New Virtual Table Implementations</h3>

<p>Follow these steps to create your own virtual table:

<p>
<ol>
<li> Write all necessary methods.
<li> Create an instance of the [sqlite3_module] structure containing pointers
................................................................................
(for testing purposes). You might use one of those as a guide. Locate 
these test virtual table implementations by searching 
for "sqlite3_create_module".

<p>You might also want to implement your new virtual table as a 
[sqlite3_load_extension | loadable extension].

<h2>2.0 Virtual Table Methods</h2>

<tcl>hd_fragment xcreate {sqlite3_module.xCreate} {xCreate}</tcl>
<h3>2.1 The xCreate Method</h3>

<blockquote><pre>
  int (*xCreate)(sqlite3 *db, void *pAux,
               int argc, char **argv,
               sqlite3_vtab **ppVTab,
               char **pzErr);
</pre></blockquote>

<p>This method is called to create a new instance of a virtual table 
in response to a [CREATE VIRTUAL TABLE] statement. 






The db parameter is a pointer to the SQLite [database connection] that 
is executing the [CREATE VIRTUAL TABLE] statement. 
The pAux argument is the copy of the client data pointer that was the 
fourth argument to the [sqlite3_create_module()] or
[sqlite3_create_module_v2()] call that registered the 
[sqlite3_module | virtual table module]. 
The argv parameter is an array of argc pointers to null terminated strings. 
The first string, argv[0], is the name of the module being invoked.   The
................................................................................
<p>
If the xCreate method is the exact same pointer as the [xConnect] method,
that indicates that the virtual table does not need to initialize backing
store.  Such a virtual table can be used as an [eponymous virtual table]
or as a named virtual table using [CREATE VIRTUAL TABLE] or both.

<tcl>hd_fragment hiddencol {hidden column} {hidden columns}</tcl>
<h4>2.1.1 Hidden columns in virtual tables</h4>
<p>If a column datatype contains the special keyword "HIDDEN"
(in any combination of upper and lower case letters) then that keyword
it is omitted from the column datatype name and the column is marked 
as a hidden column internally. 
A hidden column differs from a normal column in three respects:

<p>
................................................................................
<p>An example use of hidden columns can be seen in the [FTS3] virtual 
table implementation, where every FTS virtual table
contains an [FTS hidden column] that is used to pass information from the
virtual table into [FTS auxiliary functions] and to the [FTS MATCH] operator.

<tcl>hd_fragment tabfunc2 {table-valued functions} \
         {table-valued function}</tcl>
<h4>2.1.2 Table-valued functions</h4>

<p>A [virtual table] that contains [hidden columns] can be used like
a table-valued function in the FROM clause of a [SELECT] statement.
The arguments to the table-valued function become constraints on 
the HIDDEN columns of the virtual table.

<p>For example, the "generate_series" extension (located in the
................................................................................

<p>Arguments on the virtual table name are matched to [hidden columns]
in order.  The number of arguments can be less than the
number of hidden columns, in which case the latter hidden columns are
unconstrained.  However, an error results if there are more arguments
than there are hidden columns in the virtual table.


























<tcl>############################################################# xConnect
hd_fragment xconnect {sqlite3_module.xConnect} {xConnect}</tcl>
<h3>2.2 The xConnect Method</h3>

<blockquote><pre>
  int (*xConnect)(sqlite3*, void *pAux,
               int argc, char **argv,
               sqlite3_vtab **ppVTab,
               char **pzErr);
</pre></blockquote>
................................................................................
<p>The xConnect method is required for every virtual table implementation, 
though the [xCreate] and xConnect pointers of the [sqlite3_module] object
may point to the same function if the virtual table does not need to
initialize backing store.

<tcl>############################################################ xBestIndex
hd_fragment xbestindex {sqlite3_module.xBestIndex} {xBestIndex}</tcl>
<h3>2.3 The xBestIndex Method</h3>

<p>SQLite uses the xBestIndex method of a virtual table module to determine
the best way to access the virtual table. 
The xBestIndex method has a prototype like this:

<blockquote><pre>
  int (*xBestIndex)(sqlite3_vtab *pVTab, sqlite3_index_info*);
................................................................................
<p>Note that xBestIndex will always be called before [xFilter], since
the idxNum and idxStr outputs from xBestIndex are required inputs to
xFilter.  However, there is no guarantee that xFilter will be called
following a successful xBestIndex.  

<p>The xBestIndex method is required for every virtual table implementation.

<h4>2.3.1 Inputs</h4>

<p>The main thing that the SQLite core is trying to communicate to 
the virtual table is the constraints that are available to limit 
the number of rows that need to be searched. The aConstraint[] array 
contains one entry for each constraint. There will be exactly 
nConstraint entries in that array.

................................................................................
means that the first column is used.  The second lowest bit corresponds
to the second column.  And so forth.  If the most significant bit of
colUsed is set, that means that one or more columns other than the 
first 63 columns are used.  If column usage information is needed by the
[xFilter] method, then the required bits must be encoded into either
the idxNum or idxStr output fields.

<h4>2.3.2 Outputs</h4>

<p>Given all of the information above, the job of the xBestIndex 
method it to figure out the best way to search the virtual table.

<p>The xBestIndex method fills the idxNum and idxStr fields with 
information that communicates an indexing strategy to the [xFilter] 
method. The information in idxNum and idxStr is arbitrary as far 
................................................................................

<p>By default, the SQLite core double checks all constraints on 
each row of the virtual table that it receives. If such a check 
is redundant, the xBestFilter method can suppress that double-check by 
setting aConstraintUsage[].omit.

<tcl>hd_fragment xdisconnect {sqlite3_module.xDisconnect} {xDisconnect}</tcl>
<h3>2.4 The xDisconnect Method</h3>

<blockquote><pre>
  int (*xDisconnect)(sqlite3_vtab *pVTab);
</pre></blockquote>

<p>This method releases a connection to a virtual table. 
Only the [sqlite3_vtab] object is destroyed.
................................................................................

<p>The xDisconnect method is required for every virtual table implementation,
though it is acceptable for the xDisconnect and [xDestroy] methods to be
the same function if that makes sense for the particular virtual table.

<tcl>########################################################## xDestroy
hd_fragment {sqlite3_module.xDestroy} {xDestroy}</tcl>
<h3>2.5 The xDestroy Method</h3>

<blockquote><pre>
  int (*xDestroy)(sqlite3_vtab *pVTab);
</pre></blockquote>

<p>This method releases a connection to a virtual table, just like 
the [xDisconnect] method, and it also destroys the underlying 
................................................................................

<p>The xDestroy method is required for every virtual table implementation,
though it is acceptable for the [xDisconnect] and xDestroy methods to be
the same function if that makes sense for the particular virtual table.

<tcl>########################################################## xOpen
hd_fragment xopen {sqlite3_module.xOpen}</tcl>
<h3>2.6 The xOpen Method</h3>

<blockquote><pre>
  int (*xOpen)(sqlite3_vtab *pVTab, sqlite3_vtab_cursor **ppCursor);
</pre></blockquote>

<p>The xOpen method creates a new cursor used for accessing (read and/or
writing) a virtual table.  A successful invocation of this method 
................................................................................
The SQLite core will invoke the [xFilter] method
on the cursor prior to any attempt to position or read from the cursor.

<p>The xOpen method is required for every virtual table implementation.

<tcl>############################################################### xClose
hd_fragment xclose {sqlite3_module.xClose}</tcl>
<h3>2.7 The xClose Method</h3>

<blockquote><pre>
  int (*xClose)(sqlite3_vtab_cursor*);
</pre></blockquote>

<p>The xClose method closes a cursor previously opened by 
[sqlite3_module.xOpen | xOpen]. 
................................................................................
returns an error.  The SQLite core will not use the
[sqlite3_vtab_cursor] again after it has been closed.

<p>The xClose method is required for every virtual table implementation.

<tcl>############################################################## xEof
hd_fragment xeof {sqlite3_module.xEof} {xEof}</tcl>
<h3>2.8 The xEof Method</h3>

<blockquote><pre>
  int (*xEof)(sqlite3_vtab_cursor*);
</pre></blockquote>

<p>The xEof method must return false (zero) if the specified cursor 
currently points to a valid row of data, or true (non-zero) otherwise. 
................................................................................
This method is called by the SQL engine immediately after each 
[xFilter] and [xNext] invocation.

<p>The xEof method is required for every virtual table implementation.

<tcl>############################################################## xFilter
hd_fragment xfilter {sqlite3_module.xFilter} {xFilter}</tcl>
<h3>2.9 The xFilter Method</h3>

<blockquote><pre>
  int (*xFilter)(sqlite3_vtab_cursor*, int idxNum, const char *idxStr,
                int argc, sqlite3_value **argv);
</pre></blockquote>

<p>This method begins a search of a virtual table. 
................................................................................
<p>This method must return [SQLITE_OK] if successful, or an sqlite 
[error code] if an error occurs.

<p>The xFilter method is required for every virtual table implementation.

<tcl>############################################################### xNext
hd_fragment xnext {sqlite3_module.xNext} {xNext}</tcl>
<h3>2.10 The xNext Method</h3>

<blockquote><pre>
  int (*xNext)(sqlite3_vtab_cursor*);
</pre></blockquote>

<p>The xNext method advances a [sqlite3_vtab_cursor | virtual table cursor]
to the next row of a result set initiated by [xFilter]. 
................................................................................
<p>This method must return [SQLITE_OK] if successful, or an sqlite 
[error code] if an error occurs.

<p>The xNext method is required for every virtual table implementation.

<tcl>############################################################## xColumn
hd_fragment xcolumn {sqlite3_module.xColumn} {xColumn}</tcl>
<h3>2.11 The xColumn Method</h3>

<blockquote><pre>
  int (*xColumn)(sqlite3_vtab_cursor*, sqlite3_context*, int N);
</pre></blockquote>

<p>The SQLite core invokes this method in order to find the value for 
the N-th column of the current row. N is zero-based so the first column 
................................................................................
methods to set the error message text, then return an appropriate
[error code].  The xColumn method must return [SQLITE_OK] on success.

<p>The xColumn method is required for every virtual table implementation.

<tcl>############################################################# xRowid
hd_fragment xrowid {sqlite3_module.xRowid} {xRowid}</tcl>
<h3>2.12 The xRowid Method</h3>

<blockquote><pre>
  int (*xRowid)(sqlite3_vtab_cursor *pCur, sqlite_int64 *pRowid);
</pre></blockquote>

<p>A successful invocation of this method will cause *pRowid to be
filled with the [rowid] of row that the
................................................................................
This method returns [SQLITE_OK] on success.
It returns an appropriate [error code] on failure.</p>

<p>The xRowid method is required for every virtual table implementation.

<tcl>############################################################# xUpdate
hd_fragment xupdate {sqlite3_module.xUpdate} {xUpdate}</tcl>
<h3>2.13 The xUpdate Method</h3>

<blockquote><pre>
  int (*xUpdate)(
    sqlite3_vtab *pVTab,
    int argc,
    sqlite3_value **argv,
    sqlite_int64 *pRowid
................................................................................
<p>The xUpdate method is optional.
If the xUpdate pointer in the [sqlite3_module] for a virtual table
is a NULL pointer, then the virtual table is read-only.


<tcl>########################################################## xFindFunction
hd_fragment xfindfunction {sqlite3_module.xFindFunction} {xFindFunction}</tcl>
<h3>2.14 The xFindFunction Method</h3>

<blockquote><pre>
  int (*xFindFunction)(
    sqlite3_vtab *pVtab,
    int nArg,
    const char *zName,
    void (**pxFunc)(sqlite3_context*,int,sqlite3_value**),
................................................................................
first argument.

<p>The function pointer returned by this routine must be valid for
the lifetime of the [sqlite3_vtab] object given in the first parameter.

<tcl>############################################################ xBegin
hd_fragment xBegin {sqlite3_module.xBegin} {xBegin}</tcl>
<h3>2.15 The xBegin Method</h3>

<blockquote><pre>
  int (*xBegin)(sqlite3_vtab *pVTab);
</pre></blockquote>

<p>This method begins a transaction on a virtual table.
This is method is optional.  The xBegin pointer of [sqlite3_module]
................................................................................
on a single virtual table
without an intervening call to either [xCommit] or [xRollback].
Multiple calls to other methods can and likely will occur in between
the xBegin and the corresponding [xCommit] or [xRollback].

<tcl>############################################################ xSync
hd_fragment xsync {sqlite3_module.xSync}</tcl>
<h3>2.16 The xSync Method</h3>

<blockquote><pre>
  int (*xSync)(sqlite3_vtab *pVTab);
</pre></blockquote>


<p>This method signals the start of a two-phase commit on a virtual
................................................................................
prior to an [xCommit] or [xRollback].  In order to implement two-phase
commit, the xSync method on all virtual tables is invoked prior to
invoking the [xCommit] method on any virtual table.  If any of the 
xSync methods fail, the entire transaction is rolled back.

<tcl>########################################################### xCommit
hd_fragment xcommit {sqlite3_module.xCommit} {xCommit}</tcl>
<h3>2.17 The xCommit Method</h3>

<blockquote><pre>
  int (*xCommit)(sqlite3_vtab *pVTab);
</pre></blockquote>

<p>This method causes a virtual table transaction to commit.
This is method is optional.  The xCommit pointer of [sqlite3_module]
................................................................................

<p>A call to this method always follows a prior call to [xBegin] and
[sqlite3_module.xSync|xSync].


<tcl>############################################################## xRollback
hd_fragment xrollback {sqlite3_module.xRollback} {xRollback}</tcl>
<h3>2.18 The xRollback Method</h3>

<blockquote><pre>
  int (*xRollback)(sqlite3_vtab *pVTab);
</pre></blockquote>

<p>This method causes a virtual table transaction to rollback.
This is method is optional.  The xRollback pointer of [sqlite3_module]
................................................................................
may be NULL.

<p>A call to this method always follows a prior call to [xBegin].


<tcl>############################################################# xRename
hd_fragment xrename {sqlite3_module.xRename} {xRename}</tcl>
<h3>2.19 The xRename Method</h3>

<blockquote><pre>
  int (*xRename)(sqlite3_vtab *pVtab, const char *zNew);
</pre></blockquote>

<p>This method provides notification that the virtual table implementation
that the virtual table will be given a new name. 
................................................................................
If this method returns an [error code] then the renaming is prevented.

<p>The xRename method is required for every virtual table implementation.

<tcl>############################################################# xSavepoint
hd_fragment xsavepoint {sqlite3_module.xSavepoint} {xSavepoint}\
  xRelease xRollbackTo</tcl>
<h3>2.20 The xSavepoint, xRelease, and xRollbackTo Methods</h3>

<blockquote><pre>
  int (*xSavepoint)(sqlite3_vtab *pVtab, int);
  int (*xRelease)(sqlite3_vtab *pVtab, int);
  int (*xRollbackTo)(sqlite3_vtab *pVtab, int);
</pre></blockquote>



<


>
>
|






|







 







>
|







 







|











|







 







|













|







 







|









|







 







|


|








|
|
>
>
>
>
>
>
|







 







|







 







|







 







>
>

>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>


|







 







|







 







|







 







|







 







|







 







|







 







|







 







|







 







|







 







|







 







|







 







|







 







|







 







|







 







|







 







|







 







|







 







|







 







|







 







|







 







|







1
2

3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
..
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
..
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
...
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
...
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
...
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
...
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
...
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
...
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
...
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
...
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
...
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
...
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
...
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
...
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
...
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
...
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
...
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
...
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
....
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
....
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
....
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
....
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
....
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
....
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
....
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
....
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
....
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
....
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
<title>The Virtual Table Mechanism Of SQLite</title>


<tcl>hd_keywords {virtual table} {virtual tables}</tcl>

<table_of_contents>

<h1>Introduction</h1>

<p>A virtual table is an object that is registered with an open SQLite
[database connection]. From the perspective of an SQL statement,
the virtual table object looks like any other table or view. 
But behind the scenes, queries and updates on a virtual table
invoke callback methods of the virtual table object instead of
reading and writing on the database file.

<p>The virtual table mechanism allows an application to publish
interfaces that are accessible from SQL statements as if they were
tables. SQL statements can do almost anything to a
virtual table that they can do to a real table, with the following
exceptions:

................................................................................
     (the [dbstat virtual table])
<li> Read and/or write the content of a comma-separated value (CSV)
     file
<li> Access the filesystem of the host computer as if it were a database table
<li> Enabling SQL manipulation of data in statistics packages like R
</ul>


<h2>Usage</h2>

<p>A virtual table is created using a [CREATE VIRTUAL TABLE] statement.

<tcl>
RecursiveBubbleDiagram create-virtual-table-stmt
</tcl>

................................................................................
arguments.

<p>Once a virtual table has been created, it can be used like any other 
table with the exceptions noted above and imposed by specific virtual
table implementations. A virtual table is destroyed using the ordinary
[DROP TABLE] syntax.

<h3>Temporary virtual tables</h3>

<p>There is no "CREATE TEMP VIRTUAL TABLE" statement.  To create a
temporary virtual table, add the "temp" schema
before the virtual table name.

<blockcuqote><pre>
   CREATE VIRTUAL TABLE <b>temp.</b>tablename USING module(arg1, ...);
</pre></blockquote>

<tcl>hd_fragment epovtab {eponymous virtual tables}  \
        {eponymous virtual table}</tcl>
<h3>Eponymous virtual tables</h3>

<p>Some virtual tables exist automatically in the "main" schema of
every database connection in which their
module is registered, even without a [CREATE VIRTUAL TABLE] statement.
Such virtual tables are called "eponymous virtual tables".
To use an eponymous virtual table, simply use the 
module name as if it were a table.
................................................................................
The [xCreate] method is called when a virtual table is first created
using the [CREATE VIRTUAL TABLE] statement.  The [xConnect] method whenever
a database connection attaches to or reparses a schema. When these two methods
are the same, that indicates that the virtual table has no persistent
state that needs to be created and destroyed.

<tcl>hd_fragment epoonlyvtab {eponymous-only virtual table}</tcl>
<h3>Eponymous-only virtual tables</h3>
<p>If the [xCreate] method is NULL, then
[CREATE VIRTUAL TABLE] statements are prohibited for that virtual table,
and the virtual table is an "eponymous-only virtual table".
Eponymous-only virtual tables are useful as 
[table-valued functions].

<p>
Note that SQLite versions prior to 3.9.0 did not check the xCreate method
for NULL before invoking it.  So if an eponymous-only virtual table is
registered with SQLite version 3.8.11.1 or earlier and a [CREATE VIRTUAL TABLE]
command is attempted against that virtual table module, a jump to a NULL
pointer will occur, resulting in a crash.

<h2>Implementation</h2>

<p>Several new C-level objects are used by the virtual table implementation:

<blockquote><pre>
  typedef struct sqlite3_vtab sqlite3_vtab;
  typedef struct sqlite3_index_info sqlite3_index_info;
  typedef struct sqlite3_vtab_cursor sqlite3_vtab_cursor;
................................................................................
definition might be extended with additional methods and in that case 
the iVersion value will be increased.

<p>The rest of the module structure consists of methods used to implement
various features of the virtual table. Details on what each of these 
methods do are provided in the sequel.

<h2>Virtual Tables And Shared Cache</h2>

<p>Prior to SQLite [version 3.6.17], the virtual table mechanism assumes 
that each [database connection] kept
its own copy of the database schema. Hence, the virtual table mechanism
could not be used in a database that has [shared cache mode] enabled. 
The [sqlite3_create_module()] interface would return an error if 
[shared cache mode] is enabled.  That restriction was relaxed
beginning with SQLite [version 3.6.17].

<h2>Creating New Virtual Table Implementations</h2>

<p>Follow these steps to create your own virtual table:

<p>
<ol>
<li> Write all necessary methods.
<li> Create an instance of the [sqlite3_module] structure containing pointers
................................................................................
(for testing purposes). You might use one of those as a guide. Locate 
these test virtual table implementations by searching 
for "sqlite3_create_module".

<p>You might also want to implement your new virtual table as a 
[sqlite3_load_extension | loadable extension].

<h1>Virtual Table Methods</h1>

<tcl>hd_fragment xcreate {sqlite3_module.xCreate} {xCreate}</tcl>
<h2>The xCreate Method</h2>

<blockquote><pre>
  int (*xCreate)(sqlite3 *db, void *pAux,
               int argc, char **argv,
               sqlite3_vtab **ppVTab,
               char **pzErr);
</pre></blockquote>

<p>The xCreate method is called to create a new instance of a virtual table 
in response to a [CREATE VIRTUAL TABLE] statement.
If the xCreate method is the same pointer as the [xConnect] method, then the
virtual table is an [eponymous virtual table].
If the xCreate method is omitted (if it is a NULL pointer) then the virtual 
table is an [eponymous-only virtual table].


<p>The db parameter is a pointer to the SQLite [database connection] that 
is executing the [CREATE VIRTUAL TABLE] statement. 
The pAux argument is the copy of the client data pointer that was the 
fourth argument to the [sqlite3_create_module()] or
[sqlite3_create_module_v2()] call that registered the 
[sqlite3_module | virtual table module]. 
The argv parameter is an array of argc pointers to null terminated strings. 
The first string, argv[0], is the name of the module being invoked.   The
................................................................................
<p>
If the xCreate method is the exact same pointer as the [xConnect] method,
that indicates that the virtual table does not need to initialize backing
store.  Such a virtual table can be used as an [eponymous virtual table]
or as a named virtual table using [CREATE VIRTUAL TABLE] or both.

<tcl>hd_fragment hiddencol {hidden column} {hidden columns}</tcl>
<h3>Hidden columns in virtual tables</h3>
<p>If a column datatype contains the special keyword "HIDDEN"
(in any combination of upper and lower case letters) then that keyword
it is omitted from the column datatype name and the column is marked 
as a hidden column internally. 
A hidden column differs from a normal column in three respects:

<p>
................................................................................
<p>An example use of hidden columns can be seen in the [FTS3] virtual 
table implementation, where every FTS virtual table
contains an [FTS hidden column] that is used to pass information from the
virtual table into [FTS auxiliary functions] and to the [FTS MATCH] operator.

<tcl>hd_fragment tabfunc2 {table-valued functions} \
         {table-valued function}</tcl>
<h3>Table-valued functions</h3>

<p>A [virtual table] that contains [hidden columns] can be used like
a table-valued function in the FROM clause of a [SELECT] statement.
The arguments to the table-valued function become constraints on 
the HIDDEN columns of the virtual table.

<p>For example, the "generate_series" extension (located in the
................................................................................

<p>Arguments on the virtual table name are matched to [hidden columns]
in order.  The number of arguments can be less than the
number of hidden columns, in which case the latter hidden columns are
unconstrained.  However, an error results if there are more arguments
than there are hidden columns in the virtual table.

<tcl>hd_fragment worid {WITHOUT ROWID virtual tables}</tcl>
<h3> WITHOUT ROWID Virtual Tables </h3>

<p>Beginning with SQLite [version 3.14.0], the CREATE TABLE statement that
is passed into [sqlite3_declare_vtab()] may contain a [WITHOUT ROWID] clause.
This is useful for cases where the rows of dataset being represented by the 
virtual table cannot easily be mapped into unique integers.  A CREATE TABLE
statement that includes WITHOUT ROWID must define one or more columns a
the PRIMARY KEY.  Every column of the PRIMARY KEY must individually be
NOT NULL and all columns for each row must be collectively unique.

<p>Note that SQLite does not enforce the PRIMARY KEY for a WITHOUT ROWID
virtual table.  Enforcement is the responsibility of the underlying
virtual table implementation.  But SQLite does assume that the PRIMARY KEY
constraint is valid - that the identified columns really are UNIQUE and
NOT NULL - and it uses that assumption to optimize queries against the
virtual table.

<p>The rowid column is not accessible on a
WITHOUT ROWID virtual table (of course).  Furthermore, since the
[xUpdate] method depends on having a valid rowid, the [xUpdate] method 
must be NULL for a WITHOUT ROWID virtual table.  That in turn means that
WITHOUT ROWID virtual tables must be read-only.


<tcl>############################################################# xConnect
hd_fragment xconnect {sqlite3_module.xConnect} {xConnect}</tcl>
<h2>The xConnect Method</h2>

<blockquote><pre>
  int (*xConnect)(sqlite3*, void *pAux,
               int argc, char **argv,
               sqlite3_vtab **ppVTab,
               char **pzErr);
</pre></blockquote>
................................................................................
<p>The xConnect method is required for every virtual table implementation, 
though the [xCreate] and xConnect pointers of the [sqlite3_module] object
may point to the same function if the virtual table does not need to
initialize backing store.

<tcl>############################################################ xBestIndex
hd_fragment xbestindex {sqlite3_module.xBestIndex} {xBestIndex}</tcl>
<h2>The xBestIndex Method</h2>

<p>SQLite uses the xBestIndex method of a virtual table module to determine
the best way to access the virtual table. 
The xBestIndex method has a prototype like this:

<blockquote><pre>
  int (*xBestIndex)(sqlite3_vtab *pVTab, sqlite3_index_info*);
................................................................................
<p>Note that xBestIndex will always be called before [xFilter], since
the idxNum and idxStr outputs from xBestIndex are required inputs to
xFilter.  However, there is no guarantee that xFilter will be called
following a successful xBestIndex.  

<p>The xBestIndex method is required for every virtual table implementation.

<h3>Inputs</h3>

<p>The main thing that the SQLite core is trying to communicate to 
the virtual table is the constraints that are available to limit 
the number of rows that need to be searched. The aConstraint[] array 
contains one entry for each constraint. There will be exactly 
nConstraint entries in that array.

................................................................................
means that the first column is used.  The second lowest bit corresponds
to the second column.  And so forth.  If the most significant bit of
colUsed is set, that means that one or more columns other than the 
first 63 columns are used.  If column usage information is needed by the
[xFilter] method, then the required bits must be encoded into either
the idxNum or idxStr output fields.

<h3>Outputs</h3>

<p>Given all of the information above, the job of the xBestIndex 
method it to figure out the best way to search the virtual table.

<p>The xBestIndex method fills the idxNum and idxStr fields with 
information that communicates an indexing strategy to the [xFilter] 
method. The information in idxNum and idxStr is arbitrary as far 
................................................................................

<p>By default, the SQLite core double checks all constraints on 
each row of the virtual table that it receives. If such a check 
is redundant, the xBestFilter method can suppress that double-check by 
setting aConstraintUsage[].omit.

<tcl>hd_fragment xdisconnect {sqlite3_module.xDisconnect} {xDisconnect}</tcl>
<h2>The xDisconnect Method</h2>

<blockquote><pre>
  int (*xDisconnect)(sqlite3_vtab *pVTab);
</pre></blockquote>

<p>This method releases a connection to a virtual table. 
Only the [sqlite3_vtab] object is destroyed.
................................................................................

<p>The xDisconnect method is required for every virtual table implementation,
though it is acceptable for the xDisconnect and [xDestroy] methods to be
the same function if that makes sense for the particular virtual table.

<tcl>########################################################## xDestroy
hd_fragment {sqlite3_module.xDestroy} {xDestroy}</tcl>
<h2>The xDestroy Method</h2>

<blockquote><pre>
  int (*xDestroy)(sqlite3_vtab *pVTab);
</pre></blockquote>

<p>This method releases a connection to a virtual table, just like 
the [xDisconnect] method, and it also destroys the underlying 
................................................................................

<p>The xDestroy method is required for every virtual table implementation,
though it is acceptable for the [xDisconnect] and xDestroy methods to be
the same function if that makes sense for the particular virtual table.

<tcl>########################################################## xOpen
hd_fragment xopen {sqlite3_module.xOpen}</tcl>
<h2>The xOpen Method</h2>

<blockquote><pre>
  int (*xOpen)(sqlite3_vtab *pVTab, sqlite3_vtab_cursor **ppCursor);
</pre></blockquote>

<p>The xOpen method creates a new cursor used for accessing (read and/or
writing) a virtual table.  A successful invocation of this method 
................................................................................
The SQLite core will invoke the [xFilter] method
on the cursor prior to any attempt to position or read from the cursor.

<p>The xOpen method is required for every virtual table implementation.

<tcl>############################################################### xClose
hd_fragment xclose {sqlite3_module.xClose}</tcl>
<h2>The xClose Method</h2>

<blockquote><pre>
  int (*xClose)(sqlite3_vtab_cursor*);
</pre></blockquote>

<p>The xClose method closes a cursor previously opened by 
[sqlite3_module.xOpen | xOpen]. 
................................................................................
returns an error.  The SQLite core will not use the
[sqlite3_vtab_cursor] again after it has been closed.

<p>The xClose method is required for every virtual table implementation.

<tcl>############################################################## xEof
hd_fragment xeof {sqlite3_module.xEof} {xEof}</tcl>
<h2>The xEof Method</h2>

<blockquote><pre>
  int (*xEof)(sqlite3_vtab_cursor*);
</pre></blockquote>

<p>The xEof method must return false (zero) if the specified cursor 
currently points to a valid row of data, or true (non-zero) otherwise. 
................................................................................
This method is called by the SQL engine immediately after each 
[xFilter] and [xNext] invocation.

<p>The xEof method is required for every virtual table implementation.

<tcl>############################################################## xFilter
hd_fragment xfilter {sqlite3_module.xFilter} {xFilter}</tcl>
<h2>The xFilter Method</h2>

<blockquote><pre>
  int (*xFilter)(sqlite3_vtab_cursor*, int idxNum, const char *idxStr,
                int argc, sqlite3_value **argv);
</pre></blockquote>

<p>This method begins a search of a virtual table. 
................................................................................
<p>This method must return [SQLITE_OK] if successful, or an sqlite 
[error code] if an error occurs.

<p>The xFilter method is required for every virtual table implementation.

<tcl>############################################################### xNext
hd_fragment xnext {sqlite3_module.xNext} {xNext}</tcl>
<h2>The xNext Method</h2>

<blockquote><pre>
  int (*xNext)(sqlite3_vtab_cursor*);
</pre></blockquote>

<p>The xNext method advances a [sqlite3_vtab_cursor | virtual table cursor]
to the next row of a result set initiated by [xFilter]. 
................................................................................
<p>This method must return [SQLITE_OK] if successful, or an sqlite 
[error code] if an error occurs.

<p>The xNext method is required for every virtual table implementation.

<tcl>############################################################## xColumn
hd_fragment xcolumn {sqlite3_module.xColumn} {xColumn}</tcl>
<h2>The xColumn Method</h2>

<blockquote><pre>
  int (*xColumn)(sqlite3_vtab_cursor*, sqlite3_context*, int N);
</pre></blockquote>

<p>The SQLite core invokes this method in order to find the value for 
the N-th column of the current row. N is zero-based so the first column 
................................................................................
methods to set the error message text, then return an appropriate
[error code].  The xColumn method must return [SQLITE_OK] on success.

<p>The xColumn method is required for every virtual table implementation.

<tcl>############################################################# xRowid
hd_fragment xrowid {sqlite3_module.xRowid} {xRowid}</tcl>
<h2>The xRowid Method</h2>

<blockquote><pre>
  int (*xRowid)(sqlite3_vtab_cursor *pCur, sqlite_int64 *pRowid);
</pre></blockquote>

<p>A successful invocation of this method will cause *pRowid to be
filled with the [rowid] of row that the
................................................................................
This method returns [SQLITE_OK] on success.
It returns an appropriate [error code] on failure.</p>

<p>The xRowid method is required for every virtual table implementation.

<tcl>############################################################# xUpdate
hd_fragment xupdate {sqlite3_module.xUpdate} {xUpdate}</tcl>
<h2>The xUpdate Method</h2>

<blockquote><pre>
  int (*xUpdate)(
    sqlite3_vtab *pVTab,
    int argc,
    sqlite3_value **argv,
    sqlite_int64 *pRowid
................................................................................
<p>The xUpdate method is optional.
If the xUpdate pointer in the [sqlite3_module] for a virtual table
is a NULL pointer, then the virtual table is read-only.


<tcl>########################################################## xFindFunction
hd_fragment xfindfunction {sqlite3_module.xFindFunction} {xFindFunction}</tcl>
<h2>The xFindFunction Method</h2>

<blockquote><pre>
  int (*xFindFunction)(
    sqlite3_vtab *pVtab,
    int nArg,
    const char *zName,
    void (**pxFunc)(sqlite3_context*,int,sqlite3_value**),
................................................................................
first argument.

<p>The function pointer returned by this routine must be valid for
the lifetime of the [sqlite3_vtab] object given in the first parameter.

<tcl>############################################################ xBegin
hd_fragment xBegin {sqlite3_module.xBegin} {xBegin}</tcl>
<h2>The xBegin Method</h2>

<blockquote><pre>
  int (*xBegin)(sqlite3_vtab *pVTab);
</pre></blockquote>

<p>This method begins a transaction on a virtual table.
This is method is optional.  The xBegin pointer of [sqlite3_module]
................................................................................
on a single virtual table
without an intervening call to either [xCommit] or [xRollback].
Multiple calls to other methods can and likely will occur in between
the xBegin and the corresponding [xCommit] or [xRollback].

<tcl>############################################################ xSync
hd_fragment xsync {sqlite3_module.xSync}</tcl>
<h2>The xSync Method</h2>

<blockquote><pre>
  int (*xSync)(sqlite3_vtab *pVTab);
</pre></blockquote>


<p>This method signals the start of a two-phase commit on a virtual
................................................................................
prior to an [xCommit] or [xRollback].  In order to implement two-phase
commit, the xSync method on all virtual tables is invoked prior to
invoking the [xCommit] method on any virtual table.  If any of the 
xSync methods fail, the entire transaction is rolled back.

<tcl>########################################################### xCommit
hd_fragment xcommit {sqlite3_module.xCommit} {xCommit}</tcl>
<h2>The xCommit Method</h2>

<blockquote><pre>
  int (*xCommit)(sqlite3_vtab *pVTab);
</pre></blockquote>

<p>This method causes a virtual table transaction to commit.
This is method is optional.  The xCommit pointer of [sqlite3_module]
................................................................................

<p>A call to this method always follows a prior call to [xBegin] and
[sqlite3_module.xSync|xSync].


<tcl>############################################################## xRollback
hd_fragment xrollback {sqlite3_module.xRollback} {xRollback}</tcl>
<h2>The xRollback Method</h2>

<blockquote><pre>
  int (*xRollback)(sqlite3_vtab *pVTab);
</pre></blockquote>

<p>This method causes a virtual table transaction to rollback.
This is method is optional.  The xRollback pointer of [sqlite3_module]
................................................................................
may be NULL.

<p>A call to this method always follows a prior call to [xBegin].


<tcl>############################################################# xRename
hd_fragment xrename {sqlite3_module.xRename} {xRename}</tcl>
<h2>The xRename Method</h2>

<blockquote><pre>
  int (*xRename)(sqlite3_vtab *pVtab, const char *zNew);
</pre></blockquote>

<p>This method provides notification that the virtual table implementation
that the virtual table will be given a new name. 
................................................................................
If this method returns an [error code] then the renaming is prevented.

<p>The xRename method is required for every virtual table implementation.

<tcl>############################################################# xSavepoint
hd_fragment xsavepoint {sqlite3_module.xSavepoint} {xSavepoint}\
  xRelease xRollbackTo</tcl>
<h2>The xSavepoint, xRelease, and xRollbackTo Methods</h2>

<blockquote><pre>
  int (*xSavepoint)(sqlite3_vtab *pVtab, int);
  int (*xRelease)(sqlite3_vtab *pVtab, int);
  int (*xRollbackTo)(sqlite3_vtab *pVtab, int);
</pre></blockquote>