Documentation Source Text

<p> Such advanced searches are requested by providing a more complicated 
FTS5 query string as the text to the right of the MATCH operator. The full
query syntax is [FTS5 query syntax | described here].

<h2>Differences between FTS5 and FTS3/4</h2>

<p> Also available is the similar but more mature [fts3 | FTS3/4] module.
FTS5 is similar to FTS3/4 in that the primary task of each is to maintain
an index mapping from each unique token to a list of instances of that token 
within a set of documents, where each instance is identified by the document 
in which it appears and its position within that document. For example:

<codeblock>
  <i>-- Given the following SQL:</i>
  CREATE VIRTUAL TABLE ft USING fts5(a, b);
  INSERT INTO ft(rowid, a, b) VALUES(1, 'X Y', 'Y Z');
  INSERT INTO ft(rowid, a, b) VALUES(2, 'A Z', 'Y Y');

  <i>-- The FTS5 module creates the following mapping on disk:</i>
  A --&gt; (2, 0, 0)
  X --&gt; (1, 0, 0)
  Y --&gt; (1, 1, 0) (2, 1, 0) (2, 1, 1)
  Z --&gt; (2, 1, 1) (1, 1, 1)
</codeblock>

<p>In the example above, each triple identifies the location of a token
instance by rowid, column number (columns are numbered sequentially
starting at 0 from left to right) and position within the column value (the
first token in a column value is 0, the second is 1, and so on). Using this
index, FTS5 is able to provide timely answers to queries such as "the set
of all documents that contain the token 'A'", or "the set of all documents
that contain the sequence 'Y Z'". The list of instances associated with a
single token is called an "instance-list".

<p>The principle difference between FTS3/4 and FTS5 is that in FTS3/4,
each instance-list is stored as a single large database record, whereas
in FTS5 large instance-lists are divided betwen multiple database records.
This has the following implications for dealing with large databases that
contain large lists:

<ul>
  <li> <p>FTS5 is able to load instance-lists into memory incrementally in
       order to reduce memory usage and peak allocation size. FTS3/4 very
       often loads entire instance-lists into memory.

  <li> <p>When processing queries that feature more than one token, FTS5 is
       sometimes able to determine that the query can be answered by
       inspecting a subset of a large instance-list. FTS3/4 almost always
       has to traverse entire instance-lists.

  <li> If an instance-list grows so large that it exceeds
       the [SQLITE_MAX_LENGTH] limit, FTS3/4 is unable to handle it. FTS5
       does not have this problem. 
</ul>

<p>For these reasons, many complex queries may use less memory and run faster 
using FTS5.

<p>Some other ways in which FTS5 differs from FTS3/4 are:

<ul>
  <li> <p>FTS5 supports "ORDER BY rank" for returning results in order of
       decreasing relevancy.

  <li> <p>FTS5 features an API allowing users to create custom auxiliary 
       functions for advanced ranking and text processing applications. The

       b-trees that make up its full-text index within an INSERT, UPDATE or
       DELETE statement executed by the user. This means that any operation
       on an FTS3/4 table may turn out to be surprisingly slow, as FTS3/4 
       may unpredictably choose to merge together two or more large b-trees
       within it. FTS5 uses incremental merging by default, which limits
       the amount of processing that may take place within any given 
       INSERT, UPDATE or DELETE operation.



</ul>

<h1 tags="FTS5 query syntax">Full-text Query Syntax</h1>

<p>
The following block contains a summary of the FTS query syntax in BNF form.
A detailed explanation follows.