Documentation Source Text

<p>
  For doclists for which the term appears in more than one column of the FTS
  virtual table, term-offset lists within the doclist are stored in column 
  number order. This ensures that the term-offset list associated with 
  column 0 (if any) is always first, allowing the first two fields of the
  term-offset list to be omitted in this case.

<h1 tags="bugs">Limitations</h1>

<h2> UTF-16 byte-order-mark problem </h2>

For UTF-16 databases, when using the "simple" tokenizer, it is possible to use
malformed unicode strings to cause the integrity-check to falsely report
corruption, or for auxiliary functions to return incorrect results. More
specifically, the bug can be triggered by any of the following:

<ul>
  <li><p>A UTF-16 byte-order-mark is embedded at the beginning of an SQL string
       literal value inserted into an FTS3 table. For example:

<codeblock>
    INSERT INTO fts_table(col) VALUES('<b>{BOM}</b>text...');
</codeblock>
      <p>where {BOM} is a UTF-16 byte-order-mark, a 16-bit integer value 0xFFFE
      in either big or little endian format.

  <li><p>Malformed UTF-8 that SQLite converts to a UTF-16 byte-order-mark is
       embedded at the beginning of an SQL string literal value inserted 
       into an FTS3 table.

  <li><p>A text value created by casting a blob that begins with the two
       bytes 0xFF and 0xFE, in either possible order, is inserted into an
       FTS3 table. For example:
       
<codeblock>
    INSERT INTO fts_table(col) VALUES(CAST(X'FEFF' AS TEXT));
</codeblock>
</ul>

No problems occur if all unicode strings used with FTS3/4 are well-formed.
UTF-16 byte-order-marks may be safely used at the start of strings passed
to [sqlite3_bind_text16()], [sqlite3_prepare16()] and other similar APIs.


<h1 id=appendix_a nonumber tags="search application tips">
  Appendix A: Search Application Tips
</h1>

<p>
  FTS is primarily designed to support Boolean full-text queries - queries

  return;

<i>  /* Jump here if the wrong number of arguments are passed to this function */</i>
wrong_number_args:
  sqlite3_result_error(pCtx, "wrong number of arguments to function rank()", -1);
}
</codeblock>