Documentation Source Text

  term-offset list to be omitted in this case.

<h1 tags="bugs">Limitations</h1>

<h2> UTF-16 byte-order-mark problem </h2>

For UTF-16 databases, when using the "simple" tokenizer, it is possible to use
malformed unicode strings to cause the integrity-check to falsely report
corruption, or for auxiliary functions to return incorrect results. More

specifically, the bug can be triggered by any of the following:

<ul>
  <li><p>A UTF-16 byte-order-mark is embedded at the beginning of an SQL string
       literal value inserted into an FTS3 table. For example:

<codeblock>
    INSERT INTO fts_table(col) VALUES('<b>{BOM}</b>text...');
</codeblock>
      <p>where {BOM} is a UTF-16 byte-order-mark, a 16-bit integer value 0xFFFE
      in either big or little endian format.

  <li><p>Malformed UTF-8 that SQLite converts to a UTF-16 byte-order-mark is
       embedded at the beginning of an SQL string literal value inserted 
       into an FTS3 table.

  <li><p>A text value created by casting a blob that begins with the two
       bytes 0xFF and 0xFE, in either possible order, is inserted into an
................................................................................
       FTS3 table. For example:
       
<codeblock>
    INSERT INTO fts_table(col) VALUES(CAST(X'FEFF' AS TEXT));
</codeblock>
</ul>

No problems occur if all unicode strings used with FTS3/4 are well-formed.
UTF-16 byte-order-marks may be safely used at the start of strings passed
to [sqlite3_bind_text16()], [sqlite3_prepare16()] and other similar APIs.
















<h1 id=appendix_a nonumber tags="search application tips">
  Appendix A: Search Application Tips
</h1>

<p>
  FTS is primarily designed to support Boolean full-text queries - queries
................................................................................
  return;

<i>  /* Jump here if the wrong number of arguments are passed to this function */</i>
wrong_number_args:
  sqlite3_result_error(pCtx, "wrong number of arguments to function rank()", -1);
}
</codeblock>

  term-offset list to be omitted in this case.

<h1 tags="bugs">Limitations</h1>

<h2> UTF-16 byte-order-mark problem </h2>

For UTF-16 databases, when using the "simple" tokenizer, it is possible to use
malformed unicode strings to cause the 
<a href="#integcheck">integrity-check special command</a> to falsely report
corruption, or for <a href="#snippet">auxiliary functions</a> to return
incorrect results. More specifically, the bug can be triggered by any of the following:

<ul>
  <li><p>A UTF-16 byte-order-mark (BOM) is embedded at the beginning of an SQL string
       literal value inserted into an FTS3 table. For example:

<codeblock>
    INSERT INTO fts_table(col) VALUES(char(0xfeff)||'text...');
</codeblock>



  <li><p>Malformed UTF-8 that SQLite converts to a UTF-16 byte-order-mark is
       embedded at the beginning of an SQL string literal value inserted 
       into an FTS3 table.

  <li><p>A text value created by casting a blob that begins with the two
       bytes 0xFF and 0xFE, in either possible order, is inserted into an
................................................................................
       FTS3 table. For example:
       
<codeblock>
    INSERT INTO fts_table(col) VALUES(CAST(X'FEFF' AS TEXT));
</codeblock>
</ul>

Everything works correctly if any of the following are true:
<ul>
<li> The [encoding|database encoding] is UTF-8.

<li> All text strings are insert using one of the
     [sqlite3_bind_text()] family of functions.
<li> Literal strings contain no byte-order-marks.
<li> A tokenizer is used that recognizes byte-order-marks
     as whitespace.  (The default "simple" tokenizer for
     FTS3/4 does not think that BOMs are whitespace, but
     the unicode tokenizer does.)
</ul>
All of the above conditions must be false in order for problems
to occur.  And even if all of the conditiona above are false,
most things will still operator correctly.  Only the
<a href="#integcheck">integrity-check</a> command and the
<a href="#snippet">auxiliary functions</a> might given
unexpected results.

<h1 id=appendix_a nonumber tags="search application tips">
  Appendix A: Search Application Tips
</h1>

<p>
  FTS is primarily designed to support Boolean full-text queries - queries
................................................................................
  return;

<i>  /* Jump here if the wrong number of arguments are passed to this function */</i>
wrong_number_args:
  sqlite3_result_error(pCtx, "wrong number of arguments to function rank()", -1);
}
</codeblock>