Documentation Source Text

chng {2016-04-00 (3.12.0)} {
<p><b>Performance enhancements:</b>
<li>Enhancements to the [https://www.sqlite.org/src/doc/trunk/doc/lemon.html|Lemon]
    parser generator so that it creates a smaller and faster SQL parser.
<li>Only create [master journal] files if two or more attached databases are
(1) modified, (2) do not have [PRAGMA synchronous] set to OFF, and
(3) do not have the [journal_mode] set to OFF, MEMORY, or WAL.
<p><b>New Features:</b>
<li>Added the [SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER] option to [sqlite3_db_config()]
    which allows the two-argument version of the [fts3_tokenizer()] SQL function to
    be enabled or disabled at run-time.
<li>The [PRAGMA defer_foreign_keys=ON] statement now also disables 
    [foreign key actions|RESTRICT actions] on foreign key.
<p><b>Bug fixes:</b>
<li>Make sure the [sqlite3_set_auxdata()] values from multiple triggers
    within a single statement do not interfere with one another.
    Fix for ticket [https://www.sqlite.org/src/info/dc9b1c91|dc9b1c91].
}

chng {2016-02-15 (3.11.0)} {

COMPILE_OPTION {SQLITE_ENABLE_FTS3_TOKENIZER} {
  This option enables the two-argument version of the [fts3_tokenizer()]
  interface.  The second argument to fts3_tokenizer() is suppose to be a
  pointer to a function (encoded as a BLOB) that implements an
  application defined tokenizer.  If hostile actors are able to run
  the two-argument version of fts3_tokenizer() with an arbitrary second
  argument, they could use crash or take control of the process. 
  <p>
  Because of security concerns, the two-argument fts3_tokenizer() feature 
  was disabled beginning with [Version 3.11.0] unless this compile-time
  option is used.
  [Version 3.12.0] added the 
  [sqlite3_db_config](db,[SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER],1,0) interface
  that activates the two-argument version of [fts3_tokenizer()]
  for a specific [database connection] at run-time.
}

COMPILE_OPTION {SQLITE_ENABLE_FTS4} {
  When this option is defined in the [amalgamation], versions 3 and 4
  of the full-text search engine is added to the build automatically.
}

<p>
  The arguments passed to the "tokenchars=" or "separators=" options are 
  case-sensitive. In the example above, specifying that "X" is a separator
  character does not affect the way "x" is handled.

<tcl>hd_fragment f3tknzr {fts3_tokenizer}</tcl>
<h2>Custom (Application Defined) Tokenizers</h2>

<p>
  In addition to providing built-in "simple", "porter" and (possibly) "icu" and
  "unicode61" tokenizers,

  FTS provides an interface for applications to implement and register custom






  tokenizers written in C.  The interface used to create a new tokenizer is defined
  and described in the fts3_tokenizer.h source file.

<p>
  Registering a new FTS tokenizer is similar to registering a new
  virtual table module with SQLite. The user passes a pointer to a
  structure containing pointers to various callback functions that
  make up the implementation of the new tokenizer type. For tokenizers,

  it is registered as tokenizer <tokenizer-name> and a copy of it
  returned. If only one argument is passed, a pointer to the tokenizer
  implementation currently registered as <tokenizer-name> is returned,
  encoded as a blob. Or, if no such tokenizer exists, an SQL exception
  (error) is raised.

<p>
  Because of security concerns, SQLite version 3.11.0 only enabled the
  second form of the fts3_tokenizer() function when the library is compiled
  with the [SQLITE_ENABLE_FTS3_TOKENIZER | -DSQLITE_ENABLE_FTS3_TOKENIZER]
  option. In earlier versions it was
  always available.  Beginning with SQLite version 3.12.0, the second form of
  fts3_tokenizer() can also be activated at run-time by calling
  [sqlite3_db_config](db,[SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER],1,0).

<p>
  <b>SECURITY WARNING</b>: 
  If a version of the fts3/4 extension that supports the two-argument form of
  fts3_tokenizer() is deployed in an environment where malicious users can
  run arbitrary SQL, then those users should be prevented from invoking the 
  two-argument fts3_tokenizer() function.
  This can be done using the [sqlite3_set_authorizer()|authorization callback], 
  or by disabling the two-argument fts3_tokenizer() interface using a



  call to
  [sqlite3_db_config](db,[SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER],0,0).

<p>
  The following block contains an example of calling the fts3_tokenizer()
  function from C code:

<codeblock>
  <i>/*