Documentation Source Text

  case-sensitive. In the example above, specifying that "X" is a separator
  character does not affect the way "x" is handled.

<h2>Custom (User Implemented) Tokenizers</h2>

<p>
  As well as the built-in "simple", "porter" and (possibly) "icu" and
  "unicode61" tokenizers,







  FTS exports an interface that allows users to implement custom tokenizers
  using C. The interface used to create a new tokenizer is defined and 
  described in the fts3_tokenizer.h source file.

<p>
  Registering a new FTS tokenizer is similar to registering a new
  virtual table module with SQLite. The user passes a pointer to a
  structure containing pointers to various callback functions that
  make up the implementation of the new tokenizer type. For tokenizers,
  the structure (defined in fts3_tokenizer.h) is called
................................................................................
  it is registered as tokenizer &lt;tokenizer-name&gt; and a copy of it
  returned. If only one argument is passed, a pointer to the tokenizer
  implementation currently registered as &lt;tokenizer-name&gt; is returned,
  encoded as a blob. Or, if no such tokenizer exists, an SQL exception
  (error) is raised.

<p>






  <b>SECURITY WARNING</b>: If the fts3/4 extension is used in an environment


  where potentially malicious users may execute arbitrary SQL, they should 
  be prevented from invoking the fts3_tokenizer() function, possibly using 
  the [sqlite3_set_authorizer()|authorization callback].

<p>
  The following block contains an example of calling the fts3_tokenizer()
  function from C code:

<codeblock>
  <i>/*

  case-sensitive. In the example above, specifying that "X" is a separator
  character does not affect the way "x" is handled.

<h2>Custom (User Implemented) Tokenizers</h2>

<p>
  As well as the built-in "simple", "porter" and (possibly) "icu" and
  "unicode61" tokenizers, if the library is compiled with the following 
  compiler option:

<codeblock>
  -DSQLITE_ENABLE_FTS3_TOKENIZER
</codeblock>

<p>
  then FTS exports an interface that allows users to implement custom
  tokenizers using C. The interface used to create a new tokenizer is defined
  and described in the fts3_tokenizer.h source file.

<p>
  Registering a new FTS tokenizer is similar to registering a new
  virtual table module with SQLite. The user passes a pointer to a
  structure containing pointers to various callback functions that
  make up the implementation of the new tokenizer type. For tokenizers,
  the structure (defined in fts3_tokenizer.h) is called
................................................................................
  it is registered as tokenizer &lt;tokenizer-name&gt; and a copy of it
  returned. If only one argument is passed, a pointer to the tokenizer
  implementation currently registered as &lt;tokenizer-name&gt; is returned,
  encoded as a blob. Or, if no such tokenizer exists, an SQL exception
  (error) is raised.

<p>
  As of SQLite version 3.11.0, the second form of the fts3_tokenizer() function
  is only available if the library is compiled with the
  -DSQLITE_ENABLE_FTS3_TOKENIZER compiler switch. In earlier versions it was
  always available.

<p>
  <b>SECURITY WARNING</b>: 
  If a version of the fts3/4 extension that supports the second form of
  fts3_tokenizer() is deployed in an environment where potentially malicious
  users may execute arbitrary SQL, they should be prevented from invoking the
  fts3_tokenizer() function, possibly using the 
  [sqlite3_set_authorizer()|authorization callback].

<p>
  The following block contains an example of calling the fts3_tokenizer()
  function from C code:

<codeblock>
  <i>/*