Documentation Source Text

Check-in [23c364befb]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Add examples of creating fts3tokenize tables that use tokenizers declared with multiple arguments to fts3.in.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 23c364befb1473853d18ba36dc2536224f7755f4
User & Date: dan 2013-09-18 11:31:13.372
Context
2013-09-19
13:22
Updates to the TH3 description. (check-in: ecd17d4f41 user: drh tags: trunk)
2013-09-18
11:31
Add examples of creating fts3tokenize tables that use tokenizers declared with multiple arguments to fts3.in. (check-in: 23c364befb user: dan tags: trunk)
2013-09-16
13:32
Update the date and time function documentation to guarantee that 'now' is stable within a single sqlite3_step() call. (check-in: a2842a55e9 user: drh tags: trunk)
Changes
Unified Diff Ignore Whitespace Patch
Changes to pages/fts3.in.
2237
2238
2239
2240
2241
2242
2243
2244













2245
2246
2247
2248
2249
2250
2251
2252
   of the fts3tokenize virtual table:

<codeblock>
CREATE VIRTUAL TABLE tok1 USING fts3tokenize('porter');
</codeblock>

<p>The name of the desired tokenizer should be substituted in place of
   'porter' in the example, of course.  Once the virtual table is created,













   it can be queried as follows:

<codeblock>
SELECT token, start, end, position 
  FROM tok1
 WHERE input='This is a test sentence.';
</codeblock>








|
>
>
>
>
>
>
>
>
>
>
>
>
>
|







2237
2238
2239
2240
2241
2242
2243
2244
2245
2246
2247
2248
2249
2250
2251
2252
2253
2254
2255
2256
2257
2258
2259
2260
2261
2262
2263
2264
2265
   of the fts3tokenize virtual table:

<codeblock>
CREATE VIRTUAL TABLE tok1 USING fts3tokenize('porter');
</codeblock>

<p>The name of the desired tokenizer should be substituted in place of
   'porter' in the example, of course.  If the tokenizer requires one or
   more arguments, they should be separated by commas in the fts3tokenize
   declaration (even though they are separated by spaces in declarations
   of regular fts4 tables). The following creates fts4 and fts3tokenize
   tables that use the same tokenizer:
<codeblock>
  CREATE VIRTUAL TABLE text1 USING fts4(tokenize=icu en_AU);
  CREATE VIRTUAL TABLE tokens1 USING fts3tokenize(icu, en_AU);

  CREATE VIRTUAL TABLE text2 USING fts4(tokenize=unicode61 "tokenchars=@." "separators=123");
  CREATE VIRTUAL TABLE tokens2 USING fts3tokenize(unicode61, "tokenchars=@.", "separators=123");
</codeblock>
   
<p>
   Once the virtual table is created, it can be queried as follows:

<codeblock>
SELECT token, start, end, position 
  FROM tok1
 WHERE input='This is a test sentence.';
</codeblock>