/ Check-in [ca850d3d]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Make hi-bit characters delimiters. This is a stopgap until the tokenizer and fulltext.c recognize UTF-8 correctly. (CVS 3370)
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1:ca850d3d80f67672172d11392fcdf60bfbb94c02
User & Date: shess 2006-08-28 20:08:57
Context
2006-08-28
23:46
Make static some symbols which shouldn't have been exported. (CVS 3371) check-in: 58006e38 user: shess tags: trunk
20:08
Make hi-bit characters delimiters. This is a stopgap until the tokenizer and fulltext.c recognize UTF-8 correctly. (CVS 3370) check-in: ca850d3d user: shess tags: trunk
2006-08-27
14:10
Add sqlite3_malloc and sqlite3_realloc to the sqlite3.def file. Ticket #1943. (CVS 3369) check-in: 4a74838e user: drh tags: trunk
Changes
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Changes to ext/fts1/simple_tokenizer.c.

    57     57     ** else we need to reindex.  One solution would be a meta-table to
    58     58     ** track such information in the database, then we'd only want this
    59     59     ** information on the initial create.
    60     60     */
    61     61     if( argc>1 ){
    62     62       t->zDelim = string_dup(argv[1]);
    63     63     } else {
    64         -    /* Build a string of non-alphanumeric ASCII characters */
    65         -    char zDelim[128];               /* nul-terminated, so nul not a member */
           64  +    /* Build a string excluding alphanumeric ASCII characters */
           65  +    char zDelim[256];               /* nul-terminated, so nul not a member */
    66     66       int i, j;
    67         -    for(i=1, j=0; i<0x80; i++){
    68         -      if( !isalnum(i) ){
           67  +    for(i=1, j=0; i<0x100; i++){
           68  +      if( i>=0x80 || !isalnum(i) ){
    69     69           zDelim[j++] = i;
    70     70         }
    71     71       }
    72     72       zDelim[j++] = '\0';
    73     73       assert( j<=sizeof(zDelim) );
    74     74       t->zDelim = string_dup(zDelim);
    75     75     }