SQLite

Check-in [ca850d3d80]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Make hi-bit characters delimiters. This is a stopgap until the tokenizer and fulltext.c recognize UTF-8 correctly. (CVS 3370)
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: ca850d3d80f67672172d11392fcdf60bfbb94c02
User & Date: shess 2006-08-28 20:08:57.000
Context
2006-08-28
23:46
Make static some symbols which shouldn't have been exported. (CVS 3371) (check-in: 58006e38af user: shess tags: trunk)
20:08
Make hi-bit characters delimiters. This is a stopgap until the tokenizer and fulltext.c recognize UTF-8 correctly. (CVS 3370) (check-in: ca850d3d80 user: shess tags: trunk)
2006-08-27
14:10
Add sqlite3_malloc and sqlite3_realloc to the sqlite3.def file. Ticket #1943. (CVS 3369) (check-in: 4a74838eac user: drh tags: trunk)
Changes
Unified Diff Ignore Whitespace Patch
Changes to ext/fts1/simple_tokenizer.c.
︙︙
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

  ** else we need to reindex.  One solution would be a meta-table to
  ** track such information in the database, then we'd only want this
  ** information on the initial create.
  */
  if( argc>1 ){
    t->zDelim = string_dup(argv[1]);
  } else {
    /* Build a string of non-alphanumeric ASCII characters */
    char zDelim[128];               /* nul-terminated, so nul not a member */
    int i, j;
    for(i=1, j=0; i<0x80; i++){
      if( !isalnum(i) ){
        zDelim[j++] = i;
      }
    }
    zDelim[j++] = '\0';
    assert( j<=sizeof(zDelim) );
    t->zDelim = string_dup(zDelim);
  }








|
|

|
|








57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

  ** else we need to reindex.  One solution would be a meta-table to
  ** track such information in the database, then we'd only want this
  ** information on the initial create.
  */
  if( argc>1 ){
    t->zDelim = string_dup(argv[1]);
  } else {
    /* Build a string excluding alphanumeric ASCII characters */
    char zDelim[256];               /* nul-terminated, so nul not a member */
    int i, j;
    for(i=1, j=0; i<0x100; i++){
      if( i>=0x80 || !isalnum(i) ){
        zDelim[j++] = i;
      }
    }
    zDelim[j++] = '\0';
    assert( j<=sizeof(zDelim) );
    t->zDelim = string_dup(zDelim);
  }
︙︙