/ Check-in [3e4a0082]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Fix up obsolete comments in FTS3 to conform to the latest nomenclature. Add new comments to better explain FTS3 operation.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 3e4a0082170155b5b779afd075a3ee650530ca68
User & Date: drh 2010-03-23 15:46:41
Context
2010-03-23
18:24
More commenting and documentation enhancements in FTS3. check-in: 892e2867 user: drh tags: trunk
15:46
Fix up obsolete comments in FTS3 to conform to the latest nomenclature. Add new comments to better explain FTS3 operation. check-in: 3e4a0082 user: drh tags: trunk
15:29
Close the auxiliary database db2 at the end of the crash8.test script. check-in: 0fbdc431 user: drh tags: trunk
Changes
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Changes to ext/fts3/fts3.c.

    44     44   ** 14 bits - BA
    45     45   ** 21 bits - BBA
    46     46   ** and so on.
    47     47   **
    48     48   ** This is similar in concept to how sqlite encodes "varints" but
    49     49   ** the encoding is not the same.  SQLite varints are big-endian
    50     50   ** are are limited to 9 bytes in length whereas FTS3 varints are
    51         -** little-endian and can be upt to 10 bytes in length (in theory).
           51  +** little-endian and can be up to 10 bytes in length (in theory).
    52     52   **
    53     53   ** Example encodings:
    54     54   **
    55     55   **     1:    0x01
    56     56   **   127:    0x7f
    57     57   **   128:    0x81 0x00
    58     58   **
    59     59   **
    60     60   **** Document lists ****
    61     61   ** A doclist (document list) holds a docid-sorted list of hits for a
    62         -** given term.  Doclists hold docids, and can optionally associate
    63         -** token positions and offsets with docids.  A position is the index
    64         -** of a word within the document.  The first word of the document has
    65         -** a position of 0.
           62  +** given term.  Doclists hold docids and associated token positions.
           63  +** A docid is the unique integer identifier for a single document.
           64  +** A position is the index of a word within the document.  The first 
           65  +** word of the document has a position of 0.
    66     66   **
    67     67   ** FTS3 used to optionally store character offsets using a compile-time
    68     68   ** option.  But that functionality is no longer supported.
    69     69   **
    70         -** A DL_POSITIONS_OFFSETS doclist is stored like this:
           70  +** A doclist is stored like this:
    71     71   **
    72     72   ** array {
    73     73   **   varint docid;
    74     74   **   array {                (position list for column 0)
    75         -**     varint position;     (delta from previous position plus POS_BASE)
           75  +**     varint position;     (2 more than the delta from previous position)
    76     76   **   }
    77     77   **   array {
    78     78   **     varint POS_COLUMN;   (marks start of position list for new column)
    79     79   **     varint column;       (index of new column)
    80     80   **     array {
    81         -**       varint position;   (delta from previous position plus POS_BASE)
           81  +**       varint position;   (2 more than the delta from previous position)
    82     82   **     }
    83     83   **   }
    84     84   **   varint POS_END;        (marks end of positions for this document.
    85     85   ** }
    86     86   **
    87     87   ** Here, array { X } means zero or more occurrences of X, adjacent in
    88     88   ** memory.  A "position" is an index of a token in the token stream
    89     89   ** generated by the tokenizer. Note that POS_END and POS_COLUMN occur 
    90     90   ** in the same logical place as the position element, and act as sentinals
    91     91   ** ending a position list array.  POS_END is 0.  POS_COLUMN is 1.
    92     92   ** The positions numbers are not stored literally but rather as two more
    93         -** the difference from the prior position, or the just the position plus
           93  +** than the difference from the prior position, or the just the position plus
    94     94   ** 2 for the first position.  Example:
    95     95   **
    96     96   **   label:       A B C D E  F  G H   I  J K
    97     97   **   value:     123 5 9 1 1 14 35 0 234 72 0
    98     98   **
    99     99   ** The 123 value is the first docid.  For column zero in this document
   100    100   ** there are two matches at positions 3 and 10 (5-2 and 9-2+3).  The 1
   101    101   ** at D signals the start of a new column; the 1 at E indicates that the
   102    102   ** new column is column number 1.  There are two positions at 12 and 45
   103    103   ** (14-2 and 35-2+12).  The 0 at H indicate the end-of-document.  The
   104    104   ** 234 at I is the next docid.  It has one position 72 (72-2) and then
   105    105   ** terminates with the 0 at K.
   106    106   **
   107         -** A DL_POSITIONS doclist omits the startOffset and endOffset
   108         -** information.  A DL_DOCIDS doclist omits both the position and
   109         -** offset information, becoming an array of varint-encoded docids.
          107  +** A "position-list" is the list of positions for multiple columns for
          108  +** a single docid.  A "column-list" is the set of positions for a single
          109  +** column.  Hence, a position-list consists of one or more column-lists,
          110  +** a document record consists of a docid followed by a position-list and
          111  +** a doclist consists of one or more document records.
   110    112   **
   111         -** On-disk data is stored as type DL_DEFAULT, so we don't serialize
   112         -** the type.  Due to how deletion is implemented in the segmentation
   113         -** system, on-disk doclists MUST store at least positions.
   114         -**
          113  +** A bare doclist omits the position information, becoming an 
          114  +** array of varint-encoded docids.
   115    115   **
   116    116   **** Segment leaf nodes ****
   117    117   ** Segment leaf nodes store terms and doclists, ordered by term.  Leaf
   118    118   ** nodes are written using LeafWriter, and read using LeafReader (to
   119    119   ** iterate through a single leaf node's data) and LeavesReader (to
   120    120   ** iterate through a segment's entire leaf layer).  Leaf nodes have
   121    121   ** the format:
................................................................................
   309    309   
   310    310   #include "fts3.h"
   311    311   #ifndef SQLITE_CORE 
   312    312   # include "sqlite3ext.h"
   313    313     SQLITE_EXTENSION_INIT1
   314    314   #endif
   315    315   
          316  +/*
          317  +** The testcase() macro is only used by the amalgamation.  If undefined,
          318  +** make it a no-op.
          319  +*/
          320  +#ifndef testcase
          321  +# define testcase(X)
          322  +#endif
          323  +
          324  +/*
          325  +** Terminator values for position-lists and column-lists.
          326  +*/
          327  +#define POS_COLUMN  (1)     /* Column-list terminator */
          328  +#define POS_END     (0)     /* Position-list terminator */ 
          329  +
   316    330   /* 
   317    331   ** Write a 64-bit variable-length integer to memory starting at p[0].
   318    332   ** The length of data written will be between 1 and FTS3_VARINT_MAX bytes.
   319    333   ** The number of bytes written is returned.
   320    334   */
   321    335   int sqlite3Fts3PutVarint(char *p, sqlite_int64 v){
   322    336     unsigned char *q = (unsigned char *) p;
................................................................................
   355    369    sqlite_int64 i;
   356    370    int ret = sqlite3Fts3GetVarint(p, &i);
   357    371    *pi = (int) i;
   358    372    return ret;
   359    373   }
   360    374   
   361    375   /*
   362         -** Return the number of bytes required to store the value passed as the
   363         -** first argument in varint form.
          376  +** Return the number of bytes required to encode v as a varint
   364    377   */
   365    378   int sqlite3Fts3VarintLen(sqlite3_uint64 v){
   366    379     int i = 0;
   367    380     do{
   368    381       i++;
   369    382       v >>= 7;
   370    383     }while( v!=0 );
................................................................................
   407    420       }
   408    421       z[iOut] = '\0';
   409    422     }
   410    423   }
   411    424   
   412    425   /*
   413    426   ** Read a single varint from the doclist at *pp and advance *pp to point
   414         -** to the next element of the varlist.  Add the value of the varint
          427  +** to the first byte past the end of the varint.  Add the value of the varint
   415    428   ** to *pVal.
   416    429   */
   417    430   static void fts3GetDeltaVarint(char **pp, sqlite3_int64 *pVal){
   418    431     sqlite3_int64 iVal;
   419    432     *pp += sqlite3Fts3GetVarint(*pp, &iVal);
   420    433     *pVal += iVal;
   421    434   }
................................................................................
   543    556     return rc;
   544    557   }
   545    558   
   546    559   /*
   547    560   ** Create the backing store tables (%_content, %_segments and %_segdir)
   548    561   ** required by the FTS3 table passed as the only argument. This is done
   549    562   ** as part of the vtab xCreate() method.
          563  +**
          564  +** If the p->bHasDocsize boolean is true (indicating that this is an
          565  +** FTS4 table, not an FTS3 table) then also create the %_docsize and
          566  +** %_stat tables required by FTS4.
   550    567   */
   551    568   static int fts3CreateTables(Fts3Table *p){
   552    569     int rc = SQLITE_OK;             /* Return code */
   553    570     int i;                          /* Iterator variable */
   554    571     char *zContentCols;             /* Columns of %_content table */
   555    572     sqlite3 *db = p->db;            /* The database connection */
   556    573   
................................................................................
   635    652   
   636    653   /*
   637    654   ** This function is the implementation of both the xConnect and xCreate
   638    655   ** methods of the FTS3 virtual table.
   639    656   **
   640    657   ** The argv[] array contains the following:
   641    658   **
   642         -**   argv[0]   -> module name
          659  +**   argv[0]   -> module name  ("fts3" or "fts4")
   643    660   **   argv[1]   -> database name
   644    661   **   argv[2]   -> table name
   645    662   **   argv[...] -> "column name" and other module argument fields.
   646    663   */
   647    664   static int fts3InitVtab(
   648    665     int isCreate,                   /* True for xCreate, false for xConnect */
   649    666     sqlite3 *db,                    /* The SQLite database connection */
................................................................................
   654    671     char **pzErr                    /* Write any error message here */
   655    672   ){
   656    673     Fts3Hash *pHash = (Fts3Hash *)pAux;
   657    674     Fts3Table *p;                   /* Pointer to allocated vtab */
   658    675     int rc;                         /* Return code */
   659    676     int i;                          /* Iterator variable */
   660    677     int nByte;                      /* Size of allocation used for *p */
   661         -  int iCol;
   662         -  int nString = 0;
   663         -  int nCol = 0;
   664         -  char *zCsr;
   665         -  int nDb;
   666         -  int nName;
          678  +  int iCol;                       /* Column index */
          679  +  int nString = 0;                /* Bytes required to hold all column names */
          680  +  int nCol = 0;                   /* Number of columns in the FTS table */
          681  +  char *zCsr;                     /* Space for holding column names */
          682  +  int nDb;                        /* Bytes required to hold database name */
          683  +  int nName;                      /* Bytes required to hold table name */
   667    684   
   668    685     const char *zTokenizer = 0;               /* Name of tokenizer to use */
   669    686     sqlite3_tokenizer *pTokenizer = 0;        /* Tokenizer for this table */
   670    687   
   671    688     nDb = (int)strlen(argv[1]) + 1;
   672    689     nName = (int)strlen(argv[2]) + 1;
   673    690     for(i=3; i<argc; i++){
................................................................................
   889    906     sqlite3Fts3ExprFree(pCsr->pExpr);
   890    907     sqlite3_free(pCsr->aDoclist);
   891    908     sqlite3_free(pCsr->aMatchinfo);
   892    909     sqlite3_free(pCsr);
   893    910     return SQLITE_OK;
   894    911   }
   895    912   
          913  +/*
          914  +** Position the pCsr->pStmt statement so that it is on the row
          915  +** of the %_content table that contains the last match.  Return
          916  +** SQLITE_OK on success.  
          917  +*/
   896    918   static int fts3CursorSeek(sqlite3_context *pContext, Fts3Cursor *pCsr){
   897    919     if( pCsr->isRequireSeek ){
   898    920       pCsr->isRequireSeek = 0;
   899    921       sqlite3_bind_int64(pCsr->pStmt, 1, pCsr->iPrevId);
   900    922       if( SQLITE_ROW==sqlite3_step(pCsr->pStmt) ){
   901    923         return SQLITE_OK;
   902    924       }else{
................................................................................
   915    937         return rc;
   916    938       }
   917    939     }else{
   918    940       return SQLITE_OK;
   919    941     }
   920    942   }
   921    943   
          944  +/*
          945  +** Advance the cursor to the next row in the %_content table that
          946  +** matches the search criteria.  For a MATCH search, this will be
          947  +** the next row that matches.  For a full-table scan, this will be
          948  +** simply the next row in the %_content table.  For a docid lookup,
          949  +** this routine simply sets the EOF flag.
          950  +**
          951  +** Return SQLITE_OK if nothing goes wrong.  SQLITE_OK is returned
          952  +** even if we reach end-of-file.  The fts3EofMethod() will be called
          953  +** subsequently to determine whether or not an EOF was hit.
          954  +*/
   922    955   static int fts3NextMethod(sqlite3_vtab_cursor *pCursor){
   923    956     int rc = SQLITE_OK;             /* Return code */
   924    957     Fts3Cursor *pCsr = (Fts3Cursor *)pCursor;
   925    958   
   926    959     if( pCsr->aDoclist==0 ){
   927    960       if( SQLITE_ROW!=sqlite3_step(pCsr->pStmt) ){
   928    961         pCsr->isEof = 1;
................................................................................
  1050   1083     *piPrev = iVal;
  1051   1084   }
  1052   1085   
  1053   1086   /*
  1054   1087   ** When this function is called, *ppPoslist is assumed to point to the 
  1055   1088   ** start of a position-list. After it returns, *ppPoslist points to the
  1056   1089   ** first byte after the position-list.
         1090  +**
         1091  +** A position list is list of positions (delta encoded) and columns for 
         1092  +** a single document record of a doclist.  So, in other words, this
         1093  +** routine advances *ppPoslist so that it points to the next docid in
         1094  +** the doclist, or to the first byte past the end of the doclist.
  1057   1095   **
  1058   1096   ** If pp is not NULL, then the contents of the position list are copied
  1059   1097   ** to *pp. *pp is set to point to the first byte past the last byte copied
  1060   1098   ** before this function returns.
  1061   1099   */
  1062   1100   static void fts3PoslistCopy(char **pp, char **ppPoslist){
  1063   1101     char *pEnd = *ppPoslist;
  1064   1102     char c = 0;
  1065   1103   
  1066   1104     /* The end of a position list is marked by a zero encoded as an FTS3 
  1067         -  ** varint. A single 0x00 byte. Except, if the 0x00 byte is preceded by
         1105  +  ** varint. A single POS_END (0) byte. Except, if the 0 byte is preceded by
  1068   1106     ** a byte with the 0x80 bit set, then it is not a varint 0, but the tail
  1069   1107     ** of some other, multi-byte, value.
  1070   1108     **
  1071         -  ** The following block moves pEnd to point to the first byte that is not 
         1109  +  ** The following while-loop moves pEnd to point to the first byte that is not 
  1072   1110     ** immediately preceded by a byte with the 0x80 bit set. Then increments
  1073   1111     ** pEnd once more so that it points to the byte immediately following the
  1074   1112     ** last byte in the position-list.
  1075   1113     */
  1076         -  while( *pEnd | c ) c = *pEnd++ & 0x80;
  1077         -  pEnd++;
         1114  +  while( *pEnd | c ){
         1115  +    c = *pEnd++ & 0x80;
         1116  +    testcase( c!=0 && (*pEnd)==0 );
         1117  +  }
         1118  +  pEnd++;  /* Advance past the POS_END terminator byte */
  1078   1119   
  1079   1120     if( pp ){
  1080   1121       int n = (int)(pEnd - *ppPoslist);
  1081   1122       char *p = *pp;
  1082   1123       memcpy(p, *ppPoslist, n);
  1083   1124       p += n;
  1084   1125       *pp = p;
  1085   1126     }
  1086   1127     *ppPoslist = pEnd;
  1087   1128   }
  1088   1129   
         1130  +/*
         1131  +** When this function is called, *ppPoslist is assumed to point to the 
         1132  +** start of a column-list. After it returns, *ppPoslist points to the
         1133  +** to the terminator (POS_COLUMN or POS_END) byte of the column-list.
         1134  +**
         1135  +** A column-list is list of delta-encoded positions for a single column
         1136  +** within a single document within a doclist.
         1137  +**
         1138  +** The column-list is terminated either by a POS_COLUMN varint (1) or
         1139  +** a POS_END varint (0).  This routine leaves *ppPoslist pointing to
         1140  +** the POS_COLUMN or POS_END that terminates the column-list.
         1141  +**
         1142  +** If pp is not NULL, then the contents of the column-list are copied
         1143  +** to *pp. *pp is set to point to the first byte past the last byte copied
         1144  +** before this function returns.  The POS_COLUMN or POS_END terminator
         1145  +** is not copied into *pp.
         1146  +*/
  1089   1147   static void fts3ColumnlistCopy(char **pp, char **ppPoslist){
  1090   1148     char *pEnd = *ppPoslist;
  1091   1149     char c = 0;
  1092   1150   
  1093         -  /* A column-list is terminated by either a 0x01 or 0x00. */
  1094         -  while( 0xFE & (*pEnd | c) ) c = *pEnd++ & 0x80;
         1151  +  /* A column-list is terminated by either a 0x01 or 0x00 byte that is
         1152  +  ** not part of a multi-byte varint.
         1153  +  */
         1154  +  while( 0xFE & (*pEnd | c) ){
         1155  +    c = *pEnd++ & 0x80;
         1156  +    testcase( c!=0 && ((*pEnd)&0xfe)==0 );
         1157  +  }
  1095   1158     if( pp ){
  1096   1159       int n = (int)(pEnd - *ppPoslist);
  1097   1160       char *p = *pp;
  1098   1161       memcpy(p, *ppPoslist, n);
  1099   1162       p += n;
  1100   1163       *pp = p;
  1101   1164     }
  1102   1165     *ppPoslist = pEnd;
  1103   1166   }
  1104   1167   
  1105   1168   /*
  1106         -** Value used to signify the end of an offset-list. This is safe because
         1169  +** Value used to signify the end of an position-list. This is safe because
  1107   1170   ** it is not possible to have a document with 2^31 terms.
  1108   1171   */
  1109         -#define OFFSET_LIST_END 0x7fffffff
         1172  +#define POSITION_LIST_END 0x7fffffff
  1110   1173   
  1111   1174   /*
  1112         -** This function is used to help parse offset-lists. When this function is
  1113         -** called, *pp may point to the start of the next varint in the offset-list
  1114         -** being parsed, or it may point to 1 byte past the end of the offset-list
  1115         -** (in which case **pp will be 0x00 or 0x01).
         1175  +** This function is used to help parse position-lists. When this function is
         1176  +** called, *pp may point to the start of the next varint in the position-list
         1177  +** being parsed, or it may point to 1 byte past the end of the position-list
         1178  +** (in which case **pp will be a terminator bytes POS_END (0) or
         1179  +** (1)).
  1116   1180   **
  1117         -** If *pp points past the end of the current offset list, set *pi to 
  1118         -** OFFSET_LIST_END and return. Otherwise, read the next varint from *pp,
         1181  +** If *pp points past the end of the current position-list, set *pi to 
         1182  +** POSITION_LIST_END and return. Otherwise, read the next varint from *pp,
  1119   1183   ** increment the current value of *pi by the value read, and set *pp to
  1120   1184   ** point to the next value before returning.
         1185  +**
         1186  +** Before calling this routine *pi must be initialized to the value of
         1187  +** the previous position, or zero if we are reading the first position
         1188  +** in the position-list.  Because positions are delta-encoded, the value
         1189  +** of the previous position is needed in order to compute the value of
         1190  +** the next position.
  1121   1191   */
  1122   1192   static void fts3ReadNextPos(
  1123         -  char **pp,                      /* IN/OUT: Pointer into offset-list buffer */
  1124         -  sqlite3_int64 *pi               /* IN/OUT: Value read from offset-list */
         1193  +  char **pp,                    /* IN/OUT: Pointer into position-list buffer */
         1194  +  sqlite3_int64 *pi             /* IN/OUT: Value read from position-list */
  1125   1195   ){
  1126         -  if( **pp&0xFE ){
         1196  +  if( (**pp)&0xFE ){
  1127   1197       fts3GetDeltaVarint(pp, pi);
  1128   1198       *pi -= 2;
  1129   1199     }else{
  1130         -    *pi = OFFSET_LIST_END;
         1200  +    *pi = POSITION_LIST_END;
  1131   1201     }
  1132   1202   }
  1133   1203   
  1134   1204   /*
  1135         -** If parameter iCol is not 0, write an 0x01 byte followed by the value of
  1136         -** iCol encoded as a varint to *pp. 
         1205  +** If parameter iCol is not 0, write an POS_COLUMN (1) byte followed by
         1206  +** the value of iCol encoded as a varint to *pp.   This will start a new
         1207  +** column list.
  1137   1208   **
  1138   1209   ** Set *pp to point to the byte just after the last byte written before 
  1139   1210   ** returning (do not modify it if iCol==0). Return the total number of bytes
  1140   1211   ** written (0 if iCol==0).
  1141   1212   */
  1142   1213   static int fts3PutColNumber(char **pp, int iCol){
  1143   1214     int n = 0;                      /* Number of bytes written */
................................................................................
  1147   1218       *p = 0x01;
  1148   1219       *pp = &p[n];
  1149   1220     }
  1150   1221     return n;
  1151   1222   }
  1152   1223   
  1153   1224   /*
  1154         -**
         1225  +** Compute the union of two position lists.  The output written
         1226  +** into *pp contains all positions of both *pp1 and *pp2 in sorted
         1227  +** order and with any duplicates removed.  All pointers are
         1228  +** updated appropriately.   The caller is responsible for insuring
         1229  +** that there is enough space in *pp to hold the complete output.
  1155   1230   */
  1156   1231   static void fts3PoslistMerge(
  1157   1232     char **pp,                      /* Output buffer */
  1158   1233     char **pp1,                     /* Left input list */
  1159   1234     char **pp2                      /* Right input list */
  1160   1235   ){
  1161   1236     char *p = *pp;
  1162   1237     char *p1 = *pp1;
  1163   1238     char *p2 = *pp2;
  1164   1239   
  1165   1240     while( *p1 || *p2 ){
  1166         -    int iCol1;
  1167         -    int iCol2;
         1241  +    int iCol1;         /* The current column index in pp1 */
         1242  +    int iCol2;         /* The current column index in pp2 */
  1168   1243   
  1169         -    if( *p1==0x01 ) sqlite3Fts3GetVarint32(&p1[1], &iCol1);
  1170         -    else if( *p1==0x00 ) iCol1 = OFFSET_LIST_END;
         1244  +    if( *p1==POS_COLUMN ) sqlite3Fts3GetVarint32(&p1[1], &iCol1);
         1245  +    else if( *p1==POS_END ) iCol1 = POSITION_LIST_END;
  1171   1246       else iCol1 = 0;
  1172   1247   
  1173         -    if( *p2==0x01 ) sqlite3Fts3GetVarint32(&p2[1], &iCol2);
  1174         -    else if( *p2==0x00 ) iCol2 = OFFSET_LIST_END;
         1248  +    if( *p2==POS_COLUMN ) sqlite3Fts3GetVarint32(&p2[1], &iCol2);
         1249  +    else if( *p2==POS_END ) iCol2 = POSITION_LIST_END;
  1175   1250       else iCol2 = 0;
  1176   1251   
  1177   1252       if( iCol1==iCol2 ){
  1178         -      sqlite3_int64 i1 = 0;
  1179         -      sqlite3_int64 i2 = 0;
         1253  +      sqlite3_int64 i1 = 0;       /* Last position from pp1 */
         1254  +      sqlite3_int64 i2 = 0;       /* Last position from pp2 */
  1180   1255         sqlite3_int64 iPrev = 0;
  1181   1256         int n = fts3PutColNumber(&p, iCol1);
  1182   1257         p1 += n;
  1183   1258         p2 += n;
  1184   1259   
  1185         -      /* At this point, both p1 and p2 point to the start of offset-lists.
  1186         -      ** An offset-list is a list of non-negative delta-encoded varints, each 
  1187         -      ** incremented by 2 before being stored. Each list is terminated by a 0 
  1188         -      ** or 1 value (0x00 or 0x01). The following block merges the two lists
         1260  +      /* At this point, both p1 and p2 point to the start of column-lists
         1261  +      ** for the same column (the column with index iCol1 and iCol2).
         1262  +      ** A column-list is a list of non-negative delta-encoded varints, each 
         1263  +      ** incremented by 2 before being stored. Each list is terminated by a
         1264  +      ** POS_END (0) or POS_COLUMN (1). The following block merges the two lists
  1189   1265         ** and writes the results to buffer p. p is left pointing to the byte
  1190         -      ** after the list written. No terminator (0x00 or 0x01) is written to
  1191         -      ** the output.
         1266  +      ** after the list written. No terminator (POS_END or POS_COLUMN) is
         1267  +      ** written to the output.
  1192   1268         */
  1193   1269         fts3GetDeltaVarint(&p1, &i1);
  1194   1270         fts3GetDeltaVarint(&p2, &i2);
  1195   1271         do {
  1196   1272           fts3PutDeltaVarint(&p, &iPrev, (i1<i2) ? i1 : i2); 
  1197   1273           iPrev -= 2;
  1198   1274           if( i1==i2 ){
................................................................................
  1199   1275             fts3ReadNextPos(&p1, &i1);
  1200   1276             fts3ReadNextPos(&p2, &i2);
  1201   1277           }else if( i1<i2 ){
  1202   1278             fts3ReadNextPos(&p1, &i1);
  1203   1279           }else{
  1204   1280             fts3ReadNextPos(&p2, &i2);
  1205   1281           }
  1206         -      }while( i1!=OFFSET_LIST_END || i2!=OFFSET_LIST_END );
         1282  +      }while( i1!=POSITION_LIST_END || i2!=POSITION_LIST_END );
  1207   1283       }else if( iCol1<iCol2 ){
  1208   1284         p1 += fts3PutColNumber(&p, iCol1);
  1209   1285         fts3ColumnlistCopy(&p, &p1);
  1210   1286       }else{
  1211   1287         p2 += fts3PutColNumber(&p, iCol2);
  1212   1288         fts3ColumnlistCopy(&p, &p2);
  1213   1289       }
  1214   1290     }
  1215   1291   
  1216         -  *p++ = '\0';
         1292  +  *p++ = POS_END;
  1217   1293     *pp = p;
  1218   1294     *pp1 = p1 + 1;
  1219   1295     *pp2 = p2 + 1;
  1220   1296   }
  1221   1297   
  1222   1298   /*
  1223   1299   ** nToken==1 searches for adjacent positions.