/ Check-in [96499b1d]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Update the Apple OS-X branch to include all of the latest changes in trunk.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | apple-osx
Files: files | file ages | folders
SHA1:96499b1dd69a0954e30e4b5b4f2272ab306afcdb
User & Date: drh 2010-01-20 01:26:07
Context
2010-01-20
13:20
Align the os_unix.c source file with the version found on trunk. check-in: fa0f6c14 user: drh tags: apple-osx
01:26
Update the Apple OS-X branch to include all of the latest changes in trunk. check-in: 96499b1d user: drh tags: apple-osx
01:20
Manually copy over the rebustness fixes from the apple-osx branch. check-in: 095c74ea user: drh tags: trunk
2010-01-19
23:50
robustness fixes for preventing a finalized statement from being reused check-in: a7a0c8d6 user: adam tags: apple-osx
Changes
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Changes to ext/fts3/fts3.c.

    19     19   **     * The FTS3 module is being built as an extension
    20     20   **       (in which case SQLITE_CORE is not defined), or
    21     21   **
    22     22   **     * The FTS3 module is being built into the core of
    23     23   **       SQLite (in which case SQLITE_ENABLE_FTS3 is defined).
    24     24   */
    25     25   
    26         -/* TODO(shess) Consider exporting this comment to an HTML file or the
    27         -** wiki.
    28         -*/
    29     26   /* The full-text index is stored in a series of b+tree (-like)
    30     27   ** structures called segments which map terms to doclists.  The
    31     28   ** structures are like b+trees in layout, but are constructed from the
    32     29   ** bottom up in optimal fashion and are not updatable.  Since trees
    33     30   ** are built from the bottom up, things will be described from the
    34     31   ** bottom up.
    35     32   **
................................................................................
    44     41   **         B = 1xxxxxxx    7 bits of data and one flag bit
    45     42   **
    46     43   **  7 bits - A
    47     44   ** 14 bits - BA
    48     45   ** 21 bits - BBA
    49     46   ** and so on.
    50     47   **
    51         -** This is identical to how sqlite encodes varints (see util.c).
           48  +** This is similar in concept to how sqlite encodes "varints" but
           49  +** the encoding is not the same.  SQLite varints are big-endian
           50  +** are are limited to 9 bytes in length whereas FTS3 varints are
           51  +** little-endian and can be upt to 10 bytes in length (in theory).
           52  +**
           53  +** Example encodings:
           54  +**
           55  +**     1:    0x01
           56  +**   127:    0x7f
           57  +**   128:    0x81 0x00
    52     58   **
    53     59   **
    54     60   **** Document lists ****
    55     61   ** A doclist (document list) holds a docid-sorted list of hits for a
    56     62   ** given term.  Doclists hold docids, and can optionally associate
    57         -** token positions and offsets with docids.
           63  +** token positions and offsets with docids.  A position is the index
           64  +** of a word within the document.  The first word of the document has
           65  +** a position of 0.
           66  +**
           67  +** FTS3 used to optionally store character offsets using a compile-time
           68  +** option.  But that functionality is no longer supported.
    58     69   **
    59     70   ** A DL_POSITIONS_OFFSETS doclist is stored like this:
    60     71   **
    61     72   ** array {
    62     73   **   varint docid;
    63     74   **   array {                (position list for column 0)
    64     75   **     varint position;     (delta from previous position plus POS_BASE)
    65         -**     varint startOffset;  (delta from previous startOffset)
    66         -**     varint endOffset;    (delta from startOffset)
    67     76   **   }
    68     77   **   array {
    69     78   **     varint POS_COLUMN;   (marks start of position list for new column)
    70     79   **     varint column;       (index of new column)
    71     80   **     array {
    72     81   **       varint position;   (delta from previous position plus POS_BASE)
    73         -**       varint startOffset;(delta from previous startOffset)
    74         -**       varint endOffset;  (delta from startOffset)
    75     82   **     }
    76     83   **   }
    77     84   **   varint POS_END;        (marks end of positions for this document.
    78     85   ** }
    79     86   **
    80     87   ** Here, array { X } means zero or more occurrences of X, adjacent in
    81     88   ** memory.  A "position" is an index of a token in the token stream
    82         -** generated by the tokenizer, while an "offset" is a byte offset,
    83         -** both based at 0.  Note that POS_END and POS_COLUMN occur in the
    84         -** same logical place as the position element, and act as sentinals
    85         -** ending a position list array.
           89  +** generated by the tokenizer. Note that POS_END and POS_COLUMN occur 
           90  +** in the same logical place as the position element, and act as sentinals
           91  +** ending a position list array.  POS_END is 0.  POS_COLUMN is 1.
           92  +** The positions numbers are not stored literally but rather as two more
           93  +** the difference from the prior position, or the just the position plus
           94  +** 2 for the first position.  Example:
           95  +**
           96  +**   label:       A B C D E  F  G H   I  J K
           97  +**   value:     123 5 9 1 1 14 35 0 234 72 0
           98  +**
           99  +** The 123 value is the first docid.  For column zero in this document
          100  +** there are two matches at positions 3 and 10 (5-2 and 9-2+3).  The 1
          101  +** at D signals the start of a new column; the 1 at E indicates that the
          102  +** new column is column number 1.  There are two positions at 12 and 45
          103  +** (14-2 and 35-2+12).  The 0 at H indicate the end-of-document.  The
          104  +** 234 at I is the next docid.  It has one position 72 (72-2) and then
          105  +** terminates with the 0 at K.
    86    106   **
    87    107   ** A DL_POSITIONS doclist omits the startOffset and endOffset
    88    108   ** information.  A DL_DOCIDS doclist omits both the position and
    89    109   ** offset information, becoming an array of varint-encoded docids.
    90    110   **
    91    111   ** On-disk data is stored as type DL_DEFAULT, so we don't serialize
    92    112   ** the type.  Due to how deletion is implemented in the segmentation
................................................................................
   384    404           z[iOut++] = z[iIn++];
   385    405         }
   386    406       }
   387    407       z[iOut] = '\0';
   388    408     }
   389    409   }
   390    410   
          411  +/*
          412  +** Read a single varint from the doclist at *pp and advance *pp to point
          413  +** to the next element of the varlist.  Add the value of the varint
          414  +** to *pVal.
          415  +*/
   391    416   static void fts3GetDeltaVarint(char **pp, sqlite3_int64 *pVal){
   392    417     sqlite3_int64 iVal;
   393    418     *pp += sqlite3Fts3GetVarint(*pp, &iVal);
   394    419     *pVal += iVal;
   395    420   }
   396    421   
          422  +/*
          423  +** As long as *pp has not reached its end (pEnd), then do the same
          424  +** as fts3GetDeltaVarint(): read a single varint and add it to *pVal.
          425  +** But if we have reached the end of the varint, just set *pp=0 and
          426  +** leave *pVal unchanged.
          427  +*/
   397    428   static void fts3GetDeltaVarint2(char **pp, char *pEnd, sqlite3_int64 *pVal){
   398    429     if( *pp>=pEnd ){
   399    430       *pp = 0;
   400    431     }else{
   401    432       fts3GetDeltaVarint(pp, pVal);
   402    433     }
   403    434   }
................................................................................
   777    808     *ppCsr = pCsr = (sqlite3_vtab_cursor *)sqlite3_malloc(sizeof(Fts3Cursor));
   778    809     if( !pCsr ){
   779    810       return SQLITE_NOMEM;
   780    811     }
   781    812     memset(pCsr, 0, sizeof(Fts3Cursor));
   782    813     return SQLITE_OK;
   783    814   }
   784         -
   785         -/****************************************************************/
   786         -/****************************************************************/
   787         -/****************************************************************/
   788         -/****************************************************************/
   789         -
   790    815   
   791    816   /*
   792    817   ** Close the cursor.  For additional information see the documentation
   793    818   ** on the xClose method of the virtual table interface.
   794    819   */
   795    820   static int fulltextClose(sqlite3_vtab_cursor *pCursor){
   796    821     Fts3Cursor *pCsr = (Fts3Cursor *)pCursor;
................................................................................
  1704   1729       *paOut = pOut;
  1705   1730       *pnOut = nOut;
  1706   1731     }else{
  1707   1732       sqlite3_free(pOut);
  1708   1733     }
  1709   1734     return rc;
  1710   1735   }
         1736  +
         1737  +static int fts3NearMerge(
         1738  +  int mergetype,                  /* MERGE_POS_NEAR or MERGE_NEAR */
         1739  +  int nNear,                      /* Parameter to NEAR operator */
         1740  +  int nTokenLeft,                 /* Number of tokens in LHS phrase arg */
         1741  +  char *aLeft,                    /* Doclist for LHS (incl. positions) */
         1742  +  int nLeft,                      /* Size of LHS doclist in bytes */
         1743  +  int nTokenRight,                /* As nTokenLeft */
         1744  +  char *aRight,                   /* As aLeft */
         1745  +  int nRight,                     /* As nRight */
         1746  +  char **paOut,                   /* OUT: Results of merge (malloced) */
         1747  +  int *pnOut                      /* OUT: Sized of output buffer */
         1748  +){
         1749  +  char *aOut;
         1750  +  int rc;
         1751  +
         1752  +  assert( mergetype==MERGE_POS_NEAR || MERGE_NEAR );
         1753  +
         1754  +  aOut = sqlite3_malloc(nLeft+nRight+1);
         1755  +  if( aOut==0 ){
         1756  +    rc = SQLITE_NOMEM;
         1757  +  }else{
         1758  +    rc = fts3DoclistMerge(mergetype, nNear+nTokenRight, nNear+nTokenLeft, 
         1759  +      aOut, pnOut, aLeft, nLeft, aRight, nRight
         1760  +    );
         1761  +    if( rc!=SQLITE_OK ){
         1762  +      sqlite3_free(aOut);
         1763  +      aOut = 0;
         1764  +    }
         1765  +  }
         1766  +
         1767  +  *paOut = aOut;
         1768  +  return rc;
         1769  +}
         1770  +
         1771  +int sqlite3Fts3ExprNearTrim(Fts3Expr *pLeft, Fts3Expr *pRight, int nNear){
         1772  +  int rc;
         1773  +  if( pLeft->aDoclist==0 || pRight->aDoclist==0 ){
         1774  +    sqlite3_free(pLeft->aDoclist);
         1775  +    sqlite3_free(pRight->aDoclist);
         1776  +    pRight->aDoclist = 0;
         1777  +    pLeft->aDoclist = 0;
         1778  +    rc = SQLITE_OK;
         1779  +  }else{
         1780  +    char *aOut;
         1781  +    int nOut;
         1782  +
         1783  +    rc = fts3NearMerge(MERGE_POS_NEAR, nNear, 
         1784  +        pLeft->pPhrase->nToken, pLeft->aDoclist, pLeft->nDoclist,
         1785  +        pRight->pPhrase->nToken, pRight->aDoclist, pRight->nDoclist,
         1786  +        &aOut, &nOut
         1787  +    );
         1788  +    if( rc!=SQLITE_OK ) return rc;
         1789  +    sqlite3_free(pRight->aDoclist);
         1790  +    pRight->aDoclist = aOut;
         1791  +    pRight->nDoclist = nOut;
         1792  +
         1793  +    rc = fts3NearMerge(MERGE_POS_NEAR, nNear, 
         1794  +        pRight->pPhrase->nToken, pRight->aDoclist, pRight->nDoclist,
         1795  +        pLeft->pPhrase->nToken, pLeft->aDoclist, pLeft->nDoclist,
         1796  +        &aOut, &nOut
         1797  +    );
         1798  +    sqlite3_free(pLeft->aDoclist);
         1799  +    pLeft->aDoclist = aOut;
         1800  +    pLeft->nDoclist = nOut;
         1801  +  }
         1802  +  return rc;
         1803  +}
  1711   1804   
  1712   1805   /*
  1713   1806   ** Evaluate the full-text expression pExpr against fts3 table pTab. Store
  1714   1807   ** the resulting doclist in *paOut and *pnOut.
  1715   1808   */
  1716   1809   static int evalFts3Expr(
  1717   1810     Fts3Table *p,                   /* Virtual table handle */
................................................................................
  1749   1842               || pExpr->eType==FTSQUERY_AND  || pExpr->eType==FTSQUERY_NOT
  1750   1843           );
  1751   1844           switch( pExpr->eType ){
  1752   1845             case FTSQUERY_NEAR: {
  1753   1846               Fts3Expr *pLeft;
  1754   1847               Fts3Expr *pRight;
  1755   1848               int mergetype = isReqPos ? MERGE_POS_NEAR : MERGE_NEAR;
  1756         -            int nParam1;
  1757         -            int nParam2;
  1758         -            char *aBuffer;
  1759   1849              
  1760   1850               if( pExpr->pParent && pExpr->pParent->eType==FTSQUERY_NEAR ){
  1761   1851                 mergetype = MERGE_POS_NEAR;
  1762   1852               }
  1763   1853               pLeft = pExpr->pLeft;
  1764   1854               while( pLeft->eType==FTSQUERY_NEAR ){ 
  1765   1855                 pLeft=pLeft->pRight;
  1766   1856               }
  1767   1857               pRight = pExpr->pRight;
  1768   1858               assert( pRight->eType==FTSQUERY_PHRASE );
  1769   1859               assert( pLeft->eType==FTSQUERY_PHRASE );
  1770   1860   
  1771         -            nParam1 = pExpr->nNear+1;
  1772         -            nParam2 = nParam1+pLeft->pPhrase->nToken+pRight->pPhrase->nToken-2;
  1773         -            aBuffer = sqlite3_malloc(nLeft+nRight+1);
  1774         -            rc = fts3DoclistMerge(mergetype, nParam1, nParam2, aBuffer,
  1775         -                pnOut, aLeft, nLeft, aRight, nRight
         1861  +            rc = fts3NearMerge(mergetype, pExpr->nNear, 
         1862  +                pLeft->pPhrase->nToken, aLeft, nLeft,
         1863  +                pRight->pPhrase->nToken, aRight, nRight,
         1864  +                paOut, pnOut
  1776   1865               );
  1777         -            if( rc!=SQLITE_OK ){
  1778         -              sqlite3_free(aBuffer);
  1779         -            }else{
  1780         -              *paOut = aBuffer;
  1781         -            }
  1782   1866               sqlite3_free(aLeft);
  1783   1867               break;
  1784   1868             }
  1785   1869   
  1786   1870             case FTSQUERY_OR: {
  1787   1871               /* Allocate a buffer for the output. The maximum size is the
  1788   1872               ** sum of the sizes of the two input buffers. The +1 term is
................................................................................
  2025   2109   */
  2026   2110   int sqlite3Fts3ExprLoadDoclist(Fts3Table *pTab, Fts3Expr *pExpr){
  2027   2111     return evalFts3Expr(pTab, pExpr, &pExpr->aDoclist, &pExpr->nDoclist, 1);
  2028   2112   }
  2029   2113   
  2030   2114   /*
  2031   2115   ** After ExprLoadDoclist() (see above) has been called, this function is
  2032         -** used to iterate through the position lists that make up the doclist
         2116  +** used to iterate/search through the position lists that make up the doclist
  2033   2117   ** stored in pExpr->aDoclist.
  2034   2118   */
  2035   2119   char *sqlite3Fts3FindPositions(
  2036   2120     Fts3Expr *pExpr,                /* Access this expressions doclist */
  2037   2121     sqlite3_int64 iDocid,           /* Docid associated with requested pos-list */
  2038   2122     int iCol                        /* Column of requested pos-list */
  2039   2123   ){
................................................................................
  2060   2144             }
  2061   2145             while( iThis<iCol ){
  2062   2146               fts3ColumnlistCopy(0, &pCsr);
  2063   2147               if( *pCsr==0x00 ) return 0;
  2064   2148               pCsr++;
  2065   2149               pCsr += sqlite3Fts3GetVarint32(pCsr, &iThis);
  2066   2150             }
  2067         -          if( iCol==iThis ) return pCsr;
         2151  +          if( iCol==iThis && (*pCsr&0xFE) ) return pCsr;
  2068   2152           }
  2069   2153           return 0;
  2070   2154         }
  2071   2155       }
  2072   2156     }
  2073   2157   
  2074   2158     return 0;
................................................................................
  2112   2196     int nVal,                       /* Size of apVal[] array */
  2113   2197     sqlite3_value **apVal           /* Array of arguments */
  2114   2198   ){
  2115   2199     Fts3Cursor *pCsr;               /* Cursor handle passed through apVal[0] */
  2116   2200     const char *zStart = "<b>";
  2117   2201     const char *zEnd = "</b>";
  2118   2202     const char *zEllipsis = "<b>...</b>";
  2119         -
  2120         -  /* There must be at least one argument passed to this function (otherwise
  2121         -  ** the non-overloaded version would have been called instead of this one).
  2122         -  */
  2123         -  assert( nVal>=1 );
  2124         -
  2125         -  if( nVal>4 ){
  2126         -    sqlite3_result_error(pContext, 
  2127         -        "wrong number of arguments to function snippet()", -1);
  2128         -    return;
  2129         -  }
  2130         -  if( fts3FunctionArg(pContext, "snippet", apVal[0], &pCsr) ) return;
  2131         -
  2132         -  switch( nVal ){
  2133         -    case 4: zEllipsis = (const char*)sqlite3_value_text(apVal[3]);
  2134         -    case 3: zEnd = (const char*)sqlite3_value_text(apVal[2]);
  2135         -    case 2: zStart = (const char*)sqlite3_value_text(apVal[1]);
  2136         -  }
  2137         -  if( !zEllipsis || !zEnd || !zStart ){
  2138         -    sqlite3_result_error_nomem(pContext);
  2139         -  }else if( SQLITE_OK==fts3CursorSeek(pContext, pCsr) ){
  2140         -    sqlite3Fts3Snippet(pContext, pCsr, zStart, zEnd, zEllipsis);
  2141         -  }
  2142         -}
  2143         -
  2144         -/*
  2145         -** Implementation of the snippet2() function for FTS3
  2146         -*/
  2147         -static void fts3Snippet2Func(
  2148         -  sqlite3_context *pContext,      /* SQLite function call context */
  2149         -  int nVal,                       /* Size of apVal[] array */
  2150         -  sqlite3_value **apVal           /* Array of arguments */
  2151         -){
  2152         -  Fts3Cursor *pCsr;               /* Cursor handle passed through apVal[0] */
  2153         -  const char *zStart = "<b>";
  2154         -  const char *zEnd = "</b>";
  2155         -  const char *zEllipsis = "<b>...</b>";
  2156   2203     int iCol = -1;
  2157         -  int nToken = 10;
         2204  +  int nToken = 15;                /* Default number of tokens in snippet */
  2158   2205   
  2159   2206     /* There must be at least one argument passed to this function (otherwise
  2160   2207     ** the non-overloaded version would have been called instead of this one).
  2161   2208     */
  2162   2209     assert( nVal>=1 );
  2163   2210   
  2164   2211     if( nVal>6 ){
................................................................................
  2174   2221       case 4: zEllipsis = (const char*)sqlite3_value_text(apVal[3]);
  2175   2222       case 3: zEnd = (const char*)sqlite3_value_text(apVal[2]);
  2176   2223       case 2: zStart = (const char*)sqlite3_value_text(apVal[1]);
  2177   2224     }
  2178   2225     if( !zEllipsis || !zEnd || !zStart ){
  2179   2226       sqlite3_result_error_nomem(pContext);
  2180   2227     }else if( SQLITE_OK==fts3CursorSeek(pContext, pCsr) ){
  2181         -    sqlite3Fts3Snippet2(pContext, pCsr, zStart, zEnd, zEllipsis, iCol, nToken);
         2228  +    sqlite3Fts3Snippet(pContext, pCsr, zStart, zEnd, zEllipsis, iCol, nToken);
  2182   2229     }
  2183   2230   }
  2184   2231   
  2185   2232   /*
  2186   2233   ** Implementation of the offsets() function for FTS3
  2187   2234   */
  2188   2235   static void fts3OffsetsFunc(
................................................................................
  2275   2322     void **ppArg                    /* Unused */
  2276   2323   ){
  2277   2324     struct Overloaded {
  2278   2325       const char *zName;
  2279   2326       void (*xFunc)(sqlite3_context*,int,sqlite3_value**);
  2280   2327     } aOverload[] = {
  2281   2328       { "snippet", fts3SnippetFunc },
  2282         -    { "snippet2", fts3Snippet2Func },
  2283   2329       { "offsets", fts3OffsetsFunc },
  2284   2330       { "optimize", fts3OptimizeFunc },
  2285   2331       { "matchinfo", fts3MatchinfoFunc },
  2286   2332     };
  2287   2333     int i;                          /* Iterator variable */
  2288   2334   
  2289   2335     UNUSED_PARAMETER(pVtab);
................................................................................
  2425   2471     /* Create the virtual table wrapper around the hash-table and overload 
  2426   2472     ** the two scalar functions. If this is successful, register the
  2427   2473     ** module with sqlite.
  2428   2474     */
  2429   2475     if( SQLITE_OK==rc 
  2430   2476      && SQLITE_OK==(rc = sqlite3Fts3InitHashTable(db, pHash, "fts3_tokenizer"))
  2431   2477      && SQLITE_OK==(rc = sqlite3_overload_function(db, "snippet", -1))
  2432         -   && SQLITE_OK==(rc = sqlite3_overload_function(db, "snippet2", -1))
  2433   2478      && SQLITE_OK==(rc = sqlite3_overload_function(db, "offsets", 1))
  2434   2479      && SQLITE_OK==(rc = sqlite3_overload_function(db, "matchinfo", -1))
  2435   2480      && SQLITE_OK==(rc = sqlite3_overload_function(db, "optimize", 1))
  2436   2481     ){
  2437   2482       return sqlite3_create_module_v2(
  2438   2483           db, "fts3", &fts3Module, (void *)pHash, hashDestroy
  2439   2484       );

Changes to ext/fts3/fts3Int.h.

   275    275   int sqlite3Fts3GetVarint(const char *, sqlite_int64 *);
   276    276   int sqlite3Fts3GetVarint32(const char *, int *);
   277    277   int sqlite3Fts3VarintLen(sqlite3_uint64);
   278    278   void sqlite3Fts3Dequote(char *);
   279    279   
   280    280   char *sqlite3Fts3FindPositions(Fts3Expr *, sqlite3_int64, int);
   281    281   int sqlite3Fts3ExprLoadDoclist(Fts3Table *, Fts3Expr *);
          282  +int sqlite3Fts3ExprNearTrim(Fts3Expr *, Fts3Expr *, int);
   282    283   
   283    284   /* fts3_tokenizer.c */
   284    285   const char *sqlite3Fts3NextToken(const char *, int *);
   285    286   int sqlite3Fts3InitHashTable(sqlite3 *, Fts3Hash *, const char *);
   286    287   int sqlite3Fts3InitTokenizer(Fts3Hash *pHash, 
   287    288     const char *, sqlite3_tokenizer **, const char **, char **
   288    289   );
   289    290   
   290    291   /* fts3_snippet.c */
   291    292   void sqlite3Fts3Offsets(sqlite3_context*, Fts3Cursor*);
   292         -void sqlite3Fts3Snippet(sqlite3_context*, Fts3Cursor*, 
   293         -  const char *, const char *, const char *
   294         -);
   295         -void sqlite3Fts3Snippet2(sqlite3_context *, Fts3Cursor *, const char *,
          293  +void sqlite3Fts3Snippet(sqlite3_context *, Fts3Cursor *, const char *,
   296    294     const char *, const char *, int, int
   297    295   );
   298    296   void sqlite3Fts3Matchinfo(sqlite3_context *, Fts3Cursor *);
   299    297   
   300    298   /* fts3_expr.c */
   301    299   int sqlite3Fts3ExprParse(sqlite3_tokenizer *, 
   302    300     char **, int, int, const char *, int, Fts3Expr **

Changes to ext/fts3/fts3_snippet.c.

    14     14   #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3)
    15     15   
    16     16   #include "fts3Int.h"
    17     17   #include <string.h>
    18     18   #include <assert.h>
    19     19   #include <ctype.h>
    20     20   
    21         -typedef struct Snippet Snippet;
    22         -
    23         -/*
    24         -** An instance of the following structure keeps track of generated
    25         -** matching-word offset information and snippets.
    26         -*/
    27         -struct Snippet {
    28         -  int nMatch;                     /* Total number of matches */
    29         -  int nAlloc;                     /* Space allocated for aMatch[] */
    30         -  struct snippetMatch {  /* One entry for each matching term */
    31         -    char snStatus;       /* Status flag for use while constructing snippets */
    32         -    short int nByte;     /* Number of bytes in the term */
    33         -    short int iCol;      /* The column that contains the match */
    34         -    short int iTerm;     /* The index in Query.pTerms[] of the matching term */
    35         -    int iToken;          /* The index of the matching document token */
    36         -    int iStart;          /* The offset to the first character of the term */
    37         -  } *aMatch;                      /* Points to space obtained from malloc */
    38         -  char *zOffset;                  /* Text rendering of aMatch[] */
    39         -  int nOffset;                    /* strlen(zOffset) */
    40         -  char *zSnippet;                 /* Snippet text */
    41         -  int nSnippet;                   /* strlen(zSnippet) */
           21  +
           22  +/*
           23  +** Used as an fts3ExprIterate() context when loading phrase doclists to
           24  +** Fts3Expr.aDoclist[]/nDoclist.
           25  +*/
           26  +typedef struct LoadDoclistCtx LoadDoclistCtx;
           27  +struct LoadDoclistCtx {
           28  +  Fts3Table *pTab;                /* FTS3 Table */
           29  +  int nPhrase;                    /* Number of phrases seen so far */
           30  +  int nToken;                     /* Number of tokens seen so far */
           31  +};
           32  +
           33  +/*
           34  +** The following types are used as part of the implementation of the 
           35  +** fts3BestSnippet() routine.
           36  +*/
           37  +typedef struct SnippetIter SnippetIter;
           38  +typedef struct SnippetPhrase SnippetPhrase;
           39  +typedef struct SnippetFragment SnippetFragment;
           40  +
           41  +struct SnippetIter {
           42  +  Fts3Cursor *pCsr;               /* Cursor snippet is being generated from */
           43  +  int iCol;                       /* Extract snippet from this column */
           44  +  int nSnippet;                   /* Requested snippet length (in tokens) */
           45  +  int nPhrase;                    /* Number of phrases in query */
           46  +  SnippetPhrase *aPhrase;         /* Array of size nPhrase */
           47  +  int iCurrent;                   /* First token of current snippet */
           48  +};
           49  +
           50  +struct SnippetPhrase {
           51  +  int nToken;                     /* Number of tokens in phrase */
           52  +  char *pList;                    /* Pointer to start of phrase position list */
           53  +  int iHead;                      /* Next value in position list */
           54  +  char *pHead;                    /* Position list data following iHead */
           55  +  int iTail;                      /* Next value in trailing position list */
           56  +  char *pTail;                    /* Position list data following iTail */
           57  +};
           58  +
           59  +struct SnippetFragment {
           60  +  int iCol;                       /* Column snippet is extracted from */
           61  +  int iPos;                       /* Index of first token in snippet */
           62  +  u64 covered;                    /* Mask of query phrases covered */
           63  +  u64 hlmask;                     /* Mask of snippet terms to highlight */
           64  +};
           65  +
           66  +/*
           67  +** This type is used as an fts3ExprIterate() context object while 
           68  +** accumulating the data returned by the matchinfo() function.
           69  +*/
           70  +typedef struct MatchInfo MatchInfo;
           71  +struct MatchInfo {
           72  +  Fts3Cursor *pCursor;            /* FTS3 Cursor */
           73  +  int nCol;                       /* Number of columns in table */
           74  +  u32 *aMatchinfo;                /* Pre-allocated buffer */
           75  +};
           76  +
           77  +
           78  +
           79  +/*
           80  +** The snippet() and offsets() functions both return text values. An instance
           81  +** of the following structure is used to accumulate those values while the
           82  +** functions are running. See fts3StringAppend() for details.
           83  +*/
           84  +typedef struct StrBuffer StrBuffer;
           85  +struct StrBuffer {
           86  +  char *z;                        /* Pointer to buffer containing string */
           87  +  int n;                          /* Length of z in bytes (excl. nul-term) */
           88  +  int nAlloc;                     /* Allocated size of buffer z in bytes */
    42     89   };
    43     90   
    44     91   
    45         -/* It is not safe to call isspace(), tolower(), or isalnum() on
    46         -** hi-bit-set characters.  This is the same solution used in the
    47         -** tokenizer.
    48         -*/
    49         -static int fts3snippetIsspace(char c){
    50         -  return (c&0x80)==0 ? isspace(c) : 0;
    51         -}
    52         -
    53         -
    54         -/*
    55         -** A StringBuffer object holds a zero-terminated string that grows
    56         -** arbitrarily by appending.  Space to hold the string is obtained
    57         -** from sqlite3_malloc().  After any memory allocation failure, 
    58         -** StringBuffer.z is set to NULL and no further allocation is attempted.
    59         -*/
    60         -typedef struct StringBuffer {
    61         -  char *z;         /* Text of the string.  Space from malloc. */
    62         -  int nUsed;       /* Number bytes of z[] used, not counting \000 terminator */
    63         -  int nAlloc;      /* Bytes allocated for z[] */
    64         -} StringBuffer;
    65         -
    66         -
    67         -/*
    68         -** Initialize a new StringBuffer.
    69         -*/
    70         -static void fts3SnippetSbInit(StringBuffer *p){
    71         -  p->nAlloc = 100;
    72         -  p->nUsed = 0;
    73         -  p->z = sqlite3_malloc( p->nAlloc );
    74         -}
    75         -
    76         -/*
    77         -** Append text to the string buffer.
    78         -*/
    79         -static void fts3SnippetAppend(StringBuffer *p, const char *zNew, int nNew){
    80         -  if( p->z==0 ) return;
    81         -  if( nNew<0 ) nNew = (int)strlen(zNew);
    82         -  if( p->nUsed + nNew >= p->nAlloc ){
    83         -    int nAlloc;
    84         -    char *zNew;
    85         -
    86         -    nAlloc = p->nUsed + nNew + p->nAlloc;
    87         -    zNew = sqlite3_realloc(p->z, nAlloc);
    88         -    if( zNew==0 ){
    89         -      sqlite3_free(p->z);
    90         -      p->z = 0;
    91         -      return;
    92         -    }
    93         -    p->z = zNew;
    94         -    p->nAlloc = nAlloc;
    95         -  }
    96         -  memcpy(&p->z[p->nUsed], zNew, nNew);
    97         -  p->nUsed += nNew;
    98         -  p->z[p->nUsed] = 0;
    99         -}
   100         -
   101         -/* If the StringBuffer ends in something other than white space, add a
   102         -** single space character to the end.
   103         -*/
   104         -static void fts3SnippetAppendWhiteSpace(StringBuffer *p){
   105         -  if( p->z && p->nUsed && !fts3snippetIsspace(p->z[p->nUsed-1]) ){
   106         -    fts3SnippetAppend(p, " ", 1);
   107         -  }
   108         -}
   109         -
   110         -/* Remove white space from the end of the StringBuffer */
   111         -static void fts3SnippetTrimWhiteSpace(StringBuffer *p){
   112         -  if( p->z ){
   113         -    while( p->nUsed && fts3snippetIsspace(p->z[p->nUsed-1]) ){
   114         -      p->nUsed--;
   115         -    }
   116         -    p->z[p->nUsed] = 0;
   117         -  }
   118         -}
   119         -
   120         -/* 
   121         -** Release all memory associated with the Snippet structure passed as
   122         -** an argument.
   123         -*/
   124         -static void fts3SnippetFree(Snippet *p){
   125         -  if( p ){
   126         -    sqlite3_free(p->aMatch);
   127         -    sqlite3_free(p->zOffset);
   128         -    sqlite3_free(p->zSnippet);
   129         -    sqlite3_free(p);
   130         -  }
   131         -}
   132         -
   133         -/*
   134         -** Append a single entry to the p->aMatch[] log.
   135         -*/
   136         -static int snippetAppendMatch(
   137         -  Snippet *p,               /* Append the entry to this snippet */
   138         -  int iCol, int iTerm,      /* The column and query term */
   139         -  int iToken,               /* Matching token in document */
   140         -  int iStart, int nByte     /* Offset and size of the match */
   141         -){
   142         -  int i;
   143         -  struct snippetMatch *pMatch;
   144         -  if( p->nMatch+1>=p->nAlloc ){
   145         -    struct snippetMatch *pNew;
   146         -    p->nAlloc = p->nAlloc*2 + 10;
   147         -    pNew = sqlite3_realloc(p->aMatch, p->nAlloc*sizeof(p->aMatch[0]) );
   148         -    if( pNew==0 ){
   149         -      p->aMatch = 0;
   150         -      p->nMatch = 0;
   151         -      p->nAlloc = 0;
   152         -      return SQLITE_NOMEM;
   153         -    }
   154         -    p->aMatch = pNew;
   155         -  }
   156         -  i = p->nMatch++;
   157         -  pMatch = &p->aMatch[i];
   158         -  pMatch->iCol = (short)iCol;
   159         -  pMatch->iTerm = (short)iTerm;
   160         -  pMatch->iToken = iToken;
   161         -  pMatch->iStart = iStart;
   162         -  pMatch->nByte = (short)nByte;
   163         -  return SQLITE_OK;
   164         -}
   165         -
   166         -/*
   167         -** Sizing information for the circular buffer used in snippetOffsetsOfColumn()
   168         -*/
   169         -#define FTS3_ROTOR_SZ   (32)
   170         -#define FTS3_ROTOR_MASK (FTS3_ROTOR_SZ-1)
   171         -
   172         -/*
   173         -** Function to iterate through the tokens of a compiled expression.
   174         -**
   175         -** Except, skip all tokens on the right-hand side of a NOT operator.
   176         -** This function is used to find tokens as part of snippet and offset
   177         -** generation and we do nt want snippets and offsets to report matches
   178         -** for tokens on the RHS of a NOT.
   179         -*/
   180         -static int fts3NextExprToken(Fts3Expr **ppExpr, int *piToken){
   181         -  Fts3Expr *p = *ppExpr;
   182         -  int iToken = *piToken;
   183         -  if( iToken<0 ){
   184         -    /* In this case the expression p is the root of an expression tree.
   185         -    ** Move to the first token in the expression tree.
   186         -    */
   187         -    while( p->pLeft ){
   188         -      p = p->pLeft;
   189         -    }
   190         -    iToken = 0;
   191         -  }else{
   192         -    assert(p && p->eType==FTSQUERY_PHRASE );
   193         -    if( iToken<(p->pPhrase->nToken-1) ){
   194         -      iToken++;
   195         -    }else{
   196         -      iToken = 0;
   197         -      while( p->pParent && p->pParent->pLeft!=p ){
   198         -        assert( p->pParent->pRight==p );
   199         -        p = p->pParent;
   200         -      }
   201         -      p = p->pParent;
   202         -      if( p ){
   203         -        assert( p->pRight!=0 );
   204         -        p = p->pRight;
   205         -        while( p->pLeft ){
   206         -          p = p->pLeft;
   207         -        }
   208         -      }
   209         -    }
   210         -  }
   211         -
   212         -  *ppExpr = p;
   213         -  *piToken = iToken;
   214         -  return p?1:0;
   215         -}
   216         -
   217         -/*
   218         -** Return TRUE if the expression node pExpr is located beneath the
   219         -** RHS of a NOT operator.
   220         -*/
   221         -static int fts3ExprBeneathNot(Fts3Expr *p){
   222         -  Fts3Expr *pParent;
   223         -  while( p ){
   224         -    pParent = p->pParent;
   225         -    if( pParent && pParent->eType==FTSQUERY_NOT && pParent->pRight==p ){
   226         -      return 1;
   227         -    }
   228         -    p = pParent;
   229         -  }
   230         -  return 0;
   231         -}
   232         -
   233         -/*
   234         -** Add entries to pSnippet->aMatch[] for every match that occurs against
   235         -** document zDoc[0..nDoc-1] which is stored in column iColumn.
   236         -*/
   237         -static int snippetOffsetsOfColumn(
   238         -  Fts3Cursor *pCur,         /* The fulltest search cursor */
   239         -  Snippet *pSnippet,             /* The Snippet object to be filled in */
   240         -  int iColumn,                   /* Index of fulltext table column */
   241         -  const char *zDoc,              /* Text of the fulltext table column */
   242         -  int nDoc                       /* Length of zDoc in bytes */
   243         -){
   244         -  const sqlite3_tokenizer_module *pTModule;  /* The tokenizer module */
   245         -  sqlite3_tokenizer *pTokenizer;             /* The specific tokenizer */
   246         -  sqlite3_tokenizer_cursor *pTCursor;        /* Tokenizer cursor */
   247         -  Fts3Table *pVtab;                /* The full text index */
   248         -  int nColumn;                         /* Number of columns in the index */
   249         -  int i, j;                            /* Loop counters */
   250         -  int rc;                              /* Return code */
   251         -  unsigned int match, prevMatch;       /* Phrase search bitmasks */
   252         -  const char *zToken;                  /* Next token from the tokenizer */
   253         -  int nToken;                          /* Size of zToken */
   254         -  int iBegin, iEnd, iPos;              /* Offsets of beginning and end */
   255         -
   256         -  /* The following variables keep a circular buffer of the last
   257         -  ** few tokens */
   258         -  unsigned int iRotor = 0;             /* Index of current token */
   259         -  int iRotorBegin[FTS3_ROTOR_SZ];      /* Beginning offset of token */
   260         -  int iRotorLen[FTS3_ROTOR_SZ];        /* Length of token */
   261         -
   262         -  pVtab =  (Fts3Table *)pCur->base.pVtab;
   263         -  nColumn = pVtab->nColumn;
   264         -  pTokenizer = pVtab->pTokenizer;
   265         -  pTModule = pTokenizer->pModule;
   266         -  rc = pTModule->xOpen(pTokenizer, zDoc, nDoc, &pTCursor);
   267         -  if( rc ) return rc;
   268         -  pTCursor->pTokenizer = pTokenizer;
   269         -
   270         -  prevMatch = 0;
   271         -  while( (rc = pTModule->xNext(pTCursor, &zToken, &nToken,
   272         -                               &iBegin, &iEnd, &iPos))==SQLITE_OK ){
   273         -    Fts3Expr *pIter = pCur->pExpr;
   274         -    int iIter = -1;
   275         -    iRotorBegin[iRotor&FTS3_ROTOR_MASK] = iBegin;
   276         -    iRotorLen[iRotor&FTS3_ROTOR_MASK] = iEnd-iBegin;
   277         -    match = 0;
   278         -    for(i=0; i<(FTS3_ROTOR_SZ-1) && fts3NextExprToken(&pIter, &iIter); i++){
   279         -      int nPhrase;                    /* Number of tokens in current phrase */
   280         -      struct PhraseToken *pToken;     /* Current token */
   281         -      int iCol;                       /* Column index */
   282         -
   283         -      if( fts3ExprBeneathNot(pIter) ) continue;
   284         -      nPhrase = pIter->pPhrase->nToken;
   285         -      pToken = &pIter->pPhrase->aToken[iIter];
   286         -      iCol = pIter->pPhrase->iColumn;
   287         -      if( iCol>=0 && iCol<nColumn && iCol!=iColumn ) continue;
   288         -      if( pToken->n>nToken ) continue;
   289         -      if( !pToken->isPrefix && pToken->n<nToken ) continue;
   290         -      assert( pToken->n<=nToken );
   291         -      if( memcmp(pToken->z, zToken, pToken->n) ) continue;
   292         -      if( iIter>0 && (prevMatch & (1<<i))==0 ) continue;
   293         -      match |= 1<<i;
   294         -      if( i==(FTS3_ROTOR_SZ-2) || nPhrase==iIter+1 ){
   295         -        for(j=nPhrase-1; j>=0; j--){
   296         -          int k = (iRotor-j) & FTS3_ROTOR_MASK;
   297         -          rc = snippetAppendMatch(pSnippet, iColumn, i-j, iPos-j,
   298         -                                  iRotorBegin[k], iRotorLen[k]);
   299         -          if( rc ) goto end_offsets_of_column;
   300         -        }
   301         -      }
   302         -    }
   303         -    prevMatch = match<<1;
   304         -    iRotor++;
   305         -  }
   306         -end_offsets_of_column:
   307         -  pTModule->xClose(pTCursor);  
   308         -  return rc==SQLITE_DONE ? SQLITE_OK : rc;
   309         -}
   310         -
   311         -/*
   312         -** Remove entries from the pSnippet structure to account for the NEAR
   313         -** operator. When this is called, pSnippet contains the list of token 
   314         -** offsets produced by treating all NEAR operators as AND operators.
   315         -** This function removes any entries that should not be present after
   316         -** accounting for the NEAR restriction. For example, if the queried
   317         -** document is:
   318         -**
   319         -**     "A B C D E A"
   320         -**
   321         -** and the query is:
   322         -** 
   323         -**     A NEAR/0 E
   324         -**
   325         -** then when this function is called the Snippet contains token offsets
   326         -** 0, 4 and 5. This function removes the "0" entry (because the first A
   327         -** is not near enough to an E).
   328         -**
   329         -** When this function is called, the value pointed to by parameter piLeft is
   330         -** the integer id of the left-most token in the expression tree headed by
   331         -** pExpr. This function increments *piLeft by the total number of tokens
   332         -** in the expression tree headed by pExpr.
   333         -**
   334         -** Return 1 if any trimming occurs.  Return 0 if no trimming is required.
   335         -*/
   336         -static int trimSnippetOffsets(
   337         -  Fts3Expr *pExpr,      /* The search expression */
   338         -  Snippet *pSnippet,    /* The set of snippet offsets to be trimmed */
   339         -  int *piLeft           /* Index of left-most token in pExpr */
   340         -){
   341         -  if( pExpr ){
   342         -    if( trimSnippetOffsets(pExpr->pLeft, pSnippet, piLeft) ){
   343         -      return 1;
   344         -    }
   345         -
   346         -    switch( pExpr->eType ){
   347         -      case FTSQUERY_PHRASE:
   348         -        *piLeft += pExpr->pPhrase->nToken;
   349         -        break;
   350         -      case FTSQUERY_NEAR: {
   351         -        /* The right-hand-side of a NEAR operator is always a phrase. The
   352         -        ** left-hand-side is either a phrase or an expression tree that is 
   353         -        ** itself headed by a NEAR operator. The following initializations
   354         -        ** set local variable iLeft to the token number of the left-most
   355         -        ** token in the right-hand phrase, and iRight to the right most
   356         -        ** token in the same phrase. For example, if we had:
   357         -        **
   358         -        **     <col> MATCH '"abc def" NEAR/2 "ghi jkl"'
   359         -        **
   360         -        ** then iLeft will be set to 2 (token number of ghi) and nToken will
   361         -        ** be set to 4.
   362         -        */
   363         -        Fts3Expr *pLeft = pExpr->pLeft;
   364         -        Fts3Expr *pRight = pExpr->pRight;
   365         -        int iLeft = *piLeft;
   366         -        int nNear = pExpr->nNear;
   367         -        int nToken = pRight->pPhrase->nToken;
   368         -        int jj, ii;
   369         -        if( pLeft->eType==FTSQUERY_NEAR ){
   370         -          pLeft = pLeft->pRight;
   371         -        }
   372         -        assert( pRight->eType==FTSQUERY_PHRASE );
   373         -        assert( pLeft->eType==FTSQUERY_PHRASE );
   374         -        nToken += pLeft->pPhrase->nToken;
   375         -
   376         -        for(ii=0; ii<pSnippet->nMatch; ii++){
   377         -          struct snippetMatch *p = &pSnippet->aMatch[ii];
   378         -          if( p->iTerm==iLeft ){
   379         -            int isOk = 0;
   380         -            /* Snippet ii is an occurence of query term iLeft in the document.
   381         -            ** It occurs at position (p->iToken) of the document. We now
   382         -            ** search for an instance of token (iLeft-1) somewhere in the 
   383         -            ** range (p->iToken - nNear)...(p->iToken + nNear + nToken) within 
   384         -            ** the set of snippetMatch structures. If one is found, proceed. 
   385         -            ** If one cannot be found, then remove snippets ii..(ii+N-1) 
   386         -            ** from the matching snippets, where N is the number of tokens 
   387         -            ** in phrase pRight->pPhrase.
   388         -            */
   389         -            for(jj=0; isOk==0 && jj<pSnippet->nMatch; jj++){
   390         -              struct snippetMatch *p2 = &pSnippet->aMatch[jj];
   391         -              if( p2->iTerm==(iLeft-1) ){
   392         -                if( p2->iToken>=(p->iToken-nNear-1) 
   393         -                 && p2->iToken<(p->iToken+nNear+nToken) 
   394         -                ){
   395         -                  isOk = 1;
   396         -                }
   397         -              }
   398         -            }
   399         -            if( !isOk ){
   400         -              int kk;
   401         -              for(kk=0; kk<pRight->pPhrase->nToken; kk++){
   402         -                pSnippet->aMatch[kk+ii].iTerm = -2;
   403         -              }
   404         -              return 1;
   405         -            }
   406         -          }
   407         -          if( p->iTerm==(iLeft-1) ){
   408         -            int isOk = 0;
   409         -            for(jj=0; isOk==0 && jj<pSnippet->nMatch; jj++){
   410         -              struct snippetMatch *p2 = &pSnippet->aMatch[jj];
   411         -              if( p2->iTerm==iLeft ){
   412         -                if( p2->iToken<=(p->iToken+nNear+1) 
   413         -                 && p2->iToken>(p->iToken-nNear-nToken) 
   414         -                ){
   415         -                  isOk = 1;
   416         -                }
   417         -              }
   418         -            }
   419         -            if( !isOk ){
   420         -              int kk;
   421         -              for(kk=0; kk<pLeft->pPhrase->nToken; kk++){
   422         -                pSnippet->aMatch[ii-kk].iTerm = -2;
   423         -              }
   424         -              return 1;
   425         -            }
   426         -          }
   427         -        }
   428         -        break;
   429         -      }
   430         -    }
   431         -
   432         -    if( trimSnippetOffsets(pExpr->pRight, pSnippet, piLeft) ){
   433         -      return 1;
   434         -    }
   435         -  }
   436         -  return 0;
   437         -}
   438         -
   439         -/*
   440         -** Compute all offsets for the current row of the query.  
   441         -** If the offsets have already been computed, this routine is a no-op.
   442         -*/
   443         -static int snippetAllOffsets(Fts3Cursor *pCsr, Snippet **ppSnippet){
   444         -  Fts3Table *p = (Fts3Table *)pCsr->base.pVtab;  /* The FTS3 virtual table */
   445         -  int nColumn;           /* Number of columns.  Docid does count */
   446         -  int iColumn;           /* Index of of a column */
   447         -  int i;                 /* Loop index */
   448         -  int iFirst;            /* First column to search */
   449         -  int iLast;             /* Last coumn to search */
   450         -  int iTerm = 0;
   451         -  Snippet *pSnippet;
   452         -  int rc = SQLITE_OK;
   453         -
   454         -  if( pCsr->pExpr==0 ){
   455         -    return SQLITE_OK;
   456         -  }
   457         -
   458         -  pSnippet = (Snippet *)sqlite3_malloc(sizeof(Snippet));
   459         -  *ppSnippet = pSnippet;
   460         -  if( !pSnippet ){
   461         -    return SQLITE_NOMEM;
   462         -  }
   463         -  memset(pSnippet, 0, sizeof(Snippet));
   464         -
   465         -  nColumn = p->nColumn;
   466         -  iColumn = (pCsr->eSearch - 2);
   467         -  if( iColumn<0 || iColumn>=nColumn ){
   468         -    /* Look for matches over all columns of the full-text index */
   469         -    iFirst = 0;
   470         -    iLast = nColumn-1;
   471         -  }else{
   472         -    /* Look for matches in the iColumn-th column of the index only */
   473         -    iFirst = iColumn;
   474         -    iLast = iColumn;
   475         -  }
   476         -  for(i=iFirst; rc==SQLITE_OK && i<=iLast; i++){
   477         -    const char *zDoc;
   478         -    int nDoc;
   479         -    zDoc = (const char*)sqlite3_column_text(pCsr->pStmt, i+1);
   480         -    nDoc = sqlite3_column_bytes(pCsr->pStmt, i+1);
   481         -    if( zDoc==0 && sqlite3_column_type(pCsr->pStmt, i+1)!=SQLITE_NULL ){
   482         -      rc = SQLITE_NOMEM;
   483         -    }else{
   484         -      rc = snippetOffsetsOfColumn(pCsr, pSnippet, i, zDoc, nDoc);
   485         -    }
   486         -  }
   487         -
   488         -  while( trimSnippetOffsets(pCsr->pExpr, pSnippet, &iTerm) ){
   489         -    iTerm = 0;
   490         -  }
   491         -
   492         -  return rc;
   493         -}
   494         -
   495         -/*
   496         -** Convert the information in the aMatch[] array of the snippet
   497         -** into the string zOffset[0..nOffset-1]. This string is used as
   498         -** the return of the SQL offsets() function.
   499         -*/
   500         -static void snippetOffsetText(Snippet *p){
   501         -  int i;
   502         -  int cnt = 0;
   503         -  StringBuffer sb;
   504         -  char zBuf[200];
   505         -  if( p->zOffset ) return;
   506         -  fts3SnippetSbInit(&sb);
   507         -  for(i=0; i<p->nMatch; i++){
   508         -    struct snippetMatch *pMatch = &p->aMatch[i];
   509         -    if( pMatch->iTerm>=0 ){
   510         -      /* If snippetMatch.iTerm is less than 0, then the match was 
   511         -      ** discarded as part of processing the NEAR operator (see the 
   512         -      ** trimSnippetOffsetsForNear() function for details). Ignore 
   513         -      ** it in this case
   514         -      */
   515         -      zBuf[0] = ' ';
   516         -      sqlite3_snprintf(sizeof(zBuf)-1, &zBuf[cnt>0], "%d %d %d %d",
   517         -          pMatch->iCol, pMatch->iTerm, pMatch->iStart, pMatch->nByte);
   518         -      fts3SnippetAppend(&sb, zBuf, -1);
   519         -      cnt++;
   520         -    }
   521         -  }
   522         -  p->zOffset = sb.z;
   523         -  p->nOffset = sb.z ? sb.nUsed : 0;
   524         -}
   525         -
   526         -/*
   527         -** zDoc[0..nDoc-1] is phrase of text.  aMatch[0..nMatch-1] are a set
   528         -** of matching words some of which might be in zDoc.  zDoc is column
   529         -** number iCol.
   530         -**
   531         -** iBreak is suggested spot in zDoc where we could begin or end an
   532         -** excerpt.  Return a value similar to iBreak but possibly adjusted
   533         -** to be a little left or right so that the break point is better.
   534         -*/
   535         -static int wordBoundary(
   536         -  int iBreak,                   /* The suggested break point */
   537         -  const char *zDoc,             /* Document text */
   538         -  int nDoc,                     /* Number of bytes in zDoc[] */
   539         -  struct snippetMatch *aMatch,  /* Matching words */
   540         -  int nMatch,                   /* Number of entries in aMatch[] */
   541         -  int iCol                      /* The column number for zDoc[] */
   542         -){
   543         -  int i;
   544         -  if( iBreak<=10 ){
   545         -    return 0;
   546         -  }
   547         -  if( iBreak>=nDoc-10 ){
   548         -    return nDoc;
   549         -  }
   550         -  for(i=0; ALWAYS(i<nMatch) && aMatch[i].iCol<iCol; i++){}
   551         -  while( i<nMatch && aMatch[i].iStart+aMatch[i].nByte<iBreak ){ i++; }
   552         -  if( i<nMatch ){
   553         -    if( aMatch[i].iStart<iBreak+10 ){
   554         -      return aMatch[i].iStart;
   555         -    }
   556         -    if( i>0 && aMatch[i-1].iStart+aMatch[i-1].nByte>=iBreak ){
   557         -      return aMatch[i-1].iStart;
   558         -    }
   559         -  }
   560         -  for(i=1; i<=10; i++){
   561         -    if( fts3snippetIsspace(zDoc[iBreak-i]) ){
   562         -      return iBreak - i + 1;
   563         -    }
   564         -    if( fts3snippetIsspace(zDoc[iBreak+i]) ){
   565         -      return iBreak + i + 1;
   566         -    }
   567         -  }
   568         -  return iBreak;
   569         -}
   570         -
   571         -
   572         -
   573         -/*
   574         -** Allowed values for Snippet.aMatch[].snStatus
   575         -*/
   576         -#define SNIPPET_IGNORE  0   /* It is ok to omit this match from the snippet */
   577         -#define SNIPPET_DESIRED 1   /* We want to include this match in the snippet */
   578         -
   579         -/*
   580         -** Generate the text of a snippet.
   581         -*/
   582         -static void snippetText(
   583         -  Fts3Cursor *pCursor,   /* The cursor we need the snippet for */
   584         -  Snippet *pSnippet,
   585         -  const char *zStartMark,     /* Markup to appear before each match */
   586         -  const char *zEndMark,       /* Markup to appear after each match */
   587         -  const char *zEllipsis       /* Ellipsis mark */
   588         -){
   589         -  int i, j;
   590         -  struct snippetMatch *aMatch;
   591         -  int nMatch;
   592         -  int nDesired;
   593         -  StringBuffer sb;
   594         -  int tailCol;
   595         -  int tailOffset;
   596         -  int iCol;
   597         -  int nDoc;
   598         -  const char *zDoc;
   599         -  int iStart, iEnd;
   600         -  int tailEllipsis = 0;
   601         -  int iMatch;
   602         -  
   603         -
   604         -  sqlite3_free(pSnippet->zSnippet);
   605         -  pSnippet->zSnippet = 0;
   606         -  aMatch = pSnippet->aMatch;
   607         -  nMatch = pSnippet->nMatch;
   608         -  fts3SnippetSbInit(&sb);
   609         -
   610         -  for(i=0; i<nMatch; i++){
   611         -    aMatch[i].snStatus = SNIPPET_IGNORE;
   612         -  }
   613         -  nDesired = 0;
   614         -  for(i=0; i<FTS3_ROTOR_SZ; i++){
   615         -    for(j=0; j<nMatch; j++){
   616         -      if( aMatch[j].iTerm==i ){
   617         -        aMatch[j].snStatus = SNIPPET_DESIRED;
   618         -        nDesired++;
   619         -        break;
   620         -      }
   621         -    }
   622         -  }
   623         -
   624         -  iMatch = 0;
   625         -  tailCol = -1;
   626         -  tailOffset = 0;
   627         -  for(i=0; i<nMatch && nDesired>0; i++){
   628         -    if( aMatch[i].snStatus!=SNIPPET_DESIRED ) continue;
   629         -    nDesired--;
   630         -    iCol = aMatch[i].iCol;
   631         -    zDoc = (const char*)sqlite3_column_text(pCursor->pStmt, iCol+1);
   632         -    nDoc = sqlite3_column_bytes(pCursor->pStmt, iCol+1);
   633         -    iStart = aMatch[i].iStart - 40;
   634         -    iStart = wordBoundary(iStart, zDoc, nDoc, aMatch, nMatch, iCol);
   635         -    if( iStart<=10 ){
   636         -      iStart = 0;
   637         -    }
   638         -    if( iCol==tailCol && iStart<=tailOffset+20 ){
   639         -      iStart = tailOffset;
   640         -    }
   641         -    if( (iCol!=tailCol && tailCol>=0) || iStart!=tailOffset ){
   642         -      fts3SnippetTrimWhiteSpace(&sb);
   643         -      fts3SnippetAppendWhiteSpace(&sb);
   644         -      fts3SnippetAppend(&sb, zEllipsis, -1);
   645         -      fts3SnippetAppendWhiteSpace(&sb);
   646         -    }
   647         -    iEnd = aMatch[i].iStart + aMatch[i].nByte + 40;
   648         -    iEnd = wordBoundary(iEnd, zDoc, nDoc, aMatch, nMatch, iCol);
   649         -    if( iEnd>=nDoc-10 ){
   650         -      iEnd = nDoc;
   651         -      tailEllipsis = 0;
   652         -    }else{
   653         -      tailEllipsis = 1;
   654         -    }
   655         -    while( iMatch<nMatch && aMatch[iMatch].iCol<iCol ){ iMatch++; }
   656         -    while( iStart<iEnd ){
   657         -      while( iMatch<nMatch && aMatch[iMatch].iStart<iStart
   658         -             && aMatch[iMatch].iCol<=iCol ){
   659         -        iMatch++;
   660         -      }
   661         -      if( iMatch<nMatch && aMatch[iMatch].iStart<iEnd
   662         -             && aMatch[iMatch].iCol==iCol ){
   663         -        fts3SnippetAppend(&sb, &zDoc[iStart], aMatch[iMatch].iStart - iStart);
   664         -        iStart = aMatch[iMatch].iStart;
   665         -        fts3SnippetAppend(&sb, zStartMark, -1);
   666         -        fts3SnippetAppend(&sb, &zDoc[iStart], aMatch[iMatch].nByte);
   667         -        fts3SnippetAppend(&sb, zEndMark, -1);
   668         -        iStart += aMatch[iMatch].nByte;
   669         -        for(j=iMatch+1; j<nMatch; j++){
   670         -          if( aMatch[j].iTerm==aMatch[iMatch].iTerm
   671         -              && aMatch[j].snStatus==SNIPPET_DESIRED ){
   672         -            nDesired--;
   673         -            aMatch[j].snStatus = SNIPPET_IGNORE;
   674         -          }
   675         -        }
   676         -      }else{
   677         -        fts3SnippetAppend(&sb, &zDoc[iStart], iEnd - iStart);
   678         -        iStart = iEnd;
   679         -      }
   680         -    }
   681         -    tailCol = iCol;
   682         -    tailOffset = iEnd;
   683         -  }
   684         -  fts3SnippetTrimWhiteSpace(&sb);
   685         -  if( tailEllipsis ){
   686         -    fts3SnippetAppendWhiteSpace(&sb);
   687         -    fts3SnippetAppend(&sb, zEllipsis, -1);
   688         -  }
   689         -  pSnippet->zSnippet = sb.z;
   690         -  pSnippet->nSnippet = sb.z ? sb.nUsed : 0;
   691         -}
   692         -
   693         -void sqlite3Fts3Offsets(
   694         -  sqlite3_context *pCtx,          /* SQLite function call context */
   695         -  Fts3Cursor *pCsr                /* Cursor object */
   696         -){
   697         -  Snippet *p;                     /* Snippet structure */
   698         -  int rc = snippetAllOffsets(pCsr, &p);
   699         -  if( rc==SQLITE_OK ){
   700         -    snippetOffsetText(p);
   701         -    if( p->zOffset ){
   702         -      sqlite3_result_text(pCtx, p->zOffset, p->nOffset, SQLITE_TRANSIENT);
   703         -    }else{
   704         -      sqlite3_result_error_nomem(pCtx);
   705         -    }
   706         -  }else{
   707         -    sqlite3_result_error_nomem(pCtx);
   708         -  }
   709         -  fts3SnippetFree(p);
   710         -}
   711         -
   712         -void sqlite3Fts3Snippet(
   713         -  sqlite3_context *pCtx,          /* SQLite function call context */
   714         -  Fts3Cursor *pCsr,               /* Cursor object */
   715         -  const char *zStart,             /* Snippet start text - "<b>" */
   716         -  const char *zEnd,               /* Snippet end text - "</b>" */
   717         -  const char *zEllipsis           /* Snippet ellipsis text - "<b>...</b>" */
   718         -){
   719         -  Snippet *p;                     /* Snippet structure */
   720         -  int rc = snippetAllOffsets(pCsr, &p);
   721         -  if( rc==SQLITE_OK ){
   722         -    snippetText(pCsr, p, zStart, zEnd, zEllipsis);
   723         -    if( p->zSnippet ){
   724         -      sqlite3_result_text(pCtx, p->zSnippet, p->nSnippet, SQLITE_TRANSIENT);
   725         -    }else{
   726         -      sqlite3_result_error_nomem(pCtx);
   727         -    }
   728         -  }else{
   729         -    sqlite3_result_error_nomem(pCtx);
   730         -  }
   731         -  fts3SnippetFree(p);
   732         -}
   733         -
   734         -/*************************************************************************
   735         -** Below this point is the alternative, experimental snippet() implementation.
   736         -*/
   737         -
   738         -#define SNIPPET_BUFFER_CHUNK  64
   739         -#define SNIPPET_BUFFER_SIZE   SNIPPET_BUFFER_CHUNK*4
   740         -#define SNIPPET_BUFFER_MASK   (SNIPPET_BUFFER_SIZE-1)
   741         -
           92  +/*
           93  +** This function is used to help iterate through a position-list. A position
           94  +** list is a list of unique integers, sorted from smallest to largest. Each
           95  +** element of the list is represented by an FTS3 varint that takes the value
           96  +** of the difference between the current element and the previous one plus
           97  +** two. For example, to store the position-list:
           98  +**
           99  +**     4 9 113
          100  +**
          101  +** the three varints:
          102  +**
          103  +**     6 7 106
          104  +**
          105  +** are encoded.
          106  +**
          107  +** When this function is called, *pp points to the start of an element of
          108  +** the list. *piPos contains the value of the previous entry in the list.
          109  +** After it returns, *piPos contains the value of the next element of the
          110  +** list and *pp is advanced to the following varint.
          111  +*/
   742    112   static void fts3GetDeltaPosition(char **pp, int *piPos){
   743    113     int iVal;
   744    114     *pp += sqlite3Fts3GetVarint32(*pp, &iVal);
   745    115     *piPos += (iVal-2);
   746    116   }
          117  +
          118  +/*
          119  +** Helper function for fts3ExprIterate() (see below).
          120  +*/
          121  +static int fts3ExprIterate2(
          122  +  Fts3Expr *pExpr,                /* Expression to iterate phrases of */
          123  +  int *piPhrase,                  /* Pointer to phrase counter */
          124  +  int (*x)(Fts3Expr*,int,void*),  /* Callback function to invoke for phrases */
          125  +  void *pCtx                      /* Second argument to pass to callback */
          126  +){
          127  +  int rc;                         /* Return code */
          128  +  int eType = pExpr->eType;       /* Type of expression node pExpr */
          129  +
          130  +  if( eType!=FTSQUERY_PHRASE ){
          131  +    assert( pExpr->pLeft && pExpr->pRight );
          132  +    rc = fts3ExprIterate2(pExpr->pLeft, piPhrase, x, pCtx);
          133  +    if( rc==SQLITE_OK && eType!=FTSQUERY_NOT ){
          134  +      rc = fts3ExprIterate2(pExpr->pRight, piPhrase, x, pCtx);
          135  +    }
          136  +  }else{
          137  +    rc = x(pExpr, *piPhrase, pCtx);
          138  +    (*piPhrase)++;
          139  +  }
          140  +  return rc;
          141  +}
   747    142   
   748    143   /*
   749    144   ** Iterate through all phrase nodes in an FTS3 query, except those that
   750    145   ** are part of a sub-tree that is the right-hand-side of a NOT operator.
   751    146   ** For each phrase node found, the supplied callback function is invoked.
   752    147   **
   753    148   ** If the callback function returns anything other than SQLITE_OK, 
   754    149   ** the iteration is abandoned and the error code returned immediately.
   755    150   ** Otherwise, SQLITE_OK is returned after a callback has been made for
   756    151   ** all eligible phrase nodes.
   757    152   */
   758    153   static int fts3ExprIterate(
   759    154     Fts3Expr *pExpr,                /* Expression to iterate phrases of */
   760         -  int (*x)(Fts3Expr *, void *),   /* Callback function to invoke for phrases */
          155  +  int (*x)(Fts3Expr*,int,void*),  /* Callback function to invoke for phrases */
   761    156     void *pCtx                      /* Second argument to pass to callback */
   762    157   ){
   763         -  int rc;
   764         -  int eType = pExpr->eType;
   765         -  if( eType==FTSQUERY_NOT ){
   766         -    rc = SQLITE_OK;
   767         -  }else if( eType!=FTSQUERY_PHRASE ){
   768         -    assert( pExpr->pLeft && pExpr->pRight );
   769         -    rc = fts3ExprIterate(pExpr->pLeft, x, pCtx);
   770         -    if( rc==SQLITE_OK ){
   771         -      rc = fts3ExprIterate(pExpr->pRight, x, pCtx);
   772         -    }
   773         -  }else{
   774         -    rc = x(pExpr, pCtx);
   775         -  }
          158  +  int iPhrase = 0;                /* Variable used as the phrase counter */
          159  +  return fts3ExprIterate2(pExpr, &iPhrase, x, pCtx);
          160  +}
          161  +
          162  +/*
          163  +** The argument to this function is always a phrase node. Its doclist 
          164  +** (Fts3Expr.aDoclist[]) and the doclists associated with all phrase nodes
          165  +** to the left of this one in the query tree have already been loaded.
          166  +**
          167  +** If this phrase node is part of a series of phrase nodes joined by 
          168  +** NEAR operators (and is not the left-most of said series), then elements are
          169  +** removed from the phrases doclist consistent with the NEAR restriction. If
          170  +** required, elements may be removed from the doclists of phrases to the
          171  +** left of this one that are part of the same series of NEAR operator 
          172  +** connected phrases.
          173  +**
          174  +** If an OOM error occurs, SQLITE_NOMEM is returned. Otherwise, SQLITE_OK.
          175  +*/
          176  +static int fts3ExprNearTrim(Fts3Expr *pExpr){
          177  +  int rc = SQLITE_OK;
          178  +  Fts3Expr *pParent = pExpr->pParent;
          179  +
          180  +  assert( pExpr->eType==FTSQUERY_PHRASE );
          181  +  while( rc==SQLITE_OK
          182  +   && pParent 
          183  +   && pParent->eType==FTSQUERY_NEAR 
          184  +   && pParent->pRight==pExpr 
          185  +  ){
          186  +    /* This expression (pExpr) is the right-hand-side of a NEAR operator. 
          187  +    ** Find the expression to the left of the same operator.
          188  +    */
          189  +    int nNear = pParent->nNear;
          190  +    Fts3Expr *pLeft = pParent->pLeft;
          191  +
          192  +    if( pLeft->eType!=FTSQUERY_PHRASE ){
          193  +      assert( pLeft->eType==FTSQUERY_NEAR );
          194  +      assert( pLeft->pRight->eType==FTSQUERY_PHRASE );
          195  +      pLeft = pLeft->pRight;
          196  +    }
          197  +
          198  +    rc = sqlite3Fts3ExprNearTrim(pLeft, pExpr, nNear);
          199  +
          200  +    pExpr = pLeft;
          201  +    pParent = pExpr->pParent;
          202  +  }
          203  +
   776    204     return rc;
   777    205   }
   778    206   
   779         -typedef struct LoadDoclistCtx LoadDoclistCtx;
   780         -struct LoadDoclistCtx {
   781         -  Fts3Table *pTab;                /* FTS3 Table */
   782         -  int nPhrase;                    /* Number of phrases so far */
   783         -};
   784         -
   785         -static int fts3ExprLoadDoclistsCb(Fts3Expr *pExpr, void *ctx){
          207  +/*
          208  +** This is an fts3ExprIterate() callback used while loading the doclists
          209  +** for each phrase into Fts3Expr.aDoclist[]/nDoclist. See also
          210  +** fts3ExprLoadDoclists().
          211  +*/
          212  +static int fts3ExprLoadDoclistsCb1(Fts3Expr *pExpr, int iPhrase, void *ctx){
   786    213     int rc = SQLITE_OK;
   787    214     LoadDoclistCtx *p = (LoadDoclistCtx *)ctx;
          215  +
   788    216     p->nPhrase++;
          217  +  p->nToken += pExpr->pPhrase->nToken;
          218  +
   789    219     if( pExpr->isLoaded==0 ){
   790    220       rc = sqlite3Fts3ExprLoadDoclist(p->pTab, pExpr);
   791    221       pExpr->isLoaded = 1;
   792         -    if( rc==SQLITE_OK && pExpr->aDoclist ){
   793         -      pExpr->pCurrent = pExpr->aDoclist;
   794         -      pExpr->pCurrent += sqlite3Fts3GetVarint(pExpr->pCurrent,&pExpr->iCurrent);
          222  +    if( rc==SQLITE_OK ){
          223  +      rc = fts3ExprNearTrim(pExpr);
   795    224       }
   796    225     }
          226  +
   797    227     return rc;
   798    228   }
   799    229   
   800         -static int fts3ExprLoadDoclists(Fts3Cursor *pCsr, int *pnPhrase){
   801         -  int rc;
   802         -  LoadDoclistCtx sCtx = {0, 0};
          230  +/*
          231  +** This is an fts3ExprIterate() callback used while loading the doclists
          232  +** for each phrase into Fts3Expr.aDoclist[]/nDoclist. See also
          233  +** fts3ExprLoadDoclists().
          234  +*/
          235  +static int fts3ExprLoadDoclistsCb2(Fts3Expr *pExpr, int iPhrase, void *ctx){
          236  +  if( pExpr->aDoclist ){
          237  +    pExpr->pCurrent = pExpr->aDoclist;
          238  +    pExpr->iCurrent = 0;
          239  +    pExpr->pCurrent += sqlite3Fts3GetVarint(pExpr->pCurrent, &pExpr->iCurrent);
          240  +  }
          241  +  return SQLITE_OK;
          242  +}
          243  +
          244  +/*
          245  +** Load the doclists for each phrase in the query associated with FTS3 cursor
          246  +** pCsr. 
          247  +**
          248  +** If pnPhrase is not NULL, then *pnPhrase is set to the number of matchable 
          249  +** phrases in the expression (all phrases except those directly or 
          250  +** indirectly descended from the right-hand-side of a NOT operator). If 
          251  +** pnToken is not NULL, then it is set to the number of tokens in all
          252  +** matchable phrases of the expression.
          253  +*/
          254  +static int fts3ExprLoadDoclists(
          255  +  Fts3Cursor *pCsr,               /* Fts3 cursor for current query */
          256  +  int *pnPhrase,                  /* OUT: Number of phrases in query */
          257  +  int *pnToken                    /* OUT: Number of tokens in query */
          258  +){
          259  +  int rc;                         /* Return Code */
          260  +  LoadDoclistCtx sCtx = {0,0,0};  /* Context for fts3ExprIterate() */
   803    261     sCtx.pTab = (Fts3Table *)pCsr->base.pVtab;
   804         -  rc = fts3ExprIterate(pCsr->pExpr, fts3ExprLoadDoclistsCb, (void *)&sCtx);
   805         -  *pnPhrase = sCtx.nPhrase;
          262  +  rc = fts3ExprIterate(pCsr->pExpr, fts3ExprLoadDoclistsCb1, (void *)&sCtx);
          263  +  if( rc==SQLITE_OK ){
          264  +    (void)fts3ExprIterate(pCsr->pExpr, fts3ExprLoadDoclistsCb2, 0);
          265  +  }
          266  +  if( pnPhrase ) *pnPhrase = sCtx.nPhrase;
          267  +  if( pnToken ) *pnToken = sCtx.nToken;
   806    268     return rc;
   807    269   }
   808    270   
   809    271   /*
   810         -** Each call to this function populates a chunk of a snippet-buffer 
   811         -** SNIPPET_BUFFER_CHUNK bytes in size.
   812         -**
   813         -** Return true if the end of the data has been reached (and all subsequent
   814         -** calls to fts3LoadSnippetBuffer() with the same arguments will be no-ops), 
   815         -** or false otherwise.
   816         -*/
   817         -static int fts3LoadSnippetBuffer(
   818         -  int iPos,                       /* Document token offset to load data for */
   819         -  u8 *aBuffer,                    /* Circular snippet buffer to populate */
   820         -  int nList,                      /* Number of position lists in appList */
   821         -  char **apList,                  /* IN/OUT: nList position list pointers */
   822         -  int *aiPrev                     /* IN/OUT: Previous positions read */
   823         -){
   824         -  int i;
   825         -  int nFin = 0;
   826         -
   827         -  assert( (iPos&(SNIPPET_BUFFER_CHUNK-1))==0 );
   828         -
   829         -  memset(&aBuffer[iPos&SNIPPET_BUFFER_MASK], 0, SNIPPET_BUFFER_CHUNK);
   830         -
   831         -  for(i=0; i<nList; i++){
   832         -    int iPrev = aiPrev[i];
   833         -    char *pList = apList[i];
   834         -
   835         -    if( !pList ){
   836         -      nFin++;
   837         -      continue;
   838         -    }
   839         -
   840         -    while( iPrev<(iPos+SNIPPET_BUFFER_CHUNK) ){
   841         -      if( iPrev>=iPos ){
   842         -        aBuffer[iPrev&SNIPPET_BUFFER_MASK] = (u8)(i+1);
   843         -      }
   844         -      if( 0==((*pList)&0xFE) ){
   845         -        nFin++;
          272  +** Advance the position list iterator specified by the first two 
          273  +** arguments so that it points to the first element with a value greater
          274  +** than or equal to parameter iNext.
          275  +*/
          276  +static void fts3SnippetAdvance(char **ppIter, int *piIter, int iNext){
          277  +  char *pIter = *ppIter;
          278  +  if( pIter ){
          279  +    int iIter = *piIter;
          280  +
          281  +    while( iIter<iNext ){
          282  +      if( 0==(*pIter & 0xFE) ){
          283  +        iIter = -1;
          284  +        pIter = 0;
   846    285           break;
   847    286         }
   848         -      fts3GetDeltaPosition(&pList, &iPrev); 
   849         -    }
   850         -
   851         -    aiPrev[i] = iPrev;
   852         -    apList[i] = pList;
   853         -  }
   854         -
   855         -  return (nFin==nList);
   856         -}
   857         -
   858         -typedef struct SnippetCtx SnippetCtx;
   859         -struct SnippetCtx {
   860         -  Fts3Cursor *pCsr;
   861         -  int iCol;
   862         -  int iPhrase;
   863         -  int *aiPrev;
   864         -  int *anToken;
   865         -  char **apList;
   866         -};
   867         -
   868         -static int fts3SnippetFindPositions(Fts3Expr *pExpr, void *ctx){
   869         -  SnippetCtx *p = (SnippetCtx *)ctx;
   870         -  int iPhrase = p->iPhrase++;
          287  +      fts3GetDeltaPosition(&pIter, &iIter);
          288  +    }
          289  +
          290  +    *piIter = iIter;
          291  +    *ppIter = pIter;
          292  +  }
          293  +}
          294  +
          295  +/*
          296  +** Advance the snippet iterator to the next candidate snippet.
          297  +*/
          298  +static int fts3SnippetNextCandidate(SnippetIter *pIter){
          299  +  int i;                          /* Loop counter */
          300  +
          301  +  if( pIter->iCurrent<0 ){
          302  +    /* The SnippetIter object has just been initialized. The first snippet
          303  +    ** candidate always starts at offset 0 (even if this candidate has a
          304  +    ** score of 0.0).
          305  +    */
          306  +    pIter->iCurrent = 0;
          307  +
          308  +    /* Advance the 'head' iterator of each phrase to the first offset that
          309  +    ** is greater than or equal to (iNext+nSnippet).
          310  +    */
          311  +    for(i=0; i<pIter->nPhrase; i++){
          312  +      SnippetPhrase *pPhrase = &pIter->aPhrase[i];
          313  +      fts3SnippetAdvance(&pPhrase->pHead, &pPhrase->iHead, pIter->nSnippet);
          314  +    }
          315  +  }else{
          316  +    int iStart;
          317  +    int iEnd = 0x7FFFFFFF;
          318  +
          319  +    for(i=0; i<pIter->nPhrase; i++){
          320  +      SnippetPhrase *pPhrase = &pIter->aPhrase[i];
          321  +      if( pPhrase->pHead && pPhrase->iHead<iEnd ){
          322  +        iEnd = pPhrase->iHead;
          323  +      }
          324  +    }
          325  +    if( iEnd==0x7FFFFFFF ){
          326  +      return 1;
          327  +    }
          328  +
          329  +    pIter->iCurrent = iStart = iEnd - pIter->nSnippet + 1;
          330  +    for(i=0; i<pIter->nPhrase; i++){
          331  +      SnippetPhrase *pPhrase = &pIter->aPhrase[i];
          332  +      fts3SnippetAdvance(&pPhrase->pHead, &pPhrase->iHead, iEnd+1);
          333  +      fts3SnippetAdvance(&pPhrase->pTail, &pPhrase->iTail, iStart);
          334  +    }
          335  +  }
          336  +
          337  +  return 0;
          338  +}
          339  +
          340  +/*
          341  +** Retrieve information about the current candidate snippet of snippet 
          342  +** iterator pIter.
          343  +*/
          344  +static void fts3SnippetDetails(
          345  +  SnippetIter *pIter,             /* Snippet iterator */
          346  +  u64 mCovered,                   /* Bitmask of phrases already covered */
          347  +  int *piToken,                   /* OUT: First token of proposed snippet */
          348  +  int *piScore,                   /* OUT: "Score" for this snippet */
          349  +  u64 *pmCover,                   /* OUT: Bitmask of phrases covered */
          350  +  u64 *pmHighlight                /* OUT: Bitmask of terms to highlight */
          351  +){
          352  +  int iStart = pIter->iCurrent;   /* First token of snippet */
          353  +  int iScore = 0;                 /* Score of this snippet */
          354  +  int i;                          /* Loop counter */
          355  +  u64 mCover = 0;                 /* Mask of phrases covered by this snippet */
          356  +  u64 mHighlight = 0;             /* Mask of tokens to highlight in snippet */
          357  +
          358  +  for(i=0; i<pIter->nPhrase; i++){
          359  +    SnippetPhrase *pPhrase = &pIter->aPhrase[i];
          360  +    if( pPhrase->pTail ){
          361  +      char *pCsr = pPhrase->pTail;
          362  +      int iCsr = pPhrase->iTail;
          363  +
          364  +      while( iCsr<(iStart+pIter->nSnippet) ){
          365  +        int j;
          366  +        u64 mPhrase = (u64)1 << i;
          367  +        u64 mPos = (u64)1 << (iCsr - iStart);
          368  +        assert( iCsr>=iStart );
          369  +        if( (mCover|mCovered)&mPhrase ){
          370  +          iScore++;
          371  +        }else{
          372  +          iScore += 1000;
          373  +        }
          374  +        mCover |= mPhrase;
          375  +
          376  +        for(j=0; j<pPhrase->nToken; j++){
          377  +          mHighlight |= (mPos>>j);
          378  +        }
          379  +
          380  +        if( 0==(*pCsr & 0x0FE) ) break;
          381  +        fts3GetDeltaPosition(&pCsr, &iCsr);
          382  +      }
          383  +    }
          384  +  }
          385  +
          386  +  /* Set the output variables before returning. */
          387  +  *piToken = iStart;
          388  +  *piScore = iScore;
          389  +  *pmCover = mCover;
          390  +  *pmHighlight = mHighlight;
          391  +}
          392  +
          393  +/*
          394  +** This function is an fts3ExprIterate() callback used by fts3BestSnippet().
          395  +** Each invocation populates an element of the SnippetIter.aPhrase[] array.
          396  +*/
          397  +static int fts3SnippetFindPositions(Fts3Expr *pExpr, int iPhrase, void *ctx){
          398  +  SnippetIter *p = (SnippetIter *)ctx;
          399  +  SnippetPhrase *pPhrase = &p->aPhrase[iPhrase];
   871    400     char *pCsr;
   872    401   
   873         -  p->anToken[iPhrase] = pExpr->pPhrase->nToken;
          402  +  pPhrase->nToken = pExpr->pPhrase->nToken;
          403  +
   874    404     pCsr = sqlite3Fts3FindPositions(pExpr, p->pCsr->iPrevId, p->iCol);
   875         -
   876    405     if( pCsr ){
   877         -    int iVal;
   878         -    pCsr += sqlite3Fts3GetVarint32(pCsr, &iVal);
   879         -    p->apList[iPhrase] = pCsr;
   880         -    p->aiPrev[iPhrase] = iVal-2;
          406  +    int iFirst = 0;
          407  +    pPhrase->pList = pCsr;
          408  +    fts3GetDeltaPosition(&pCsr, &iFirst);
          409  +    pPhrase->pHead = pCsr;
          410  +    pPhrase->pTail = pCsr;
          411  +    pPhrase->iHead = iFirst;
          412  +    pPhrase->iTail = iFirst;
          413  +  }else{
          414  +    assert( pPhrase->pList==0 && pPhrase->pHead==0 && pPhrase->pTail==0 );
   881    415     }
          416  +
   882    417     return SQLITE_OK;
   883    418   }
   884    419   
   885         -static void fts3SnippetCnt(
   886         -  int iIdx, 
   887         -  int nSnippet, 
   888         -  int *anCnt, 
   889         -  u8 *aBuffer,
   890         -  int *anToken,
   891         -  u64 *pHlmask
   892         -){
   893         -  int iSub =  (iIdx-1)&SNIPPET_BUFFER_MASK;
   894         -  int iAdd =  (iIdx+nSnippet-1)&SNIPPET_BUFFER_MASK;
   895         -  int iSub2 = (iIdx+(nSnippet/3)-1)&SNIPPET_BUFFER_MASK;
   896         -  int iAdd2 = (iIdx+(nSnippet*2/3)-1)&SNIPPET_BUFFER_MASK;
   897         -
   898         -  u64 h = *pHlmask;
   899         -
   900         -  anCnt[ aBuffer[iSub]  ]--;
   901         -  anCnt[ aBuffer[iSub2] ]--;
   902         -  anCnt[ aBuffer[iAdd]  ]++;
   903         -  anCnt[ aBuffer[iAdd2] ]++;
   904         -
   905         -  h = h >> 1;
   906         -  if( aBuffer[iAdd] ){
   907         -    int j;
   908         -    for(j=anToken[aBuffer[iAdd]-1]; j>=1; j--){
   909         -      h |= (u64)1 << (nSnippet-j);
   910         -    }
   911         -  }
   912         -  *pHlmask = h;
   913         -}
   914         -
   915         -static int fts3SnippetScore(int n, int *anCnt){
   916         -  int j;
   917         -  int iScore = 0;
   918         -  for(j=1; j<=n; j++){
   919         -    int nCnt = anCnt[j];
   920         -    iScore += nCnt + (nCnt ? 1000 : 0);
   921         -  }
   922         -  return iScore;
   923         -}
   924         -
          420  +/*
          421  +** Select the fragment of text consisting of nFragment contiguous tokens 
          422  +** from column iCol that represent the "best" snippet. The best snippet
          423  +** is the snippet with the highest score, where scores are calculated
          424  +** by adding:
          425  +**
          426  +**   (a) +1 point for each occurence of a matchable phrase in the snippet.
          427  +**
          428  +**   (b) +1000 points for the first occurence of each matchable phrase in 
          429  +**       the snippet for which the corresponding mCovered bit is not set.
          430  +**
          431  +** The selected snippet parameters are stored in structure *pFragment before
          432  +** returning. The score of the selected snippet is stored in *piScore
          433  +** before returning.
          434  +*/
   925    435   static int fts3BestSnippet(
   926    436     int nSnippet,                   /* Desired snippet length */
   927    437     Fts3Cursor *pCsr,               /* Cursor to create snippet for */
   928    438     int iCol,                       /* Index of column to create snippet from */
   929         -  int *piPos,                     /* OUT: Starting token for best snippet */
   930         -  u64 *pHlmask                    /* OUT: Highlight mask for best snippet */
          439  +  u64 mCovered,                   /* Mask of phrases already covered */
          440  +  u64 *pmSeen,                    /* IN/OUT: Mask of phrases seen */
          441  +  SnippetFragment *pFragment,     /* OUT: Best snippet found */
          442  +  int *piScore                    /* OUT: Score of snippet pFragment */
   931    443   ){
   932    444     int rc;                         /* Return Code */
   933         -  u8 aBuffer[SNIPPET_BUFFER_SIZE];/* Circular snippet buffer */
   934         -  int *aiPrev;                    /* Used by fts3LoadSnippetBuffer() */
   935         -  int *anToken;                   /* Number of tokens in each phrase */
   936         -  char **apList;                  /* Array of position lists */
   937         -  int *anCnt;                     /* Running totals of phrase occurences */
   938         -  int nList;
          445  +  int nList;                      /* Number of phrases in expression */
          446  +  SnippetIter sIter;              /* Iterates through snippet candidates */
          447  +  int nByte;                      /* Number of bytes of space to allocate */
          448  +  int iBestScore = -1;            /* Best snippet score found so far */
          449  +  int i;                          /* Loop counter */
   939    450   
   940         -  int i;
   941         -
   942         -  u64 hlmask = 0;                 /* Current mask of highlighted terms */
   943         -  u64 besthlmask = 0;             /* Mask of highlighted terms for iBestPos */
   944         -  int iBestPos = 0;               /* Starting position of 'best' snippet */
   945         -  int iBestScore = 0;             /* Score of best snippet higher->better */
   946         -  SnippetCtx sCtx;
          451  +  memset(&sIter, 0, sizeof(sIter));
   947    452   
   948    453     /* Iterate through the phrases in the expression to count them. The same
   949    454     ** callback makes sure the doclists are loaded for each phrase.
   950    455     */
   951         -  rc = fts3ExprLoadDoclists(pCsr, &nList);
          456  +  rc = fts3ExprLoadDoclists(pCsr, &nList, 0);
   952    457     if( rc!=SQLITE_OK ){
   953    458       return rc;
   954    459     }
   955    460   
   956    461     /* Now that it is known how many phrases there are, allocate and zero
   957         -  ** the required arrays using malloc().
          462  +  ** the required space using malloc().
   958    463     */
   959         -  apList = sqlite3_malloc(
   960         -      sizeof(u8*)*nList +         /* apList */
   961         -      sizeof(int)*(nList) +       /* anToken */
   962         -      sizeof(int)*nList +         /* aiPrev */
   963         -      sizeof(int)*(nList+1)       /* anCnt */
   964         -  );
   965         -  if( !apList ){
          464  +  nByte = sizeof(SnippetPhrase) * nList;
          465  +  sIter.aPhrase = (SnippetPhrase *)sqlite3_malloc(nByte);
          466  +  if( !sIter.aPhrase ){
   966    467       return SQLITE_NOMEM;
   967    468     }
   968         -  memset(apList, 0, sizeof(u8*)*nList+sizeof(int)*nList+sizeof(int)*nList);
   969         -  anToken = (int *)&apList[nList];
   970         -  aiPrev = &anToken[nList];
   971         -  anCnt = &aiPrev[nList];
   972         -
   973         -  /* Initialize the contents of the aiPrev and aiList arrays. */
   974         -  sCtx.pCsr = pCsr;
   975         -  sCtx.iCol = iCol;
   976         -  sCtx.apList = apList;
   977         -  sCtx.aiPrev = aiPrev;
   978         -  sCtx.anToken = anToken;
   979         -  sCtx.iPhrase = 0;
   980         -  (void)fts3ExprIterate(pCsr->pExpr, fts3SnippetFindPositions, (void *)&sCtx);
   981         -
   982         -  /* Load the first two chunks of data into the buffer. */
   983         -  memset(aBuffer, 0, SNIPPET_BUFFER_SIZE);
   984         -  fts3LoadSnippetBuffer(0, aBuffer, nList, apList, aiPrev);
   985         -  fts3LoadSnippetBuffer(SNIPPET_BUFFER_CHUNK, aBuffer, nList, apList, aiPrev);
   986         -
   987         -  /* Set the initial contents of the highlight-mask and anCnt[] array. */
   988         -  for(i=1-nSnippet; i<=0; i++){
   989         -    fts3SnippetCnt(i, nSnippet, anCnt, aBuffer, anToken, &hlmask);
   990         -  }
   991         -  iBestScore = fts3SnippetScore(nList, anCnt);
   992         -  besthlmask = hlmask;
   993         -  iBestPos = 0;
   994         -
   995         -  for(i=1; 1; i++){
          469  +  memset(sIter.aPhrase, 0, nByte);
          470  +
          471  +  /* Initialize the contents of the SnippetIter object. Then iterate through
          472  +  ** the set of phrases in the expression to populate the aPhrase[] array.
          473  +  */
          474  +  sIter.pCsr = pCsr;
          475  +  sIter.iCol = iCol;
          476  +  sIter.nSnippet = nSnippet;
          477  +  sIter.nPhrase = nList;
          478  +  sIter.iCurrent = -1;
          479  +  (void)fts3ExprIterate(pCsr->pExpr, fts3SnippetFindPositions, (void *)&sIter);
          480  +
          481  +  /* Set the *pmSeen output variable. */
          482  +  for(i=0; i<nList; i++){
          483  +    if( sIter.aPhrase[i].pHead ){
          484  +      *pmSeen |= (u64)1 << i;
          485  +    }
          486  +  }
          487  +
          488  +  /* Loop through all candidate snippets. Store the best snippet in 
          489  +  ** *pFragment. Store its associated 'score' in iBestScore.
          490  +  */
          491  +  pFragment->iCol = iCol;
          492  +  while( !fts3SnippetNextCandidate(&sIter) ){
          493  +    int iPos;
   996    494       int iScore;
   997         -
   998         -    if( 0==(i&(SNIPPET_BUFFER_CHUNK-1)) ){
   999         -      int iLoad = i + SNIPPET_BUFFER_CHUNK;
  1000         -      if( fts3LoadSnippetBuffer(iLoad, aBuffer, nList, apList, aiPrev) ) break;
  1001         -    }
  1002         -
  1003         -    /* Figure out how highly a snippet starting at token offset i scores
  1004         -    ** according to fts3SnippetScore(). If it is higher than any previously
  1005         -    ** considered position, save the current position, score and hlmask as 
  1006         -    ** the best snippet candidate found so far.
  1007         -    */
  1008         -    fts3SnippetCnt(i, nSnippet, anCnt, aBuffer, anToken, &hlmask);
  1009         -    iScore = fts3SnippetScore(nList, anCnt);
          495  +    u64 mCover;
          496  +    u64 mHighlight;
          497  +    fts3SnippetDetails(&sIter, mCovered, &iPos, &iScore, &mCover, &mHighlight);
          498  +    assert( iScore>=0 );
  1010    499       if( iScore>iBestScore ){
  1011         -      iBestPos = i;
          500  +      pFragment->iPos = iPos;
          501  +      pFragment->hlmask = mHighlight;
          502  +      pFragment->covered = mCover;
  1012    503         iBestScore = iScore;
  1013         -      besthlmask = hlmask;
  1014    504       }
  1015    505     }
  1016    506   
  1017         -  sqlite3_free(apList);
  1018         -  *piPos = iBestPos;
  1019         -  *pHlmask = besthlmask;
          507  +  sqlite3_free(sIter.aPhrase);
          508  +  *piScore = iBestScore;
  1020    509     return SQLITE_OK;
  1021    510   }
  1022    511   
  1023         -typedef struct StrBuffer StrBuffer;
  1024         -struct StrBuffer {
  1025         -  char *z;
  1026         -  int n;
  1027         -  int nAlloc;
  1028         -};
  1029    512   
          513  +/*
          514  +** Append a string to the string-buffer passed as the first argument.
          515  +**
          516  +** If nAppend is negative, then the length of the string zAppend is
          517  +** determined using strlen().
          518  +*/
  1030    519   static int fts3StringAppend(
  1031         -  StrBuffer *pStr, 
  1032         -  const char *zAppend, 
  1033         -  int nAppend
          520  +  StrBuffer *pStr,                /* Buffer to append to */
          521  +  const char *zAppend,            /* Pointer to data to append to buffer */
          522  +  int nAppend                     /* Size of zAppend in bytes (or -1) */
  1034    523   ){
  1035    524     if( nAppend<0 ){
  1036         -    nAppend = (int)strlen(zAppend);
          525  +    nAppend = strlen(zAppend);
  1037    526     }
  1038    527   
          528  +  /* If there is insufficient space allocated at StrBuffer.z, use realloc()
          529  +  ** to grow the buffer until so that it is big enough to accomadate the
          530  +  ** appended data.
          531  +  */
  1039    532     if( pStr->n+nAppend+1>=pStr->nAlloc ){
  1040    533       int nAlloc = pStr->nAlloc+nAppend+100;
  1041    534       char *zNew = sqlite3_realloc(pStr->z, nAlloc);
  1042    535       if( !zNew ){
  1043    536         return SQLITE_NOMEM;
  1044    537       }
  1045    538       pStr->z = zNew;
  1046    539       pStr->nAlloc = nAlloc;
  1047    540     }
  1048    541   
          542  +  /* Append the data to the string buffer. */
  1049    543     memcpy(&pStr->z[pStr->n], zAppend, nAppend);
  1050    544     pStr->n += nAppend;
  1051    545     pStr->z[pStr->n] = '\0';
  1052    546   
  1053    547     return SQLITE_OK;
  1054    548   }
  1055    549   
          550  +/*
          551  +** The fts3BestSnippet() function often selects snippets that end with a
          552  +** query term. That is, the final term of the snippet is always a term
          553  +** that requires highlighting. For example, if 'X' is a highlighted term
          554  +** and '.' is a non-highlighted term, BestSnippet() may select:
          555  +**
          556  +**     ........X.....X
          557  +**
          558  +** This function "shifts" the beginning of the snippet forward in the 
          559  +** document so that there are approximately the same number of 
          560  +** non-highlighted terms to the right of the final highlighted term as there
          561  +** are to the left of the first highlighted term. For example, to this:
          562  +**
          563  +**     ....X.....X....
          564  +**
          565  +** This is done as part of extracting the snippet text, not when selecting
          566  +** the snippet. Snippet selection is done based on doclists only, so there
          567  +** is no way for fts3BestSnippet() to know whether or not the document 
          568  +** actually contains terms that follow the final highlighted term. 
          569  +*/
          570  +int fts3SnippetShift(
          571  +  Fts3Table *pTab,                /* FTS3 table snippet comes from */
          572  +  int nSnippet,                   /* Number of tokens desired for snippet */
          573  +  const char *zDoc,               /* Document text to extract snippet from */
          574  +  int nDoc,                       /* Size of buffer zDoc in bytes */
          575  +  int *piPos,                     /* IN/OUT: First token of snippet */
          576  +  u64 *pHlmask                    /* IN/OUT: Mask of tokens to highlight */
          577  +){
          578  +  u64 hlmask = *pHlmask;          /* Local copy of initial highlight-mask */
          579  +
          580  +  if( hlmask ){
          581  +    int nLeft;                    /* Tokens to the left of first highlight */
          582  +    int nRight;                   /* Tokens to the right of last highlight */
          583  +    int nDesired;                 /* Ideal number of tokens to shift forward */
          584  +
          585  +    for(nLeft=0; !(hlmask & ((u64)1 << nLeft)); nLeft++);
          586  +    for(nRight=0; !(hlmask & ((u64)1 << (nSnippet-1-nRight))); nRight++);
          587  +    nDesired = (nLeft-nRight)/2;
          588  +
          589  +    /* Ideally, the start of the snippet should be pushed forward in the
          590  +    ** document nDesired tokens. This block checks if there are actually
          591  +    ** nDesired tokens to the right of the snippet. If so, *piPos and
          592  +    ** *pHlMask are updated to shift the snippet nDesired tokens to the
          593  +    ** right. Otherwise, the snippet is shifted by the number of tokens
          594  +    ** available.
          595  +    */
          596  +    if( nDesired>0 ){
          597  +      int nShift;                 /* Number of tokens to shift snippet by */
          598  +      int iCurrent = 0;           /* Token counter */
          599  +      int rc;                     /* Return Code */
          600  +      sqlite3_tokenizer_module *pMod;
          601  +      sqlite3_tokenizer_cursor *pC;
          602  +      pMod = (sqlite3_tokenizer_module *)pTab->pTokenizer->pModule;
          603  +
          604  +      /* Open a cursor on zDoc/nDoc. Check if there are (nSnippet+nDesired)
          605  +      ** or more tokens in zDoc/nDoc.
          606  +      */
          607  +      rc = pMod->xOpen(pTab->pTokenizer, zDoc, nDoc, &pC);
          608  +      if( rc!=SQLITE_OK ){
          609  +        return rc;
          610  +      }
          611  +      pC->pTokenizer = pTab->pTokenizer;
          612  +      while( rc==SQLITE_OK && iCurrent<(nSnippet+nDesired) ){
          613  +        const char *ZDUMMY; int DUMMY1, DUMMY2, DUMMY3;
          614  +        rc = pMod->xNext(pC, &ZDUMMY, &DUMMY1, &DUMMY2, &DUMMY3, &iCurrent);
          615  +      }
          616  +      pMod->xClose(pC);
          617  +      if( rc!=SQLITE_OK && rc!=SQLITE_DONE ){ return rc; }
          618  +
          619  +      nShift = (rc==SQLITE_DONE)+iCurrent-nSnippet;
          620  +      assert( nShift<=nDesired );
          621  +      if( nShift>0 ){
          622  +        *piPos += nShift;
          623  +        *pHlmask = hlmask >> nShift;
          624  +      }
          625  +    }
          626  +  }
          627  +  return SQLITE_OK;
          628  +}
          629  +
          630  +/*
          631  +** Extract the snippet text for fragment pFragment from cursor pCsr and
          632  +** append it to string buffer pOut.
          633  +*/
  1056    634   static int fts3SnippetText(
  1057    635     Fts3Cursor *pCsr,               /* FTS3 Cursor */
  1058         -  const char *zDoc,               /* Document to extract snippet from */
  1059         -  int nDoc,                       /* Size of zDoc in bytes */
          636  +  SnippetFragment *pFragment,     /* Snippet to extract */
          637  +  int iFragment,                  /* Fragment number */
          638  +  int isLast,                     /* True for final fragment in snippet */
  1060    639     int nSnippet,                   /* Number of tokens in extracted snippet */
  1061         -  int iPos,                       /* Index of first document token in snippet */
  1062         -  u64 hlmask,                     /* Bitmask of terms to highlight in snippet */
  1063    640     const char *zOpen,              /* String inserted before highlighted term */
  1064    641     const char *zClose,             /* String inserted after highlighted term */
  1065         -  const char *zEllipsis,
  1066         -  char **pzSnippet                /* OUT: Snippet text */
          642  +  const char *zEllipsis,          /* String inserted between snippets */
          643  +  StrBuffer *pOut                 /* Write output here */
  1067    644   ){
  1068    645     Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab;
  1069    646     int rc;                         /* Return code */
  1070         -  int iCurrent = 0;
  1071         -  int iStart = 0;
  1072         -  int iEnd;
  1073         -
          647  +  const char *zDoc;               /* Document text to extract snippet from */
          648  +  int nDoc;                       /* Size of zDoc in bytes */
          649  +  int iCurrent = 0;               /* Current token number of document */
          650  +  int iEnd = 0;                   /* Byte offset of end of current token */
          651  +  int isShiftDone = 0;            /* True after snippet is shifted */
          652  +  int iPos = pFragment->iPos;     /* First token of snippet */
          653  +  u64 hlmask = pFragment->hlmask; /* Highlight-mask for snippet */
          654  +  int iCol = pFragment->iCol+1;   /* Query column to extract text from */
  1074    655     sqlite3_tokenizer_module *pMod; /* Tokenizer module methods object */
  1075    656     sqlite3_tokenizer_cursor *pC;   /* Tokenizer cursor open on zDoc/nDoc */
  1076         -  const char *ZDUMMY;             /* Dummy arguments used with tokenizer */
  1077         -  int DUMMY1, DUMMY2, DUMMY3;     /* Dummy arguments used with tokenizer */
          657  +  const char *ZDUMMY;             /* Dummy argument used with tokenizer */
          658  +  int DUMMY1;                     /* Dummy argument used with tokenizer */
          659  +  
          660  +  zDoc = (const char *)sqlite3_column_text(pCsr->pStmt, iCol);
          661  +  if( zDoc==0 ){
          662  +    if( sqlite3_column_type(pCsr->pStmt, iCol)!=SQLITE_NULL ){
          663  +      return SQLITE_NOMEM;
          664  +    }
          665  +    return SQLITE_OK;
          666  +  }
          667  +  nDoc = sqlite3_column_bytes(pCsr->pStmt, iCol);
  1078    668   
  1079         -  StrBuffer res = {0, 0, 0};   /* Result string */
  1080         -
  1081         -  /* Open a token cursor on the document. Read all tokens up to and 
  1082         -  ** including token iPos (the first token of the snippet). Set variable
  1083         -  ** iStart to the byte offset in zDoc of the start of token iPos.
  1084         -  */
          669  +  /* Open a token cursor on the document. */
  1085    670     pMod = (sqlite3_tokenizer_module *)pTab->pTokenizer->pModule;
  1086    671     rc = pMod->xOpen(pTab->pTokenizer, zDoc, nDoc, &pC);
  1087         -  while( rc==SQLITE_OK && iCurrent<iPos ){
  1088         -    rc = pMod->xNext(pC, &ZDUMMY, &DUMMY1, &iStart, &DUMMY2, &iCurrent);
          672  +  if( rc!=SQLITE_OK ){
          673  +    return rc;
  1089    674     }
  1090         -  iEnd = iStart;
  1091         -
  1092         -  if( rc==SQLITE_OK && iStart>0 ){
  1093         -    rc = fts3StringAppend(&res, zEllipsis, -1);
  1094         -  }
          675  +  pC->pTokenizer = pTab->pTokenizer;
  1095    676   
  1096    677     while( rc==SQLITE_OK ){
  1097         -    int iBegin;
  1098         -    int iFin;
          678  +    int iBegin;                   /* Offset in zDoc of start of token */
          679  +    int iFin;                     /* Offset in zDoc of end of token */
          680  +    int isHighlight;              /* True for highlighted terms */
          681  +
  1099    682       rc = pMod->xNext(pC, &ZDUMMY, &DUMMY1, &iBegin, &iFin, &iCurrent);
          683  +    if( rc!=SQLITE_OK ){
          684  +      if( rc==SQLITE_DONE ){
          685  +        /* Special case - the last token of the snippet is also the last token
          686  +        ** of the column. Append any punctuation that occurred between the end
          687  +        ** of the previous token and the end of the document to the output. 
          688  +        ** Then break out of the loop. */
          689  +        rc = fts3StringAppend(pOut, &zDoc[iEnd], -1);
          690  +      }
          691  +      break;
          692  +    }
          693  +    if( iCurrent<iPos ){ continue; }
  1100    694   
  1101         -    if( rc==SQLITE_OK ){
  1102         -      if( iCurrent>=(iPos+nSnippet) ){
  1103         -        rc = SQLITE_DONE;
  1104         -      }else{
  1105         -        iEnd = iFin;
  1106         -        if( hlmask & ((u64)1 << (iCurrent-iPos)) ){
  1107         -          if( fts3StringAppend(&res, &zDoc[iStart], iBegin-iStart)
  1108         -           || fts3StringAppend(&res, zOpen, -1)
  1109         -           || fts3StringAppend(&res, &zDoc[iBegin], iEnd-iBegin)
  1110         -           || fts3StringAppend(&res, zClose, -1)
  1111         -          ){
  1112         -            rc = SQLITE_NOMEM;
  1113         -          }
  1114         -          iStart = iEnd;
  1115         -        }
          695  +    if( !isShiftDone ){
          696  +      int n = nDoc - iBegin;
          697  +      rc = fts3SnippetShift(pTab, nSnippet, &zDoc[iBegin], n, &iPos, &hlmask);
          698  +      isShiftDone = 1;
          699  +
          700  +      /* Now that the shift has been done, check if the initial "..." are
          701  +      ** required. They are required if (a) this is not the first fragment,
          702  +      ** or (b) this fragment does not begin at position 0 of its column. 
          703  +      */
          704  +      if( rc==SQLITE_OK && (iPos>0 || iFragment>0) ){
          705  +        rc = fts3StringAppend(pOut, zEllipsis, -1);
  1116    706         }
          707  +      if( rc!=SQLITE_OK || iCurrent<iPos ) continue;
  1117    708       }
  1118         -  }
  1119         -  assert( rc!=SQLITE_OK );
  1120         -  if( rc==SQLITE_DONE ){
  1121         -    rc = fts3StringAppend(&res, &zDoc[iStart], iEnd-iStart);
  1122         -    if( rc==SQLITE_OK ){
  1123         -      rc = pMod->xNext(pC, &ZDUMMY, &DUMMY1, &DUMMY2, &DUMMY3, &iCurrent);
  1124         -      if( rc==SQLITE_OK ){
  1125         -        rc = fts3StringAppend(&res, zEllipsis, -1);
  1126         -      }else if( rc==SQLITE_DONE ){
  1127         -        rc = fts3StringAppend(&res, &zDoc[iEnd], -1);
          709  +
          710  +    if( iCurrent>=(iPos+nSnippet) ){
          711  +      if( isLast ){
          712  +        rc = fts3StringAppend(pOut, zEllipsis, -1);
  1128    713         }
          714  +      break;
  1129    715       }
          716  +
          717  +    /* Set isHighlight to true if this term should be highlighted. */
          718  +    isHighlight = (hlmask & ((u64)1 << (iCurrent-iPos)))!=0;
          719  +
          720  +    if( iCurrent>iPos ) rc = fts3StringAppend(pOut, &zDoc[iEnd], iBegin-iEnd);
          721  +    if( rc==SQLITE_OK && isHighlight ) rc = fts3StringAppend(pOut, zOpen, -1);
          722  +    if( rc==SQLITE_OK ) rc = fts3StringAppend(pOut, &zDoc[iBegin], iFin-iBegin);
          723  +    if( rc==SQLITE_OK && isHighlight ) rc = fts3StringAppend(pOut, zClose, -1);
          724  +
          725  +    iEnd = iFin;
  1130    726     }
  1131    727   
  1132    728     pMod->xClose(pC);
  1133         -  if( rc!=SQLITE_OK ){
  1134         -    sqlite3_free(res.z);
  1135         -  }else{
  1136         -    *pzSnippet = res.z;
  1137         -  }
  1138    729     return rc;
  1139    730   }
  1140    731   
  1141    732   
  1142    733   /*
  1143         -** An instance of this structure is used to collect the 'global' part of
  1144         -** the matchinfo statistics. The 'global' part consists of the following:
  1145         -**
  1146         -**   1. The number of phrases in the query (nPhrase).
  1147         -**
  1148         -**   2. The number of columns in the FTS3 table (nCol).
  1149         -**
  1150         -**   3. A matrix of (nPhrase*nCol) integers containing the sum of the
  1151         -**      number of hits for each phrase in each column across all rows
  1152         -**      of the table.
          734  +** This function is used to count the entries in a column-list (a 
          735  +** delta-encoded list of term offsets within a single column of a single 
          736  +** row). When this function is called, *ppCollist should point to the
          737  +** beginning of the first varint in the column-list (the varint that
          738  +** contains the position of the first matching term in the column data).
          739  +** Before returning, *ppCollist is set to point to the first byte after
          740  +** the last varint in the column-list (either the 0x00 signifying the end
          741  +** of the position-list, or the 0x01 that precedes the column number of
          742  +** the next column in the position-list).
  1153    743   **
  1154         -** The total size of the global matchinfo array, assuming the number of
  1155         -** columns is N and the number of phrases is P is:
  1156         -**
  1157         -**   2 + P*(N+1)
  1158         -**
  1159         -** The number of hits for the 3rd phrase in the second column is found
  1160         -** using the expression:
  1161         -**
  1162         -**   aGlobal[2 + P*(1+2) + 1]
  1163         -*/
  1164         -typedef struct MatchInfo MatchInfo;
  1165         -struct MatchInfo {
  1166         -  Fts3Table *pTab;                /* FTS3 Table */
  1167         -  Fts3Cursor *pCursor;            /* FTS3 Cursor */
  1168         -  int iPhrase;                    /* Number of phrases so far */
  1169         -  int nCol;                       /* Number of columns in table */
  1170         -  u32 *aGlobal;                   /* Pre-allocated buffer */
  1171         -};
  1172         -
  1173         -/*
  1174         -** This function is used to count the entries in a column-list (delta-encoded
  1175         -** list of term offsets within a single column of a single row).
          744  +** The number of elements in the column-list is returned.
  1176    745   */
  1177    746   static int fts3ColumnlistCount(char **ppCollist){
  1178    747     char *pEnd = *ppCollist;
  1179    748     char c = 0;
  1180    749     int nEntry = 0;
  1181    750   
  1182    751     /* A column-list is terminated by either a 0x01 or 0x00. */
................................................................................
  1185    754       if( !c ) nEntry++;
  1186    755     }
  1187    756   
  1188    757     *ppCollist = pEnd;
  1189    758     return nEntry;
  1190    759   }
  1191    760   
  1192         -static void fts3LoadColumnlistCounts(char **pp, u32 *aOut){
          761  +static void fts3LoadColumnlistCounts(char **pp, u32 *aOut, int isGlobal){
  1193    762     char *pCsr = *pp;
  1194    763     while( *pCsr ){
          764  +    int nHit;
  1195    765       sqlite3_int64 iCol = 0;
  1196    766       if( *pCsr==0x01 ){
  1197    767         pCsr++;
  1198    768         pCsr += sqlite3Fts3GetVarint(pCsr, &iCol);
  1199    769       }
  1200         -    aOut[iCol] += fts3ColumnlistCount(&pCsr);
          770  +    nHit = fts3ColumnlistCount(&pCsr);
          771  +    assert( nHit>0 );
          772  +    if( isGlobal ){
          773  +      aOut[iCol*3+1]++;
          774  +    }
          775  +    aOut[iCol*3] += nHit;
  1201    776     }
  1202    777     pCsr++;
  1203    778     *pp = pCsr;
  1204    779   }
  1205    780   
  1206    781   /*
  1207    782   ** fts3ExprIterate() callback used to collect the "global" matchinfo stats
  1208         -** for a single query.
          783  +** for a single query. The "global" stats are those elements of the matchinfo
          784  +** array that are constant for all rows returned by the current query.
  1209    785   */
  1210    786   static int fts3ExprGlobalMatchinfoCb(
  1211    787     Fts3Expr *pExpr,                /* Phrase expression node */
          788  +  int iPhrase,                    /* Phrase number (numbered from zero) */
  1212    789     void *pCtx                      /* Pointer to MatchInfo structure */
  1213    790   ){
  1214    791     MatchInfo *p = (MatchInfo *)pCtx;
  1215    792     char *pCsr;
  1216    793     char *pEnd;
  1217         -  const int iStart = 2 + p->nCol*p->iPhrase;
          794  +  const int iStart = 2 + (iPhrase * p->nCol * 3) + 1;
  1218    795   
  1219    796     assert( pExpr->isLoaded );
  1220    797   
  1221    798     /* Fill in the global hit count matrix row for this phrase. */
  1222    799     pCsr = pExpr->aDoclist;
  1223    800     pEnd = &pExpr->aDoclist[pExpr->nDoclist];
  1224    801     while( pCsr<pEnd ){
  1225         -    while( *pCsr++ & 0x80 );
  1226         -    fts3LoadColumnlistCounts(&pCsr, &p->aGlobal[iStart]);
          802  +    while( *pCsr++ & 0x80 );      /* Skip past docid. */
          803  +    fts3LoadColumnlistCounts(&pCsr, &p->aMatchinfo[iStart], 1);
  1227    804     }
  1228    805   
  1229         -  p->iPhrase++;
  1230    806     return SQLITE_OK;
  1231    807   }
  1232    808   
          809  +/*
          810  +** fts3ExprIterate() callback used to collect the "local" matchinfo stats
          811  +** for a single query. The "local" stats are those elements of the matchinfo
          812  +** array that are different for each row returned by the query.
          813  +*/
  1233    814   static int fts3ExprLocalMatchinfoCb(
  1234    815     Fts3Expr *pExpr,                /* Phrase expression node */
          816  +  int iPhrase,                    /* Phrase number */
  1235    817     void *pCtx                      /* Pointer to MatchInfo structure */
  1236    818   ){
  1237    819     MatchInfo *p = (MatchInfo *)pCtx;
  1238         -  int iPhrase = p->iPhrase++;
  1239    820   
  1240    821     if( pExpr->aDoclist ){
  1241    822       char *pCsr;
  1242         -    int iOffset = 2 + p->nCol*(p->aGlobal[0]+iPhrase);
          823  +    int iStart = 2 + (iPhrase * p->nCol * 3);
          824  +    int i;
  1243    825   
  1244         -    memset(&p->aGlobal[iOffset], 0, p->nCol*sizeof(u32));
          826  +    for(i=0; i<p->nCol; i++) p->aMatchinfo[iStart+i*3] = 0;
          827  +
  1245    828       pCsr = sqlite3Fts3FindPositions(pExpr, p->pCursor->iPrevId, -1);
  1246         -    if( pCsr ) fts3LoadColumnlistCounts(&pCsr, &p->aGlobal[iOffset]);
          829  +    if( pCsr ){
          830  +      fts3LoadColumnlistCounts(&pCsr, &p->aMatchinfo[iStart], 0);
          831  +    }
  1247    832     }
  1248    833   
  1249    834     return SQLITE_OK;
  1250    835   }
  1251    836   
  1252    837   /*
  1253         -** Populate pCsr->aMatchinfo[] with data for the current row. The 'matchinfo'
  1254         -** data is an array of 32-bit unsigned integers (C type u32).
          838  +** Populate pCsr->aMatchinfo[] with data for the current row. The 
          839  +** 'matchinfo' data is an array of 32-bit unsigned integers (C type u32).
  1255    840   */
  1256    841   static int fts3GetMatchinfo(Fts3Cursor *pCsr){
  1257         -  MatchInfo g;
          842  +  MatchInfo sInfo;
  1258    843     Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab;
          844  +
          845  +  sInfo.pCursor = pCsr;
          846  +  sInfo.nCol = pTab->nColumn;
          847  +
  1259    848     if( pCsr->aMatchinfo==0 ){
  1260         -    int rc;
  1261         -    int nPhrase;
  1262         -    int nMatchinfo;
          849  +    /* If Fts3Cursor.aMatchinfo[] is NULL, then this is the first time the
          850  +    ** matchinfo function has been called for this query. In this case 
          851  +    ** allocate the array used to accumulate the matchinfo data and
          852  +    ** initialize those elements that are constant for every row.
          853  +    */
          854  +    int rc;                       /* Return Code */
          855  +    int nPhrase;                  /* Number of phrases */
          856  +    int nMatchinfo;               /* Number of u32 elements in match-info */
  1263    857   
  1264         -    g.pTab = pTab;
  1265         -    g.nCol = pTab->nColumn;
  1266         -    g.iPhrase = 0;
  1267         -    rc = fts3ExprLoadDoclists(pCsr, &nPhrase);
          858  +    /* Load doclists for each phrase in the query. */
          859  +    rc = fts3ExprLoadDoclists(pCsr, &nPhrase, 0);
  1268    860       if( rc!=SQLITE_OK ){
  1269    861         return rc;
  1270    862       }
          863  +    nMatchinfo = 2 + 3*sInfo.nCol*nPhrase;
  1271    864   
  1272         -    nMatchinfo = 2 + 2*g.nCol*nPhrase;
  1273         -
  1274         -    g.iPhrase = 0;
  1275         -    g.aGlobal = (u32 *)sqlite3_malloc(sizeof(u32)*nMatchinfo);
  1276         -    if( !g.aGlobal ){ 
          865  +    sInfo.aMatchinfo = (u32 *)sqlite3_malloc(sizeof(u32)*nMatchinfo);
          866  +    if( !sInfo.aMatchinfo ){ 
  1277    867         return SQLITE_NOMEM;
  1278    868       }
  1279         -    memset(g.aGlobal, 0, sizeof(u32)*nMatchinfo);
          869  +    memset(sInfo.aMatchinfo, 0, sizeof(u32)*nMatchinfo);
  1280    870   
  1281         -    g.aGlobal[0] = nPhrase;
  1282         -    g.aGlobal[1] = g.nCol;
  1283         -    (void)fts3ExprIterate(pCsr->pExpr, fts3ExprGlobalMatchinfoCb, (void *)&g);
  1284    871   
  1285         -    pCsr->aMatchinfo = g.aGlobal;
          872  +    /* First element of match-info is the number of phrases in the query */
          873  +    sInfo.aMatchinfo[0] = nPhrase;
          874  +    sInfo.aMatchinfo[1] = sInfo.nCol;
          875  +    (void)fts3ExprIterate(pCsr->pExpr, fts3ExprGlobalMatchinfoCb,(void*)&sInfo);
          876  +
          877  +    pCsr->aMatchinfo = sInfo.aMatchinfo;
  1286    878     }
  1287    879   
  1288         -  g.pTab = pTab;
  1289         -  g.pCursor = pCsr;
  1290         -  g.nCol = pTab->nColumn;
  1291         -  g.iPhrase = 0;
  1292         -  g.aGlobal = pCsr->aMatchinfo;
  1293         -
          880  +  sInfo.aMatchinfo = pCsr->aMatchinfo;
  1294    881     if( pCsr->isMatchinfoOk ){
  1295         -    (void)fts3ExprIterate(pCsr->pExpr, fts3ExprLocalMatchinfoCb, (void *)&g);
          882  +    (void)fts3ExprIterate(pCsr->pExpr, fts3ExprLocalMatchinfoCb, (void*)&sInfo);
  1296    883       pCsr->isMatchinfoOk = 0;
  1297    884     }
  1298    885   
  1299    886     return SQLITE_OK;
  1300    887   }
  1301    888   
  1302         -void sqlite3Fts3Snippet2(
          889  +/*
          890  +** Implementation of snippet() function.
          891  +*/
          892  +void sqlite3Fts3Snippet(
  1303    893     sqlite3_context *pCtx,          /* SQLite function call context */
  1304    894     Fts3Cursor *pCsr,               /* Cursor object */
  1305    895     const char *zStart,             /* Snippet start text - "<b>" */
  1306    896     const char *zEnd,               /* Snippet end text - "</b>" */
  1307    897     const char *zEllipsis,          /* Snippet ellipsis text - "<b>...</b>" */
  1308    898     int iCol,                       /* Extract snippet from this column */
  1309    899     int nToken                      /* Approximate number of tokens in snippet */
  1310    900   ){
  1311         -  int rc;
  1312         -  int iPos = 0;
  1313         -  u64 hlmask = 0;
  1314         -  char *z = 0;
  1315         -  int nDoc;
  1316         -  const char *zDoc;
  1317         -
  1318         -  rc = fts3BestSnippet(nToken, pCsr, iCol, &iPos, &hlmask);
  1319         -
  1320         -  nDoc = sqlite3_column_bytes(pCsr->pStmt, iCol+1);
  1321         -  zDoc = (const char *)sqlite3_column_text(pCsr->pStmt, iCol+1);
  1322         -
  1323         -  if( rc==SQLITE_OK ){
  1324         -    rc = fts3SnippetText(
  1325         -        pCsr, zDoc, nDoc, nToken, iPos, hlmask, zStart, zEnd, zEllipsis, &z);
          901  +  Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab;
          902  +  int rc = SQLITE_OK;
          903  +  int i;
          904  +  StrBuffer res = {0, 0, 0};
          905  +
          906  +  /* The returned text includes up to four fragments of text extracted from
          907  +  ** the data in the current row. The first iteration of the for(...) loop
          908  +  ** below attempts to locate a single fragment of text nToken tokens in 
          909  +  ** size that contains at least one instance of all phrases in the query
          910  +  ** expression that appear in the current row. If such a fragment of text
          911  +  ** cannot be found, the second iteration of the loop attempts to locate
          912  +  ** a pair of fragments, and so on.
          913  +  */
          914  +  int nSnippet = 0;               /* Number of fragments in this snippet */
          915  +  SnippetFragment aSnippet[4];    /* Maximum of 4 fragments per snippet */
          916  +  int nFToken = -1;               /* Number of tokens in each fragment */
          917  +
          918  +  for(nSnippet=1; 1; nSnippet++){
          919  +
          920  +    int iSnip;                    /* Loop counter 0..nSnippet-1 */
          921  +    u64 mCovered = 0;             /* Bitmask of phrases covered by snippet */
          922  +    u64 mSeen = 0;                /* Bitmask of phrases seen by BestSnippet() */
          923  +
          924  +    if( nToken>=0 ){
          925  +      nFToken = (nToken+nSnippet-1) / nSnippet;
          926  +    }else{
          927  +      nFToken = -1 * nToken;
          928  +    }
          929  +
          930  +    for(iSnip=0; iSnip<nSnippet; iSnip++){
          931  +      int iBestScore = -1;        /* Best score of columns checked so far */
          932  +      int iRead;                  /* Used to iterate through columns */
          933  +      SnippetFragment *pFragment = &aSnippet[iSnip];
          934  +
          935  +      memset(pFragment, 0, sizeof(*pFragment));
          936  +
          937  +      /* Loop through all columns of the table being considered for snippets.
          938  +      ** If the iCol argument to this function was negative, this means all
          939  +      ** columns of the FTS3 table. Otherwise, only column iCol is considered.
          940  +      */
          941  +      for(iRead=0; iRead<pTab->nColumn; iRead++){
          942  +        SnippetFragment sF;
          943  +        int iS;
          944  +        if( iCol>=0 && iRead!=iCol ) continue;
          945  +
          946  +        /* Find the best snippet of nFToken tokens in column iRead. */
          947  +        rc = fts3BestSnippet(nFToken, pCsr, iRead, mCovered, &mSeen, &sF, &iS);
          948  +        if( rc!=SQLITE_OK ){
          949  +          goto snippet_out;
          950  +        }
          951  +        if( iS>iBestScore ){
          952  +          *pFragment = sF;
          953  +          iBestScore = iS;
          954  +        }
          955  +      }
          956  +
          957  +      mCovered |= pFragment->covered;
          958  +    }
          959  +
          960  +    /* If all query phrases seen by fts3BestSnippet() are present in at least
          961  +    ** one of the nSnippet snippet fragments, break out of the loop.
          962  +    */
          963  +    assert( (mCovered&mSeen)==mCovered );
          964  +    if( mSeen==mCovered || nSnippet==SizeofArray(aSnippet) ) break;
          965  +  }
          966  +
          967  +  assert( nFToken>0 );
          968  +
          969  +  for(i=0; i<nSnippet && rc==SQLITE_OK; i++){
          970  +    rc = fts3SnippetText(pCsr, &aSnippet[i], 
          971  +        i, (i==nSnippet-1), nFToken, zStart, zEnd, zEllipsis, &res
          972  +    );
  1326    973     }
          974  +
          975  + snippet_out:
  1327    976     if( rc!=SQLITE_OK ){
  1328    977       sqlite3_result_error_code(pCtx, rc);
          978  +    sqlite3_free(res.z);
          979  +  }else{
          980  +    sqlite3_result_text(pCtx, res.z, -1, sqlite3_free);
          981  +  }
          982  +}
          983  +
          984  +
          985  +typedef struct TermOffset TermOffset;
          986  +typedef struct TermOffsetCtx TermOffsetCtx;
          987  +
          988  +struct TermOffset {
          989  +  char *pList;                    /* Position-list */
          990  +  int iPos;                       /* Position just read from pList */
          991  +  int iOff;                       /* Offset of this term from read positions */
          992  +};
          993  +
          994  +struct TermOffsetCtx {
          995  +  int iCol;                       /* Column of table to populate aTerm for */
          996  +  int iTerm;
          997  +  sqlite3_int64 iDocid;
          998  +  TermOffset *aTerm;
          999  +};
         1000  +
         1001  +/*
         1002  +** This function is an fts3ExprIterate() callback used by sqlite3Fts3Offsets().
         1003  +*/
         1004  +static int fts3ExprTermOffsetInit(Fts3Expr *pExpr, int iPhrase, void *ctx){
         1005  +  TermOffsetCtx *p = (TermOffsetCtx *)ctx;
         1006  +  int nTerm;                      /* Number of tokens in phrase */
         1007  +  int iTerm;                      /* For looping through nTerm phrase terms */
         1008  +  char *pList;                    /* Pointer to position list for phrase */
         1009  +  int iPos = 0;                   /* First position in position-list */
         1010  +
         1011  +  pList = sqlite3Fts3FindPositions(pExpr, p->iDocid, p->iCol);
         1012  +  nTerm = pExpr->pPhrase->nToken;
         1013  +  if( pList ){
         1014  +    fts3GetDeltaPosition(&pList, &iPos);
         1015  +    assert( iPos>=0 );
         1016  +  }
         1017  +
         1018  +  for(iTerm=0; iTerm<nTerm; iTerm++){
         1019  +    TermOffset *pT = &p->aTerm[p->iTerm++];
         1020  +    pT->iOff = nTerm-iTerm-1;
         1021  +    pT->pList = pList;
         1022  +    pT->iPos = iPos;
         1023  +  }
         1024  +
         1025  +  return SQLITE_OK;
         1026  +}
         1027  +
         1028  +/*
         1029  +** Implementation of offsets() function.
         1030  +*/
         1031  +void sqlite3Fts3Offsets(
         1032  +  sqlite3_context *pCtx,          /* SQLite function call context */
         1033  +  Fts3Cursor *pCsr                /* Cursor object */
         1034  +){
         1035  +  Fts3Table *pTab = (Fts3Table *)pCsr->base.pVtab;
         1036  +  sqlite3_tokenizer_module const *pMod = pTab->pTokenizer->pModule;
         1037  +  const char *ZDUMMY;             /* Dummy argument used with xNext() */
         1038  +  int NDUMMY;                     /* Dummy argument used with xNext() */
         1039  +  int rc;                         /* Return Code */
         1040  +  int nToken;                     /* Number of tokens in query */
         1041  +  int iCol;                       /* Column currently being processed */
         1042  +  StrBuffer res = {0, 0, 0};      /* Result string */
         1043  +  TermOffsetCtx sCtx;             /* Context for fts3ExprTermOffsetInit() */
         1044  +
         1045  +  memset(&sCtx, 0, sizeof(sCtx));
         1046  +  assert( pCsr->isRequireSeek==0 );
         1047  +
         1048  +  /* Count the number of terms in the query */
         1049  +  rc = fts3ExprLoadDoclists(pCsr, 0, &nToken);
         1050  +  if( rc!=SQLITE_OK ) goto offsets_out;
         1051  +
         1052  +  /* Allocate the array of TermOffset iterators. */
         1053  +  sCtx.aTerm = (TermOffset *)sqlite3_malloc(sizeof(TermOffset)*nToken);
         1054  +  if( 0==sCtx.aTerm ){
         1055  +    rc = SQLITE_NOMEM;
         1056  +    goto offsets_out;
         1057  +  }
         1058  +  sCtx.iDocid = pCsr->iPrevId;
         1059  +
         1060  +  /* Loop through the table columns, appending offset information to 
         1061  +  ** string-buffer res for each column.
         1062  +  */
         1063  +  for(iCol=0; iCol<pTab->nColumn; iCol++){
         1064  +    sqlite3_tokenizer_cursor *pC; /* Tokenizer cursor */
         1065  +    int iStart;
         1066  +    int iEnd;
         1067  +    int iCurrent;
         1068  +    const char *zDoc;
         1069  +    int nDoc;
         1070  +
         1071  +    /* Initialize the contents of sCtx.aTerm[] for column iCol. There is 
         1072  +    ** no way that this operation can fail, so the return code from
         1073  +    ** fts3ExprIterate() can be discarded.
         1074  +    */
         1075  +    sCtx.iCol = iCol;
         1076  +    sCtx.iTerm = 0;
         1077  +    (void)fts3ExprIterate(pCsr->pExpr, fts3ExprTermOffsetInit, (void *)&sCtx);
         1078  +
         1079  +    /* Retreive the text stored in column iCol. If an SQL NULL is stored 
         1080  +    ** in column iCol, jump immediately to the next iteration of the loop.
         1081  +    ** If an OOM occurs while retrieving the data (this can happen if SQLite
         1082  +    ** needs to transform the data from utf-16 to utf-8), return SQLITE_NOMEM 
         1083  +    ** to the caller. 
         1084  +    */
         1085  +    zDoc = (const char *)sqlite3_column_text(pCsr->pStmt, iCol+1);
         1086  +    nDoc = sqlite3_column_bytes(pCsr->pStmt, iCol+1);
         1087  +    if( zDoc==0 ){
         1088  +      if( sqlite3_column_type(pCsr->pStmt, iCol+1)==SQLITE_NULL ){
         1089  +        continue;
         1090  +      }
         1091  +      rc = SQLITE_NOMEM;
         1092  +      goto offsets_out;
         1093  +    }
         1094  +
         1095  +    /* Initialize a tokenizer iterator to iterate through column iCol. */
         1096  +    rc = pMod->xOpen(pTab->pTokenizer, zDoc, nDoc, &pC);
         1097  +    if( rc!=SQLITE_OK ) goto offsets_out;
         1098  +    pC->pTokenizer = pTab->pTokenizer;
         1099  +
         1100  +    rc = pMod->xNext(pC, &ZDUMMY, &NDUMMY, &iStart, &iEnd, &iCurrent);
         1101  +    while( rc==SQLITE_OK ){
         1102  +      int i;                      /* Used to loop through terms */
         1103  +      int iMinPos = 0x7FFFFFFF;   /* Position of next token */
         1104  +      TermOffset *pTerm = 0;      /* TermOffset associated with next token */
         1105  +
         1106  +      for(i=0; i<nToken; i++){
         1107  +        TermOffset *pT = &sCtx.aTerm[i];
         1108  +        if( pT->pList && (pT->iPos-pT->iOff)<iMinPos ){
         1109  +          iMinPos = pT->iPos-pT->iOff;
         1110  +          pTerm = pT;
         1111  +        }
         1112  +      }
         1113  +
         1114  +      if( !pTerm ){
         1115  +        /* All offsets for this column have been gathered. */
         1116  +        break;
         1117  +      }else{
         1118  +        assert( iCurrent<=iMinPos );
         1119  +        if( 0==(0xFE&*pTerm->pList) ){
         1120  +          pTerm->pList = 0;
         1121  +        }else{
         1122  +          fts3GetDeltaPosition(&pTerm->pList, &pTerm->iPos);
         1123  +        }
         1124  +        while( rc==SQLITE_OK && iCurrent<iMinPos ){
         1125  +          rc = pMod->xNext(pC, &ZDUMMY, &NDUMMY, &iStart, &iEnd, &iCurrent);
         1126  +        }
         1127  +        if( rc==SQLITE_OK ){
         1128  +          char aBuffer[64];
         1129  +          sqlite3_snprintf(sizeof(aBuffer), aBuffer, 
         1130  +              "%d %d %d %d ", iCol, pTerm-sCtx.aTerm, iStart, iEnd-iStart
         1131  +          );
         1132  +          rc = fts3StringAppend(&res, aBuffer, -1);
         1133  +        }
         1134  +      }
         1135  +    }
         1136  +    if( rc==SQLITE_DONE ){
         1137  +      rc = SQLITE_CORRUPT;
         1138  +    }
         1139  +
         1140  +    pMod->xClose(pC);
         1141  +    if( rc!=SQLITE_OK ) goto offsets_out;
         1142  +  }
         1143  +
         1144  + offsets_out:
         1145  +  sqlite3_free(sCtx.aTerm);
         1146  +  assert( rc!=SQLITE_DONE );
         1147  +  if( rc!=SQLITE_OK ){
         1148  +    sqlite3_result_error_code(pCtx,  rc);
         1149  +    sqlite3_free(res.z);
  1329   1150     }else{
  1330         -    sqlite3_result_text(pCtx, z, -1, sqlite3_free);
         1151  +    sqlite3_result_text(pCtx, res.z, res.n-1, sqlite3_free);
  1331   1152     }
         1153  +  return;
  1332   1154   }
  1333   1155   
         1156  +/*
         1157  +** Implementation of matchinfo() function.
         1158  +*/
  1334   1159   void sqlite3Fts3Matchinfo(sqlite3_context *pContext, Fts3Cursor *pCsr){
  1335   1160     int rc = fts3GetMatchinfo(pCsr);
  1336   1161     if( rc!=SQLITE_OK ){
  1337   1162       sqlite3_result_error_code(pContext, rc);
  1338   1163     }else{
  1339         -    int n = sizeof(u32)*(2+pCsr->aMatchinfo[0]*pCsr->aMatchinfo[1]*2);
         1164  +    int n = sizeof(u32)*(2+pCsr->aMatchinfo[0]*pCsr->aMatchinfo[1]*3);
  1340   1165       sqlite3_result_blob(pContext, pCsr->aMatchinfo, n, SQLITE_TRANSIENT);
  1341   1166     }
  1342   1167   }
  1343   1168   
  1344   1169   #endif

Changes to src/date.c.

  1088   1088       FUNCTION(datetime,         -1, 0, 0, datetimeFunc  ),
  1089   1089       FUNCTION(strftime,         -1, 0, 0, strftimeFunc  ),
  1090   1090       FUNCTION(current_time,      0, 0, 0, ctimeFunc     ),
  1091   1091       FUNCTION(current_timestamp, 0, 0, 0, ctimestampFunc),
  1092   1092       FUNCTION(current_date,      0, 0, 0, cdateFunc     ),
  1093   1093   #else
  1094   1094       STR_FUNCTION(current_time,      0, "%H:%M:%S",          0, currentTimeFunc),
  1095         -    STR_FUNCTION(current_timestamp, 0, "%Y-%m-%d",          0, currentTimeFunc),
  1096         -    STR_FUNCTION(current_date,      0, "%Y-%m-%d %H:%M:%S", 0, currentTimeFunc),
         1095  +    STR_FUNCTION(current_date,      0, "%Y-%m-%d",          0, currentTimeFunc),
         1096  +    STR_FUNCTION(current_timestamp, 0, "%Y-%m-%d %H:%M:%S", 0, currentTimeFunc),
  1097   1097   #endif
  1098   1098     };
  1099   1099     int i;
  1100   1100     FuncDefHash *pHash = &GLOBAL(FuncDefHash, sqlite3GlobalFunctions);
  1101   1101     FuncDef *aFunc = (FuncDef*)&GLOBAL(FuncDef, aDateTimeFuncs);
  1102   1102   
  1103   1103     for(i=0; i<ArraySize(aDateTimeFuncs); i++){
  1104   1104       sqlite3FuncDefInsert(pHash, &aFunc[i]);
  1105   1105     }
  1106   1106   }

Changes to src/expr.c.

  1878   1878     char *out = sqlite3DbMallocRaw(sqlite3VdbeDb(v), 8);
  1879   1879     if( out ){
  1880   1880       memcpy(out, in, 8);
  1881   1881     }
  1882   1882     return out;
  1883   1883   }
  1884   1884   
         1885  +#ifndef SQLITE_OMIT_FLOATING_POINT
  1885   1886   /*
  1886   1887   ** Generate an instruction that will put the floating point
  1887   1888   ** value described by z[0..n-1] into register iMem.
  1888   1889   **
  1889   1890   ** The z[] string will probably not be zero-terminated.  But the 
  1890   1891   ** z[n] character is guaranteed to be something that does not look
  1891   1892   ** like the continuation of the number.
................................................................................
  1897   1898       sqlite3AtoF(z, &value);
  1898   1899       assert( !sqlite3IsNaN(value) ); /* The new AtoF never returns NaN */
  1899   1900       if( negateFlag ) value = -value;
  1900   1901       zV = dup8bytes(v, (char*)&value);
  1901   1902       sqlite3VdbeAddOp4(v, OP_Real, 0, iMem, 0, zV, P4_REAL);
  1902   1903     }
  1903   1904   }
         1905  +#endif
  1904   1906   
  1905   1907   
  1906   1908   /*
  1907   1909   ** Generate an instruction that will put the integer describe by
  1908   1910   ** text z[0..n-1] into register iMem.
  1909   1911   **
  1910   1912   ** The z[] string will probably not be zero-terminated.  But the 
  1911   1913   ** z[n] character is guaranteed to be something that does not look
  1912   1914   ** like the continuation of the number.
  1913   1915   */
  1914         -static void codeInteger(Vdbe *v, Expr *pExpr, int negFlag, int iMem){
         1916  +static void codeInteger(Parse *pParse, Expr *pExpr, int negFlag, int iMem){
         1917  +  Vdbe *v = pParse->pVdbe;
  1915   1918     if( pExpr->flags & EP_IntValue ){
  1916   1919       int i = pExpr->u.iValue;
  1917   1920       if( negFlag ) i = -i;
  1918   1921       sqlite3VdbeAddOp2(v, OP_Integer, i, iMem);
  1919   1922     }else{
  1920   1923       const char *z = pExpr->u.zToken;
  1921   1924       assert( z!=0 );
................................................................................
  1923   1926         i64 value;
  1924   1927         char *zV;
  1925   1928         sqlite3Atoi64(z, &value);
  1926   1929         if( negFlag ) value = -value;
  1927   1930         zV = dup8bytes(v, (char*)&value);
  1928   1931         sqlite3VdbeAddOp4(v, OP_Int64, 0, iMem, 0, zV, P4_INT64);
  1929   1932       }else{
         1933  +#ifdef SQLITE_OMIT_FLOATING_POINT
         1934  +      sqlite3ErrorMsg(pParse, "oversized integer: %s%s", negFlag ? "-" : "", z);
         1935  +#else
  1930   1936         codeReal(v, z, negFlag, iMem);
         1937  +#endif
  1931   1938       }
  1932   1939     }
  1933   1940   }
  1934   1941   
  1935   1942   /*
  1936   1943   ** Clear a cache entry.
  1937   1944   */
................................................................................
  2310   2317         }else{
  2311   2318           inReg = sqlite3ExprCodeGetColumn(pParse, pExpr->pTab,
  2312   2319                                    pExpr->iColumn, pExpr->iTable, target);
  2313   2320         }
  2314   2321         break;
  2315   2322       }
  2316   2323       case TK_INTEGER: {
  2317         -      codeInteger(v, pExpr, 0, target);
         2324  +      codeInteger(pParse, pExpr, 0, target);
  2318   2325         break;
  2319   2326       }
         2327  +#ifndef SQLITE_OMIT_FLOATING_POINT
  2320   2328       case TK_FLOAT: {
  2321   2329         assert( !ExprHasProperty(pExpr, EP_IntValue) );
  2322   2330         codeReal(v, pExpr->u.zToken, 0, target);
  2323   2331         break;
  2324   2332       }
         2333  +#endif
  2325   2334       case TK_STRING: {
  2326   2335         assert( !ExprHasProperty(pExpr, EP_IntValue) );
  2327   2336         sqlite3VdbeAddOp4(v, OP_String8, 0, target, 0, pExpr->u.zToken, 0);
  2328   2337         break;
  2329   2338       }
  2330   2339       case TK_NULL: {
  2331   2340         sqlite3VdbeAddOp2(v, OP_Null, 0, target);
................................................................................
  2487   2496         testcase( regFree1==0 );
  2488   2497         testcase( regFree2==0 );
  2489   2498         break;
  2490   2499       }
  2491   2500       case TK_UMINUS: {
  2492   2501         Expr *pLeft = pExpr->pLeft;
  2493   2502         assert( pLeft );
  2494         -      if( pLeft->op==TK_FLOAT ){
         2503  +      if( pLeft->op==TK_INTEGER ){
         2504  +        codeInteger(pParse, pLeft, 1, target);
         2505  +#ifndef SQLITE_OMIT_FLOATING_POINT
         2506  +      }else if( pLeft->op==TK_FLOAT ){
  2495   2507           assert( !ExprHasProperty(pExpr, EP_IntValue) );
  2496   2508           codeReal(v, pLeft->u.zToken, 1, target);
  2497         -      }else if( pLeft->op==TK_INTEGER ){
  2498         -        codeInteger(v, pLeft, 1, target);
         2509  +#endif
  2499   2510         }else{
  2500   2511           regFree1 = r1 = sqlite3GetTempReg(pParse);
  2501   2512           sqlite3VdbeAddOp2(v, OP_Integer, 0, r1);
  2502   2513           r2 = sqlite3ExprCodeTemp(pParse, pExpr->pLeft, &regFree2);
  2503   2514           sqlite3VdbeAddOp3(v, OP_Subtract, r2, r1, target);
  2504   2515           testcase( regFree2==0 );
  2505   2516         }
................................................................................
  2739   2750         sqlite3VdbeAddOp2(v, OP_Param, p1, target);
  2740   2751         VdbeComment((v, "%s.%s -> $%d",
  2741   2752           (pExpr->iTable ? "new" : "old"),
  2742   2753           (pExpr->iColumn<0 ? "rowid" : pExpr->pTab->aCol[pExpr->iColumn].zName),
  2743   2754           target
  2744   2755         ));
  2745   2756   
         2757  +#ifndef SQLITE_OMIT_FLOATING_POINT
  2746   2758         /* If the column has REAL affinity, it may currently be stored as an
  2747   2759         ** integer. Use OP_RealAffinity to make sure it is really real.  */
  2748   2760         if( pExpr->iColumn>=0 
  2749   2761          && pTab->aCol[pExpr->iColumn].affinity==SQLITE_AFF_REAL
  2750   2762         ){
  2751   2763           sqlite3VdbeAddOp1(v, OP_RealAffinity, target);
  2752   2764         }
         2765  +#endif
  2753   2766         break;
  2754   2767       }
  2755   2768   
  2756   2769   
  2757   2770       /*
  2758   2771       ** Form A:
  2759   2772       **   CASE x WHEN e1 THEN r1 WHEN e2 THEN r2 ... WHEN eN THEN rN ELSE y END
................................................................................
  3411   3424       }
  3412   3425     }
  3413   3426     sqlite3ReleaseTempReg(pParse, regFree1);
  3414   3427     sqlite3ReleaseTempReg(pParse, regFree2);
  3415   3428   }
  3416   3429   
  3417   3430   /*
  3418         -** Do a deep comparison of two expression trees.  Return TRUE (non-zero)
  3419         -** if they are identical and return FALSE if they differ in any way.
         3431  +** Do a deep comparison of two expression trees.  Return 0 if the two
         3432  +** expressions are completely identical.  Return 1 if they differ only
         3433  +** by a COLLATE operator at the top level.  Return 2 if there are differences
         3434  +** other than the top-level COLLATE operator.
  3420   3435   **
  3421         -** Sometimes this routine will return FALSE even if the two expressions
         3436  +** Sometimes this routine will return 2 even if the two expressions
  3422   3437   ** really are equivalent.  If we cannot prove that the expressions are
  3423         -** identical, we return FALSE just to be safe.  So if this routine
  3424         -** returns false, then you do not really know for certain if the two
  3425         -** expressions are the same.  But if you get a TRUE return, then you
         3438  +** identical, we return 2 just to be safe.  So if this routine
         3439  +** returns 2, then you do not really know for certain if the two
         3440  +** expressions are the same.  But if you get a 0 or 1 return, then you
  3426   3441   ** can be sure the expressions are the same.  In the places where
  3427         -** this routine is used, it does not hurt to get an extra FALSE - that
         3442  +** this routine is used, it does not hurt to get an extra 2 - that
  3428   3443   ** just might result in some slightly slower code.  But returning
  3429         -** an incorrect TRUE could lead to a malfunction.
         3444  +** an incorrect 0 or 1 could lead to a malfunction.
  3430   3445   */
  3431   3446   int sqlite3ExprCompare(Expr *pA, Expr *pB){
  3432   3447     int i;
  3433   3448     if( pA==0||pB==0 ){
  3434         -    return pB==pA;
         3449  +    return pB==pA ? 0 : 2;
  3435   3450     }
  3436   3451     assert( !ExprHasAnyProperty(pA, EP_TokenOnly|EP_Reduced) );
  3437   3452     assert( !ExprHasAnyProperty(pB, EP_TokenOnly|EP_Reduced) );
  3438   3453     if( ExprHasProperty(pA, EP_xIsSelect) || ExprHasProperty(pB, EP_xIsSelect) ){
  3439         -    return 0;
         3454  +    return 2;
  3440   3455     }
  3441         -  if( (pA->flags & EP_Distinct)!=(pB->flags & EP_Distinct) ) return 0;
  3442         -  if( pA->op!=pB->op ) return 0;
  3443         -  if( !sqlite3ExprCompare(pA->pLeft, pB->pLeft) ) return 0;
  3444         -  if( !sqlite3ExprCompare(pA->pRight, pB->pRight) ) return 0;
         3456  +  if( (pA->flags & EP_Distinct)!=(pB->flags & EP_Distinct) ) return 2;
         3457  +  if( pA->op!=pB->op ) return 2;
         3458  +  if( sqlite3ExprCompare(pA->pLeft, pB->pLeft) ) return 2;
         3459  +  if( sqlite3ExprCompare(pA->pRight, pB->pRight) ) return 2;
  3445   3460   
  3446   3461     if( pA->x.pList && pB->x.pList ){
  3447         -    if( pA->x.pList->nExpr!=pB->x.pList->nExpr ) return 0;
         3462  +    if( pA->x.pList->nExpr!=pB->x.pList->nExpr ) return 2;
  3448   3463       for(i=0; i<pA->x.pList->nExpr; i++){
  3449   3464         Expr *pExprA = pA->x.pList->a[i].pExpr;
  3450   3465         Expr *pExprB = pB->x.pList->a[i].pExpr;
  3451         -      if( !sqlite3ExprCompare(pExprA, pExprB) ) return 0;
         3466  +      if( sqlite3ExprCompare(pExprA, pExprB) ) return 2;
  3452   3467       }
  3453   3468     }else if( pA->x.pList || pB->x.pList ){
  3454         -    return 0;
         3469  +    return 2;
  3455   3470     }
  3456   3471   
  3457         -  if( pA->iTable!=pB->iTable || pA->iColumn!=pB->iColumn ) return 0;
         3472  +  if( pA->iTable!=pB->iTable || pA->iColumn!=pB->iColumn ) return 2;
  3458   3473     if( ExprHasProperty(pA, EP_IntValue) ){
  3459   3474       if( !ExprHasProperty(pB, EP_IntValue) || pA->u.iValue!=pB->u.iValue ){
  3460         -      return 0;
         3475  +      return 2;
  3461   3476       }
  3462   3477     }else if( pA->op!=TK_COLUMN && pA->u.zToken ){
  3463         -    if( ExprHasProperty(pB, EP_IntValue) || NEVER(pB->u.zToken==0) ) return 0;
         3478  +    if( ExprHasProperty(pB, EP_IntValue) || NEVER(pB->u.zToken==0) ) return 2;
  3464   3479       if( sqlite3StrICmp(pA->u.zToken,pB->u.zToken)!=0 ){
  3465         -      return 0;
         3480  +      return 2;
  3466   3481       }
  3467   3482     }
  3468         -  return 1;
         3483  +  if( (pA->flags & EP_ExpCollate)!=(pB->flags & EP_ExpCollate) ) return 1;
         3484  +  if( (pA->flags & EP_ExpCollate)!=0 && pA->pColl!=pB->pColl ) return 2;
         3485  +  return 0;
  3469   3486   }
  3470   3487   
  3471   3488   
  3472   3489   /*
  3473   3490   ** Add a new element to the pAggInfo->aCol[] array.  Return the index of
  3474   3491   ** the new element.  Return a negative number if malloc fails.
  3475   3492   */
................................................................................
  3592   3609         ** to be ignored */
  3593   3610         if( pNC->nDepth==0 ){
  3594   3611           /* Check to see if pExpr is a duplicate of another aggregate 
  3595   3612           ** function that is already in the pAggInfo structure
  3596   3613           */
  3597   3614           struct AggInfo_func *pItem = pAggInfo->aFunc;
  3598   3615           for(i=0; i<pAggInfo->nFunc; i++, pItem++){
  3599         -          if( sqlite3ExprCompare(pItem->pExpr, pExpr) ){
         3616  +          if( sqlite3ExprCompare(pItem->pExpr, pExpr)==0 ){
  3600   3617               break;
  3601   3618             }
  3602   3619           }
  3603   3620           if( i>=pAggInfo->nFunc ){
  3604   3621             /* pExpr is original.  Make a new entry in pAggInfo->aFunc[]
  3605   3622             */
  3606   3623             u8 enc = ENC(pParse->db);

Changes to src/func.c.

   439    439   static void last_insert_rowid(
   440    440     sqlite3_context *context, 
   441    441     int NotUsed, 
   442    442     sqlite3_value **NotUsed2
   443    443   ){
   444    444     sqlite3 *db = sqlite3_context_db_handle(context);
   445    445     UNUSED_PARAMETER2(NotUsed, NotUsed2);
          446  +  /* IMP: R-51513-12026 The last_insert_rowid() SQL function is a
          447  +  ** wrapper around the sqlite3_last_insert_rowid() C/C++ interface
          448  +  ** function. */
   446    449     sqlite3_result_int64(context, sqlite3_last_insert_rowid(db));
   447    450   }
   448    451   
   449    452   /*
   450         -** Implementation of the changes() SQL function.  The return value is the
   451         -** same as the sqlite3_changes() API function.
          453  +** Implementation of the changes() SQL function.
          454  +**
          455  +** IMP: R-62073-11209 The changes() SQL function is a wrapper
          456  +** around the sqlite3_changes() C/C++ function and hence follows the same
          457  +** rules for counting changes.
   452    458   */
   453    459   static void changes(
   454    460     sqlite3_context *context,
   455    461     int NotUsed,
   456    462     sqlite3_value **NotUsed2
   457    463   ){
   458    464     sqlite3 *db = sqlite3_context_db_handle(context);
................................................................................
   467    473   static void total_changes(
   468    474     sqlite3_context *context,
   469    475     int NotUsed,
   470    476     sqlite3_value **NotUsed2
   471    477   ){
   472    478     sqlite3 *db = sqlite3_context_db_handle(context);
   473    479     UNUSED_PARAMETER2(NotUsed, NotUsed2);
          480  +  /* IMP: R-52756-41993 This function is a wrapper around the
          481  +  ** sqlite3_total_changes() C/C++ interface. */
   474    482     sqlite3_result_int(context, sqlite3_total_changes(db));
   475    483   }
   476    484   
   477    485   /*
   478    486   ** A structure defining how to do GLOB-style comparisons.
   479    487   */
   480    488   struct compareInfo {
................................................................................
   734    742   */
   735    743   static void versionFunc(
   736    744     sqlite3_context *context,
   737    745     int NotUsed,
   738    746     sqlite3_value **NotUsed2
   739    747   ){
   740    748     UNUSED_PARAMETER2(NotUsed, NotUsed2);
   741         -  sqlite3_result_text(context, sqlite3_version, -1, SQLITE_STATIC);
          749  +  /* IMP: R-48699-48617 This function is an SQL wrapper around the
          750  +  ** sqlite3_libversion() C-interface. */
          751  +  sqlite3_result_text(context, sqlite3_libversion(), -1, SQLITE_STATIC);
   742    752   }
   743    753   
   744    754   /*
   745    755   ** Implementation of the sqlite_source_id() function. The result is a string
   746    756   ** that identifies the particular version of the source code used to build
   747    757   ** SQLite.
   748    758   */
   749    759   static void sourceidFunc(
   750    760     sqlite3_context *context,
   751    761     int NotUsed,
   752    762     sqlite3_value **NotUsed2
   753    763   ){
   754    764     UNUSED_PARAMETER2(NotUsed, NotUsed2);
   755         -  sqlite3_result_text(context, SQLITE_SOURCE_ID, -1, SQLITE_STATIC);
          765  +  /* IMP: R-24470-31136 This function is an SQL wrapper around the
          766  +  ** sqlite3_sourceid() C interface. */
          767  +  sqlite3_result_text(context, sqlite3_sourceid(), -1, SQLITE_STATIC);
   756    768   }
   757    769   
   758    770   /* Array for converting from half-bytes (nybbles) into ASCII hex
   759    771   ** digits. */
   760    772   static const char hexdigits[] = {
   761    773     '0', '1', '2', '3', '4', '5', '6', '7',
   762    774     '8', '9', 'A', 'B', 'C', 'D', 'E', 'F' 
................................................................................
   877    889     UNUSED_PARAMETER(argc);
   878    890     n = sqlite3_value_int64(argv[0]);
   879    891     testcase( n==db->aLimit[SQLITE_LIMIT_LENGTH] );
   880    892     testcase( n==db->aLimit[SQLITE_LIMIT_LENGTH]+1 );
   881    893     if( n>db->aLimit[SQLITE_LIMIT_LENGTH] ){
   882    894       sqlite3_result_error_toobig(context);
   883    895     }else{
   884         -    sqlite3_result_zeroblob(context, (int)n);
          896  +    sqlite3_result_zeroblob(context, (int)n); /* IMP: R-00293-64994 */
   885    897     }
   886    898   }
   887    899   
   888    900   /*
   889    901   ** The replace() function.  Three arguments are all strings: call
   890    902   ** them A, B, and C. The result is also a string which is derived
   891    903   ** from A by replacing every occurance of B with C.  The match

Changes to src/insert.c.

  1711   1711         if( xferCompatibleIndex(pDestIdx, pSrcIdx) ) break;
  1712   1712       }
  1713   1713       if( pSrcIdx==0 ){
  1714   1714         return 0;    /* pDestIdx has no corresponding index in pSrc */
  1715   1715       }
  1716   1716     }
  1717   1717   #ifndef SQLITE_OMIT_CHECK
  1718         -  if( pDest->pCheck && !sqlite3ExprCompare(pSrc->pCheck, pDest->pCheck) ){
         1718  +  if( pDest->pCheck && sqlite3ExprCompare(pSrc->pCheck, pDest->pCheck) ){
  1719   1719       return 0;   /* Tables have different CHECK constraints.  Ticket #2252 */
  1720   1720     }
  1721   1721   #endif
  1722   1722   
  1723   1723     /* If we get this far, it means either:
  1724   1724     **
  1725   1725     **    *   We can always do the transfer if the table contains an

Changes to src/legacy.c.

    40     40     int rc = SQLITE_OK;         /* Return code */
    41     41     const char *zLeftover;      /* Tail of unprocessed SQL */
    42     42     sqlite3_stmt *pStmt = 0;    /* The current SQL statement */
    43     43     char **azCols = 0;          /* Names of result columns */
    44     44     int nRetry = 0;             /* Number of retry attempts */
    45     45     int callbackIsInit;         /* True if callback data is initialized */
    46     46   
    47         -  if (!sqlite3SafetyCheckOk(db)) {
    48         -    return SQLITE_MISUSE;
    49         -  }
    50         -  	
           47  +  if( !sqlite3SafetyCheckOk(db) ) return SQLITE_MISUSE;
    51     48     if( zSql==0 ) zSql = "";
    52     49   #ifdef SQLITE_ENABLE_SQLRR
    53     50     SRRecExec(db, zSql);
    54     51   #endif  
    55     52     sqlite3_mutex_enter(db->mutex);
    56     53     sqlite3Error(db, SQLITE_OK, 0);
    57     54     while( (rc==SQLITE_OK || (rc==SQLITE_SCHEMA && (++nRetry)<2)) && zSql[0] ){

Changes to src/prepare.c.

   314    314   
   315    315     /* Read the schema information out of the schema tables
   316    316     */
   317    317     assert( db->init.busy );
   318    318     {
   319    319       char *zSql;
   320    320       zSql = sqlite3MPrintf(db, 
   321         -        "SELECT name, rootpage, sql FROM '%q'.%s",
          321  +        "SELECT name, rootpage, sql FROM '%q'.%s ORDER BY rowid",
   322    322           db->aDb[iDb].zName, zMasterName);
   323    323       (void)sqlite3SafetyOff(db);
   324    324   #ifndef SQLITE_OMIT_AUTHORIZATION
   325    325       {
   326    326         int (*xAuth)(void*,int,const char*,const char*,const char*,const char*);
   327    327         xAuth = db->xAuth;
   328    328         db->xAuth = 0;

Changes to src/printf.c.

   602    602             for(i=width; i>=nPad; i--){
   603    603               bufpt[i] = bufpt[i-nPad];
   604    604             }
   605    605             i = prefix!=0;
   606    606             while( nPad-- ) bufpt[i++] = '0';
   607    607             length = width;
   608    608           }
   609         -#endif
          609  +#else
          610  +        length = 0;
          611  +#endif /* SQLITE_OMIT_FLOATING_POINT */
   610    612           break;
   611    613         case etSIZE:
   612    614           *(va_arg(ap,int*)) = pAccum->nChar;
   613    615           length = width = 0;
   614    616           break;
   615    617         case etPERCENT:
   616    618           buf[0] = '%';

Changes to src/resolve.c.

   682    682     }
   683    683   
   684    684     /* Try to match the ORDER BY expression against an expression
   685    685     ** in the result set.  Return an 1-based index of the matching
   686    686     ** result-set entry.
   687    687     */
   688    688     for(i=0; i<pEList->nExpr; i++){
   689         -    if( sqlite3ExprCompare(pEList->a[i].pExpr, pE) ){
          689  +    if( sqlite3ExprCompare(pEList->a[i].pExpr, pE)<2 ){
   690    690         return i+1;
   691    691       }
   692    692     }
   693    693   
   694    694     /* If no match, return 0. */
   695    695     return 0;
   696    696   }

Changes to src/shell.c.

  1497   1497   ** This is the callback routine that the shell
  1498   1498   ** invokes for each row of a query result.
  1499   1499   */
  1500   1500   static int shell_callback(void *pArg, int nArg, char **azArg, char **azCol, int *aiType){
  1501   1501     int i;
  1502   1502     struct callback_data *p = (struct callback_data*)pArg;
  1503   1503   
  1504         -  if( p->echoOn && p->cnt==0  && p->pStmt){
  1505         -    printf("%s\n", sqlite3_sql(p->pStmt));
  1506         -  }
  1507         -
  1508   1504     switch( p->mode ){
  1509   1505       case MODE_Line: {
  1510   1506         int w = 5;
  1511   1507         if( azArg==0 ) break;
  1512   1508         for(i=0; i<nArg; i++){
  1513   1509           int len = strlen30(azCol[i] ? azCol[i] : "");
  1514   1510           if( len>w ) w = len;
................................................................................
  1850   1846       }else{
  1851   1847         if( !pStmt ){
  1852   1848           /* this happens for a comment or white-space */
  1853   1849           zSql = zLeftover;
  1854   1850           while( isspace(zSql[0]) ) zSql++;
  1855   1851           continue;
  1856   1852         }
         1853  +
         1854  +      /* echo the sql statement if echo on */
         1855  +      if( pArg->echoOn ){
         1856  +        char *zStmtSql = sqlite3_sql(pStmt);
         1857  +        fprintf(pArg->out,"%s\n", zStmtSql ? zStmtSql : zSql);
         1858  +      }
  1857   1859   
  1858   1860         /* perform the first step.  this will tell us if we
  1859   1861         ** have a result set or not and how wide it is.
  1860   1862         */
  1861   1863         rc = sqlite3_step(pStmt);
  1862   1864         /* if we have a result set... */
  1863   1865         if( SQLITE_ROW == rc ){

Changes to src/sqliteInt.h.

    72     72   #ifdef HAVE_STDINT_H
    73     73   #include <stdint.h>
    74     74   #endif
    75     75   #ifdef HAVE_INTTYPES_H
    76     76   #include <inttypes.h>
    77     77   #endif
    78     78   
           79  +/*
           80  +** The number of samples of an index that SQLite takes in order to 
           81  +** construct a histogram of the table content when running ANALYZE
           82  +** and with SQLITE_ENABLE_STAT2
           83  +*/
    79     84   #define SQLITE_INDEX_SAMPLES 10
    80     85   
    81     86   /*
    82     87   ** This macro is used to "hide" some ugliness in casting an int
    83     88   ** value to a ptr value under the MSVC 64-bit compiler.   Casting
    84     89   ** non 64-bit values to ptr types results in a "hard" error with 
    85     90   ** the MSVC 64-bit compiler which this attempts to avoid.  
................................................................................
   136    141   
   137    142   /*
   138    143   ** Exactly one of the following macros must be defined in order to
   139    144   ** specify which memory allocation subsystem to use.
   140    145   **
   141    146   **     SQLITE_SYSTEM_MALLOC          // Use normal system malloc()
   142    147   **     SQLITE_MEMDEBUG               // Debugging version of system malloc()
   143         -**     SQLITE_MEMORY_SIZE            // internal allocator #1
   144         -**     SQLITE_MMAP_HEAP_SIZE         // internal mmap() allocator
   145         -**     SQLITE_POW2_MEMORY_SIZE       // internal power-of-two allocator
          148  +**
          149  +** (Historical note:  There used to be several other options, but we've
          150  +** pared it down to just these two.)
   146    151   **
   147    152   ** If none of the above are defined, then set SQLITE_SYSTEM_MALLOC as
   148    153   ** the default.
   149    154   */
   150         -#if defined(SQLITE_SYSTEM_MALLOC)+defined(SQLITE_MEMDEBUG)+\
   151         -    defined(SQLITE_MEMORY_SIZE)+defined(SQLITE_MMAP_HEAP_SIZE)+\
   152         -    defined(SQLITE_POW2_MEMORY_SIZE)>1
          155  +#if defined(SQLITE_SYSTEM_MALLOC)+defined(SQLITE_MEMDEBUG)>1
   153    156   # error "At most one of the following compile-time configuration options\
   154         - is allows: SQLITE_SYSTEM_MALLOC, SQLITE_MEMDEBUG, SQLITE_MEMORY_SIZE,\
   155         - SQLITE_MMAP_HEAP_SIZE, SQLITE_POW2_MEMORY_SIZE"
          157  + is allows: SQLITE_SYSTEM_MALLOC, SQLITE_MEMDEBUG"
   156    158   #endif
   157         -#if defined(SQLITE_SYSTEM_MALLOC)+defined(SQLITE_MEMDEBUG)+\
   158         -    defined(SQLITE_MEMORY_SIZE)+defined(SQLITE_MMAP_HEAP_SIZE)+\
   159         -    defined(SQLITE_POW2_MEMORY_SIZE)==0
          159  +#if defined(SQLITE_SYSTEM_MALLOC)+defined(SQLITE_MEMDEBUG)==0
   160    160   # define SQLITE_SYSTEM_MALLOC 1
   161    161   #endif
   162    162   
   163    163   /*
   164    164   ** If SQLITE_MALLOC_SOFT_LIMIT is not zero, then try to keep the
   165    165   ** sizes of memory allocations below this value where possible.
   166    166   */
................................................................................
   324    324   */
   325    325   #ifdef SQLITE_OMIT_TEMPDB
   326    326   #define OMIT_TEMPDB 1
   327    327   #else
   328    328   #define OMIT_TEMPDB 0
   329    329   #endif
   330    330   
   331         -/*
   332         -** If the following macro is set to 1, then NULL values are considered
   333         -** distinct when determining whether or not two entries are the same
   334         -** in a UNIQUE index.  This is the way PostgreSQL, Oracle, DB2, MySQL,
   335         -** OCELOT, and Firebird all work.  The SQL92 spec explicitly says this
   336         -** is the way things are suppose to work.
   337         -**
   338         -** If the following macro is set to 0, the NULLs are indistinct for
   339         -** a UNIQUE index.  In this mode, you can only have a single NULL entry
   340         -** for a column declared UNIQUE.  This is the way Informix and SQL Server
   341         -** work.
   342         -*/
   343         -#define NULL_DISTINCT_FOR_UNIQUE 1
   344         -
   345    331   /*
   346    332   ** The "file format" number is an integer that is incremented whenever
   347    333   ** the VDBE-level file format changes.  The following macros define the
   348    334   ** the default file format for new databases and the maximum file format
   349    335   ** that the library can read.
   350    336   */
   351    337   #define SQLITE_MAX_FILE_FORMAT 4
   352    338   #ifndef SQLITE_DEFAULT_FILE_FORMAT
   353    339   # define SQLITE_DEFAULT_FILE_FORMAT 1
   354    340   #endif
   355    341   
          342  +/*
          343  +** Determine whether triggers are recursive by default.  This can be
          344  +** changed at run-time using a pragma.
          345  +*/
   356    346   #ifndef SQLITE_DEFAULT_RECURSIVE_TRIGGERS
   357    347   # define SQLITE_DEFAULT_RECURSIVE_TRIGGERS 0
   358    348   #endif
   359    349   
   360    350   /*
   361    351   ** Provide a default value for SQLITE_TEMP_STORE in case it is not specified
   362    352   ** on the command-line
................................................................................
   593    583   /*
   594    584   ** Forward references to structures
   595    585   */
   596    586   typedef struct AggInfo AggInfo;
   597    587   typedef struct AuthContext AuthContext;
   598    588   typedef struct AutoincInfo AutoincInfo;
   599    589   typedef struct Bitvec Bitvec;
   600         -typedef struct RowSet RowSet;
   601    590   typedef struct CollSeq CollSeq;
   602    591   typedef struct Column Column;
   603    592   typedef struct Db Db;
   604    593   typedef struct Schema Schema;
   605    594   typedef struct Expr Expr;
   606    595   typedef struct ExprList ExprList;
   607    596   typedef struct ExprSpan ExprSpan;
................................................................................
   614    603   typedef struct KeyClass KeyClass;
   615    604   typedef struct KeyInfo KeyInfo;
   616    605   typedef struct Lookaside Lookaside;
   617    606   typedef struct LookasideSlot LookasideSlot;
   618    607   typedef struct Module Module;
   619    608   typedef struct NameContext NameContext;
   620    609   typedef struct Parse Parse;
          610  +typedef struct RowSet RowSet;
   621    611   typedef struct Savepoint Savepoint;
   622    612   typedef struct Select Select;
   623    613   typedef struct SrcList SrcList;
   624    614   typedef struct StrAccum StrAccum;
   625    615   typedef struct Table Table;
   626    616   typedef struct TableLock TableLock;
   627    617   typedef struct Token Token;
          618  +typedef struct Trigger Trigger;
   628    619   typedef struct TriggerPrg TriggerPrg;
   629    620   typedef struct TriggerStep TriggerStep;
   630         -typedef struct Trigger Trigger;
   631    621   typedef struct UnpackedRecord UnpackedRecord;
   632    622   typedef struct VTable VTable;
   633    623   typedef struct Walker Walker;
   634    624   typedef struct WherePlan WherePlan;
   635    625   typedef struct WhereInfo WhereInfo;
   636    626   typedef struct WhereLevel WhereLevel;
   637    627   
................................................................................
   689    679   #ifndef SQLITE_OMIT_VIRTUALTABLE
   690    680     sqlite3 *db;         /* "Owner" connection. See comment above */
   691    681   #endif
   692    682   };
   693    683   
   694    684   /*
   695    685   ** These macros can be used to test, set, or clear bits in the 
   696         -** Db.flags field.
          686  +** Db.pSchema->flags field.
   697    687   */
   698    688   #define DbHasProperty(D,I,P)     (((D)->aDb[I].pSchema->flags&(P))==(P))
   699    689   #define DbHasAnyProperty(D,I,P)  (((D)->aDb[I].pSchema->flags&(P))!=0)
   700    690   #define DbSetProperty(D,I,P)     (D)->aDb[I].pSchema->flags|=(P)
   701    691   #define DbClearProperty(D,I,P)   (D)->aDb[I].pSchema->flags&=~(P)
   702    692   
   703    693   /*
   704         -** Allowed values for the DB.flags field.
          694  +** Allowed values for the DB.pSchema->flags field.
   705    695   **
   706    696   ** The DB_SchemaLoaded flag is set after the database schema has been
   707    697   ** read into internal hash tables.
   708    698   **
   709    699   ** DB_UnresetViews means that one or more views have column names that
   710    700   ** have been filled out.  If the schema changes, these column names might
   711    701   ** changes and so the view will need to be reset.
................................................................................
   761    751   ** Collisions are on the FuncDef.pHash chain.
   762    752   */
   763    753   struct FuncDefHash {
   764    754     FuncDef *a[23];       /* Hash table for functions */
   765    755   };
   766    756   
   767    757   /*
   768         -** Each database is an instance of the following structure.
          758  +** Each database connection is an instance of the following structure.
   769    759   **
   770    760   ** The sqlite.lastRowid records the last insert rowid generated by an
   771    761   ** insert statement.  Inserts on views do not affect its value.  Each
   772    762   ** trigger has its own context, so that lastRowid can be updated inside
   773    763   ** triggers as usual.  The previous value will be restored once the trigger
   774    764   ** exits.  Upon entering a before or instead of trigger, lastRowid is no
   775    765   ** longer (since after version 2.8.12) reset to -1.
................................................................................
  2528   2518     int sqlite3MutexEnd(void);
  2529   2519   #endif
  2530   2520   
  2531   2521   int sqlite3StatusValue(int);
  2532   2522   void sqlite3StatusAdd(int, int);
  2533   2523   void sqlite3StatusSet(int, int);
  2534   2524   
  2535         -int sqlite3IsNaN(double);
         2525  +#ifndef SQLITE_OMIT_FLOATING_POINT
         2526  +  int sqlite3IsNaN(double);
         2527  +#else
         2528  +# define sqlite3IsNaN(X)  0
         2529  +#endif
  2536   2530   
  2537   2531   void sqlite3VXPrintf(StrAccum*, int, const char*, va_list);
  2538   2532   #ifndef SQLITE_OMIT_TRACE
  2539   2533   void sqlite3XPrintf(StrAccum*, const char*, ...);
  2540   2534   #endif
  2541   2535   char *sqlite3MPrintf(sqlite3*,const char*, ...);
  2542   2536   char *sqlite3VMPrintf(sqlite3*,const char*, va_list);

Changes to src/test3.c.

   215    215     Tcl_Interp *interp,    /* The TCL interpreter that invoked this command */
   216    216     int argc,              /* Number of arguments */
   217    217     const char **argv      /* Text of each argument */
   218    218   ){
   219    219     Btree *pBt;
   220    220     int iTable;
   221    221     BtCursor *pCur;
   222         -  int rc;
          222  +  int rc = SQLITE_OK;
   223    223     int wrFlag;
   224    224     char zBuf[30];
   225    225   
   226    226     if( argc!=4 ){
   227    227       Tcl_AppendResult(interp, "wrong # args: should be \"", argv[0],
   228    228          " ID TABLENUM WRITEABLE\"", 0);
   229    229       return TCL_ERROR;
................................................................................
   230    230     }
   231    231     pBt = sqlite3TestTextToPtr(argv[1]);
   232    232     if( Tcl_GetInt(interp, argv[2], &iTable) ) return TCL_ERROR;
   233    233     if( Tcl_GetBoolean(interp, argv[3], &wrFlag) ) return TCL_ERROR;
   234    234     pCur = (BtCursor *)ckalloc(sqlite3BtreeCursorSize());
   235    235     memset(pCur, 0, sqlite3BtreeCursorSize());
   236    236     sqlite3BtreeEnter(pBt);
          237  +#ifndef SQLITE_OMIT_SHARED_CACHE
   237    238     rc = sqlite3BtreeLockTable(pBt, iTable, wrFlag);
          239  +#endif
   238    240     if( rc==SQLITE_OK ){
   239    241       rc = sqlite3BtreeCursor(pBt, iTable, wrFlag, 0, pCur);
   240    242     }
   241    243     sqlite3BtreeLeave(pBt);
   242    244     if( rc ){
   243    245       ckfree((char *)pCur);
   244    246       Tcl_AppendResult(interp, errorName(rc), 0);

Changes to src/test_func.c.

   337    337   
   338    338   /*
   339    339   **      hex_to_utf16be(HEX)
   340    340   **
   341    341   ** Convert the input string from HEX into binary.  Then return the
   342    342   ** result using sqlite3_result_text16le().
   343    343   */
          344  +#ifndef SQLITE_OMIT_UTF16
   344    345   static void testHexToUtf16be(
   345    346     sqlite3_context *pCtx, 
   346    347     int nArg,
   347    348     sqlite3_value **argv
   348    349   ){
   349    350     int n;
   350    351     const char *zIn;
................................................................................
   356    357     if( zOut==0 ){
   357    358       sqlite3_result_error_nomem(pCtx);
   358    359     }else{
   359    360       testHexToBin(zIn, zOut);
   360    361       sqlite3_result_text16be(pCtx, zOut, n/2, sqlite3_free);
   361    362     }
   362    363   }
          364  +#endif
   363    365   
   364    366   /*
   365    367   **      hex_to_utf8(HEX)
   366    368   **
   367    369   ** Convert the input string from HEX into binary.  Then return the
   368    370   ** result using sqlite3_result_text16le().
   369    371   */
................................................................................
   389    391   
   390    392   /*
   391    393   **      hex_to_utf16le(HEX)
   392    394   **
   393    395   ** Convert the input string from HEX into binary.  Then return the
   394    396   ** result using sqlite3_result_text16le().
   395    397   */
          398  +#ifndef SQLITE_OMIT_UTF16
   396    399   static void testHexToUtf16le(
   397    400     sqlite3_context *pCtx, 
   398    401     int nArg,
   399    402     sqlite3_value **argv
   400    403   ){
   401    404     int n;
   402    405     const char *zIn;
................................................................................
   408    411     if( zOut==0 ){
   409    412       sqlite3_result_error_nomem(pCtx);
   410    413     }else{
   411    414       testHexToBin(zIn, zOut);
   412    415       sqlite3_result_text16le(pCtx, zOut, n/2, sqlite3_free);
   413    416     }
   414    417   }
          418  +#endif
   415    419   
   416    420   static int registerTestFunctions(sqlite3 *db){
   417    421     static const struct {
   418    422        char *zName;
   419    423        signed char nArg;
   420    424        unsigned char eTextRep; /* 1: UTF-16.  0: UTF-8 */
   421    425        void (*xFunc)(sqlite3_context*,int,sqlite3_value **);

Changes to src/util.c.

    27     27   #ifdef SQLITE_COVERAGE_TEST
    28     28   void sqlite3Coverage(int x){
    29     29     static int dummy = 0;
    30     30     dummy += x;
    31     31   }
    32     32   #endif
    33     33   
           34  +#ifndef SQLITE_OMIT_FLOATING_POINT
    34     35   /*
    35     36   ** Return true if the floating point value is Not a Number (NaN).
    36     37   **
    37     38   ** Use the math library isnan() function if compiled with SQLITE_HAVE_ISNAN.
    38     39   ** Otherwise, we have our own implementation that works on most systems.
    39     40   */
    40     41   int sqlite3IsNaN(double x){
................................................................................
    71     72     rc = (y!=z);
    72     73   #else  /* if defined(SQLITE_HAVE_ISNAN) */
    73     74     rc = isnan(x);
    74     75   #endif /* SQLITE_HAVE_ISNAN */
    75     76     testcase( rc );
    76     77     return rc;
    77     78   }
           79  +#endif /* SQLITE_OMIT_FLOATING_POINT */
    78     80   
    79     81   /*
    80     82   ** Compute a string length that is limited to what can be stored in
    81     83   ** lower 30 bits of a 32-bit signed integer.
    82     84   **
    83     85   ** The value returned will never be negative.  Nor will it ever be greater
    84     86   ** than the actual length of the string.  For very long strings (greater
................................................................................
   251    253     if( *z=='-' || *z=='+' ) z += incr;
   252    254     if( !sqlite3Isdigit(*z) ){
   253    255       return 0;
   254    256     }
   255    257     z += incr;
   256    258     *realnum = 0;
   257    259     while( sqlite3Isdigit(*z) ){ z += incr; }
          260  +#ifndef SQLITE_OMIT_FLOATING_POINT
   258    261     if( *z=='.' ){
   259    262       z += incr;
   260    263       if( !sqlite3Isdigit(*z) ) return 0;
   261    264       while( sqlite3Isdigit(*z) ){ z += incr; }
   262    265       *realnum = 1;
   263    266     }
   264    267     if( *z=='e' || *z=='E' ){
   265    268       z += incr;
   266    269       if( *z=='+' || *z=='-' ) z += incr;
   267    270       if( !sqlite3Isdigit(*z) ) return 0;
   268    271       while( sqlite3Isdigit(*z) ){ z += incr; }
   269    272       *realnum = 1;
   270    273     }
          274  +#endif
   271    275     return *z==0;
   272    276   }
   273    277   
   274    278   /*
   275    279   ** The string z[] is an ASCII representation of a real number.
   276    280   ** Convert this string to a double.
   277    281   **
................................................................................
   425    429   ** will return -8.
   426    430   */
   427    431   static int compare2pow63(const char *zNum){
   428    432     int c;
   429    433     c = memcmp(zNum,"922337203685477580",18)*10;
   430    434     if( c==0 ){
   431    435       c = zNum[18] - '8';
          436  +    testcase( c==(-1) );
          437  +    testcase( c==0 );
          438  +    testcase( c==(+1) );
   432    439     }
   433    440     return c;
   434    441   }
   435    442   
   436    443   
   437    444   /*
   438    445   ** Return TRUE if zNum is a 64-bit signed integer and write
................................................................................
   461    468     }
   462    469     zStart = zNum;
   463    470     while( zNum[0]=='0' ){ zNum++; } /* Skip over leading zeros. Ticket #2454 */
   464    471     for(i=0; (c=zNum[i])>='0' && c<='9'; i++){
   465    472       v = v*10 + c - '0';
   466    473     }
   467    474     *pNum = neg ? -v : v;
          475  +  testcase( i==18 );
          476  +  testcase( i==19 );
          477  +  testcase( i==20 );
   468    478     if( c!=0 || (i==0 && zStart==zNum) || i>19 ){
   469    479       /* zNum is empty or contains non-numeric text or is longer
   470    480       ** than 19 digits (thus guaranting that it is too large) */
   471    481       return 0;
   472    482     }else if( i<19 ){
   473    483       /* Less than 19 digits, so we know that it fits in 64 bits */
   474    484       return 1;
................................................................................
   504    514     assert( zNum[0]>='0' && zNum[0]<='9' ); /* zNum is an unsigned number */
   505    515   
   506    516     if( negFlag ) neg = 1-neg;
   507    517     while( *zNum=='0' ){
   508    518       zNum++;   /* Skip leading zeros.  Ticket #2454 */
   509    519     }
   510    520     for(i=0; zNum[i]; i++){ assert( zNum[i]>='0' && zNum[i]<='9' ); }
          521  +  testcase( i==18 );
          522  +  testcase( i==19 );
          523  +  testcase( i==20 );
   511    524     if( i<19 ){
   512    525       /* Guaranteed to fit if less than 19 digits */
   513    526       return 1;
   514    527     }else if( i>19 ){
   515    528       /* Guaranteed to be too big if greater than 19 digits */
   516    529       return 0;
   517    530     }else{
................................................................................
   544    557     }
   545    558   
   546    559     /* The longest decimal representation of a 32 bit integer is 10 digits:
   547    560     **
   548    561     **             1234567890
   549    562     **     2^31 -> 2147483648
   550    563     */
          564  +  testcase( i==10 );
   551    565     if( i>10 ){
   552    566       return 0;
   553    567     }
          568  +  testcase( v-neg==2147483647 );
   554    569     if( v-neg>2147483647 ){
   555    570       return 0;
   556    571     }
   557    572     if( neg ){
   558    573       v = -v;
   559    574     }
   560    575     *pValue = (int)v;

Changes to src/vacuum.c.

   100    100     ** restored before returning. Then set the writable-schema flag, and
   101    101     ** disable CHECK and foreign key constraints.  */
   102    102     saved_flags = db->flags;
   103    103     saved_nChange = db->nChange;
   104    104     saved_nTotalChange = db->nTotalChange;
   105    105     saved_xTrace = db->xTrace;
   106    106     db->flags |= SQLITE_WriteSchema | SQLITE_IgnoreChecks;
   107         -  db->flags &= ~SQLITE_ForeignKeys;
          107  +  db->flags &= ~(SQLITE_ForeignKeys | SQLITE_ReverseOrder);
   108    108     db->xTrace = 0;
   109    109   
   110    110     pMain = db->aDb[0].pBt;
   111    111     isMemDb = sqlite3PagerIsMemdb(sqlite3BtreePager(pMain));
   112    112   
   113    113     /* Attach the temporary database as 'vacuum_db'. The synchronous pragma
   114    114     ** can be set to 'off' for this file, as it is not recovered if a crash

Changes to src/vdbe.c.

   880    880   */
   881    881   case OP_Int64: {           /* out2-prerelease */
   882    882     assert( pOp->p4.pI64!=0 );
   883    883     pOut->u.i = *pOp->p4.pI64;
   884    884     break;
   885    885   }
   886    886   
          887  +#ifndef SQLITE_OMIT_FLOATING_POINT
   887    888   /* Opcode: Real * P2 * P4 *
   888    889   **
   889    890   ** P4 is a pointer to a 64-bit floating point value.
   890    891   ** Write that value into register P2.
   891    892   */
   892    893   case OP_Real: {            /* same as TK_FLOAT, out2-prerelease */
   893    894     pOut->flags = MEM_Real;
   894    895     assert( !sqlite3IsNaN(*pOp->p4.pReal) );
   895    896     pOut->r = *pOp->p4.pReal;
   896    897     break;
   897    898   }
          899  +#endif
   898    900   
   899    901   /* Opcode: String8 * P2 * P4 *
   900    902   **
   901    903   ** P4 points to a nul terminated UTF-8 string. This opcode is transformed 
   902    904   ** into an OP_String before it is executed for the first time.
   903    905   */
   904    906   case OP_String8: {         /* same as TK_STRING, out2-prerelease */
................................................................................
  1291   1293           iB = (i64)rB;
  1292   1294           if( iA==0 ) goto arithmetic_result_is_null;
  1293   1295           if( iA==-1 ) iA = 1;
  1294   1296           rB = (double)(iB % iA);
  1295   1297           break;
  1296   1298         }
  1297   1299       }
         1300  +#ifdef SQLITE_OMIT_FLOATING_POINT
         1301  +    pOut->u.i = rB;
         1302  +    MemSetTypeFlag(pOut, MEM_Int);
         1303  +#else
  1298   1304       if( sqlite3IsNaN(rB) ){
  1299   1305         goto arithmetic_result_is_null;
  1300   1306       }
  1301   1307       pOut->r = rB;
  1302   1308       MemSetTypeFlag(pOut, MEM_Real);
  1303   1309       if( (flags & MEM_Real)==0 ){
  1304   1310         sqlite3VdbeIntegerAffinity(pOut);
  1305   1311       }
         1312  +#endif
  1306   1313     }
  1307   1314     break;
  1308   1315   
  1309   1316   arithmetic_result_is_null:
  1310   1317     sqlite3VdbeMemSetNull(pOut);
  1311   1318     break;
  1312   1319   }
................................................................................
  1526   1533       }
  1527   1534     }else{
  1528   1535       MemSetTypeFlag(pIn1, MEM_Int);
  1529   1536     }
  1530   1537     break;
  1531   1538   }
  1532   1539   
         1540  +#ifndef SQLITE_OMIT_FLOATING_POINT
  1533   1541   /* Opcode: RealAffinity P1 * * * *
  1534   1542   **
  1535   1543   ** If register P1 holds an integer convert it to a real value.
  1536   1544   **
  1537   1545   ** This opcode is used when extracting information from a column that
  1538   1546   ** has REAL affinity.  Such column values may still be stored as
  1539   1547   ** integers, for space efficiency, but after extraction we want them
................................................................................
  1542   1550   case OP_RealAffinity: {                  /* in1 */
  1543   1551     pIn1 = &aMem[pOp->p1];
  1544   1552     if( pIn1->flags & MEM_Int ){
  1545   1553       sqlite3VdbeMemRealify(pIn1);
  1546   1554     }
  1547   1555     break;
  1548   1556   }
         1557  +#endif
  1549   1558   
  1550   1559   #ifndef SQLITE_OMIT_CAST
  1551   1560   /* Opcode: ToText P1 * * * *
  1552   1561   **
  1553   1562   ** Force the value in register P1 to be text.
  1554   1563   ** If the value is numeric, convert it to a string using the
  1555   1564   ** equivalent of printf().  Blob values are unchanged and
................................................................................
  1625   1634     pIn1 = &aMem[pOp->p1];
  1626   1635     if( (pIn1->flags & MEM_Null)==0 ){
  1627   1636       sqlite3VdbeMemIntegerify(pIn1);
  1628   1637     }
  1629   1638     break;
  1630   1639   }
  1631   1640   
  1632         -#ifndef SQLITE_OMIT_CAST
         1641  +#if !defined(SQLITE_OMIT_CAST) && !defined(SQLITE_OMIT_FLOATING_POINT)
  1633   1642   /* Opcode: ToReal P1 * * * *
  1634   1643   **
  1635   1644   ** Force the value in register P1 to be a floating point number.
  1636   1645   ** If The value is currently an integer, convert it.
  1637   1646   ** If the value is text or blob, try to convert it to an integer using the
  1638   1647   ** equivalent of atoi() and store 0.0 if no such conversion is possible.
  1639   1648   **
................................................................................
  1642   1651   case OP_ToReal: {                  /* same as TK_TO_REAL, in1 */
  1643   1652     pIn1 = &aMem[pOp->p1];
  1644   1653     if( (pIn1->flags & MEM_Null)==0 ){
  1645   1654       sqlite3VdbeMemRealify(pIn1);
  1646   1655     }
  1647   1656     break;
  1648   1657   }
  1649         -#endif /* SQLITE_OMIT_CAST */
         1658  +#endif /* !defined(SQLITE_OMIT_CAST) && !defined(SQLITE_OMIT_FLOATING_POINT) */
  1650   1659   
  1651   1660   /* Opcode: Lt P1 P2 P3 P4 P5
  1652   1661   **
  1653   1662   ** Compare the values in register P1 and P3.  If reg(P3)<reg(P1) then
  1654   1663   ** jump to address P2.  
  1655   1664   **
  1656   1665   ** If the SQLITE_JUMPIFNULL bit of P5 is set and either reg(P1) or
................................................................................
  4568   4577     sqlite3BtreeEnterAll(db);
  4569   4578     if( pOp->p2 || DbHasProperty(db, iDb, DB_SchemaLoaded) ){
  4570   4579       zMaster = SCHEMA_TABLE(iDb);
  4571   4580       initData.db = db;
  4572   4581       initData.iDb = pOp->p1;
  4573   4582       initData.pzErrMsg = &p->zErrMsg;
  4574   4583       zSql = sqlite3MPrintf(db,
  4575         -       "SELECT name, rootpage, sql FROM '%q'.%s WHERE %s",
         4584  +       "SELECT name, rootpage, sql FROM '%q'.%s WHERE %s ORDER BY rowid",
  4576   4585          db->aDb[iDb].zName, zMaster, pOp->p4.z);
  4577   4586       if( zSql==0 ){
  4578   4587         rc = SQLITE_NOMEM;
  4579   4588       }else{
  4580   4589         (void)sqlite3SafetyOff(db);
  4581   4590         assert( db->init.busy==0 );
  4582   4591         db->init.busy = 1;
................................................................................
  5660   5669   /*
  5661   5670   ** The magic Explain opcode are only inserted when explain==2 (which
  5662   5671   ** is to say when the EXPLAIN QUERY PLAN syntax is used.)
  5663   5672   ** This opcode records information from the optimizer.  It is the
  5664   5673   ** the same as a no-op.  This opcodesnever appears in a real VM program.
  5665   5674   */
  5666   5675   default: {          /* This is really OP_Noop and OP_Explain */
         5676  +  assert( pOp->opcode==OP_Noop || pOp->opcode==OP_Explain );
  5667   5677     break;
  5668   5678   }
  5669   5679   
  5670   5680   /*****************************************************************************
  5671   5681   ** The cases of the switch statement above this line should all be indented
  5672   5682   ** by 6 spaces.  But the left-most 6 spaces have been removed to improve the
  5673   5683   ** readability.  From this point on down, the normal indentation rules are

Changes to src/vdbeInt.h.

   358    358   int sqlite3VdbeMemTooBig(Mem*);
   359    359   int sqlite3VdbeMemCopy(Mem*, const Mem*);
   360    360   void sqlite3VdbeMemShallowCopy(Mem*, const Mem*, int);
   361    361   void sqlite3VdbeMemMove(Mem*, Mem*);
   362    362   int sqlite3VdbeMemNulTerminate(Mem*);
   363    363   int sqlite3VdbeMemSetStr(Mem*, const char*, int, u8, void(*)(void*));
   364    364   void sqlite3VdbeMemSetInt64(Mem*, i64);
   365         -void sqlite3VdbeMemSetDouble(Mem*, double);
          365  +#ifdef SQLITE_OMIT_FLOATING_POINT
          366  +# define sqlite3VdbeMemSetDouble sqlite3VdbeMemSetInt64
          367  +#else
          368  +  void sqlite3VdbeMemSetDouble(Mem*, double);
          369  +#endif
   366    370   void sqlite3VdbeMemSetNull(Mem*);
   367    371   void sqlite3VdbeMemSetZeroBlob(Mem*,int);
   368    372   void sqlite3VdbeMemSetRowSet(Mem*);
   369    373   int sqlite3VdbeMemMakeWriteable(Mem*);
   370    374   int sqlite3VdbeMemStringify(Mem*, int);
   371    375   i64 sqlite3VdbeIntValue(Mem*);
   372    376   int sqlite3VdbeMemIntegerify(Mem*);

Changes to src/vdbeapi.c.

    49     49       rc = SQLITE_OK;
    50     50     }else{
    51     51       Vdbe *v = (Vdbe*)pStmt;
    52     52   #ifdef SQLITE_ENABLE_SQLRR
    53     53       SRRecFinalize(pStmt);
    54     54   #endif
    55     55       sqlite3 *db = v->db;
    56         -    if( db==0 ){
    57         -      return SQLITE_MISUSE;
    58         -    }
           56  +    if( db==0 ) return SQLITE_MISUSE;
    59     57   #if SQLITE_THREADSAFE
    60     58       sqlite3_mutex *mutex = v->db->mutex;
    61     59   #endif
    62     60       sqlite3_mutex_enter(mutex);
    63     61       rc = sqlite3VdbeFinalize(v);
    64     62       rc = sqlite3ApiExit(db, rc);
    65     63       sqlite3_mutex_leave(mutex);
................................................................................
   414    412   ** This is the top-level implementation of sqlite3_step().  Call
   415    413   ** sqlite3Step() to do most of the work.  If a schema error occurs,
   416    414   ** call sqlite3Reprepare() and try again.
   417    415   */
   418    416   int sqlite3_step(sqlite3_stmt *pStmt){
   419    417     int rc = SQLITE_MISUSE;
   420    418     Vdbe *v = (Vdbe*)pStmt;
   421         -  sqlite3 *db;
   422         -  if( v && ((db = v->db) != NULL)){
          419  +  if( v && (v->db)!=0 ){
   423    420       int cnt = 0;
          421  +    sqlite3 *db = v->db;
   424    422   #ifdef SQLITE_ENABLE_SQLRR
   425    423       SRRecStep(pStmt);
   426    424   #endif
   427         -    
   428    425       sqlite3_mutex_enter(db->mutex);
   429    426       while( (rc = sqlite3Step(v))==SQLITE_SCHEMA
   430    427              && cnt++ < 5
   431    428              && (rc = sqlite3Reprepare(v))==SQLITE_OK ){
   432    429         sqlite3_reset(pStmt);
   433    430         v->expired = 0;
   434    431       }

Changes to src/vdbeaux.c.

    62     62   }
    63     63   
    64     64   /*
    65     65   ** Return the SQL associated with a prepared statement
    66     66   */
    67     67   const char *sqlite3_sql(sqlite3_stmt *pStmt){
    68     68     Vdbe *p = (Vdbe *)pStmt;
    69         -  return ((p && p->isPrepareV2) ? p->zSql : 0);
           69  +  return (p && p->isPrepareV2) ? p->zSql : 0;
    70     70   }
    71     71   
    72     72   /*
    73     73   ** Swap all content between two VDBE structures.
    74     74   */
    75     75   void sqlite3VdbeSwap(Vdbe *pA, Vdbe *pB){
    76     76     Vdbe tmp, *pTmp;
................................................................................
  2332   2332     releaseMemArray(p->aColName, p->nResColumn*COLNAME_N);
  2333   2333     vdbeFreeOpArray(db, p->aOp, p->nOp);
  2334   2334     sqlite3DbFree(db, p->aLabel);
  2335   2335     sqlite3DbFree(db, p->aColName);
  2336   2336     sqlite3DbFree(db, p->zSql);
  2337   2337     p->magic = VDBE_MAGIC_DEAD;
  2338   2338     sqlite3DbFree(db, p->pFree);
  2339         -  p->db = NULL;
         2339  +  p->db = 0;
  2340   2340     sqlite3DbFree(db, p);
  2341   2341   }
  2342   2342   
  2343   2343   /*
  2344   2344   ** Make sure the cursor p is ready to read or write the row to which it
  2345   2345   ** was last positioned.  Return an error code if an OOM fault or I/O error
  2346   2346   ** prevents us from positioning the cursor to its correct position.

Changes to src/vdbemem.c.

   311    311   ** there are reports that windows throws an expection
   312    312   ** if the floating point value is out of range. (See ticket #2880.)
   313    313   ** Because we do not completely understand the problem, we will
   314    314   ** take the conservative approach and always do range tests
   315    315   ** before attempting the conversion.
   316    316   */
   317    317   static i64 doubleToInt64(double r){
          318  +#ifdef SQLITE_OMIT_FLOATING_POINT
          319  +  /* When floating-point is omitted, double and int64 are the same thing */
          320  +  return r;
          321  +#else
   318    322     /*
   319    323     ** Many compilers we encounter do not define constants for the
   320    324     ** minimum and maximum 64-bit integers, or they define them
   321    325     ** inconsistently.  And many do not understand the "LL" notation.
   322    326     ** So we define our own static constants here using nothing
   323    327     ** larger than a 32-bit integer constant.
   324    328     */
................................................................................
   332    336       ** a very large positive number to an integer results in a very large
   333    337       ** negative integer.  This makes no sense, but it is what x86 hardware
   334    338       ** does so for compatibility we will do the same in software. */
   335    339       return minInt;
   336    340     }else{
   337    341       return (i64)r;
   338    342     }
          343  +#endif
   339    344   }
   340    345   
   341    346   /*
   342    347   ** Return some kind of integer value which is the best we can do
   343    348   ** at representing the value that *pMem describes as an integer.
   344    349   ** If pMem is an integer, then the value is exact.  If pMem is
   345    350   ** a floating-point then the value returned is the integer part.
................................................................................
   525    530   void sqlite3VdbeMemSetInt64(Mem *pMem, i64 val){
   526    531     sqlite3VdbeMemRelease(pMem);
   527    532     pMem->u.i = val;
   528    533     pMem->flags = MEM_Int;
   529    534     pMem->type = SQLITE_INTEGER;
   530    535   }
   531    536   
          537  +#ifndef SQLITE_OMIT_FLOATING_POINT
   532    538   /*
   533    539   ** Delete any previous value and set the value stored in *pMem to val,
   534    540   ** manifest type REAL.
   535    541   */
   536    542   void sqlite3VdbeMemSetDouble(Mem *pMem, double val){
   537    543     if( sqlite3IsNaN(val) ){
   538    544       sqlite3VdbeMemSetNull(pMem);
................................................................................
   539    545     }else{
   540    546       sqlite3VdbeMemRelease(pMem);
   541    547       pMem->r = val;
   542    548       pMem->flags = MEM_Real;
   543    549       pMem->type = SQLITE_FLOAT;
   544    550     }
   545    551   }
          552  +#endif
   546    553   
   547    554   /*
   548    555   ** Delete any previous value and set the value of pMem to be an
   549    556   ** empty boolean index.
   550    557   */
   551    558   void sqlite3VdbeMemSetRowSet(Mem *pMem){
   552    559     sqlite3 *db = pMem->db;

Changes to test/auth2.test.

    98     98   SQLITE_UPDATE sqlite_master rootpage main {}
    99     99   SQLITE_UPDATE sqlite_master sql main {}
   100    100   SQLITE_READ sqlite_master ROWID main {}
   101    101   SQLITE_READ sqlite_master name main {}
   102    102   SQLITE_READ sqlite_master rootpage main {}
   103    103   SQLITE_READ sqlite_master sql main {}
   104    104   SQLITE_READ sqlite_master tbl_name main {}
          105  +SQLITE_READ sqlite_master ROWID main {}
   105    106   }
   106    107   do_test auth2-2.2 {
   107    108     set ::authargs {}
   108    109     db eval {
   109    110       CREATE VIEW v2 AS SELECT x+y AS a, y+z AS b from t2;
   110    111     }
   111    112     set ::authargs
................................................................................
   117    118   SQLITE_UPDATE sqlite_master rootpage main {}
   118    119   SQLITE_UPDATE sqlite_master sql main {}
   119    120   SQLITE_READ sqlite_master ROWID main {}
   120    121   SQLITE_READ sqlite_master name main {}
   121    122   SQLITE_READ sqlite_master rootpage main {}
   122    123   SQLITE_READ sqlite_master sql main {}
   123    124   SQLITE_READ sqlite_master tbl_name main {}
          125  +SQLITE_READ sqlite_master ROWID main {}
   124    126   }
   125    127   do_test auth2-2.3 {
   126    128     set ::authargs {}
   127    129     db eval {
   128    130       SELECT a, b FROM v2;
   129    131     }
   130    132     set ::authargs

Changes to test/backup.test.

   900    900       execsql { PRAGMA integrity_check } db3
   901    901     } {ok}
   902    902   
   903    903     db2 close
   904    904     db3 close
   905    905   }
   906    906   
          907  +
          908  +# Test that if the database is written to via the same database handle being
          909  +# used as the source by a backup operation:
          910  +#
          911  +#   10.1.*: If the db is in-memory, the backup is restarted.
          912  +#   10.2.*: If the db is a file, the backup is not restarted.
          913  +#
          914  +db close
          915  +file delete -force test.db test.db-journal
          916  +foreach {tn file rc} {
          917  +  1 test.db  SQLITE_DONE
          918  +  2 :memory: SQLITE_OK
          919  +} {
          920  +  do_test backup-10.$tn.1 {
          921  +    sqlite3 db $file
          922  +    execsql { 
          923  +      CREATE TABLE t1(a INTEGER PRIMARY KEY, b BLOB);
          924  +      BEGIN;
          925  +        INSERT INTO t1 VALUES(NULL, randomblob(200));
          926  +        INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1;
          927  +        INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1;
          928  +        INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1;
          929  +        INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1;
          930  +        INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1;
          931  +        INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1;
          932  +        INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1;
          933  +        INSERT INTO t1 SELECT NULL, randomblob(200) FROM t1;
          934  +      COMMIT;
          935  +      SELECT count(*) FROM t1;
          936  +    }
          937  +  } {256}
          938  +
          939  +  do_test backup-10.$tn.2 {
          940  +    set pgs [execsql {pragma page_count}]
          941  +    expr {$pgs > 50 && $pgs < 75}
          942  +  } {1}
          943  +
          944  +  do_test backup-10.$tn.3 {
          945  +    file delete -force bak.db bak.db-journal
          946  +    sqlite3 db2 bak.db
          947  +    sqlite3_backup B db2 main db main
          948  +    B step 50
          949  +  } {SQLITE_OK}
          950  +
          951  +  do_test backup-10.$tn.4 {
          952  +    execsql { UPDATE t1 SET b = randomblob(200) WHERE a IN (1, 250) }
          953  +  } {}
          954  +
          955  +  do_test backup-10.$tn.5 {
          956  +    B step 50
          957  +  } $rc
          958  +
          959  +  do_test backup-10.6 {
          960  +    B finish
          961  +  } {SQLITE_OK}
          962  +}
          963  +
          964  +db2 close
   907    965   finish_test

Changes to test/e_fts3.test.

    60     60   # [foreach] loop is testing with OOM errors, disable the lookaside buffer.
    61     61   #
    62     62   db close
    63     63   file delete -force test.db test.db-journal
    64     64   sqlite3 db test.db
    65     65   if {$DO_MALLOC_TEST} { sqlite3_db_config_lookaside db 0 0 0 }
    66     66   db eval "PRAGMA encoding = '$enc'"
           67  +
           68  +proc mit {blob} {
           69  +  set scan(littleEndian) i*
           70  +  set scan(bigEndian) I*
           71  +  binary scan $blob $scan($::tcl_platform(byteOrder)) r
           72  +  return $r
           73  +}
           74  +db func mit mit
    67     75   
    68     76   ##########################################################################
    69     77   # Test the example CREATE VIRTUAL TABLE statements in section 1.1 
    70     78   # of fts3.in.
    71     79   #
    72     80   ddl_test   1.1.1.1 {CREATE VIRTUAL TABLE data USING fts3()}
    73     81   read_test  1.1.1.2 {PRAGMA table_info(data)} {0 content {} 0 {} 0}
................................................................................
   407    415   } {{1 0 5 7 1 0 30 7}}
   408    416   read_test  1.7.1.6 { 
   409    417     SELECT offsets(mail) FROM mail WHERE mail MATCH '"serious mail"'
   410    418   } {{1 0 28 7 1 1 36 4}}
   411    419   
   412    420   ddl_test   1.7.2.1 { CREATE VIRTUAL TABLE text USING fts3() }
   413    421   
   414         -write_test 3.2.2 text_content {
          422  +write_test 1.7.2.2 text_content {
   415    423     INSERT INTO text VALUES('
   416    424       During 30 Nov-1 Dec, 2-3oC drops. Cool in the upper portion, minimum temperature 14-16oC and cool elsewhere, minimum temperature 17-20oC. Cold to very cold on mountaintops, minimum temperature 6-12oC. Northeasterly winds 15-30 km/hr. After that, temperature increases. Northeasterly winds 15-30 km/hr.
   417    425     ');
   418    426   }
   419    427   
   420    428   read_test  1.7.2.3 {
   421    429     SELECT snippet(text) FROM text WHERE text MATCH 'cold'
   422         -} {{<b>...</b> elsewhere, minimum temperature 17-20oC. <b>Cold</b> to very <b>cold</b> on mountaintops, minimum <b>...</b>}}
          430  +} {{<b>...</b>cool elsewhere, minimum temperature 17-20oC. <b>Cold</b> to very <b>cold</b> on mountaintops, minimum temperature 6<b>...</b>}}
   423    431   
   424    432   read_test  1.7.2.4 {
   425    433     SELECT snippet(text, '[', ']', '...') FROM text WHERE text MATCH '"min* tem*"'
   426         -} {{... 2-3oC drops. Cool in the upper portion, [minimum] [temperature] 14-16oC and cool elsewhere, [minimum] ...}}
          434  +} {{...the upper portion, [minimum] [temperature] 14-16oC and cool elsewhere, [minimum] [temperature] 17-20oC. Cold...}}
          435  +
          436  +ddl_test   1.7.3.1 { DROP TABLE IF EXISTS t1 }
          437  +ddl_test   1.7.3.2 { CREATE VIRTUAL TABLE t1 USING fts3(a, b) }
          438  +write_test 1.7.3.3 t1_content { 
          439  +  INSERT INTO t1 VALUES(
          440  +    'transaction default models default', 'Non transaction reads');
          441  +}
          442  +write_test 1.7.3.4 t1_content { 
          443  +  INSERT INTO t1 VALUES('the default transaction', 'these semantics present');
          444  +}
          445  +write_test 1.7.3.5 t1_content { 
          446  +  INSERT INTO t1 VALUES('single request', 'default data');
          447  +}
          448  +read_test  1.7.3.6 { 
          449  +  SELECT mit(matchinfo(t1)) FROM t1 
          450  +    WHERE t1 MATCH 'default transaction "these semantics"';
          451  +} {{3 2 1 3 2 0 1 1 1 2 2 0 1 1 0 0 0 1 1 1}}
   427    452   
   428    453   ##########################################################################
   429    454   # Test the example in section 5 (custom tokenizers).
   430    455   #
   431    456   ddl_test   1.8.1.1 { CREATE VIRTUAL TABLE simple USING fts3(tokenize=simple) } 
   432    457   write_test 1.8.1.2 simple_content { 
   433    458     INSERT INTO simple VALUES('Right now they''re very frustrated')
................................................................................
   448    473   #-------------------------------------------------------------------------
   449    474   
   450    475   #-------------------------------------------------------------------------
   451    476   # Test that errors in the arguments passed to the snippet and offsets
   452    477   # functions are handled correctly.
   453    478   #
   454    479   set DO_MALLOC_TEST 0
          480  +ddl_test   2.1.0 { DROP TABLE IF EXISTS t1 }
   455    481   ddl_test   2.1.1 { CREATE VIRTUAL TABLE t1 USING fts3(a, b) }
   456    482   write_test 2.1.2 t1_content { 
   457    483     INSERT INTO t1 VALUES('one two three', x'A1B2C3D4E5F6');
   458    484   }
   459    485   error_test 2.1.3 {
   460    486     SELECT offsets(a) FROM t1 WHERE a MATCH 'one'
   461    487   } {illegal first argument to offsets}
................................................................................
   468    494   error_test 2.1.6 {
   469    495     SELECT snippet(a) FROM t1 WHERE a MATCH 'one'
   470    496   } {illegal first argument to snippet}
   471    497   error_test 2.1.7 {
   472    498     SELECT snippet() FROM t1 WHERE a MATCH 'one'
   473    499   } {unable to use function snippet in the requested context}
   474    500   error_test 2.1.8 {
   475         -  SELECT snippet(a, b, 'A', 'B', 'C') FROM t1 WHERE a MATCH 'one'
          501  +  SELECT snippet(a, b, 'A', 'B', 'C', 'D', 'E') FROM t1 WHERE a MATCH 'one'
   476    502   } {wrong number of arguments to function snippet()}
   477    503   #-------------------------------------------------------------------------
   478    504   
   479    505   #-------------------------------------------------------------------------
   480    506   # Test the effect of an OOM error while installing the FTS3 module (i.e.
   481    507   # opening a database handle). This case was not tested by the OOM testing
   482    508   # of the document examples above.
................................................................................
   514    540   set DO_MALLOC_TEST 0
   515    541   ddl_test   5.1 { CREATE VIRTUAL TABLE t5 USING fts3(x) }
   516    542   write_test 5.2 t5_content {
   517    543     INSERT INTO t5 VALUES('In Xanadu did Kubla Khan A stately pleasure-dome decree Where Alph, the sacred river, ran Through caverns measureless to man Down to a sunless sea.  So twice five miles of fertile ground With walls and towers were girdled round : And there were gardens bright with sinuous rills, Where blossomed many an incense-bearing tree ; And here were forests ancient as the hills, Enfolding sunny spots of greenery.');
   518    544   }
   519    545   read_test 5.3 { 
   520    546     SELECT snippet(t5) FROM t5 WHERE t5 MATCH 'miles'
   521         -} {{<b>...</b> Down to a sunless sea.  So twice five <b>miles</b> of fertile ground With walls and towers were <b>...</b>}}
          547  +} {{<b>...</b>to a sunless sea.  So twice five <b>miles</b> of fertile ground With walls and towers<b>...</b>}}
   522    548   read_test 5.4 { 
   523    549     SELECT snippet(t5, '<i>') FROM t5 WHERE t5 MATCH 'miles'
   524         -} {{<b>...</b> Down to a sunless sea.  So twice five <i>miles</b> of fertile ground With walls and towers were <b>...</b>}}
          550  +} {{<b>...</b>to a sunless sea.  So twice five <i>miles</b> of fertile ground With walls and towers<b>...</b>}}
   525    551   read_test 5.5 { 
   526    552     SELECT snippet(t5, '<i>', '</i>') FROM t5 WHERE t5 MATCH 'miles'
   527         -} {{<b>...</b> Down to a sunless sea.  So twice five <i>miles</i> of fertile ground With walls and towers were <b>...</b>}}
          553  +} {{<b>...</b>to a sunless sea.  So twice five <i>miles</i> of fertile ground With walls and towers<b>...</b>}}
   528    554   read_test 5.6 { 
   529    555     SELECT snippet(t5, '<i>', '</i>', 'XXX') FROM t5 WHERE t5 MATCH 'miles'
   530         -} {{XXX Down to a sunless sea.  So twice five <i>miles</i> of fertile ground With walls and towers were XXX}}
          556  +} {{XXXto a sunless sea.  So twice five <i>miles</i> of fertile ground With walls and towersXXX}}
   531    557   #-------------------------------------------------------------------------
   532    558   
   533    559   #-------------------------------------------------------------------------
   534    560   # Test that an empty MATCH expression returns an empty result set. As
   535    561   # does passing a NULL value as a MATCH expression.
   536    562   #
   537    563   set DO_MALLOC_TEST 0

Changes to test/fts3_common.tcl.

   322    322   }
   323    323   
   324    324   proc doPassiveTest {isRestart name sql catchres} {
   325    325     if {![info exists ::DO_MALLOC_TEST]} { set ::DO_MALLOC_TEST 1 }
   326    326   
   327    327     switch $::DO_MALLOC_TEST {
   328    328       0 { # No malloc failures.
   329         -      do_test $name [list catchsql $sql] $catchres
          329  +      do_test $name [list set {} [uplevel [list catchsql $sql]]] $catchres
   330    330         return
   331    331       }
   332    332       1 { # Simulate transient failures.
   333    333         set nRepeat 1
   334    334         set zName "transient"
   335    335         set nStartLimit 100000
   336    336         set nBackup 1

Changes to test/fts3ac.test.

  1127   1127     }
  1128   1128   } {{Alert Posted 10:00 AM November 20,2000: E-<b>GAS</b> Request <b>Reminder</b>}}
  1129   1129   do_test fts3ac-4.2 {
  1130   1130     execsql {
  1131   1131       SELECT snippet(email) FROM email
  1132   1132        WHERE email MATCH 'christmas candlelight'
  1133   1133     }
  1134         -} {{<b>...</b> place.? What do you think about going here <b>Christmas</b> 
  1135         -eve?? They have an 11:00 a.m. service and a <b>candlelight</b> service at 5:00 p.m., 
  1136         -among others. <b>...</b>}}
         1134  +} {{<b>...</b>here <b>Christmas</b> 
         1135  +eve?? They have an 11:00 a.m. service and a <b>candlelight</b> service<b>...</b>}}
  1137   1136   
  1138   1137   do_test fts3ac-4.3 {
  1139   1138     execsql {
  1140   1139       SELECT snippet(email) FROM email
  1141   1140        WHERE email MATCH 'deal sheet potential reuse'
  1142   1141     }
  1143         -} {{EOL-Accenture <b>Deal</b> <b>Sheet</b> <b>...</b> intent
  1144         -     Review Enron asset base for <b>potential</b> <b>reuse</b>/ licensing
  1145         -     Contract negotiations <b>...</b>}}
         1142  +} {{EOL-Accenture <b>Deal</b> <b>Sheet</b><b>...</b>asset base for <b>potential</b> <b>reuse</b>/ licensing
         1143  +     Contract negotiations<b>...</b>}}
  1146   1144   do_test fts3ac-4.4 {
  1147   1145     execsql {
  1148   1146       SELECT snippet(email,'<<<','>>>',' ') FROM email
  1149   1147        WHERE email MATCH 'deal sheet potential reuse'
  1150   1148     }
  1151         -} {{EOL-Accenture <<<Deal>>> <<<Sheet>>>  intent
  1152         -     Review Enron asset base for <<<potential>>> <<<reuse>>>/ licensing
  1153         -     Contract negotiations  }}
         1149  +} {{EOL-Accenture <<<Deal>>> <<<Sheet>>> asset base for <<<potential>>> <<<reuse>>>/ licensing
         1150  +     Contract negotiations }}
  1154   1151   do_test fts3ac-4.5 {
  1155   1152     execsql {
  1156   1153       SELECT snippet(email,'<<<','>>>',' ') FROM email
  1157   1154        WHERE email MATCH 'first things'
  1158   1155     }
  1159         -} {{Re: <<<First>>> Polish Deal!  Congrats!  <<<Things>>> seem to be building rapidly now on the  }}
         1156  +} {{Re: <<<First>>> Polish Deal! Congrats!  <<<Things>>> seem to be building rapidly now }}
  1160   1157   do_test fts3ac-4.6 {
  1161   1158     execsql {
  1162   1159       SELECT snippet(email) FROM email
  1163   1160        WHERE email MATCH 'chris is here'
  1164   1161     }
  1165         -} {{<b>chris</b>.germany@enron.com <b>...</b> Sounds good to me.  I bet this <b>is</b> next to the Warick?? Hotel. <b>...</b> place.? What do you think about going <b>here</b> Christmas 
  1166         -eve?? They have an 11:00 a.m. <b>...</b>}}
         1162  +} {{<b>...</b><b>chris</b>.germany@enron.com'" <<b>chris</b><b>...</b>bet this <b>is</b> next to<b>...</b>about going <b>here</b> Christmas 
         1163  +eve<b>...</b>}}
  1167   1164   do_test fts3ac-4.7 {
  1168   1165     execsql {
  1169   1166       SELECT snippet(email) FROM email
  1170   1167        WHERE email MATCH '"pursuant to"'
  1171   1168     }
  1172   1169   } {{Erin:
  1173   1170   
  1174         -<b>Pursuant</b> <b>to</b> your request, attached are the Schedule to <b>...</b>}}
         1171  +<b>Pursuant</b> <b>to</b> your request, attached are the Schedule to the ISDA Master Agreement, together<b>...</b>}}
  1175   1172   do_test fts3ac-4.8 {
  1176   1173     execsql {
  1177   1174       SELECT snippet(email) FROM email
  1178   1175        WHERE email MATCH 'ancillary load davis'
  1179   1176     }
  1180         -} {{pete.<b>davis</b>@enron.com <b>...</b> Start Date: 4/22/01; HourAhead hour: 3;  No <b>ancillary</b> schedules awarded.  
  1181         -Variances detected.
  1182         -Variances detected in <b>Load</b> schedule.
         1177  +} {{pete.<b>davis</b>@enron.com<b>...</b>3;  No <b>ancillary</b> schedules awarded<b>...</b>detected in <b>Load</b> schedule.
  1183   1178   
  1184         -    LOG MESSAGES:
  1185         -
  1186         -PARSING <b>...</b>}}
         1179  +    LOG<b>...</b>}}
  1187   1180   
  1188   1181   # Combinations of AND and OR operators:
  1189   1182   #
  1190   1183   do_test fts3ac-5.1 {
  1191   1184     execsql {
  1192   1185       SELECT snippet(email) FROM email
  1193   1186        WHERE email MATCH 'questar enron OR com'
  1194   1187     }
  1195         -} {{matt.smith@<b>enron</b>.<b>com</b> <b>...</b> six reports:  
  1196         -
  1197         -31 Keystone Receipts
         1188  +} {{matt.smith@<b>enron</b>.<b>com</b><b>...</b>31 Keystone Receipts
  1198   1189   15 <b>Questar</b> Pipeline
  1199         -40 Rockies Production
  1200         -22 West_2 <b>...</b>}}
         1190  +40 Rockies<b>...</b>}}
         1191  +
  1201   1192   do_test fts3ac-5.2 {
  1202   1193     execsql {
  1203   1194       SELECT snippet(email) FROM email
  1204   1195        WHERE email MATCH 'enron OR com questar'
  1205   1196     }
  1206         -} {{matt.smith@<b>enron</b>.<b>com</b> <b>...</b> six reports:  
  1207         -
  1208         -31 Keystone Receipts
         1197  +} {{matt.smith@<b>enron</b>.<b>com</b><b>...</b>31 Keystone Receipts
  1209   1198   15 <b>Questar</b> Pipeline
  1210         -40 Rockies Production
  1211         -22 West_2 <b>...</b>}}
         1199  +40 Rockies<b>...</b>}}
  1212   1200   
  1213   1201   finish_test

Changes to test/fts3al.test.

    49     49   # exercised.  The version with a couple extra spaces should cause the
    50     50   # other isspace() call to be exercised.  [Both cases have been tested
    51     51   # in the debugger, but I'm hoping to continue to catch it if simple
    52     52   # constant changes change things slightly.
    53     53   #
    54     54   # The trailing and leading hi-bit chars help with code which tests for
    55     55   # isspace() to coalesce multiple spaces.
           56  +#
           57  +# UPDATE: The above is no longer true; there is no such code in fts3.
           58  +# But leave the test in just the same.
           59  +# 
    56     60   
    57     61   set word "\x80xxxxx\x80xxxxx\x80xxxxx\x80xxxxx\x80xxxxx\x80xxxxx\x80"
    58     62   set phrase1 "$word $word $word target $word $word $word"
    59     63   set phrase2 "$word $word $word    target    $word $word $word"
    60     64   
    61     65   db eval {CREATE VIRTUAL TABLE t4 USING fts3(content)}
    62     66   db eval "INSERT INTO t4 (content) VALUES ('$phrase1')"
    63     67   db eval "INSERT INTO t4 (content) VALUES ('$phrase2')"
    64     68   
    65     69   do_test fts3al-1.4 {
    66     70     execsql {SELECT rowid, length(snippet(t4)) FROM t4 WHERE t4 MATCH 'target'}
    67         -} {1 111 2 117}
           71  +} {1 241 2 247}
    68     72   
    69     73   finish_test

Changes to test/fts3near.test.

    72     72   do_test fts3near-1.14 {
    73     73     execsql {SELECT docid FROM t1 WHERE content MATCH 'four NEAR four'}
    74     74   } {} 
    75     75   do_test fts3near-1.15 {
    76     76     execsql {SELECT docid FROM t1 WHERE content MATCH 'one NEAR two NEAR one'}
    77     77   } {3} 
    78     78   
           79  +do_test fts3near-1.16 {
           80  +  execsql {
           81  +    SELECT docid FROM t1 WHERE content MATCH '"one three" NEAR/0 "four five"'
           82  +  }
           83  +} {1} 
           84  +do_test fts3near-1.17 {
           85  +  execsql {
           86  +    SELECT docid FROM t1 WHERE content MATCH '"four five" NEAR/0 "one three"'
           87  +  }
           88  +} {1} 
           89  +
    79     90   
    80     91   # Output format of the offsets() function:
    81     92   #
    82     93   #     <column number> <term number> <starting offset> <number of bytes>
    83     94   #
    84     95   db eval {
    85     96     INSERT INTO t1(content) VALUES('A X B C D A B');
................................................................................
   150    161     execsql {SELECT offsets(t1) FROM t1 WHERE content MATCH 'two NEAR/2 three'}
   151    162   } {{0 0 4 3 0 1 8 5 0 0 14 3 0 1 27 5}}
   152    163   do_test fts3near-3.6 {
   153    164     execsql {
   154    165       SELECT offsets(t1) FROM t1 WHERE content MATCH 'three NEAR/0 "two four"'
   155    166     }
   156    167   } {{0 0 8 5 0 1 14 3 0 2 18 4}}
          168  +breakpoint
   157    169   do_test fts3near-3.7 {
   158    170     execsql {
   159    171       SELECT offsets(t1) FROM t1 WHERE content MATCH '"two four" NEAR/0 three'}
   160    172   } {{0 2 8 5 0 0 14 3 0 1 18 4}}
   161    173   
   162    174   db eval {
   163    175     INSERT INTO t1(content) VALUES('
................................................................................
   166    178       CSS2 builds on CSS1 (see [CSS1]) and, with very few exceptions, all valid CSS1 style sheets are valid CSS2 style sheets. CSS2 supports media-specific style sheets so that authors may tailor the presentation of their documents to visual browsers, aural devices, printers, braille devices, handheld devices, etc. This specification also supports content positioning, downloadable fonts, table layout, features for internationalization, automatic counters and numbering, and some properties related to user interface.
   167    179     ') 
   168    180   }
   169    181   do_test fts3near-4.1 {
   170    182     execsql {
   171    183       SELECT snippet(t1) FROM t1 WHERE content MATCH 'specification NEAR supports'
   172    184     }
   173         -} {{<b>...</b> devices, handheld devices, etc. This <b>specification</b> also <b>supports</b> content positioning, downloadable fonts, <b>...</b>}}
          185  +} {{<b>...</b>braille devices, handheld devices, etc. This <b>specification</b> also <b>supports</b> content positioning, downloadable fonts, table layout<b>...</b>}}
   174    186   
   175    187   do_test fts3near-5.1 {
   176    188     execsql {
   177    189       SELECT docid FROM t1 WHERE content MATCH 'specification attach'
   178    190     }
   179    191   } {2}
   180    192   do_test fts3near-5.2 {

Changes to test/fts3query.test.

    95     95     binary scan $blob $scan($::tcl_platform(byteOrder)) r
    96     96     return $r
    97     97   }
    98     98   db func mit mit
    99     99   
   100    100   do_test fts3query-3.3 {
   101    101     execsql { SELECT mit(matchinfo(foobar)) FROM foobar WHERE foobar MATCH 'the' }
   102         -} {{1 1 3 3}}
          102  +} {{1 1 3 3 1}}
   103    103   
   104    104   finish_test
   105    105   

Changes to test/fts3rnd.test.

   155    155       }
   156    156     }
   157    157   
   158    158     #lsort -uniq -integer $ret
   159    159     set ret
   160    160   }
   161    161   
          162  +# This [proc] is used to test the FTS3 matchinfo() function.
          163  +# 
   162    164   proc simple_token_matchinfo {zToken} {
   163         -  set total(0) 0
   164         -  set total(1) 0
   165         -  set total(2) 0
   166    165   
   167         -  foreach key [lsort -integer [array names ::t1]] {
          166  +  set nDoc(0) 0
          167  +  set nDoc(1) 0
          168  +  set nDoc(2) 0
          169  +  set nHit(0) 0
          170  +  set nHit(1) 0
          171  +  set nHit(2) 0
          172  +
          173  +
          174  +  foreach key [array names ::t1] {
   168    175       set value $::t1($key)
   169         -    set cnt [list]
          176  +    set a($key) [list]
   170    177       foreach i {0 1 2} col $value {
   171         -      set n [llength [lsearch -all $col $zToken]]
   172         -      lappend cnt $n
   173         -      incr total($i) $n
          178  +      set hit [llength [lsearch -all $col $zToken]]
          179  +      lappend a($key) $hit
          180  +      incr nHit($i) $hit
          181  +      if {$hit>0} { incr nDoc($i) }
   174    182       }
   175         -    if {[lindex [lsort $cnt] end]} {
   176         -      lappend ret $key [concat 1 3 XXX $cnt]
          183  +  }
          184  +
          185  +  set ret [list]
          186  +  foreach docid [lsort -integer [array names a]] {
          187  +    if { [lindex [lsort -integer $a($docid)] end] } {
          188  +      set matchinfo [list 1 3]
          189  +      foreach i {0 1 2} hit $a($docid) {
          190  +        lappend matchinfo $hit $nHit($i) $nDoc($i)
          191  +      }
          192  +      lappend ret $docid $matchinfo
   177    193       }
   178    194     }
   179         -  
   180         -  string map [list XXX "$total(0) $total(1) $total(2)"] $ret
          195  +
          196  +  set ret
   181    197   } 
   182    198   
   183    199   proc simple_near {termlist nNear} {
   184    200     set ret [list]
   185    201   
   186    202     foreach {key value} [array get ::t1] {
   187    203       foreach v $value {

Added test/fts3snippet.test.

            1  +# 2010 January 07
            2  +#
            3  +# The author disclaims copyright to this source code.  In place of
            4  +# a legal notice, here is a blessing:
            5  +#
            6  +#    May you do good and not evil.
            7  +#    May you find forgiveness for yourself and forgive others.
            8  +#    May you share freely, never taking more than you give.
            9  +#
           10  +#*************************************************************************
           11  +#
           12  +# The tests in this file test the FTS3 auxillary functions offsets(), 
           13  +# snippet() and matchinfo() work. At time of writing, running this file 
           14  +# provides full coverage of fts3_snippet.c.
           15  +#
           16  +
           17  +set testdir [file dirname $argv0]
           18  +source $testdir/tester.tcl
           19  +
           20  +# If SQLITE_ENABLE_FTS3 is not defined, omit this file.
           21  +ifcapable !fts3 { finish_test ; return }
           22  +source $testdir/fts3_common.tcl
           23  +source $testdir/malloc_common.tcl
           24  +
           25  +set sqlite_fts3_enable_parentheses 1
           26  +set DO_MALLOC_TEST 0
           27  +
           28  +# Transform the list $L to its "normal" form. So that it can be compared to
           29  +# another list with the same set of elements using [string compare].
           30  +#
           31  +proc normalize {L} {
           32  +  set ret [list]
           33  +  foreach l $L {lappend ret $l}
           34  +  return $ret
           35  +}
           36  +
           37  +proc do_offsets_test {name expr args} {
           38  +  set result [list]
           39  +  foreach a $args {
           40  +    lappend result [normalize $a]
           41  +  }
           42  +  do_select_test $name {
           43  +    SELECT offsets(ft) FROM ft WHERE ft MATCH $expr
           44  +  } $result
           45  +}
           46  +  
           47  +# Document text used by a few tests. Contains the English names of all
           48  +# integers between 1 and 300.
           49  +#
           50  +set numbers [normalize {
           51  +  one two three four five six seven eight nine ten eleven twelve thirteen
           52  +  fourteen fifteen sixteen seventeen eighteen nineteen twenty twentyone
           53  +  twentytwo twentythree twentyfour twentyfive twentysix twentyseven
           54  +  twentyeight twentynine thirty thirtyone thirtytwo thirtythree thirtyfour
           55  +  thirtyfive thirtysix thirtyseven thirtyeight thirtynine forty fortyone
           56  +  fortytwo fortythree fortyfour fortyfive fortysix fortyseven fortyeight
           57  +  fortynine fifty fiftyone fiftytwo fiftythree fiftyfour fiftyfive fiftysix
           58  +  fiftyseven fiftyeight fiftynine sixty sixtyone sixtytwo sixtythree sixtyfour
           59  +  sixtyfive sixtysix sixtyseven sixtyeight sixtynine seventy seventyone
           60  +  seventytwo seventythree seventyfour seventyfive seventysix seventyseven
           61  +  seventyeight seventynine eighty eightyone eightytwo eightythree eightyfour
           62  +  eightyfive eightysix eightyseven eightyeight eightynine ninety ninetyone
           63  +  ninetytwo ninetythree ninetyfour ninetyfive ninetysix ninetyseven
           64  +  ninetyeight ninetynine onehundred onehundredone onehundredtwo
           65  +  onehundredthree onehundredfour onehundredfive onehundredsix onehundredseven
           66  +  onehundredeight onehundrednine onehundredten onehundredeleven
           67  +  onehundredtwelve onehundredthirteen onehundredfourteen onehundredfifteen
           68  +  onehundredsixteen onehundredseventeen onehundredeighteen onehundrednineteen
           69  +  onehundredtwenty onehundredtwentyone onehundredtwentytwo
           70  +  onehundredtwentythree onehundredtwentyfour onehundredtwentyfive
           71  +  onehundredtwentysix onehundredtwentyseven onehundredtwentyeight
           72  +  onehundredtwentynine onehundredthirty onehundredthirtyone
           73  +  onehundredthirtytwo onehundredthirtythree onehundredthirtyfour
           74  +  onehundredthirtyfive onehundredthirtysix onehundredthirtyseven
           75  +  onehundredthirtyeight onehundredthirtynine onehundredforty
           76  +  onehundredfortyone onehundredfortytwo onehundredfortythree
           77  +  onehundredfortyfour onehundredfortyfive onehundredfortysix
           78  +  onehundredfortyseven onehundredfortyeight onehundredfortynine
           79  +  onehundredfifty onehundredfiftyone onehundredfiftytwo onehundredfiftythree
           80  +  onehundredfiftyfour onehundredfiftyfive onehundredfiftysix
           81  +  onehundredfiftyseven onehundredfiftyeight onehundredfiftynine
           82  +  onehundredsixty onehundredsixtyone onehundredsixtytwo onehundredsixtythree
           83  +  onehundredsixtyfour onehundredsixtyfive onehundredsixtysix
           84  +  onehundredsixtyseven onehundredsixtyeight onehundredsixtynine
           85  +  onehundredseventy onehundredseventyone onehundredseventytwo
           86  +  onehundredseventythree onehundredseventyfour onehundredseventyfive
           87  +  onehundredseventysix onehundredseventyseven onehundredseventyeight
           88  +  onehundredseventynine onehundredeighty onehundredeightyone
           89  +  onehundredeightytwo onehundredeightythree onehundredeightyfour
           90  +  onehundredeightyfive onehundredeightysix onehundredeightyseven
           91  +  onehundredeightyeight onehundredeightynine onehundredninety
           92  +  onehundredninetyone onehundredninetytwo onehundredninetythree
           93  +  onehundredninetyfour onehundredninetyfive onehundredninetysix
           94  +  onehundredninetyseven onehundredninetyeight onehundredninetynine twohundred
           95  +  twohundredone twohundredtwo twohundredthree twohundredfour twohundredfive
           96  +  twohundredsix twohundredseven twohundredeight twohundrednine twohundredten
           97  +  twohundredeleven twohundredtwelve twohundredthirteen twohundredfourteen
           98  +  twohundredfifteen twohundredsixteen twohundredseventeen twohundredeighteen
           99  +  twohundrednineteen twohundredtwenty twohundredtwentyone twohundredtwentytwo
          100  +  twohundredtwentythree twohundredtwentyfour twohundredtwentyfive
          101  +  twohundredtwentysix twohundredtwentyseven twohundredtwentyeight
          102  +  twohundredtwentynine twohundredthirty twohundredthirtyone
          103  +  twohundredthirtytwo twohundredthirtythree twohundredthirtyfour
          104  +  twohundredthirtyfive twohundredthirtysix twohundredthirtyseven
          105  +  twohundredthirtyeight twohundredthirtynine twohundredforty
          106  +  twohundredfortyone twohundredfortytwo twohundredfortythree
          107  +  twohundredfortyfour twohundredfortyfive twohundredfortysix
          108  +  twohundredfortyseven twohundredfortyeight twohundredfortynine
          109  +  twohundredfifty twohundredfiftyone twohundredfiftytwo twohundredfiftythree
          110  +  twohundredfiftyfour twohundredfiftyfive twohundredfiftysix
          111  +  twohundredfiftyseven twohundredfiftyeight twohundredfiftynine
          112  +  twohundredsixty twohundredsixtyone twohundredsixtytwo twohundredsixtythree
          113  +  twohundredsixtyfour twohundredsixtyfive twohundredsixtysix
          114  +  twohundredsixtyseven twohundredsixtyeight twohundredsixtynine
          115  +  twohundredseventy twohundredseventyone twohundredseventytwo
          116  +  twohundredseventythree twohundredseventyfour twohundredseventyfive
          117  +  twohundredseventysix twohundredseventyseven twohundredseventyeight
          118  +  twohundredseventynine twohundredeighty twohundredeightyone
          119  +  twohundredeightytwo twohundredeightythree twohundredeightyfour
          120  +  twohundredeightyfive twohundredeightysix twohundredeightyseven
          121  +  twohundredeightyeight twohundredeightynine twohundredninety
          122  +  twohundredninetyone twohundredninetytwo twohundredninetythree
          123  +  twohundredninetyfour twohundredninetyfive twohundredninetysix
          124  +  twohundredninetyseven twohundredninetyeight twohundredninetynine
          125  +  threehundred
          126  +}]
          127  +
          128  +foreach {DO_MALLOC_TEST enc} {
          129  +  0 utf8
          130  +  1 utf8
          131  +  1 utf16
          132  +} {
          133  +
          134  +  db close
          135  +  file delete -force test.db
          136  +  sqlite3 db test.db
          137  +  db eval "PRAGMA encoding = \"$enc\""
          138  +
          139  +  # Set variable $T to the test name prefix for this iteration of the loop.
          140  +  #
          141  +  set T "fts3snippet-$enc"
          142  +
          143  +  ##########################################################################
          144  +  # Test the offset function.
          145  +  #
          146  +  do_test $T.1.1 {
          147  +    execsql {
          148  +      CREATE VIRTUAL TABLE ft USING fts3;
          149  +      INSERT INTO ft VALUES('xxx xxx xxx xxx');
          150  +    }
          151  +  } {}
          152  +  do_offsets_test $T.1.2 {xxx} {0 0 0 3 0 0 4 3 0 0 8 3 0 0 12 3}
          153  +  do_offsets_test $T.1.3 {"xxx xxx"} {
          154  +      0 0  0 3     0 0  4 3     0 1  4 3     0 0  8 3 
          155  +      0 1  8 3     0 1 12 3
          156  +  }
          157  +  do_offsets_test $T.1.4 {"xxx xxx" xxx} {
          158  +      0 0  0 3     0 2  0 3     0 0  4 3     0 1  4 3 
          159  +      0 2  4 3     0 0  8 3     0 1  8 3     0 2  8 3 
          160  +      0 1 12 3     0 2 12 3
          161  +  }
          162  +  do_offsets_test $T.1.5 {xxx "xxx xxx"} {
          163  +      0 0  0 3     0 1  0 3     0 0  4 3     0 1  4 3 
          164  +      0 2  4 3     0 0  8 3     0 1  8 3     0 2  8 3 
          165  +      0 0 12 3     0 2 12 3
          166  +  }
          167  +
          168  +  do_test $T.2.1 {
          169  +    set v1 [lrange $numbers 0 99]
          170  +    execsql {
          171  +      DROP TABLE IF EXISTS ft;
          172  +      CREATE VIRTUAL TABLE ft USING fts3(a, b);
          173  +      INSERT INTO ft VALUES($v1, $numbers);
          174  +      INSERT INTO ft VALUES($v1, NULL);
          175  +    }
          176  +  } {}
          177  +
          178  +  set off [string first "twohundred " $numbers]
          179  +  do_offsets_test $T.2.1 {twohundred} [list 1 0 $off 10]
          180  +
          181  +  set off [string first "onehundred " $numbers]
          182  +  do_offsets_test $T.2.2 {onehundred} \
          183  +    [list 0 0 $off 10 1 0 $off 10] [list 0 0 $off 10]
          184  +
          185  +  # Test a corruption case:
          186  +  execsql { UPDATE ft_content SET c1b = 'hello world' WHERE c1b = $numbers }
          187  +  do_error_test $T.2.3 {
          188  +    SELECT offsets(ft) FROM ft WHERE ft MATCH 'onehundred'
          189  +  } {database disk image is malformed}
          190  +  
          191  +  ##########################################################################
          192  +  # Test the snippet function.
          193  +  #
          194  +  proc do_snippet_test {name expr iCol nTok args} {
          195  +    set res [list]
          196  +    foreach a $args { lappend res [string trim $a] }
          197  +    do_select_test $name {
          198  +      SELECT snippet(ft,'{','}','...',$iCol,$nTok) FROM ft WHERE ft MATCH $expr
          199  +    } $res
          200  +  }
          201  +  do_test $T.3.1 {
          202  +    execsql {
          203  +      DROP TABLE IF EXISTS ft;
          204  +      CREATE VIRTUAL TABLE ft USING fts3;
          205  +      INSERT INTO ft VALUES('one two three four five six seven eight nine ten');
          206  +    }
          207  +  } {}
          208  +  do_snippet_test $T.3.2  one    0 5 "{one} two three four five..."
          209  +  do_snippet_test $T.3.3  two    0 5 "one {two} three four five..."
          210  +  do_snippet_test $T.3.4  three  0 5 "one two {three} four five..."
          211  +  do_snippet_test $T.3.5  four   0 5 "...two three {four} five six..."
          212  +  do_snippet_test $T.3.6  five   0 5 "...three four {five} six seven..."
          213  +  do_snippet_test $T.3.7  six    0 5 "...four five {six} seven eight..."
          214  +  do_snippet_test $T.3.8  seven  0 5 "...five six {seven} eight nine..."
          215  +  do_snippet_test $T.3.9  eight  0 5 "...six seven {eight} nine ten"
          216  +  do_snippet_test $T.3.10 nine   0 5 "...six seven eight {nine} ten"
          217  +  do_snippet_test $T.3.11 ten    0 5 "...six seven eight nine {ten}"
          218  +  
          219  +  do_test $T.4.1 {
          220  +    execsql {
          221  +      INSERT INTO ft VALUES(
          222  +           'one two three four five '
          223  +        || 'six seven eight nine ten '
          224  +        || 'eleven twelve thirteen fourteen fifteen '
          225  +        || 'sixteen seventeen eighteen nineteen twenty '
          226  +        || 'one two three four five '
          227  +        || 'six seven eight nine ten '
          228  +        || 'eleven twelve thirteen fourteen fifteen '
          229  +        || 'sixteen seventeen eighteen nineteen twenty'
          230  +      );
          231  +    }
          232  +  } {}
          233  +  
          234  +  do_snippet_test $T.4.2 {one nine} 0 5 {
          235  +     {one} two three...eight {nine} ten
          236  +  } {
          237  +     {one} two three...eight {nine} ten...
          238  +  }
          239  +  
          240  +  do_snippet_test $T.4.3 {one nine} 0 -5 {
          241  +     {one} two three four five...six seven eight {nine} ten
          242  +  } {
          243  +     {one} two three four five...seven eight {nine} ten eleven...
          244  +  }
          245  +  do_snippet_test $T.4.3 {one nineteen} 0 -5 {
          246  +     ...eighteen {nineteen} twenty {one} two...
          247  +  }
          248  +  do_snippet_test $T.4.4 {two nineteen} 0 -5 {
          249  +     ...eighteen {nineteen} twenty one {two}...
          250  +  }
          251  +  do_snippet_test $T.4.5 {three nineteen} 0 -5 {
          252  +     ...{nineteen} twenty one two {three}...
          253  +  }
          254  +  
          255  +  do_snippet_test $T.4.6 {four nineteen} 0 -5 {
          256  +     ...two three {four} five six...seventeen eighteen {nineteen} twenty one...
          257  +  }
          258  +  do_snippet_test $T.4.7 {four NEAR nineteen} 0 -5 {
          259  +     ...seventeen eighteen {nineteen} twenty one...two three {four} five six...
          260  +  }
          261  +  
          262  +  do_snippet_test $T.4.8 {four nineteen} 0 5 {
          263  +     ...three {four} five...eighteen {nineteen} twenty...
          264  +  }
          265  +  do_snippet_test $T.4.9 {four NEAR nineteen} 0 5 {
          266  +     ...eighteen {nineteen} twenty...three {four} five...
          267  +  }
          268  +  do_snippet_test $T.4.10 {four NEAR nineteen} 0 -5 {
          269  +     ...seventeen eighteen {nineteen} twenty one...two three {four} five six...
          270  +  }
          271  +  do_snippet_test $T.4.11 {four NOT (nineteen twentyone)} 0 5 {
          272  +     ...two three {four} five six...
          273  +  } {
          274  +     ...two three {four} five six...
          275  +  }
          276  +  do_snippet_test $T.4.12 {four OR nineteen NEAR twentyone} 0 5 {
          277  +     ...two three {four} five six...
          278  +  } {
          279  +     ...two three {four} five six...
          280  +  }
          281  +  
          282  +  do_test $T.5.1 {
          283  +    execsql {
          284  +      DROP TABLE IF EXISTS ft;
          285  +      CREATE VIRTUAL TABLE ft USING fts3(a, b, c);
          286  +      INSERT INTO ft VALUES(
          287  +        'one two three four five', 
          288  +        'four five six seven eight', 
          289  +        'seven eight nine ten eleven'
          290  +      );
          291  +    }
          292  +  } {}
          293  +  
          294  +  do_snippet_test $T.5.2 {five} -1 3 {...three four {five}}
          295  +  do_snippet_test $T.5.3 {five}  0 3 {...three four {five}}
          296  +  do_snippet_test $T.5.4 {five}  1 3 {four {five} six...}
          297  +  do_snippet_test $T.5.5 {five}  2 3 {seven eight nine...}
          298  +  
          299  +  do_test $T.5.6 {
          300  +    execsql { UPDATE ft SET b = NULL }
          301  +  } {}
          302  +  
          303  +  do_snippet_test $T.5.7  {five} -1 3 {...three four {five}}
          304  +  do_snippet_test $T.5.8  {five}  0 3 {...three four {five}}
          305  +  do_snippet_test $T.5.9  {five}  1 3 {}
          306  +  do_snippet_test $T.5.10 {five}  2 3 {seven eight nine...}
          307  +  
          308  +  do_snippet_test $T.5.11 {one "seven eight nine"} -1 -3 {
          309  +    {one} two three...{seven} {eight} {nine}...
          310  +  }
          311  +
          312  +  do_test $T.6.1 {
          313  +    execsql {
          314  +      DROP TABLE IF EXISTS ft;
          315  +      CREATE VIRTUAL TABLE ft USING fts3(x);
          316  +      INSERT INTO ft VALUES($numbers);
          317  +    }
          318  +  } {}
          319  +  do_snippet_test $T.6.2 {
          320  +    one fifty onehundred onehundredfifty twohundredfifty threehundred
          321  +  } -1 4 {
          322  +    {one}...{fifty}...{onehundred}...{onehundredfifty}...
          323  +  }
          324  +  do_snippet_test $T.6.3 {
          325  +    one fifty onehundred onehundredfifty twohundredfifty threehundred
          326  +  } -1 -4 {
          327  +    {one} two three four...fortyeight fortynine {fifty} fiftyone...ninetyeight ninetynine {onehundred} onehundredone...onehundredfortyeight onehundredfortynine {onehundredfifty} onehundredfiftyone...
          328  +  }
          329  +
          330  +  do_test $T.7.1 {
          331  +    execsql {
          332  +      BEGIN;
          333  +        DROP TABLE IF EXISTS ft;
          334  +        CREATE VIRTUAL TABLE ft USING fts3(x);
          335  +    }
          336  +    set testresults [list]
          337  +    for {set i 1} {$i < 150} {incr i} {
          338  +      set commas [string repeat , $i]
          339  +      execsql {INSERT INTO ft VALUES('one' || $commas || 'two')}
          340  +      lappend testresults "{one}$commas{two}"
          341  +    }
          342  +    execsql COMMIT
          343  +  } {}
          344  +  do_snippet_test $T.7.2 {one two} -1 3 {*}$testresults
          345  +  
          346  +  ##########################################################################
          347  +  # Test the matchinfo function.
          348  +  #
          349  +  proc mit {blob} {
          350  +    set scan(littleEndian) i*
          351  +    set scan(bigEndian) I*
          352  +    binary scan $blob $scan($::tcl_platform(byteOrder)) r
          353  +    return $r
          354  +  }
          355  +  db func mit mit
          356  +  proc do_matchinfo_test {name expr args} {
          357  +    set res [list]
          358  +    foreach a $args { lappend res [normalize $a] }
          359  +    do_select_test $name {
          360  +      SELECT mit(matchinfo(ft)) FROM ft WHERE ft MATCH $expr
          361  +    } $res
          362  +  }
          363  +  do_test $T.8.1 {
          364  +    set ten {one two three four five six seven eight nine ten}
          365  +    execsql {
          366  +      DROP TABLE IF EXISTS ft;
          367  +      CREATE VIRTUAL TABLE ft USING fts3;
          368  +      INSERT INTO ft VALUES($ten);
          369  +      INSERT INTO ft VALUES($ten || ' ' || $ten);
          370  +    }
          371  +  } {}
          372  +  
          373  +  do_matchinfo_test $T.8.2 "one" {1 1  1 3 2} {1 1  2 3 2}
          374  +  do_matchinfo_test $T.8.3 "one NEAR/3 ten" {2 1  1 1 1 1 1 1}
          375  +  do_matchinfo_test $T.8.4 "five NEAR/4 ten" \
          376  +    {2 1  1 3 2  1 3 2} {2 1  2 3 2  2 3 2}
          377  +  do_matchinfo_test $T.8.5 "six NEAR/3 ten NEAR/3 two" \
          378  +    {3 1  1 1 1  1 1 1  1 1 1}
          379  +  do_matchinfo_test $T.8.6 "five NEAR/4 ten NEAR/3 two" \
          380  +    {3 1  2 2 1  1 1 1  1 1 1}
          381  +
          382  +  do_test $T.9.1 {
          383  +    execsql {
          384  +      DROP TABLE IF EXISTS ft;
          385  +      CREATE VIRTUAL TABLE ft USING fts3(x, y);
          386  +    }
          387  +    foreach n {1 2 3} {
          388  +      set v1 [lrange $numbers 0 [expr $n*100]]
          389  +      set v2 [string trim [string repeat "$numbers " $n]]
          390  +      set docid [expr $n * 1000000]
          391  +      execsql { INSERT INTO ft(docid, x, y) VALUES($docid, $v1, $v2) }
          392  +    }
          393  +  } {}
          394  +  do_matchinfo_test $T.9.2 {two*}     \
          395  +    { 1 2    1   105 3   101 606 3}   \
          396  +    { 1 2    3   105 3   202 606 3}   \
          397  +    { 1 2    101 105 3   303 606 3}
          398  +
          399  +  do_matchinfo_test $T.9.4 {"one* two*"}  \
          400  +    { 1 2    1 5 3   2 12 3}              \
          401  +    { 1 2    2 5 3   4 12 3}              \
          402  +    { 1 2    2 5 3   6 12 3}
          403  +
          404  +  do_matchinfo_test $T.9.5 {twohundredfifty}  \
          405  +    { 1 2    0 1 1   1 6 3}                   \
          406  +    { 1 2    0 1 1   2 6 3}                   \
          407  +    { 1 2    1 1 1   3 6 3}
          408  +
          409  +  do_matchinfo_test $T.9.6 {"threehundred one"} \
          410  +    { 1 2    0 0 0   1 3 2}                     \
          411  +    { 1 2    0 0 0   2 3 2}
          412  +
          413  +  do_matchinfo_test $T.9.7 {one OR fivehundred} \
          414  +    { 2 2    1 3 3   1 6 3   0 0 0   0 0 0 }    \
          415  +    { 2 2    1 3 3   2 6 3   0 0 0   0 0 0 }    \
          416  +    { 2 2    1 3 3   3 6 3   0 0 0   0 0 0 }
          417  +
          418  +  do_matchinfo_test $T.9.8 {two OR "threehundred one"} \
          419  +    { 2 2    1 3 3   1 6 3   0 0 0   0 3 2 }           \
          420  +    { 2 2    1 3 3   2 6 3   0 0 0   1 3 2 }           \
          421  +    { 2 2    1 3 3   3 6 3   0 0 0   2 3 2 }
          422  +
          423  +  do_select_test $T.9.9 {
          424  +    SELECT mit(matchinfo(ft)), mit(matchinfo(ft))
          425  +    FROM ft WHERE ft MATCH 'two OR "threehundred one"' 
          426  +  } [normalize {
          427  +    {2 2 1 3 3 1 6 3 0 0 0 0 3 2}
          428  +    {2 2 1 3 3 1 6 3 0 0 0 0 3 2}
          429  +    {2 2 1 3 3 2 6 3 0 0 0 1 3 2}
          430  +    {2 2 1 3 3 2 6 3 0 0 0 1 3 2}
          431  +    {2 2 1 3 3 3 6 3 0 0 0 2 3 2}          
          432  +    {2 2 1 3 3 3 6 3 0 0 0 2 3 2}
          433  +  }]
          434  +}
          435  +
          436  +set sqlite_fts3_enable_parentheses 0
          437  +finish_test

Changes to test/insert4.test.

   297    297       CREATE INDEX t2_i1 ON t2(x ASC, y COLLATE RTRIM);
   298    298       INSERT INTO t2 SELECT * FROM t3;
   299    299     }
   300    300     set ::sqlite3_xferopt_count
   301    301   } {0}
   302    302   
   303    303   
   304         -
          304  +do_test insert4-6.5 {
          305  +  execsql {
          306  +    CREATE TABLE t6a(x CHECK( x<>'abc' ));
          307  +    INSERT INTO t6a VALUES('ABC');
          308  +    SELECT * FROM t6a;
          309  +  }
          310  +} {ABC}
          311  +do_test insert4-6.6 {
          312  +  execsql {
          313  +    CREATE TABLE t6b(x CHECK( x<>'abc' COLLATE nocase ));
          314  +  }
          315  +  catchsql {
          316  +    INSERT INTO t6b SELECT * FROM t6a;
          317  +  }
          318  +} {1 {constraint failed}}
          319  +do_test insert4-6.7 {
          320  +  execsql {
          321  +    DROP TABLE t6b;
          322  +    CREATE TABLE t6b(x CHECK( x COLLATE nocase <>'abc' ));
          323  +  }
          324  +  catchsql {
          325  +    INSERT INTO t6b SELECT * FROM t6a;
          326  +  }
          327  +} {1 {constraint failed}}
   305    328   
   306    329   finish_test

Changes to test/minmax3.test.

   170    170   do_test minmax3-2.7 {
   171    171     execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b<1; }
   172    172   } {{}}
   173    173   do_test minmax3-2.8 {
   174    174     execsql { SELECT min(b) FROM t2 WHERE a = 3 AND b<1; }
   175    175   } {{}}
   176    176   
   177         -do_test minmax3-2.1 {
          177  +do_test minmax3-3.1 {
   178    178     execsql {
   179    179       DROP TABLE t2;
   180    180       CREATE TABLE t2(a, b);
   181    181       CREATE INDEX i3 ON t2(a, b DESC);
   182    182       INSERT INTO t2 VALUES(1, NULL);
   183    183       INSERT INTO t2 VALUES(1, 1);
   184    184       INSERT INTO t2 VALUES(1, 2);
................................................................................
   188    188       INSERT INTO t2 VALUES(2, 2);
   189    189       INSERT INTO t2 VALUES(2, 3);
   190    190       INSERT INTO t2 VALUES(3, 1);
   191    191       INSERT INTO t2 VALUES(3, 2);
   192    192       INSERT INTO t2 VALUES(3, 3);
   193    193     }
   194    194   } {}
   195         -do_test minmax3-2.2 {
          195  +do_test minmax3-3.2 {
   196    196     execsql { SELECT min(b) FROM t2 WHERE a = 1; }
   197    197   } {1}
   198         -do_test minmax3-2.3 {
          198  +do_test minmax3-3.3 {
   199    199     execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b>1; }
   200    200   } {2}
   201         -do_test minmax3-2.4 {
          201  +do_test minmax3-3.4 {
   202    202     execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b>-1; }
   203    203   } {1}
   204         -do_test minmax3-2.5 {
          204  +do_test minmax3-3.5 {
   205    205     execsql { SELECT min(b) FROM t2 WHERE a = 1; }
   206    206   } {1}
   207         -do_test minmax3-2.6 {
          207  +do_test minmax3-3.6 {
   208    208     execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b<2; }
   209    209   } {1}
   210         -do_test minmax3-2.7 {
          210  +do_test minmax3-3.7 {
   211    211     execsql { SELECT min(b) FROM t2 WHERE a = 1 AND b<1; }
   212    212   } {{}}
   213         -do_test minmax3-2.8 {
          213  +do_test minmax3-3.8 {
   214    214     execsql { SELECT min(b) FROM t2 WHERE a = 3 AND b<1; }
   215    215   } {{}}
          216  +
          217  +do_test minmax3-4.1 {
          218  +  execsql {
          219  +    CREATE TABLE t4(x);
          220  +    INSERT INTO t4 VALUES('abc');
          221  +    INSERT INTO t4 VALUES('BCD');
          222  +    SELECT max(x) FROM t4;
          223  +  }
          224  +} {abc}
          225  +do_test minmax3-4.2 {
          226  +  execsql {
          227  +    SELECT max(x COLLATE nocase) FROM t4;
          228  +  }
          229  +} {BCD}
          230  +do_test minmax3-4.3 {
          231  +  execsql {
          232  +    SELECT max(x), max(x COLLATE nocase) FROM t4;
          233  +  }
          234  +} {abc BCD}
          235  +do_test minmax3-4.4 {
          236  +  execsql {
          237  +    SELECT max(x COLLATE binary), max(x COLLATE nocase) FROM t4;
          238  +  }
          239  +} {abc BCD}
          240  +do_test minmax3-4.5 {
          241  +  execsql {
          242  +    SELECT max(x COLLATE nocase), max(x COLLATE rtrim) FROM t4;
          243  +  }
          244  +} {BCD abc}
          245  +do_test minmax3-4.6 {
          246  +  execsql {
          247  +    SELECT max(x COLLATE nocase), max(x) FROM t4;
          248  +  }
          249  +} {BCD abc}
          250  +do_test minmax3-4.10 {
          251  +  execsql {
          252  +    SELECT min(x) FROM t4;
          253  +  }
          254  +} {BCD}
          255  +do_test minmax3-4.11 {
          256  +  execsql {
          257  +    SELECT min(x COLLATE nocase) FROM t4;
          258  +  }
          259  +} {abc}
          260  +do_test minmax3-4.12 {
          261  +  execsql {
          262  +    SELECT min(x), min(x COLLATE nocase) FROM t4;
          263  +  }
          264  +} {BCD abc}
          265  +do_test minmax3-4.13 {
          266  +  execsql {
          267  +    SELECT min(x COLLATE binary), min(x COLLATE nocase) FROM t4;
          268  +  }
          269  +} {BCD abc}
          270  +do_test minmax3-4.14 {
          271  +  execsql {
          272  +    SELECT min(x COLLATE nocase), min(x COLLATE rtrim) FROM t4;
          273  +  }
          274  +} {abc BCD}
          275  +do_test minmax3-4.15 {
          276  +  execsql {
          277  +    SELECT min(x COLLATE nocase), min(x) FROM t4;
          278  +  }
          279  +} {abc BCD}
          280  +
   216    281   
   217    282   finish_test

Changes to test/whereA.test.

    29     29     db eval {
    30     30       PRAGMA reverse_unordered_selects=1;
    31     31       SELECT * FROM t1;
    32     32     }
    33     33   } {3 4.53 {} 2 hello world 1 2 3}
    34     34   
    35     35   do_test whereA-1.3 {
           36  +  db close
           37  +  sqlite3 db test.db
           38  +  db eval {
           39  +    PRAGMA reverse_unordered_selects=1;
           40  +    SELECT * FROM t1;
           41  +  }
           42  +} {3 4.53 {} 2 hello world 1 2 3}
           43  +do_test whereA-1.4 {
           44  +  db close
           45  +  sqlite3 db test.db
    36     46     db eval {
    37     47       PRAGMA reverse_unordered_selects=1;
    38     48       SELECT * FROM t1 ORDER BY rowid;
    39     49     }
           50  +} {1 2 3 2 hello world 3 4.53 {}}
           51  +do_test whereA-1.5 {
           52  +  db eval {
           53  +    VACUUM;
           54  +    SELECT * FROM t1 ORDER BY rowid;
           55  +  }
    40     56   } {1 2 3 2 hello world 3 4.53 {}}
           57  +do_test whereA-1.6 {
           58  +  db eval {
           59  +    PRAGMA reverse_unordered_selects;
           60  +  }
           61  +} {1}
           62  +do_test whereA-1.7 {
           63  +  db close
           64  +  sqlite3 db test.db
           65  +  db eval {
           66  +    PRAGMA reverse_unordered_selects=1;
           67  +    VACUUM;
           68  +    SELECT * FROM t1;
           69  +  }
           70  +} {3 4.53 {} 2 hello world 1 2 3}
    41     71   
    42     72   do_test whereA-2.1 {
    43     73     db eval {
    44     74       PRAGMA reverse_unordered_selects=0;
    45     75       SELECT * FROM t1 WHERE a>0;
    46     76     }
    47     77   } {1 2 3 2 hello world 3 4.53 {}}

Changes to tool/lemon.c.

   405    405   /********************** New code to implement the "acttab" module ***********/
   406    406   /*
   407    407   ** This module implements routines use to construct the yy_action[] table.
   408    408   */
   409    409   
   410    410   /*
   411    411   ** The state of the yy_action table under construction is an instance of
   412         -** the following structure
          412  +** the following structure.
          413  +**
          414  +** The yy_action table maps the pair (state_number, lookahead) into an
          415  +** action_number.  The table is an array of integers pairs.  The state_number
          416  +** determines an initial offset into the yy_action array.  The lookahead
          417  +** value is then added to this initial offset to get an index X into the
          418  +** yy_action array. If the aAction[X].lookahead equals the value of the
          419  +** of the lookahead input, then the value of the action_number output is
          420  +** aAction[X].action.  If the lookaheads do not match then the
          421  +** default action for the state_number is returned.
          422  +**
          423  +** All actions associated with a single state_number are first entered
          424  +** into aLookahead[] using multiple calls to acttab_action().  Then the 
          425  +** actions for that single state_number are placed into the aAction[] 
          426  +** array with a single call to acttab_insert().  The acttab_insert() call
          427  +** also resets the aLookahead[] array in preparation for the next
          428  +** state number.
   413    429   */
   414    430   typedef struct acttab acttab;
   415    431   struct acttab {
   416    432     int nAction;                 /* Number of used slots in aAction[] */
   417    433     int nActionAlloc;            /* Slots allocated for aAction[] */
   418    434     struct {
   419    435       int lookahead;             /* Value of the lookahead token */
................................................................................
   450    466       fprintf(stderr,"Unable to allocate memory for a new acttab.");
   451    467       exit(1);
   452    468     }
   453    469     memset(p, 0, sizeof(*p));
   454    470     return p;
   455    471   }
   456    472   
   457         -/* Add a new action to the current transaction set
          473  +/* Add a new action to the current transaction set.  
          474  +**
          475  +** This routine is called once for each lookahead for a particular
          476  +** state.
   458    477   */
   459    478   void acttab_action(acttab *p, int lookahead, int action){
   460    479     if( p->nLookahead>=p->nLookaheadAlloc ){
   461    480       p->nLookaheadAlloc += 25;
   462    481       p->aLookahead = realloc( p->aLookahead,
   463    482                                sizeof(p->aLookahead[0])*p->nLookaheadAlloc );
   464    483       if( p->aLookahead==0 ){
................................................................................
   487    506   ** into the current action table.  Then reset the transaction set back
   488    507   ** to an empty set in preparation for a new round of acttab_action() calls.
   489    508   **
   490    509   ** Return the offset into the action table of the new transaction.
   491    510   */
   492    511   int acttab_insert(acttab *p){
   493    512     int i, j, k, n;
   494         -  int nActtab;     /* Number of slots in the p->aAction[] table */
   495    513     assert( p->nLookahead>0 );
   496    514   
   497    515     /* Make sure we have enough space to hold the expanded action table
   498    516     ** in the worst case.  The worst case occurs if the transaction set
   499    517     ** must be appended to the current action table
   500    518     */
   501    519     n = p->mxLookahead + 1;
   502         -  nActtab = p->nAction + n;
   503         -  if( nActtab >= p->nActionAlloc ){
          520  +  if( p->nAction + n >= p->nActionAlloc ){
   504    521       int oldAlloc = p->nActionAlloc;
   505    522       p->nActionAlloc = p->nAction + n + p->nActionAlloc + 20;
   506    523       p->aAction = realloc( p->aAction,
   507    524                             sizeof(p->aAction[0])*p->nActionAlloc);
   508    525       if( p->aAction==0 ){
   509    526         fprintf(stderr,"malloc failed\n");
   510    527         exit(1);
................................................................................
   511    528       }
   512    529       for(i=oldAlloc; i<p->nActionAlloc; i++){
   513    530         p->aAction[i].lookahead = -1;
   514    531         p->aAction[i].action = -1;
   515    532       }
   516    533     }
   517    534   
   518         -  /* Scan the existing action table looking for an offset where we can
   519         -  ** insert the current transaction set.  Fall out of the loop when that
   520         -  ** offset is found.  In the worst case, we fall out of the loop when
   521         -  ** i reaches nActtab, which means we append the new transaction set.
          535  +  /* Scan the existing action table looking for an offset that is a 
          536  +  ** duplicate of the current transaction set.  Fall out of the loop
          537  +  ** if and when the duplicate is found.
   522    538     **
   523    539     ** i is the index in p->aAction[] where p->mnLookahead is inserted.
   524    540     */
   525         -  for(i=nActtab-1; i>=0; i--){
   526         -    /* First look for an existing action table entry that can be reused */
          541  +  for(i=p->nAction-1; i>=0; i--){
   527    542       if( p->aAction[i].lookahead==p->mnLookahead ){
          543  +      /* All lookaheads and actions in the aLookahead[] transaction
          544  +      ** must match against the candidate aAction[i] entry. */
   528    545         if( p->aAction[i].action!=p->mnAction ) continue;
   529    546         for(j=0; j<p->nLookahead; j++){
   530    547           k = p->aLookahead[j].lookahead - p->mnLookahead + i;
   531    548           if( k<0 || k>=p->nAction ) break;
   532    549           if( p->aLookahead[j].lookahead!=p->aAction[k].lookahead ) break;
   533    550           if( p->aLookahead[j].action!=p->aAction[k].action ) break;
   534    551         }
   535    552         if( j<p->nLookahead ) continue;
          553  +
          554  +      /* No possible lookahead value that is not in the aLookahead[]
          555  +      ** transaction is allowed to match aAction[i] */
   536    556         n = 0;
   537    557         for(j=0; j<p->nAction; j++){
   538    558           if( p->aAction[j].lookahead<0 ) continue;
   539    559           if( p->aAction[j].lookahead==j+p->mnLookahead-i ) n++;
   540    560         }
   541    561         if( n==p->nLookahead ){
   542         -        break;  /* Same as a prior transaction set */
          562  +        break;  /* An exact match is found at offset i */
   543    563         }
   544    564       }
   545    565     }
          566  +
          567  +  /* If no existing offsets exactly match the current transaction, find an
          568  +  ** an empty offset in the aAction[] table in which we can add the
          569  +  ** aLookahead[] transaction.
          570  +  */
   546    571     if( i<0 ){
   547         -    /* If no reusable entry is found, look for an empty slot */
   548         -    for(i=0; i<nActtab; i++){
          572  +    /* Look for holes in the aAction[] table that fit the current
          573  +    ** aLookahead[] transaction.  Leave i set to the offset of the hole.
          574  +    ** If no holes are found, i is left at p->nAction, which means the
          575  +    ** transaction will be appended. */
          576  +    for(i=0; i<p->nActionAlloc - p->mxLookahead; i++){
   549    577         if( p->aAction[i].lookahead<0 ){
   550    578           for(j=0; j<p->nLookahead; j++){
   551    579             k = p->aLookahead[j].lookahead - p->mnLookahead + i;
   552    580             if( k<0 ) break;
   553    581             if( p->aAction[k].lookahead>=0 ) break;
   554    582           }
   555    583           if( j<p->nLookahead ) continue;

Added tool/restore_jrnl.tcl.

            1  +# 2010 January 7
            2  +#
            3  +# The author disclaims copyright to this source code.  In place of
            4  +# a legal notice, here is a blessing:
            5  +#
            6  +#    May you do good and not evil.
            7  +#    May you find forgiveness for yourself and forgive others.
            8  +#    May you share freely, never taking more than you give.
            9  +#
           10  +#***********************************************************************
           11  +# This file implements utility functions for SQLite library.
           12  +#
           13  +# This file attempts to restore the header of a journal.
           14  +# This may be useful for rolling-back the last committed 
           15  +# transaction from a recovered journal.
           16  +#
           17  +
           18  +package require sqlite3
           19  +
           20  +set parm_error 0
           21  +set fix_chksums 0
           22  +set dump_pages 0
           23  +set db_name ""
           24  +
           25  +for {set i 0} {$i<$argc} {incr i} {
           26  +  if {[lindex $argv $i] == "-fix_chksums"} {
           27  +    set fix_chksums -1
           28  +  } elseif {[lindex $argv $i] == "-dump_pages"} {
           29  +    set dump_pages -1
           30  +  } elseif {$db_name == ""} {
           31  +    set db_name [lindex $argv $i]
           32  +    set jrnl_name $db_name-journal
           33  +  } else {
           34  +    set parm_error -1
           35  +  }
           36  +}
           37  +if {$parm_error || $db_name == ""} {
           38  +  puts "USAGE: restore_jrnl.tcl \[-fix_chksums\] \[-dump_pages\] db_name"
           39  +  puts "Example: restore_jrnl.tcl foo.sqlite"
           40  +  return
           41  +}
           42  +
           43  +# is there a way to determine this?
           44  +set sectsz 512
           45  +
           46  +# Copy file $from into $to
           47  +#
           48  +proc copy_file {from to} {
           49  +  file copy -force $from $to
           50  +}
           51  +
           52  +# Execute some SQL
           53  +#
           54  +proc catchsql {sql} {
           55  +  set rc [catch {uplevel [list db eval $sql]} msg]
           56  +  list $rc $msg
           57  +}
           58  +
           59  +# Perform a test
           60  +#
           61  +proc do_test {name cmd expected} {
           62  +  puts -nonewline "$name ..."
           63  +  set res [uplevel $cmd]
           64  +  if {$res eq $expected} {
           65  +    puts Ok
           66  +  } else {
           67  +    puts Error
           68  +    puts "  Got: $res"
           69  +    puts "  Expected: $expected"
           70  +  }
           71  +}
           72  +
           73  +# Calc checksum nonce from journal page data.
           74  +#
           75  +proc calc_nonce {jrnl_pgno} {
           76  +  global sectsz
           77  +  global db_pgsz
           78  +  global jrnl_name
           79  +  set jrnl_pg_offset [expr $sectsz+((4+$db_pgsz+4)*$jrnl_pgno)]
           80  +  set nonce [hexio_get_int [hexio_read $jrnl_name [expr $jrnl_pg_offset+4+$db_pgsz] 4]]
           81  +  for {set i [expr $db_pgsz-200]} {$i>0} {set i [expr $i-200]} {
           82  +    set byte [hexio_get_int [hexio_read $jrnl_name [expr $jrnl_pg_offset+4+$i] 1]]
           83  +    set nonce [expr $nonce-$byte]
           84  +  }
           85  +  return $nonce
           86  +}
           87  +
           88  +# Calc checksum from journal page data.
           89  +#
           90  +proc calc_chksum {jrnl_pgno} {
           91  +  global sectsz
           92  +  global db_pgsz
           93  +  global jrnl_name
           94  +  global nonce
           95  +  set jrnl_pg_offset [expr $sectsz+((4+$db_pgsz+4)*$jrnl_pgno)]
           96  +  set chksum $nonce
           97  +  for {set i [expr $db_pgsz-200]} {$i>0} {set i [expr $i-200]} {
           98  +    set byte [hexio_get_int [hexio_read $jrnl_name [expr $jrnl_pg_offset+4+$i] 1]]
           99  +    set chksum [expr $chksum+$byte]
          100  +  }
          101  +  return $chksum
          102  +}
          103  +
          104  +# Print journal page data in hex dump form
          105  +#
          106  +proc dump_jrnl_page {jrnl_pgno} {
          107  +  global sectsz
          108  +  global db_pgsz
          109  +  global jrnl_name
          110  +
          111  +  # print a header block for the page
          112  +  puts [string repeat "-" 79]
          113  +  set jrnl_pg_offset [expr $sectsz+((4+$db_pgsz+4)*$jrnl_pgno)]
          114  +  set db_pgno [hexio_get_int [hexio_read $jrnl_name [expr $jrnl_pg_offset] 4]]
          115  +  set chksum [hexio_get_int [hexio_read $jrnl_name [expr $jrnl_pg_offset+4+$db_pgsz] 4]]
          116  +  set nonce [calc_nonce $jrnl_pgno]
          117  +  puts [ format {jrnl_pg_offset: %08x (%d)  jrnl_pgno: %d  db_pgno: %d} \
          118  +      $jrnl_pg_offset $jrnl_pg_offset \
          119  +      $jrnl_pgno $db_pgno]
          120  +  puts [ format {nonce: %08x chksum: %08x} \
          121  +      $nonce $chksum]
          122  +
          123  +  # now hex dump the data
          124  +  # This is derived from the Tcler's WIKI
          125  +  set fid [open $jrnl_name r]
          126  +  fconfigure $fid -translation binary -encoding binary
          127  +  seek $fid [expr $jrnl_pg_offset+4]
          128  +  set data [read $fid $db_pgsz]
          129  +  close $fid
          130  +  for {set addr 0} {$addr<$db_pgsz} {set addr [expr $addr+16]} {
          131  +    # get 16 bytes of data
          132  +    set s [string range $data $addr [expr $addr+16]]
          133  +    
          134  +    # Convert the data to hex and to characters.
          135  +    binary scan $s H*@0a* hex ascii
          136  +
          137  +    # Replace non-printing characters in the data.
          138  +    regsub -all -- {[^[:graph:] ]} $ascii {.} ascii
          139  +
          140  +    # Split the 16 bytes into two 8-byte chunks
          141  +    regexp -- {(.{16})(.{0,16})} $hex -> hex1 hex2
          142  +
          143  +    # Convert the hex to pairs of hex digits
          144  +    regsub -all -- {..} $hex1 {& } hex1
          145  +    regsub -all -- {..} $hex2 {& } hex2
          146  +
          147  +    # Print the hex and ascii data
          148  +    puts [ format {%08x %-24s %-24s %-16s} \
          149  +        $addr $hex1 $hex2 $ascii ]
          150  +  }
          151  +}
          152  +
          153  +# Setup for the tests.  Make a backup copy of the files.
          154  +#
          155  +if [file exist $db_name.org] {
          156  +  puts "ERROR: during back-up: $db_name.org exists already."
          157  +  return;
          158  +}
          159  +if [file exist $jrnl_name.org] {
          160  +  puts "ERROR: during back-up: $jrnl_name.org exists already."
          161  +  return
          162  +}
          163  +copy_file $db_name $db_name.org
          164  +copy_file $jrnl_name $jrnl_name.org
          165  +
          166  +set db_fsize [file size $db_name]
          167  +set db_pgsz [hexio_get_int [hexio_read $db_name 16 2]]
          168  +set db_npage [expr {$db_fsize / $db_pgsz}]
          169  +
          170  +set jrnl_fsize [file size $jrnl_name]
          171  +set jrnl_npage [expr {($jrnl_fsize - $sectsz) / (4 + $db_pgsz + 4)}]
          172  +
          173  +# calculate checksum nonce for first page
          174  +set nonce [calc_nonce 0]
          175  +
          176  +# verify all the pages in the journal use the same nonce
          177  +for {set i 1} {$i<$jrnl_npage} {incr i} {
          178  +  set tnonce [calc_nonce $i]
          179  +  if {$tnonce != $nonce} {
          180  +    puts "WARNING: different nonces: 0=$nonce $i=$tnonce"
          181  +    if {$fix_chksums } {
          182  +      set jrnl_pg_offset [expr $sectsz+((4+$db_pgsz+4)*$i)]
          183  +      set tchksum [calc_chksum $i]
          184  +      hexio_write $jrnl_name [expr $jrnl_pg_offset+4+$db_pgsz] [format %08x $tchksum]
          185  +      puts "INFO: fixing chksum: $i=$tchksum"
          186  +    }
          187  +  }
          188  +}
          189  +
          190  +# verify all the page numbers in the journal
          191  +for {set i 0} {$i<$jrnl_npage} {incr i} {
          192  +  set jrnl_pg_offset [expr $sectsz+((4+$db_pgsz+4)*$i)]
          193  +  set db_pgno [hexio_get_int [hexio_read $jrnl_name $jrnl_pg_offset 4]]
          194  +  if {$db_pgno < 1} {
          195  +    puts "WARNING: page number < 1: $i=$db_pgno"
          196  +  }
          197  +  if {$db_pgno >= $db_npage} {
          198  +    puts "WARNING: page number >= $db_npage: $i=$db_pgno"
          199  +  }
          200  +}
          201  +
          202  +# dump page data
          203  +if {$dump_pages} {
          204  +  for {set i 0} {$i<$jrnl_npage} {incr i} {
          205  +    dump_jrnl_page $i
          206  +  }
          207  +}
          208  +
          209  +# write the 8 byte magic string
          210  +hexio_write $jrnl_name 0 d9d505f920a163d7
          211  +
          212  +# write -1 for number of records
          213  +hexio_write $jrnl_name 8 ffffffff
          214  +
          215  +# write 00 for checksum nonce
          216  +hexio_write $jrnl_name 12 [format %08x $nonce]
          217  +
          218  +# write page count
          219  +hexio_write $jrnl_name 16 [format %08x $db_npage]
          220  +
          221  +# write sector size
          222  +hexio_write $jrnl_name 20 [format %08x $sectsz]
          223  +
          224  +# write page size
          225  +hexio_write $jrnl_name 24 [format %08x $db_pgsz]
          226  +
          227  +# check the integrity of the database with the patched journal
          228  +sqlite3 db $db_name
          229  +do_test restore_jrnl-1.0 {
          230  +  catchsql {PRAGMA integrity_check}
          231  +} {0 ok}
          232  +db close
          233  +

Changes to tool/shell2.test.

    95     95       END;
    96     96   
    97     97       UPDATE OR REPLACE t5 SET a = 4 WHERE a = 1;
    98     98     }
    99     99   } {1 {Error: near line 9: too many levels of trigger recursion}}
   100    100   
   101    101   
          102  +
          103  +# Shell not echoing all commands with echo on.
          104  +# Ticket [eb620916be].
          105  +
          106  +# Test with echo off
          107  +# NB. whitespace is important
          108  +do_test shell2-1.4.1 {
          109  +  file delete -force foo.db
          110  +  catchcmd "foo.db" {CREATE TABLE foo(a);
          111  +INSERT INTO foo(a) VALUES(1);
          112  +SELECT * FROM foo;}
          113  +} {0 1}
          114  +
          115  +# Test with echo on using command line option
          116  +# NB. whitespace is important
          117  +do_test shell2-1.4.2 {
          118  +  file delete -force foo.db
          119  +  catchcmd "-echo foo.db" {CREATE TABLE foo(a);
          120  +INSERT INTO foo(a) VALUES(1);
          121  +SELECT * FROM foo;}
          122  +} {0 {CREATE TABLE foo(a);
          123  +INSERT INTO foo(a) VALUES(1);
          124  +SELECT * FROM foo;
          125  +1}}
          126  +
          127  +# Test with echo on using dot command
          128  +# NB. whitespace is important
          129  +do_test shell2-1.4.3 {
          130  +  file delete -force foo.db
          131  +  catchcmd "foo.db" {.echo ON
          132  +CREATE TABLE foo(a);
          133  +INSERT INTO foo(a) VALUES(1);
          134  +SELECT * FROM foo;}
          135  +} {0 {CREATE TABLE foo(a);
          136  +INSERT INTO foo(a) VALUES(1);
          137  +SELECT * FROM foo;
          138  +1}}
          139  +
          140  +# Test with echo on using dot command and 
          141  +# turning off mid- processing.
          142  +# NB. whitespace is important
          143  +do_test shell2-1.4.4 {
          144  +  file delete -force foo.db
          145  +  catchcmd "foo.db" {.echo ON
          146  +CREATE TABLE foo(a);
          147  +.echo OFF
          148  +INSERT INTO foo(a) VALUES(1);
          149  +SELECT * FROM foo;}
          150  +} {0 {CREATE TABLE foo(a);
          151  +.echo OFF
          152  +1}}
          153  +
          154  +# Test with echo on using dot command and 
          155  +# multiple commands per line.
          156  +# NB. whitespace is important
          157  +do_test shell2-1.4.5 {
          158  +  file delete -force foo.db
          159  +  catchcmd "foo.db" {.echo ON
          160  +CREATE TABLE foo1(a);
          161  +INSERT INTO foo1(a) VALUES(1);
          162  +CREATE TABLE foo2(b);
          163  +INSERT INTO foo2(b) VALUES(1);
          164  +SELECT * FROM foo1; SELECT * FROM foo2;
          165  +INSERT INTO foo1(a) VALUES(2); INSERT INTO foo2(b) VALUES(2);
          166  +SELECT * FROM foo1; SELECT * FROM foo2;
          167  +}
          168  +} {0 {CREATE TABLE foo1(a);
          169  +INSERT INTO foo1(a) VALUES(1);
          170  +CREATE TABLE foo2(b);
          171  +INSERT INTO foo2(b) VALUES(1);
          172  +SELECT * FROM foo1;
          173  +1
          174  +SELECT * FROM foo2;
          175  +1
          176  +INSERT INTO foo1(a) VALUES(2);
          177  +INSERT INTO foo2(b) VALUES(2);
          178  +SELECT * FROM foo1;
          179  +1
          180  +2
          181  +SELECT * FROM foo2;
          182  +1
          183  +2}}
          184  +
          185  +# Test with echo on and headers on using dot command and 
          186  +# multiple commands per line.
          187  +# NB. whitespace is important
          188  +do_test shell2-1.4.6 {
          189  +  file delete -force foo.db
          190  +  catchcmd "foo.db" {.echo ON
          191  +.headers ON
          192  +CREATE TABLE foo1(a);
          193  +INSERT INTO foo1(a) VALUES(1);
          194  +CREATE TABLE foo2(b);
          195  +INSERT INTO foo2(b) VALUES(1);
          196  +SELECT * FROM foo1; SELECT * FROM foo2;
          197  +INSERT INTO foo1(a) VALUES(2); INSERT INTO foo2(b) VALUES(2);
          198  +SELECT * FROM foo1; SELECT * FROM foo2;
          199  +}
          200  +} {0 {.headers ON
          201  +CREATE TABLE foo1(a);
          202  +INSERT INTO foo1(a) VALUES(1);
          203  +CREATE TABLE foo2(b);
          204  +INSERT INTO foo2(b) VALUES(1);
          205  +SELECT * FROM foo1;
          206  +a
          207  +1
          208  +SELECT * FROM foo2;
          209  +b
          210  +1
          211  +INSERT INTO foo1(a) VALUES(2);
          212  +INSERT INTO foo2(b) VALUES(2);
          213  +SELECT * FROM foo1;
          214  +a
          215  +1
          216  +2
          217  +SELECT * FROM foo2;
          218  +b
          219  +1
          220  +2}}