/ Check-in [86ab8643]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Merge updates from trunk.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | sqlite_system_errno
Files: files | file ages | folders
SHA1: 86ab8643969bd2e51a257d80da9316c668437f7b
User & Date: drh 2016-03-21 10:49:49
Context
2016-03-21
11:38
Do a better job of capturing all system errno values regardless of when they occur. check-in: 7d49998d user: drh tags: sqlite_system_errno
10:49
Merge updates from trunk. check-in: 86ab8643 user: drh tags: sqlite_system_errno
09:56
Change the way fts5 internally allocates segment ids in order to eliminated non-determinism from the module. check-in: d6e2637d user: dan tags: trunk
2016-03-17
19:28
Change the xGetLastError() method on the windows VFS so that it continues to format an error message (if requested) but returns the system error code, rather than an SQLite status code. check-in: 1602f6b5 user: drh tags: sqlite_system_errno
Changes
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Changes to doc/lemon.html.

     1      1   <html>
     2      2   <head>
     3      3   <title>The Lemon Parser Generator</title>
     4      4   </head>
     5      5   <body bgcolor=white>
     6      6   <h1 align=center>The Lemon Parser Generator</h1>
     7      7   
     8         -<p>Lemon is an LALR(1) parser generator for C or C++.  
     9         -It does the same job as ``bison'' and ``yacc''.
    10         -But lemon is not another bison or yacc clone.  It
            8  +<p>Lemon is an LALR(1) parser generator for C.
            9  +It does the same job as "bison" and "yacc".
           10  +But lemon is not a bison or yacc clone.  Lemon
    11     11   uses a different grammar syntax which is designed to
    12         -reduce the number of coding errors.  Lemon also uses a more
    13         -sophisticated parsing engine that is faster than yacc and
    14         -bison and which is both reentrant and thread-safe.
    15         -Furthermore, Lemon implements features that can be used
           12  +reduce the number of coding errors.  Lemon also uses a
           13  +parsing engine that is faster than yacc and
           14  +bison and which is both reentrant and threadsafe.
           15  +(Update: Since the previous sentence was written, bison
           16  +has also been updated so that it too can generate a
           17  +reentrant and threadsafe parser.)
           18  +Lemon also implements features that can be used
    16     19   to eliminate resource leaks, making is suitable for use
    17     20   in long-running programs such as graphical user interfaces
    18     21   or embedded controllers.</p>
    19     22   
    20     23   <p>This document is an introduction to the Lemon
    21     24   parser generator.</p>
    22     25   
................................................................................
    40     43   <ul>
    41     44   <li>C code to implement the parser.
    42     45   <li>A header file defining an integer ID for each terminal symbol.
    43     46   <li>An information file that describes the states of the generated parser
    44     47       automaton.
    45     48   </ul>
    46     49   By default, all three of these output files are generated.
    47         -The header file is suppressed if the ``-m'' command-line option is
    48         -used and the report file is omitted when ``-q'' is selected.</p>
           50  +The header file is suppressed if the "-m" command-line option is
           51  +used and the report file is omitted when "-q" is selected.</p>
    49     52   
    50         -<p>The grammar specification file uses a ``.y'' suffix, by convention.
           53  +<p>The grammar specification file uses a ".y" suffix, by convention.
    51     54   In the examples used in this document, we'll assume the name of the
    52         -grammar file is ``gram.y''.  A typical use of Lemon would be the
           55  +grammar file is "gram.y".  A typical use of Lemon would be the
    53     56   following command:
    54     57   <pre>
    55     58      lemon gram.y
    56     59   </pre>
    57         -This command will generate three output files named ``gram.c'',
    58         -``gram.h'' and ``gram.out''.
           60  +This command will generate three output files named "gram.c",
           61  +"gram.h" and "gram.out".
    59     62   The first is C code to implement the parser.  The second
    60     63   is the header file that defines numerical values for all
    61     64   terminal symbols, and the last is the report that explains
    62     65   the states used by the parser automaton.</p>
    63     66   
    64     67   <h3>Command Line Options</h3>
    65     68   
................................................................................
    67     70   You can obtain a list of the available command-line options together
    68     71   with a brief explanation of what each does by typing
    69     72   <pre>
    70     73      lemon -?
    71     74   </pre>
    72     75   As of this writing, the following command-line options are supported:
    73     76   <ul>
    74         -<li><tt>-b</tt>
    75         -<li><tt>-c</tt>
    76         -<li><tt>-g</tt>
    77         -<li><tt>-m</tt>
    78         -<li><tt>-q</tt>
    79         -<li><tt>-s</tt>
    80         -<li><tt>-x</tt>
    81         -</ul>
    82         -The ``-b'' option reduces the amount of text in the report file by
    83         -printing only the basis of each parser state, rather than the full
    84         -configuration.
    85         -The ``-c'' option suppresses action table compression.  Using -c
    86         -will make the parser a little larger and slower but it will detect
    87         -syntax errors sooner.
    88         -The ``-g'' option causes no output files to be generated at all.
    89         -Instead, the input grammar file is printed on standard output but
    90         -with all comments, actions and other extraneous text deleted.  This
    91         -is a useful way to get a quick summary of a grammar.
    92         -The ``-m'' option causes the output C source file to be compatible
    93         -with the ``makeheaders'' program.
    94         -Makeheaders is a program that automatically generates header files
    95         -from C source code.  When the ``-m'' option is used, the header
    96         -file is not output since the makeheaders program will take care
    97         -of generated all header files automatically.
    98         -The ``-q'' option suppresses the report file.
    99         -Using ``-s'' causes a brief summary of parser statistics to be
   100         -printed.  Like this:
   101         -<pre>
   102         -   Parser statistics: 74 terminals, 70 nonterminals, 179 rules
   103         -                      340 states, 2026 parser table entries, 0 conflicts
   104         -</pre>
   105         -Finally, the ``-x'' option causes Lemon to print its version number
   106         -and then stops without attempting to read the grammar or generate a parser.</p>
           77  +<li><b>-b</b>
           78  +Show only the basis for each parser state in the report file.
           79  +<li><b>-c</b>
           80  +Do not compress the generated action tables.
           81  +<li><b>-D<i>name</i></b>
           82  +Define C preprocessor macro <i>name</i>.  This macro is useable by
           83  +"%ifdef" lines in the grammar file.
           84  +<li><b>-g</b>
           85  +Do not generate a parser.  Instead write the input grammar to standard
           86  +output with all comments, actions, and other extraneous text removed.
           87  +<li><b>-l</b>
           88  +Omit "#line" directives int the generated parser C code.
           89  +<li><b>-m</b>
           90  +Cause the output C source code to be compatible with the "makeheaders"
           91  +program. 
           92  +<li><b>-p</b>
           93  +Display all conflicts that are resolved by 
           94  +<a href='#precrules'>precedence rules</a>.
           95  +<li><b>-q</b>
           96  +Suppress generation of the report file.
           97  +<li><b>-r</b>
           98  +Do not sort or renumber the parser states as part of optimization.
           99  +<li><b>-s</b>
          100  +Show parser statistics before existing.
          101  +<li><b>-T<i>file</i></b>
          102  +Use <i>file</i> as the template for the generated C-code parser implementation.
          103  +<li><b>-x</b>
          104  +Print the Lemon version number.
          105  +</ul>
   107    106   
   108    107   <h3>The Parser Interface</h3>
   109    108   
   110    109   <p>Lemon doesn't generate a complete, working program.  It only generates
   111    110   a few subroutines that implement a parser.  This section describes
   112    111   the interface to those subroutines.  It is up to the programmer to
   113    112   call these subroutines in an appropriate way in order to produce a
................................................................................
   117    116   must first create the parser.
   118    117   A new parser is created as follows:
   119    118   <pre>
   120    119      void *pParser = ParseAlloc( malloc );
   121    120   </pre>
   122    121   The ParseAlloc() routine allocates and initializes a new parser and
   123    122   returns a pointer to it.
   124         -The actual data structure used to represent a parser is opaque --
          123  +The actual data structure used to represent a parser is opaque &mdash;
   125    124   its internal structure is not visible or usable by the calling routine.
   126    125   For this reason, the ParseAlloc() routine returns a pointer to void
   127    126   rather than a pointer to some particular structure.
   128    127   The sole argument to the ParseAlloc() routine is a pointer to the
   129         -subroutine used to allocate memory.  Typically this means ``malloc()''.</p>
          128  +subroutine used to allocate memory.  Typically this means malloc().</p>
   130    129   
   131    130   <p>After a program is finished using a parser, it can reclaim all
   132    131   memory allocated by that parser by calling
   133    132   <pre>
   134    133      ParseFree(pParser, free);
   135    134   </pre>
   136    135   The first argument is the same pointer returned by ParseAlloc().  The
................................................................................
   147    146   The first argument to the Parse() routine is the pointer returned by
   148    147   ParseAlloc().
   149    148   The second argument is a small positive integer that tells the parse the
   150    149   type of the next token in the data stream.
   151    150   There is one token type for each terminal symbol in the grammar.
   152    151   The gram.h file generated by Lemon contains #define statements that
   153    152   map symbolic terminal symbol names into appropriate integer values.
   154         -(A value of 0 for the second argument is a special flag to the
   155         -parser to indicate that the end of input has been reached.)
          153  +A value of 0 for the second argument is a special flag to the
          154  +parser to indicate that the end of input has been reached.
   156    155   The third argument is the value of the given token.  By default,
   157    156   the type of the third argument is integer, but the grammar will
   158    157   usually redefine this type to be some kind of structure.
   159    158   Typically the second argument will be a broad category of tokens
   160         -such as ``identifier'' or ``number'' and the third argument will
          159  +such as "identifier" or "number" and the third argument will
   161    160   be the name of the identifier or the value of the number.</p>
   162    161   
   163    162   <p>The Parse() function may have either three or four arguments,
   164    163   depending on the grammar.  If the grammar specification file requests
   165    164   it (via the <a href='#extraarg'><tt>extra_argument</tt> directive</a>),
   166    165   the Parse() function will have a fourth parameter that can be
   167    166   of any type chosen by the programmer.  The parser doesn't do anything
................................................................................
   189    188      15    ParseFree(pParser, free );
   190    189      16    TokenizerFree(pTokenizer);
   191    190      17    return sState.treeRoot;
   192    191      18 }
   193    192   </pre>
   194    193   This example shows a user-written routine that parses a file of
   195    194   text and returns a pointer to the parse tree.
   196         -(We've omitted all error-handling from this example to keep it
          195  +(All error-handling code is omitted from this example to keep it
   197    196   simple.)
   198    197   We assume the existence of some kind of tokenizer which is created
   199    198   using TokenizerCreate() on line 8 and deleted by TokenizerFree()
   200    199   on line 16.  The GetNextToken() function on line 11 retrieves the
   201    200   next token from the input file and puts its type in the 
   202    201   integer variable hTokenId.  The sToken variable is assumed to be
   203    202   some kind of structure that contains details about each token,
................................................................................
   283    282   declaration can occur at any point in the file.
   284    283   Lemon ignores whitespace (except where it is needed to separate
   285    284   tokens) and it honors the same commenting conventions as C and C++.</p>
   286    285   
   287    286   <h3>Terminals and Nonterminals</h3>
   288    287   
   289    288   <p>A terminal symbol (token) is any string of alphanumeric
   290         -and underscore characters
          289  +and/or underscore characters
   291    290   that begins with an upper case letter.
   292    291   A terminal can contain lowercase letters after the first character,
   293    292   but the usual convention is to make terminals all upper case.
   294    293   A nonterminal, on the other hand, is any string of alphanumeric
   295    294   and underscore characters than begins with a lower case letter.
   296    295   Again, the usual convention is to make nonterminals use all lower
   297    296   case letters.</p>
................................................................................
   310    309   must have alphanumeric names.</p>
   311    310   
   312    311   <h3>Grammar Rules</h3>
   313    312   
   314    313   <p>The main component of a Lemon grammar file is a sequence of grammar
   315    314   rules.
   316    315   Each grammar rule consists of a nonterminal symbol followed by
   317         -the special symbol ``::='' and then a list of terminals and/or nonterminals.
          316  +the special symbol "::=" and then a list of terminals and/or nonterminals.
   318    317   The rule is terminated by a period.
   319    318   The list of terminals and nonterminals on the right-hand side of the
   320    319   rule can be empty.
   321    320   Rules can occur in any order, except that the left-hand side of the
   322    321   first rule is assumed to be the start symbol for the grammar (unless
   323    322   specified otherwise using the <tt>%start</tt> directive described below.)
   324    323   A typical sequence of grammar rules might look something like this:
................................................................................
   326    325     expr ::= expr PLUS expr.
   327    326     expr ::= expr TIMES expr.
   328    327     expr ::= LPAREN expr RPAREN.
   329    328     expr ::= VALUE.
   330    329   </pre>
   331    330   </p>
   332    331   
   333         -<p>There is one non-terminal in this example, ``expr'', and five
   334         -terminal symbols or tokens: ``PLUS'', ``TIMES'', ``LPAREN'',
   335         -``RPAREN'' and ``VALUE''.</p>
          332  +<p>There is one non-terminal in this example, "expr", and five
          333  +terminal symbols or tokens: "PLUS", "TIMES", "LPAREN",
          334  +"RPAREN" and "VALUE".</p>
   336    335   
   337    336   <p>Like yacc and bison, Lemon allows the grammar to specify a block
   338    337   of C code that will be executed whenever a grammar rule is reduced
   339    338   by the parser.
   340    339   In Lemon, this action is specified by putting the C code (contained
   341    340   within curly braces <tt>{...}</tt>) immediately after the
   342    341   period that closes the rule.
................................................................................
   344    343   <pre>
   345    344     expr ::= expr PLUS expr.   { printf("Doing an addition...\n"); }
   346    345   </pre>
   347    346   </p>
   348    347   
   349    348   <p>In order to be useful, grammar actions must normally be linked to
   350    349   their associated grammar rules.
   351         -In yacc and bison, this is accomplished by embedding a ``$$'' in the
          350  +In yacc and bison, this is accomplished by embedding a "$$" in the
   352    351   action to stand for the value of the left-hand side of the rule and
   353         -symbols ``$1'', ``$2'', and so forth to stand for the value of
          352  +symbols "$1", "$2", and so forth to stand for the value of
   354    353   the terminal or nonterminal at position 1, 2 and so forth on the
   355    354   right-hand side of the rule.
   356    355   This idea is very powerful, but it is also very error-prone.  The
   357    356   single most common source of errors in a yacc or bison grammar is
   358    357   to miscount the number of symbols on the right-hand side of a grammar
   359         -rule and say ``$7'' when you really mean ``$8''.</p>
          358  +rule and say "$7" when you really mean "$8".</p>
   360    359   
   361    360   <p>Lemon avoids the need to count grammar symbols by assigning symbolic
   362    361   names to each symbol in a grammar rule and then using those symbolic
   363    362   names in the action.
   364    363   In yacc or bison, one would write this:
   365    364   <pre>
   366    365     expr -> expr PLUS expr  { $$ = $1 + $3; };
................................................................................
   382    381   includes a linking symbol in parentheses but that linking symbol
   383    382   is not actually used in the reduce action, then an error message
   384    383   is generated.
   385    384   For example, the rule
   386    385   <pre>
   387    386     expr(A) ::= expr(B) PLUS expr(C).  { A = B; }
   388    387   </pre>
   389         -will generate an error because the linking symbol ``C'' is used
          388  +will generate an error because the linking symbol "C" is used
   390    389   in the grammar rule but not in the reduce action.</p>
   391    390   
   392    391   <p>The Lemon notation for linking grammar rules to reduce actions
   393    392   also facilitates the use of destructors for reclaiming memory
   394    393   allocated by the values of terminals and nonterminals on the
   395    394   right-hand side of a rule.</p>
   396    395   
          396  +<a name='precrules'></a>
   397    397   <h3>Precedence Rules</h3>
   398    398   
   399    399   <p>Lemon resolves parsing ambiguities in exactly the same way as
   400    400   yacc and bison.  A shift-reduce conflict is resolved in favor
   401    401   of the shift, and a reduce-reduce conflict is resolved by reducing
   402    402   whichever rule comes first in the grammar file.</p>
   403    403   
   404    404   <p>Just like in
   405    405   yacc and bison, Lemon allows a measure of control 
   406    406   over the resolution of paring conflicts using precedence rules.
   407    407   A precedence value can be assigned to any terminal symbol
   408         -using the %left, %right or %nonassoc directives.  Terminal symbols
          408  +using the 
          409  +<a href='#pleft'>%left</a>,
          410  +<a href='#pright'>%right</a> or
          411  +<a href='#pnonassoc'>%nonassoc</a> directives.  Terminal symbols
   409    412   mentioned in earlier directives have a lower precedence that
   410    413   terminal symbols mentioned in later directives.  For example:</p>
   411    414   
   412    415   <p><pre>
   413    416      %left AND.
   414    417      %left OR.
   415    418      %nonassoc EQ NE GT GE LT LE.
................................................................................
   521    524   
   522    525   <p>Lemon supports the following special directives:
   523    526   <ul>
   524    527   <li><tt>%code</tt>
   525    528   <li><tt>%default_destructor</tt>
   526    529   <li><tt>%default_type</tt>
   527    530   <li><tt>%destructor</tt>
          531  +<li><tt>%endif</tt>
   528    532   <li><tt>%extra_argument</tt>
          533  +<li><tt>%fallback</tt>
          534  +<li><tt>%ifdef</tt>
          535  +<li><tt>%ifndef</tt>
   529    536   <li><tt>%include</tt>
   530    537   <li><tt>%left</tt>
   531    538   <li><tt>%name</tt>
   532    539   <li><tt>%nonassoc</tt>
   533    540   <li><tt>%parse_accept</tt>
   534    541   <li><tt>%parse_failure </tt>
   535    542   <li><tt>%right</tt>
   536    543   <li><tt>%stack_overflow</tt>
   537    544   <li><tt>%stack_size</tt>
   538    545   <li><tt>%start_symbol</tt>
   539    546   <li><tt>%syntax_error</tt>
          547  +<li><tt>%token_class</tt>
   540    548   <li><tt>%token_destructor</tt>
   541    549   <li><tt>%token_prefix</tt>
   542    550   <li><tt>%token_type</tt>
   543    551   <li><tt>%type</tt>
          552  +<li><tt>%wildcard</tt>
   544    553   </ul>
   545    554   Each of these directives will be described separately in the
   546    555   following sections:</p>
   547    556   
          557  +<a name='pcode'></a>
   548    558   <h4>The <tt>%code</tt> directive</h4>
   549    559   
   550         -<p>The %code directive is used to specify addition C/C++ code that
          560  +<p>The %code directive is used to specify addition C code that
   551    561   is added to the end of the main output file.  This is similar to
   552         -the %include directive except that %include is inserted at the
   553         -beginning of the main output file.</p>
          562  +the <a href='#pinclude'>%include</a> directive except that %include
          563  +is inserted at the beginning of the main output file.</p>
   554    564   
   555    565   <p>%code is typically used to include some action routines or perhaps
   556         -a tokenizer as part of the output file.</p>
          566  +a tokenizer or even the "main()" function 
          567  +as part of the output file.</p>
   557    568   
          569  +<a name='default_destructor'></a>
   558    570   <h4>The <tt>%default_destructor</tt> directive</h4>
   559    571   
   560    572   <p>The %default_destructor directive specifies a destructor to 
   561    573   use for non-terminals that do not have their own destructor
   562    574   specified by a separate %destructor directive.  See the documentation
   563         -on the %destructor directive below for additional information.</p>
          575  +on the <a name='#destructor'>%destructor</a> directive below for
          576  +additional information.</p>
   564    577   
   565    578   <p>In some grammers, many different non-terminal symbols have the
   566    579   same datatype and hence the same destructor.  This directive is
   567    580   a convenience way to specify the same destructor for all those
   568    581   non-terminals using a single statement.</p>
   569    582   
          583  +<a name='default_type'></a>
   570    584   <h4>The <tt>%default_type</tt> directive</h4>
   571    585   
   572    586   <p>The %default_type directive specifies the datatype of non-terminal
   573    587   symbols that do no have their own datatype defined using a separate
   574         -%type directive.  See the documentation on %type below for addition
   575         -information.</p>
          588  +<a href='#ptype'>%type</a> directive.  
          589  +</p>
   576    590   
          591  +<a name='destructor'></a>
   577    592   <h4>The <tt>%destructor</tt> directive</h4>
   578    593   
   579    594   <p>The %destructor directive is used to specify a destructor for
   580    595   a non-terminal symbol.
   581         -(See also the %token_destructor directive which is used to
   582         -specify a destructor for terminal symbols.)</p>
          596  +(See also the <a href='#token_destructor'>%token_destructor</a>
          597  +directive which is used to specify a destructor for terminal symbols.)</p>
   583    598   
   584    599   <p>A non-terminal's destructor is called to dispose of the
   585    600   non-terminal's value whenever the non-terminal is popped from
   586    601   the stack.  This includes all of the following circumstances:
   587    602   <ul>
   588    603   <li> When a rule reduces and the value of a non-terminal on
   589    604        the right-hand side is not linked to C code.
................................................................................
   598    613   <pre>
   599    614      %type nt {void*}
   600    615      %destructor nt { free($$); }
   601    616      nt(A) ::= ID NUM.   { A = malloc( 100 ); }
   602    617   </pre>
   603    618   This example is a bit contrived but it serves to illustrate how
   604    619   destructors work.  The example shows a non-terminal named
   605         -``nt'' that holds values of type ``void*''.  When the rule for
   606         -an ``nt'' reduces, it sets the value of the non-terminal to
          620  +"nt" that holds values of type "void*".  When the rule for
          621  +an "nt" reduces, it sets the value of the non-terminal to
   607    622   space obtained from malloc().  Later, when the nt non-terminal
   608    623   is popped from the stack, the destructor will fire and call
   609    624   free() on this malloced space, thus avoiding a memory leak.
   610         -(Note that the symbol ``$$'' in the destructor code is replaced
          625  +(Note that the symbol "$$" in the destructor code is replaced
   611    626   by the value of the non-terminal.)</p>
   612    627   
   613    628   <p>It is important to note that the value of a non-terminal is passed
   614    629   to the destructor whenever the non-terminal is removed from the
   615    630   stack, unless the non-terminal is used in a C-code action.  If
   616    631   the non-terminal is used by C-code, then it is assumed that the
   617         -C-code will take care of destroying it if it should really
   618         -be destroyed.  More commonly, the value is used to build some
          632  +C-code will take care of destroying it.
          633  +More commonly, the value is used to build some
   619    634   larger structure and we don't want to destroy it, which is why
   620    635   the destructor is not called in this circumstance.</p>
   621    636   
   622         -<p>By appropriate use of destructors, it is possible to
   623         -build a parser using Lemon that can be used within a long-running
   624         -program, such as a GUI, that will not leak memory or other resources.
          637  +<p>Destructors help avoid memory leaks by automatically freeing
          638  +allocated objects when they go out of scope.
   625    639   To do the same using yacc or bison is much more difficult.</p>
   626    640   
   627    641   <a name="extraarg"></a>
   628    642   <h4>The <tt>%extra_argument</tt> directive</h4>
   629    643   
   630    644   The %extra_argument directive instructs Lemon to add a 4th parameter
   631    645   to the parameter list of the Parse() function it generates.  Lemon
................................................................................
   634    648   and so forth.  For example, if the grammar file contains:</p>
   635    649   
   636    650   <p><pre>
   637    651       %extra_argument { MyStruct *pAbc }
   638    652   </pre></p>
   639    653   
   640    654   <p>Then the Parse() function generated will have an 4th parameter
   641         -of type ``MyStruct*'' and all action routines will have access to
   642         -a variable named ``pAbc'' that is the value of the 4th parameter
          655  +of type "MyStruct*" and all action routines will have access to
          656  +a variable named "pAbc" that is the value of the 4th parameter
   643    657   in the most recent call to Parse().</p>
   644    658   
          659  +<a name='pfallback'></a>
          660  +<h4>The <tt>%fallback</tt> directive</h4>
          661  +
          662  +<p>The %fallback directive specifies an alternative meaning for one
          663  +or more tokens.  The alternative meaning is tried if the original token
          664  +would have generated a syntax error.
          665  +
          666  +<p>The %fallback directive was added to support robust parsing of SQL
          667  +syntax in <a href="https://www.sqlite.org/">SQLite</a>.
          668  +The SQL language contains a large assortment of keywords, each of which
          669  +appears as a different token to the language parser.  SQL contains so
          670  +many keywords, that it can be difficult for programmers to keep up with
          671  +them all.  Programmers will, therefore, sometimes mistakenly use an
          672  +obscure language keyword for an identifier.  The %fallback directive
          673  +provides a mechanism to tell the parser:  "If you are unable to parse
          674  +this keyword, try treating it as an identifier instead."
          675  +
          676  +<p>The syntax of %fallback is as follows:
          677  +
          678  +<blockquote>
          679  +<tt>%fallback</tt>  <i>ID</i> <i>TOKEN...</i> <b>.</b>
          680  +</blockquote>
          681  +
          682  +<p>In words, the %fallback directive is followed by a list of token names
          683  +terminated by a period.  The first token name is the fallback token - the
          684  +token to which all the other tokens fall back to.  The second and subsequent
          685  +arguments are tokens which fall back to the token identified by the first
          686  +argument.
          687  +
          688  +<a name='pifdef'></a>
          689  +<h4>The <tt>%ifdef</tt>, <tt>%ifndef</tt>, and <tt>%endif</tt> directives.</h4>
          690  +
          691  +<p>The %ifdef, %ifndef, and %endif directives are similar to
          692  +#ifdef, #ifndef, and #endif in the C-preprocessor, just not as general.
          693  +Each of these directives must begin at the left margin.  No whitespace
          694  +is allowed between the "%" and the directive name.
          695  +
          696  +<p>Grammar text in between "%ifdef MACRO" and the next nested "%endif" is
          697  +ignored unless the "-DMACRO" command-line option is used.  Grammar text
          698  +betwen "%ifndef MACRO" and the next nested "%endif" is included except when
          699  +the "-DMACRO" command-line option is used.
          700  +
          701  +<p>Note that the argument to %ifdef and %ifndef must be a single 
          702  +preprocessor symbol name, not a general expression.  There is no "%else"
          703  +directive.
          704  +
          705  +
          706  +<a name='pinclude'></a>
   645    707   <h4>The <tt>%include</tt> directive</h4>
   646    708   
   647    709   <p>The %include directive specifies C code that is included at the
   648    710   top of the generated parser.  You can include any text you want --
   649    711   the Lemon parser generator copies it blindly.  If you have multiple
   650         -%include directives in your grammar file the value of the last
   651         -%include directive overwrites all the others.</p.
          712  +%include directives in your grammar file, their values are concatenated
          713  +so that all %include code ultimately appears near the top of the
          714  +generated parser, in the same order as it appeared in the grammer.</p>
   652    715   
   653    716   <p>The %include directive is very handy for getting some extra #include
   654    717   preprocessor statements at the beginning of the generated parser.
   655    718   For example:</p>
   656    719   
   657    720   <p><pre>
   658    721      %include {#include &lt;unistd.h&gt;}
   659    722   </pre></p>
   660    723   
   661    724   <p>This might be needed, for example, if some of the C actions in the
   662    725   grammar call functions that are prototyed in unistd.h.</p>
   663    726   
          727  +<a name='pleft'></a>
   664    728   <h4>The <tt>%left</tt> directive</h4>
   665    729   
   666         -The %left directive is used (along with the %right and
   667         -%nonassoc directives) to declare precedences of terminal
   668         -symbols.  Every terminal symbol whose name appears after
   669         -a %left directive but before the next period (``.'') is
          730  +The %left directive is used (along with the <a href='#pright'>%right</a> and
          731  +<a href='#pnonassoc'>%nonassoc</a> directives) to declare precedences of 
          732  +terminal symbols.  Every terminal symbol whose name appears after
          733  +a %left directive but before the next period (".") is
   670    734   given the same left-associative precedence value.  Subsequent
   671    735   %left directives have higher precedence.  For example:</p>
   672    736   
   673    737   <p><pre>
   674    738      %left AND.
   675    739      %left OR.
   676    740      %nonassoc EQ NE GT GE LT LE.
................................................................................
   683    747   directive.</p>
   684    748   
   685    749   <p>LALR(1) grammars can get into a situation where they require
   686    750   a large amount of stack space if you make heavy use or right-associative
   687    751   operators.  For this reason, it is recommended that you use %left
   688    752   rather than %right whenever possible.</p>
   689    753   
          754  +<a name='pname'></a>
   690    755   <h4>The <tt>%name</tt> directive</h4>
   691    756   
   692    757   <p>By default, the functions generated by Lemon all begin with the
   693         -five-character string ``Parse''.  You can change this string to something
          758  +five-character string "Parse".  You can change this string to something
   694    759   different using the %name directive.  For instance:</p>
   695    760   
   696    761   <p><pre>
   697    762      %name Abcde
   698    763   </pre></p>
   699    764   
   700    765   <p>Putting this directive in the grammar file will cause Lemon to generate
................................................................................
   705    770   <li> AbcdeTrace(), and
   706    771   <li> Abcde().
   707    772   </ul>
   708    773   The %name directive allows you to generator two or more different
   709    774   parsers and link them all into the same executable.
   710    775   </p>
   711    776   
          777  +<a name='pnonassoc'></a>
   712    778   <h4>The <tt>%nonassoc</tt> directive</h4>
   713    779   
   714    780   <p>This directive is used to assign non-associative precedence to
   715         -one or more terminal symbols.  See the section on precedence rules
   716         -or on the %left directive for additional information.</p>
          781  +one or more terminal symbols.  See the section on 
          782  +<a href='#precrules'>precedence rules</a>
          783  +or on the <a href='#pleft'>%left</a> directive for additional information.</p>
   717    784   
          785  +<a name='parse_accept'></a>
   718    786   <h4>The <tt>%parse_accept</tt> directive</h4>
   719    787   
   720    788   <p>The %parse_accept directive specifies a block of C code that is
   721         -executed whenever the parser accepts its input string.  To ``accept''
          789  +executed whenever the parser accepts its input string.  To "accept"
   722    790   an input string means that the parser was able to process all tokens
   723    791   without error.</p>
   724    792   
   725    793   <p>For example:</p>
   726    794   
   727    795   <p><pre>
   728    796      %parse_accept {
   729    797         printf("parsing complete!\n");
   730    798      }
   731    799   </pre></p>
   732    800   
   733         -
          801  +<a name='parse_failure'></a>
   734    802   <h4>The <tt>%parse_failure</tt> directive</h4>
   735    803   
   736    804   <p>The %parse_failure directive specifies a block of C code that
   737    805   is executed whenever the parser fails complete.  This code is not
   738    806   executed until the parser has tried and failed to resolve an input
   739    807   error using is usual error recovery strategy.  The routine is
   740    808   only invoked when parsing is unable to continue.</p>
................................................................................
   741    809   
   742    810   <p><pre>
   743    811      %parse_failure {
   744    812        fprintf(stderr,"Giving up.  Parser is hopelessly lost...\n");
   745    813      }
   746    814   </pre></p>
   747    815   
          816  +<a name='pright'></a>
   748    817   <h4>The <tt>%right</tt> directive</h4>
   749    818   
   750    819   <p>This directive is used to assign right-associative precedence to
   751         -one or more terminal symbols.  See the section on precedence rules
   752         -or on the %left directive for additional information.</p>
          820  +one or more terminal symbols.  See the section on 
          821  +<a href='#precrules'>precedence rules</a>
          822  +or on the <a href='#pleft'>%left</a> directive for additional information.</p>
   753    823   
          824  +<a name='stack_overflow'></a>
   754    825   <h4>The <tt>%stack_overflow</tt> directive</h4>
   755    826   
   756    827   <p>The %stack_overflow directive specifies a block of C code that
   757    828   is executed if the parser's internal stack ever overflows.  Typically
   758    829   this just prints an error message.  After a stack overflow, the parser
   759    830   will be unable to continue and must be reset.</p>
   760    831   
................................................................................
   775    846   </pre>
   776    847   Not like this:
   777    848   <pre>
   778    849      list ::= element list.      // right-recursion.  Bad!
   779    850      list ::= .
   780    851   </pre>
   781    852   
          853  +<a name='stack_size'></a>
   782    854   <h4>The <tt>%stack_size</tt> directive</h4>
   783    855   
   784    856   <p>If stack overflow is a problem and you can't resolve the trouble
   785    857   by using left-recursion, then you might want to increase the size
   786    858   of the parser's stack using this directive.  Put an positive integer
   787    859   after the %stack_size directive and Lemon will generate a parse
   788    860   with a stack of the requested size.  The default value is 100.</p>
   789    861   
   790    862   <p><pre>
   791    863      %stack_size 2000
   792    864   </pre></p>
   793    865   
          866  +<a name='start_symbol'></a>
   794    867   <h4>The <tt>%start_symbol</tt> directive</h4>
   795    868   
   796    869   <p>By default, the start-symbol for the grammar that Lemon generates
   797    870   is the first non-terminal that appears in the grammar file.  But you
   798    871   can choose a different start-symbol using the %start_symbol directive.</p>
   799    872   
   800    873   <p><pre>
   801    874      %start_symbol  prog
   802    875   </pre></p>
   803    876   
          877  +<a name='token_destructor'></a>
   804    878   <h4>The <tt>%token_destructor</tt> directive</h4>
   805    879   
   806    880   <p>The %destructor directive assigns a destructor to a non-terminal
   807    881   symbol.  (See the description of the %destructor directive above.)
   808    882   This directive does the same thing for all terminal symbols.</p>
   809    883   
   810    884   <p>Unlike non-terminal symbols which may each have a different data type
   811    885   for their values, terminals all use the same data type (defined by
   812    886   the %token_type directive) and so they use a common destructor.  Other
   813    887   than that, the token destructor works just like the non-terminal
   814    888   destructors.</p>
   815    889   
          890  +<a name='token_prefix'></a>
   816    891   <h4>The <tt>%token_prefix</tt> directive</h4>
   817    892   
   818    893   <p>Lemon generates #defines that assign small integer constants
   819    894   to each terminal symbol in the grammar.  If desired, Lemon will
   820    895   add a prefix specified by this directive
   821    896   to each of the #defines it generates.
   822    897   So if the default output of Lemon looked like this:
................................................................................
   834    909   <pre>
   835    910       #define TOKEN_AND        1
   836    911       #define TOKEN_MINUS      2
   837    912       #define TOKEN_OR         3
   838    913       #define TOKEN_PLUS       4
   839    914   </pre>
   840    915   
          916  +<a name='token_type'></a><a name='ptype'></a>
   841    917   <h4>The <tt>%token_type</tt> and <tt>%type</tt> directives</h4>
   842    918   
   843    919   <p>These directives are used to specify the data types for values
   844    920   on the parser's stack associated with terminal and non-terminal
   845    921   symbols.  The values of all terminal symbols must be of the same
   846    922   type.  This turns out to be the same data type as the 3rd parameter
   847    923   to the Parse() function generated by Lemon.  Typically, you will
................................................................................
   849    925   token structure.  Like this:</p>
   850    926   
   851    927   <p><pre>
   852    928      %token_type    {Token*}
   853    929   </pre></p>
   854    930   
   855    931   <p>If the data type of terminals is not specified, the default value
   856         -is ``int''.</p>
          932  +is "int".</p>
   857    933   
   858    934   <p>Non-terminal symbols can each have their own data types.  Typically
   859    935   the data type  of a non-terminal is a pointer to the root of a parse-tree
   860    936   structure that contains all information about that non-terminal.
   861    937   For example:</p>
   862    938   
   863    939   <p><pre>
................................................................................
   870    946   on what the corresponding non-terminal or terminal symbol is.  But
   871    947   the grammar designer should keep in mind that the size of the union
   872    948   will be the size of its largest element.  So if you have a single
   873    949   non-terminal whose data type requires 1K of storage, then your 100
   874    950   entry parser stack will require 100K of heap space.  If you are willing
   875    951   and able to pay that price, fine.  You just need to know.</p>
   876    952   
          953  +<a name='pwildcard'></a>
          954  +<h4>The <tt>%wildcard</tt> directive</h4>
          955  +
          956  +<p>The %wildcard directive is followed by a single token name and a
          957  +period.  This directive specifies that the identified token should 
          958  +match any input token.
          959  +
          960  +<p>When the generated parser has the choice of matching an input against
          961  +the wildcard token and some other token, the other token is always used.
          962  +The wildcard token is only matched if there are no other alternatives.
          963  +
   877    964   <h3>Error Processing</h3>
   878    965   
   879    966   <p>After extensive experimentation over several years, it has been
   880    967   discovered that the error recovery strategy used by yacc is about
   881    968   as good as it gets.  And so that is what Lemon uses.</p>
   882    969   
   883    970   <p>When a Lemon-generated parser encounters a syntax error, it
   884    971   first invokes the code specified by the %syntax_error directive, if
   885    972   any.  It then enters its error recovery strategy.  The error recovery
   886    973   strategy is to begin popping the parsers stack until it enters a
   887    974   state where it is permitted to shift a special non-terminal symbol
   888         -named ``error''.  It then shifts this non-terminal and continues
          975  +named "error".  It then shifts this non-terminal and continues
   889    976   parsing.  But the %syntax_error routine will not be called again
   890    977   until at least three new tokens have been successfully shifted.</p>
   891    978   
   892    979   <p>If the parser pops its stack until the stack is empty, and it still
   893    980   is unable to shift the error symbol, then the %parse_failed routine
   894    981   is invoked and the parser resets itself to its start state, ready
   895    982   to begin parsing a new file.  This is what will happen at the very
   896    983   first syntax error, of course, if there are no instances of the 
   897         -``error'' non-terminal in your grammar.</p>
          984  +"error" non-terminal in your grammar.</p>
   898    985   
   899    986   </body>
   900    987   </html>

Changes to ext/fts5/fts5_index.c.

  3449   3449   static int fts5AllocateSegid(Fts5Index *p, Fts5Structure *pStruct){
  3450   3450     int iSegid = 0;
  3451   3451   
  3452   3452     if( p->rc==SQLITE_OK ){
  3453   3453       if( pStruct->nSegment>=FTS5_MAX_SEGMENT ){
  3454   3454         p->rc = SQLITE_FULL;
  3455   3455       }else{
  3456         -      while( iSegid==0 ){
  3457         -        int iLvl, iSeg;
  3458         -        sqlite3_randomness(sizeof(u32), (void*)&iSegid);
  3459         -        iSegid = iSegid & ((1 << FTS5_DATA_ID_B)-1);
  3460         -        for(iLvl=0; iLvl<pStruct->nLevel; iLvl++){
  3461         -          for(iSeg=0; iSeg<pStruct->aLevel[iLvl].nSeg; iSeg++){
  3462         -            if( iSegid==pStruct->aLevel[iLvl].aSeg[iSeg].iSegid ){
  3463         -              iSegid = 0;
  3464         -            }
         3456  +      /* FTS5_MAX_SEGMENT is currently defined as 2000. So the following
         3457  +      ** array is 63 elements, or 252 bytes, in size.  */
         3458  +      u32 aUsed[(FTS5_MAX_SEGMENT+31) / 32];
         3459  +      int iLvl, iSeg;
         3460  +      int i;
         3461  +      u32 mask;
         3462  +      memset(aUsed, 0, sizeof(aUsed));
         3463  +      for(iLvl=0; iLvl<pStruct->nLevel; iLvl++){
         3464  +        for(iSeg=0; iSeg<pStruct->aLevel[iLvl].nSeg; iSeg++){
         3465  +          int iId = pStruct->aLevel[iLvl].aSeg[iSeg].iSegid;
         3466  +          if( iId<=FTS5_MAX_SEGMENT ){
         3467  +            aUsed[(iId-1) / 32] |= 1 << ((iId-1) % 32);
  3465   3468             }
  3466   3469           }
  3467   3470         }
         3471  +
         3472  +      for(i=0; aUsed[i]==0xFFFFFFFF; i++);
         3473  +      mask = aUsed[i];
         3474  +      for(iSegid=0; mask & (1 << iSegid); iSegid++);
         3475  +      iSegid += 1 + i*32;
         3476  +
         3477  +#ifdef SQLITE_DEBUG
         3478  +      for(iLvl=0; iLvl<pStruct->nLevel; iLvl++){
         3479  +        for(iSeg=0; iSeg<pStruct->aLevel[iLvl].nSeg; iSeg++){
         3480  +          assert( iSegid!=pStruct->aLevel[iLvl].aSeg[iSeg].iSegid );
         3481  +        }
         3482  +      }
         3483  +      assert( iSegid>0 && iSegid<=FTS5_MAX_SEGMENT );
         3484  +#endif
  3468   3485       }
  3469   3486     }
  3470   3487   
  3471   3488     return iSegid;
  3472   3489   }
  3473   3490   
  3474   3491   /*
................................................................................
  3905   3922     Fts5PageWriter *pLeaf = &pWriter->writer;
  3906   3923     if( p->rc==SQLITE_OK ){
  3907   3924       assert( pLeaf->pgno>=1 );
  3908   3925       if( pLeaf->buf.n>4 ){
  3909   3926         fts5WriteFlushLeaf(p, pWriter);
  3910   3927       }
  3911   3928       *pnLeaf = pLeaf->pgno-1;
  3912         -    fts5WriteFlushBtree(p, pWriter);
         3929  +    if( pLeaf->pgno>1 ){
         3930  +      fts5WriteFlushBtree(p, pWriter);
         3931  +    }
  3913   3932     }
  3914   3933     fts5BufferFree(&pLeaf->term);
  3915   3934     fts5BufferFree(&pLeaf->buf);
  3916   3935     fts5BufferFree(&pLeaf->pgidx);
  3917   3936     fts5BufferFree(&pWriter->btterm);
  3918   3937   
  3919   3938     for(i=0; i<pWriter->nDlidx; i++){

Changes to ext/fts5/fts5_storage.c.

   141    141         if( rc!=SQLITE_OK && pzErrMsg ){
   142    142           *pzErrMsg = sqlite3_mprintf("%s", sqlite3_errmsg(pC->db));
   143    143         }
   144    144       }
   145    145     }
   146    146   
   147    147     *ppStmt = p->aStmt[eStmt];
          148  +  sqlite3_reset(*ppStmt);
   148    149     return rc;
   149    150   }
   150    151   
   151    152   
   152    153   static int fts5ExecPrintf(
   153    154     sqlite3 *db,
   154    155     char **pzErr,
................................................................................
  1117   1118       rc = sqlite3Fts5IndexSetCookie(p->pIndex, iNew);
  1118   1119       if( rc==SQLITE_OK ){
  1119   1120         p->pConfig->iCookie = iNew;
  1120   1121       }
  1121   1122     }
  1122   1123     return rc;
  1123   1124   }
  1124         -
  1125         -

Changes to ext/fts5/tool/fts5txt2db.tcl.

    13     13       {fts5                 "use fts5 (this is the default)"}
    14     14       {fts4                 "use fts4"}
    15     15       {colsize   "10 10 10" "list of column sizes"}
    16     16       {tblname   "t1"       "table name to create"}
    17     17       {detail    "full"     "Fts5 detail mode to use"}
    18     18       {repeat    1          "Load each file this many times"}
    19     19       {prefix    ""         "Fts prefix= option"}
           20  +    {trans     1          "True to use a transaction"}
    20     21       database
    21     22       file...
    22     23     } {
    23     24     This script is designed to create fts4/5 tables with more than one column.
    24     25     The -colsize option should be set to a Tcl list of integer values, one for
    25     26     each column in the table. Each value is the number of tokens that will be
    26     27     inserted into the column value for each row. For example, setting the -colsize
................................................................................
   210    211   set cols [create_table]
   211    212   set sql "INSERT INTO $A(tblname) VALUES(\$R([lindex $cols 0])"
   212    213   foreach c [lrange $cols 1 end] {
   213    214     append sql ", \$R($c)"
   214    215   }
   215    216   append sql ")"
   216    217   
   217         -db eval BEGIN
          218  +if {$A(trans)} { db eval BEGIN }
   218    219     while {$i < $N} {
   219    220       foreach c $cols s $A(colsize) {
   220    221         set R($c) [lrange $tokens $i [expr $i+$s-1]]
   221    222         incr i $s
   222    223       }
   223    224       db eval $sql
   224    225     }
   225         -db eval COMMIT
          226  +if {$A(trans)} { db eval COMMIT }
   226    227   
   227    228   
   228    229   

Changes to ext/rbu/rbu1.test.

   607    607            CREATE TABLE rbu.data_t1(a, b, rbu_control);
   608    608            INSERT INTO rbu.data_t1 VALUES(1, 2, 4);
   609    609          } {SQLITE_ERROR - invalid rbu_control value}
   610    610   
   611    611          9 {
   612    612            CREATE TABLE t1(a, b PRIMARY KEY) WITHOUT ROWID;
   613    613            CREATE TABLE rbu.data_t1(a, b, rbu_control);
   614         -         INSERT INTO rbu.data_t1 VALUES(1, 2, 2);
          614  +         INSERT INTO rbu.data_t1 VALUES(1, 2, 3);
   615    615          } {SQLITE_ERROR - invalid rbu_control value}
   616    616   
   617    617          10 {
   618    618            CREATE TABLE t2(a, b);
   619    619            CREATE TABLE rbu.data_t1(a, b, rbu_control);
   620    620            INSERT INTO rbu.data_t1 VALUES(1, 2, 2);
   621    621          } {SQLITE_ERROR - no such table: t1}

Changes to ext/rbu/rbudiff.test.

    14     14   #
    15     15   if {![info exists testdir]} {
    16     16     set testdir [file join [file dirname [info script]] .. .. test]
    17     17   }
    18     18   source $testdir/tester.tcl
    19     19   set testprefix rbudiff
    20     20   
    21         -if {$tcl_platform(platform)=="windows"} {
    22         -  set PROG "sqldiff.exe"
    23         -} else {
    24         -  set PROG "./sqldiff"
    25         -}
    26         -if {![file exe $PROG]} {
    27         -  puts "rbudiff.test cannot run because $PROG is not available"
    28         -  finish_test
    29         -  return
    30         -}
           21  +set PROG [test_find_sqldiff]
    31     22   db close
    32     23   
    33     24   proc get_rbudiff_sql {db1 db2} {
    34     25     exec $::PROG --rbu $db1 $db2
    35     26   }
    36     27   
    37     28   proc step_rbu {target rbu} {
................................................................................
    41     32       rbu close
    42     33       if {$rc != "SQLITE_OK"} break
    43     34     }
    44     35     set rc
    45     36   }
    46     37   
    47     38   proc apply_rbudiff {sql target} {
           39  +  test_rbucount $sql
    48     40     forcedelete rbu.db
    49     41     sqlite3 rbudb rbu.db
    50     42     rbudb eval $sql
    51     43     rbudb close
    52     44     step_rbu $target rbu.db
    53     45   }
           46  +
           47  +# The only argument is the output of an [sqldiff -rbu] run. This command
           48  +# tests that the contents of the rbu_count table is correct. An exception
           49  +# is thrown if it is not.
           50  +#
           51  +proc test_rbucount {sql} {
           52  +  sqlite3 tmpdb ""
           53  +  tmpdb eval $sql
           54  +  tmpdb eval {
           55  +    SELECT name FROM sqlite_master WHERE name LIKE 'data%' AND type='table'
           56  +  } {
           57  +    set a [tmpdb eval "SELECT count(*) FROM $name"]
           58  +    set b [tmpdb eval {SELECT cnt FROM rbu_count WHERE tbl = $name}]
           59  +    if {$a != $b} { 
           60  +      tmpdb close
           61  +      error "rbu_count error - tbl = $name" 
           62  +    }
           63  +  }
           64  +  tmpdb close
           65  +  return ""
           66  +}
    54     67   
    55     68   proc rbudiff_cksum {db1} {
    56     69     set txt ""
    57     70   
    58     71     sqlite3 dbtmp $db1
    59     72     foreach tbl [dbtmp eval {SELECT name FROM sqlite_master WHERE type='table'}] {
    60     73       set cols [list]

Added ext/rbu/rbuprogress.test.

            1  +# 2016 March 18
            2  +#
            3  +# The author disclaims copyright to this source code.  In place of
            4  +# a legal notice, here is a blessing:
            5  +#
            6  +#    May you do good and not evil.
            7  +#    May you find forgiveness for yourself and forgive others.
            8  +#    May you share freely, never taking more than you give.
            9  +#
           10  +#***********************************************************************
           11  +#
           12  +
           13  +source [file join [file dirname [info script]] rbu_common.tcl]
           14  +set ::testprefix rbuprogress
           15  +
           16  +
           17  +proc create_db_file {filename sql} {
           18  +  forcedelete $filename
           19  +  sqlite3 tmpdb $filename  
           20  +  tmpdb eval $sql
           21  +  tmpdb close
           22  +}
           23  +
           24  +# Create a simple RBU database. That expects to write to a table:
           25  +#
           26  +#   CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
           27  +#
           28  +proc create_rbu1 {filename} {
           29  +  create_db_file $filename {
           30  +    CREATE TABLE data_t1(a, b, c, rbu_control);
           31  +    INSERT INTO data_t1 VALUES(1, 2, 3, 0);
           32  +    INSERT INTO data_t1 VALUES(2, 'two', 'three', 0);
           33  +    INSERT INTO data_t1 VALUES(3, NULL, 8.2, 0);
           34  +
           35  +    CREATE TABLE rbu_count(tbl, cnt);
           36  +    INSERT INTO rbu_count VALUES('data_t1', 3);
           37  +  }
           38  +  return $filename
           39  +}
           40  +
           41  +
           42  +do_execsql_test 1.0 {
           43  +  CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
           44  +}
           45  +
           46  +do_test 1.1 {
           47  +  create_rbu1 rbu.db
           48  +  sqlite3rbu rbu test.db rbu.db
           49  +  rbu bp_progress
           50  +} {0 0}
           51  +do_test 1.2 { rbu step ; rbu bp_progress } {3333 0}
           52  +do_test 1.3 { rbu step ; rbu bp_progress } {6666 0}
           53  +do_test 1.4 { rbu step ; rbu bp_progress } {10000 0}
           54  +do_test 1.5 { rbu step ; rbu bp_progress } {10000 0}
           55  +do_test 1.6 { rbu step ; rbu bp_progress } {10000 0}
           56  +do_test 1.7 { rbu step ; rbu bp_progress } {10000 5000}
           57  +do_test 1.8 { rbu step ; rbu bp_progress } {10000 10000}
           58  +do_test 1.9 { rbu step ; rbu bp_progress } {10000 10000}
           59  +
           60  +do_test 1.10 {
           61  +  rbu close
           62  +} {SQLITE_DONE}
           63  +
           64  +#-------------------------------------------------------------------------
           65  +#
           66  +proc do_sp_test {tn bReopen target rbu reslist} {
           67  +  uplevel [list do_test $tn [subst -nocommands {
           68  +    if {$bReopen==0} { sqlite3rbu rbu $target $rbu }
           69  +    set res [list]
           70  +    while 1 {
           71  +      if {$bReopen} { sqlite3rbu rbu $target $rbu }
           72  +      set rc [rbu step]
           73  +      if {[set rc] != "SQLITE_OK"} { rbu close ; error "error 1" }
           74  +      lappend res [lindex [rbu bp_progress] 0]
           75  +      if {[lindex [set res] end]==10000} break
           76  +      if {$bReopen} { rbu close }
           77  +    }
           78  +    if {[set res] != [list $reslist]} {
           79  +      rbu close
           80  +      error "1. reslist incorrect (expect=$reslist got=[set res])"
           81  +    }
           82  +
           83  +    # One step to clean up the temporary tables used to update the only
           84  +    # target table in the rbu database. And one more to move the *-oal 
           85  +    # file to *-wal. After each of these steps, the progress remains
           86  +    # at "10000 0".
           87  +    #
           88  +    if {[lindex [list $reslist] 0]!=-1} {
           89  +      rbu step
           90  +      set res [rbu bp_progress]
           91  +      if {[set res] != [list 10000 0]} {
           92  +        rbu close
           93  +        error "2. reslist incorrect (expect=10000 0 got=[set res])"
           94  +      }
           95  +    }
           96  +
           97  +    rbu step
           98  +    set res [rbu bp_progress]
           99  +    if {[set res] != [list 10000 0]} {
          100  +      rbu close
          101  +      error "3. reslist incorrect (expect=10000 0 got=[set res])"
          102  +    }
          103  +
          104  +    # Do the checkpoint.
          105  +    while {[rbu step]=="SQLITE_OK"} { 
          106  +      foreach {a b} [rbu bp_progress] {}
          107  +      if {[set a]!=10000 || [set b]<=0 || [set b]>10000} {
          108  +        rbu close
          109  +        error "4. reslist incorrect (expect=10000 1..10000 got=[set a] [set b])"
          110  +      }
          111  +    }
          112  +
          113  +    set res [rbu bp_progress]
          114  +    if {[set res] != [list 10000 10000]} {
          115  +      rbu close
          116  +      error "5. reslist is incorrect (expect=10000 10000 got=[set res])"
          117  +    }
          118  +
          119  +    rbu close
          120  +  }] {SQLITE_DONE}]
          121  +}
          122  +
          123  +foreach {bReopen} { 0 1 } {
          124  +  reset_db
          125  +  do_test 2.$bReopen.1.0 {
          126  +    execsql {
          127  +      CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
          128  +    }
          129  +    create_db_file rbu.db {
          130  +      CREATE TABLE data_t1(a, b, c, rbu_control);
          131  +      INSERT INTO data_t1 VALUES(4, 4, 4, 0);
          132  +      INSERT INTO data_t1 VALUES(5, 5, 5, 0);
          133  +  
          134  +      CREATE TABLE rbu_count(tbl, cnt);
          135  +      INSERT INTO rbu_count VALUES('data_t1', 2);
          136  +    }
          137  +  } {}
          138  +  do_sp_test 2.$bReopen.1.1 $bReopen test.db rbu.db {5000 10000}
          139  +  
          140  +  reset_db
          141  +  do_test 2.$bReopen.2.0 {
          142  +    execsql { CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c) }
          143  +    create_rbu1 rbu.db
          144  +  } {rbu.db}
          145  +  do_sp_test 2.$bReopen.2.1 $bReopen test.db rbu.db {3333 6666 10000}
          146  +  
          147  +  reset_db
          148  +  do_test 2.$bReopen.3.0 {
          149  +    execsql { 
          150  +      CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
          151  +      CREATE INDEX i1 ON t1(b);
          152  +      INSERT INTO t1 VALUES(1, 1, 1);
          153  +      INSERT INTO t1 VALUES(2, 2, 2);
          154  +      INSERT INTO t1 VALUES(3, 3, 3);
          155  +    }
          156  +    create_db_file rbu.db {
          157  +      CREATE TABLE data_t1(a, b, c, rbu_control);
          158  +      INSERT INTO data_t1 VALUES(4, 4, 4, 0);
          159  +      INSERT INTO data_t1 VALUES(2, NULL, NULL, 1);
          160  +      INSERT INTO data_t1 VALUES(5, NULL, NULL, 1);
          161  +  
          162  +      CREATE TABLE rbu_count(tbl, cnt);
          163  +      INSERT INTO rbu_count VALUES('data_t1', 3);
          164  +    }
          165  +  } {}
          166  +  do_sp_test 2.$bReopen.3.1 $bReopen test.db rbu.db {1666 3333 6000 8000 10000}
          167  +  
          168  +  reset_db
          169  +  do_test 2.$bReopen.4.0 {
          170  +    execsql { 
          171  +      CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
          172  +      CREATE INDEX i1 ON t1(b);
          173  +      INSERT INTO t1 VALUES(1, 1, 1);
          174  +      INSERT INTO t1 VALUES(2, 2, 2);
          175  +      INSERT INTO t1 VALUES(3, 3, 3);
          176  +    }
          177  +    create_db_file rbu.db {
          178  +      CREATE TABLE data_t1(a, b, c, rbu_control);
          179  +      INSERT INTO data_t1 VALUES(2, 4, 4, '.xx');
          180  +  
          181  +      CREATE TABLE rbu_count(tbl, cnt);
          182  +      INSERT INTO rbu_count VALUES('data_t1', 1);
          183  +    }
          184  +  } {}
          185  +  do_sp_test 2.$bReopen.4.1 $bReopen test.db rbu.db {3333 6666 10000}
          186  +  
          187  +  reset_db
          188  +  do_test 2.$bReopen.5.0 {
          189  +    execsql { 
          190  +      CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
          191  +      CREATE INDEX i1 ON t1(b);
          192  +      INSERT INTO t1 VALUES(1, 1, 1);
          193  +      INSERT INTO t1 VALUES(2, 2, 2);
          194  +      INSERT INTO t1 VALUES(3, 3, 3);
          195  +    }
          196  +    create_db_file rbu.db {
          197  +      CREATE TABLE data_t1(a, b, c, rbu_control);
          198  +      INSERT INTO data_t1 VALUES(4, NULL, 4, '.xx');
          199  +  
          200  +      CREATE TABLE rbu_count(tbl, cnt);
          201  +      INSERT INTO rbu_count VALUES('data_t1', 1);
          202  +    }
          203  +  } {}
          204  +  do_sp_test 2.$bReopen.5.1 $bReopen test.db rbu.db {10000}
          205  +
          206  +  reset_db
          207  +  do_test 2.$bReopen.6.0 {
          208  +    execsql { 
          209  +      CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
          210  +      CREATE INDEX i1 ON t1(b);
          211  +      INSERT INTO t1 VALUES(1, 1, 1);
          212  +      INSERT INTO t1 VALUES(2, 2, 2);
          213  +      INSERT INTO t1 VALUES(3, 3, 3);
          214  +    }
          215  +    create_db_file rbu.db {
          216  +      CREATE TABLE data_t1(a, b, c, rbu_control);
          217  +      INSERT INTO data_t1 VALUES(4, 4, 4, 0);
          218  +      INSERT INTO data_t1 VALUES(2, NULL, NULL, 1);
          219  +      INSERT INTO data_t1 VALUES(5, NULL, NULL, 1);
          220  +    }
          221  +  } {}
          222  +  do_sp_test 2.$bReopen.6.1 $bReopen test.db rbu.db {-1 -1 -1 -1 -1 10000}
          223  +}
          224  +
          225  +#-------------------------------------------------------------------------
          226  +# The following tests verify that the API works when resuming an update
          227  +# during the incremental checkpoint stage.
          228  +#
          229  +proc do_phase2_test {tn bReopen target rbu nStep} {
          230  +  uplevel [list do_test $tn [subst -nocommands {
          231  +
          232  +    # Build the OAL/WAL file:
          233  +    sqlite3rbu rbu $target $rbu
          234  +    while {[lindex [rbu bp_progress] 0]<10000} { 
          235  +      set rc [rbu step]
          236  +      if {"SQLITE_OK" != [set rc]} { rbu close }
          237  +    }
          238  +
          239  +    # Clean up the temp tables and move the *-oal file to *-wal.
          240  +    rbu step
          241  +    rbu step
          242  +
          243  +    for {set i 0} {[set i] < $nStep} {incr i} {
          244  +      if {$bReopen} {
          245  +        rbu close
          246  +        sqlite3rbu rbu $target $rbu
          247  +      }
          248  +      rbu step
          249  +      set res [rbu bp_progress]
          250  +      set expect [expr (1 + [set i]) * 10000 / $nStep]
          251  +      if {[lindex [set res] 1] != [set expect]} {
          252  +        error "Have [set res], expected 10000 [set expect]"
          253  +      }
          254  +    }
          255  +
          256  +    set rc [rbu step]
          257  +    if {[set rc] != "SQLITE_DONE"} {
          258  +      error "Have [set rc], expected SQLITE_DONE" 
          259  +    }
          260  +
          261  +    rbu close
          262  +  }] {SQLITE_DONE}]
          263  +}
          264  +
          265  +foreach bReopen {0 1} {
          266  +  do_test 3.$bReopen.1.0 {
          267  +    reset_db
          268  +    execsql {
          269  +      CREATE TABLE t1(a INTEGER PRIMARY KEY, b);
          270  +      CREATE TABLE t2(a INTEGER PRIMARY KEY, b);
          271  +      CREATE TABLE t3(a INTEGER PRIMARY KEY, b);
          272  +      CREATE TABLE t4(a INTEGER PRIMARY KEY, b);
          273  +    }
          274  +    create_db_file rbu.db {
          275  +      CREATE TABLE data_t1(a, b, rbu_control);
          276  +      CREATE TABLE data_t2(a, b, rbu_control);
          277  +      CREATE TABLE data_t3(a, b, rbu_control);
          278  +      CREATE TABLE data_t4(a, b, rbu_control);
          279  +      INSERT INTO data_t1 VALUES(1, 2, 0);
          280  +      INSERT INTO data_t2 VALUES(1, 2, 0);
          281  +      INSERT INTO data_t3 VALUES(1, 2, 0);
          282  +      INSERT INTO data_t4 VALUES(1, 2, 0);
          283  +  
          284  +      CREATE TABLE rbu_count(tbl, cnt);
          285  +      INSERT INTO rbu_count VALUES('data_t1', 1);
          286  +      INSERT INTO rbu_count VALUES('data_t2', 1);
          287  +      INSERT INTO rbu_count VALUES('data_t3', 1);
          288  +      INSERT INTO rbu_count VALUES('data_t4', 1);
          289  +    }
          290  +  } {}
          291  +  do_phase2_test 3.$bReopen.1.1 $bReopen test.db rbu.db 5
          292  +}
          293  +
          294  +
          295  +foreach {bReopen} { 0 1 } {
          296  +  foreach {tn tbl} {
          297  +    ipk { CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c) }
          298  +    wr  { CREATE TABLE t1(a INT PRIMARY KEY, b, c) WITHOUT ROWID }
          299  +    pk  { CREATE TABLE t1(a INT PRIMARY KEY, b, c) }
          300  +  } {
          301  +
          302  +    foreach {tn2 rbusql r1 r3} {
          303  +      1 {
          304  +        CREATE TABLE data0_t1(a, b, c, rbu_control);
          305  +        INSERT INTO data0_t1 VALUES(15, 15, 15, 0);
          306  +        INSERT INTO data0_t1 VALUES(20, 20, 20, 0);
          307  +        CREATE TABLE rbu_count(tbl, cnt);
          308  +        INSERT INTO rbu_count VALUES('data0_t1', 2); 
          309  +      } 
          310  +      {2500 5000 7500 10000}
          311  +      {1666 3333 5000 6666 8333 10000}
          312  +
          313  +      2 {
          314  +        CREATE TABLE data0_t1(a, b, c, rbu_control);
          315  +        INSERT INTO data0_t1 VALUES(10, 10, 10, 2);
          316  +        CREATE TABLE rbu_count(tbl, cnt);
          317  +        INSERT INTO rbu_count VALUES('data0_t1', 1); 
          318  +      } 
          319  +      {3333 6666 10000}
          320  +      {2000 4000 6000 8000 10000}
          321  +
          322  +      3 {
          323  +        CREATE TABLE data0_t1(a, b, c, rbu_control);
          324  +        INSERT INTO data0_t1 VALUES(7, 7, 7, 2);
          325  +        INSERT INTO data0_t1 VALUES(10, 10, 10, 2);
          326  +        CREATE TABLE rbu_count(tbl, cnt);
          327  +        INSERT INTO rbu_count VALUES('data0_t1', 2); 
          328  +      } 
          329  +      {2500 4000 6000 8000 10000}
          330  +      {1666 2500 3750 5000 6250 7500 8750 10000}
          331  +
          332  +    } {
          333  +
          334  +      reset_db ; execsql $tbl
          335  +      do_test 4.$tn.$bReopen.$tn2.0 {
          336  +        execsql {
          337  +          CREATE INDEX t1c ON t1(c);
          338  +          INSERT INTO t1 VALUES(1, 1, 1);
          339  +          INSERT INTO t1 VALUES(5, 5, 5);
          340  +          INSERT INTO t1 VALUES(10, 10, 10);
          341  +        }
          342  +        create_db_file rbu.db $rbusql
          343  +      } {}
          344  +
          345  +      set R(ipk) $r1
          346  +      set R(wr) $r1
          347  +      set R(pk) $r3
          348  +      do_sp_test 4.$tn.$bReopen.$tn2.1 $bReopen test.db rbu.db $R($tn)
          349  +    }
          350  +  }
          351  +}
          352  +
          353  +foreach {bReopen} { 0 1 } {
          354  +  foreach {tn tbl} {
          355  +    nopk { 
          356  +      CREATE TABLE t1(a, b, c);
          357  +      CREATE INDEX t1c ON t1(c);
          358  +    }
          359  +    vtab { 
          360  +      CREATE VIRTUAL TABLE t1 USING fts5(a, b, c);
          361  +    }
          362  +  } {
          363  +
          364  +    foreach {tn2 rbusql r1 r2} {
          365  +      1 {
          366  +        CREATE TABLE data0_t1(a, b, c, rbu_rowid, rbu_control);
          367  +        INSERT INTO data0_t1 VALUES(15, 15, 15, 4, 0);
          368  +        INSERT INTO data0_t1 VALUES(20, 20, 20, 5, 0);
          369  +        CREATE TABLE rbu_count(tbl, cnt);
          370  +        INSERT INTO rbu_count VALUES('data0_t1', 2); 
          371  +      } 
          372  +      {2500 5000 7500 10000}
          373  +      {5000 10000}
          374  +
          375  +      2 {
          376  +        CREATE TABLE data0_t1(rbu_rowid, a, b, c, rbu_control);
          377  +        INSERT INTO data0_t1 VALUES(0, 7, 7, 7, 2);
          378  +        INSERT INTO data0_t1 VALUES(2, 10, 10, 10, 2);
          379  +        CREATE TABLE rbu_count(tbl, cnt);
          380  +        INSERT INTO rbu_count VALUES('data0_t1', 2); 
          381  +      } 
          382  +      {2500 4000 6000 8000 10000}
          383  +      {5000 10000}
          384  +
          385  +      3 {
          386  +        CREATE TABLE data0_t1(rbu_rowid, a, b, c, rbu_control);
          387  +        INSERT INTO data0_t1 VALUES(1, NULL, NULL, NULL, 1);
          388  +        INSERT INTO data0_t1 VALUES(2, NULL, NULL, 7, '..x');
          389  +        CREATE TABLE rbu_count(tbl, cnt);
          390  +        INSERT INTO rbu_count VALUES('data0_t1', 2); 
          391  +      } 
          392  +      {2500 4000 6000 8000 10000}
          393  +      {5000 10000}
          394  +    } {
          395  +
          396  +      reset_db ; execsql $tbl
          397  +      do_test 5.$tn.$bReopen.$tn2.0 {
          398  +        execsql {
          399  +          INSERT INTO t1 VALUES(1, 1, 1);
          400  +          INSERT INTO t1 VALUES(5, 5, 5);
          401  +          INSERT INTO t1 VALUES(10, 10, 10);
          402  +        }
          403  +        create_db_file rbu.db $rbusql
          404  +      } {}
          405  +
          406  +      set R(nopk) $r1
          407  +      set R(vtab) $r2
          408  +      do_sp_test 5.$tn.$bReopen.$tn2.1 $bReopen test.db rbu.db $R($tn)
          409  +    }
          410  +  }
          411  +}
          412  +
          413  +
          414  +finish_test
          415  +

Changes to ext/rbu/sqlite3rbu.c.

   143    143   ** RBU_STATE_COOKIE:
   144    144   **   Valid if STAGE==1. The current change-counter cookie value in the 
   145    145   **   target db file.
   146    146   **
   147    147   ** RBU_STATE_OALSZ:
   148    148   **   Valid if STAGE==1. The size in bytes of the *-oal file.
   149    149   */
   150         -#define RBU_STATE_STAGE       1
   151         -#define RBU_STATE_TBL         2
   152         -#define RBU_STATE_IDX         3
   153         -#define RBU_STATE_ROW         4
   154         -#define RBU_STATE_PROGRESS    5
   155         -#define RBU_STATE_CKPT        6
   156         -#define RBU_STATE_COOKIE      7
   157         -#define RBU_STATE_OALSZ       8
          150  +#define RBU_STATE_STAGE        1
          151  +#define RBU_STATE_TBL          2
          152  +#define RBU_STATE_IDX          3
          153  +#define RBU_STATE_ROW          4
          154  +#define RBU_STATE_PROGRESS     5
          155  +#define RBU_STATE_CKPT         6
          156  +#define RBU_STATE_COOKIE       7
          157  +#define RBU_STATE_OALSZ        8
          158  +#define RBU_STATE_PHASEONESTEP 9
   158    159   
   159    160   #define RBU_STAGE_OAL         1
   160    161   #define RBU_STAGE_MOVE        2
   161    162   #define RBU_STAGE_CAPTURE     3
   162    163   #define RBU_STAGE_CKPT        4
   163    164   #define RBU_STAGE_DONE        5
   164    165   
................................................................................
   196    197     char *zTbl;
   197    198     char *zIdx;
   198    199     i64 iWalCksum;
   199    200     int nRow;
   200    201     i64 nProgress;
   201    202     u32 iCookie;
   202    203     i64 iOalSz;
          204  +  i64 nPhaseOneStep;
   203    205   };
   204    206   
   205    207   struct RbuUpdateStmt {
   206    208     char *zMask;                    /* Copy of update mask used with pUpdate */
   207    209     sqlite3_stmt *pUpdate;          /* Last update statement (or NULL) */
   208    210     RbuUpdateStmt *pNext;
   209    211   };
................................................................................
   240    242     int bCleanup;                   /* True in "cleanup" state */
   241    243     const char *zTbl;               /* Name of target db table */
   242    244     const char *zDataTbl;           /* Name of rbu db table (or null) */
   243    245     const char *zIdx;               /* Name of target db index (or null) */
   244    246     int iTnum;                      /* Root page of current object */
   245    247     int iPkTnum;                    /* If eType==EXTERNAL, root of PK index */
   246    248     int bUnique;                    /* Current index is unique */
          249  +  int nIndex;                     /* Number of aux. indexes on table zTbl */
   247    250   
   248    251     /* Statements created by rbuObjIterPrepareAll() */
   249    252     int nCol;                       /* Number of columns in current object */
   250    253     sqlite3_stmt *pSelect;          /* Source data */
   251    254     sqlite3_stmt *pInsert;          /* Statement for INSERT operations */
   252    255     sqlite3_stmt *pDelete;          /* Statement for DELETE ops */
   253    256     sqlite3_stmt *pTmpInsert;       /* Insert into rbu_tmp_$zDataTbl */
................................................................................
   293    296   struct RbuFrame {
   294    297     u32 iDbPage;
   295    298     u32 iWalFrame;
   296    299   };
   297    300   
   298    301   /*
   299    302   ** RBU handle.
          303  +**
          304  +** nPhaseOneStep:
          305  +**   If the RBU database contains an rbu_count table, this value is set to
          306  +**   a running estimate of the number of b-tree operations required to 
          307  +**   finish populating the *-oal file. This allows the sqlite3_bp_progress()
          308  +**   API to calculate the permyriadage progress of populating the *-oal file
          309  +**   using the formula:
          310  +**
          311  +**     permyriadage = (10000 * nProgress) / nPhaseOneStep
          312  +**
          313  +**   nPhaseOneStep is initialized to the sum of:
          314  +**
          315  +**     nRow * (nIndex + 1)
          316  +**
          317  +**   for all source tables in the RBU database, where nRow is the number
          318  +**   of rows in the source table and nIndex the number of indexes on the
          319  +**   corresponding target database table.
          320  +**
          321  +**   This estimate is accurate if the RBU update consists entirely of
          322  +**   INSERT operations. However, it is inaccurate if:
          323  +**
          324  +**     * the RBU update contains any UPDATE operations. If the PK specified
          325  +**       for an UPDATE operation does not exist in the target table, then
          326  +**       no b-tree operations are required on index b-trees. Or if the 
          327  +**       specified PK does exist, then (nIndex*2) such operations are
          328  +**       required (one delete and one insert on each index b-tree).
          329  +**
          330  +**     * the RBU update contains any DELETE operations for which the specified
          331  +**       PK does not exist. In this case no operations are required on index
          332  +**       b-trees.
          333  +**
          334  +**     * the RBU update contains REPLACE operations. These are similar to
          335  +**       UPDATE operations.
          336  +**
          337  +**   nPhaseOneStep is updated to account for the conditions above during the
          338  +**   first pass of each source table. The updated nPhaseOneStep value is
          339  +**   stored in the rbu_state table if the RBU update is suspended.
   300    340   */
   301    341   struct sqlite3rbu {
   302    342     int eStage;                     /* Value of RBU_STATE_STAGE field */
   303    343     sqlite3 *dbMain;                /* target database handle */
   304    344     sqlite3 *dbRbu;                 /* rbu database handle */
   305    345     char *zTarget;                  /* Path to target db */
   306    346     char *zRbu;                     /* Path to rbu db */
................................................................................
   310    350     char *zErrmsg;                  /* Error message if rc!=SQLITE_OK */
   311    351     int nStep;                      /* Rows processed for current object */
   312    352     int nProgress;                  /* Rows processed for all objects */
   313    353     RbuObjIter objiter;             /* Iterator for skipping through tbl/idx */
   314    354     const char *zVfsName;           /* Name of automatically created rbu vfs */
   315    355     rbu_file *pTargetFd;            /* File handle open on target db */
   316    356     i64 iOalSz;
          357  +  i64 nPhaseOneStep;
   317    358   
   318    359     /* The following state variables are used as part of the incremental
   319    360     ** checkpoint stage (eStage==RBU_STAGE_CKPT). See comments surrounding
   320    361     ** function rbuSetupCheckpoint() for details.  */
   321    362     u32 iMaxFrame;                  /* Largest iWalFrame value in aFrame[] */
   322    363     u32 mLock;
   323    364     int nFrame;                     /* Entries in aFrame[] array */
................................................................................
  1140   1181     if( p->rc==SQLITE_OK ){
  1141   1182       memcpy(pIter->abIndexed, pIter->abTblPk, sizeof(u8)*pIter->nTblCol);
  1142   1183       p->rc = prepareFreeAndCollectError(p->dbMain, &pList, &p->zErrmsg,
  1143   1184           sqlite3_mprintf("PRAGMA main.index_list = %Q", pIter->zTbl)
  1144   1185       );
  1145   1186     }
  1146   1187   
         1188  +  pIter->nIndex = 0;
  1147   1189     while( p->rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pList) ){
  1148   1190       const char *zIdx = (const char*)sqlite3_column_text(pList, 1);
  1149   1191       sqlite3_stmt *pXInfo = 0;
  1150   1192       if( zIdx==0 ) break;
  1151   1193       p->rc = prepareFreeAndCollectError(p->dbMain, &pXInfo, &p->zErrmsg,
  1152   1194           sqlite3_mprintf("PRAGMA main.index_xinfo = %Q", zIdx)
  1153   1195       );
  1154   1196       while( p->rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pXInfo) ){
  1155   1197         int iCid = sqlite3_column_int(pXInfo, 1);
  1156   1198         if( iCid>=0 ) pIter->abIndexed[iCid] = 1;
  1157   1199       }
  1158   1200       rbuFinalize(p, pXInfo);
  1159   1201       bIndex = 1;
         1202  +    pIter->nIndex++;
         1203  +  }
         1204  +
         1205  +  if( pIter->eType==RBU_PK_WITHOUT_ROWID ){
         1206  +    /* "PRAGMA index_list" includes the main PK b-tree */
         1207  +    pIter->nIndex--;
  1160   1208     }
  1161   1209   
  1162   1210     rbuFinalize(p, pList);
  1163   1211     if( bIndex==0 ) pIter->abIndexed = 0;
  1164   1212   }
  1165   1213   
  1166   1214   
................................................................................
  1266   1314           iOrder++;
  1267   1315         }
  1268   1316       }
  1269   1317   
  1270   1318       rbuFinalize(p, pStmt);
  1271   1319       rbuObjIterCacheIndexedCols(p, pIter);
  1272   1320       assert( pIter->eType!=RBU_PK_VTAB || pIter->abIndexed==0 );
         1321  +    assert( pIter->eType!=RBU_PK_VTAB || pIter->nIndex==0 );
  1273   1322     }
  1274   1323   
  1275   1324     return p->rc;
  1276   1325   }
  1277   1326   
  1278   1327   /*
  1279   1328   ** This function constructs and returns a pointer to a nul-terminated 
................................................................................
  1818   1867     sqlite3_context *pCtx, 
  1819   1868     int nVal,
  1820   1869     sqlite3_value **apVal
  1821   1870   ){
  1822   1871     sqlite3rbu *p = sqlite3_user_data(pCtx);
  1823   1872     int rc = SQLITE_OK;
  1824   1873     int i;
         1874  +
         1875  +  assert( sqlite3_value_int(apVal[0])!=0
         1876  +      || p->objiter.eType==RBU_PK_EXTERNAL 
         1877  +      || p->objiter.eType==RBU_PK_NONE 
         1878  +  );
         1879  +  if( sqlite3_value_int(apVal[0])!=0 ){
         1880  +    p->nPhaseOneStep += p->objiter.nIndex;
         1881  +  }
  1825   1882   
  1826   1883     for(i=0; rc==SQLITE_OK && i<nVal; i++){
  1827   1884       rc = sqlite3_bind_value(p->objiter.pTmpInsert, i+1, apVal[i]);
  1828   1885     }
  1829   1886     if( rc==SQLITE_OK ){
  1830   1887       sqlite3_step(p->objiter.pTmpInsert);
  1831   1888       rc = sqlite3_reset(p->objiter.pTmpInsert);
................................................................................
  2563   2620     RbuObjIter *pIter = &p->objiter;
  2564   2621     sqlite3_value *pVal;
  2565   2622     sqlite3_stmt *pWriter;
  2566   2623     int i;
  2567   2624   
  2568   2625     assert( p->rc==SQLITE_OK );
  2569   2626     assert( eType!=RBU_DELETE || pIter->zIdx==0 );
         2627  +  assert( eType==RBU_DELETE || eType==RBU_IDX_DELETE
         2628  +       || eType==RBU_INSERT || eType==RBU_IDX_INSERT
         2629  +  );
         2630  +
         2631  +  /* If this is a delete, decrement nPhaseOneStep by nIndex. If the DELETE
         2632  +  ** statement below does actually delete a row, nPhaseOneStep will be
         2633  +  ** incremented by the same amount when SQL function rbu_tmp_insert()
         2634  +  ** is invoked by the trigger.  */
         2635  +  if( eType==RBU_DELETE ){
         2636  +    p->nPhaseOneStep -= p->objiter.nIndex;
         2637  +  }
  2570   2638   
  2571   2639     if( eType==RBU_IDX_DELETE || eType==RBU_DELETE ){
  2572   2640       pWriter = pIter->pDelete;
  2573   2641     }else{
  2574   2642       pWriter = pIter->pInsert;
  2575   2643     }
  2576   2644   
................................................................................
  2634   2702     if( eType ){
  2635   2703       assert( eType==RBU_INSERT     || eType==RBU_DELETE
  2636   2704            || eType==RBU_REPLACE    || eType==RBU_IDX_DELETE
  2637   2705            || eType==RBU_IDX_INSERT || eType==RBU_UPDATE
  2638   2706       );
  2639   2707       assert( eType!=RBU_UPDATE || pIter->zIdx==0 );
  2640   2708   
  2641         -    if( pIter->zIdx==0 && eType==RBU_IDX_DELETE ){
         2709  +    if( pIter->zIdx==0 && (eType==RBU_IDX_DELETE || eType==RBU_IDX_INSERT) ){
  2642   2710         rbuBadControlError(p);
  2643   2711       }
  2644   2712       else if( eType==RBU_REPLACE ){
  2645         -      if( pIter->zIdx==0 ) rbuStepOneOp(p, RBU_DELETE);
         2713  +      if( pIter->zIdx==0 ){
         2714  +        p->nPhaseOneStep += p->objiter.nIndex;
         2715  +        rbuStepOneOp(p, RBU_DELETE);
         2716  +      }
  2646   2717         if( p->rc==SQLITE_OK ) rbuStepOneOp(p, RBU_INSERT);
  2647   2718       }
  2648   2719       else if( eType!=RBU_UPDATE ){
  2649   2720         rbuStepOneOp(p, eType);
  2650   2721       }
  2651   2722       else{
  2652   2723         sqlite3_value *pVal;
  2653   2724         sqlite3_stmt *pUpdate = 0;
  2654   2725         assert( eType==RBU_UPDATE );
         2726  +      p->nPhaseOneStep -= p->objiter.nIndex;
  2655   2727         rbuGetUpdateStmt(p, pIter, zMask, &pUpdate);
  2656   2728         if( pUpdate ){
  2657   2729           int i;
  2658   2730           for(i=0; p->rc==SQLITE_OK && i<pIter->nCol; i++){
  2659   2731             char c = zMask[pIter->aiSrcOrder[i]];
  2660   2732             pVal = sqlite3_column_value(pIter->pSelect, i);
  2661   2733             if( pIter->abTblPk[i] || c!='.' ){
................................................................................
  2725   2797             "(%d, %d), "
  2726   2798             "(%d, %Q), "
  2727   2799             "(%d, %Q), "
  2728   2800             "(%d, %d), "
  2729   2801             "(%d, %d), "
  2730   2802             "(%d, %lld), "
  2731   2803             "(%d, %lld), "
         2804  +          "(%d, %lld), "
  2732   2805             "(%d, %lld) ",
  2733   2806             p->zStateDb,
  2734   2807             RBU_STATE_STAGE, eStage,
  2735   2808             RBU_STATE_TBL, p->objiter.zTbl, 
  2736   2809             RBU_STATE_IDX, p->objiter.zIdx, 
  2737   2810             RBU_STATE_ROW, p->nStep, 
  2738   2811             RBU_STATE_PROGRESS, p->nProgress,
  2739   2812             RBU_STATE_CKPT, p->iWalCksum,
  2740   2813             RBU_STATE_COOKIE, (i64)p->pTargetFd->iCookie,
  2741         -          RBU_STATE_OALSZ, p->iOalSz
         2814  +          RBU_STATE_OALSZ, p->iOalSz,
         2815  +          RBU_STATE_PHASEONESTEP, p->nPhaseOneStep
  2742   2816         )
  2743   2817       );
  2744   2818       assert( pInsert==0 || rc==SQLITE_OK );
  2745   2819   
  2746   2820       if( rc==SQLITE_OK ){
  2747   2821         sqlite3_step(pInsert);
  2748   2822         rc = sqlite3_finalize(pInsert);
................................................................................
  2921   2995           pRet->iCookie = (u32)sqlite3_column_int64(pStmt, 1);
  2922   2996           break;
  2923   2997   
  2924   2998         case RBU_STATE_OALSZ:
  2925   2999           pRet->iOalSz = (u32)sqlite3_column_int64(pStmt, 1);
  2926   3000           break;
  2927   3001   
         3002  +      case RBU_STATE_PHASEONESTEP:
         3003  +        pRet->nPhaseOneStep = sqlite3_column_int64(pStmt, 1);
         3004  +        break;
         3005  +
  2928   3006         default:
  2929   3007           rc = SQLITE_CORRUPT;
  2930   3008           break;
  2931   3009       }
  2932   3010     }
  2933   3011     rc2 = sqlite3_finalize(pStmt);
  2934   3012     if( rc==SQLITE_OK ) rc = rc2;
................................................................................
  3027   3105   */
  3028   3106   static void rbuDeleteVfs(sqlite3rbu *p){
  3029   3107     if( p->zVfsName ){
  3030   3108       sqlite3rbu_destroy_vfs(p->zVfsName);
  3031   3109       p->zVfsName = 0;
  3032   3110     }
  3033   3111   }
         3112  +
         3113  +/*
         3114  +** This user-defined SQL function is invoked with a single argument - the
         3115  +** name of a table expected to appear in the target database. It returns
         3116  +** the number of auxilliary indexes on the table.
         3117  +*/
         3118  +static void rbuIndexCntFunc(
         3119  +  sqlite3_context *pCtx, 
         3120  +  int nVal,
         3121  +  sqlite3_value **apVal
         3122  +){
         3123  +  sqlite3rbu *p = (sqlite3rbu*)sqlite3_user_data(pCtx);
         3124  +  sqlite3_stmt *pStmt = 0;
         3125  +  char *zErrmsg = 0;
         3126  +  int rc;
         3127  +
         3128  +  assert( nVal==1 );
         3129  +  
         3130  +  rc = prepareFreeAndCollectError(p->dbMain, &pStmt, &zErrmsg, 
         3131  +      sqlite3_mprintf("SELECT count(*) FROM sqlite_master "
         3132  +        "WHERE type='index' AND tbl_name = %Q", sqlite3_value_text(apVal[0]))
         3133  +  );
         3134  +  if( rc!=SQLITE_OK ){
         3135  +    sqlite3_result_error(pCtx, zErrmsg, -1);
         3136  +  }else{
         3137  +    int nIndex = 0;
         3138  +    if( SQLITE_ROW==sqlite3_step(pStmt) ){
         3139  +      nIndex = sqlite3_column_int(pStmt, 0);
         3140  +    }
         3141  +    rc = sqlite3_finalize(pStmt);
         3142  +    if( rc==SQLITE_OK ){
         3143  +      sqlite3_result_int(pCtx, nIndex);
         3144  +    }else{
         3145  +      sqlite3_result_error(pCtx, sqlite3_errmsg(p->dbMain), -1);
         3146  +    }
         3147  +  }
         3148  +
         3149  +  sqlite3_free(zErrmsg);
         3150  +}
         3151  +
         3152  +/*
         3153  +** If the RBU database contains the rbu_count table, use it to initialize
         3154  +** the sqlite3rbu.nPhaseOneStep variable. The schema of the rbu_count table
         3155  +** is assumed to contain the same columns as:
         3156  +**
         3157  +**   CREATE TABLE rbu_count(tbl TEXT PRIMARY KEY, cnt INTEGER) WITHOUT ROWID;
         3158  +**
         3159  +** There should be one row in the table for each data_xxx table in the
         3160  +** database. The 'tbl' column should contain the name of a data_xxx table,
         3161  +** and the cnt column the number of rows it contains.
         3162  +**
         3163  +** sqlite3rbu.nPhaseOneStep is initialized to the sum of (1 + nIndex) * cnt
         3164  +** for all rows in the rbu_count table, where nIndex is the number of 
         3165  +** indexes on the corresponding target database table.
         3166  +*/
         3167  +static void rbuInitPhaseOneSteps(sqlite3rbu *p){
         3168  +  if( p->rc==SQLITE_OK ){
         3169  +    sqlite3_stmt *pStmt = 0;
         3170  +    int bExists = 0;                /* True if rbu_count exists */
         3171  +
         3172  +    p->nPhaseOneStep = -1;
         3173  +
         3174  +    p->rc = sqlite3_create_function(p->dbRbu, 
         3175  +        "rbu_index_cnt", 1, SQLITE_UTF8, (void*)p, rbuIndexCntFunc, 0, 0
         3176  +    );
         3177  +  
         3178  +    /* Check for the rbu_count table. If it does not exist, or if an error
         3179  +    ** occurs, nPhaseOneStep will be left set to -1. */
         3180  +    if( p->rc==SQLITE_OK ){
         3181  +      p->rc = prepareAndCollectError(p->dbRbu, &pStmt, &p->zErrmsg,
         3182  +          "SELECT 1 FROM sqlite_master WHERE tbl_name = 'rbu_count'"
         3183  +      );
         3184  +    }
         3185  +    if( p->rc==SQLITE_OK ){
         3186  +      if( SQLITE_ROW==sqlite3_step(pStmt) ){
         3187  +        bExists = 1;
         3188  +      }
         3189  +      p->rc = sqlite3_finalize(pStmt);
         3190  +    }
         3191  +  
         3192  +    if( p->rc==SQLITE_OK && bExists ){
         3193  +      p->rc = prepareAndCollectError(p->dbRbu, &pStmt, &p->zErrmsg,
         3194  +          "SELECT sum(cnt * (1 + rbu_index_cnt(rbu_target_name(tbl))))"
         3195  +          "FROM rbu_count"
         3196  +      );
         3197  +      if( p->rc==SQLITE_OK ){
         3198  +        if( SQLITE_ROW==sqlite3_step(pStmt) ){
         3199  +          p->nPhaseOneStep = sqlite3_column_int64(pStmt, 0);
         3200  +        }
         3201  +        p->rc = sqlite3_finalize(pStmt);
         3202  +      }
         3203  +    }
         3204  +  }
         3205  +}
  3034   3206   
  3035   3207   /*
  3036   3208   ** Open and return a new RBU handle. 
  3037   3209   */
  3038   3210   sqlite3rbu *sqlite3rbu_open(
  3039   3211     const char *zTarget, 
  3040   3212     const char *zRbu,
................................................................................
  3073   3245       if( p->rc==SQLITE_OK ){
  3074   3246         pState = rbuLoadState(p);
  3075   3247         assert( pState || p->rc!=SQLITE_OK );
  3076   3248         if( p->rc==SQLITE_OK ){
  3077   3249   
  3078   3250           if( pState->eStage==0 ){ 
  3079   3251             rbuDeleteOalFile(p);
         3252  +          rbuInitPhaseOneSteps(p);
  3080   3253             p->eStage = RBU_STAGE_OAL;
  3081   3254           }else{
  3082   3255             p->eStage = pState->eStage;
         3256  +          p->nPhaseOneStep = pState->nPhaseOneStep;
  3083   3257           }
  3084   3258           p->nProgress = pState->nProgress;
  3085   3259           p->iOalSz = pState->iOalSz;
  3086   3260         }
  3087   3261       }
  3088   3262       assert( p->rc!=SQLITE_OK || p->eStage!=0 );
  3089   3263   
................................................................................
  3238   3412   ** Return the total number of key-value operations (inserts, deletes or 
  3239   3413   ** updates) that have been performed on the target database since the
  3240   3414   ** current RBU update was started.
  3241   3415   */
  3242   3416   sqlite3_int64 sqlite3rbu_progress(sqlite3rbu *pRbu){
  3243   3417     return pRbu->nProgress;
  3244   3418   }
         3419  +
         3420  +/*
         3421  +** Return permyriadage progress indications for the two main stages of
         3422  +** an RBU update.
         3423  +*/
         3424  +void sqlite3rbu_bp_progress(sqlite3rbu *p, int *pnOne, int *pnTwo){
         3425  +  const int MAX_PROGRESS = 10000;
         3426  +  switch( p->eStage ){
         3427  +    case RBU_STAGE_OAL:
         3428  +      if( p->nPhaseOneStep>0 ){
         3429  +        *pnOne = (int)(MAX_PROGRESS * (i64)p->nProgress/(i64)p->nPhaseOneStep);
         3430  +      }else{
         3431  +        *pnOne = -1;
         3432  +      }
         3433  +      *pnTwo = 0;
         3434  +      break;
         3435  +
         3436  +    case RBU_STAGE_MOVE:
         3437  +      *pnOne = MAX_PROGRESS;
         3438  +      *pnTwo = 0;
         3439  +      break;
         3440  +
         3441  +    case RBU_STAGE_CKPT:
         3442  +      *pnOne = MAX_PROGRESS;
         3443  +      *pnTwo = (int)(MAX_PROGRESS * (i64)p->nStep / (i64)p->nFrame);
         3444  +      break;
         3445  +
         3446  +    case RBU_STAGE_DONE:
         3447  +      *pnOne = MAX_PROGRESS;
         3448  +      *pnTwo = MAX_PROGRESS;
         3449  +      break;
         3450  +
         3451  +    default:
         3452  +      assert( 0 );
         3453  +  }
         3454  +}
  3245   3455   
  3246   3456   int sqlite3rbu_savestate(sqlite3rbu *p){
  3247   3457     int rc = p->rc;
  3248   3458     
  3249   3459     if( rc==SQLITE_DONE ) return SQLITE_OK;
  3250   3460   
  3251   3461     assert( p->eStage>=RBU_STAGE_OAL && p->eStage<=RBU_STAGE_DONE );

Changes to ext/rbu/sqlite3rbu.h.

   396    396   /*
   397    397   ** Return the total number of key-value operations (inserts, deletes or 
   398    398   ** updates) that have been performed on the target database since the
   399    399   ** current RBU update was started.
   400    400   */
   401    401   sqlite3_int64 sqlite3rbu_progress(sqlite3rbu *pRbu);
   402    402   
          403  +/*
          404  +** Obtain permyriadage (permyriadage is to 10000 as percentage is to 100) 
          405  +** progress indications for the two stages of an RBU update. This API may
          406  +** be useful for driving GUI progress indicators and similar.
          407  +**
          408  +** An RBU update is divided into two stages:
          409  +**
          410  +**   * Stage 1, in which changes are accumulated in an oal/wal file, and
          411  +**   * Stage 2, in which the contents of the wal file are copied into the
          412  +**     main database.
          413  +**
          414  +** The update is visible to non-RBU clients during stage 2. During stage 1
          415  +** non-RBU reader clients may see the original database.
          416  +**
          417  +** If this API is called during stage 2 of the update, output variable 
          418  +** (*pnOne) is set to 10000 to indicate that stage 1 has finished and (*pnTwo)
          419  +** to a value between 0 and 10000 to indicate the permyriadage progress of
          420  +** stage 2. A value of 5000 indicates that stage 2 is half finished, 
          421  +** 9000 indicates that it is 90% finished, and so on.
          422  +**
          423  +** If this API is called during stage 1 of the update, output variable 
          424  +** (*pnTwo) is set to 0 to indicate that stage 2 has not yet started. The
          425  +** value to which (*pnOne) is set depends on whether or not the RBU 
          426  +** database contains an "rbu_count" table. The rbu_count table, if it 
          427  +** exists, must contain the same columns as the following:
          428  +**
          429  +**   CREATE TABLE rbu_count(tbl TEXT PRIMARY KEY, cnt INTEGER) WITHOUT ROWID;
          430  +**
          431  +** There must be one row in the table for each source (data_xxx) table within
          432  +** the RBU database. The 'tbl' column should contain the name of the source
          433  +** table. The 'cnt' column should contain the number of rows within the
          434  +** source table.
          435  +**
          436  +** If the rbu_count table is present and populated correctly and this
          437  +** API is called during stage 1, the *pnOne output variable is set to the
          438  +** permyriadage progress of the same stage. If the rbu_count table does
          439  +** not exist, then (*pnOne) is set to -1 during stage 1. If the rbu_count
          440  +** table exists but is not correctly populated, the value of the *pnOne
          441  +** output variable during stage 1 is undefined.
          442  +*/
          443  +void sqlite3rbu_bp_progress(sqlite3rbu *pRbu, int *pnOne, int *pnTwo);
          444  +
   403    445   /*
   404    446   ** Create an RBU VFS named zName that accesses the underlying file-system
   405    447   ** via existing VFS zParent. Or, if the zParent parameter is passed NULL, 
   406    448   ** then the new RBU VFS uses the default system VFS to access the file-system.
   407    449   ** The new object is registered as a non-default VFS with SQLite before 
   408    450   ** returning.
   409    451   **

Changes to ext/rbu/test_rbu.c.

    62     62       const char *zUsage;
    63     63     } aCmd[] = {
    64     64       {"step", 2, ""},              /* 0 */
    65     65       {"close", 2, ""},             /* 1 */
    66     66       {"create_rbu_delta", 2, ""},  /* 2 */
    67     67       {"savestate", 2, ""},         /* 3 */
    68     68       {"dbMain_eval", 3, "SQL"},    /* 4 */
           69  +    {"bp_progress", 2, ""},    /* 5 */
    69     70       {0,0,0}
    70     71     };
    71     72     int iCmd;
    72     73   
    73     74     if( objc<2 ){
    74     75       Tcl_WrongNumArgs(interp, 1, objv, "METHOD");
    75     76       return TCL_ERROR;
................................................................................
   131    132         int rc = sqlite3_exec(db, Tcl_GetString(objv[2]), 0, 0, 0);
   132    133         if( rc!=SQLITE_OK ){
   133    134           Tcl_SetObjResult(interp, Tcl_NewStringObj(sqlite3_errmsg(db), -1));
   134    135           ret = TCL_ERROR;
   135    136         }
   136    137         break;
   137    138       }
          139  +
          140  +    case 5: /* bp_progress */ {
          141  +      int one, two;
          142  +      Tcl_Obj *pObj;
          143  +      sqlite3rbu_bp_progress(pRbu, &one, &two);
          144  +
          145  +      pObj = Tcl_NewObj();
          146  +      Tcl_ListObjAppendElement(interp, pObj, Tcl_NewIntObj(one));
          147  +      Tcl_ListObjAppendElement(interp, pObj, Tcl_NewIntObj(two));
          148  +      Tcl_SetObjResult(interp, pObj);
          149  +      break;
          150  +    }
   138    151   
   139    152       default: /* seems unlikely */
   140    153         assert( !"cannot happen" );
   141    154         break;
   142    155     }
   143    156   
   144    157     return ret;

Changes to src/expr.c.

  1279   1279   */
  1280   1280   u32 sqlite3ExprListFlags(const ExprList *pList){
  1281   1281     int i;
  1282   1282     u32 m = 0;
  1283   1283     if( pList ){
  1284   1284       for(i=0; i<pList->nExpr; i++){
  1285   1285          Expr *pExpr = pList->a[i].pExpr;
  1286         -       if( ALWAYS(pExpr) ) m |= pExpr->flags;
         1286  +       assert( pExpr!=0 );
         1287  +       m |= pExpr->flags;
  1287   1288       }
  1288   1289     }
  1289   1290     return m;
  1290   1291   }
  1291   1292   
  1292   1293   /*
  1293   1294   ** These routines are Walker callbacks used to check expressions to
................................................................................
  3544   3545         break;
  3545   3546       }
  3546   3547       case TK_NOT: {
  3547   3548         testcase( jumpIfNull==0 );
  3548   3549         sqlite3ExprIfFalse(pParse, pExpr->pLeft, dest, jumpIfNull);
  3549   3550         break;
  3550   3551       }
         3552  +    case TK_IS:
         3553  +    case TK_ISNOT:
         3554  +      testcase( op==TK_IS );
         3555  +      testcase( op==TK_ISNOT );
         3556  +      op = (op==TK_IS) ? TK_EQ : TK_NE;
         3557  +      jumpIfNull = SQLITE_NULLEQ;
         3558  +      /* Fall thru */
  3551   3559       case TK_LT:
  3552   3560       case TK_LE:
  3553   3561       case TK_GT:
  3554   3562       case TK_GE:
  3555   3563       case TK_NE:
  3556   3564       case TK_EQ: {
  3557   3565         testcase( jumpIfNull==0 );
................................................................................
  3559   3567         r2 = sqlite3ExprCodeTemp(pParse, pExpr->pRight, &regFree2);
  3560   3568         codeCompare(pParse, pExpr->pLeft, pExpr->pRight, op,
  3561   3569                     r1, r2, dest, jumpIfNull);
  3562   3570         assert(TK_LT==OP_Lt); testcase(op==OP_Lt); VdbeCoverageIf(v,op==OP_Lt);
  3563   3571         assert(TK_LE==OP_Le); testcase(op==OP_Le); VdbeCoverageIf(v,op==OP_Le);
  3564   3572         assert(TK_GT==OP_Gt); testcase(op==OP_Gt); VdbeCoverageIf(v,op==OP_Gt);
  3565   3573         assert(TK_GE==OP_Ge); testcase(op==OP_Ge); VdbeCoverageIf(v,op==OP_Ge);
  3566         -      assert(TK_EQ==OP_Eq); testcase(op==OP_Eq); VdbeCoverageIf(v,op==OP_Eq);
  3567         -      assert(TK_NE==OP_Ne); testcase(op==OP_Ne); VdbeCoverageIf(v,op==OP_Ne);
  3568         -      testcase( regFree1==0 );
  3569         -      testcase( regFree2==0 );
  3570         -      break;
  3571         -    }
  3572         -    case TK_IS:
  3573         -    case TK_ISNOT: {
  3574         -      testcase( op==TK_IS );
  3575         -      testcase( op==TK_ISNOT );
  3576         -      r1 = sqlite3ExprCodeTemp(pParse, pExpr->pLeft, &regFree1);
  3577         -      r2 = sqlite3ExprCodeTemp(pParse, pExpr->pRight, &regFree2);
  3578         -      op = (op==TK_IS) ? TK_EQ : TK_NE;
  3579         -      codeCompare(pParse, pExpr->pLeft, pExpr->pRight, op,
  3580         -                  r1, r2, dest, SQLITE_NULLEQ);
  3581         -      VdbeCoverageIf(v, op==TK_EQ);
  3582         -      VdbeCoverageIf(v, op==TK_NE);
         3574  +      assert(TK_EQ==OP_Eq); testcase(op==OP_Eq);
         3575  +      VdbeCoverageIf(v, op==OP_Eq && jumpIfNull==SQLITE_NULLEQ);
         3576  +      VdbeCoverageIf(v, op==OP_Eq && jumpIfNull!=SQLITE_NULLEQ);
         3577  +      assert(TK_NE==OP_Ne); testcase(op==OP_Ne);
         3578  +      VdbeCoverageIf(v, op==OP_Ne && jumpIfNull==SQLITE_NULLEQ);
         3579  +      VdbeCoverageIf(v, op==OP_Ne && jumpIfNull!=SQLITE_NULLEQ);
  3583   3580         testcase( regFree1==0 );
  3584   3581         testcase( regFree2==0 );
  3585   3582         break;
  3586   3583       }
  3587   3584       case TK_ISNULL:
  3588   3585       case TK_NOTNULL: {
  3589   3586         assert( TK_ISNULL==OP_IsNull );   testcase( op==TK_ISNULL );
................................................................................
  3700   3697         break;
  3701   3698       }
  3702   3699       case TK_NOT: {
  3703   3700         testcase( jumpIfNull==0 );
  3704   3701         sqlite3ExprIfTrue(pParse, pExpr->pLeft, dest, jumpIfNull);
  3705   3702         break;
  3706   3703       }
         3704  +    case TK_IS:
         3705  +    case TK_ISNOT:
         3706  +      testcase( pExpr->op==TK_IS );
         3707  +      testcase( pExpr->op==TK_ISNOT );
         3708  +      op = (pExpr->op==TK_IS) ? TK_NE : TK_EQ;
         3709  +      jumpIfNull = SQLITE_NULLEQ;
         3710  +      /* Fall thru */
  3707   3711       case TK_LT:
  3708   3712       case TK_LE:
  3709   3713       case TK_GT:
  3710   3714       case TK_GE:
  3711   3715       case TK_NE:
  3712   3716       case TK_EQ: {
  3713   3717         testcase( jumpIfNull==0 );
................................................................................
  3715   3719         r2 = sqlite3ExprCodeTemp(pParse, pExpr->pRight, &regFree2);
  3716   3720         codeCompare(pParse, pExpr->pLeft, pExpr->pRight, op,
  3717   3721                     r1, r2, dest, jumpIfNull);
  3718   3722         assert(TK_LT==OP_Lt); testcase(op==OP_Lt); VdbeCoverageIf(v,op==OP_Lt);
  3719   3723         assert(TK_LE==OP_Le); testcase(op==OP_Le); VdbeCoverageIf(v,op==OP_Le);
  3720   3724         assert(TK_GT==OP_Gt); testcase(op==OP_Gt); VdbeCoverageIf(v,op==OP_Gt);
  3721   3725         assert(TK_GE==OP_Ge); testcase(op==OP_Ge); VdbeCoverageIf(v,op==OP_Ge);
  3722         -      assert(TK_EQ==OP_Eq); testcase(op==OP_Eq); VdbeCoverageIf(v,op==OP_Eq);
  3723         -      assert(TK_NE==OP_Ne); testcase(op==OP_Ne); VdbeCoverageIf(v,op==OP_Ne);
  3724         -      testcase( regFree1==0 );
  3725         -      testcase( regFree2==0 );
  3726         -      break;
  3727         -    }
  3728         -    case TK_IS:
  3729         -    case TK_ISNOT: {
  3730         -      testcase( pExpr->op==TK_IS );
  3731         -      testcase( pExpr->op==TK_ISNOT );
  3732         -      r1 = sqlite3ExprCodeTemp(pParse, pExpr->pLeft, &regFree1);
  3733         -      r2 = sqlite3ExprCodeTemp(pParse, pExpr->pRight, &regFree2);
  3734         -      op = (pExpr->op==TK_IS) ? TK_NE : TK_EQ;
  3735         -      codeCompare(pParse, pExpr->pLeft, pExpr->pRight, op,
  3736         -                  r1, r2, dest, SQLITE_NULLEQ);
  3737         -      VdbeCoverageIf(v, op==TK_EQ);
  3738         -      VdbeCoverageIf(v, op==TK_NE);
         3726  +      assert(TK_EQ==OP_Eq); testcase(op==OP_Eq);
         3727  +      VdbeCoverageIf(v, op==OP_Eq && jumpIfNull!=SQLITE_NULLEQ);
         3728  +      VdbeCoverageIf(v, op==OP_Eq && jumpIfNull==SQLITE_NULLEQ);
         3729  +      assert(TK_NE==OP_Ne); testcase(op==OP_Ne);
         3730  +      VdbeCoverageIf(v, op==OP_Ne && jumpIfNull!=SQLITE_NULLEQ);
         3731  +      VdbeCoverageIf(v, op==OP_Ne && jumpIfNull==SQLITE_NULLEQ);
  3739   3732         testcase( regFree1==0 );
  3740   3733         testcase( regFree2==0 );
  3741   3734         break;
  3742   3735       }
  3743   3736       case TK_ISNULL:
  3744   3737       case TK_NOTNULL: {
  3745   3738         r1 = sqlite3ExprCodeTemp(pParse, pExpr->pLeft, &regFree1);
................................................................................
  4254   4247   /*
  4255   4248   ** Mark all temporary registers as being unavailable for reuse.
  4256   4249   */
  4257   4250   void sqlite3ClearTempRegCache(Parse *pParse){
  4258   4251     pParse->nTempReg = 0;
  4259   4252     pParse->nRangeReg = 0;
  4260   4253   }
         4254  +
         4255  +/*
         4256  +** Validate that no temporary register falls within the range of
         4257  +** iFirst..iLast, inclusive.  This routine is only call from within assert()
         4258  +** statements.
         4259  +*/
         4260  +#ifdef SQLITE_DEBUG
         4261  +int sqlite3NoTempsInRange(Parse *pParse, int iFirst, int iLast){
         4262  +  int i;
         4263  +  if( pParse->nRangeReg>0
         4264  +   && pParse->iRangeReg+pParse->nRangeReg<iLast
         4265  +   && pParse->iRangeReg>=iFirst
         4266  +  ){
         4267  +     return 0;
         4268  +  }
         4269  +  for(i=0; i<pParse->nTempReg; i++){
         4270  +    if( pParse->aTempReg[i]>=iFirst && pParse->aTempReg[i]<=iLast ){
         4271  +      return 0;
         4272  +    }
         4273  +  }
         4274  +  return 1;
         4275  +}
         4276  +#endif /* SQLITE_DEBUG */

Changes to src/pcache1.c.

   344    344     return p;
   345    345   }
   346    346   
   347    347   /*
   348    348   ** Free an allocated buffer obtained from pcache1Alloc().
   349    349   */
   350    350   static void pcache1Free(void *p){
   351         -  int nFreed = 0;
   352    351     if( p==0 ) return;
   353    352     if( SQLITE_WITHIN(p, pcache1.pStart, pcache1.pEnd) ){
   354    353       PgFreeslot *pSlot;
   355    354       sqlite3_mutex_enter(pcache1.mutex);
   356    355       sqlite3StatusDown(SQLITE_STATUS_PAGECACHE_USED, 1);
   357    356       pSlot = (PgFreeslot*)p;
   358    357       pSlot->pNext = pcache1.pFree;
................................................................................
   361    360       pcache1.bUnderPressure = pcache1.nFreeSlot<pcache1.nReserve;
   362    361       assert( pcache1.nFreeSlot<=pcache1.nSlot );
   363    362       sqlite3_mutex_leave(pcache1.mutex);
   364    363     }else{
   365    364       assert( sqlite3MemdebugHasType(p, MEMTYPE_PCACHE) );
   366    365       sqlite3MemdebugSetType(p, MEMTYPE_HEAP);
   367    366   #ifndef SQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS
   368         -    nFreed = sqlite3MallocSize(p);
   369         -    sqlite3_mutex_enter(pcache1.mutex);
   370         -    sqlite3StatusDown(SQLITE_STATUS_PAGECACHE_OVERFLOW, nFreed);
   371         -    sqlite3_mutex_leave(pcache1.mutex);
          367  +    {
          368  +      int nFreed = 0;
          369  +      nFreed = sqlite3MallocSize(p);
          370  +      sqlite3_mutex_enter(pcache1.mutex);
          371  +      sqlite3StatusDown(SQLITE_STATUS_PAGECACHE_OVERFLOW, nFreed);
          372  +      sqlite3_mutex_leave(pcache1.mutex);
          373  +    }
   372    374   #endif
   373    375       sqlite3_free(p);
   374    376     }
   375    377   }
   376    378   
   377    379   #ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT
   378    380   /*

Changes to src/pragma.c.

  1437   1437   
  1438   1438       /* Do an integrity check on each database file */
  1439   1439       for(i=0; i<db->nDb; i++){
  1440   1440         HashElem *x;
  1441   1441         Hash *pTbls;
  1442   1442         int *aRoot;
  1443   1443         int cnt = 0;
         1444  +      int mxIdx = 0;
         1445  +      int nIdx;
  1444   1446   
  1445   1447         if( OMIT_TEMPDB && i==1 ) continue;
  1446   1448         if( iDb>=0 && i!=iDb ) continue;
  1447   1449   
  1448   1450         sqlite3CodeVerifySchema(pParse, i);
  1449   1451         addr = sqlite3VdbeAddOp1(v, OP_IfPos, 1); /* Halt if out of errors */
  1450   1452         VdbeCoverage(v);
................................................................................
  1458   1460         */
  1459   1461         assert( sqlite3SchemaMutexHeld(db, i, 0) );
  1460   1462         pTbls = &db->aDb[i].pSchema->tblHash;
  1461   1463         for(cnt=0, x=sqliteHashFirst(pTbls); x; x=sqliteHashNext(x)){
  1462   1464           Table *pTab = sqliteHashData(x);
  1463   1465           Index *pIdx;
  1464   1466           if( HasRowid(pTab) ) cnt++;
  1465         -        for(pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext){ cnt++; }
         1467  +        for(nIdx=0, pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext, nIdx++){ cnt++; }
         1468  +        if( nIdx>mxIdx ) mxIdx = nIdx;
  1466   1469         }
  1467   1470         aRoot = sqlite3DbMallocRawNN(db, sizeof(int)*(cnt+1));
  1468   1471         if( aRoot==0 ) break;
  1469   1472         for(cnt=0, x=sqliteHashFirst(pTbls); x; x=sqliteHashNext(x)){
  1470   1473           Table *pTab = sqliteHashData(x);
  1471   1474           Index *pIdx;
  1472   1475           if( HasRowid(pTab) ) aRoot[cnt++] = pTab->tnum;
................................................................................
  1473   1476           for(pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext){
  1474   1477             aRoot[cnt++] = pIdx->tnum;
  1475   1478           }
  1476   1479         }
  1477   1480         aRoot[cnt] = 0;
  1478   1481   
  1479   1482         /* Make sure sufficient number of registers have been allocated */
  1480         -      pParse->nMem = MAX( pParse->nMem, 14 );
         1483  +      pParse->nMem = MAX( pParse->nMem, 8+mxIdx );
  1481   1484   
  1482   1485         /* Do the b-tree integrity checks */
  1483   1486         sqlite3VdbeAddOp4(v, OP_IntegrityCk, 2, cnt, 1, (char*)aRoot,P4_INTARRAY);
  1484   1487         sqlite3VdbeChangeP5(v, (u8)i);
  1485   1488         addr = sqlite3VdbeAddOp1(v, OP_IsNull, 2); VdbeCoverage(v);
  1486   1489         sqlite3VdbeAddOp4(v, OP_String8, 0, 3, 0,
  1487   1490            sqlite3MPrintf(db, "*** in database %s ***\n", db->aDb[i].zName),
................................................................................
  1510   1513           sqlite3ExprCacheClear(pParse);
  1511   1514           sqlite3OpenTableAndIndices(pParse, pTab, OP_OpenRead, 0,
  1512   1515                                      1, 0, &iDataCur, &iIdxCur);
  1513   1516           sqlite3VdbeAddOp2(v, OP_Integer, 0, 7);
  1514   1517           for(j=0, pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext, j++){
  1515   1518             sqlite3VdbeAddOp2(v, OP_Integer, 0, 8+j); /* index entries counter */
  1516   1519           }
  1517         -        pParse->nMem = MAX(pParse->nMem, 8+j);
         1520  +        assert( pParse->nMem>=8+j );
         1521  +        assert( sqlite3NoTempsInRange(pParse,1,7+j) );
  1518   1522           sqlite3VdbeAddOp2(v, OP_Rewind, iDataCur, 0); VdbeCoverage(v);
  1519   1523           loopTop = sqlite3VdbeAddOp2(v, OP_AddImm, 7, 1);
  1520   1524           /* Verify that all NOT NULL columns really are NOT NULL */
  1521   1525           for(j=0; j<pTab->nCol; j++){
  1522   1526             char *zErr;
  1523   1527             int jmp2, jmp3;
  1524   1528             if( j==pTab->iPKey ) continue;

Changes to src/sqliteInt.h.

  3416   3416   int sqlite3RunParser(Parse*, const char*, char **);
  3417   3417   void sqlite3FinishCoding(Parse*);
  3418   3418   int sqlite3GetTempReg(Parse*);
  3419   3419   void sqlite3ReleaseTempReg(Parse*,int);
  3420   3420   int sqlite3GetTempRange(Parse*,int);
  3421   3421   void sqlite3ReleaseTempRange(Parse*,int,int);
  3422   3422   void sqlite3ClearTempRegCache(Parse*);
         3423  +#ifdef SQLITE_DEBUG
         3424  +int sqlite3NoTempsInRange(Parse*,int,int);
         3425  +#endif
  3423   3426   Expr *sqlite3ExprAlloc(sqlite3*,int,const Token*,int);
  3424   3427   Expr *sqlite3Expr(sqlite3*,int,const char*);
  3425   3428   void sqlite3ExprAttachSubtrees(sqlite3*,Expr*,Expr*,Expr*);
  3426   3429   Expr *sqlite3PExpr(Parse*, int, Expr*, Expr*, const Token*);
  3427   3430   Expr *sqlite3ExprAnd(sqlite3*,Expr*, Expr*);
  3428   3431   Expr *sqlite3ExprFunction(Parse*,ExprList*, Token*);
  3429   3432   void sqlite3ExprAssignVarNumber(Parse*, Expr*);

Changes to src/test1.c.

  1927   1927     /* Call the underlying C function. If an error occurs, set rc to 
  1928   1928     ** TCL_ERROR and load any error string into the interpreter. If no 
  1929   1929     ** error occurs, set rc to TCL_OK.
  1930   1930     */
  1931   1931   #ifdef SQLITE_OMIT_LOAD_EXTENSION
  1932   1932     rc = SQLITE_ERROR;
  1933   1933     zErr = sqlite3_mprintf("this build omits sqlite3_load_extension()");
         1934  +  (void)zProc;
         1935  +  (void)zFile;
  1934   1936   #else
  1935   1937     rc = sqlite3_load_extension(db, zFile, zProc, &zErr);
  1936   1938   #endif
  1937   1939     if( rc!=SQLITE_OK ){
  1938   1940       Tcl_SetResult(interp, zErr ? zErr : "", TCL_VOLATILE);
  1939   1941       rc = TCL_ERROR;
  1940   1942     }else{

Changes to src/test_config.c.

    74     74   #endif
    75     75   
    76     76   #ifdef SQLITE_DEBUG
    77     77     Tcl_SetVar2(interp, "sqlite_options", "debug", "1", TCL_GLOBAL_ONLY);
    78     78   #else
    79     79     Tcl_SetVar2(interp, "sqlite_options", "debug", "0", TCL_GLOBAL_ONLY);
    80     80   #endif
           81  +
           82  +#ifdef SQLITE_DEFAULT_CKPTFULLFSYNC
           83  +  Tcl_SetVar2(interp, "sqlite_options", "default_ckptfullfsync", 
           84  +              SQLITE_DEFAULT_CKPTFULLFSYNC ? "1" : "0", TCL_GLOBAL_ONLY);
           85  +#else
           86  +  Tcl_SetVar2(interp, "sqlite_options", "default_ckptfullfsync", "0", TCL_GLOBAL_ONLY);
           87  +#endif
    81     88   
    82     89   #ifdef SQLITE_DIRECT_OVERFLOW_READ
    83     90     Tcl_SetVar2(interp, "sqlite_options", "direct_read", "1", TCL_GLOBAL_ONLY);
    84     91   #else
    85     92     Tcl_SetVar2(interp, "sqlite_options", "direct_read", "0", TCL_GLOBAL_ONLY);
    86     93   #endif
    87     94   
................................................................................
    92     99   #endif
    93    100   
    94    101   #ifdef SQLITE_DISABLE_LFS
    95    102     Tcl_SetVar2(interp, "sqlite_options", "lfs", "0", TCL_GLOBAL_ONLY);
    96    103   #else
    97    104     Tcl_SetVar2(interp, "sqlite_options", "lfs", "1", TCL_GLOBAL_ONLY);
    98    105   #endif
          106  +
          107  +#ifdef SQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS
          108  +  Tcl_SetVar2(interp, "sqlite_options", "pagecache_overflow_stats","0",TCL_GLOBAL_ONLY);
          109  +#else
          110  +  Tcl_SetVar2(interp, "sqlite_options", "pagecache_overflow_stats","1",TCL_GLOBAL_ONLY);
          111  +#endif
    99    112   
   100    113   #if SQLITE_MAX_MMAP_SIZE>0
   101    114     Tcl_SetVar2(interp, "sqlite_options", "mmap", "1", TCL_GLOBAL_ONLY);
   102    115   #else
   103    116     Tcl_SetVar2(interp, "sqlite_options", "mmap", "0", TCL_GLOBAL_ONLY);
   104    117   #endif
   105    118   
................................................................................
   572    585   #ifdef SQLITE_OMIT_TCL_VARIABLE
   573    586     Tcl_SetVar2(interp, "sqlite_options", "tclvar", "0", TCL_GLOBAL_ONLY);
   574    587   #else
   575    588     Tcl_SetVar2(interp, "sqlite_options", "tclvar", "1", TCL_GLOBAL_ONLY);
   576    589   #endif
   577    590   
   578    591     Tcl_SetVar2(interp, "sqlite_options", "threadsafe", 
   579         -      STRINGVALUE(SQLITE_THREADSAFE), TCL_GLOBAL_ONLY);
          592  +      SQLITE_THREADSAFE ? "1" : "0", TCL_GLOBAL_ONLY);
          593  +  Tcl_SetVar2(interp, "sqlite_options", "threadsafe1", 
          594  +      SQLITE_THREADSAFE==1 ? "1" : "0", TCL_GLOBAL_ONLY);
          595  +  Tcl_SetVar2(interp, "sqlite_options", "threadsafe2", 
          596  +      SQLITE_THREADSAFE==2 ? "1" : "0", TCL_GLOBAL_ONLY);
   580    597     assert( sqlite3_threadsafe()==SQLITE_THREADSAFE );
   581    598   
   582    599   #ifdef SQLITE_OMIT_TEMPDB
   583    600     Tcl_SetVar2(interp, "sqlite_options", "tempdb", "0", TCL_GLOBAL_ONLY);
   584    601   #else
   585    602     Tcl_SetVar2(interp, "sqlite_options", "tempdb", "1", TCL_GLOBAL_ONLY);
   586    603   #endif

Changes to src/vdbe.c.

   188    188     **     different sized allocations. Memory cells provide growable
   189    189     **     allocations.
   190    190     **
   191    191     **   * When using ENABLE_MEMORY_MANAGEMENT, memory cell buffers can
   192    192     **     be freed lazily via the sqlite3_release_memory() API. This
   193    193     **     minimizes the number of malloc calls made by the system.
   194    194     **
   195         -  ** Memory cells for cursors are allocated at the top of the address
   196         -  ** space. Memory cell (p->nMem) corresponds to cursor 0. Space for
   197         -  ** cursor 1 is managed by memory cell (p->nMem-1), etc.
          195  +  ** The memory cell for cursor 0 is aMem[0]. The rest are allocated from
          196  +  ** the top of the register space.  Cursor 1 is at Mem[p->nMem-1].
          197  +  ** Cursor 2 is at Mem[p->nMem-2]. And so forth.
   198    198     */
   199         -  Mem *pMem = &p->aMem[p->nMem-iCur];
          199  +  Mem *pMem = iCur>0 ? &p->aMem[p->nMem-iCur] : p->aMem;
   200    200   
   201    201     int nByte;
   202    202     VdbeCursor *pCx = 0;
   203    203     nByte = 
   204    204         ROUND8(sizeof(VdbeCursor)) + 2*sizeof(u32)*nField + 
   205    205         (eCurType==CURTYPE_BTREE?sqlite3BtreeCursorSize():0);
   206    206   
   207         -  assert( iCur<p->nCursor );
          207  +  assert( iCur>=0 && iCur<p->nCursor );
   208    208     if( p->apCsr[iCur] ){
   209    209       sqlite3VdbeFreeCursor(p, p->apCsr[iCur]);
   210    210       p->apCsr[iCur] = 0;
   211    211     }
   212    212     if( SQLITE_OK==sqlite3VdbeMemClearAndResize(pMem, nByte) ){
   213    213       p->apCsr[iCur] = pCx = (VdbeCursor*)pMem->z;
   214    214       memset(pCx, 0, sizeof(VdbeCursor));
................................................................................
   525    525     sqlite3VdbeMemSetNull(pOut);
   526    526     pOut->flags = MEM_Int;
   527    527     return pOut;
   528    528   }
   529    529   static Mem *out2Prerelease(Vdbe *p, VdbeOp *pOp){
   530    530     Mem *pOut;
   531    531     assert( pOp->p2>0 );
   532         -  assert( pOp->p2<=(p->nMem-p->nCursor) );
          532  +  assert( pOp->p2<=(p->nMem+1 - p->nCursor) );
   533    533     pOut = &p->aMem[pOp->p2];
   534    534     memAboutToChange(p, pOut);
   535    535     if( VdbeMemDynamic(pOut) ){
   536    536       return out2PrereleaseWithClear(pOut);
   537    537     }else{
   538    538       pOut->flags = MEM_Int;
   539    539       return pOut;
................................................................................
   663    663   #endif
   664    664   
   665    665       /* Sanity checking on other operands */
   666    666   #ifdef SQLITE_DEBUG
   667    667       assert( pOp->opflags==sqlite3OpcodeProperty[pOp->opcode] );
   668    668       if( (pOp->opflags & OPFLG_IN1)!=0 ){
   669    669         assert( pOp->p1>0 );
   670         -      assert( pOp->p1<=(p->nMem-p->nCursor) );
          670  +      assert( pOp->p1<=(p->nMem+1 - p->nCursor) );
   671    671         assert( memIsValid(&aMem[pOp->p1]) );
   672    672         assert( sqlite3VdbeCheckMemInvariants(&aMem[pOp->p1]) );
   673    673         REGISTER_TRACE(pOp->p1, &aMem[pOp->p1]);
   674    674       }
   675    675       if( (pOp->opflags & OPFLG_IN2)!=0 ){
   676    676         assert( pOp->p2>0 );
   677         -      assert( pOp->p2<=(p->nMem-p->nCursor) );
          677  +      assert( pOp->p2<=(p->nMem+1 - p->nCursor) );
   678    678         assert( memIsValid(&aMem[pOp->p2]) );
   679    679         assert( sqlite3VdbeCheckMemInvariants(&aMem[pOp->p2]) );
   680    680         REGISTER_TRACE(pOp->p2, &aMem[pOp->p2]);
   681    681       }
   682    682       if( (pOp->opflags & OPFLG_IN3)!=0 ){
   683    683         assert( pOp->p3>0 );
   684         -      assert( pOp->p3<=(p->nMem-p->nCursor) );
          684  +      assert( pOp->p3<=(p->nMem+1 - p->nCursor) );
   685    685         assert( memIsValid(&aMem[pOp->p3]) );
   686    686         assert( sqlite3VdbeCheckMemInvariants(&aMem[pOp->p3]) );
   687    687         REGISTER_TRACE(pOp->p3, &aMem[pOp->p3]);
   688    688       }
   689    689       if( (pOp->opflags & OPFLG_OUT2)!=0 ){
   690    690         assert( pOp->p2>0 );
   691         -      assert( pOp->p2<=(p->nMem-p->nCursor) );
          691  +      assert( pOp->p2<=(p->nMem+1 - p->nCursor) );
   692    692         memAboutToChange(p, &aMem[pOp->p2]);
   693    693       }
   694    694       if( (pOp->opflags & OPFLG_OUT3)!=0 ){
   695    695         assert( pOp->p3>0 );
   696         -      assert( pOp->p3<=(p->nMem-p->nCursor) );
          696  +      assert( pOp->p3<=(p->nMem+1 - p->nCursor) );
   697    697         memAboutToChange(p, &aMem[pOp->p3]);
   698    698       }
   699    699   #endif
   700    700   #if defined(SQLITE_DEBUG) || defined(VDBE_PROFILE)
   701    701       pOrigOp = pOp;
   702    702   #endif
   703    703     
................................................................................
   788    788   
   789    789   /* Opcode:  Gosub P1 P2 * * *
   790    790   **
   791    791   ** Write the current address onto register P1
   792    792   ** and then jump to address P2.
   793    793   */
   794    794   case OP_Gosub: {            /* jump */
   795         -  assert( pOp->p1>0 && pOp->p1<=(p->nMem-p->nCursor) );
          795  +  assert( pOp->p1>0 && pOp->p1<=(p->nMem+1 - p->nCursor) );
   796    796     pIn1 = &aMem[pOp->p1];
   797    797     assert( VdbeMemDynamic(pIn1)==0 );
   798    798     memAboutToChange(p, pIn1);
   799    799     pIn1->flags = MEM_Int;
   800    800     pIn1->u.i = (int)(pOp-aOp);
   801    801     REGISTER_TRACE(pOp->p1, pIn1);
   802    802   
................................................................................
   828    828   ** If P2!=0 then the coroutine implementation immediately follows
   829    829   ** this opcode.  So jump over the coroutine implementation to
   830    830   ** address P2.
   831    831   **
   832    832   ** See also: EndCoroutine
   833    833   */
   834    834   case OP_InitCoroutine: {     /* jump */
   835         -  assert( pOp->p1>0 &&  pOp->p1<=(p->nMem-p->nCursor) );
          835  +  assert( pOp->p1>0 &&  pOp->p1<=(p->nMem+1 - p->nCursor) );
   836    836     assert( pOp->p2>=0 && pOp->p2<p->nOp );
   837    837     assert( pOp->p3>=0 && pOp->p3<p->nOp );
   838    838     pOut = &aMem[pOp->p1];
   839    839     assert( !VdbeMemDynamic(pOut) );
   840    840     pOut->u.i = pOp->p3 - 1;
   841    841     pOut->flags = MEM_Int;
   842    842     if( pOp->p2 ) goto jump_to_p2;
................................................................................
  1097   1097     pOut->z = pOp->p4.z;
  1098   1098     pOut->n = pOp->p1;
  1099   1099     pOut->enc = encoding;
  1100   1100     UPDATE_MAX_BLOBSIZE(pOut);
  1101   1101   #ifndef SQLITE_LIKE_DOESNT_MATCH_BLOBS
  1102   1102     if( pOp->p5 ){
  1103   1103       assert( pOp->p3>0 );
  1104         -    assert( pOp->p3<=(p->nMem-p->nCursor) );
         1104  +    assert( pOp->p3<=(p->nMem+1 - p->nCursor) );
  1105   1105       pIn3 = &aMem[pOp->p3];
  1106   1106       assert( pIn3->flags & MEM_Int );
  1107   1107       if( pIn3->u.i ) pOut->flags = MEM_Blob|MEM_Static|MEM_Term;
  1108   1108     }
  1109   1109   #endif
  1110   1110     break;
  1111   1111   }
................................................................................
  1123   1123   ** OP_Ne or OP_Eq.
  1124   1124   */
  1125   1125   case OP_Null: {           /* out2 */
  1126   1126     int cnt;
  1127   1127     u16 nullFlag;
  1128   1128     pOut = out2Prerelease(p, pOp);
  1129   1129     cnt = pOp->p3-pOp->p2;
  1130         -  assert( pOp->p3<=(p->nMem-p->nCursor) );
         1130  +  assert( pOp->p3<=(p->nMem+1 - p->nCursor) );
  1131   1131     pOut->flags = nullFlag = pOp->p1 ? (MEM_Null|MEM_Cleared) : MEM_Null;
  1132   1132     while( cnt>0 ){
  1133   1133       pOut++;
  1134   1134       memAboutToChange(p, pOut);
  1135   1135       sqlite3VdbeMemSetNull(pOut);
  1136   1136       pOut->flags = nullFlag;
  1137   1137       cnt--;
................................................................................
  1144   1144   **
  1145   1145   ** Set register P1 to have the value NULL as seen by the OP_MakeRecord
  1146   1146   ** instruction, but do not free any string or blob memory associated with
  1147   1147   ** the register, so that if the value was a string or blob that was
  1148   1148   ** previously copied using OP_SCopy, the copies will continue to be valid.
  1149   1149   */
  1150   1150   case OP_SoftNull: {
  1151         -  assert( pOp->p1>0 && pOp->p1<=(p->nMem-p->nCursor) );
         1151  +  assert( pOp->p1>0 && pOp->p1<=(p->nMem+1 - p->nCursor) );
  1152   1152     pOut = &aMem[pOp->p1];
  1153   1153     pOut->flags = (pOut->flags|MEM_Null)&~MEM_Undefined;
  1154   1154     break;
  1155   1155   }
  1156   1156   
  1157   1157   /* Opcode: Blob P1 P2 * P4 *
  1158   1158   ** Synopsis: r[P2]=P4 (len=P1)
................................................................................
  1211   1211     p2 = pOp->p2;
  1212   1212     assert( n>0 && p1>0 && p2>0 );
  1213   1213     assert( p1+n<=p2 || p2+n<=p1 );
  1214   1214   
  1215   1215     pIn1 = &aMem[p1];
  1216   1216     pOut = &aMem[p2];
  1217   1217     do{
  1218         -    assert( pOut<=&aMem[(p->nMem-p->nCursor)] );
  1219         -    assert( pIn1<=&aMem[(p->nMem-p->nCursor)] );
         1218  +    assert( pOut<=&aMem[(p->nMem+1 - p->nCursor)] );
         1219  +    assert( pIn1<=&aMem[(p->nMem+1 - p->nCursor)] );
  1220   1220       assert( memIsValid(pIn1) );
  1221   1221       memAboutToChange(p, pOut);
  1222   1222       sqlite3VdbeMemMove(pOut, pIn1);
  1223   1223   #ifdef SQLITE_DEBUG
  1224   1224       if( pOut->pScopyFrom>=&aMem[p1] && pOut->pScopyFrom<pOut ){
  1225   1225         pOut->pScopyFrom += pOp->p2 - p1;
  1226   1226       }
................................................................................
  1312   1312   ** the result row.
  1313   1313   */
  1314   1314   case OP_ResultRow: {
  1315   1315     Mem *pMem;
  1316   1316     int i;
  1317   1317     assert( p->nResColumn==pOp->p2 );
  1318   1318     assert( pOp->p1>0 );
  1319         -  assert( pOp->p1+pOp->p2<=(p->nMem-p->nCursor)+1 );
         1319  +  assert( pOp->p1+pOp->p2<=(p->nMem+1 - p->nCursor)+1 );
  1320   1320   
  1321   1321   #ifndef SQLITE_OMIT_PROGRESS_CALLBACK
  1322   1322     /* Run the progress counter just before returning.
  1323   1323     */
  1324   1324     if( db->xProgress!=0
  1325   1325      && nVmStep>=nProgressLimit
  1326   1326      && db->xProgress(db->pProgressArg)!=0
................................................................................
  1624   1624   */
  1625   1625   case OP_Function0: {
  1626   1626     int n;
  1627   1627     sqlite3_context *pCtx;
  1628   1628   
  1629   1629     assert( pOp->p4type==P4_FUNCDEF );
  1630   1630     n = pOp->p5;
  1631         -  assert( pOp->p3>0 && pOp->p3<=(p->nMem-p->nCursor) );
  1632         -  assert( n==0 || (pOp->p2>0 && pOp->p2+n<=(p->nMem-p->nCursor)+1) );
         1631  +  assert( pOp->p3>0 && pOp->p3<=(p->nMem+1 - p->nCursor) );
         1632  +  assert( n==0 || (pOp->p2>0 && pOp->p2+n<=(p->nMem+1 - p->nCursor)+1) );
  1633   1633     assert( pOp->p3<pOp->p2 || pOp->p3>=pOp->p2+n );
  1634   1634     pCtx = sqlite3DbMallocRawNN(db, sizeof(*pCtx) + (n-1)*sizeof(sqlite3_value*));
  1635   1635     if( pCtx==0 ) goto no_mem;
  1636   1636     pCtx->pOut = 0;
  1637   1637     pCtx->pFunc = pOp->p4.pFunc;
  1638   1638     pCtx->iOp = (int)(pOp - aOp);
  1639   1639     pCtx->pVdbe = p;
................................................................................
  2125   2125     assert( pKeyInfo!=0 );
  2126   2126     p1 = pOp->p1;
  2127   2127     p2 = pOp->p2;
  2128   2128   #if SQLITE_DEBUG
  2129   2129     if( aPermute ){
  2130   2130       int k, mx = 0;
  2131   2131       for(k=0; k<n; k++) if( aPermute[k]>mx ) mx = aPermute[k];
  2132         -    assert( p1>0 && p1+mx<=(p->nMem-p->nCursor)+1 );
  2133         -    assert( p2>0 && p2+mx<=(p->nMem-p->nCursor)+1 );
         2132  +    assert( p1>0 && p1+mx<=(p->nMem+1 - p->nCursor)+1 );
         2133  +    assert( p2>0 && p2+mx<=(p->nMem+1 - p->nCursor)+1 );
  2134   2134     }else{
  2135         -    assert( p1>0 && p1+n<=(p->nMem-p->nCursor)+1 );
  2136         -    assert( p2>0 && p2+n<=(p->nMem-p->nCursor)+1 );
         2135  +    assert( p1>0 && p1+n<=(p->nMem+1 - p->nCursor)+1 );
         2136  +    assert( p2>0 && p2+n<=(p->nMem+1 - p->nCursor)+1 );
  2137   2137     }
  2138   2138   #endif /* SQLITE_DEBUG */
  2139   2139     for(i=0; i<n; i++){
  2140   2140       idx = aPermute ? aPermute[i] : i;
  2141   2141       assert( memIsValid(&aMem[p1+idx]) );
  2142   2142       assert( memIsValid(&aMem[p2+idx]) );
  2143   2143       REGISTER_TRACE(p1+idx, &aMem[p1+idx]);
................................................................................
  2391   2391   
  2392   2392     pC = p->apCsr[pOp->p1];
  2393   2393     p2 = pOp->p2;
  2394   2394   
  2395   2395     /* If the cursor cache is stale, bring it up-to-date */
  2396   2396     rc = sqlite3VdbeCursorMoveto(&pC, &p2);
  2397   2397   
  2398         -  assert( pOp->p3>0 && pOp->p3<=(p->nMem-p->nCursor) );
         2398  +  assert( pOp->p3>0 && pOp->p3<=(p->nMem+1 - p->nCursor) );
  2399   2399     pDest = &aMem[pOp->p3];
  2400   2400     memAboutToChange(p, pDest);
  2401   2401     assert( pOp->p1>=0 && pOp->p1<p->nCursor );
  2402   2402     assert( pC!=0 );
  2403   2403     assert( p2<pC->nField );
  2404   2404     aOffset = pC->aOffset;
  2405   2405     assert( pC->eCurType!=CURTYPE_VTAB );
................................................................................
  2634   2634     char cAff;               /* A single character of affinity */
  2635   2635   
  2636   2636     zAffinity = pOp->p4.z;
  2637   2637     assert( zAffinity!=0 );
  2638   2638     assert( zAffinity[pOp->p2]==0 );
  2639   2639     pIn1 = &aMem[pOp->p1];
  2640   2640     while( (cAff = *(zAffinity++))!=0 ){
  2641         -    assert( pIn1 <= &p->aMem[(p->nMem-p->nCursor)] );
         2641  +    assert( pIn1 <= &p->aMem[(p->nMem+1 - p->nCursor)] );
  2642   2642       assert( memIsValid(pIn1) );
  2643   2643       applyAffinity(pIn1, cAff, encoding);
  2644   2644       pIn1++;
  2645   2645     }
  2646   2646     break;
  2647   2647   }
  2648   2648   
................................................................................
  2696   2696     ** of the record to data0.
  2697   2697     */
  2698   2698     nData = 0;         /* Number of bytes of data space */
  2699   2699     nHdr = 0;          /* Number of bytes of header space */
  2700   2700     nZero = 0;         /* Number of zero bytes at the end of the record */
  2701   2701     nField = pOp->p1;
  2702   2702     zAffinity = pOp->p4.z;
  2703         -  assert( nField>0 && pOp->p2>0 && pOp->p2+nField<=(p->nMem-p->nCursor)+1 );
         2703  +  assert( nField>0 && pOp->p2>0 && pOp->p2+nField<=(p->nMem+1 - p->nCursor)+1 );
  2704   2704     pData0 = &aMem[nField];
  2705   2705     nField = pOp->p2;
  2706   2706     pLast = &pData0[nField-1];
  2707   2707     file_format = p->minWriteFileFormat;
  2708   2708   
  2709   2709     /* Identify the output register */
  2710   2710     assert( pOp->p3<pOp->p1 || pOp->p3>=pOp->p1+pOp->p2 );
................................................................................
  2786   2786       /* EVIDENCE-OF: R-64536-51728 The values for each column in the record
  2787   2787       ** immediately follow the header. */
  2788   2788       j += sqlite3VdbeSerialPut(&zNewRecord[j], pRec, serial_type); /* content */
  2789   2789     }while( (++pRec)<=pLast );
  2790   2790     assert( i==nHdr );
  2791   2791     assert( j==nByte );
  2792   2792   
  2793         -  assert( pOp->p3>0 && pOp->p3<=(p->nMem-p->nCursor) );
         2793  +  assert( pOp->p3>0 && pOp->p3<=(p->nMem+1 - p->nCursor) );
  2794   2794     pOut->n = (int)nByte;
  2795   2795     pOut->flags = MEM_Blob;
  2796   2796     if( nZero ){
  2797   2797       pOut->u.nZero = nZero;
  2798   2798       pOut->flags |= MEM_Zero;
  2799   2799     }
  2800   2800     pOut->enc = SQLITE_UTF8;  /* In case the blob is ever converted to text */
................................................................................
  3372   3372         p->minWriteFileFormat = pDb->pSchema->file_format;
  3373   3373       }
  3374   3374     }else{
  3375   3375       wrFlag = 0;
  3376   3376     }
  3377   3377     if( pOp->p5 & OPFLAG_P2ISREG ){
  3378   3378       assert( p2>0 );
  3379         -    assert( p2<=(p->nMem-p->nCursor) );
         3379  +    assert( p2<=(p->nMem+1 - p->nCursor) );
  3380   3380       pIn2 = &aMem[p2];
  3381   3381       assert( memIsValid(pIn2) );
  3382   3382       assert( (pIn2->flags & MEM_Int)!=0 );
  3383   3383       sqlite3VdbeMemIntegerify(pIn2);
  3384   3384       p2 = (int)pIn2->u.i;
  3385   3385       /* The p2 value always comes from a prior OP_CreateTable opcode and
  3386   3386       ** that opcode will always set the p2 value to 2 or more or else fail.
................................................................................
  4167   4167         if( p->pFrame ){
  4168   4168           for(pFrame=p->pFrame; pFrame->pParent; pFrame=pFrame->pParent);
  4169   4169           /* Assert that P3 is a valid memory cell. */
  4170   4170           assert( pOp->p3<=pFrame->nMem );
  4171   4171           pMem = &pFrame->aMem[pOp->p3];
  4172   4172         }else{
  4173   4173           /* Assert that P3 is a valid memory cell. */
  4174         -        assert( pOp->p3<=(p->nMem-p->nCursor) );
         4174  +        assert( pOp->p3<=(p->nMem+1 - p->nCursor) );
  4175   4175           pMem = &aMem[pOp->p3];
  4176   4176           memAboutToChange(p, pMem);
  4177   4177         }
  4178   4178         assert( memIsValid(pMem) );
  4179   4179   
  4180   4180         REGISTER_TRACE(pOp->p3, pMem);
  4181   4181         sqlite3VdbeMemIntegerify(pMem);
................................................................................
  4943   4943   case OP_IdxDelete: {
  4944   4944     VdbeCursor *pC;
  4945   4945     BtCursor *pCrsr;
  4946   4946     int res;
  4947   4947     UnpackedRecord r;
  4948   4948   
  4949   4949     assert( pOp->p3>0 );
  4950         -  assert( pOp->p2>0 && pOp->p2+pOp->p3<=(p->nMem-p->nCursor)+1 );
         4950  +  assert( pOp->p2>0 && pOp->p2+pOp->p3<=(p->nMem+1 - p->nCursor)+1 );
  4951   4951     assert( pOp->p1>=0 && pOp->p1<p->nCursor );
  4952   4952     pC = p->apCsr[pOp->p1];
  4953   4953     assert( pC!=0 );
  4954   4954     assert( pC->eCurType==CURTYPE_BTREE );
  4955   4955     pCrsr = pC->uc.pCursor;
  4956   4956     assert( pCrsr!=0 );
  4957   4957     assert( pOp->p5==0 );
................................................................................
  5449   5449     Mem *pnErr;     /* Register keeping track of errors remaining */
  5450   5450   
  5451   5451     assert( p->bIsReader );
  5452   5452     nRoot = pOp->p2;
  5453   5453     aRoot = pOp->p4.ai;
  5454   5454     assert( nRoot>0 );
  5455   5455     assert( aRoot[nRoot]==0 );
  5456         -  assert( pOp->p3>0 && pOp->p3<=(p->nMem-p->nCursor) );
         5456  +  assert( pOp->p3>0 && pOp->p3<=(p->nMem+1 - p->nCursor) );
  5457   5457     pnErr = &aMem[pOp->p3];
  5458   5458     assert( (pnErr->flags & MEM_Int)!=0 );
  5459   5459     assert( (pnErr->flags & (MEM_Str|MEM_Blob))==0 );
  5460   5460     pIn1 = &aMem[pOp->p1];
  5461   5461     assert( pOp->p5<db->nDb );
  5462   5462     assert( DbMaskTest(p->btreeMask, pOp->p5) );
  5463   5463     z = sqlite3BtreeIntegrityCheck(db->aDb[pOp->p5].pBt, aRoot, nRoot,
................................................................................
  5639   5639     if( (pRt->flags&MEM_Frame)==0 ){
  5640   5640       /* SubProgram.nMem is set to the number of memory cells used by the 
  5641   5641       ** program stored in SubProgram.aOp. As well as these, one memory
  5642   5642       ** cell is required for each cursor used by the program. Set local
  5643   5643       ** variable nMem (and later, VdbeFrame.nChildMem) to this value.
  5644   5644       */
  5645   5645       nMem = pProgram->nMem + pProgram->nCsr;
         5646  +    assert( nMem>0 );
         5647  +    if( pProgram->nCsr==0 ) nMem++;
  5646   5648       nByte = ROUND8(sizeof(VdbeFrame))
  5647   5649                 + nMem * sizeof(Mem)
  5648   5650                 + pProgram->nCsr * sizeof(VdbeCursor *)
  5649   5651                 + pProgram->nOnce * sizeof(u8);
  5650   5652       pFrame = sqlite3DbMallocZero(db, nByte);
  5651   5653       if( !pFrame ){
  5652   5654         goto no_mem;
................................................................................
  5675   5677       pEnd = &VdbeFrameMem(pFrame)[pFrame->nChildMem];
  5676   5678       for(pMem=VdbeFrameMem(pFrame); pMem!=pEnd; pMem++){
  5677   5679         pMem->flags = MEM_Undefined;
  5678   5680         pMem->db = db;
  5679   5681       }
  5680   5682     }else{
  5681   5683       pFrame = pRt->u.pFrame;
  5682         -    assert( pProgram->nMem+pProgram->nCsr==pFrame->nChildMem );
         5684  +    assert( pProgram->nMem+pProgram->nCsr==pFrame->nChildMem 
         5685  +        || (pProgram->nCsr==0 && pProgram->nMem+1==pFrame->nChildMem) );
  5683   5686       assert( pProgram->nCsr==pFrame->nChildCsr );
  5684   5687       assert( (int)(pOp - aOp)==pFrame->pc );
  5685   5688     }
  5686   5689   
  5687   5690     p->nFrame++;
  5688   5691     pFrame->pParent = p->pFrame;
  5689   5692     pFrame->lastRowid = lastRowid;
................................................................................
  5690   5693     pFrame->nChange = p->nChange;
  5691   5694     pFrame->nDbChange = p->db->nChange;
  5692   5695     assert( pFrame->pAuxData==0 );
  5693   5696     pFrame->pAuxData = p->pAuxData;
  5694   5697     p->pAuxData = 0;
  5695   5698     p->nChange = 0;
  5696   5699     p->pFrame = pFrame;
  5697         -  p->aMem = aMem = &VdbeFrameMem(pFrame)[-1];
         5700  +  p->aMem = aMem = VdbeFrameMem(pFrame);
  5698   5701     p->nMem = pFrame->nChildMem;
  5699   5702     p->nCursor = (u16)pFrame->nChildCsr;
  5700         -  p->apCsr = (VdbeCursor **)&aMem[p->nMem+1];
         5703  +  p->apCsr = (VdbeCursor **)&aMem[p->nMem];
  5701   5704     p->aOp = aOp = pProgram->aOp;
  5702   5705     p->nOp = pProgram->nOp;
  5703   5706     p->aOnceFlag = (u8 *)&p->apCsr[p->nCursor];
  5704   5707     p->nOnceFlag = pProgram->nOnce;
  5705   5708   #ifdef SQLITE_ENABLE_STMT_SCANSTATUS
  5706   5709     p->anExec = 0;
  5707   5710   #endif
................................................................................
  5939   5942   */
  5940   5943   case OP_AggStep0: {
  5941   5944     int n;
  5942   5945     sqlite3_context *pCtx;
  5943   5946   
  5944   5947     assert( pOp->p4type==P4_FUNCDEF );
  5945   5948     n = pOp->p5;
  5946         -  assert( pOp->p3>0 && pOp->p3<=(p->nMem-p->nCursor) );
  5947         -  assert( n==0 || (pOp->p2>0 && pOp->p2+n<=(p->nMem-p->nCursor)+1) );
         5949  +  assert( pOp->p3>0 && pOp->p3<=(p->nMem+1 - p->nCursor) );
         5950  +  assert( n==0 || (pOp->p2>0 && pOp->p2+n<=(p->nMem+1 - p->nCursor)+1) );
  5948   5951     assert( pOp->p3<pOp->p2 || pOp->p3>=pOp->p2+n );
  5949   5952     pCtx = sqlite3DbMallocRawNN(db, sizeof(*pCtx) + (n-1)*sizeof(sqlite3_value*));
  5950   5953     if( pCtx==0 ) goto no_mem;
  5951   5954     pCtx->pMem = 0;
  5952   5955     pCtx->pFunc = pOp->p4.pFunc;
  5953   5956     pCtx->iOp = (int)(pOp - aOp);
  5954   5957     pCtx->pVdbe = p;
................................................................................
  6019   6022   ** argument is not used by this opcode.  It is only there to disambiguate
  6020   6023   ** functions that can take varying numbers of arguments.  The
  6021   6024   ** P4 argument is only needed for the degenerate case where
  6022   6025   ** the step function was not previously called.
  6023   6026   */
  6024   6027   case OP_AggFinal: {
  6025   6028     Mem *pMem;
  6026         -  assert( pOp->p1>0 && pOp->p1<=(p->nMem-p->nCursor) );
         6029  +  assert( pOp->p1>0 && pOp->p1<=(p->nMem+1 - p->nCursor) );
  6027   6030     pMem = &aMem[pOp->p1];
  6028   6031     assert( (pMem->flags & ~(MEM_Null|MEM_Agg))==0 );
  6029   6032     rc = sqlite3VdbeMemFinalize(pMem, pOp->p4.pFunc);
  6030   6033     if( rc ){
  6031   6034       sqlite3VdbeError(p, "%s", sqlite3_value_text(pMem));
  6032   6035       goto abort_due_to_error;
  6033   6036     }
................................................................................
  6461   6464     sqlite3_vtab *pVtab;
  6462   6465     const sqlite3_module *pModule;
  6463   6466     Mem *pDest;
  6464   6467     sqlite3_context sContext;
  6465   6468   
  6466   6469     VdbeCursor *pCur = p->apCsr[pOp->p1];
  6467   6470     assert( pCur->eCurType==CURTYPE_VTAB );
  6468         -  assert( pOp->p3>0 && pOp->p3<=(p->nMem-p->nCursor) );
         6471  +  assert( pOp->p3>0 && pOp->p3<=(p->nMem+1 - p->nCursor) );
  6469   6472     pDest = &aMem[pOp->p3];
  6470   6473     memAboutToChange(p, pDest);
  6471   6474     if( pCur->nullRow ){
  6472   6475       sqlite3VdbeMemSetNull(pDest);
  6473   6476       break;
  6474   6477     }
  6475   6478     pVtab = pCur->uc.pVCur->pVtab;

Changes to src/vdbeaux.c.

  1797   1797     */
  1798   1798     assert( p->nOp>0 );
  1799   1799   
  1800   1800     /* Set the magic to VDBE_MAGIC_RUN sooner rather than later. */
  1801   1801     p->magic = VDBE_MAGIC_RUN;
  1802   1802   
  1803   1803   #ifdef SQLITE_DEBUG
  1804         -  for(i=1; i<p->nMem; i++){
         1804  +  for(i=0; i<p->nMem; i++){
  1805   1805       assert( p->aMem[i].db==p->db );
  1806   1806     }
  1807   1807   #endif
  1808   1808     p->pc = -1;
  1809   1809     p->rc = SQLITE_OK;
  1810   1810     p->errorAction = OE_Abort;
  1811   1811     p->nChange = 0;
................................................................................
  1862   1862     nVar = pParse->nVar;
  1863   1863     nMem = pParse->nMem;
  1864   1864     nCursor = pParse->nTab;
  1865   1865     nArg = pParse->nMaxArg;
  1866   1866     nOnce = pParse->nOnce;
  1867   1867     if( nOnce==0 ) nOnce = 1; /* Ensure at least one byte in p->aOnceFlag[] */
  1868   1868     
  1869         -  /* For each cursor required, also allocate a memory cell. Memory
  1870         -  ** cells (nMem+1-nCursor)..nMem, inclusive, will never be used by
  1871         -  ** the vdbe program. Instead they are used to allocate memory for
  1872         -  ** VdbeCursor/BtCursor structures. The blob of memory associated with 
  1873         -  ** cursor 0 is stored in memory cell nMem. Memory cell (nMem-1)
  1874         -  ** stores the blob of memory associated with cursor 1, etc.
  1875         -  **
         1869  +  /* Each cursor uses a memory cell.  The first cursor (cursor 0) can
         1870  +  ** use aMem[0] which is not otherwise used by the VDBE program.  Allocate
         1871  +  ** space at the end of aMem[] for cursors 1 and greater.
  1876   1872     ** See also: allocateCursor().
  1877   1873     */
  1878   1874     nMem += nCursor;
         1875  +  if( nCursor==0 && nMem>0 ) nMem++;  /* Space for aMem[0] even if not used */
  1879   1876   
  1880   1877     /* Figure out how much reusable memory is available at the end of the
  1881   1878     ** opcode array.  This extra memory will be reallocated for other elements
  1882   1879     ** of the prepared statement.
  1883   1880     */
  1884   1881     n = ROUND8(sizeof(Op)*p->nOp);              /* Bytes of opcode memory used */
  1885   1882     x.pSpace = &((u8*)p->aOp)[n];               /* Unused opcode memory */
................................................................................
  1933   1930       }
  1934   1931     }
  1935   1932     p->nzVar = pParse->nzVar;
  1936   1933     p->azVar = pParse->azVar;
  1937   1934     pParse->nzVar =  0;
  1938   1935     pParse->azVar = 0;
  1939   1936     if( p->aMem ){
  1940         -    p->aMem--;                      /* aMem[] goes from 1..nMem */
  1941         -    p->nMem = nMem;                 /*       not from 0..nMem-1 */
  1942         -    for(n=1; n<=nMem; n++){
         1937  +    p->nMem = nMem;
         1938  +    for(n=0; n<nMem; n++){
  1943   1939         p->aMem[n].flags = MEM_Undefined;
  1944   1940         p->aMem[n].db = db;
  1945   1941       }
  1946   1942     }
  1947   1943     p->explain = pParse->explain;
  1948   1944     sqlite3VdbeRewind(p);
  1949   1945   }
................................................................................
  2045   2041       sqlite3VdbeFrameRestore(pFrame);
  2046   2042       p->pFrame = 0;
  2047   2043       p->nFrame = 0;
  2048   2044     }
  2049   2045     assert( p->nFrame==0 );
  2050   2046     closeCursorsInFrame(p);
  2051   2047     if( p->aMem ){
  2052         -    releaseMemArray(&p->aMem[1], p->nMem);
         2048  +    releaseMemArray(p->aMem, p->nMem);
  2053   2049     }
  2054   2050     while( p->pDelFrame ){
  2055   2051       VdbeFrame *pDel = p->pDelFrame;
  2056   2052       p->pDelFrame = pDel->pParent;
  2057   2053       sqlite3VdbeFrameDelete(pDel);
  2058   2054     }
  2059   2055   
................................................................................
  2070   2066   
  2071   2067   #ifdef SQLITE_DEBUG
  2072   2068     /* Execute assert() statements to ensure that the Vdbe.apCsr[] and 
  2073   2069     ** Vdbe.aMem[] arrays have already been cleaned up.  */
  2074   2070     int i;
  2075   2071     if( p->apCsr ) for(i=0; i<p->nCursor; i++) assert( p->apCsr[i]==0 );
  2076   2072     if( p->aMem ){
  2077         -    for(i=1; i<=p->nMem; i++) assert( p->aMem[i].flags==MEM_Undefined );
         2073  +    for(i=0; i<p->nMem; i++) assert( p->aMem[i].flags==MEM_Undefined );
  2078   2074     }
  2079   2075   #endif
  2080   2076   
  2081   2077     sqlite3DbFree(db, p->zErrMsg);
  2082   2078     p->zErrMsg = 0;
  2083   2079     p->pResultSet = 0;
  2084   2080   }

Changes to src/vdbemem.c.

   757    757   **
   758    758   ** This is used for testing and debugging only - to make sure shallow
   759    759   ** copies are not misused.
   760    760   */
   761    761   void sqlite3VdbeMemAboutToChange(Vdbe *pVdbe, Mem *pMem){
   762    762     int i;
   763    763     Mem *pX;
   764         -  for(i=1, pX=&pVdbe->aMem[1]; i<=pVdbe->nMem; i++, pX++){
          764  +  for(i=0, pX=pVdbe->aMem; i<pVdbe->nMem; i++, pX++){
   765    765       if( pX->pScopyFrom==pMem ){
   766    766         pX->flags |= MEM_Undefined;
   767    767         pX->pScopyFrom = 0;
   768    768       }
   769    769     }
   770    770     pMem->pScopyFrom = 0;
   771    771   }

Changes to test/capi3.test.

   922    922   } {0 {}}
   923    923   do_test capi3-11.9.3 {
   924    924     sqlite3_get_autocommit $DB
   925    925   } 1
   926    926   do_test capi3-11.10 {
   927    927     sqlite3_step $STMT
   928    928   } {SQLITE_ROW}
   929         -ifcapable !autoreset {
   930         -  # If SQLITE_OMIT_AUTORESET is defined, then the statement must be
   931         -  # reset() before it can be passed to step() again.
   932         -  do_test capi3-11.11a { sqlite3_step $STMT } {SQLITE_MISUSE}
   933         -  do_test capi3-11.11b { sqlite3_reset $STMT } {SQLITE_ABORT}
   934         -}
   935    929   do_test capi3-11.11 {
   936    930     sqlite3_step $STMT
   937    931   } {SQLITE_DONE}
   938         -do_test capi3-11.12 {
   939         -  sqlite3_step $STMT
   940         -  sqlite3_step $STMT
   941         -} {SQLITE_ROW}
          932  +ifcapable api_armor {
          933  +  do_test capi3-11.12armor {
          934  +    sqlite3_step $STMT
          935  +    sqlite3_step $STMT
          936  +  } {SQLITE_MISUSE}
          937  +} else {
          938  +  do_test capi3-11.12 {
          939  +    sqlite3_step $STMT
          940  +    sqlite3_step $STMT
          941  +  } {SQLITE_ROW}
          942  +}
   942    943   do_test capi3-11.13 {
   943    944     sqlite3_finalize $STMT
   944    945   } {SQLITE_OK}
   945    946   do_test capi3-11.14 {
   946    947     execsql {
   947    948       SELECT a FROM t2;
   948    949     }

Changes to test/capi3c.test.

   861    861   } {0 {}}
   862    862   do_test capi3c-11.9.3 {
   863    863     sqlite3_get_autocommit $DB
   864    864   } 1
   865    865   do_test capi3c-11.10 {
   866    866     sqlite3_step $STMT
   867    867   } {SQLITE_ROW}
   868         -ifcapable !autoreset {
   869         -  # If SQLITE_OMIT_AUTORESET is defined, then the statement must be
   870         -  # reset() before it can be passed to step() again.
   871         -  do_test capi3-11.11a { sqlite3_step $STMT } {SQLITE_MISUSE}
   872         -  do_test capi3-11.11b { sqlite3_reset $STMT } {SQLITE_ABORT}
   873         -}
   874    868   do_test capi3c-11.11 {
   875    869     sqlite3_step $STMT
   876    870   } {SQLITE_DONE}
   877         -do_test capi3c-11.12 {
   878         -  sqlite3_step $STMT
   879         -  sqlite3_step $STMT
   880         -} {SQLITE_ROW}
          871  +ifcapable api_armor {
          872  +  do_test capi3c-11.12armor {
          873  +    sqlite3_step $STMT
          874  +    sqlite3_step $STMT
          875  +  } {SQLITE_MISUSE}
          876  +} else {
          877  +  do_test capi3c-11.12 {
          878  +    sqlite3_step $STMT
          879  +    sqlite3_step $STMT
          880  +  } {SQLITE_ROW}
          881  +}
   881    882   do_test capi3c-11.13 {
   882    883     sqlite3_finalize $STMT
   883    884   } {SQLITE_OK}
   884    885   do_test capi3c-11.14 {
   885    886     execsql {
   886    887       SELECT a FROM t2;
   887    888     }

Changes to test/exclusive.test.

   419    419     }
   420    420   } {}
   421    421   do_test exclusive-5.1 {
   422    422     # Three files are open: The db, journal and statement-journal.
   423    423     # (2016-03-04) The statement-journal is now opened lazily
   424    424     set sqlite_open_file_count
   425    425     expr $sqlite_open_file_count-$extrafds
   426         -} [expr 2 - ($TEMP_STORE>=2)]
          426  +} {2}
   427    427   do_test exclusive-5.2 {
   428    428     execsql {
   429    429       COMMIT;
   430    430     }
   431    431     # One file open: the db.
   432    432     set sqlite_open_file_count
   433    433     expr $sqlite_open_file_count-$extrafds
................................................................................
   446    446     execsql {
   447    447       INSERT INTO abc SELECT a+10, b+10, c+10 FROM abc;
   448    448     }
   449    449     # Three files are open: The db, journal and statement-journal.
   450    450     # 2016-03-04: The statement-journal open is deferred
   451    451     set sqlite_open_file_count
   452    452     expr $sqlite_open_file_count-$extrafds
   453         -} [expr 2 - ($TEMP_STORE>=2)]
          453  +} {2}
   454    454   do_test exclusive-5.5 {
   455    455     execsql {
   456    456       COMMIT;
   457    457     }
   458    458     # Three files are still open: The db, journal and statement-journal.
   459    459     # 2016-03-04: The statement-journal open is deferred
   460    460     set sqlite_open_file_count
   461    461     expr $sqlite_open_file_count-$extrafds
   462         -} [expr 2 - ($TEMP_STORE>=2)]
          462  +} {2}
   463    463   do_test exclusive-5.6 {
   464    464     execsql {
   465    465       PRAGMA locking_mode = normal;
   466    466       SELECT * FROM abc;
   467    467     }
   468    468   } {normal 1 2 3 2 3 4 5 6 7 11 12 13 12 13 14 15 16 17}
   469    469   do_test exclusive-5.7 {

Changes to test/memsubsys1.test.

    96     96   sqlite3_shutdown
    97     97   sqlite3_config_pagecache [expr 1024+$xtra_size] 20
    98     98   sqlite3_initialize
    99     99   reset_highwater_marks
   100    100   build_test_db memsubsys1-2 {PRAGMA page_size=1024; PRAGMA mmap_size=0}
   101    101   #show_memstats
   102    102   set MEMORY_MANAGEMENT $sqlite_options(memorymanage)
   103         -ifcapable !malloc_usable_size {
   104         - do_test memsubsys1-2.3 {
   105         -    set pg_ovfl [lindex [sqlite3_status SQLITE_STATUS_PAGECACHE_OVERFLOW 0] 2]
   106         -  } [expr ($TEMP_STORE>1 || $MEMORY_MANAGEMENT==0)*1024]
          103  +ifcapable pagecache_overflow_stats {
          104  +  ifcapable !malloc_usable_size {
          105  +    do_test memsubsys1-2.3 {
          106  +      set pg_ovfl [lindex [sqlite3_status SQLITE_STATUS_PAGECACHE_OVERFLOW 0] 2]
          107  +    } [expr ($TEMP_STORE>1 || $MEMORY_MANAGEMENT==0)*1024]
          108  +  }
   107    109   }
   108    110   do_test memsubsys1-2.4 {
   109    111     set pg_used [lindex [sqlite3_status SQLITE_STATUS_PAGECACHE_USED 0] 2]
   110    112   } 20
   111    113   do_test memsubsys1-2.5 {
   112    114     set s_used [lindex [sqlite3_status SQLITE_STATUS_SCRATCH_USED 0] 2]
   113    115   } 0

Changes to test/mutex1.test.

    93     93   # Tests mutex1-2.* test the three thread-safety related modes that
    94     94   # can be selected using sqlite3_config:
    95     95   #
    96     96   #   * Serialized mode,
    97     97   #   * Multi-threaded mode,
    98     98   #   * Single-threaded mode.
    99     99   #
   100         -ifcapable threadsafe&&shared_cache {
          100  +ifcapable threadsafe1&&shared_cache {
   101    101     set enable_shared_cache [sqlite3_enable_shared_cache 1]
   102    102     foreach {mode mutexes} {
   103    103       singlethread {}
   104    104       multithread  {
   105    105         fast static_app1 static_app2 static_app3
   106    106         static_lru static_master static_mem static_open
   107    107         static_prng static_pmem static_vfs1 static_vfs2

Changes to test/releasetest.tcl.

   170    170       -DSQLITE_THREADSAFE=2
   171    171       --enable-json1 --enable-fts5
   172    172     }
   173    173     "Locking-Style" {
   174    174       -O2
   175    175       -DSQLITE_ENABLE_LOCKING_STYLE=1
   176    176     }
   177         -  "OS-X" {
          177  +  "Apple" {
   178    178       -O1   # Avoid a compiler bug in gcc 4.2.1 build 5658
   179         -    -DSQLITE_OMIT_LOAD_EXTENSION=1
   180         -    -DSQLITE_DEFAULT_MEMSTATUS=0
   181         -    -DSQLITE_THREADSAFE=2
   182         -    -DSQLITE_OS_UNIX=1
   183         -    -DSQLITE_ENABLE_JSON1=1
   184         -    -DSQLITE_ENABLE_LOCKING_STYLE=1
   185         -    -DUSE_PREAD=1
   186         -    -DSQLITE_ENABLE_RTREE=1
          179  +    -DHAVE_GMTIME_R=1
          180  +    -DHAVE_ISNAN=1
          181  +    -DHAVE_LOCALTIME_R=1
          182  +    -DHAVE_PREAD=1
          183  +    -DHAVE_PWRITE=1
          184  +    -DHAVE_USLEEP=1
          185  +    -DHAVE_USLEEP=1
          186  +    -DHAVE_UTIME=1
          187  +    -DSQLITE_DEFAULT_CACHE_SIZE=1000
          188  +    -DSQLITE_DEFAULT_CKPTFULLFSYNC=1
          189  +    -DSQLITE_DEFAULT_MEMSTATUS=1
          190  +    -DSQLITE_DEFAULT_PAGE_SIZE=1024
          191  +    -DSQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS=1
          192  +    -DSQLITE_ENABLE_API_ARMOR=1
          193  +    -DSQLITE_ENABLE_AUTO_PROFILE=1
          194  +    -DSQLITE_ENABLE_FLOCKTIMEOUT=1
   187    195       -DSQLITE_ENABLE_FTS3=1
   188    196       -DSQLITE_ENABLE_FTS3_PARENTHESIS=1
   189         -    -DSQLITE_DEFAULT_CACHE_SIZE=1000
          197  +    -DSQLITE_ENABLE_FTS3_TOKENIZER=1
          198  +    if:os=="Darwin" -DSQLITE_ENABLE_LOCKING_STYLE=1
          199  +    -DSQLITE_ENABLE_PERSIST_WAL=1
          200  +    -DSQLITE_ENABLE_PURGEABLE_PCACHE=1
          201  +    -DSQLITE_ENABLE_RTREE=1
          202  +    -DSQLITE_ENABLE_SNAPSHOT=1
          203  +    # -DSQLITE_ENABLE_SQLLOG=1
          204  +    -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT=1
   190    205       -DSQLITE_MAX_LENGTH=2147483645
   191    206       -DSQLITE_MAX_VARIABLE_NUMBER=500000
   192         -    -DSQLITE_DEBUG=1
          207  +    # -DSQLITE_MEMDEBUG=1
          208  +    -DSQLITE_NO_SYNC=1
          209  +    -DSQLITE_OMIT_AUTORESET=1
          210  +    -DSQLITE_OMIT_LOAD_EXTENSION=1
   193    211       -DSQLITE_PREFER_PROXY_LOCKING=1
   194         -    -DSQLITE_ENABLE_API_ARMOR=1
          212  +    -DSQLITE_SERIES_CONSTRAINT_VERIFY=1
          213  +    -DSQLITE_THREADSAFE=2
          214  +    -DSQLITE_USE_URI=1
          215  +    -DSQLITE_WRITE_WALFRAME_PREBUFFERED=1
          216  +    -DUSE_GUARDED_FD=1
          217  +    -DUSE_PREAD=1
   195    218       --enable-json1 --enable-fts5
   196    219     }
   197    220     "Extra-Robustness" {
   198    221       -DSQLITE_ENABLE_OVERSIZE_CELL_CHECK=1
   199    222       -DSQLITE_MAX_ATTACHED=62
   200    223     }
   201    224     "Devkit" {
................................................................................
   244    267       "Secure-Delete"           test
   245    268       "Unlock-Notify"           "QUICKTEST_INCLUDE=notify2.test test"
   246    269       "Update-Delete-Limit"     test
   247    270       "Extra-Robustness"        test
   248    271       "Device-Two"              test
   249    272       "No-lookaside"            test
   250    273       "Devkit"                  test
          274  +    "Apple"                   test
   251    275       "Sanitize"                {QUICKTEST_OMIT=func4.test,nan.test test}
   252    276       "Device-One"              fulltest
   253    277       "Default"                 "threadtest fulltest"
   254    278       "Valgrind"                valgrindtest
   255    279     }
   256    280     Linux-i686 {
   257    281       "Devkit"                  test
................................................................................
   260    284       "Device-One"              test
   261    285       "Device-Two"              test
   262    286       "Default"                 "threadtest fulltest"
   263    287     }
   264    288     Darwin-i386 {
   265    289       "Locking-Style"           "mptest test"
   266    290       "Have-Not"                test
   267         -    "OS-X"                    "threadtest fulltest"
          291  +    "Apple"                   "threadtest fulltest"
   268    292     }
   269    293     Darwin-x86_64 {
   270    294       "Locking-Style"           "mptest test"
   271    295       "Have-Not"                test
   272         -    "OS-X"                    "threadtest fulltest"
          296  +    "Apple"                   "threadtest fulltest"
   273    297     }
   274    298     "Windows NT-intel" {
   275    299       "Have-Not"                test
   276    300       "Default"                 "mptest fulltestonly"
   277    301     }
   278    302     "Windows NT-amd64" {
   279    303       "Have-Not"                test
................................................................................
   584    608     # CFLAGS is only passed to gcc.
   585    609     #
   586    610     set makeOpts ""
   587    611     set cflags [expr {$::MSVC ? "-Zi" : "-g"}]
   588    612     set opts ""
   589    613     set title ${name}($testtarget)
   590    614     set configOpts $::WITHTCL
          615  +  set skip 0
   591    616   
   592    617     regsub -all {#[^\n]*\n} $config \n config
   593    618     foreach arg $config {
          619  +    if {$skip} {
          620  +      set skip 0
          621  +      continue
          622  +    }
   594    623       if {[regexp {^-[UD]} $arg]} {
   595    624         lappend opts $arg
   596    625       } elseif {[regexp {^[A-Z]+=} $arg]} {
   597    626         lappend testtarget $arg
          627  +    } elseif {[regexp {^if:([a-z]+)(.*)} $arg all key tail]} {
          628  +      # Arguments of the form 'if:os=="Linux"' will cause the subsequent
          629  +      # argument to be skipped if the $tcl_platform(os) is not "Linux", for
          630  +      # example...
          631  +      set skip [expr !(\$::tcl_platform($key)$tail)]
   598    632       } elseif {[regexp {^--(enable|disable)-} $arg]} {
   599    633         if {$::MSVC} {
   600    634           if {$arg eq "--disable-amalgamation"} {
   601    635             lappend makeOpts USE_AMALGAMATION=0
   602    636             continue
   603    637           }
   604    638           if {$arg eq "--disable-shared"} {

Changes to test/sqldiff1.test.

    10     10   #***********************************************************************
    11     11   #
    12     12   # Quick tests for the sqldiff tool
    13     13   #
    14     14   set testdir [file dirname $argv0]
    15     15   source $testdir/tester.tcl
    16     16   
    17         -if {$tcl_platform(platform)=="windows"} {
    18         -  set PROG "sqldiff.exe"
    19         -} else {
    20         -  set PROG "./sqldiff"
    21         -}
    22         -if {![file exe $PROG]} {
    23         -  puts "sqldiff cannot run because $PROG is not available"
    24         -  finish_test
    25         -  return
    26         -}
           17  +set PROG [test_find_sqldiff]
           18  +
    27     19   db close
    28     20   forcedelete test.db test2.db
    29     21   sqlite3 db test.db
    30     22   
    31     23   do_test sqldiff-1.0 {
    32     24     db eval {
    33     25       CREATE TABLE t1(a INTEGER PRIMARY KEY, b);

Changes to test/tester.tcl.

  2131   2131     sqlite3_shutdown
  2132   2132     eval sqlite3_config_pagecache $::old_pagecache_config
  2133   2133     unset ::old_pagecache_config 
  2134   2134     sqlite3_initialize
  2135   2135     autoinstall_test_functions
  2136   2136     sqlite3 db test.db
  2137   2137   }
         2138  +
         2139  +proc test_find_binary {nm} {
         2140  +  if {$::tcl_platform(platform)=="windows"} {
         2141  +    set ret "$nm.exe"
         2142  +  } else {
         2143  +    set ret $nm
         2144  +  }
         2145  +  set ret [file normalize [file join $::cmdlinearg(TESTFIXTURE_HOME) $ret]]
         2146  +  if {![file executable $ret]} {
         2147  +    finish_test
         2148  +    return ""
         2149  +  }
         2150  +  return $ret
         2151  +}
  2138   2152   
  2139   2153   # Find the name of the 'shell' executable (e.g. "sqlite3.exe") to use for
  2140   2154   # the tests in shell[1-5].test. If no such executable can be found, invoke
  2141   2155   # [finish_test ; return] in the callers context.
  2142   2156   #
  2143   2157   proc test_find_cli {} {
  2144         -  if {$::tcl_platform(platform)=="windows"} {
  2145         -    set ret "sqlite3.exe"
  2146         -  } else {
  2147         -    set ret "sqlite3"
  2148         -  }
  2149         -  set ret [file normalize [file join $::cmdlinearg(TESTFIXTURE_HOME) $ret]]
  2150         -  if {![file executable $ret]} {
  2151         -    finish_test
  2152         -    return -code return
  2153         -  }
  2154         -  return $ret
         2158  +  set prog [test_find_binary sqlite3]
         2159  +  if {$prog==""} { return -code return }
         2160  +  return $prog
         2161  +}
         2162  +
         2163  +# Find the name of the 'sqldiff' executable (e.g. "sqlite3.exe") to use for
         2164  +# the tests in sqldiff tests. If no such executable can be found, invoke
         2165  +# [finish_test ; return] in the callers context.
         2166  +#
         2167  +proc test_find_sqldiff {} {
         2168  +  set prog [test_find_binary sqldiff]
         2169  +  if {$prog==""} { return -code return }
         2170  +  return $prog
  2155   2171   }
         2172  +
  2156   2173   
  2157   2174   # If the library is compiled with the SQLITE_DEFAULT_AUTOVACUUM macro set
  2158   2175   # to non-zero, then set the global variable $AUTOVACUUM to 1.
  2159   2176   set AUTOVACUUM $sqlite_options(default_autovacuum)
  2160   2177   
  2161   2178   # Make sure the FTS enhanced query syntax is disabled.
  2162   2179   set sqlite_fts3_enable_parentheses 0

Changes to test/wal2.test.

  1190   1190   # Test that "PRAGMA checkpoint_fullsync" appears to be working.
  1191   1191   #
  1192   1192   foreach {tn sql reslist} {
  1193   1193     1 { }                                 {10 0 4 0 6 0}
  1194   1194     2 { PRAGMA checkpoint_fullfsync = 1 } {10 4 4 2 6 2}
  1195   1195     3 { PRAGMA checkpoint_fullfsync = 0 } {10 0 4 0 6 0}
  1196   1196   } {
         1197  +  ifcapable default_ckptfullfsync {
         1198  +    if {[string trim $sql]==""} continue
         1199  +  }
  1197   1200     faultsim_delete_and_reopen
  1198   1201   
  1199   1202     execsql {PRAGMA auto_vacuum = 0; PRAGMA synchronous = FULL;}
  1200   1203     execsql $sql
  1201   1204     do_execsql_test wal2-14.$tn.0 { PRAGMA page_size = 4096 }   {}
  1202   1205     do_execsql_test wal2-14.$tn.1 { PRAGMA journal_mode = WAL } {wal}
  1203   1206   

Changes to test/wal3.test.

   216    216     
   217    217       testvfs T
   218    218       T filter {} 
   219    219       T script sync_counter
   220    220       sqlite3 db test.db -vfs T
   221    221     
   222    222       execsql "PRAGMA synchronous = $syncmode"
          223  +    execsql "PRAGMA checkpoint_fullfsync = 0"
   223    224       execsql { PRAGMA journal_mode = WAL }
   224    225       execsql { CREATE TABLE filler(a,b,c); }
   225    226   
   226    227       set ::syncs [list]
   227    228       T filter xSync
   228    229       execsql {
   229    230         CREATE TABLE x(y);

Changes to tool/sqldiff.c.

  1240   1240     char **azCol;                   /* NULL terminated array of col names */
  1241   1241     int i;
  1242   1242     int nCol;
  1243   1243     Str ct = {0, 0, 0};             /* The "CREATE TABLE data_xxx" statement */
  1244   1244     Str sql = {0, 0, 0};            /* Query to find differences */
  1245   1245     Str insert = {0, 0, 0};         /* First part of output INSERT statement */
  1246   1246     sqlite3_stmt *pStmt = 0;
         1247  +  int nRow = 0;                   /* Total rows in data_xxx table */
  1247   1248   
  1248   1249     /* --rbu mode must use real primary keys. */
  1249   1250     g.bSchemaPK = 1;
  1250   1251   
  1251   1252     /* Check that the schemas of the two tables match. Exit early otherwise. */
  1252   1253     checkSchemasMatch(zTab);
  1253   1254   
................................................................................
  1285   1286       if( ct.z ){
  1286   1287         fprintf(out, "%s\n", ct.z);
  1287   1288         strFree(&ct);
  1288   1289       }
  1289   1290   
  1290   1291       /* Output the first part of the INSERT statement */
  1291   1292       fprintf(out, "%s", insert.z);
         1293  +    nRow++;
  1292   1294   
  1293   1295       if( sqlite3_column_type(pStmt, nCol)==SQLITE_INTEGER ){
  1294   1296         for(i=0; i<=nCol; i++){
  1295   1297           if( i>0 ) fprintf(out, ", ");
  1296   1298           printQuoted(out, sqlite3_column_value(pStmt, i));
  1297   1299         }
  1298   1300       }else{
................................................................................
  1338   1340       }
  1339   1341   
  1340   1342       /* And the closing bracket of the insert statement */
  1341   1343       fprintf(out, ");\n");
  1342   1344     }
  1343   1345   
  1344   1346     sqlite3_finalize(pStmt);
         1347  +  if( nRow>0 ){
         1348  +    Str cnt = {0, 0, 0};
         1349  +    strPrintf(&cnt, "INSERT INTO rbu_count VALUES('data_%q', %d);", zTab, nRow);
         1350  +    fprintf(out, "%s\n", cnt.z);
         1351  +    strFree(&cnt);
         1352  +  }
  1345   1353   
  1346   1354     strFree(&ct);
  1347   1355     strFree(&sql);
  1348   1356     strFree(&insert);
  1349   1357   }
  1350   1358   
  1351   1359   /*
................................................................................
  1753   1761     int rc;
  1754   1762     char *zErrMsg = 0;
  1755   1763     char *zSql;
  1756   1764     sqlite3_stmt *pStmt;
  1757   1765     char *zTab = 0;
  1758   1766     FILE *out = stdout;
  1759   1767     void (*xDiff)(const char*,FILE*) = diff_one_table;
         1768  +#ifndef SQLITE_OMIT_LOAD_EXTENSION
  1760   1769     int nExt = 0;
  1761   1770     char **azExt = 0;
         1771  +#endif
  1762   1772     int useTransaction = 0;
  1763   1773     int neverUseTransaction = 0;
  1764   1774   
  1765   1775     g.zArgv0 = argv[0];
  1766   1776     sqlite3_config(SQLITE_CONFIG_SINGLETHREAD);
  1767   1777     for(i=1; i<argc; i++){
  1768   1778       const char *z = argv[i];
................................................................................
  1837   1847     sqlite3_enable_load_extension(g.db, 1);
  1838   1848     for(i=0; i<nExt; i++){
  1839   1849       rc = sqlite3_load_extension(g.db, azExt[i], 0, &zErrMsg);
  1840   1850       if( rc || zErrMsg ){
  1841   1851         cmdlineError("error loading %s: %s", azExt[i], zErrMsg);
  1842   1852       }
  1843   1853     }
  1844         -#endif
  1845   1854     free(azExt);
         1855  +#endif
  1846   1856     zSql = sqlite3_mprintf("ATTACH %Q as aux;", zDb2);
  1847   1857     rc = sqlite3_exec(g.db, zSql, 0, 0, &zErrMsg);
  1848   1858     if( rc || zErrMsg ){
  1849   1859       cmdlineError("cannot attach database \"%s\"", zDb2);
  1850   1860     }
  1851   1861     rc = sqlite3_exec(g.db, "SELECT * FROM aux.sqlite_master", 0, 0, &zErrMsg);
  1852   1862     if( rc || zErrMsg ){
  1853   1863       cmdlineError("\"%s\" does not appear to be a valid SQLite database", zDb2);
  1854   1864     }
  1855   1865   
  1856   1866     if( neverUseTransaction ) useTransaction = 0;
  1857         -  if( useTransaction ) printf("BEGIN TRANSACTION;\n");
         1867  +  if( useTransaction ) fprintf(out, "BEGIN TRANSACTION;\n");
         1868  +  if( xDiff==rbudiff_one_table ){
         1869  +    fprintf(out, "CREATE TABLE IF NOT EXISTS rbu_count"
         1870  +           "(tbl TEXT PRIMARY KEY COLLATE NOCASE, cnt INTEGER) "
         1871  +           "WITHOUT ROWID;\n"
         1872  +    );
         1873  +  }
  1858   1874     if( zTab ){
  1859   1875       xDiff(zTab, out);
  1860   1876     }else{
  1861   1877       /* Handle tables one by one */
  1862   1878       pStmt = db_prepare(
  1863   1879         "SELECT name FROM main.sqlite_master\n"
  1864   1880         " WHERE type='table' AND sql NOT LIKE 'CREATE VIRTUAL%%'\n"