/ Check-in [3296a0ce]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Merge 3.12.0 beta changes from trunk.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | sessions
Files: files | file ages | folders
SHA1: 3296a0ceedef43c2790f0b36471f91138a575243
User & Date: drh 2016-03-21 15:32:19
Context
2016-03-24
14:34
Merge the beta changes into sessions. check-in: beb5ea14 user: drh tags: sessions
2016-03-21
15:32
Merge 3.12.0 beta changes from trunk. check-in: 3296a0ce user: drh tags: sessions
14:46
Add the sqlite3_system_errno() interface. check-in: 4bd12b57 user: drh tags: trunk
2016-03-16
01:16
Merge all recent enhancements from trunk. check-in: 6a7ee04b user: drh tags: sessions
Changes
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Changes to doc/lemon.html.

     1      1   <html>
     2      2   <head>
     3      3   <title>The Lemon Parser Generator</title>
     4      4   </head>
     5      5   <body bgcolor=white>
     6      6   <h1 align=center>The Lemon Parser Generator</h1>
     7      7   
     8         -<p>Lemon is an LALR(1) parser generator for C or C++.  
     9         -It does the same job as ``bison'' and ``yacc''.
    10         -But lemon is not another bison or yacc clone.  It
            8  +<p>Lemon is an LALR(1) parser generator for C.
            9  +It does the same job as "bison" and "yacc".
           10  +But lemon is not a bison or yacc clone.  Lemon
    11     11   uses a different grammar syntax which is designed to
    12         -reduce the number of coding errors.  Lemon also uses a more
    13         -sophisticated parsing engine that is faster than yacc and
    14         -bison and which is both reentrant and thread-safe.
    15         -Furthermore, Lemon implements features that can be used
           12  +reduce the number of coding errors.  Lemon also uses a
           13  +parsing engine that is faster than yacc and
           14  +bison and which is both reentrant and threadsafe.
           15  +(Update: Since the previous sentence was written, bison
           16  +has also been updated so that it too can generate a
           17  +reentrant and threadsafe parser.)
           18  +Lemon also implements features that can be used
    16     19   to eliminate resource leaks, making is suitable for use
    17     20   in long-running programs such as graphical user interfaces
    18     21   or embedded controllers.</p>
    19     22   
    20     23   <p>This document is an introduction to the Lemon
    21     24   parser generator.</p>
    22     25   
................................................................................
    40     43   <ul>
    41     44   <li>C code to implement the parser.
    42     45   <li>A header file defining an integer ID for each terminal symbol.
    43     46   <li>An information file that describes the states of the generated parser
    44     47       automaton.
    45     48   </ul>
    46     49   By default, all three of these output files are generated.
    47         -The header file is suppressed if the ``-m'' command-line option is
    48         -used and the report file is omitted when ``-q'' is selected.</p>
           50  +The header file is suppressed if the "-m" command-line option is
           51  +used and the report file is omitted when "-q" is selected.</p>
    49     52   
    50         -<p>The grammar specification file uses a ``.y'' suffix, by convention.
           53  +<p>The grammar specification file uses a ".y" suffix, by convention.
    51     54   In the examples used in this document, we'll assume the name of the
    52         -grammar file is ``gram.y''.  A typical use of Lemon would be the
           55  +grammar file is "gram.y".  A typical use of Lemon would be the
    53     56   following command:
    54     57   <pre>
    55     58      lemon gram.y
    56     59   </pre>
    57         -This command will generate three output files named ``gram.c'',
    58         -``gram.h'' and ``gram.out''.
           60  +This command will generate three output files named "gram.c",
           61  +"gram.h" and "gram.out".
    59     62   The first is C code to implement the parser.  The second
    60     63   is the header file that defines numerical values for all
    61     64   terminal symbols, and the last is the report that explains
    62     65   the states used by the parser automaton.</p>
    63     66   
    64     67   <h3>Command Line Options</h3>
    65     68   
................................................................................
    67     70   You can obtain a list of the available command-line options together
    68     71   with a brief explanation of what each does by typing
    69     72   <pre>
    70     73      lemon -?
    71     74   </pre>
    72     75   As of this writing, the following command-line options are supported:
    73     76   <ul>
    74         -<li><tt>-b</tt>
    75         -<li><tt>-c</tt>
    76         -<li><tt>-g</tt>
    77         -<li><tt>-m</tt>
    78         -<li><tt>-q</tt>
    79         -<li><tt>-s</tt>
    80         -<li><tt>-x</tt>
    81         -</ul>
    82         -The ``-b'' option reduces the amount of text in the report file by
    83         -printing only the basis of each parser state, rather than the full
    84         -configuration.
    85         -The ``-c'' option suppresses action table compression.  Using -c
    86         -will make the parser a little larger and slower but it will detect
    87         -syntax errors sooner.
    88         -The ``-g'' option causes no output files to be generated at all.
    89         -Instead, the input grammar file is printed on standard output but
    90         -with all comments, actions and other extraneous text deleted.  This
    91         -is a useful way to get a quick summary of a grammar.
    92         -The ``-m'' option causes the output C source file to be compatible
    93         -with the ``makeheaders'' program.
    94         -Makeheaders is a program that automatically generates header files
    95         -from C source code.  When the ``-m'' option is used, the header
    96         -file is not output since the makeheaders program will take care
    97         -of generated all header files automatically.
    98         -The ``-q'' option suppresses the report file.
    99         -Using ``-s'' causes a brief summary of parser statistics to be
   100         -printed.  Like this:
   101         -<pre>
   102         -   Parser statistics: 74 terminals, 70 nonterminals, 179 rules
   103         -                      340 states, 2026 parser table entries, 0 conflicts
   104         -</pre>
   105         -Finally, the ``-x'' option causes Lemon to print its version number
   106         -and then stops without attempting to read the grammar or generate a parser.</p>
           77  +<li><b>-b</b>
           78  +Show only the basis for each parser state in the report file.
           79  +<li><b>-c</b>
           80  +Do not compress the generated action tables.
           81  +<li><b>-D<i>name</i></b>
           82  +Define C preprocessor macro <i>name</i>.  This macro is useable by
           83  +"%ifdef" lines in the grammar file.
           84  +<li><b>-g</b>
           85  +Do not generate a parser.  Instead write the input grammar to standard
           86  +output with all comments, actions, and other extraneous text removed.
           87  +<li><b>-l</b>
           88  +Omit "#line" directives int the generated parser C code.
           89  +<li><b>-m</b>
           90  +Cause the output C source code to be compatible with the "makeheaders"
           91  +program. 
           92  +<li><b>-p</b>
           93  +Display all conflicts that are resolved by 
           94  +<a href='#precrules'>precedence rules</a>.
           95  +<li><b>-q</b>
           96  +Suppress generation of the report file.
           97  +<li><b>-r</b>
           98  +Do not sort or renumber the parser states as part of optimization.
           99  +<li><b>-s</b>
          100  +Show parser statistics before existing.
          101  +<li><b>-T<i>file</i></b>
          102  +Use <i>file</i> as the template for the generated C-code parser implementation.
          103  +<li><b>-x</b>
          104  +Print the Lemon version number.
          105  +</ul>
   107    106   
   108    107   <h3>The Parser Interface</h3>
   109    108   
   110    109   <p>Lemon doesn't generate a complete, working program.  It only generates
   111    110   a few subroutines that implement a parser.  This section describes
   112    111   the interface to those subroutines.  It is up to the programmer to
   113    112   call these subroutines in an appropriate way in order to produce a
................................................................................
   117    116   must first create the parser.
   118    117   A new parser is created as follows:
   119    118   <pre>
   120    119      void *pParser = ParseAlloc( malloc );
   121    120   </pre>
   122    121   The ParseAlloc() routine allocates and initializes a new parser and
   123    122   returns a pointer to it.
   124         -The actual data structure used to represent a parser is opaque --
          123  +The actual data structure used to represent a parser is opaque &mdash;
   125    124   its internal structure is not visible or usable by the calling routine.
   126    125   For this reason, the ParseAlloc() routine returns a pointer to void
   127    126   rather than a pointer to some particular structure.
   128    127   The sole argument to the ParseAlloc() routine is a pointer to the
   129         -subroutine used to allocate memory.  Typically this means ``malloc()''.</p>
          128  +subroutine used to allocate memory.  Typically this means malloc().</p>
   130    129   
   131    130   <p>After a program is finished using a parser, it can reclaim all
   132    131   memory allocated by that parser by calling
   133    132   <pre>
   134    133      ParseFree(pParser, free);
   135    134   </pre>
   136    135   The first argument is the same pointer returned by ParseAlloc().  The
................................................................................
   147    146   The first argument to the Parse() routine is the pointer returned by
   148    147   ParseAlloc().
   149    148   The second argument is a small positive integer that tells the parse the
   150    149   type of the next token in the data stream.
   151    150   There is one token type for each terminal symbol in the grammar.
   152    151   The gram.h file generated by Lemon contains #define statements that
   153    152   map symbolic terminal symbol names into appropriate integer values.
   154         -(A value of 0 for the second argument is a special flag to the
   155         -parser to indicate that the end of input has been reached.)
          153  +A value of 0 for the second argument is a special flag to the
          154  +parser to indicate that the end of input has been reached.
   156    155   The third argument is the value of the given token.  By default,
   157    156   the type of the third argument is integer, but the grammar will
   158    157   usually redefine this type to be some kind of structure.
   159    158   Typically the second argument will be a broad category of tokens
   160         -such as ``identifier'' or ``number'' and the third argument will
          159  +such as "identifier" or "number" and the third argument will
   161    160   be the name of the identifier or the value of the number.</p>
   162    161   
   163    162   <p>The Parse() function may have either three or four arguments,
   164    163   depending on the grammar.  If the grammar specification file requests
   165    164   it (via the <a href='#extraarg'><tt>extra_argument</tt> directive</a>),
   166    165   the Parse() function will have a fourth parameter that can be
   167    166   of any type chosen by the programmer.  The parser doesn't do anything
................................................................................
   189    188      15    ParseFree(pParser, free );
   190    189      16    TokenizerFree(pTokenizer);
   191    190      17    return sState.treeRoot;
   192    191      18 }
   193    192   </pre>
   194    193   This example shows a user-written routine that parses a file of
   195    194   text and returns a pointer to the parse tree.
   196         -(We've omitted all error-handling from this example to keep it
          195  +(All error-handling code is omitted from this example to keep it
   197    196   simple.)
   198    197   We assume the existence of some kind of tokenizer which is created
   199    198   using TokenizerCreate() on line 8 and deleted by TokenizerFree()
   200    199   on line 16.  The GetNextToken() function on line 11 retrieves the
   201    200   next token from the input file and puts its type in the 
   202    201   integer variable hTokenId.  The sToken variable is assumed to be
   203    202   some kind of structure that contains details about each token,
................................................................................
   283    282   declaration can occur at any point in the file.
   284    283   Lemon ignores whitespace (except where it is needed to separate
   285    284   tokens) and it honors the same commenting conventions as C and C++.</p>
   286    285   
   287    286   <h3>Terminals and Nonterminals</h3>
   288    287   
   289    288   <p>A terminal symbol (token) is any string of alphanumeric
   290         -and underscore characters
          289  +and/or underscore characters
   291    290   that begins with an upper case letter.
   292    291   A terminal can contain lowercase letters after the first character,
   293    292   but the usual convention is to make terminals all upper case.
   294    293   A nonterminal, on the other hand, is any string of alphanumeric
   295    294   and underscore characters than begins with a lower case letter.
   296    295   Again, the usual convention is to make nonterminals use all lower
   297    296   case letters.</p>
................................................................................
   310    309   must have alphanumeric names.</p>
   311    310   
   312    311   <h3>Grammar Rules</h3>
   313    312   
   314    313   <p>The main component of a Lemon grammar file is a sequence of grammar
   315    314   rules.
   316    315   Each grammar rule consists of a nonterminal symbol followed by
   317         -the special symbol ``::='' and then a list of terminals and/or nonterminals.
          316  +the special symbol "::=" and then a list of terminals and/or nonterminals.
   318    317   The rule is terminated by a period.
   319    318   The list of terminals and nonterminals on the right-hand side of the
   320    319   rule can be empty.
   321    320   Rules can occur in any order, except that the left-hand side of the
   322    321   first rule is assumed to be the start symbol for the grammar (unless
   323    322   specified otherwise using the <tt>%start</tt> directive described below.)
   324    323   A typical sequence of grammar rules might look something like this:
................................................................................
   326    325     expr ::= expr PLUS expr.
   327    326     expr ::= expr TIMES expr.
   328    327     expr ::= LPAREN expr RPAREN.
   329    328     expr ::= VALUE.
   330    329   </pre>
   331    330   </p>
   332    331   
   333         -<p>There is one non-terminal in this example, ``expr'', and five
   334         -terminal symbols or tokens: ``PLUS'', ``TIMES'', ``LPAREN'',
   335         -``RPAREN'' and ``VALUE''.</p>
          332  +<p>There is one non-terminal in this example, "expr", and five
          333  +terminal symbols or tokens: "PLUS", "TIMES", "LPAREN",
          334  +"RPAREN" and "VALUE".</p>
   336    335   
   337    336   <p>Like yacc and bison, Lemon allows the grammar to specify a block
   338    337   of C code that will be executed whenever a grammar rule is reduced
   339    338   by the parser.
   340    339   In Lemon, this action is specified by putting the C code (contained
   341    340   within curly braces <tt>{...}</tt>) immediately after the
   342    341   period that closes the rule.
................................................................................
   344    343   <pre>
   345    344     expr ::= expr PLUS expr.   { printf("Doing an addition...\n"); }
   346    345   </pre>
   347    346   </p>
   348    347   
   349    348   <p>In order to be useful, grammar actions must normally be linked to
   350    349   their associated grammar rules.
   351         -In yacc and bison, this is accomplished by embedding a ``$$'' in the
          350  +In yacc and bison, this is accomplished by embedding a "$$" in the
   352    351   action to stand for the value of the left-hand side of the rule and
   353         -symbols ``$1'', ``$2'', and so forth to stand for the value of
          352  +symbols "$1", "$2", and so forth to stand for the value of
   354    353   the terminal or nonterminal at position 1, 2 and so forth on the
   355    354   right-hand side of the rule.
   356    355   This idea is very powerful, but it is also very error-prone.  The
   357    356   single most common source of errors in a yacc or bison grammar is
   358    357   to miscount the number of symbols on the right-hand side of a grammar
   359         -rule and say ``$7'' when you really mean ``$8''.</p>
          358  +rule and say "$7" when you really mean "$8".</p>
   360    359   
   361    360   <p>Lemon avoids the need to count grammar symbols by assigning symbolic
   362    361   names to each symbol in a grammar rule and then using those symbolic
   363    362   names in the action.
   364    363   In yacc or bison, one would write this:
   365    364   <pre>
   366    365     expr -> expr PLUS expr  { $$ = $1 + $3; };
................................................................................
   382    381   includes a linking symbol in parentheses but that linking symbol
   383    382   is not actually used in the reduce action, then an error message
   384    383   is generated.
   385    384   For example, the rule
   386    385   <pre>
   387    386     expr(A) ::= expr(B) PLUS expr(C).  { A = B; }
   388    387   </pre>
   389         -will generate an error because the linking symbol ``C'' is used
          388  +will generate an error because the linking symbol "C" is used
   390    389   in the grammar rule but not in the reduce action.</p>
   391    390   
   392    391   <p>The Lemon notation for linking grammar rules to reduce actions
   393    392   also facilitates the use of destructors for reclaiming memory
   394    393   allocated by the values of terminals and nonterminals on the
   395    394   right-hand side of a rule.</p>
   396    395   
          396  +<a name='precrules'></a>
   397    397   <h3>Precedence Rules</h3>
   398    398   
   399    399   <p>Lemon resolves parsing ambiguities in exactly the same way as
   400    400   yacc and bison.  A shift-reduce conflict is resolved in favor
   401    401   of the shift, and a reduce-reduce conflict is resolved by reducing
   402    402   whichever rule comes first in the grammar file.</p>
   403    403   
   404    404   <p>Just like in
   405    405   yacc and bison, Lemon allows a measure of control 
   406    406   over the resolution of paring conflicts using precedence rules.
   407    407   A precedence value can be assigned to any terminal symbol
   408         -using the %left, %right or %nonassoc directives.  Terminal symbols
          408  +using the 
          409  +<a href='#pleft'>%left</a>,
          410  +<a href='#pright'>%right</a> or
          411  +<a href='#pnonassoc'>%nonassoc</a> directives.  Terminal symbols
   409    412   mentioned in earlier directives have a lower precedence that
   410    413   terminal symbols mentioned in later directives.  For example:</p>
   411    414   
   412    415   <p><pre>
   413    416      %left AND.
   414    417      %left OR.
   415    418      %nonassoc EQ NE GT GE LT LE.
................................................................................
   521    524   
   522    525   <p>Lemon supports the following special directives:
   523    526   <ul>
   524    527   <li><tt>%code</tt>
   525    528   <li><tt>%default_destructor</tt>
   526    529   <li><tt>%default_type</tt>
   527    530   <li><tt>%destructor</tt>
          531  +<li><tt>%endif</tt>
   528    532   <li><tt>%extra_argument</tt>
          533  +<li><tt>%fallback</tt>
          534  +<li><tt>%ifdef</tt>
          535  +<li><tt>%ifndef</tt>
   529    536   <li><tt>%include</tt>
   530    537   <li><tt>%left</tt>
   531    538   <li><tt>%name</tt>
   532    539   <li><tt>%nonassoc</tt>
   533    540   <li><tt>%parse_accept</tt>
   534    541   <li><tt>%parse_failure </tt>
   535    542   <li><tt>%right</tt>
   536    543   <li><tt>%stack_overflow</tt>
   537    544   <li><tt>%stack_size</tt>
   538    545   <li><tt>%start_symbol</tt>
   539    546   <li><tt>%syntax_error</tt>
          547  +<li><tt>%token_class</tt>
   540    548   <li><tt>%token_destructor</tt>
   541    549   <li><tt>%token_prefix</tt>
   542    550   <li><tt>%token_type</tt>
   543    551   <li><tt>%type</tt>
          552  +<li><tt>%wildcard</tt>
   544    553   </ul>
   545    554   Each of these directives will be described separately in the
   546    555   following sections:</p>
   547    556   
          557  +<a name='pcode'></a>
   548    558   <h4>The <tt>%code</tt> directive</h4>
   549    559   
   550         -<p>The %code directive is used to specify addition C/C++ code that
          560  +<p>The %code directive is used to specify addition C code that
   551    561   is added to the end of the main output file.  This is similar to
   552         -the %include directive except that %include is inserted at the
   553         -beginning of the main output file.</p>
          562  +the <a href='#pinclude'>%include</a> directive except that %include
          563  +is inserted at the beginning of the main output file.</p>
   554    564   
   555    565   <p>%code is typically used to include some action routines or perhaps
   556         -a tokenizer as part of the output file.</p>
          566  +a tokenizer or even the "main()" function 
          567  +as part of the output file.</p>
   557    568   
          569  +<a name='default_destructor'></a>
   558    570   <h4>The <tt>%default_destructor</tt> directive</h4>
   559    571   
   560    572   <p>The %default_destructor directive specifies a destructor to 
   561    573   use for non-terminals that do not have their own destructor
   562    574   specified by a separate %destructor directive.  See the documentation
   563         -on the %destructor directive below for additional information.</p>
          575  +on the <a name='#destructor'>%destructor</a> directive below for
          576  +additional information.</p>
   564    577   
   565    578   <p>In some grammers, many different non-terminal symbols have the
   566    579   same datatype and hence the same destructor.  This directive is
   567    580   a convenience way to specify the same destructor for all those
   568    581   non-terminals using a single statement.</p>
   569    582   
          583  +<a name='default_type'></a>
   570    584   <h4>The <tt>%default_type</tt> directive</h4>
   571    585   
   572    586   <p>The %default_type directive specifies the datatype of non-terminal
   573    587   symbols that do no have their own datatype defined using a separate
   574         -%type directive.  See the documentation on %type below for addition
   575         -information.</p>
          588  +<a href='#ptype'>%type</a> directive.  
          589  +</p>
   576    590   
          591  +<a name='destructor'></a>
   577    592   <h4>The <tt>%destructor</tt> directive</h4>
   578    593   
   579    594   <p>The %destructor directive is used to specify a destructor for
   580    595   a non-terminal symbol.
   581         -(See also the %token_destructor directive which is used to
   582         -specify a destructor for terminal symbols.)</p>
          596  +(See also the <a href='#token_destructor'>%token_destructor</a>
          597  +directive which is used to specify a destructor for terminal symbols.)</p>
   583    598   
   584    599   <p>A non-terminal's destructor is called to dispose of the
   585    600   non-terminal's value whenever the non-terminal is popped from
   586    601   the stack.  This includes all of the following circumstances:
   587    602   <ul>
   588    603   <li> When a rule reduces and the value of a non-terminal on
   589    604        the right-hand side is not linked to C code.
................................................................................
   598    613   <pre>
   599    614      %type nt {void*}
   600    615      %destructor nt { free($$); }
   601    616      nt(A) ::= ID NUM.   { A = malloc( 100 ); }
   602    617   </pre>
   603    618   This example is a bit contrived but it serves to illustrate how
   604    619   destructors work.  The example shows a non-terminal named
   605         -``nt'' that holds values of type ``void*''.  When the rule for
   606         -an ``nt'' reduces, it sets the value of the non-terminal to
          620  +"nt" that holds values of type "void*".  When the rule for
          621  +an "nt" reduces, it sets the value of the non-terminal to
   607    622   space obtained from malloc().  Later, when the nt non-terminal
   608    623   is popped from the stack, the destructor will fire and call
   609    624   free() on this malloced space, thus avoiding a memory leak.
   610         -(Note that the symbol ``$$'' in the destructor code is replaced
          625  +(Note that the symbol "$$" in the destructor code is replaced
   611    626   by the value of the non-terminal.)</p>
   612    627   
   613    628   <p>It is important to note that the value of a non-terminal is passed
   614    629   to the destructor whenever the non-terminal is removed from the
   615    630   stack, unless the non-terminal is used in a C-code action.  If
   616    631   the non-terminal is used by C-code, then it is assumed that the
   617         -C-code will take care of destroying it if it should really
   618         -be destroyed.  More commonly, the value is used to build some
          632  +C-code will take care of destroying it.
          633  +More commonly, the value is used to build some
   619    634   larger structure and we don't want to destroy it, which is why
   620    635   the destructor is not called in this circumstance.</p>
   621    636   
   622         -<p>By appropriate use of destructors, it is possible to
   623         -build a parser using Lemon that can be used within a long-running
   624         -program, such as a GUI, that will not leak memory or other resources.
          637  +<p>Destructors help avoid memory leaks by automatically freeing
          638  +allocated objects when they go out of scope.
   625    639   To do the same using yacc or bison is much more difficult.</p>
   626    640   
   627    641   <a name="extraarg"></a>
   628    642   <h4>The <tt>%extra_argument</tt> directive</h4>
   629    643   
   630    644   The %extra_argument directive instructs Lemon to add a 4th parameter
   631    645   to the parameter list of the Parse() function it generates.  Lemon
................................................................................
   634    648   and so forth.  For example, if the grammar file contains:</p>
   635    649   
   636    650   <p><pre>
   637    651       %extra_argument { MyStruct *pAbc }
   638    652   </pre></p>
   639    653   
   640    654   <p>Then the Parse() function generated will have an 4th parameter
   641         -of type ``MyStruct*'' and all action routines will have access to
   642         -a variable named ``pAbc'' that is the value of the 4th parameter
          655  +of type "MyStruct*" and all action routines will have access to
          656  +a variable named "pAbc" that is the value of the 4th parameter
   643    657   in the most recent call to Parse().</p>
   644    658   
          659  +<a name='pfallback'></a>
          660  +<h4>The <tt>%fallback</tt> directive</h4>
          661  +
          662  +<p>The %fallback directive specifies an alternative meaning for one
          663  +or more tokens.  The alternative meaning is tried if the original token
          664  +would have generated a syntax error.
          665  +
          666  +<p>The %fallback directive was added to support robust parsing of SQL
          667  +syntax in <a href="https://www.sqlite.org/">SQLite</a>.
          668  +The SQL language contains a large assortment of keywords, each of which
          669  +appears as a different token to the language parser.  SQL contains so
          670  +many keywords, that it can be difficult for programmers to keep up with
          671  +them all.  Programmers will, therefore, sometimes mistakenly use an
          672  +obscure language keyword for an identifier.  The %fallback directive
          673  +provides a mechanism to tell the parser:  "If you are unable to parse
          674  +this keyword, try treating it as an identifier instead."
          675  +
          676  +<p>The syntax of %fallback is as follows:
          677  +
          678  +<blockquote>
          679  +<tt>%fallback</tt>  <i>ID</i> <i>TOKEN...</i> <b>.</b>
          680  +</blockquote>
          681  +
          682  +<p>In words, the %fallback directive is followed by a list of token names
          683  +terminated by a period.  The first token name is the fallback token - the
          684  +token to which all the other tokens fall back to.  The second and subsequent
          685  +arguments are tokens which fall back to the token identified by the first
          686  +argument.
          687  +
          688  +<a name='pifdef'></a>
          689  +<h4>The <tt>%ifdef</tt>, <tt>%ifndef</tt>, and <tt>%endif</tt> directives.</h4>
          690  +
          691  +<p>The %ifdef, %ifndef, and %endif directives are similar to
          692  +#ifdef, #ifndef, and #endif in the C-preprocessor, just not as general.
          693  +Each of these directives must begin at the left margin.  No whitespace
          694  +is allowed between the "%" and the directive name.
          695  +
          696  +<p>Grammar text in between "%ifdef MACRO" and the next nested "%endif" is
          697  +ignored unless the "-DMACRO" command-line option is used.  Grammar text
          698  +betwen "%ifndef MACRO" and the next nested "%endif" is included except when
          699  +the "-DMACRO" command-line option is used.
          700  +
          701  +<p>Note that the argument to %ifdef and %ifndef must be a single 
          702  +preprocessor symbol name, not a general expression.  There is no "%else"
          703  +directive.
          704  +
          705  +
          706  +<a name='pinclude'></a>
   645    707   <h4>The <tt>%include</tt> directive</h4>
   646    708   
   647    709   <p>The %include directive specifies C code that is included at the
   648    710   top of the generated parser.  You can include any text you want --
   649    711   the Lemon parser generator copies it blindly.  If you have multiple
   650         -%include directives in your grammar file the value of the last
   651         -%include directive overwrites all the others.</p.
          712  +%include directives in your grammar file, their values are concatenated
          713  +so that all %include code ultimately appears near the top of the
          714  +generated parser, in the same order as it appeared in the grammer.</p>
   652    715   
   653    716   <p>The %include directive is very handy for getting some extra #include
   654    717   preprocessor statements at the beginning of the generated parser.
   655    718   For example:</p>
   656    719   
   657    720   <p><pre>
   658    721      %include {#include &lt;unistd.h&gt;}
   659    722   </pre></p>
   660    723   
   661    724   <p>This might be needed, for example, if some of the C actions in the
   662    725   grammar call functions that are prototyed in unistd.h.</p>
   663    726   
          727  +<a name='pleft'></a>
   664    728   <h4>The <tt>%left</tt> directive</h4>
   665    729   
   666         -The %left directive is used (along with the %right and
   667         -%nonassoc directives) to declare precedences of terminal
   668         -symbols.  Every terminal symbol whose name appears after
   669         -a %left directive but before the next period (``.'') is
          730  +The %left directive is used (along with the <a href='#pright'>%right</a> and
          731  +<a href='#pnonassoc'>%nonassoc</a> directives) to declare precedences of 
          732  +terminal symbols.  Every terminal symbol whose name appears after
          733  +a %left directive but before the next period (".") is
   670    734   given the same left-associative precedence value.  Subsequent
   671    735   %left directives have higher precedence.  For example:</p>
   672    736   
   673    737   <p><pre>
   674    738      %left AND.
   675    739      %left OR.
   676    740      %nonassoc EQ NE GT GE LT LE.
................................................................................
   683    747   directive.</p>
   684    748   
   685    749   <p>LALR(1) grammars can get into a situation where they require
   686    750   a large amount of stack space if you make heavy use or right-associative
   687    751   operators.  For this reason, it is recommended that you use %left
   688    752   rather than %right whenever possible.</p>
   689    753   
          754  +<a name='pname'></a>
   690    755   <h4>The <tt>%name</tt> directive</h4>
   691    756   
   692    757   <p>By default, the functions generated by Lemon all begin with the
   693         -five-character string ``Parse''.  You can change this string to something
          758  +five-character string "Parse".  You can change this string to something
   694    759   different using the %name directive.  For instance:</p>
   695    760   
   696    761   <p><pre>
   697    762      %name Abcde
   698    763   </pre></p>
   699    764   
   700    765   <p>Putting this directive in the grammar file will cause Lemon to generate
................................................................................
   705    770   <li> AbcdeTrace(), and
   706    771   <li> Abcde().
   707    772   </ul>
   708    773   The %name directive allows you to generator two or more different
   709    774   parsers and link them all into the same executable.
   710    775   </p>
   711    776   
          777  +<a name='pnonassoc'></a>
   712    778   <h4>The <tt>%nonassoc</tt> directive</h4>
   713    779   
   714    780   <p>This directive is used to assign non-associative precedence to
   715         -one or more terminal symbols.  See the section on precedence rules
   716         -or on the %left directive for additional information.</p>
          781  +one or more terminal symbols.  See the section on 
          782  +<a href='#precrules'>precedence rules</a>
          783  +or on the <a href='#pleft'>%left</a> directive for additional information.</p>
   717    784   
          785  +<a name='parse_accept'></a>
   718    786   <h4>The <tt>%parse_accept</tt> directive</h4>
   719    787   
   720    788   <p>The %parse_accept directive specifies a block of C code that is
   721         -executed whenever the parser accepts its input string.  To ``accept''
          789  +executed whenever the parser accepts its input string.  To "accept"
   722    790   an input string means that the parser was able to process all tokens
   723    791   without error.</p>
   724    792   
   725    793   <p>For example:</p>
   726    794   
   727    795   <p><pre>
   728    796      %parse_accept {
   729    797         printf("parsing complete!\n");
   730    798      }
   731    799   </pre></p>
   732    800   
   733         -
          801  +<a name='parse_failure'></a>
   734    802   <h4>The <tt>%parse_failure</tt> directive</h4>
   735    803   
   736    804   <p>The %parse_failure directive specifies a block of C code that
   737    805   is executed whenever the parser fails complete.  This code is not
   738    806   executed until the parser has tried and failed to resolve an input
   739    807   error using is usual error recovery strategy.  The routine is
   740    808   only invoked when parsing is unable to continue.</p>
................................................................................
   741    809   
   742    810   <p><pre>
   743    811      %parse_failure {
   744    812        fprintf(stderr,"Giving up.  Parser is hopelessly lost...\n");
   745    813      }
   746    814   </pre></p>
   747    815   
          816  +<a name='pright'></a>
   748    817   <h4>The <tt>%right</tt> directive</h4>
   749    818   
   750    819   <p>This directive is used to assign right-associative precedence to
   751         -one or more terminal symbols.  See the section on precedence rules
   752         -or on the %left directive for additional information.</p>
          820  +one or more terminal symbols.  See the section on 
          821  +<a href='#precrules'>precedence rules</a>
          822  +or on the <a href='#pleft'>%left</a> directive for additional information.</p>
   753    823   
          824  +<a name='stack_overflow'></a>
   754    825   <h4>The <tt>%stack_overflow</tt> directive</h4>
   755    826   
   756    827   <p>The %stack_overflow directive specifies a block of C code that
   757    828   is executed if the parser's internal stack ever overflows.  Typically
   758    829   this just prints an error message.  After a stack overflow, the parser
   759    830   will be unable to continue and must be reset.</p>
   760    831   
................................................................................
   775    846   </pre>
   776    847   Not like this:
   777    848   <pre>
   778    849      list ::= element list.      // right-recursion.  Bad!
   779    850      list ::= .
   780    851   </pre>
   781    852   
          853  +<a name='stack_size'></a>
   782    854   <h4>The <tt>%stack_size</tt> directive</h4>
   783    855   
   784    856   <p>If stack overflow is a problem and you can't resolve the trouble
   785    857   by using left-recursion, then you might want to increase the size
   786    858   of the parser's stack using this directive.  Put an positive integer
   787    859   after the %stack_size directive and Lemon will generate a parse
   788    860   with a stack of the requested size.  The default value is 100.</p>
   789    861   
   790    862   <p><pre>
   791    863      %stack_size 2000
   792    864   </pre></p>
   793    865   
          866  +<a name='start_symbol'></a>
   794    867   <h4>The <tt>%start_symbol</tt> directive</h4>
   795    868   
   796    869   <p>By default, the start-symbol for the grammar that Lemon generates
   797    870   is the first non-terminal that appears in the grammar file.  But you
   798    871   can choose a different start-symbol using the %start_symbol directive.</p>
   799    872   
   800    873   <p><pre>
   801    874      %start_symbol  prog
   802    875   </pre></p>
   803    876   
          877  +<a name='token_destructor'></a>
   804    878   <h4>The <tt>%token_destructor</tt> directive</h4>
   805    879   
   806    880   <p>The %destructor directive assigns a destructor to a non-terminal
   807    881   symbol.  (See the description of the %destructor directive above.)
   808    882   This directive does the same thing for all terminal symbols.</p>
   809    883   
   810    884   <p>Unlike non-terminal symbols which may each have a different data type
   811    885   for their values, terminals all use the same data type (defined by
   812    886   the %token_type directive) and so they use a common destructor.  Other
   813    887   than that, the token destructor works just like the non-terminal
   814    888   destructors.</p>
   815    889   
          890  +<a name='token_prefix'></a>
   816    891   <h4>The <tt>%token_prefix</tt> directive</h4>
   817    892   
   818    893   <p>Lemon generates #defines that assign small integer constants
   819    894   to each terminal symbol in the grammar.  If desired, Lemon will
   820    895   add a prefix specified by this directive
   821    896   to each of the #defines it generates.
   822    897   So if the default output of Lemon looked like this:
................................................................................
   834    909   <pre>
   835    910       #define TOKEN_AND        1
   836    911       #define TOKEN_MINUS      2
   837    912       #define TOKEN_OR         3
   838    913       #define TOKEN_PLUS       4
   839    914   </pre>
   840    915   
          916  +<a name='token_type'></a><a name='ptype'></a>
   841    917   <h4>The <tt>%token_type</tt> and <tt>%type</tt> directives</h4>
   842    918   
   843    919   <p>These directives are used to specify the data types for values
   844    920   on the parser's stack associated with terminal and non-terminal
   845    921   symbols.  The values of all terminal symbols must be of the same
   846    922   type.  This turns out to be the same data type as the 3rd parameter
   847    923   to the Parse() function generated by Lemon.  Typically, you will
................................................................................
   849    925   token structure.  Like this:</p>
   850    926   
   851    927   <p><pre>
   852    928      %token_type    {Token*}
   853    929   </pre></p>
   854    930   
   855    931   <p>If the data type of terminals is not specified, the default value
   856         -is ``int''.</p>
          932  +is "int".</p>
   857    933   
   858    934   <p>Non-terminal symbols can each have their own data types.  Typically
   859    935   the data type  of a non-terminal is a pointer to the root of a parse-tree
   860    936   structure that contains all information about that non-terminal.
   861    937   For example:</p>
   862    938   
   863    939   <p><pre>
................................................................................
   870    946   on what the corresponding non-terminal or terminal symbol is.  But
   871    947   the grammar designer should keep in mind that the size of the union
   872    948   will be the size of its largest element.  So if you have a single
   873    949   non-terminal whose data type requires 1K of storage, then your 100
   874    950   entry parser stack will require 100K of heap space.  If you are willing
   875    951   and able to pay that price, fine.  You just need to know.</p>
   876    952   
          953  +<a name='pwildcard'></a>
          954  +<h4>The <tt>%wildcard</tt> directive</h4>
          955  +
          956  +<p>The %wildcard directive is followed by a single token name and a
          957  +period.  This directive specifies that the identified token should 
          958  +match any input token.
          959  +
          960  +<p>When the generated parser has the choice of matching an input against
          961  +the wildcard token and some other token, the other token is always used.
          962  +The wildcard token is only matched if there are no other alternatives.
          963  +
   877    964   <h3>Error Processing</h3>
   878    965   
   879    966   <p>After extensive experimentation over several years, it has been
   880    967   discovered that the error recovery strategy used by yacc is about
   881    968   as good as it gets.  And so that is what Lemon uses.</p>
   882    969   
   883    970   <p>When a Lemon-generated parser encounters a syntax error, it
   884    971   first invokes the code specified by the %syntax_error directive, if
   885    972   any.  It then enters its error recovery strategy.  The error recovery
   886    973   strategy is to begin popping the parsers stack until it enters a
   887    974   state where it is permitted to shift a special non-terminal symbol
   888         -named ``error''.  It then shifts this non-terminal and continues
          975  +named "error".  It then shifts this non-terminal and continues
   889    976   parsing.  But the %syntax_error routine will not be called again
   890    977   until at least three new tokens have been successfully shifted.</p>
   891    978   
   892    979   <p>If the parser pops its stack until the stack is empty, and it still
   893    980   is unable to shift the error symbol, then the %parse_failed routine
   894    981   is invoked and the parser resets itself to its start state, ready
   895    982   to begin parsing a new file.  This is what will happen at the very
   896    983   first syntax error, of course, if there are no instances of the 
   897         -``error'' non-terminal in your grammar.</p>
          984  +"error" non-terminal in your grammar.</p>
   898    985   
   899    986   </body>
   900    987   </html>

Changes to ext/fts5/fts5_expr.c.

  2496   2496     UNUSED_PARAM2(iUnused1, iUnused2);
  2497   2497   
  2498   2498     if( (tflags & FTS5_TOKEN_COLOCATED)==0 ) p->iOff++;
  2499   2499     for(i=0; i<pExpr->nPhrase; i++){
  2500   2500       Fts5ExprTerm *pTerm;
  2501   2501       if( p->aPopulator[i].bOk==0 ) continue;
  2502   2502       for(pTerm=&pExpr->apExprPhrase[i]->aTerm[0]; pTerm; pTerm=pTerm->pSynonym){
  2503         -      int nTerm = strlen(pTerm->zTerm);
         2503  +      int nTerm = (int)strlen(pTerm->zTerm);
  2504   2504         if( (nTerm==nToken || (nTerm<nToken && pTerm->bPrefix))
  2505   2505          && memcmp(pTerm->zTerm, pToken, nTerm)==0
  2506   2506         ){
  2507   2507           int rc = sqlite3Fts5PoslistWriterAppend(
  2508   2508               &pExpr->apExprPhrase[i]->poslist, &p->aPopulator[i].writer, p->iOff
  2509   2509           );
  2510   2510           if( rc ) return rc;

Changes to ext/fts5/fts5_index.c.

  3449   3449   static int fts5AllocateSegid(Fts5Index *p, Fts5Structure *pStruct){
  3450   3450     int iSegid = 0;
  3451   3451   
  3452   3452     if( p->rc==SQLITE_OK ){
  3453   3453       if( pStruct->nSegment>=FTS5_MAX_SEGMENT ){
  3454   3454         p->rc = SQLITE_FULL;
  3455   3455       }else{
  3456         -      while( iSegid==0 ){
  3457         -        int iLvl, iSeg;
  3458         -        sqlite3_randomness(sizeof(u32), (void*)&iSegid);
  3459         -        iSegid = iSegid & ((1 << FTS5_DATA_ID_B)-1);
  3460         -        for(iLvl=0; iLvl<pStruct->nLevel; iLvl++){
  3461         -          for(iSeg=0; iSeg<pStruct->aLevel[iLvl].nSeg; iSeg++){
  3462         -            if( iSegid==pStruct->aLevel[iLvl].aSeg[iSeg].iSegid ){
  3463         -              iSegid = 0;
  3464         -            }
         3456  +      /* FTS5_MAX_SEGMENT is currently defined as 2000. So the following
         3457  +      ** array is 63 elements, or 252 bytes, in size.  */
         3458  +      u32 aUsed[(FTS5_MAX_SEGMENT+31) / 32];
         3459  +      int iLvl, iSeg;
         3460  +      int i;
         3461  +      u32 mask;
         3462  +      memset(aUsed, 0, sizeof(aUsed));
         3463  +      for(iLvl=0; iLvl<pStruct->nLevel; iLvl++){
         3464  +        for(iSeg=0; iSeg<pStruct->aLevel[iLvl].nSeg; iSeg++){
         3465  +          int iId = pStruct->aLevel[iLvl].aSeg[iSeg].iSegid;
         3466  +          if( iId<=FTS5_MAX_SEGMENT ){
         3467  +            aUsed[(iId-1) / 32] |= 1 << ((iId-1) % 32);
  3465   3468             }
  3466   3469           }
  3467   3470         }
         3471  +
         3472  +      for(i=0; aUsed[i]==0xFFFFFFFF; i++);
         3473  +      mask = aUsed[i];
         3474  +      for(iSegid=0; mask & (1 << iSegid); iSegid++);
         3475  +      iSegid += 1 + i*32;
         3476  +
         3477  +#ifdef SQLITE_DEBUG
         3478  +      for(iLvl=0; iLvl<pStruct->nLevel; iLvl++){
         3479  +        for(iSeg=0; iSeg<pStruct->aLevel[iLvl].nSeg; iSeg++){
         3480  +          assert( iSegid!=pStruct->aLevel[iLvl].aSeg[iSeg].iSegid );
         3481  +        }
         3482  +      }
         3483  +      assert( iSegid>0 && iSegid<=FTS5_MAX_SEGMENT );
         3484  +#endif
  3468   3485       }
  3469   3486     }
  3470   3487   
  3471   3488     return iSegid;
  3472   3489   }
  3473   3490   
  3474   3491   /*
................................................................................
  3905   3922     Fts5PageWriter *pLeaf = &pWriter->writer;
  3906   3923     if( p->rc==SQLITE_OK ){
  3907   3924       assert( pLeaf->pgno>=1 );
  3908   3925       if( pLeaf->buf.n>4 ){
  3909   3926         fts5WriteFlushLeaf(p, pWriter);
  3910   3927       }
  3911   3928       *pnLeaf = pLeaf->pgno-1;
  3912         -    fts5WriteFlushBtree(p, pWriter);
         3929  +    if( pLeaf->pgno>1 ){
         3930  +      fts5WriteFlushBtree(p, pWriter);
         3931  +    }
  3913   3932     }
  3914   3933     fts5BufferFree(&pLeaf->term);
  3915   3934     fts5BufferFree(&pLeaf->buf);
  3916   3935     fts5BufferFree(&pLeaf->pgidx);
  3917   3936     fts5BufferFree(&pWriter->btterm);
  3918   3937   
  3919   3938     for(i=0; i<pWriter->nDlidx; i++){

Changes to ext/fts5/fts5_storage.c.

   141    141         if( rc!=SQLITE_OK && pzErrMsg ){
   142    142           *pzErrMsg = sqlite3_mprintf("%s", sqlite3_errmsg(pC->db));
   143    143         }
   144    144       }
   145    145     }
   146    146   
   147    147     *ppStmt = p->aStmt[eStmt];
          148  +  sqlite3_reset(*ppStmt);
   148    149     return rc;
   149    150   }
   150    151   
   151    152   
   152    153   static int fts5ExecPrintf(
   153    154     sqlite3 *db,
   154    155     char **pzErr,
................................................................................
  1117   1118       rc = sqlite3Fts5IndexSetCookie(p->pIndex, iNew);
  1118   1119       if( rc==SQLITE_OK ){
  1119   1120         p->pConfig->iCookie = iNew;
  1120   1121       }
  1121   1122     }
  1122   1123     return rc;
  1123   1124   }
  1124         -
  1125         -

Changes to ext/fts5/tool/fts5txt2db.tcl.

    13     13       {fts5                 "use fts5 (this is the default)"}
    14     14       {fts4                 "use fts4"}
    15     15       {colsize   "10 10 10" "list of column sizes"}
    16     16       {tblname   "t1"       "table name to create"}
    17     17       {detail    "full"     "Fts5 detail mode to use"}
    18     18       {repeat    1          "Load each file this many times"}
    19     19       {prefix    ""         "Fts prefix= option"}
           20  +    {trans     1          "True to use a transaction"}
    20     21       database
    21     22       file...
    22     23     } {
    23     24     This script is designed to create fts4/5 tables with more than one column.
    24     25     The -colsize option should be set to a Tcl list of integer values, one for
    25     26     each column in the table. Each value is the number of tokens that will be
    26     27     inserted into the column value for each row. For example, setting the -colsize
................................................................................
   210    211   set cols [create_table]
   211    212   set sql "INSERT INTO $A(tblname) VALUES(\$R([lindex $cols 0])"
   212    213   foreach c [lrange $cols 1 end] {
   213    214     append sql ", \$R($c)"
   214    215   }
   215    216   append sql ")"
   216    217   
   217         -db eval BEGIN
          218  +if {$A(trans)} { db eval BEGIN }
   218    219     while {$i < $N} {
   219    220       foreach c $cols s $A(colsize) {
   220    221         set R($c) [lrange $tokens $i [expr $i+$s-1]]
   221    222         incr i $s
   222    223       }
   223    224       db eval $sql
   224    225     }
   225         -db eval COMMIT
          226  +if {$A(trans)} { db eval COMMIT }
   226    227   
   227    228   
   228    229   

Changes to ext/rbu/rbu1.test.

   607    607            CREATE TABLE rbu.data_t1(a, b, rbu_control);
   608    608            INSERT INTO rbu.data_t1 VALUES(1, 2, 4);
   609    609          } {SQLITE_ERROR - invalid rbu_control value}
   610    610   
   611    611          9 {
   612    612            CREATE TABLE t1(a, b PRIMARY KEY) WITHOUT ROWID;
   613    613            CREATE TABLE rbu.data_t1(a, b, rbu_control);
   614         -         INSERT INTO rbu.data_t1 VALUES(1, 2, 2);
          614  +         INSERT INTO rbu.data_t1 VALUES(1, 2, 3);
   615    615          } {SQLITE_ERROR - invalid rbu_control value}
   616    616   
   617    617          10 {
   618    618            CREATE TABLE t2(a, b);
   619    619            CREATE TABLE rbu.data_t1(a, b, rbu_control);
   620    620            INSERT INTO rbu.data_t1 VALUES(1, 2, 2);
   621    621          } {SQLITE_ERROR - no such table: t1}

Changes to ext/rbu/rbudiff.test.

    14     14   #
    15     15   if {![info exists testdir]} {
    16     16     set testdir [file join [file dirname [info script]] .. .. test]
    17     17   }
    18     18   source $testdir/tester.tcl
    19     19   set testprefix rbudiff
    20     20   
    21         -if {$tcl_platform(platform)=="windows"} {
    22         -  set PROG "sqldiff.exe"
    23         -} else {
    24         -  set PROG "./sqldiff"
    25         -}
    26         -if {![file exe $PROG]} {
    27         -  puts "rbudiff.test cannot run because $PROG is not available"
    28         -  finish_test
    29         -  return
    30         -}
           21  +set PROG [test_find_sqldiff]
    31     22   db close
    32     23   
    33     24   proc get_rbudiff_sql {db1 db2} {
    34     25     exec $::PROG --rbu $db1 $db2
    35     26   }
    36     27   
    37     28   proc step_rbu {target rbu} {
................................................................................
    41     32       rbu close
    42     33       if {$rc != "SQLITE_OK"} break
    43     34     }
    44     35     set rc
    45     36   }
    46     37   
    47     38   proc apply_rbudiff {sql target} {
           39  +  test_rbucount $sql
    48     40     forcedelete rbu.db
    49     41     sqlite3 rbudb rbu.db
    50     42     rbudb eval $sql
    51     43     rbudb close
    52     44     step_rbu $target rbu.db
    53     45   }
           46  +
           47  +# The only argument is the output of an [sqldiff -rbu] run. This command
           48  +# tests that the contents of the rbu_count table is correct. An exception
           49  +# is thrown if it is not.
           50  +#
           51  +proc test_rbucount {sql} {
           52  +  sqlite3 tmpdb ""
           53  +  tmpdb eval $sql
           54  +  tmpdb eval {
           55  +    SELECT name FROM sqlite_master WHERE name LIKE 'data%' AND type='table'
           56  +  } {
           57  +    set a [tmpdb eval "SELECT count(*) FROM $name"]
           58  +    set b [tmpdb eval {SELECT cnt FROM rbu_count WHERE tbl = $name}]
           59  +    if {$a != $b} { 
           60  +      tmpdb close
           61  +      error "rbu_count error - tbl = $name" 
           62  +    }
           63  +  }
           64  +  tmpdb close
           65  +  return ""
           66  +}
    54     67   
    55     68   proc rbudiff_cksum {db1} {
    56     69     set txt ""
    57     70   
    58     71     sqlite3 dbtmp $db1
    59     72     foreach tbl [dbtmp eval {SELECT name FROM sqlite_master WHERE type='table'}] {
    60     73       set cols [list]

Added ext/rbu/rbuprogress.test.

            1  +# 2016 March 18
            2  +#
            3  +# The author disclaims copyright to this source code.  In place of
            4  +# a legal notice, here is a blessing:
            5  +#
            6  +#    May you do good and not evil.
            7  +#    May you find forgiveness for yourself and forgive others.
            8  +#    May you share freely, never taking more than you give.
            9  +#
           10  +#***********************************************************************
           11  +#
           12  +
           13  +source [file join [file dirname [info script]] rbu_common.tcl]
           14  +set ::testprefix rbuprogress
           15  +
           16  +
           17  +proc create_db_file {filename sql} {
           18  +  forcedelete $filename
           19  +  sqlite3 tmpdb $filename  
           20  +  tmpdb eval $sql
           21  +  tmpdb close
           22  +}
           23  +
           24  +# Create a simple RBU database. That expects to write to a table:
           25  +#
           26  +#   CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
           27  +#
           28  +proc create_rbu1 {filename} {
           29  +  create_db_file $filename {
           30  +    CREATE TABLE data_t1(a, b, c, rbu_control);
           31  +    INSERT INTO data_t1 VALUES(1, 2, 3, 0);
           32  +    INSERT INTO data_t1 VALUES(2, 'two', 'three', 0);
           33  +    INSERT INTO data_t1 VALUES(3, NULL, 8.2, 0);
           34  +
           35  +    CREATE TABLE rbu_count(tbl, cnt);
           36  +    INSERT INTO rbu_count VALUES('data_t1', 3);
           37  +  }
           38  +  return $filename
           39  +}
           40  +
           41  +
           42  +do_execsql_test 1.0 {
           43  +  CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
           44  +}
           45  +
           46  +do_test 1.1 {
           47  +  create_rbu1 rbu.db
           48  +  sqlite3rbu rbu test.db rbu.db
           49  +  rbu bp_progress
           50  +} {0 0}
           51  +do_test 1.2 { rbu step ; rbu bp_progress } {3333 0}
           52  +do_test 1.3 { rbu step ; rbu bp_progress } {6666 0}
           53  +do_test 1.4 { rbu step ; rbu bp_progress } {10000 0}
           54  +do_test 1.5 { rbu step ; rbu bp_progress } {10000 0}
           55  +do_test 1.6 { rbu step ; rbu bp_progress } {10000 0}
           56  +do_test 1.7 { rbu step ; rbu bp_progress } {10000 5000}
           57  +do_test 1.8 { rbu step ; rbu bp_progress } {10000 10000}
           58  +do_test 1.9 { rbu step ; rbu bp_progress } {10000 10000}
           59  +
           60  +do_test 1.10 {
           61  +  rbu close
           62  +} {SQLITE_DONE}
           63  +
           64  +#-------------------------------------------------------------------------
           65  +#
           66  +proc do_sp_test {tn bReopen target rbu reslist} {
           67  +  uplevel [list do_test $tn [subst -nocommands {
           68  +    if {$bReopen==0} { sqlite3rbu rbu $target $rbu }
           69  +    set res [list]
           70  +    while 1 {
           71  +      if {$bReopen} { sqlite3rbu rbu $target $rbu }
           72  +      set rc [rbu step]
           73  +      if {[set rc] != "SQLITE_OK"} { rbu close ; error "error 1" }
           74  +      lappend res [lindex [rbu bp_progress] 0]
           75  +      if {[lindex [set res] end]==10000} break
           76  +      if {$bReopen} { rbu close }
           77  +    }
           78  +    if {[set res] != [list $reslist]} {
           79  +      rbu close
           80  +      error "1. reslist incorrect (expect=$reslist got=[set res])"
           81  +    }
           82  +
           83  +    # One step to clean up the temporary tables used to update the only
           84  +    # target table in the rbu database. And one more to move the *-oal 
           85  +    # file to *-wal. After each of these steps, the progress remains
           86  +    # at "10000 0".
           87  +    #
           88  +    if {[lindex [list $reslist] 0]!=-1} {
           89  +      rbu step
           90  +      set res [rbu bp_progress]
           91  +      if {[set res] != [list 10000 0]} {
           92  +        rbu close
           93  +        error "2. reslist incorrect (expect=10000 0 got=[set res])"
           94  +      }
           95  +    }
           96  +
           97  +    rbu step
           98  +    set res [rbu bp_progress]
           99  +    if {[set res] != [list 10000 0]} {
          100  +      rbu close
          101  +      error "3. reslist incorrect (expect=10000 0 got=[set res])"
          102  +    }
          103  +
          104  +    # Do the checkpoint.
          105  +    while {[rbu step]=="SQLITE_OK"} { 
          106  +      foreach {a b} [rbu bp_progress] {}
          107  +      if {[set a]!=10000 || [set b]<=0 || [set b]>10000} {
          108  +        rbu close
          109  +        error "4. reslist incorrect (expect=10000 1..10000 got=[set a] [set b])"
          110  +      }
          111  +    }
          112  +
          113  +    set res [rbu bp_progress]
          114  +    if {[set res] != [list 10000 10000]} {
          115  +      rbu close
          116  +      error "5. reslist is incorrect (expect=10000 10000 got=[set res])"
          117  +    }
          118  +
          119  +    rbu close
          120  +  }] {SQLITE_DONE}]
          121  +}
          122  +
          123  +foreach {bReopen} { 0 1 } {
          124  +  reset_db
          125  +  do_test 2.$bReopen.1.0 {
          126  +    execsql {
          127  +      CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
          128  +    }
          129  +    create_db_file rbu.db {
          130  +      CREATE TABLE data_t1(a, b, c, rbu_control);
          131  +      INSERT INTO data_t1 VALUES(4, 4, 4, 0);
          132  +      INSERT INTO data_t1 VALUES(5, 5, 5, 0);
          133  +  
          134  +      CREATE TABLE rbu_count(tbl, cnt);
          135  +      INSERT INTO rbu_count VALUES('data_t1', 2);
          136  +    }
          137  +  } {}
          138  +  do_sp_test 2.$bReopen.1.1 $bReopen test.db rbu.db {5000 10000}
          139  +  
          140  +  reset_db
          141  +  do_test 2.$bReopen.2.0 {
          142  +    execsql { CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c) }
          143  +    create_rbu1 rbu.db
          144  +  } {rbu.db}
          145  +  do_sp_test 2.$bReopen.2.1 $bReopen test.db rbu.db {3333 6666 10000}
          146  +  
          147  +  reset_db
          148  +  do_test 2.$bReopen.3.0 {
          149  +    execsql { 
          150  +      CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
          151  +      CREATE INDEX i1 ON t1(b);
          152  +      INSERT INTO t1 VALUES(1, 1, 1);
          153  +      INSERT INTO t1 VALUES(2, 2, 2);
          154  +      INSERT INTO t1 VALUES(3, 3, 3);
          155  +    }
          156  +    create_db_file rbu.db {
          157  +      CREATE TABLE data_t1(a, b, c, rbu_control);
          158  +      INSERT INTO data_t1 VALUES(4, 4, 4, 0);
          159  +      INSERT INTO data_t1 VALUES(2, NULL, NULL, 1);
          160  +      INSERT INTO data_t1 VALUES(5, NULL, NULL, 1);
          161  +  
          162  +      CREATE TABLE rbu_count(tbl, cnt);
          163  +      INSERT INTO rbu_count VALUES('data_t1', 3);
          164  +    }
          165  +  } {}
          166  +  do_sp_test 2.$bReopen.3.1 $bReopen test.db rbu.db {1666 3333 6000 8000 10000}
          167  +  
          168  +  reset_db
          169  +  do_test 2.$bReopen.4.0 {
          170  +    execsql { 
          171  +      CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
          172  +      CREATE INDEX i1 ON t1(b);
          173  +      INSERT INTO t1 VALUES(1, 1, 1);
          174  +      INSERT INTO t1 VALUES(2, 2, 2);
          175  +      INSERT INTO t1 VALUES(3, 3, 3);
          176  +    }
          177  +    create_db_file rbu.db {
          178  +      CREATE TABLE data_t1(a, b, c, rbu_control);
          179  +      INSERT INTO data_t1 VALUES(2, 4, 4, '.xx');
          180  +  
          181  +      CREATE TABLE rbu_count(tbl, cnt);
          182  +      INSERT INTO rbu_count VALUES('data_t1', 1);
          183  +    }
          184  +  } {}
          185  +  do_sp_test 2.$bReopen.4.1 $bReopen test.db rbu.db {3333 6666 10000}
          186  +  
          187  +  reset_db
          188  +  do_test 2.$bReopen.5.0 {
          189  +    execsql { 
          190  +      CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
          191  +      CREATE INDEX i1 ON t1(b);
          192  +      INSERT INTO t1 VALUES(1, 1, 1);
          193  +      INSERT INTO t1 VALUES(2, 2, 2);
          194  +      INSERT INTO t1 VALUES(3, 3, 3);
          195  +    }
          196  +    create_db_file rbu.db {
          197  +      CREATE TABLE data_t1(a, b, c, rbu_control);
          198  +      INSERT INTO data_t1 VALUES(4, NULL, 4, '.xx');
          199  +  
          200  +      CREATE TABLE rbu_count(tbl, cnt);
          201  +      INSERT INTO rbu_count VALUES('data_t1', 1);
          202  +    }
          203  +  } {}
          204  +  do_sp_test 2.$bReopen.5.1 $bReopen test.db rbu.db {10000}
          205  +
          206  +  reset_db
          207  +  do_test 2.$bReopen.6.0 {
          208  +    execsql { 
          209  +      CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c);
          210  +      CREATE INDEX i1 ON t1(b);
          211  +      INSERT INTO t1 VALUES(1, 1, 1);
          212  +      INSERT INTO t1 VALUES(2, 2, 2);
          213  +      INSERT INTO t1 VALUES(3, 3, 3);
          214  +    }
          215  +    create_db_file rbu.db {
          216  +      CREATE TABLE data_t1(a, b, c, rbu_control);
          217  +      INSERT INTO data_t1 VALUES(4, 4, 4, 0);
          218  +      INSERT INTO data_t1 VALUES(2, NULL, NULL, 1);
          219  +      INSERT INTO data_t1 VALUES(5, NULL, NULL, 1);
          220  +    }
          221  +  } {}
          222  +  do_sp_test 2.$bReopen.6.1 $bReopen test.db rbu.db {-1 -1 -1 -1 -1 10000}
          223  +}
          224  +
          225  +#-------------------------------------------------------------------------
          226  +# The following tests verify that the API works when resuming an update
          227  +# during the incremental checkpoint stage.
          228  +#
          229  +proc do_phase2_test {tn bReopen target rbu nStep} {
          230  +  uplevel [list do_test $tn [subst -nocommands {
          231  +
          232  +    # Build the OAL/WAL file:
          233  +    sqlite3rbu rbu $target $rbu
          234  +    while {[lindex [rbu bp_progress] 0]<10000} { 
          235  +      set rc [rbu step]
          236  +      if {"SQLITE_OK" != [set rc]} { rbu close }
          237  +    }
          238  +
          239  +    # Clean up the temp tables and move the *-oal file to *-wal.
          240  +    rbu step
          241  +    rbu step
          242  +
          243  +    for {set i 0} {[set i] < $nStep} {incr i} {
          244  +      if {$bReopen} {
          245  +        rbu close
          246  +        sqlite3rbu rbu $target $rbu
          247  +      }
          248  +      rbu step
          249  +      set res [rbu bp_progress]
          250  +      set expect [expr (1 + [set i]) * 10000 / $nStep]
          251  +      if {[lindex [set res] 1] != [set expect]} {
          252  +        error "Have [set res], expected 10000 [set expect]"
          253  +      }
          254  +    }
          255  +
          256  +    set rc [rbu step]
          257  +    if {[set rc] != "SQLITE_DONE"} {
          258  +      error "Have [set rc], expected SQLITE_DONE" 
          259  +    }
          260  +
          261  +    rbu close
          262  +  }] {SQLITE_DONE}]
          263  +}
          264  +
          265  +foreach bReopen {0 1} {
          266  +  do_test 3.$bReopen.1.0 {
          267  +    reset_db
          268  +    execsql {
          269  +      CREATE TABLE t1(a INTEGER PRIMARY KEY, b);
          270  +      CREATE TABLE t2(a INTEGER PRIMARY KEY, b);
          271  +      CREATE TABLE t3(a INTEGER PRIMARY KEY, b);
          272  +      CREATE TABLE t4(a INTEGER PRIMARY KEY, b);
          273  +    }
          274  +    create_db_file rbu.db {
          275  +      CREATE TABLE data_t1(a, b, rbu_control);
          276  +      CREATE TABLE data_t2(a, b, rbu_control);
          277  +      CREATE TABLE data_t3(a, b, rbu_control);
          278  +      CREATE TABLE data_t4(a, b, rbu_control);
          279  +      INSERT INTO data_t1 VALUES(1, 2, 0);
          280  +      INSERT INTO data_t2 VALUES(1, 2, 0);
          281  +      INSERT INTO data_t3 VALUES(1, 2, 0);
          282  +      INSERT INTO data_t4 VALUES(1, 2, 0);
          283  +  
          284  +      CREATE TABLE rbu_count(tbl, cnt);
          285  +      INSERT INTO rbu_count VALUES('data_t1', 1);
          286  +      INSERT INTO rbu_count VALUES('data_t2', 1);
          287  +      INSERT INTO rbu_count VALUES('data_t3', 1);
          288  +      INSERT INTO rbu_count VALUES('data_t4', 1);
          289  +    }
          290  +  } {}
          291  +  do_phase2_test 3.$bReopen.1.1 $bReopen test.db rbu.db 5
          292  +}
          293  +
          294  +
          295  +foreach {bReopen} { 0 1 } {
          296  +  foreach {tn tbl} {
          297  +    ipk { CREATE TABLE t1(a INTEGER PRIMARY KEY, b, c) }
          298  +    wr  { CREATE TABLE t1(a INT PRIMARY KEY, b, c) WITHOUT ROWID }
          299  +    pk  { CREATE TABLE t1(a INT PRIMARY KEY, b, c) }
          300  +  } {
          301  +
          302  +    foreach {tn2 rbusql r1 r3} {
          303  +      1 {
          304  +        CREATE TABLE data0_t1(a, b, c, rbu_control);
          305  +        INSERT INTO data0_t1 VALUES(15, 15, 15, 0);
          306  +        INSERT INTO data0_t1 VALUES(20, 20, 20, 0);
          307  +        CREATE TABLE rbu_count(tbl, cnt);
          308  +        INSERT INTO rbu_count VALUES('data0_t1', 2); 
          309  +      } 
          310  +      {2500 5000 7500 10000}
          311  +      {1666 3333 5000 6666 8333 10000}
          312  +
          313  +      2 {
          314  +        CREATE TABLE data0_t1(a, b, c, rbu_control);
          315  +        INSERT INTO data0_t1 VALUES(10, 10, 10, 2);
          316  +        CREATE TABLE rbu_count(tbl, cnt);
          317  +        INSERT INTO rbu_count VALUES('data0_t1', 1); 
          318  +      } 
          319  +      {3333 6666 10000}
          320  +      {2000 4000 6000 8000 10000}
          321  +
          322  +      3 {
          323  +        CREATE TABLE data0_t1(a, b, c, rbu_control);
          324  +        INSERT INTO data0_t1 VALUES(7, 7, 7, 2);
          325  +        INSERT INTO data0_t1 VALUES(10, 10, 10, 2);
          326  +        CREATE TABLE rbu_count(tbl, cnt);
          327  +        INSERT INTO rbu_count VALUES('data0_t1', 2); 
          328  +      } 
          329  +      {2500 4000 6000 8000 10000}
          330  +      {1666 2500 3750 5000 6250 7500 8750 10000}
          331  +
          332  +    } {
          333  +
          334  +      reset_db ; execsql $tbl
          335  +      do_test 4.$tn.$bReopen.$tn2.0 {
          336  +        execsql {
          337  +          CREATE INDEX t1c ON t1(c);
          338  +          INSERT INTO t1 VALUES(1, 1, 1);
          339  +          INSERT INTO t1 VALUES(5, 5, 5);
          340  +          INSERT INTO t1 VALUES(10, 10, 10);
          341  +        }
          342  +        create_db_file rbu.db $rbusql
          343  +      } {}
          344  +
          345  +      set R(ipk) $r1
          346  +      set R(wr) $r1
          347  +      set R(pk) $r3
          348  +      do_sp_test 4.$tn.$bReopen.$tn2.1 $bReopen test.db rbu.db $R($tn)
          349  +    }
          350  +  }
          351  +}
          352  +
          353  +foreach {bReopen} { 0 1 } {
          354  +  foreach {tn tbl} {
          355  +    nopk { 
          356  +      CREATE TABLE t1(a, b, c);
          357  +      CREATE INDEX t1c ON t1(c);
          358  +    }
          359  +    vtab { 
          360  +      CREATE VIRTUAL TABLE t1 USING fts5(a, b, c);
          361  +    }
          362  +  } {
          363  +
          364  +    foreach {tn2 rbusql r1 r2} {
          365  +      1 {
          366  +        CREATE TABLE data0_t1(a, b, c, rbu_rowid, rbu_control);
          367  +        INSERT INTO data0_t1 VALUES(15, 15, 15, 4, 0);
          368  +        INSERT INTO data0_t1 VALUES(20, 20, 20, 5, 0);
          369  +        CREATE TABLE rbu_count(tbl, cnt);
          370  +        INSERT INTO rbu_count VALUES('data0_t1', 2); 
          371  +      } 
          372  +      {2500 5000 7500 10000}
          373  +      {5000 10000}
          374  +
          375  +      2 {
          376  +        CREATE TABLE data0_t1(rbu_rowid, a, b, c, rbu_control);
          377  +        INSERT INTO data0_t1 VALUES(0, 7, 7, 7, 2);
          378  +        INSERT INTO data0_t1 VALUES(2, 10, 10, 10, 2);
          379  +        CREATE TABLE rbu_count(tbl, cnt);
          380  +        INSERT INTO rbu_count VALUES('data0_t1', 2); 
          381  +      } 
          382  +      {2500 4000 6000 8000 10000}
          383  +      {5000 10000}
          384  +
          385  +      3 {
          386  +        CREATE TABLE data0_t1(rbu_rowid, a, b, c, rbu_control);
          387  +        INSERT INTO data0_t1 VALUES(1, NULL, NULL, NULL, 1);
          388  +        INSERT INTO data0_t1 VALUES(2, NULL, NULL, 7, '..x');
          389  +        CREATE TABLE rbu_count(tbl, cnt);
          390  +        INSERT INTO rbu_count VALUES('data0_t1', 2); 
          391  +      } 
          392  +      {2500 4000 6000 8000 10000}
          393  +      {5000 10000}
          394  +    } {
          395  +
          396  +      reset_db ; execsql $tbl
          397  +      do_test 5.$tn.$bReopen.$tn2.0 {
          398  +        execsql {
          399  +          INSERT INTO t1 VALUES(1, 1, 1);
          400  +          INSERT INTO t1 VALUES(5, 5, 5);
          401  +          INSERT INTO t1 VALUES(10, 10, 10);
          402  +        }
          403  +        create_db_file rbu.db $rbusql
          404  +      } {}
          405  +
          406  +      set R(nopk) $r1
          407  +      set R(vtab) $r2
          408  +      do_sp_test 5.$tn.$bReopen.$tn2.1 $bReopen test.db rbu.db $R($tn)
          409  +    }
          410  +  }
          411  +}
          412  +
          413  +
          414  +finish_test
          415  +

Changes to ext/rbu/sqlite3rbu.c.

   143    143   ** RBU_STATE_COOKIE:
   144    144   **   Valid if STAGE==1. The current change-counter cookie value in the 
   145    145   **   target db file.
   146    146   **
   147    147   ** RBU_STATE_OALSZ:
   148    148   **   Valid if STAGE==1. The size in bytes of the *-oal file.
   149    149   */
   150         -#define RBU_STATE_STAGE       1
   151         -#define RBU_STATE_TBL         2
   152         -#define RBU_STATE_IDX         3
   153         -#define RBU_STATE_ROW         4
   154         -#define RBU_STATE_PROGRESS    5
   155         -#define RBU_STATE_CKPT        6
   156         -#define RBU_STATE_COOKIE      7
   157         -#define RBU_STATE_OALSZ       8
          150  +#define RBU_STATE_STAGE        1
          151  +#define RBU_STATE_TBL          2
          152  +#define RBU_STATE_IDX          3
          153  +#define RBU_STATE_ROW          4
          154  +#define RBU_STATE_PROGRESS     5
          155  +#define RBU_STATE_CKPT         6
          156  +#define RBU_STATE_COOKIE       7
          157  +#define RBU_STATE_OALSZ        8
          158  +#define RBU_STATE_PHASEONESTEP 9
   158    159   
   159    160   #define RBU_STAGE_OAL         1
   160    161   #define RBU_STAGE_MOVE        2
   161    162   #define RBU_STAGE_CAPTURE     3
   162    163   #define RBU_STAGE_CKPT        4
   163    164   #define RBU_STAGE_DONE        5
   164    165   
................................................................................
   196    197     char *zTbl;
   197    198     char *zIdx;
   198    199     i64 iWalCksum;
   199    200     int nRow;
   200    201     i64 nProgress;
   201    202     u32 iCookie;
   202    203     i64 iOalSz;
          204  +  i64 nPhaseOneStep;
   203    205   };
   204    206   
   205    207   struct RbuUpdateStmt {
   206    208     char *zMask;                    /* Copy of update mask used with pUpdate */
   207    209     sqlite3_stmt *pUpdate;          /* Last update statement (or NULL) */
   208    210     RbuUpdateStmt *pNext;
   209    211   };
................................................................................
   240    242     int bCleanup;                   /* True in "cleanup" state */
   241    243     const char *zTbl;               /* Name of target db table */
   242    244     const char *zDataTbl;           /* Name of rbu db table (or null) */
   243    245     const char *zIdx;               /* Name of target db index (or null) */
   244    246     int iTnum;                      /* Root page of current object */
   245    247     int iPkTnum;                    /* If eType==EXTERNAL, root of PK index */
   246    248     int bUnique;                    /* Current index is unique */
          249  +  int nIndex;                     /* Number of aux. indexes on table zTbl */
   247    250   
   248    251     /* Statements created by rbuObjIterPrepareAll() */
   249    252     int nCol;                       /* Number of columns in current object */
   250    253     sqlite3_stmt *pSelect;          /* Source data */
   251    254     sqlite3_stmt *pInsert;          /* Statement for INSERT operations */
   252    255     sqlite3_stmt *pDelete;          /* Statement for DELETE ops */
   253    256     sqlite3_stmt *pTmpInsert;       /* Insert into rbu_tmp_$zDataTbl */
................................................................................
   293    296   struct RbuFrame {
   294    297     u32 iDbPage;
   295    298     u32 iWalFrame;
   296    299   };
   297    300   
   298    301   /*
   299    302   ** RBU handle.
          303  +**
          304  +** nPhaseOneStep:
          305  +**   If the RBU database contains an rbu_count table, this value is set to
          306  +**   a running estimate of the number of b-tree operations required to 
          307  +**   finish populating the *-oal file. This allows the sqlite3_bp_progress()
          308  +**   API to calculate the permyriadage progress of populating the *-oal file
          309  +**   using the formula:
          310  +**
          311  +**     permyriadage = (10000 * nProgress) / nPhaseOneStep
          312  +**
          313  +**   nPhaseOneStep is initialized to the sum of:
          314  +**
          315  +**     nRow * (nIndex + 1)
          316  +**
          317  +**   for all source tables in the RBU database, where nRow is the number
          318  +**   of rows in the source table and nIndex the number of indexes on the
          319  +**   corresponding target database table.
          320  +**
          321  +**   This estimate is accurate if the RBU update consists entirely of
          322  +**   INSERT operations. However, it is inaccurate if:
          323  +**
          324  +**     * the RBU update contains any UPDATE operations. If the PK specified
          325  +**       for an UPDATE operation does not exist in the target table, then
          326  +**       no b-tree operations are required on index b-trees. Or if the 
          327  +**       specified PK does exist, then (nIndex*2) such operations are
          328  +**       required (one delete and one insert on each index b-tree).
          329  +**
          330  +**     * the RBU update contains any DELETE operations for which the specified
          331  +**       PK does not exist. In this case no operations are required on index
          332  +**       b-trees.
          333  +**
          334  +**     * the RBU update contains REPLACE operations. These are similar to
          335  +**       UPDATE operations.
          336  +**
          337  +**   nPhaseOneStep is updated to account for the conditions above during the
          338  +**   first pass of each source table. The updated nPhaseOneStep value is
          339  +**   stored in the rbu_state table if the RBU update is suspended.
   300    340   */
   301    341   struct sqlite3rbu {
   302    342     int eStage;                     /* Value of RBU_STATE_STAGE field */
   303    343     sqlite3 *dbMain;                /* target database handle */
   304    344     sqlite3 *dbRbu;                 /* rbu database handle */
   305    345     char *zTarget;                  /* Path to target db */
   306    346     char *zRbu;                     /* Path to rbu db */
................................................................................
   310    350     char *zErrmsg;                  /* Error message if rc!=SQLITE_OK */
   311    351     int nStep;                      /* Rows processed for current object */
   312    352     int nProgress;                  /* Rows processed for all objects */
   313    353     RbuObjIter objiter;             /* Iterator for skipping through tbl/idx */
   314    354     const char *zVfsName;           /* Name of automatically created rbu vfs */
   315    355     rbu_file *pTargetFd;            /* File handle open on target db */
   316    356     i64 iOalSz;
          357  +  i64 nPhaseOneStep;
   317    358   
   318    359     /* The following state variables are used as part of the incremental
   319    360     ** checkpoint stage (eStage==RBU_STAGE_CKPT). See comments surrounding
   320    361     ** function rbuSetupCheckpoint() for details.  */
   321    362     u32 iMaxFrame;                  /* Largest iWalFrame value in aFrame[] */
   322    363     u32 mLock;
   323    364     int nFrame;                     /* Entries in aFrame[] array */
................................................................................
  1140   1181     if( p->rc==SQLITE_OK ){
  1141   1182       memcpy(pIter->abIndexed, pIter->abTblPk, sizeof(u8)*pIter->nTblCol);
  1142   1183       p->rc = prepareFreeAndCollectError(p->dbMain, &pList, &p->zErrmsg,
  1143   1184           sqlite3_mprintf("PRAGMA main.index_list = %Q", pIter->zTbl)
  1144   1185       );
  1145   1186     }
  1146   1187   
         1188  +  pIter->nIndex = 0;
  1147   1189     while( p->rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pList) ){
  1148   1190       const char *zIdx = (const char*)sqlite3_column_text(pList, 1);
  1149   1191       sqlite3_stmt *pXInfo = 0;
  1150   1192       if( zIdx==0 ) break;
  1151   1193       p->rc = prepareFreeAndCollectError(p->dbMain, &pXInfo, &p->zErrmsg,
  1152   1194           sqlite3_mprintf("PRAGMA main.index_xinfo = %Q", zIdx)
  1153   1195       );
  1154   1196       while( p->rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pXInfo) ){
  1155   1197         int iCid = sqlite3_column_int(pXInfo, 1);
  1156   1198         if( iCid>=0 ) pIter->abIndexed[iCid] = 1;
  1157   1199       }
  1158   1200       rbuFinalize(p, pXInfo);
  1159   1201       bIndex = 1;
         1202  +    pIter->nIndex++;
         1203  +  }
         1204  +
         1205  +  if( pIter->eType==RBU_PK_WITHOUT_ROWID ){
         1206  +    /* "PRAGMA index_list" includes the main PK b-tree */
         1207  +    pIter->nIndex--;
  1160   1208     }
  1161   1209   
  1162   1210     rbuFinalize(p, pList);
  1163   1211     if( bIndex==0 ) pIter->abIndexed = 0;
  1164   1212   }
  1165   1213   
  1166   1214   
................................................................................
  1266   1314           iOrder++;
  1267   1315         }
  1268   1316       }
  1269   1317   
  1270   1318       rbuFinalize(p, pStmt);
  1271   1319       rbuObjIterCacheIndexedCols(p, pIter);
  1272   1320       assert( pIter->eType!=RBU_PK_VTAB || pIter->abIndexed==0 );
         1321  +    assert( pIter->eType!=RBU_PK_VTAB || pIter->nIndex==0 );
  1273   1322     }
  1274   1323   
  1275   1324     return p->rc;
  1276   1325   }
  1277   1326   
  1278   1327   /*
  1279   1328   ** This function constructs and returns a pointer to a nul-terminated 
................................................................................
  1818   1867     sqlite3_context *pCtx, 
  1819   1868     int nVal,
  1820   1869     sqlite3_value **apVal
  1821   1870   ){
  1822   1871     sqlite3rbu *p = sqlite3_user_data(pCtx);
  1823   1872     int rc = SQLITE_OK;
  1824   1873     int i;
         1874  +
         1875  +  assert( sqlite3_value_int(apVal[0])!=0
         1876  +      || p->objiter.eType==RBU_PK_EXTERNAL 
         1877  +      || p->objiter.eType==RBU_PK_NONE 
         1878  +  );
         1879  +  if( sqlite3_value_int(apVal[0])!=0 ){
         1880  +    p->nPhaseOneStep += p->objiter.nIndex;
         1881  +  }
  1825   1882   
  1826   1883     for(i=0; rc==SQLITE_OK && i<nVal; i++){
  1827   1884       rc = sqlite3_bind_value(p->objiter.pTmpInsert, i+1, apVal[i]);
  1828   1885     }
  1829   1886     if( rc==SQLITE_OK ){
  1830   1887       sqlite3_step(p->objiter.pTmpInsert);
  1831   1888       rc = sqlite3_reset(p->objiter.pTmpInsert);
................................................................................
  2563   2620     RbuObjIter *pIter = &p->objiter;
  2564   2621     sqlite3_value *pVal;
  2565   2622     sqlite3_stmt *pWriter;
  2566   2623     int i;
  2567   2624   
  2568   2625     assert( p->rc==SQLITE_OK );
  2569   2626     assert( eType!=RBU_DELETE || pIter->zIdx==0 );
         2627  +  assert( eType==RBU_DELETE || eType==RBU_IDX_DELETE
         2628  +       || eType==RBU_INSERT || eType==RBU_IDX_INSERT
         2629  +  );
         2630  +
         2631  +  /* If this is a delete, decrement nPhaseOneStep by nIndex. If the DELETE
         2632  +  ** statement below does actually delete a row, nPhaseOneStep will be
         2633  +  ** incremented by the same amount when SQL function rbu_tmp_insert()
         2634  +  ** is invoked by the trigger.  */
         2635  +  if( eType==RBU_DELETE ){
         2636  +    p->nPhaseOneStep -= p->objiter.nIndex;
         2637  +  }
  2570   2638   
  2571   2639     if( eType==RBU_IDX_DELETE || eType==RBU_DELETE ){
  2572   2640       pWriter = pIter->pDelete;
  2573   2641     }else{
  2574   2642       pWriter = pIter->pInsert;
  2575   2643     }
  2576   2644   
................................................................................
  2634   2702     if( eType ){
  2635   2703       assert( eType==RBU_INSERT     || eType==RBU_DELETE
  2636   2704            || eType==RBU_REPLACE    || eType==RBU_IDX_DELETE
  2637   2705            || eType==RBU_IDX_INSERT || eType==RBU_UPDATE
  2638   2706       );
  2639   2707       assert( eType!=RBU_UPDATE || pIter->zIdx==0 );
  2640   2708   
  2641         -    if( pIter->zIdx==0 && eType==RBU_IDX_DELETE ){
         2709  +    if( pIter->zIdx==0 && (eType==RBU_IDX_DELETE || eType==RBU_IDX_INSERT) ){
  2642   2710         rbuBadControlError(p);
  2643   2711       }
  2644   2712       else if( eType==RBU_REPLACE ){
  2645         -      if( pIter->zIdx==0 ) rbuStepOneOp(p, RBU_DELETE);
         2713  +      if( pIter->zIdx==0 ){
         2714  +        p->nPhaseOneStep += p->objiter.nIndex;
         2715  +        rbuStepOneOp(p, RBU_DELETE);
         2716  +      }
  2646   2717         if( p->rc==SQLITE_OK ) rbuStepOneOp(p, RBU_INSERT);
  2647   2718       }
  2648   2719       else if( eType!=RBU_UPDATE ){
  2649   2720         rbuStepOneOp(p, eType);
  2650   2721       }
  2651   2722       else{
  2652   2723         sqlite3_value *pVal;
  2653   2724         sqlite3_stmt *pUpdate = 0;
  2654   2725         assert( eType==RBU_UPDATE );
         2726  +      p->nPhaseOneStep -= p->objiter.nIndex;
  2655   2727         rbuGetUpdateStmt(p, pIter, zMask, &pUpdate);
  2656   2728         if( pUpdate ){
  2657   2729           int i;
  2658   2730           for(i=0; p->rc==SQLITE_OK && i<pIter->nCol; i++){
  2659   2731             char c = zMask[pIter->aiSrcOrder[i]];
  2660   2732             pVal = sqlite3_column_value(pIter->pSelect, i);
  2661   2733             if( pIter->abTblPk[i] || c!='.' ){
................................................................................
  2725   2797             "(%d, %d), "
  2726   2798             "(%d, %Q), "
  2727   2799             "(%d, %Q), "
  2728   2800             "(%d, %d), "
  2729   2801             "(%d, %d), "
  2730   2802             "(%d, %lld), "
  2731   2803             "(%d, %lld), "
         2804  +          "(%d, %lld), "
  2732   2805             "(%d, %lld) ",
  2733   2806             p->zStateDb,
  2734   2807             RBU_STATE_STAGE, eStage,
  2735   2808             RBU_STATE_TBL, p->objiter.zTbl, 
  2736   2809             RBU_STATE_IDX, p->objiter.zIdx, 
  2737   2810             RBU_STATE_ROW, p->nStep, 
  2738   2811             RBU_STATE_PROGRESS, p->nProgress,
  2739   2812             RBU_STATE_CKPT, p->iWalCksum,
  2740   2813             RBU_STATE_COOKIE, (i64)p->pTargetFd->iCookie,
  2741         -          RBU_STATE_OALSZ, p->iOalSz
         2814  +          RBU_STATE_OALSZ, p->iOalSz,
         2815  +          RBU_STATE_PHASEONESTEP, p->nPhaseOneStep
  2742   2816         )
  2743   2817       );
  2744   2818       assert( pInsert==0 || rc==SQLITE_OK );
  2745   2819   
  2746   2820       if( rc==SQLITE_OK ){
  2747   2821         sqlite3_step(pInsert);
  2748   2822         rc = sqlite3_finalize(pInsert);
................................................................................
  2921   2995           pRet->iCookie = (u32)sqlite3_column_int64(pStmt, 1);
  2922   2996           break;
  2923   2997   
  2924   2998         case RBU_STATE_OALSZ:
  2925   2999           pRet->iOalSz = (u32)sqlite3_column_int64(pStmt, 1);
  2926   3000           break;
  2927   3001   
         3002  +      case RBU_STATE_PHASEONESTEP:
         3003  +        pRet->nPhaseOneStep = sqlite3_column_int64(pStmt, 1);
         3004  +        break;
         3005  +
  2928   3006         default:
  2929   3007           rc = SQLITE_CORRUPT;
  2930   3008           break;
  2931   3009       }
  2932   3010     }
  2933   3011     rc2 = sqlite3_finalize(pStmt);
  2934   3012     if( rc==SQLITE_OK ) rc = rc2;
................................................................................
  3027   3105   */
  3028   3106   static void rbuDeleteVfs(sqlite3rbu *p){
  3029   3107     if( p->zVfsName ){
  3030   3108       sqlite3rbu_destroy_vfs(p->zVfsName);
  3031   3109       p->zVfsName = 0;
  3032   3110     }
  3033   3111   }
         3112  +
         3113  +/*
         3114  +** This user-defined SQL function is invoked with a single argument - the
         3115  +** name of a table expected to appear in the target database. It returns
         3116  +** the number of auxilliary indexes on the table.
         3117  +*/
         3118  +static void rbuIndexCntFunc(
         3119  +  sqlite3_context *pCtx, 
         3120  +  int nVal,
         3121  +  sqlite3_value **apVal
         3122  +){
         3123  +  sqlite3rbu *p = (sqlite3rbu*)sqlite3_user_data(pCtx);
         3124  +  sqlite3_stmt *pStmt = 0;
         3125  +  char *zErrmsg = 0;
         3126  +  int rc;
         3127  +
         3128  +  assert( nVal==1 );
         3129  +  
         3130  +  rc = prepareFreeAndCollectError(p->dbMain, &pStmt, &zErrmsg, 
         3131  +      sqlite3_mprintf("SELECT count(*) FROM sqlite_master "
         3132  +        "WHERE type='index' AND tbl_name = %Q", sqlite3_value_text(apVal[0]))
         3133  +  );
         3134  +  if( rc!=SQLITE_OK ){
         3135  +    sqlite3_result_error(pCtx, zErrmsg, -1);
         3136  +  }else{
         3137  +    int nIndex = 0;
         3138  +    if( SQLITE_ROW==sqlite3_step(pStmt) ){
         3139  +      nIndex = sqlite3_column_int(pStmt, 0);
         3140  +    }
         3141  +    rc = sqlite3_finalize(pStmt);
         3142  +    if( rc==SQLITE_OK ){
         3143  +      sqlite3_result_int(pCtx, nIndex);
         3144  +    }else{
         3145  +      sqlite3_result_error(pCtx, sqlite3_errmsg(p->dbMain), -1);
         3146  +    }
         3147  +  }
         3148  +
         3149  +  sqlite3_free(zErrmsg);
         3150  +}
         3151  +
         3152  +/*
         3153  +** If the RBU database contains the rbu_count table, use it to initialize
         3154  +** the sqlite3rbu.nPhaseOneStep variable. The schema of the rbu_count table
         3155  +** is assumed to contain the same columns as:
         3156  +**
         3157  +**   CREATE TABLE rbu_count(tbl TEXT PRIMARY KEY, cnt INTEGER) WITHOUT ROWID;
         3158  +**
         3159  +** There should be one row in the table for each data_xxx table in the
         3160  +** database. The 'tbl' column should contain the name of a data_xxx table,
         3161  +** and the cnt column the number of rows it contains.
         3162  +**
         3163  +** sqlite3rbu.nPhaseOneStep is initialized to the sum of (1 + nIndex) * cnt
         3164  +** for all rows in the rbu_count table, where nIndex is the number of 
         3165  +** indexes on the corresponding target database table.
         3166  +*/
         3167  +static void rbuInitPhaseOneSteps(sqlite3rbu *p){
         3168  +  if( p->rc==SQLITE_OK ){
         3169  +    sqlite3_stmt *pStmt = 0;
         3170  +    int bExists = 0;                /* True if rbu_count exists */
         3171  +
         3172  +    p->nPhaseOneStep = -1;
         3173  +
         3174  +    p->rc = sqlite3_create_function(p->dbRbu, 
         3175  +        "rbu_index_cnt", 1, SQLITE_UTF8, (void*)p, rbuIndexCntFunc, 0, 0
         3176  +    );
         3177  +  
         3178  +    /* Check for the rbu_count table. If it does not exist, or if an error
         3179  +    ** occurs, nPhaseOneStep will be left set to -1. */
         3180  +    if( p->rc==SQLITE_OK ){
         3181  +      p->rc = prepareAndCollectError(p->dbRbu, &pStmt, &p->zErrmsg,
         3182  +          "SELECT 1 FROM sqlite_master WHERE tbl_name = 'rbu_count'"
         3183  +      );
         3184  +    }
         3185  +    if( p->rc==SQLITE_OK ){
         3186  +      if( SQLITE_ROW==sqlite3_step(pStmt) ){
         3187  +        bExists = 1;
         3188  +      }
         3189  +      p->rc = sqlite3_finalize(pStmt);
         3190  +    }
         3191  +  
         3192  +    if( p->rc==SQLITE_OK && bExists ){
         3193  +      p->rc = prepareAndCollectError(p->dbRbu, &pStmt, &p->zErrmsg,
         3194  +          "SELECT sum(cnt * (1 + rbu_index_cnt(rbu_target_name(tbl))))"
         3195  +          "FROM rbu_count"
         3196  +      );
         3197  +      if( p->rc==SQLITE_OK ){
         3198  +        if( SQLITE_ROW==sqlite3_step(pStmt) ){
         3199  +          p->nPhaseOneStep = sqlite3_column_int64(pStmt, 0);
         3200  +        }
         3201  +        p->rc = sqlite3_finalize(pStmt);
         3202  +      }
         3203  +    }
         3204  +  }
         3205  +}
  3034   3206   
  3035   3207   /*
  3036   3208   ** Open and return a new RBU handle. 
  3037   3209   */
  3038   3210   sqlite3rbu *sqlite3rbu_open(
  3039   3211     const char *zTarget, 
  3040   3212     const char *zRbu,
................................................................................
  3073   3245       if( p->rc==SQLITE_OK ){
  3074   3246         pState = rbuLoadState(p);
  3075   3247         assert( pState || p->rc!=SQLITE_OK );
  3076   3248         if( p->rc==SQLITE_OK ){
  3077   3249   
  3078   3250           if( pState->eStage==0 ){ 
  3079   3251             rbuDeleteOalFile(p);
         3252  +          rbuInitPhaseOneSteps(p);
  3080   3253             p->eStage = RBU_STAGE_OAL;
  3081   3254           }else{
  3082   3255             p->eStage = pState->eStage;
         3256  +          p->nPhaseOneStep = pState->nPhaseOneStep;
  3083   3257           }
  3084   3258           p->nProgress = pState->nProgress;
  3085   3259           p->iOalSz = pState->iOalSz;
  3086   3260         }
  3087   3261       }
  3088   3262       assert( p->rc!=SQLITE_OK || p->eStage!=0 );
  3089   3263   
................................................................................
  3238   3412   ** Return the total number of key-value operations (inserts, deletes or 
  3239   3413   ** updates) that have been performed on the target database since the
  3240   3414   ** current RBU update was started.
  3241   3415   */
  3242   3416   sqlite3_int64 sqlite3rbu_progress(sqlite3rbu *pRbu){
  3243   3417     return pRbu->nProgress;
  3244   3418   }
         3419  +
         3420  +/*
         3421  +** Return permyriadage progress indications for the two main stages of
         3422  +** an RBU update.
         3423  +*/
         3424  +void sqlite3rbu_bp_progress(sqlite3rbu *p, int *pnOne, int *pnTwo){
         3425  +  const int MAX_PROGRESS = 10000;
         3426  +  switch( p->eStage ){
         3427  +    case RBU_STAGE_OAL:
         3428  +      if( p->nPhaseOneStep>0 ){
         3429  +        *pnOne = (int)(MAX_PROGRESS * (i64)p->nProgress/(i64)p->nPhaseOneStep);
         3430  +      }else{
         3431  +        *pnOne = -1;
         3432  +      }
         3433  +      *pnTwo = 0;
         3434  +      break;
         3435  +
         3436  +    case RBU_STAGE_MOVE:
         3437  +      *pnOne = MAX_PROGRESS;
         3438  +      *pnTwo = 0;
         3439  +      break;
         3440  +
         3441  +    case RBU_STAGE_CKPT:
         3442  +      *pnOne = MAX_PROGRESS;
         3443  +      *pnTwo = (int)(MAX_PROGRESS * (i64)p->nStep / (i64)p->nFrame);
         3444  +      break;
         3445  +
         3446  +    case RBU_STAGE_DONE:
         3447  +      *pnOne = MAX_PROGRESS;
         3448  +      *pnTwo = MAX_PROGRESS;
         3449  +      break;
         3450  +
         3451  +    default:
         3452  +      assert( 0 );
         3453  +  }
         3454  +}
  3245   3455   
  3246   3456   int sqlite3rbu_savestate(sqlite3rbu *p){
  3247   3457     int rc = p->rc;
  3248   3458     
  3249   3459     if( rc==SQLITE_DONE ) return SQLITE_OK;
  3250   3460   
  3251   3461     assert( p->eStage>=RBU_STAGE_OAL && p->eStage<=RBU_STAGE_DONE );

Changes to ext/rbu/sqlite3rbu.h.

   396    396   /*
   397    397   ** Return the total number of key-value operations (inserts, deletes or 
   398    398   ** updates) that have been performed on the target database since the
   399    399   ** current RBU update was started.
   400    400   */
   401    401   sqlite3_int64 sqlite3rbu_progress(sqlite3rbu *pRbu);
   402    402   
          403  +/*
          404  +** Obtain permyriadage (permyriadage is to 10000 as percentage is to 100) 
          405  +** progress indications for the two stages of an RBU update. This API may
          406  +** be useful for driving GUI progress indicators and similar.
          407  +**
          408  +** An RBU update is divided into two stages:
          409  +**
          410  +**   * Stage 1, in which changes are accumulated in an oal/wal file, and
          411  +**   * Stage 2, in which the contents of the wal file are copied into the
          412  +**     main database.
          413  +**
          414  +** The update is visible to non-RBU clients during stage 2. During stage 1
          415  +** non-RBU reader clients may see the original database.
          416  +**
          417  +** If this API is called during stage 2 of the update, output variable 
          418  +** (*pnOne) is set to 10000 to indicate that stage 1 has finished and (*pnTwo)
          419  +** to a value between 0 and 10000 to indicate the permyriadage progress of
          420  +** stage 2. A value of 5000 indicates that stage 2 is half finished, 
          421  +** 9000 indicates that it is 90% finished, and so on.
          422  +**
          423  +** If this API is called during stage 1 of the update, output variable 
          424  +** (*pnTwo) is set to 0 to indicate that stage 2 has not yet started. The
          425  +** value to which (*pnOne) is set depends on whether or not the RBU 
          426  +** database contains an "rbu_count" table. The rbu_count table, if it 
          427  +** exists, must contain the same columns as the following:
          428  +**
          429  +**   CREATE TABLE rbu_count(tbl TEXT PRIMARY KEY, cnt INTEGER) WITHOUT ROWID;
          430  +**
          431  +** There must be one row in the table for each source (data_xxx) table within
          432  +** the RBU database. The 'tbl' column should contain the name of the source
          433  +** table. The 'cnt' column should contain the number of rows within the
          434  +** source table.
          435  +**
          436  +** If the rbu_count table is present and populated correctly and this
          437  +** API is called during stage 1, the *pnOne output variable is set to the
          438  +** permyriadage progress of the same stage. If the rbu_count table does
          439  +** not exist, then (*pnOne) is set to -1 during stage 1. If the rbu_count
          440  +** table exists but is not correctly populated, the value of the *pnOne
          441  +** output variable during stage 1 is undefined.
          442  +*/
          443  +void sqlite3rbu_bp_progress(sqlite3rbu *pRbu, int *pnOne, int *pnTwo);
          444  +
   403    445   /*
   404    446   ** Create an RBU VFS named zName that accesses the underlying file-system
   405    447   ** via existing VFS zParent. Or, if the zParent parameter is passed NULL, 
   406    448   ** then the new RBU VFS uses the default system VFS to access the file-system.
   407    449   ** The new object is registered as a non-default VFS with SQLite before 
   408    450   ** returning.
   409    451   **

Changes to ext/rbu/test_rbu.c.

    62     62       const char *zUsage;
    63     63     } aCmd[] = {
    64     64       {"step", 2, ""},              /* 0 */
    65     65       {"close", 2, ""},             /* 1 */
    66     66       {"create_rbu_delta", 2, ""},  /* 2 */
    67     67       {"savestate", 2, ""},         /* 3 */
    68     68       {"dbMain_eval", 3, "SQL"},    /* 4 */
           69  +    {"bp_progress", 2, ""},    /* 5 */
    69     70       {0,0,0}
    70     71     };
    71     72     int iCmd;
    72     73   
    73     74     if( objc<2 ){
    74     75       Tcl_WrongNumArgs(interp, 1, objv, "METHOD");
    75     76       return TCL_ERROR;
................................................................................
   131    132         int rc = sqlite3_exec(db, Tcl_GetString(objv[2]), 0, 0, 0);
   132    133         if( rc!=SQLITE_OK ){
   133    134           Tcl_SetObjResult(interp, Tcl_NewStringObj(sqlite3_errmsg(db), -1));
   134    135           ret = TCL_ERROR;
   135    136         }
   136    137         break;
   137    138       }
          139  +
          140  +    case 5: /* bp_progress */ {
          141  +      int one, two;
          142  +      Tcl_Obj *pObj;
          143  +      sqlite3rbu_bp_progress(pRbu, &one, &two);
          144  +
          145  +      pObj = Tcl_NewObj();
          146  +      Tcl_ListObjAppendElement(interp, pObj, Tcl_NewIntObj(one));
          147  +      Tcl_ListObjAppendElement(interp, pObj, Tcl_NewIntObj(two));
          148  +      Tcl_SetObjResult(interp, pObj);
          149  +      break;
          150  +    }
   138    151   
   139    152       default: /* seems unlikely */
   140    153         assert( !"cannot happen" );
   141    154         break;
   142    155     }
   143    156   
   144    157     return ret;

Changes to src/expr.c.

  1279   1279   */
  1280   1280   u32 sqlite3ExprListFlags(const ExprList *pList){
  1281   1281     int i;
  1282   1282     u32 m = 0;
  1283   1283     if( pList ){
  1284   1284       for(i=0; i<pList->nExpr; i++){
  1285   1285          Expr *pExpr = pList->a[i].pExpr;
  1286         -       if( ALWAYS(pExpr) ) m |= pExpr->flags;
         1286  +       assert( pExpr!=0 );
         1287  +       m |= pExpr->flags;
  1287   1288       }
  1288   1289     }
  1289   1290     return m;
  1290   1291   }
  1291   1292   
  1292   1293   /*
  1293   1294   ** These routines are Walker callbacks used to check expressions to
................................................................................
  3544   3545         break;
  3545   3546       }
  3546   3547       case TK_NOT: {
  3547   3548         testcase( jumpIfNull==0 );
  3548   3549         sqlite3ExprIfFalse(pParse, pExpr->pLeft, dest, jumpIfNull);
  3549   3550         break;
  3550   3551       }
         3552  +    case TK_IS:
         3553  +    case TK_ISNOT:
         3554  +      testcase( op==TK_IS );
         3555  +      testcase( op==TK_ISNOT );
         3556  +      op = (op==TK_IS) ? TK_EQ : TK_NE;
         3557  +      jumpIfNull = SQLITE_NULLEQ;
         3558  +      /* Fall thru */
  3551   3559       case TK_LT:
  3552   3560       case TK_LE:
  3553   3561       case TK_GT:
  3554   3562       case TK_GE:
  3555   3563       case TK_NE:
  3556   3564       case TK_EQ: {
  3557   3565         testcase( jumpIfNull==0 );
................................................................................
  3559   3567         r2 = sqlite3ExprCodeTemp(pParse, pExpr->pRight, &regFree2);
  3560   3568         codeCompare(pParse, pExpr->pLeft, pExpr->pRight, op,
  3561   3569                     r1, r2, dest, jumpIfNull);
  3562   3570         assert(TK_LT==OP_Lt); testcase(op==OP_Lt); VdbeCoverageIf(v,op==OP_Lt);
  3563   3571         assert(TK_LE==OP_Le); testcase(op==OP_Le); VdbeCoverageIf(v,op==OP_Le);
  3564   3572         assert(TK_GT==OP_Gt); testcase(op==OP_Gt); VdbeCoverageIf(v,op==OP_Gt);
  3565   3573         assert(TK_GE==OP_Ge); testcase(op==OP_Ge); VdbeCoverageIf(v,op==OP_Ge);
  3566         -      assert(TK_EQ==OP_Eq); testcase(op==OP_Eq); VdbeCoverageIf(v,op==OP_Eq);
  3567         -      assert(TK_NE==OP_Ne); testcase(op==OP_Ne); VdbeCoverageIf(v,op==OP_Ne);
  3568         -      testcase( regFree1==0 );
  3569         -      testcase( regFree2==0 );
  3570         -      break;
  3571         -    }
  3572         -    case TK_IS:
  3573         -    case TK_ISNOT: {
  3574         -      testcase( op==TK_IS );
  3575         -      testcase( op==TK_ISNOT );
  3576         -      r1 = sqlite3ExprCodeTemp(pParse, pExpr->pLeft, &regFree1);
  3577         -      r2 = sqlite3ExprCodeTemp(pParse, pExpr->pRight, &regFree2);
  3578         -      op = (op==TK_IS) ? TK_EQ : TK_NE;
  3579         -      codeCompare(pParse, pExpr->pLeft, pExpr->pRight, op,
  3580         -                  r1, r2, dest, SQLITE_NULLEQ);
  3581         -      VdbeCoverageIf(v, op==TK_EQ);
  3582         -      VdbeCoverageIf(v, op==TK_NE);
         3574  +      assert(TK_EQ==OP_Eq); testcase(op==OP_Eq);
         3575  +      VdbeCoverageIf(v, op==OP_Eq && jumpIfNull==SQLITE_NULLEQ);
         3576  +      VdbeCoverageIf(v, op==OP_Eq && jumpIfNull!=SQLITE_NULLEQ);
         3577  +      assert(TK_NE==OP_Ne); testcase(op==OP_Ne);
         3578  +      VdbeCoverageIf(v, op==OP_Ne && jumpIfNull==SQLITE_NULLEQ);
         3579  +      VdbeCoverageIf(v, op==OP_Ne && jumpIfNull!=SQLITE_NULLEQ);
  3583   3580         testcase( regFree1==0 );
  3584   3581         testcase( regFree2==0 );
  3585   3582         break;
  3586   3583       }
  3587   3584       case TK_ISNULL:
  3588   3585       case TK_NOTNULL: {
  3589   3586         assert( TK_ISNULL==OP_IsNull );   testcase( op==TK_ISNULL );
................................................................................
  3700   3697         break;
  3701   3698       }
  3702   3699       case TK_NOT: {
  3703   3700         testcase( jumpIfNull==0 );
  3704   3701         sqlite3ExprIfTrue(pParse, pExpr->pLeft, dest, jumpIfNull);
  3705   3702         break;
  3706   3703       }
         3704  +    case TK_IS:
         3705  +    case TK_ISNOT:
         3706  +      testcase( pExpr->op==TK_IS );
         3707  +      testcase( pExpr->op==TK_ISNOT );
         3708  +      op = (pExpr->op==TK_IS) ? TK_NE : TK_EQ;
         3709  +      jumpIfNull = SQLITE_NULLEQ;
         3710  +      /* Fall thru */
  3707   3711       case TK_LT:
  3708   3712       case TK_LE:
  3709   3713       case TK_GT:
  3710   3714       case TK_GE:
  3711   3715       case TK_NE:
  3712   3716       case TK_EQ: {
  3713   3717         testcase( jumpIfNull==0 );
................................................................................
  3715   3719         r2 = sqlite3ExprCodeTemp(pParse, pExpr->pRight, &regFree2);
  3716   3720         codeCompare(pParse, pExpr->pLeft, pExpr->pRight, op,
  3717   3721                     r1, r2, dest, jumpIfNull);
  3718   3722         assert(TK_LT==OP_Lt); testcase(op==OP_Lt); VdbeCoverageIf(v,op==OP_Lt);
  3719   3723         assert(TK_LE==OP_Le); testcase(op==OP_Le); VdbeCoverageIf(v,op==OP_Le);
  3720   3724         assert(TK_GT==OP_Gt); testcase(op==OP_Gt); VdbeCoverageIf(v,op==OP_Gt);
  3721   3725         assert(TK_GE==OP_Ge); testcase(op==OP_Ge); VdbeCoverageIf(v,op==OP_Ge);
  3722         -      assert(TK_EQ==OP_Eq); testcase(op==OP_Eq); VdbeCoverageIf(v,op==OP_Eq);
  3723         -      assert(TK_NE==OP_Ne); testcase(op==OP_Ne); VdbeCoverageIf(v,op==OP_Ne);
  3724         -      testcase( regFree1==0 );
  3725         -      testcase( regFree2==0 );
  3726         -      break;
  3727         -    }
  3728         -    case TK_IS:
  3729         -    case TK_ISNOT: {
  3730         -      testcase( pExpr->op==TK_IS );
  3731         -      testcase( pExpr->op==TK_ISNOT );
  3732         -      r1 = sqlite3ExprCodeTemp(pParse, pExpr->pLeft, &regFree1);
  3733         -      r2 = sqlite3ExprCodeTemp(pParse, pExpr->pRight, &regFree2);
  3734         -      op = (pExpr->op==TK_IS) ? TK_NE : TK_EQ;
  3735         -      codeCompare(pParse, pExpr->pLeft, pExpr->pRight, op,
  3736         -                  r1, r2, dest, SQLITE_NULLEQ);
  3737         -      VdbeCoverageIf(v, op==TK_EQ);
  3738         -      VdbeCoverageIf(v, op==TK_NE);
         3726  +      assert(TK_EQ==OP_Eq); testcase(op==OP_Eq);
         3727  +      VdbeCoverageIf(v, op==OP_Eq && jumpIfNull!=SQLITE_NULLEQ);
         3728  +      VdbeCoverageIf(v, op==OP_Eq && jumpIfNull==SQLITE_NULLEQ);
         3729  +      assert(TK_NE==OP_Ne); testcase(op==OP_Ne);
         3730  +      VdbeCoverageIf(v, op==OP_Ne && jumpIfNull!=SQLITE_NULLEQ);
         3731  +      VdbeCoverageIf(v, op==OP_Ne && jumpIfNull==SQLITE_NULLEQ);
  3739   3732         testcase( regFree1==0 );
  3740   3733         testcase( regFree2==0 );
  3741   3734         break;
  3742   3735       }
  3743   3736       case TK_ISNULL:
  3744   3737       case TK_NOTNULL: {
  3745   3738         r1 = sqlite3ExprCodeTemp(pParse, pExpr->pLeft, &regFree1);
................................................................................
  4254   4247   /*
  4255   4248   ** Mark all temporary registers as being unavailable for reuse.
  4256   4249   */
  4257   4250   void sqlite3ClearTempRegCache(Parse *pParse){
  4258   4251     pParse->nTempReg = 0;
  4259   4252     pParse->nRangeReg = 0;
  4260   4253   }
         4254  +
         4255  +/*
         4256  +** Validate that no temporary register falls within the range of
         4257  +** iFirst..iLast, inclusive.  This routine is only call from within assert()
         4258  +** statements.
         4259  +*/
         4260  +#ifdef SQLITE_DEBUG
         4261  +int sqlite3NoTempsInRange(Parse *pParse, int iFirst, int iLast){
         4262  +  int i;
         4263  +  if( pParse->nRangeReg>0
         4264  +   && pParse->iRangeReg+pParse->nRangeReg<iLast
         4265  +   && pParse->iRangeReg>=iFirst
         4266  +  ){
         4267  +     return 0;
         4268  +  }
         4269  +  for(i=0; i<pParse->nTempReg; i++){
         4270  +    if( pParse->aTempReg[i]>=iFirst && pParse->aTempReg[i]<=iLast ){
         4271  +      return 0;
         4272  +    }
         4273  +  }
         4274  +  return 1;
         4275  +}
         4276  +#endif /* SQLITE_DEBUG */

Changes to src/loadext.c.

   410    410     sqlite3_bind_zeroblob64,
   411    411     /* Version 3.9.0 and later */
   412    412     sqlite3_value_subtype,
   413    413     sqlite3_result_subtype,
   414    414     /* Version 3.10.0 and later */
   415    415     sqlite3_status64,
   416    416     sqlite3_strlike,
   417         -  sqlite3_db_cacheflush
          417  +  sqlite3_db_cacheflush,
          418  +  /* Version 3.12.0 and later */
          419  +  sqlite3_system_errno
   418    420   };
   419    421   
   420    422   /*
   421    423   ** Attempt to load an SQLite extension library contained in the file
   422    424   ** zFile.  The entry point is zProc.  zProc may be 0 in which case a
   423    425   ** default entry point name (sqlite3_extension_init) is used.  Use
   424    426   ** of the default name is recommended.

Changes to src/main.c.

  2257   2257       return SQLITE_MISUSE_BKPT;
  2258   2258     }
  2259   2259     if( !db || db->mallocFailed ){
  2260   2260       return SQLITE_NOMEM_BKPT;
  2261   2261     }
  2262   2262     return db->errCode;
  2263   2263   }
         2264  +int sqlite3_system_errno(sqlite3 *db){
         2265  +  return db ? db->iSysErrno : 0;
         2266  +}  
  2264   2267   
  2265   2268   /*
  2266   2269   ** Return a string that describes the kind of error specified in the
  2267   2270   ** argument.  For now, this simply calls the internal sqlite3ErrStr()
  2268   2271   ** function.
  2269   2272   */
  2270   2273   const char *sqlite3_errstr(int rc){

Changes to src/os.c.

   258    258   #endif /* SQLITE_OMIT_LOAD_EXTENSION */
   259    259   int sqlite3OsRandomness(sqlite3_vfs *pVfs, int nByte, char *zBufOut){
   260    260     return pVfs->xRandomness(pVfs, nByte, zBufOut);
   261    261   }
   262    262   int sqlite3OsSleep(sqlite3_vfs *pVfs, int nMicro){
   263    263     return pVfs->xSleep(pVfs, nMicro);
   264    264   }
          265  +int sqlite3OsGetLastError(sqlite3_vfs *pVfs){
          266  +  return pVfs->xGetLastError ? pVfs->xGetLastError(pVfs, 0, 0) : 0;
          267  +}
   265    268   int sqlite3OsCurrentTimeInt64(sqlite3_vfs *pVfs, sqlite3_int64 *pTimeOut){
   266    269     int rc;
   267    270     /* IMPLEMENTATION-OF: R-49045-42493 SQLite will use the xCurrentTimeInt64()
   268    271     ** method to get the current date and time if that method is available
   269    272     ** (if iVersion is 2 or greater and the function pointer is not NULL) and
   270    273     ** will fall back to xCurrentTime() if xCurrentTimeInt64() is
   271    274     ** unavailable.

Changes to src/os.h.

   193    193   void *sqlite3OsDlOpen(sqlite3_vfs *, const char *);
   194    194   void sqlite3OsDlError(sqlite3_vfs *, int, char *);
   195    195   void (*sqlite3OsDlSym(sqlite3_vfs *, void *, const char *))(void);
   196    196   void sqlite3OsDlClose(sqlite3_vfs *, void *);
   197    197   #endif /* SQLITE_OMIT_LOAD_EXTENSION */
   198    198   int sqlite3OsRandomness(sqlite3_vfs *, int, char *);
   199    199   int sqlite3OsSleep(sqlite3_vfs *, int);
          200  +int sqlite3OsGetLastError(sqlite3_vfs*);
   200    201   int sqlite3OsCurrentTimeInt64(sqlite3_vfs *, sqlite3_int64*);
   201    202   
   202    203   /*
   203    204   ** Convenience functions for opening and closing files using 
   204    205   ** sqlite3_malloc() to obtain space for the file-handle structure.
   205    206   */
   206    207   int sqlite3OsOpenMalloc(sqlite3_vfs *, const char *, sqlite3_file **, int,int*);
   207    208   int sqlite3OsCloseFree(sqlite3_file *);
   208    209   
   209    210   #endif /* _SQLITE_OS_H_ */

Changes to src/os_unix.c.

  1353   1353   ** (3) The file has not been renamed or unlinked
  1354   1354   **
  1355   1355   ** Issue sqlite3_log(SQLITE_WARNING,...) messages if anything is not right.
  1356   1356   */
  1357   1357   static void verifyDbFile(unixFile *pFile){
  1358   1358     struct stat buf;
  1359   1359     int rc;
         1360  +
         1361  +  /* These verifications occurs for the main database only */
         1362  +  if( pFile->ctrlFlags & UNIXFILE_NOLOCK ) return;
         1363  +
  1360   1364     rc = osFstat(pFile->h, &buf);
  1361   1365     if( rc!=0 ){
  1362   1366       sqlite3_log(SQLITE_WARNING, "cannot fstat db file %s", pFile->zPath);
  1363   1367       return;
  1364   1368     }
  1365   1369     if( buf.st_nlink==0 && (pFile->ctrlFlags & UNIXFILE_DELETE)==0 ){
  1366   1370       sqlite3_log(SQLITE_WARNING, "file unlinked while open: %s", pFile->zPath);
................................................................................
  5799   5803   #endif
  5800   5804     }
  5801   5805   #if SQLITE_ENABLE_LOCKING_STYLE
  5802   5806     else{
  5803   5807       p->openFlags = openFlags;
  5804   5808     }
  5805   5809   #endif
  5806         -
  5807         -  noLock = eType!=SQLITE_OPEN_MAIN_DB;
  5808         -
  5809   5810     
  5810   5811   #if defined(__APPLE__) || SQLITE_ENABLE_LOCKING_STYLE
  5811   5812     if( fstatfs(fd, &fsInfo) == -1 ){
  5812   5813       storeLastErrno(p, errno);
  5813   5814       robust_close(p, fd, __LINE__);
  5814   5815       return SQLITE_IOERR_ACCESS;
  5815   5816     }
................................................................................
  5820   5821       ((unixFile*)pFile)->fsFlags |= SQLITE_FSFLAGS_IS_MSDOS;
  5821   5822     }
  5822   5823   #endif
  5823   5824   
  5824   5825     /* Set up appropriate ctrlFlags */
  5825   5826     if( isDelete )                ctrlFlags |= UNIXFILE_DELETE;
  5826   5827     if( isReadonly )              ctrlFlags |= UNIXFILE_RDONLY;
         5828  +  noLock = eType!=SQLITE_OPEN_MAIN_DB;
  5827   5829     if( noLock )                  ctrlFlags |= UNIXFILE_NOLOCK;
  5828   5830     if( syncDir )                 ctrlFlags |= UNIXFILE_DIRSYNC;
  5829   5831     if( flags & SQLITE_OPEN_URI ) ctrlFlags |= UNIXFILE_URI;
  5830   5832   
  5831   5833   #if SQLITE_ENABLE_LOCKING_STYLE
  5832   5834   #if SQLITE_PREFER_PROXY_LOCKING
  5833   5835     isAutoProxy = 1;
................................................................................
  6258   6260     *prNow = i/86400000.0;
  6259   6261     return rc;
  6260   6262   }
  6261   6263   #else
  6262   6264   # define unixCurrentTime 0
  6263   6265   #endif
  6264   6266   
  6265         -#ifndef SQLITE_OMIT_DEPRECATED
  6266   6267   /*
  6267         -** We added the xGetLastError() method with the intention of providing
  6268         -** better low-level error messages when operating-system problems come up
  6269         -** during SQLite operation.  But so far, none of that has been implemented
  6270         -** in the core.  So this routine is never called.  For now, it is merely
  6271         -** a place-holder.
         6268  +** The xGetLastError() method is designed to return a better
         6269  +** low-level error message when operating-system problems come up
         6270  +** during SQLite operation.  Only the integer return code is currently
         6271  +** used.
  6272   6272   */
  6273   6273   static int unixGetLastError(sqlite3_vfs *NotUsed, int NotUsed2, char *NotUsed3){
  6274   6274     UNUSED_PARAMETER(NotUsed);
  6275   6275     UNUSED_PARAMETER(NotUsed2);
  6276   6276     UNUSED_PARAMETER(NotUsed3);
  6277         -  return 0;
         6277  +  return errno;
  6278   6278   }
  6279         -#else
  6280         -# define unixGetLastError 0
  6281         -#endif
  6282   6279   
  6283   6280   
  6284   6281   /*
  6285   6282   ************************ End of sqlite3_vfs methods ***************************
  6286   6283   ******************************************************************************/
  6287   6284   
  6288   6285   /******************************************************************************

Changes to src/os_win.c.

  5580   5580   **   }
  5581   5581   **
  5582   5582   ** However if an error message is supplied, it will be incorporated
  5583   5583   ** by sqlite into the error message available to the user using
  5584   5584   ** sqlite3_errmsg(), possibly making IO errors easier to debug.
  5585   5585   */
  5586   5586   static int winGetLastError(sqlite3_vfs *pVfs, int nBuf, char *zBuf){
         5587  +  DWORD e = osGetLastError();
  5587   5588     UNUSED_PARAMETER(pVfs);
  5588         -  return winGetLastErrorMsg(osGetLastError(), nBuf, zBuf);
         5589  +  if( nBuf>0 ) winGetLastErrorMsg(e, nBuf, zBuf);
         5590  +  return e;
  5589   5591   }
  5590   5592   
  5591   5593   /*
  5592   5594   ** Initialize and deinitialize the operating system interface.
  5593   5595   */
  5594   5596   int sqlite3_os_init(void){
  5595   5597     static sqlite3_vfs winVfs = {

Changes to src/pcache1.c.

   344    344     return p;
   345    345   }
   346    346   
   347    347   /*
   348    348   ** Free an allocated buffer obtained from pcache1Alloc().
   349    349   */
   350    350   static void pcache1Free(void *p){
   351         -  int nFreed = 0;
   352    351     if( p==0 ) return;
   353    352     if( SQLITE_WITHIN(p, pcache1.pStart, pcache1.pEnd) ){
   354    353       PgFreeslot *pSlot;
   355    354       sqlite3_mutex_enter(pcache1.mutex);
   356    355       sqlite3StatusDown(SQLITE_STATUS_PAGECACHE_USED, 1);
   357    356       pSlot = (PgFreeslot*)p;
   358    357       pSlot->pNext = pcache1.pFree;
................................................................................
   361    360       pcache1.bUnderPressure = pcache1.nFreeSlot<pcache1.nReserve;
   362    361       assert( pcache1.nFreeSlot<=pcache1.nSlot );
   363    362       sqlite3_mutex_leave(pcache1.mutex);
   364    363     }else{
   365    364       assert( sqlite3MemdebugHasType(p, MEMTYPE_PCACHE) );
   366    365       sqlite3MemdebugSetType(p, MEMTYPE_HEAP);
   367    366   #ifndef SQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS
   368         -    nFreed = sqlite3MallocSize(p);
   369         -    sqlite3_mutex_enter(pcache1.mutex);
   370         -    sqlite3StatusDown(SQLITE_STATUS_PAGECACHE_OVERFLOW, nFreed);
   371         -    sqlite3_mutex_leave(pcache1.mutex);
          367  +    {
          368  +      int nFreed = 0;
          369  +      nFreed = sqlite3MallocSize(p);
          370  +      sqlite3_mutex_enter(pcache1.mutex);
          371  +      sqlite3StatusDown(SQLITE_STATUS_PAGECACHE_OVERFLOW, nFreed);
          372  +      sqlite3_mutex_leave(pcache1.mutex);
          373  +    }
   372    374   #endif
   373    375       sqlite3_free(p);
   374    376     }
   375    377   }
   376    378   
   377    379   #ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT
   378    380   /*

Changes to src/pragma.c.

  1437   1437   
  1438   1438       /* Do an integrity check on each database file */
  1439   1439       for(i=0; i<db->nDb; i++){
  1440   1440         HashElem *x;
  1441   1441         Hash *pTbls;
  1442   1442         int *aRoot;
  1443   1443         int cnt = 0;
         1444  +      int mxIdx = 0;
         1445  +      int nIdx;
  1444   1446   
  1445   1447         if( OMIT_TEMPDB && i==1 ) continue;
  1446   1448         if( iDb>=0 && i!=iDb ) continue;
  1447   1449   
  1448   1450         sqlite3CodeVerifySchema(pParse, i);
  1449   1451         addr = sqlite3VdbeAddOp1(v, OP_IfPos, 1); /* Halt if out of errors */
  1450   1452         VdbeCoverage(v);
................................................................................
  1458   1460         */
  1459   1461         assert( sqlite3SchemaMutexHeld(db, i, 0) );
  1460   1462         pTbls = &db->aDb[i].pSchema->tblHash;
  1461   1463         for(cnt=0, x=sqliteHashFirst(pTbls); x; x=sqliteHashNext(x)){
  1462   1464           Table *pTab = sqliteHashData(x);
  1463   1465           Index *pIdx;
  1464   1466           if( HasRowid(pTab) ) cnt++;
  1465         -        for(pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext){ cnt++; }
         1467  +        for(nIdx=0, pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext, nIdx++){ cnt++; }
         1468  +        if( nIdx>mxIdx ) mxIdx = nIdx;
  1466   1469         }
  1467   1470         aRoot = sqlite3DbMallocRawNN(db, sizeof(int)*(cnt+1));
  1468   1471         if( aRoot==0 ) break;
  1469   1472         for(cnt=0, x=sqliteHashFirst(pTbls); x; x=sqliteHashNext(x)){
  1470   1473           Table *pTab = sqliteHashData(x);
  1471   1474           Index *pIdx;
  1472   1475           if( HasRowid(pTab) ) aRoot[cnt++] = pTab->tnum;
................................................................................
  1473   1476           for(pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext){
  1474   1477             aRoot[cnt++] = pIdx->tnum;
  1475   1478           }
  1476   1479         }
  1477   1480         aRoot[cnt] = 0;
  1478   1481   
  1479   1482         /* Make sure sufficient number of registers have been allocated */
  1480         -      pParse->nMem = MAX( pParse->nMem, 14 );
         1483  +      pParse->nMem = MAX( pParse->nMem, 8+mxIdx );
  1481   1484   
  1482   1485         /* Do the b-tree integrity checks */
  1483   1486         sqlite3VdbeAddOp4(v, OP_IntegrityCk, 2, cnt, 1, (char*)aRoot,P4_INTARRAY);
  1484   1487         sqlite3VdbeChangeP5(v, (u8)i);
  1485   1488         addr = sqlite3VdbeAddOp1(v, OP_IsNull, 2); VdbeCoverage(v);
  1486   1489         sqlite3VdbeAddOp4(v, OP_String8, 0, 3, 0,
  1487   1490            sqlite3MPrintf(db, "*** in database %s ***\n", db->aDb[i].zName),
................................................................................
  1510   1513           sqlite3ExprCacheClear(pParse);
  1511   1514           sqlite3OpenTableAndIndices(pParse, pTab, OP_OpenRead, 0,
  1512   1515                                      1, 0, &iDataCur, &iIdxCur);
  1513   1516           sqlite3VdbeAddOp2(v, OP_Integer, 0, 7);
  1514   1517           for(j=0, pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext, j++){
  1515   1518             sqlite3VdbeAddOp2(v, OP_Integer, 0, 8+j); /* index entries counter */
  1516   1519           }
  1517         -        pParse->nMem = MAX(pParse->nMem, 8+j);
         1520  +        assert( pParse->nMem>=8+j );
         1521  +        assert( sqlite3NoTempsInRange(pParse,1,7+j) );
  1518   1522           sqlite3VdbeAddOp2(v, OP_Rewind, iDataCur, 0); VdbeCoverage(v);
  1519   1523           loopTop = sqlite3VdbeAddOp2(v, OP_AddImm, 7, 1);
  1520   1524           /* Verify that all NOT NULL columns really are NOT NULL */
  1521   1525           for(j=0; j<pTab->nCol; j++){
  1522   1526             char *zErr;
  1523   1527             int jmp2, jmp3;
  1524   1528             if( j==pTab->iPKey ) continue;
................................................................................
  1702   1706     /*
  1703   1707     **   PRAGMA [schema.]schema_version
  1704   1708     **   PRAGMA [schema.]schema_version = <integer>
  1705   1709     **
  1706   1710     **   PRAGMA [schema.]user_version
  1707   1711     **   PRAGMA [schema.]user_version = <integer>
  1708   1712     **
  1709         -  **   PRAGMA [schema.]freelist_count = <integer>
         1713  +  **   PRAGMA [schema.]freelist_count
         1714  +  **
         1715  +  **   PRAGMA [schema.]data_version
  1710   1716     **
  1711   1717     **   PRAGMA [schema.]application_id
  1712   1718     **   PRAGMA [schema.]application_id = <integer>
  1713   1719     **
  1714   1720     ** The pragma's schema_version and user_version are used to set or get
  1715   1721     ** the value of the schema-version and user-version, respectively. Both
  1716   1722     ** the schema-version and the user-version are 32-bit signed integers
................................................................................
  1758   1764         aOp = sqlite3VdbeAddOpList(v, ArraySize(readCookie),readCookie,0);
  1759   1765         if( ONLY_IF_REALLOC_STRESS(aOp==0) ) break;
  1760   1766         aOp[0].p1 = iDb;
  1761   1767         aOp[1].p1 = iDb;
  1762   1768         aOp[1].p3 = iCookie;
  1763   1769         sqlite3VdbeSetNumCols(v, 1);
  1764   1770         sqlite3VdbeSetColName(v, 0, COLNAME_NAME, zLeft, SQLITE_TRANSIENT);
         1771  +      sqlite3VdbeReusable(v);
  1765   1772       }
  1766   1773     }
  1767   1774     break;
  1768   1775   #endif /* SQLITE_OMIT_SCHEMA_VERSION_PRAGMAS */
  1769   1776   
  1770   1777   #ifndef SQLITE_OMIT_COMPILEOPTION_DIAGS
  1771   1778     /*
................................................................................
  1779   1786       const char *zOpt;
  1780   1787       pParse->nMem = 1;
  1781   1788       setOneColumnName(v, "compile_option");
  1782   1789       while( (zOpt = sqlite3_compileoption_get(i++))!=0 ){
  1783   1790         sqlite3VdbeLoadString(v, 1, zOpt);
  1784   1791         sqlite3VdbeAddOp2(v, OP_ResultRow, 1, 1);
  1785   1792       }
         1793  +    sqlite3VdbeReusable(v);
  1786   1794     }
  1787   1795     break;
  1788   1796   #endif /* SQLITE_OMIT_COMPILEOPTION_DIAGS */
  1789   1797   
  1790   1798   #ifndef SQLITE_OMIT_WAL
  1791   1799     /*
  1792   1800     **   PRAGMA [schema.]wal_checkpoint = passive|full|restart|truncate

Changes to src/sqlite.h.in.

  8006   8006     void*
  8007   8007   );
  8008   8008   SQLITE_EXPERIMENTAL int sqlite3_preupdate_old(sqlite3 *, int, sqlite3_value **);
  8009   8009   SQLITE_EXPERIMENTAL int sqlite3_preupdate_count(sqlite3 *);
  8010   8010   SQLITE_EXPERIMENTAL int sqlite3_preupdate_depth(sqlite3 *);
  8011   8011   SQLITE_EXPERIMENTAL int sqlite3_preupdate_new(sqlite3 *, int, sqlite3_value **);
  8012   8012   
         8013  +/*
         8014  +** CAPI3REF: Low-level system error code
         8015  +**
         8016  +** ^Attempt to return the underlying operating system error code or error
         8017  +** number that caused the most reason I/O error or failure to open a file.
         8018  +** The return value is OS-dependent.  For example, on unix systems, after
         8019  +** [sqlite3_open_v2()] returns [SQLITE_CANTOPEN], this interface could be
         8020  +** called to get back the underlying "errno" that caused the problem, such
         8021  +** as ENOSPC, EAUTH, EISDIR, and so forth.  
         8022  +*/
         8023  +int sqlite3_system_errno(sqlite3*);
         8024  +
  8013   8025   /*
  8014   8026   ** CAPI3REF: Database Snapshot
  8015   8027   ** KEYWORDS: {snapshot}
  8016   8028   ** EXPERIMENTAL
  8017   8029   **
  8018   8030   ** An instance of the snapshot object records the state of a [WAL mode]
  8019   8031   ** database for some specific point in history.

Changes to src/sqlite3ext.h.

   275    275     /* Version 3.9.0 and later */
   276    276     unsigned int (*value_subtype)(sqlite3_value*);
   277    277     void (*result_subtype)(sqlite3_context*,unsigned int);
   278    278     /* Version 3.10.0 and later */
   279    279     int (*status64)(int,sqlite3_int64*,sqlite3_int64*,int);
   280    280     int (*strlike)(const char*,const char*,unsigned int);
   281    281     int (*db_cacheflush)(sqlite3*);
          282  +  /* Version 3.12.0 and later */
          283  +  int (*system_errno)(sqlite3*);
   282    284   };
   283    285   
   284    286   /*
   285    287   ** The following macros redefine the API routines so that they are
   286    288   ** redirected through the global sqlite3_api structure.
   287    289   **
   288    290   ** This header file is also used by the loadext.c source file
................................................................................
   518    520   /* Version 3.9.0 and later */
   519    521   #define sqlite3_value_subtype          sqlite3_api->value_subtype
   520    522   #define sqlite3_result_subtype         sqlite3_api->result_subtype
   521    523   /* Version 3.10.0 and later */
   522    524   #define sqlite3_status64               sqlite3_api->status64
   523    525   #define sqlite3_strlike                sqlite3_api->strlike
   524    526   #define sqlite3_db_cacheflush          sqlite3_api->db_cacheflush
          527  +/* Version 3.12.0 and later */
          528  +#define sqlite3_system_errno           sqlite3_api->system_errno
   525    529   #endif /* !defined(SQLITE_CORE) && !defined(SQLITE_OMIT_LOAD_EXTENSION) */
   526    530   
   527    531   #if !defined(SQLITE_CORE) && !defined(SQLITE_OMIT_LOAD_EXTENSION)
   528    532     /* This case when the file really is being compiled as a loadable 
   529    533     ** extension */
   530    534   # define SQLITE_EXTENSION_INIT1     const sqlite3_api_routines *sqlite3_api=0;
   531    535   # define SQLITE_EXTENSION_INIT2(v)  sqlite3_api=v;

Changes to src/sqliteInt.h.

  1217   1217     int nDb;                      /* Number of backends currently in use */
  1218   1218     int flags;                    /* Miscellaneous flags. See below */
  1219   1219     i64 lastRowid;                /* ROWID of most recent insert (see above) */
  1220   1220     i64 szMmap;                   /* Default mmap_size setting */
  1221   1221     unsigned int openFlags;       /* Flags passed to sqlite3_vfs.xOpen() */
  1222   1222     int errCode;                  /* Most recent error code (SQLITE_*) */
  1223   1223     int errMask;                  /* & result codes with this before returning */
         1224  +  int iSysErrno;                /* Errno value from last system error */
  1224   1225     u16 dbOptFlags;               /* Flags to enable/disable optimizations */
  1225   1226     u8 enc;                       /* Text encoding */
  1226   1227     u8 autoCommit;                /* The auto-commit flag. */
  1227   1228     u8 temp_store;                /* 1: file 2: memory 0: default */
  1228   1229     u8 mallocFailed;              /* True if we have seen a malloc failure */
  1229   1230     u8 bBenignMalloc;             /* Do not require OOMs if true */
  1230   1231     u8 dfltLockMode;              /* Default locking-mode for attached dbs */
................................................................................
  3427   3428   int sqlite3RunParser(Parse*, const char*, char **);
  3428   3429   void sqlite3FinishCoding(Parse*);
  3429   3430   int sqlite3GetTempReg(Parse*);
  3430   3431   void sqlite3ReleaseTempReg(Parse*,int);
  3431   3432   int sqlite3GetTempRange(Parse*,int);
  3432   3433   void sqlite3ReleaseTempRange(Parse*,int,int);
  3433   3434   void sqlite3ClearTempRegCache(Parse*);
         3435  +#ifdef SQLITE_DEBUG
         3436  +int sqlite3NoTempsInRange(Parse*,int,int);
         3437  +#endif
  3434   3438   Expr *sqlite3ExprAlloc(sqlite3*,int,const Token*,int);
  3435   3439   Expr *sqlite3Expr(sqlite3*,int,const char*);
  3436   3440   void sqlite3ExprAttachSubtrees(sqlite3*,Expr*,Expr*,Expr*);
  3437   3441   Expr *sqlite3PExpr(Parse*, int, Expr*, Expr*, const Token*);
  3438   3442   Expr *sqlite3ExprAnd(sqlite3*,Expr*, Expr*);
  3439   3443   Expr *sqlite3ExprFunction(Parse*,ExprList*, Token*);
  3440   3444   void sqlite3ExprAssignVarNumber(Parse*, Expr*);
................................................................................
  3771   3775   char sqlite3CompareAffinity(Expr *pExpr, char aff2);
  3772   3776   int sqlite3IndexAffinityOk(Expr *pExpr, char idx_affinity);
  3773   3777   char sqlite3ExprAffinity(Expr *pExpr);
  3774   3778   int sqlite3Atoi64(const char*, i64*, int, u8);
  3775   3779   int sqlite3DecOrHexToI64(const char*, i64*);
  3776   3780   void sqlite3ErrorWithMsg(sqlite3*, int, const char*,...);
  3777   3781   void sqlite3Error(sqlite3*,int);
         3782  +void sqlite3SystemError(sqlite3*,int);
  3778   3783   void *sqlite3HexToBlob(sqlite3*, const char *z, int n);
  3779   3784   u8 sqlite3HexToInt(int h);
  3780   3785   int sqlite3TwoPartName(Parse *, Token *, Token *, Token **);
  3781   3786   
  3782   3787   #if defined(SQLITE_NEED_ERR_NAME)
  3783   3788   const char *sqlite3ErrName(int);
  3784   3789   #endif

Changes to src/test1.c.

  1927   1927     /* Call the underlying C function. If an error occurs, set rc to 
  1928   1928     ** TCL_ERROR and load any error string into the interpreter. If no 
  1929   1929     ** error occurs, set rc to TCL_OK.
  1930   1930     */
  1931   1931   #ifdef SQLITE_OMIT_LOAD_EXTENSION
  1932   1932     rc = SQLITE_ERROR;
  1933   1933     zErr = sqlite3_mprintf("this build omits sqlite3_load_extension()");
         1934  +  (void)zProc;
         1935  +  (void)zFile;
  1934   1936   #else
  1935   1937     rc = sqlite3_load_extension(db, zFile, zProc, &zErr);
  1936   1938   #endif
  1937   1939     if( rc!=SQLITE_OK ){
  1938   1940       Tcl_SetResult(interp, zErr ? zErr : "", TCL_VOLATILE);
  1939   1941       rc = TCL_ERROR;
  1940   1942     }else{
................................................................................
  4842   4844       Tcl_SetResult(interp, (char *)sqlite3ErrStr(rc), TCL_STATIC);
  4843   4845       return TCL_ERROR;
  4844   4846     }
  4845   4847   
  4846   4848     Tcl_ResetResult(interp);
  4847   4849     return TCL_OK;
  4848   4850   }
         4851  +
         4852  +/*
         4853  +** Usage:  sqlite3_system_errno DB
         4854  +**
         4855  +** Return the low-level system errno value.
         4856  +*/
         4857  +static int test_system_errno(
         4858  +  void * clientData,
         4859  +  Tcl_Interp *interp,
         4860  +  int objc,
         4861  +  Tcl_Obj *CONST objv[]
         4862  +){
         4863  +  sqlite3 *db;
         4864  +  int iErrno;
         4865  +  if( objc!=2 ){
         4866  +    Tcl_WrongNumArgs(interp, 1, objv, "DB");
         4867  +    return TCL_ERROR;
         4868  +  }
         4869  +  if( getDbPointer(interp, Tcl_GetString(objv[1]), &db) ) return TCL_ERROR;
         4870  +  iErrno = sqlite3_system_errno(db);
         4871  +  Tcl_SetObjResult(interp, Tcl_NewIntObj(iErrno));
         4872  +  return TCL_OK;
         4873  +}
  4849   4874   
  4850   4875   /*
  4851   4876   ** Usage:  sqlite3_db_filename DB DBNAME
  4852   4877   **
  4853   4878   ** Return the name of a file associated with a database.
  4854   4879   */
  4855   4880   static int test_db_filename(
................................................................................
  7080   7105        { "sqlite3_stmt_readonly",         test_stmt_readonly ,0 },
  7081   7106        { "sqlite3_stmt_busy",             test_stmt_busy     ,0 },
  7082   7107        { "uses_stmt_journal",             uses_stmt_journal ,0 },
  7083   7108   
  7084   7109        { "sqlite3_release_memory",        test_release_memory,     0},
  7085   7110        { "sqlite3_db_release_memory",     test_db_release_memory,  0},
  7086   7111        { "sqlite3_db_cacheflush",         test_db_cacheflush,      0},
         7112  +     { "sqlite3_system_errno",          test_system_errno,       0},
  7087   7113        { "sqlite3_db_filename",           test_db_filename,        0},
  7088   7114        { "sqlite3_db_readonly",           test_db_readonly,        0},
  7089   7115        { "sqlite3_soft_heap_limit",       test_soft_heap_limit,    0},
  7090   7116        { "sqlite3_thread_cleanup",        test_thread_cleanup,     0},
  7091   7117        { "sqlite3_pager_refcounts",       test_pager_refcounts,    0},
  7092   7118   
  7093   7119        { "sqlite3_load_extension",        test_load_extension,     0},

Changes to src/test_config.c.

    74     74   #endif
    75     75   
    76     76   #ifdef SQLITE_DEBUG
    77     77     Tcl_SetVar2(interp, "sqlite_options", "debug", "1", TCL_GLOBAL_ONLY);
    78     78   #else
    79     79     Tcl_SetVar2(interp, "sqlite_options", "debug", "0", TCL_GLOBAL_ONLY);
    80     80   #endif
           81  +
           82  +#ifdef SQLITE_DEFAULT_CKPTFULLFSYNC
           83  +  Tcl_SetVar2(interp, "sqlite_options", "default_ckptfullfsync", 
           84  +              SQLITE_DEFAULT_CKPTFULLFSYNC ? "1" : "0", TCL_GLOBAL_ONLY);
           85  +#else
           86  +  Tcl_SetVar2(interp, "sqlite_options", "default_ckptfullfsync", "0", TCL_GLOBAL_ONLY);
           87  +#endif
    81     88   
    82     89   #ifdef SQLITE_DIRECT_OVERFLOW_READ
    83     90     Tcl_SetVar2(interp, "sqlite_options", "direct_read", "1", TCL_GLOBAL_ONLY);
    84     91   #else
    85     92     Tcl_SetVar2(interp, "sqlite_options", "direct_read", "0", TCL_GLOBAL_ONLY);
    86     93   #endif
    87     94   
................................................................................
    92     99   #endif
    93    100   
    94    101   #ifdef SQLITE_DISABLE_LFS
    95    102     Tcl_SetVar2(interp, "sqlite_options", "lfs", "0", TCL_GLOBAL_ONLY);
    96    103   #else
    97    104     Tcl_SetVar2(interp, "sqlite_options", "lfs", "1", TCL_GLOBAL_ONLY);
    98    105   #endif
          106  +
          107  +#ifdef SQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS
          108  +  Tcl_SetVar2(interp, "sqlite_options", "pagecache_overflow_stats","0",TCL_GLOBAL_ONLY);
          109  +#else
          110  +  Tcl_SetVar2(interp, "sqlite_options", "pagecache_overflow_stats","1",TCL_GLOBAL_ONLY);
          111  +#endif
    99    112   
   100    113   #if SQLITE_MAX_MMAP_SIZE>0
   101    114     Tcl_SetVar2(interp, "sqlite_options", "mmap", "1", TCL_GLOBAL_ONLY);
   102    115   #else
   103    116     Tcl_SetVar2(interp, "sqlite_options", "mmap", "0", TCL_GLOBAL_ONLY);
   104    117   #endif
   105    118   
................................................................................
   584    597   #ifdef SQLITE_OMIT_TCL_VARIABLE
   585    598     Tcl_SetVar2(interp, "sqlite_options", "tclvar", "0", TCL_GLOBAL_ONLY);
   586    599   #else
   587    600     Tcl_SetVar2(interp, "sqlite_options", "tclvar", "1", TCL_GLOBAL_ONLY);
   588    601   #endif
   589    602   
   590    603     Tcl_SetVar2(interp, "sqlite_options", "threadsafe", 
   591         -      STRINGVALUE(SQLITE_THREADSAFE), TCL_GLOBAL_ONLY);
          604  +      SQLITE_THREADSAFE ? "1" : "0", TCL_GLOBAL_ONLY);
          605  +  Tcl_SetVar2(interp, "sqlite_options", "threadsafe1", 
          606  +      SQLITE_THREADSAFE==1 ? "1" : "0", TCL_GLOBAL_ONLY);
          607  +  Tcl_SetVar2(interp, "sqlite_options", "threadsafe2", 
          608  +      SQLITE_THREADSAFE==2 ? "1" : "0", TCL_GLOBAL_ONLY);
   592    609     assert( sqlite3_threadsafe()==SQLITE_THREADSAFE );
   593    610   
   594    611   #ifdef SQLITE_OMIT_TEMPDB
   595    612     Tcl_SetVar2(interp, "sqlite_options", "tempdb", "0", TCL_GLOBAL_ONLY);
   596    613   #else
   597    614     Tcl_SetVar2(interp, "sqlite_options", "tempdb", "1", TCL_GLOBAL_ONLY);
   598    615   #endif

Changes to src/util.c.

   112    112   /*
   113    113   ** The string z[] is followed immediately by another string.  Return
   114    114   ** a poiner to that other string.
   115    115   */
   116    116   const char *sqlite3StrNext(const char *z){
   117    117     return z + strlen(z) + 1;
   118    118   }
          119  +
          120  +/*
          121  +** Helper function for sqlite3Error() - called rarely.  Broken out into
          122  +** a separate routine to avoid unnecessary register saves on entry to
          123  +** sqlite3Error().
          124  +*/
          125  +static SQLITE_NOINLINE void  sqlite3ErrorFinish(sqlite3 *db, int err_code){
          126  +  if( db->pErr ) sqlite3ValueSetNull(db->pErr);
          127  +  sqlite3SystemError(db, err_code);
          128  +}
   119    129   
   120    130   /*
   121    131   ** Set the current error code to err_code and clear any prior error message.
          132  +** Also set iSysErrno (by calling sqlite3System) if the err_code indicates
          133  +** that would be appropriate.
   122    134   */
   123    135   void sqlite3Error(sqlite3 *db, int err_code){
   124    136     assert( db!=0 );
   125    137     db->errCode = err_code;
   126         -  if( db->pErr ) sqlite3ValueSetNull(db->pErr);
          138  +  if( err_code || db->pErr ) sqlite3ErrorFinish(db, err_code);
          139  +}
          140  +
          141  +/*
          142  +** Load the sqlite3.iSysErrno field if that is an appropriate thing
          143  +** to do based on the SQLite error code in rc.
          144  +*/
          145  +void sqlite3SystemError(sqlite3 *db, int rc){
          146  +  if( rc==SQLITE_IOERR_NOMEM ) return;
          147  +  rc &= 0xff;
          148  +  if( rc==SQLITE_CANTOPEN || rc==SQLITE_IOERR ){
          149  +    db->iSysErrno = sqlite3OsGetLastError(db->pVfs);
          150  +  }
   127    151   }
   128    152   
   129    153   /*
   130    154   ** Set the most recent error code and error string for the sqlite
   131    155   ** handle "db". The error code is set to "err_code".
   132    156   **
   133    157   ** If it is not NULL, string zFormat specifies the format of the
................................................................................
   146    170   ** To clear the most recent error for sqlite handle "db", sqlite3Error
   147    171   ** should be called with err_code set to SQLITE_OK and zFormat set
   148    172   ** to NULL.
   149    173   */
   150    174   void sqlite3ErrorWithMsg(sqlite3 *db, int err_code, const char *zFormat, ...){
   151    175     assert( db!=0 );
   152    176     db->errCode = err_code;
          177  +  sqlite3SystemError(db, err_code);
   153    178     if( zFormat==0 ){
   154    179       sqlite3Error(db, err_code);
   155    180     }else if( db->pErr || (db->pErr = sqlite3ValueNew(db))!=0 ){
   156    181       char *z;
   157    182       va_list ap;
   158    183       va_start(ap, zFormat);
   159    184       z = sqlite3VMPrintf(db, zFormat, ap);

Changes to src/vdbe.c.

   198    198     **     different sized allocations. Memory cells provide growable
   199    199     **     allocations.
   200    200     **
   201    201     **   * When using ENABLE_MEMORY_MANAGEMENT, memory cell buffers can
   202    202     **     be freed lazily via the sqlite3_release_memory() API. This
   203    203     **     minimizes the number of malloc calls made by the system.
   204    204     **
   205         -  ** Memory cells for cursors are allocated at the top of the address
   206         -  ** space. Memory cell (p->nMem) corresponds to cursor 0. Space for
   207         -  ** cursor 1 is managed by memory cell (p->nMem-1), etc.
          205  +  ** The memory cell for cursor 0 is aMem[0]. The rest are allocated from
          206  +  ** the top of the register space.  Cursor 1 is at Mem[p->nMem-1].
          207  +  ** Cursor 2 is at Mem[p->nMem-2]. And so forth.
   208    208     */
   209         -  Mem *pMem = &p->aMem[p->nMem-iCur];
          209  +  Mem *pMem = iCur>0 ? &p->aMem[p->nMem-iCur] : p->aMem;
   210    210   
   211    211     int nByte;
   212    212     VdbeCursor *pCx = 0;
   213    213     nByte = 
   214    214         ROUND8(sizeof(VdbeCursor)) + 2*sizeof(u32)*nField + 
   215    215         (eCurType==CURTYPE_BTREE?sqlite3BtreeCursorSize():0);
   216    216   
   217         -  assert( iCur<p->nCursor );
          217  +  assert( iCur>=0 && iCur<p->nCursor );
   218    218     if( p->apCsr[iCur] ){
   219    219       sqlite3VdbeFreeCursor(p, p->apCsr[iCur]);
   220    220       p->apCsr[iCur] = 0;
   221    221     }
   222    222     if( SQLITE_OK==sqlite3VdbeMemClearAndResize(pMem, nByte) ){
   223    223       p->apCsr[iCur] = pCx = (VdbeCursor*)pMem->z;
   224    224       memset(pCx, 0, sizeof(VdbeCursor));
................................................................................
   535    535     sqlite3VdbeMemSetNull(pOut);
   536    536     pOut->flags = MEM_Int;
   537    537     return pOut;
   538    538   }
   539    539   static Mem *out2Prerelease(Vdbe *p, VdbeOp *pOp){
   540    540     Mem *pOut;
   541    541     assert( pOp->p2>0 );
   542         -  assert( pOp->p2<=(p->nMem-p->nCursor) );
          542  +  assert( pOp->p2<=(p->nMem+1 - p->nCursor) );
   543    543     pOut = &p->aMem[pOp->p2];
   544    544     memAboutToChange(p, pOut);
   545    545     if( VdbeMemDynamic(pOut) ){
   546    546       return out2PrereleaseWithClear(pOut);
   547    547     }else{
   548    548       pOut->flags = MEM_Int;
   549    549       return pOut;
................................................................................
   673    673   #endif
   674    674   
   675    675       /* Sanity checking on other operands */
   676    676   #ifdef SQLITE_DEBUG
   677    677       assert( pOp->opflags==sqlite3OpcodeProperty[pOp->opcode] );
   678    678       if( (pOp->opflags & OPFLG_IN1)!=0 ){
   679    679         assert( pOp->p1>0 );
   680         -      assert( pOp->p1<=(p->nMem-p->nCursor) );
          680  +      assert( pOp->p1<=(p->nMem+1 - p->nCursor) );
   681    681         assert( memIsValid(&aMem[pOp->p1]) );
   682    682         assert( sqlite3VdbeCheckMemInvariants(&aMem[pOp->p1]) );
   683    683         REGISTER_TRACE(pOp->p1, &aMem[pOp->p1]);
   684    684       }
   685    685       if( (pOp->opflags & OPFLG_IN2)!=0 ){
   686    686         assert( pOp->p2>0 );
   687         -      assert( pOp->p2<=(p->nMem-p->nCursor) );
          687  +      assert( pOp->p2<=(p->nMem+1 - p->nCursor) );
   688    688         assert( memIsValid(&aMem[pOp->p2]) );
   689    689         assert( sqlite3VdbeCheckMemInvariants(&aMem[pOp->p2]) );
   690    690         REGISTER_TRACE(pOp->p2, &aMem[pOp->p2]);
   691    691       }
   692    692       if( (pOp->opflags & OPFLG_IN3)!=0 ){
   693    693         assert( pOp->p3>0 );
   694         -      assert( pOp->p3<=(p->nMem-p->nCursor) );
          694  +      assert( pOp->p3<=(p->nMem+1 - p->nCursor) );
   695    695         assert( memIsValid(&aMem[pOp->p3]) );
   696    696         assert( sqlite3VdbeCheckMemInvariants(&aMem[pOp->p3]) );
   697    697         REGISTER_TRACE(pOp->p3, &aMem[pOp->p3]);
   698    698       }
   699    699       if( (pOp->opflags & OPFLG_OUT2)!=0 ){
   700    700         assert( pOp->p2>0 );
   701         -      assert( pOp->p2<=(p->nMem-p->nCursor) );
          701  +      assert( pOp->p2<=(p->nMem+1 - p->nCursor) );
   702    702         memAboutToChange(p, &aMem[pOp->p2]);
   703    703       }
   704    704       if( (pOp->opflags & OPFLG_OUT3)!=0 ){
   705    705         assert( pOp->p3>0 );
   706         -      assert( pOp->p3<=(p->nMem-p->nCursor) );
          706  +      assert( pOp->p3<=(p->nMem+1 - p->nCursor) );
   707    707         memAboutToChange(p, &aMem[pOp->p3]);
   708    708       }
   709    709   #endif
   710    710   #if defined(SQLITE_DEBUG) || defined(VDBE_PROFILE)
   711    711       pOrigOp = pOp;
   712    712   #endif
   713    713     
................................................................................
   798    798   
   799    799   /* Opcode:  Gosub P1 P2 * * *
   800    800   **
   801    801   ** Write the current address onto register P1
   802    802   ** and then jump to address P2.
   803    803   */
   804    804   case OP_Gosub: {            /* jump */
   805         -  assert( pOp->p1>0 && pOp->p1<=(p->nMem-p->nCursor) );
          805  +  assert( pOp->p1>0 && pOp->p1<=(p->nMem+1 - p->nCursor) );
   806    806     pIn1 = &aMem[pOp->p1];
   807    807     assert( VdbeMemDynamic(pIn1)==0 );
   808    808     memAboutToChange(p, pIn1);
   809    809     pIn1->flags = MEM_Int;
   810    810     pIn1->u.i = (int)(pOp-aOp);
   811    811     REGISTER_TRACE(pOp->p1, pIn1);
   812    812   
................................................................................
   838    838   ** If P2!=0 then the coroutine implementation immediately follows
   839    839   ** this opcode.  So jump over the coroutine implementation to
   840    840   ** address P2.
   841    841   **
   842    842   ** See also: EndCoroutine
   843    843   */
   844    844   case OP_InitCoroutine: {     /* jump */
   845         -  assert( pOp->p1>0 &&  pOp->p1<=(p->nMem-p->nCursor) );
          845  +  assert( pOp->p1>0 &&  pOp->p1<=(p->nMem+1 - p->nCursor) );
   846    846     assert( pOp->p2>=0 && pOp->p2<p->nOp );
   847    847     assert( pOp->p3>=0 && pOp->p3<p->nOp );
   848    848     pOut = &aMem[pOp->p1];
   849    849     assert( !VdbeMemDynamic(pOut) );
   850    850     pOut->u.i = pOp->p3 - 1;
   851    851     pOut->flags = MEM_Int;
   852    852     if( pOp->p2 ) goto jump_to_p2;
................................................................................
  1107   1107     pOut->z = pOp->p4.z;
  1108   1108     pOut->n = pOp->p1;
  1109   1109     pOut->enc = encoding;
  1110   1110     UPDATE_MAX_BLOBSIZE(pOut);
  1111   1111   #ifndef SQLITE_LIKE_DOESNT_MATCH_BLOBS
  1112   1112     if( pOp->p5 ){
  1113   1113       assert( pOp->p3>0 );
  1114         -    assert( pOp->p3<=(p->nMem-p->nCursor) );
         1114  +    assert( pOp->p3<=(p->nMem+1 - p->nCursor) );
  1115   1115       pIn3 = &aMem[pOp->p3];
  1116   1116       assert( pIn3->flags & MEM_Int );
  1117   1117       if( pIn3->u.i ) pOut->flags = MEM_Blob|MEM_Static|MEM_Term;
  1118   1118     }
  1119   1119   #endif
  1120   1120     break;
  1121   1121   }
................................................................................
  1133   1133   ** OP_Ne or OP_Eq.
  1134   1134   */
  1135   1135   case OP_Null: {           /* out2 */
  1136   1136     int cnt;
  1137   1137     u16 nullFlag;
  1138   1138     pOut = out2Prerelease(p, pOp);
  1139   1139     cnt = pOp->p3-pOp->p2;
  1140         -  assert( pOp->p3<=(p->nMem-p->nCursor) );
         1140  +  assert( pOp->p3<=(p->nMem+1 - p->nCursor) );
  1141   1141     pOut->flags = nullFlag = pOp->p1 ? (MEM_Null|MEM_Cleared) : MEM_Null;
  1142   1142     while( cnt>0 ){
  1143   1143       pOut++;
  1144   1144       memAboutToChange(p, pOut);
  1145   1145       sqlite3VdbeMemSetNull(pOut);
  1146   1146       pOut->flags = nullFlag;
  1147   1147       cnt--;
................................................................................
  1154   1154   **
  1155   1155   ** Set register P1 to have the value NULL as seen by the OP_MakeRecord
  1156   1156   ** instruction, but do not free any string or blob memory associated with
  1157   1157   ** the register, so that if the value was a string or blob that was
  1158   1158   ** previously copied using OP_SCopy, the copies will continue to be valid.
  1159   1159   */
  1160   1160   case OP_SoftNull: {
  1161         -  assert( pOp->p1>0 && pOp->p1<=(p->nMem-p->nCursor) );
         1161  +  assert( pOp->p1>0 && pOp->p1<=(p->nMem+1 - p->nCursor) );
  1162   1162     pOut = &aMem[pOp->p1];
  1163   1163     pOut->flags = (pOut->flags|MEM_Null)&~MEM_Undefined;
  1164   1164     break;
  1165   1165   }
  1166   1166   
  1167   1167   /* Opcode: Blob P1 P2 * P4 *
  1168   1168   ** Synopsis: r[P2]=P4 (len=P1)
................................................................................
  1221   1221     p2 = pOp->p2;
  1222   1222     assert( n>0 && p1>0 && p2>0 );
  1223   1223     assert( p1+n<=p2 || p2+n<=p1 );
  1224   1224   
  1225   1225     pIn1 = &aMem[p1];
  1226   1226     pOut = &aMem[p2];
  1227   1227     do{
  1228         -    assert( pOut<=&aMem[(p->nMem-p->nCursor)] );
  1229         -    assert( pIn1<=&aMem[(p->nMem-p->nCursor)] );
         1228  +    assert( pOut<=&aMem[(p->nMem+1 - p->nCursor)] );
         1229  +    assert( pIn1<=&aMem[(p->nMem+1 - p->nCursor)] );
  1230   1230       assert( memIsValid(pIn1) );
  1231   1231       memAboutToChange(p, pOut);
  1232   1232       sqlite3VdbeMemMove(pOut, pIn1);
  1233   1233   #ifdef SQLITE_DEBUG
  1234   1234       if( pOut->pScopyFrom>=&aMem[p1] && pOut->pScopyFrom<pOut ){
  1235   1235         pOut->pScopyFrom += pOp->p2 - p1;
  1236   1236       }
................................................................................
  1322   1322   ** the result row.
  1323   1323   */
  1324   1324   case OP_ResultRow: {
  1325   1325     Mem *pMem;
  1326   1326     int i;
  1327   1327     assert( p->nResColumn==pOp->p2 );
  1328   1328     assert( pOp->p1>0 );
  1329         -  assert( pOp->p1+pOp->p2<=(p->nMem-p->nCursor)+1 );
         1329  +  assert( pOp->p1+pOp->p2<=(p->nMem+1 - p->nCursor)+1 );
  1330   1330   
  1331   1331   #ifndef SQLITE_OMIT_PROGRESS_CALLBACK
  1332   1332     /* Run the progress counter just before returning.
  1333   1333     */
  1334   1334     if( db->xProgress!=0
  1335   1335      && nVmStep>=nProgressLimit
  1336   1336      && db->xProgress(db->pProgressArg)!=0
................................................................................
  1634   1634   */
  1635   1635   case OP_Function0: {
  1636   1636     int n;
  1637   1637     sqlite3_context *pCtx;
  1638   1638   
  1639   1639     assert( pOp->p4type==P4_FUNCDEF );
  1640   1640     n = pOp->p5;
  1641         -  assert( pOp->p3>0 && pOp->p3<=(p->nMem-p->nCursor) );
  1642         -  assert( n==0 || (pOp->p2>0 && pOp->p2+n<=(p->nMem-p->nCursor)+1) );
         1641  +  assert( pOp->p3>0 && pOp->p3<=(p->nMem+1 - p->nCursor) );
         1642  +  assert( n==0 || (pOp->p2>0 && pOp->p2+n<=(p->nMem+1 - p->nCursor)+1) );
  1643   1643     assert( pOp->p3<pOp->p2 || pOp->p3>=pOp->p2+n );
  1644   1644     pCtx = sqlite3DbMallocRawNN(db, sizeof(*pCtx) + (n-1)*sizeof(sqlite3_value*));
  1645   1645     if( pCtx==0 ) goto no_mem;
  1646   1646     pCtx->pOut = 0;
  1647   1647     pCtx->pFunc = pOp->p4.pFunc;
  1648   1648     pCtx->iOp = (int)(pOp - aOp);
  1649   1649     pCtx->pVdbe = p;
................................................................................
  2135   2135     assert( pKeyInfo!=0 );
  2136   2136     p1 = pOp->p1;
  2137   2137     p2 = pOp->p2;
  2138   2138   #if SQLITE_DEBUG
  2139   2139     if( aPermute ){
  2140   2140       int k, mx = 0;
  2141   2141       for(k=0; k<n; k++) if( aPermute[k]>mx ) mx = aPermute[k];
  2142         -    assert( p1>0 && p1+mx<=(p->nMem-p->nCursor)+1 );
  2143         -    assert( p2>0 && p2+mx<=(p->nMem-p->nCursor)+1 );
         2142  +    assert( p1>0 && p1+mx<=(p->nMem+1 - p->nCursor)+1 );
         2143  +    assert( p2>0 && p2+mx<=(p->nMem+1 - p->nCursor)+1 );
  2144   2144     }else{
  2145         -    assert( p1>0 && p1+n<=(p->nMem-p->nCursor)+1 );
  2146         -    assert( p2>0 && p2+n<=(p->nMem-p->nCursor)+1 );
         2145  +    assert( p1>0 && p1+n<=(p->nMem+1 - p->nCursor)+1 );
         2146  +    assert( p2>0 && p2+n<=(p->nMem+1 - p->nCursor)+1 );
  2147   2147     }
  2148   2148   #endif /* SQLITE_DEBUG */
  2149   2149     for(i=0; i<n; i++){
  2150   2150       idx = aPermute ? aPermute[i] : i;
  2151   2151       assert( memIsValid(&aMem[p1+idx]) );
  2152   2152       assert( memIsValid(&aMem[p2+idx]) );
  2153   2153       REGISTER_TRACE(p1+idx, &aMem[p1+idx]);
................................................................................
  2401   2401   
  2402   2402     pC = p->apCsr[pOp->p1];
  2403   2403     p2 = pOp->p2;
  2404   2404   
  2405   2405     /* If the cursor cache is stale, bring it up-to-date */
  2406   2406     rc = sqlite3VdbeCursorMoveto(&pC, &p2);
  2407   2407   
  2408         -  assert( pOp->p3>0 && pOp->p3<=(p->nMem-p->nCursor) );
         2408  +  assert( pOp->p3>0 && pOp->p3<=(p->nMem+1 - p->nCursor) );
  2409   2409     pDest = &aMem[pOp->p3];
  2410   2410     memAboutToChange(p, pDest);
  2411   2411     assert( pOp->p1>=0 && pOp->p1<p->nCursor );
  2412   2412     assert( pC!=0 );
  2413   2413     assert( p2<pC->nField );
  2414   2414     aOffset = pC->aOffset;
  2415   2415     assert( pC->eCurType!=CURTYPE_VTAB );
................................................................................
  2644   2644     char cAff;               /* A single character of affinity */
  2645   2645   
  2646   2646     zAffinity = pOp->p4.z;
  2647   2647     assert( zAffinity!=0 );
  2648   2648     assert( zAffinity[pOp->p2]==0 );
  2649   2649     pIn1 = &aMem[pOp->p1];
  2650   2650     while( (cAff = *(zAffinity++))!=0 ){
  2651         -    assert( pIn1 <= &p->aMem[(p->nMem-p->nCursor)] );
         2651  +    assert( pIn1 <= &p->aMem[(p->nMem+1 - p->nCursor)] );
  2652   2652       assert( memIsValid(pIn1) );
  2653   2653       applyAffinity(pIn1, cAff, encoding);
  2654   2654       pIn1++;
  2655   2655     }
  2656   2656     break;
  2657   2657   }
  2658   2658   
................................................................................
  2706   2706     ** of the record to data0.
  2707   2707     */
  2708   2708     nData = 0;         /* Number of bytes of data space */
  2709   2709     nHdr = 0;          /* Number of bytes of header space */
  2710   2710     nZero = 0;         /* Number of zero bytes at the end of the record */
  2711   2711     nField = pOp->p1;
  2712   2712     zAffinity = pOp->p4.z;
  2713         -  assert( nField>0 && pOp->p2>0 && pOp->p2+nField<=(p->nMem-p->nCursor)+1 );
         2713  +  assert( nField>0 && pOp->p2>0 && pOp->p2+nField<=(p->nMem+1 - p->nCursor)+1 );
  2714   2714     pData0 = &aMem[nField];
  2715   2715     nField = pOp->p2;
  2716   2716     pLast = &pData0[nField-1];
  2717   2717     file_format = p->minWriteFileFormat;
  2718   2718   
  2719   2719     /* Identify the output register */
  2720   2720     assert( pOp->p3<pOp->p1 || pOp->p3>=pOp->p1+pOp->p2 );
................................................................................
  2796   2796       /* EVIDENCE-OF: R-64536-51728 The values for each column in the record
  2797   2797       ** immediately follow the header. */
  2798   2798       j += sqlite3VdbeSerialPut(&zNewRecord[j], pRec, serial_type); /* content */
  2799   2799     }while( (++pRec)<=pLast );
  2800   2800     assert( i==nHdr );
  2801   2801     assert( j==nByte );
  2802   2802   
  2803         -  assert( pOp->p3>0 && pOp->p3<=(p->nMem-p->nCursor) );
         2803  +  assert( pOp->p3>0 && pOp->p3<=(p->nMem+1 - p->nCursor) );
  2804   2804     pOut->n = (int)nByte;
  2805   2805     pOut->flags = MEM_Blob;
  2806   2806     if( nZero ){
  2807   2807       pOut->u.nZero = nZero;
  2808   2808       pOut->flags |= MEM_Zero;
  2809   2809     }
  2810   2810     pOut->enc = SQLITE_UTF8;  /* In case the blob is ever converted to text */
................................................................................
  3382   3382         p->minWriteFileFormat = pDb->pSchema->file_format;
  3383   3383       }
  3384   3384     }else{
  3385   3385       wrFlag = 0;
  3386   3386     }
  3387   3387     if( pOp->p5 & OPFLAG_P2ISREG ){
  3388   3388       assert( p2>0 );
  3389         -    assert( p2<=(p->nMem-p->nCursor) );
         3389  +    assert( p2<=(p->nMem+1 - p->nCursor) );
  3390   3390       pIn2 = &aMem[p2];
  3391   3391       assert( memIsValid(pIn2) );
  3392   3392       assert( (pIn2->flags & MEM_Int)!=0 );
  3393   3393       sqlite3VdbeMemIntegerify(pIn2);
  3394   3394       p2 = (int)pIn2->u.i;
  3395   3395       /* The p2 value always comes from a prior OP_CreateTable opcode and
  3396   3396       ** that opcode will always set the p2 value to 2 or more or else fail.
................................................................................
  4177   4177         if( p->pFrame ){
  4178   4178           for(pFrame=p->pFrame; pFrame->pParent; pFrame=pFrame->pParent);
  4179   4179           /* Assert that P3 is a valid memory cell. */
  4180   4180           assert( pOp->p3<=pFrame->nMem );
  4181   4181           pMem = &pFrame->aMem[pOp->p3];
  4182   4182         }else{
  4183   4183           /* Assert that P3 is a valid memory cell. */
  4184         -        assert( pOp->p3<=(p->nMem-p->nCursor) );
         4184  +        assert( pOp->p3<=(p->nMem+1 - p->nCursor) );
  4185   4185           pMem = &aMem[pOp->p3];
  4186   4186           memAboutToChange(p, pMem);
  4187   4187         }
  4188   4188         assert( memIsValid(pMem) );
  4189   4189   
  4190   4190         REGISTER_TRACE(pOp->p3, pMem);
  4191   4191         sqlite3VdbeMemIntegerify(pMem);
................................................................................
  5003   5003   case OP_IdxDelete: {
  5004   5004     VdbeCursor *pC;
  5005   5005     BtCursor *pCrsr;
  5006   5006     int res;
  5007   5007     UnpackedRecord r;
  5008   5008   
  5009   5009     assert( pOp->p3>0 );
  5010         -  assert( pOp->p2>0 && pOp->p2+pOp->p3<=(p->nMem-p->nCursor)+1 );
         5010  +  assert( pOp->p2>0 && pOp->p2+pOp->p3<=(p->nMem+1 - p->nCursor)+1 );
  5011   5011     assert( pOp->p1>=0 && pOp->p1<p->nCursor );
  5012   5012     pC = p->apCsr[pOp->p1];
  5013   5013     assert( pC!=0 );
  5014   5014     assert( pC->eCurType==CURTYPE_BTREE );
  5015   5015     pCrsr = pC->uc.pCursor;
  5016   5016     assert( pCrsr!=0 );
  5017   5017     assert( pOp->p5==0 );
................................................................................
  5509   5509     Mem *pnErr;     /* Register keeping track of errors remaining */
  5510   5510   
  5511   5511     assert( p->bIsReader );
  5512   5512     nRoot = pOp->p2;
  5513   5513     aRoot = pOp->p4.ai;
  5514   5514     assert( nRoot>0 );
  5515   5515     assert( aRoot[nRoot]==0 );
  5516         -  assert( pOp->p3>0 && pOp->p3<=(p->nMem-p->nCursor) );
         5516  +  assert( pOp->p3>0 && pOp->p3<=(p->nMem+1 - p->nCursor) );
  5517   5517     pnErr = &aMem[pOp->p3];
  5518   5518     assert( (pnErr->flags & MEM_Int)!=0 );
  5519   5519     assert( (pnErr->flags & (MEM_Str|MEM_Blob))==0 );
  5520   5520     pIn1 = &aMem[pOp->p1];
  5521   5521     assert( pOp->p5<db->nDb );
  5522   5522     assert( DbMaskTest(p->btreeMask, pOp->p5) );
  5523   5523     z = sqlite3BtreeIntegrityCheck(db->aDb[pOp->p5].pBt, aRoot, nRoot,
................................................................................
  5699   5699     if( (pRt->flags&MEM_Frame)==0 ){
  5700   5700       /* SubProgram.nMem is set to the number of memory cells used by the 
  5701   5701       ** program stored in SubProgram.aOp. As well as these, one memory
  5702   5702       ** cell is required for each cursor used by the program. Set local
  5703   5703       ** variable nMem (and later, VdbeFrame.nChildMem) to this value.
  5704   5704       */
  5705   5705       nMem = pProgram->nMem + pProgram->nCsr;
         5706  +    assert( nMem>0 );
         5707  +    if( pProgram->nCsr==0 ) nMem++;
  5706   5708       nByte = ROUND8(sizeof(VdbeFrame))
  5707   5709                 + nMem * sizeof(Mem)
  5708   5710                 + pProgram->nCsr * sizeof(VdbeCursor *)
  5709   5711                 + pProgram->nOnce * sizeof(u8);
  5710   5712       pFrame = sqlite3DbMallocZero(db, nByte);
  5711   5713       if( !pFrame ){
  5712   5714         goto no_mem;
................................................................................
  5735   5737       pEnd = &VdbeFrameMem(pFrame)[pFrame->nChildMem];
  5736   5738       for(pMem=VdbeFrameMem(pFrame); pMem!=pEnd; pMem++){
  5737   5739         pMem->flags = MEM_Undefined;
  5738   5740         pMem->db = db;
  5739   5741       }
  5740   5742     }else{
  5741   5743       pFrame = pRt->u.pFrame;
  5742         -    assert( pProgram->nMem+pProgram->nCsr==pFrame->nChildMem );
         5744  +    assert( pProgram->nMem+pProgram->nCsr==pFrame->nChildMem 
         5745  +        || (pProgram->nCsr==0 && pProgram->nMem+1==pFrame->nChildMem) );
  5743   5746       assert( pProgram->nCsr==pFrame->nChildCsr );
  5744   5747       assert( (int)(pOp - aOp)==pFrame->pc );
  5745   5748     }
  5746   5749   
  5747   5750     p->nFrame++;
  5748   5751     pFrame->pParent = p->pFrame;
  5749   5752     pFrame->lastRowid = lastRowid;
................................................................................
  5750   5753     pFrame->nChange = p->nChange;
  5751   5754     pFrame->nDbChange = p->db->nChange;
  5752   5755     assert( pFrame->pAuxData==0 );
  5753   5756     pFrame->pAuxData = p->pAuxData;
  5754   5757     p->pAuxData = 0;
  5755   5758     p->nChange = 0;
  5756   5759     p->pFrame = pFrame;
  5757         -  p->aMem = aMem = &VdbeFrameMem(pFrame)[-1];
         5760  +  p->aMem = aMem = VdbeFrameMem(pFrame);
  5758   5761     p->nMem = pFrame->nChildMem;
  5759   5762     p->nCursor = (u16)pFrame->nChildCsr;
  5760         -  p->apCsr = (VdbeCursor **)&aMem[p->nMem+1];
         5763  +  p->apCsr = (VdbeCursor **)&aMem[p->nMem];
  5761   5764     p->aOp = aOp = pProgram->aOp;
  5762   5765     p->nOp = pProgram->nOp;
  5763   5766     p->aOnceFlag = (u8 *)&p->apCsr[p->nCursor];
  5764   5767     p->nOnceFlag = pProgram->nOnce;
  5765   5768   #ifdef SQLITE_ENABLE_STMT_SCANSTATUS
  5766   5769     p->anExec = 0;
  5767   5770   #endif
................................................................................
  5999   6002   */
  6000   6003   case OP_AggStep0: {
  6001   6004     int n;
  6002   6005     sqlite3_context *pCtx;
  6003   6006   
  6004   6007     assert( pOp->p4type==P4_FUNCDEF );
  6005   6008     n = pOp->p5;
  6006         -  assert( pOp->p3>0 && pOp->p3<=(p->nMem-p->nCursor) );
  6007         -  assert( n==0 || (pOp->p2>0 && pOp->p2+n<=(p->nMem-p->nCursor)+1) );
         6009  +  assert( pOp->p3>0 && pOp->p3<=(p->nMem+1 - p->nCursor) );
         6010  +  assert( n==0 || (pOp->p2>0 && pOp->p2+n<=(p->nMem+1 - p->nCursor)+1) );
  6008   6011     assert( pOp->p3<pOp->p2 || pOp->p3>=pOp->p2+n );
  6009   6012     pCtx = sqlite3DbMallocRawNN(db, sizeof(*pCtx) + (n-1)*sizeof(sqlite3_value*));
  6010   6013     if( pCtx==0 ) goto no_mem;
  6011   6014     pCtx->pMem = 0;
  6012   6015     pCtx->pFunc = pOp->p4.pFunc;
  6013   6016     pCtx->iOp = (int)(pOp - aOp);
  6014   6017     pCtx->pVdbe = p;
................................................................................
  6079   6082   ** argument is not used by this opcode.  It is only there to disambiguate
  6080   6083   ** functions that can take varying numbers of arguments.  The
  6081   6084   ** P4 argument is only needed for the degenerate case where
  6082   6085   ** the step function was not previously called.
  6083   6086   */
  6084   6087   case OP_AggFinal: {
  6085   6088     Mem *pMem;
  6086         -  assert( pOp->p1>0 && pOp->p1<=(p->nMem-p->nCursor) );
         6089  +  assert( pOp->p1>0 && pOp->p1<=(p->nMem+1 - p->nCursor) );
  6087   6090     pMem = &aMem[pOp->p1];
  6088   6091     assert( (pMem->flags & ~(MEM_Null|MEM_Agg))==0 );
  6089   6092     rc = sqlite3VdbeMemFinalize(pMem, pOp->p4.pFunc);
  6090   6093     if( rc ){
  6091   6094       sqlite3VdbeError(p, "%s", sqlite3_value_text(pMem));
  6092   6095       goto abort_due_to_error;
  6093   6096     }
................................................................................
  6521   6524     sqlite3_vtab *pVtab;
  6522   6525     const sqlite3_module *pModule;
  6523   6526     Mem *pDest;
  6524   6527     sqlite3_context sContext;
  6525   6528   
  6526   6529     VdbeCursor *pCur = p->apCsr[pOp->p1];
  6527   6530     assert( pCur->eCurType==CURTYPE_VTAB );
  6528         -  assert( pOp->p3>0 && pOp->p3<=(p->nMem-p->nCursor) );
         6531  +  assert( pOp->p3>0 && pOp->p3<=(p->nMem+1 - p->nCursor) );
  6529   6532     pDest = &aMem[pOp->p3];
  6530   6533     memAboutToChange(p, pDest);
  6531   6534     if( pCur->nullRow ){
  6532   6535       sqlite3VdbeMemSetNull(pDest);
  6533   6536       break;
  6534   6537     }
  6535   6538     pVtab = pCur->uc.pVCur->pVtab;
................................................................................
  6879   6882   abort_due_to_error:
  6880   6883     if( db->mallocFailed ) rc = SQLITE_NOMEM_BKPT;
  6881   6884     assert( rc );
  6882   6885     if( p->zErrMsg==0 && rc!=SQLITE_IOERR_NOMEM ){
  6883   6886       sqlite3VdbeError(p, "%s", sqlite3ErrStr(rc));
  6884   6887     }
  6885   6888     p->rc = rc;
         6889  +  sqlite3SystemError(db, rc);
  6886   6890     testcase( sqlite3GlobalConfig.xLog!=0 );
  6887   6891     sqlite3_log(rc, "statement aborts at %d: [%s] %s", 
  6888   6892                      (int)(pOp - aOp), p->zSql, p->zErrMsg);
  6889   6893     sqlite3VdbeHalt(p);
  6890   6894     if( rc==SQLITE_IOERR_NOMEM ) sqlite3OomFault(db);
  6891   6895     rc = SQLITE_ERROR;
  6892   6896     if( resetSchemaOnFault>0 ){

Changes to src/vdbe.h.

   200    200   int sqlite3VdbeDeletePriorOpcode(Vdbe*, u8 op);
   201    201   void sqlite3VdbeChangeP4(Vdbe*, int addr, const char *zP4, int N);
   202    202   void sqlite3VdbeSetP4KeyInfo(Parse*, Index*);
   203    203   void sqlite3VdbeUsesBtree(Vdbe*, int);
   204    204   VdbeOp *sqlite3VdbeGetOp(Vdbe*, int);
   205    205   int sqlite3VdbeMakeLabel(Vdbe*);
   206    206   void sqlite3VdbeRunOnlyOnce(Vdbe*);
          207  +void sqlite3VdbeReusable(Vdbe*);
   207    208   void sqlite3VdbeDelete(Vdbe*);
   208    209   void sqlite3VdbeClearObject(sqlite3*,Vdbe*);
   209    210   void sqlite3VdbeMakeReady(Vdbe*,Parse*);
   210    211   int sqlite3VdbeFinalize(Vdbe*);
   211    212   void sqlite3VdbeResolveLabel(Vdbe*, int);
   212    213   int sqlite3VdbeCurrentAddr(Vdbe*);
   213    214   #ifdef SQLITE_DEBUG

Changes to src/vdbeaux.c.

   387    387   
   388    388   /*
   389    389   ** Mark the VDBE as one that can only be run one time.
   390    390   */
   391    391   void sqlite3VdbeRunOnlyOnce(Vdbe *p){
   392    392     p->runOnlyOnce = 1;
   393    393   }
          394  +
          395  +/*
          396  +** Mark the VDBE as one that can only be run multiple times.
          397  +*/
          398  +void sqlite3VdbeReusable(Vdbe *p){
          399  +  p->runOnlyOnce = 0;
          400  +}
   394    401   
   395    402   #ifdef SQLITE_DEBUG /* sqlite3AssertMayAbort() logic */
   396    403   
   397    404   /*
   398    405   ** The following type and function are used to iterate through all opcodes
   399    406   ** in a Vdbe main program and each of the sub-programs (triggers) it may 
   400    407   ** invoke directly or indirectly. It should be used as follows:
................................................................................
  1791   1798     */
  1792   1799     assert( p->nOp>0 );
  1793   1800   
  1794   1801     /* Set the magic to VDBE_MAGIC_RUN sooner rather than later. */
  1795   1802     p->magic = VDBE_MAGIC_RUN;
  1796   1803   
  1797   1804   #ifdef SQLITE_DEBUG
  1798         -  for(i=1; i<p->nMem; i++){
         1805  +  for(i=0; i<p->nMem; i++){
  1799   1806       assert( p->aMem[i].db==p->db );
  1800   1807     }
  1801   1808   #endif
  1802   1809     p->pc = -1;
  1803   1810     p->rc = SQLITE_OK;
  1804   1811     p->errorAction = OE_Abort;
  1805   1812     p->nChange = 0;
................................................................................
  1856   1863     nVar = pParse->nVar;
  1857   1864     nMem = pParse->nMem;
  1858   1865     nCursor = pParse->nTab;
  1859   1866     nArg = pParse->nMaxArg;
  1860   1867     nOnce = pParse->nOnce;
  1861   1868     if( nOnce==0 ) nOnce = 1; /* Ensure at least one byte in p->aOnceFlag[] */
  1862   1869     
  1863         -  /* For each cursor required, also allocate a memory cell. Memory
  1864         -  ** cells (nMem+1-nCursor)..nMem, inclusive, will never be used by
  1865         -  ** the vdbe program. Instead they are used to allocate memory for
  1866         -  ** VdbeCursor/BtCursor structures. The blob of memory associated with 
  1867         -  ** cursor 0 is stored in memory cell nMem. Memory cell (nMem-1)
  1868         -  ** stores the blob of memory associated with cursor 1, etc.
  1869         -  **
         1870  +  /* Each cursor uses a memory cell.  The first cursor (cursor 0) can
         1871  +  ** use aMem[0] which is not otherwise used by the VDBE program.  Allocate
         1872  +  ** space at the end of aMem[] for cursors 1 and greater.
  1870   1873     ** See also: allocateCursor().
  1871   1874     */
  1872   1875     nMem += nCursor;
         1876  +  if( nCursor==0 && nMem>0 ) nMem++;  /* Space for aMem[0] even if not used */
  1873   1877   
  1874   1878     /* Figure out how much reusable memory is available at the end of the
  1875   1879     ** opcode array.  This extra memory will be reallocated for other elements
  1876   1880     ** of the prepared statement.
  1877   1881     */
  1878   1882     n = ROUND8(sizeof(Op)*p->nOp);              /* Bytes of opcode memory used */
  1879   1883     x.pSpace = &((u8*)p->aOp)[n];               /* Unused opcode memory */
................................................................................
  1927   1931       }
  1928   1932     }
  1929   1933     p->nzVar = pParse->nzVar;
  1930   1934     p->azVar = pParse->azVar;
  1931   1935     pParse->nzVar =  0;
  1932   1936     pParse->azVar = 0;
  1933   1937     if( p->aMem ){
  1934         -    p->aMem--;                      /* aMem[] goes from 1..nMem */
  1935         -    p->nMem = nMem;                 /*       not from 0..nMem-1 */
  1936         -    for(n=1; n<=nMem; n++){
         1938  +    p->nMem = nMem;
         1939  +    for(n=0; n<nMem; n++){
  1937   1940         p->aMem[n].flags = MEM_Undefined;
  1938   1941         p->aMem[n].db = db;
  1939   1942       }
  1940   1943     }
  1941   1944     p->explain = pParse->explain;
  1942   1945     sqlite3VdbeRewind(p);
  1943   1946   }
................................................................................
  2039   2042       sqlite3VdbeFrameRestore(pFrame);
  2040   2043       p->pFrame = 0;
  2041   2044       p->nFrame = 0;
  2042   2045     }
  2043   2046     assert( p->nFrame==0 );
  2044   2047     closeCursorsInFrame(p);
  2045   2048     if( p->aMem ){
  2046         -    releaseMemArray(&p->aMem[1], p->nMem);
         2049  +    releaseMemArray(p->aMem, p->nMem);
  2047   2050     }
  2048   2051     while( p->pDelFrame ){
  2049   2052       VdbeFrame *pDel = p->pDelFrame;
  2050   2053       p->pDelFrame = pDel->pParent;
  2051   2054       sqlite3VdbeFrameDelete(pDel);
  2052   2055     }
  2053   2056   
................................................................................
  2064   2067   
  2065   2068   #ifdef SQLITE_DEBUG
  2066   2069     /* Execute assert() statements to ensure that the Vdbe.apCsr[] and 
  2067   2070     ** Vdbe.aMem[] arrays have already been cleaned up.  */
  2068   2071     int i;
  2069   2072     if( p->apCsr ) for(i=0; i<p->nCursor; i++) assert( p->apCsr[i]==0 );
  2070   2073     if( p->aMem ){
  2071         -    for(i=1; i<=p->nMem; i++) assert( p->aMem[i].flags==MEM_Undefined );
         2074  +    for(i=0; i<p->nMem; i++) assert( p->aMem[i].flags==MEM_Undefined );
  2072   2075     }
  2073   2076   #endif
  2074   2077   
  2075   2078     sqlite3DbFree(db, p->zErrMsg);
  2076   2079     p->zErrMsg = 0;
  2077   2080     p->pResultSet = 0;
  2078   2081   }

Changes to src/vdbemem.c.

   757    757   **
   758    758   ** This is used for testing and debugging only - to make sure shallow
   759    759   ** copies are not misused.
   760    760   */
   761    761   void sqlite3VdbeMemAboutToChange(Vdbe *pVdbe, Mem *pMem){
   762    762     int i;
   763    763     Mem *pX;
   764         -  for(i=1, pX=&pVdbe->aMem[1]; i<=pVdbe->nMem; i++, pX++){
          764  +  for(i=0, pX=pVdbe->aMem; i<pVdbe->nMem; i++, pX++){
   765    765       if( pX->pScopyFrom==pMem ){
   766    766         pX->flags |= MEM_Undefined;
   767    767         pX->pScopyFrom = 0;
   768    768       }
   769    769     }
   770    770     pMem->pScopyFrom = 0;
   771    771   }

Changes to src/where.c.

   285    285     pScan->pOrigWC = pWC;
   286    286     pScan->pWC = pWC;
   287    287     pScan->pIdxExpr = 0;
   288    288     if( pIdx ){
   289    289       j = iColumn;
   290    290       iColumn = pIdx->aiColumn[j];
   291    291       if( iColumn==XN_EXPR ) pScan->pIdxExpr = pIdx->aColExpr->a[j].pExpr;
          292  +    if( iColumn==pIdx->pTable->iPKey ) iColumn = XN_ROWID;
   292    293     }
   293    294     if( pIdx && iColumn>=0 ){
   294    295       pScan->idxaff = pIdx->pTable->aCol[iColumn].affinity;
   295    296       pScan->zCollName = pIdx->azColl[j];
   296    297     }else{
   297    298       pScan->idxaff = 0;
   298    299       pScan->zCollName = 0;
................................................................................
  3925   3926     WhereClause *pWC;
  3926   3927     WhereTerm *pTerm;
  3927   3928     WhereLoop *pLoop;
  3928   3929     int iCur;
  3929   3930     int j;
  3930   3931     Table *pTab;
  3931   3932     Index *pIdx;
  3932         -  
         3933  +
  3933   3934     pWInfo = pBuilder->pWInfo;
  3934   3935     if( pWInfo->wctrlFlags & WHERE_FORCE_TABLE ) return 0;
  3935   3936     assert( pWInfo->pTabList->nSrc>=1 );
  3936   3937     pItem = pWInfo->pTabList->a;
  3937   3938     pTab = pItem->pTab;
  3938   3939     if( IsVirtual(pTab) ) return 0;
  3939   3940     if( pItem->fg.isIndexedBy ) return 0;

Changes to test/capi3.test.

   168    168   do_test capi3-3.2 {
   169    169     sqlite3_close $db2
   170    170   } {SQLITE_OK}
   171    171   do_test capi3-3.3 {
   172    172     catch {
   173    173       set db2 [sqlite3_open /bogus/path/test.db {}]
   174    174     }
   175         -  sqlite3_extended_errcode $db2
   176         -} {SQLITE_CANTOPEN}
          175  +  set ::capi3_errno [sqlite3_system_errno $db2]
          176  +  list [sqlite3_extended_errcode $db2] [expr {$::capi3_errno!=0}]
          177  +} {SQLITE_CANTOPEN 1}
   177    178   do_test capi3-3.4 {
   178    179     sqlite3_errmsg $db2
   179    180   } {unable to open database file}
   180    181   do_test capi3-3.5 {
   181         -  sqlite3_close $db2
   182         -} {SQLITE_OK}
          182  +  list [sqlite3_system_errno $db2] [sqlite3_close $db2]
          183  +} [list $::capi3_errno SQLITE_OK]
   183    184   if {[clang_sanitize_address]==0} {
   184    185     do_test capi3-3.6.1-misuse {
   185    186       sqlite3_close $db2
   186    187     } {SQLITE_MISUSE}
   187    188     do_test capi3-3.6.2-misuse {
   188    189       sqlite3_errmsg $db2
   189    190     } {library routine called out of sequence}
................................................................................
   921    922   } {0 {}}
   922    923   do_test capi3-11.9.3 {
   923    924     sqlite3_get_autocommit $DB
   924    925   } 1
   925    926   do_test capi3-11.10 {
   926    927     sqlite3_step $STMT
   927    928   } {SQLITE_ROW}
   928         -ifcapable !autoreset {
   929         -  # If SQLITE_OMIT_AUTORESET is defined, then the statement must be
   930         -  # reset() before it can be passed to step() again.
   931         -  do_test capi3-11.11a { sqlite3_step $STMT } {SQLITE_MISUSE}
   932         -  do_test capi3-11.11b { sqlite3_reset $STMT } {SQLITE_ABORT}
   933         -}
   934    929   do_test capi3-11.11 {
   935    930     sqlite3_step $STMT
   936    931   } {SQLITE_DONE}
   937         -do_test capi3-11.12 {
   938         -  sqlite3_step $STMT
   939         -  sqlite3_step $STMT
   940         -} {SQLITE_ROW}
          932  +ifcapable api_armor {
          933  +  do_test capi3-11.12armor {
          934  +    sqlite3_step $STMT
          935  +    sqlite3_step $STMT
          936  +  } {SQLITE_MISUSE}
          937  +} else {
          938  +  do_test capi3-11.12 {
          939  +    sqlite3_step $STMT
          940  +    sqlite3_step $STMT
          941  +  } {SQLITE_ROW}
          942  +}
   941    943   do_test capi3-11.13 {
   942    944     sqlite3_finalize $STMT
   943    945   } {SQLITE_OK}
   944    946   do_test capi3-11.14 {
   945    947     execsql {
   946    948       SELECT a FROM t2;
   947    949     }

Changes to test/capi3c.test.

   861    861   } {0 {}}
   862    862   do_test capi3c-11.9.3 {
   863    863     sqlite3_get_autocommit $DB
   864    864   } 1
   865    865   do_test capi3c-11.10 {
   866    866     sqlite3_step $STMT
   867    867   } {SQLITE_ROW}
   868         -ifcapable !autoreset {
   869         -  # If SQLITE_OMIT_AUTORESET is defined, then the statement must be
   870         -  # reset() before it can be passed to step() again.
   871         -  do_test capi3-11.11a { sqlite3_step $STMT } {SQLITE_MISUSE}
   872         -  do_test capi3-11.11b { sqlite3_reset $STMT } {SQLITE_ABORT}
   873         -}
   874    868   do_test capi3c-11.11 {
   875    869     sqlite3_step $STMT
   876    870   } {SQLITE_DONE}
   877         -do_test capi3c-11.12 {
   878         -  sqlite3_step $STMT
   879         -  sqlite3_step $STMT
   880         -} {SQLITE_ROW}
          871  +ifcapable api_armor {
          872  +  do_test capi3c-11.12armor {
          873  +    sqlite3_step $STMT
          874  +    sqlite3_step $STMT
          875  +  } {SQLITE_MISUSE}
          876  +} else {
          877  +  do_test capi3c-11.12 {
          878  +    sqlite3_step $STMT
          879  +    sqlite3_step $STMT
          880  +  } {SQLITE_ROW}
          881  +}
   881    882   do_test capi3c-11.13 {
   882    883     sqlite3_finalize $STMT
   883    884   } {SQLITE_OK}
   884    885   do_test capi3c-11.14 {
   885    886     execsql {
   886    887       SELECT a FROM t2;
   887    888     }

Changes to test/exclusive.test.

   419    419     }
   420    420   } {}
   421    421   do_test exclusive-5.1 {
   422    422     # Three files are open: The db, journal and statement-journal.
   423    423     # (2016-03-04) The statement-journal is now opened lazily
   424    424     set sqlite_open_file_count
   425    425     expr $sqlite_open_file_count-$extrafds
   426         -} [expr 2 - ($TEMP_STORE>=2)]
          426  +} {2}
   427    427   do_test exclusive-5.2 {
   428    428     execsql {
   429    429       COMMIT;
   430    430     }
   431    431     # One file open: the db.
   432    432     set sqlite_open_file_count
   433    433     expr $sqlite_open_file_count-$extrafds
................................................................................
   446    446     execsql {
   447    447       INSERT INTO abc SELECT a+10, b+10, c+10 FROM abc;
   448    448     }
   449    449     # Three files are open: The db, journal and statement-journal.
   450    450     # 2016-03-04: The statement-journal open is deferred
   451    451     set sqlite_open_file_count
   452    452     expr $sqlite_open_file_count-$extrafds
   453         -} [expr 2 - ($TEMP_STORE>=2)]
          453  +} {2}
   454    454   do_test exclusive-5.5 {
   455    455     execsql {
   456    456       COMMIT;
   457    457     }
   458    458     # Three files are still open: The db, journal and statement-journal.
   459    459     # 2016-03-04: The statement-journal open is deferred
   460    460     set sqlite_open_file_count
   461    461     expr $sqlite_open_file_count-$extrafds
   462         -} [expr 2 - ($TEMP_STORE>=2)]
          462  +} {2}
   463    463   do_test exclusive-5.6 {
   464    464     execsql {
   465    465       PRAGMA locking_mode = normal;
   466    466       SELECT * FROM abc;
   467    467     }
   468    468   } {normal 1 2 3 2 3 4 5 6 7 11 12 13 12 13 14 15 16 17}
   469    469   do_test exclusive-5.7 {

Changes to test/intpkey.test.

   292    292       SELECT * FROM t1 WHERE c=='world';
   293    293     }
   294    294   } {5 hello world 11 hello world 5}
   295    295   do_test intpkey-3.8 {
   296    296     count {
   297    297       SELECT * FROM t1 WHERE c=='world' AND a>7;
   298    298     }
   299         -} {11 hello world 4}
          299  +} {11 hello world 3}
   300    300   do_test intpkey-3.9 {
   301    301     count {
   302    302       SELECT * FROM t1 WHERE 7<a;
   303    303     }
   304    304   } {11 hello world 1}
   305    305   
   306    306   # Test inequality constraints on integer primary keys and rowids

Changes to test/memsubsys1.test.

    96     96   sqlite3_shutdown
    97     97   sqlite3_config_pagecache [expr 1024+$xtra_size] 20
    98     98   sqlite3_initialize
    99     99   reset_highwater_marks
   100    100   build_test_db memsubsys1-2 {PRAGMA page_size=1024; PRAGMA mmap_size=0}
   101    101   #show_memstats
   102    102   set MEMORY_MANAGEMENT $sqlite_options(memorymanage)
   103         -ifcapable !malloc_usable_size {
   104         - do_test memsubsys1-2.3 {
   105         -    set pg_ovfl [lindex [sqlite3_status SQLITE_STATUS_PAGECACHE_OVERFLOW 0] 2]
   106         -  } [expr ($TEMP_STORE>1 || $MEMORY_MANAGEMENT==0)*1024]
          103  +ifcapable pagecache_overflow_stats {
          104  +  ifcapable !malloc_usable_size {
          105  +    do_test memsubsys1-2.3 {
          106  +      set pg_ovfl [lindex [sqlite3_status SQLITE_STATUS_PAGECACHE_OVERFLOW 0] 2]
          107  +    } [expr ($TEMP_STORE>1 || $MEMORY_MANAGEMENT==0)*1024]
          108  +  }
   107    109   }
   108    110   do_test memsubsys1-2.4 {
   109    111     set pg_used [lindex [sqlite3_status SQLITE_STATUS_PAGECACHE_USED 0] 2]
   110    112   } 20
   111    113   do_test memsubsys1-2.5 {
   112    114     set s_used [lindex [sqlite3_status SQLITE_STATUS_SCRATCH_USED 0] 2]
   113    115   } 0

Changes to test/mutex1.test.

    93     93   # Tests mutex1-2.* test the three thread-safety related modes that
    94     94   # can be selected using sqlite3_config:
    95     95   #
    96     96   #   * Serialized mode,
    97     97   #   * Multi-threaded mode,
    98     98   #   * Single-threaded mode.
    99     99   #
   100         -ifcapable threadsafe&&shared_cache {
          100  +ifcapable threadsafe1&&shared_cache {
   101    101     set enable_shared_cache [sqlite3_enable_shared_cache 1]
   102    102     foreach {mode mutexes} {
   103    103       singlethread {}
   104    104       multithread  {
   105    105         fast static_app1 static_app2 static_app3
   106    106         static_lru static_master static_mem static_open
   107    107         static_prng static_pmem static_vfs1 static_vfs2

Changes to test/releasetest.tcl.

   170    170       -DSQLITE_THREADSAFE=2
   171    171       --enable-json1 --enable-fts5
   172    172     }
   173    173     "Locking-Style" {
   174    174       -O2
   175    175       -DSQLITE_ENABLE_LOCKING_STYLE=1
   176    176     }
   177         -  "OS-X" {
          177  +  "Apple" {
   178    178       -O1   # Avoid a compiler bug in gcc 4.2.1 build 5658
   179         -    -DSQLITE_OMIT_LOAD_EXTENSION=1
   180         -    -DSQLITE_DEFAULT_MEMSTATUS=0
   181         -    -DSQLITE_THREADSAFE=2
   182         -    -DSQLITE_OS_UNIX=1
   183         -    -DSQLITE_ENABLE_JSON1=1
   184         -    -DSQLITE_ENABLE_LOCKING_STYLE=1
   185         -    -DUSE_PREAD=1
   186         -    -DSQLITE_ENABLE_RTREE=1
          179  +    -DHAVE_GMTIME_R=1
          180  +    -DHAVE_ISNAN=1
          181  +    -DHAVE_LOCALTIME_R=1
          182  +    -DHAVE_PREAD=1
          183  +    -DHAVE_PWRITE=1
          184  +    -DHAVE_USLEEP=1
          185  +    -DHAVE_USLEEP=1
          186  +    -DHAVE_UTIME=1
          187  +    -DSQLITE_DEFAULT_CACHE_SIZE=1000
          188  +    -DSQLITE_DEFAULT_CKPTFULLFSYNC=1
          189  +    -DSQLITE_DEFAULT_MEMSTATUS=1
          190  +    -DSQLITE_DEFAULT_PAGE_SIZE=1024
          191  +    -DSQLITE_DISABLE_PAGECACHE_OVERFLOW_STATS=1
          192  +    -DSQLITE_ENABLE_API_ARMOR=1
          193  +    -DSQLITE_ENABLE_AUTO_PROFILE=1
          194  +    -DSQLITE_ENABLE_FLOCKTIMEOUT=1
   187    195       -DSQLITE_ENABLE_FTS3=1
   188    196       -DSQLITE_ENABLE_FTS3_PARENTHESIS=1
   189         -    -DSQLITE_DEFAULT_CACHE_SIZE=1000
          197  +    -DSQLITE_ENABLE_FTS3_TOKENIZER=1
          198  +    if:os=="Darwin" -DSQLITE_ENABLE_LOCKING_STYLE=1
          199  +    -DSQLITE_ENABLE_PERSIST_WAL=1
          200  +    -DSQLITE_ENABLE_PURGEABLE_PCACHE=1
          201  +    -DSQLITE_ENABLE_RTREE=1
          202  +    -DSQLITE_ENABLE_SNAPSHOT=1
          203  +    # -DSQLITE_ENABLE_SQLLOG=1
          204  +    -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT=1
   190    205       -DSQLITE_MAX_LENGTH=2147483645
   191    206       -DSQLITE_MAX_VARIABLE_NUMBER=500000
   192         -    -DSQLITE_DEBUG=1
          207  +    # -DSQLITE_MEMDEBUG=1
          208  +    -DSQLITE_NO_SYNC=1
          209  +    -DSQLITE_OMIT_AUTORESET=1
          210  +    -DSQLITE_OMIT_LOAD_EXTENSION=1
   193    211       -DSQLITE_PREFER_PROXY_LOCKING=1
   194         -    -DSQLITE_ENABLE_API_ARMOR=1
          212  +    -DSQLITE_SERIES_CONSTRAINT_VERIFY=1
          213  +    -DSQLITE_THREADSAFE=2
          214  +    -DSQLITE_USE_URI=1
          215  +    -DSQLITE_WRITE_WALFRAME_PREBUFFERED=1
          216  +    -DUSE_GUARDED_FD=1
          217  +    -DUSE_PREAD=1
   195    218       --enable-json1 --enable-fts5
   196    219     }
   197    220     "Extra-Robustness" {
   198    221       -DSQLITE_ENABLE_OVERSIZE_CELL_CHECK=1
   199    222       -DSQLITE_MAX_ATTACHED=62
   200    223     }
   201    224     "Devkit" {
................................................................................
   244    267       "Secure-Delete"           test
   245    268       "Unlock-Notify"           "QUICKTEST_INCLUDE=notify2.test test"
   246    269       "Update-Delete-Limit"     test
   247    270       "Extra-Robustness"        test
   248    271       "Device-Two"              test
   249    272       "No-lookaside"            test
   250    273       "Devkit"                  test
          274  +    "Apple"                   test
   251    275       "Sanitize"                {QUICKTEST_OMIT=func4.test,nan.test test}
   252    276       "Device-One"              fulltest
   253    277       "Default"                 "threadtest fulltest"
   254    278       "Valgrind"                valgrindtest
   255    279     }
   256    280     Linux-i686 {
   257    281       "Devkit"                  test
................................................................................
   260    284       "Device-One"              test
   261    285       "Device-Two"              test
   262    286       "Default"                 "threadtest fulltest"
   263    287     }
   264    288     Darwin-i386 {
   265    289       "Locking-Style"           "mptest test"
   266    290       "Have-Not"                test
   267         -    "OS-X"                    "threadtest fulltest"
          291  +    "Apple"                   "threadtest fulltest"
   268    292     }
   269    293     Darwin-x86_64 {
   270    294       "Locking-Style"           "mptest test"
   271    295       "Have-Not"                test
   272         -    "OS-X"                    "threadtest fulltest"
          296  +    "Apple"                   "threadtest fulltest"
   273    297     }
   274    298     "Windows NT-intel" {
   275    299       "Have-Not"                test
   276    300       "Default"                 "mptest fulltestonly"
   277    301     }
   278    302     "Windows NT-amd64" {
   279    303       "Have-Not"                test
................................................................................
   584    608     # CFLAGS is only passed to gcc.
   585    609     #
   586    610     set makeOpts ""
   587    611     set cflags [expr {$::MSVC ? "-Zi" : "-g"}]
   588    612     set opts ""
   589    613     set title ${name}($testtarget)
   590    614     set configOpts $::WITHTCL
          615  +  set skip 0
   591    616   
   592    617     regsub -all {#[^\n]*\n} $config \n config
   593    618     foreach arg $config {
          619  +    if {$skip} {
          620  +      set skip 0
          621  +      continue
          622  +    }
   594    623       if {[regexp {^-[UD]} $arg]} {
   595    624         lappend opts $arg
   596    625       } elseif {[regexp {^[A-Z]+=} $arg]} {
   597    626         lappend testtarget $arg
          627  +    } elseif {[regexp {^if:([a-z]+)(.*)} $arg all key tail]} {
          628  +      # Arguments of the form 'if:os=="Linux"' will cause the subsequent
          629  +      # argument to be skipped if the $tcl_platform(os) is not "Linux", for
          630  +      # example...
          631  +      set skip [expr !(\$::tcl_platform($key)$tail)]
   598    632       } elseif {[regexp {^--(enable|disable)-} $arg]} {
   599    633         if {$::MSVC} {
   600    634           if {$arg eq "--disable-amalgamation"} {
   601    635             lappend makeOpts USE_AMALGAMATION=0
   602    636             continue
   603    637           }
   604    638           if {$arg eq "--disable-shared"} {

Changes to test/sqldiff1.test.

    10     10   #***********************************************************************
    11     11   #
    12     12   # Quick tests for the sqldiff tool
    13     13   #
    14     14   set testdir [file dirname $argv0]
    15     15   source $testdir/tester.tcl
    16     16   
    17         -if {$tcl_platform(platform)=="windows"} {
    18         -  set PROG "sqldiff.exe"
    19         -} else {
    20         -  set PROG "./sqldiff"
    21         -}
    22         -if {![file exe $PROG]} {
    23         -  puts "sqldiff cannot run because $PROG is not available"
    24         -  finish_test
    25         -  return
    26         -}
           17  +set PROG [test_find_sqldiff]
           18  +
    27     19   db close
    28     20   forcedelete test.db test2.db
    29     21   sqlite3 db test.db
    30     22   
    31     23   do_test sqldiff-1.0 {
    32     24     db eval {
    33     25       CREATE TABLE t1(a INTEGER PRIMARY KEY, b);

Changes to test/tester.tcl.

  2137   2137     sqlite3_shutdown
  2138   2138     eval sqlite3_config_pagecache $::old_pagecache_config
  2139   2139     unset ::old_pagecache_config 
  2140   2140     sqlite3_initialize
  2141   2141     autoinstall_test_functions
  2142   2142     sqlite3 db test.db
  2143   2143   }
         2144  +
         2145  +proc test_find_binary {nm} {
         2146  +  if {$::tcl_platform(platform)=="windows"} {
         2147  +    set ret "$nm.exe"
         2148  +  } else {
         2149  +    set ret $nm
         2150  +  }
         2151  +  set ret [file normalize [file join $::cmdlinearg(TESTFIXTURE_HOME) $ret]]
         2152  +  if {![file executable $ret]} {
         2153  +    finish_test
         2154  +    return ""
         2155  +  }
         2156  +  return $ret
         2157  +}
  2144   2158   
  2145   2159   # Find the name of the 'shell' executable (e.g. "sqlite3.exe") to use for
  2146   2160   # the tests in shell[1-5].test. If no such executable can be found, invoke
  2147   2161   # [finish_test ; return] in the callers context.
  2148   2162   #
  2149   2163   proc test_find_cli {} {
  2150         -  if {$::tcl_platform(platform)=="windows"} {
  2151         -    set ret "sqlite3.exe"
  2152         -  } else {
  2153         -    set ret "sqlite3"
  2154         -  }
  2155         -  set ret [file normalize [file join $::cmdlinearg(TESTFIXTURE_HOME) $ret]]
  2156         -  if {![file executable $ret]} {
  2157         -    finish_test
  2158         -    return -code return
  2159         -  }
  2160         -  return $ret
         2164  +  set prog [test_find_binary sqlite3]
         2165  +  if {$prog==""} { return -code return }
         2166  +  return $prog
         2167  +}
         2168  +
         2169  +# Find the name of the 'sqldiff' executable (e.g. "sqlite3.exe") to use for
         2170  +# the tests in sqldiff tests. If no such executable can be found, invoke
         2171  +# [finish_test ; return] in the callers context.
         2172  +#
         2173  +proc test_find_sqldiff {} {
         2174  +  set prog [test_find_binary sqldiff]
         2175  +  if {$prog==""} { return -code return }
         2176  +  return $prog
  2161   2177   }
         2178  +
  2162   2179   
  2163   2180   # If the library is compiled with the SQLITE_DEFAULT_AUTOVACUUM macro set
  2164   2181   # to non-zero, then set the global variable $AUTOVACUUM to 1.
  2165   2182   set AUTOVACUUM $sqlite_options(default_autovacuum)
  2166   2183   
  2167   2184   # Make sure the FTS enhanced query syntax is disabled.
  2168   2185   set sqlite_fts3_enable_parentheses 0

Changes to test/wal2.test.

  1190   1190   # Test that "PRAGMA checkpoint_fullsync" appears to be working.
  1191   1191   #
  1192   1192   foreach {tn sql reslist} {
  1193   1193     1 { }                                 {10 0 4 0 6 0}
  1194   1194     2 { PRAGMA checkpoint_fullfsync = 1 } {10 4 4 2 6 2}
  1195   1195     3 { PRAGMA checkpoint_fullfsync = 0 } {10 0 4 0 6 0}
  1196   1196   } {
         1197  +  ifcapable default_ckptfullfsync {
         1198  +    if {[string trim $sql]==""} continue
         1199  +  }
  1197   1200     faultsim_delete_and_reopen
  1198   1201   
  1199   1202     execsql {PRAGMA auto_vacuum = 0; PRAGMA synchronous = FULL;}
  1200   1203     execsql $sql
  1201   1204     do_execsql_test wal2-14.$tn.0 { PRAGMA page_size = 4096 }   {}
  1202   1205     do_execsql_test wal2-14.$tn.1 { PRAGMA journal_mode = WAL } {wal}
  1203   1206   

Changes to test/wal3.test.

   216    216     
   217    217       testvfs T
   218    218       T filter {} 
   219    219       T script sync_counter
   220    220       sqlite3 db test.db -vfs T
   221    221     
   222    222       execsql "PRAGMA synchronous = $syncmode"
          223  +    execsql "PRAGMA checkpoint_fullfsync = 0"
   223    224       execsql { PRAGMA journal_mode = WAL }
   224    225       execsql { CREATE TABLE filler(a,b,c); }
   225    226   
   226    227       set ::syncs [list]
   227    228       T filter xSync
   228    229       execsql {
   229    230         CREATE TABLE x(y);

Changes to tool/lemon.c.

   286    286     const char **rhsalias;   /* An alias for each RHS symbol (NULL if none) */
   287    287     int line;                /* Line number at which code begins */
   288    288     const char *code;        /* The code executed when this rule is reduced */
   289    289     const char *codePrefix;  /* Setup code before code[] above */
   290    290     const char *codeSuffix;  /* Breakdown code after code[] above */
   291    291     struct symbol *precsym;  /* Precedence symbol for this rule */
   292    292     int index;               /* An index number for this rule */
          293  +  int iRule;               /* Rule number as used in the generated tables */
   293    294     Boolean canReduce;       /* True if this rule is ever reduced */
   294    295     struct rule *nextlhs;    /* Next rule with the same LHS */
   295    296     struct rule *next;       /* Next rule in the global list */
   296    297   };
   297    298   
   298    299   /* A configuration is a production rule of the grammar together with
   299    300   ** a mark (dot) showing how much of that rule has been processed so far.
................................................................................
   368    369   /* The state vector for the entire parser generator is recorded as
   369    370   ** follows.  (LEMON uses no global variables and makes little use of
   370    371   ** static variables.  Fields in the following structure can be thought
   371    372   ** of as begin global variables in the program.) */
   372    373   struct lemon {
   373    374     struct state **sorted;   /* Table of states sorted by state number */
   374    375     struct rule *rule;       /* List of all rules */
          376  +  struct rule *startRule;  /* First rule */
   375    377     int nstate;              /* Number of states */
   376    378     int nxstate;             /* nstate with tail degenerate states removed */
   377    379     int nrule;               /* Number of rules */
   378    380     int nsymbol;             /* Number of terminal and nonterminal symbols */
   379    381     int nterminal;           /* Number of terminal symbols */
   380    382     struct symbol **symbols; /* Sorted array of pointers to symbols */
   381    383     int errorcnt;            /* Number of errors */
................................................................................
   854    856     /* Find the start symbol */
   855    857     if( lemp->start ){
   856    858       sp = Symbol_find(lemp->start);
   857    859       if( sp==0 ){
   858    860         ErrorMsg(lemp->filename,0,
   859    861   "The specified start symbol \"%s\" is not \
   860    862   in a nonterminal of the grammar.  \"%s\" will be used as the start \
   861         -symbol instead.",lemp->start,lemp->rule->lhs->name);
          863  +symbol instead.",lemp->start,lemp->startRule->lhs->name);
   862    864         lemp->errorcnt++;
   863         -      sp = lemp->rule->lhs;
          865  +      sp = lemp->startRule->lhs;
   864    866       }
   865    867     }else{
   866         -    sp = lemp->rule->lhs;
          868  +    sp = lemp->startRule->lhs;
   867    869     }
   868    870   
   869    871     /* Make sure the start symbol doesn't occur on the right-hand side of
   870    872     ** any rule.  Report an error if it does.  (YACC would generate a new
   871    873     ** start symbol in this case.) */
   872    874     for(rp=lemp->rule; rp; rp=rp->next){
   873    875       int i;
................................................................................
  1113   1115         }
  1114   1116       }
  1115   1117     }
  1116   1118   
  1117   1119     /* Add the accepting token */
  1118   1120     if( lemp->start ){
  1119   1121       sp = Symbol_find(lemp->start);
  1120         -    if( sp==0 ) sp = lemp->rule->lhs;
         1122  +    if( sp==0 ) sp = lemp->startRule->lhs;
  1121   1123     }else{
  1122         -    sp = lemp->rule->lhs;
         1124  +    sp = lemp->startRule->lhs;
  1123   1125     }
  1124   1126     /* Add to the first state (which is always the starting state of the
  1125   1127     ** finite state machine) an action to ACCEPT if the lookahead is the
  1126   1128     ** start nonterminal.  */
  1127   1129     Action_add(&lemp->sorted[0]->ap,ACCEPT,sp,0);
  1128   1130   
  1129   1131     /* Resolve conflicts */
................................................................................
  1492   1494   static void handle_T_option(char *z){
  1493   1495     user_templatename = (char *) malloc( lemonStrlen(z)+1 );
  1494   1496     if( user_templatename==0 ){
  1495   1497       memory_error();
  1496   1498     }
  1497   1499     lemon_strcpy(user_templatename, z);
  1498   1500   }
         1501  +
         1502  +/* Merge together to lists of rules order by rule.iRule */
         1503  +static struct rule *Rule_merge(struct rule *pA, struct rule *pB){
         1504  +  struct rule *pFirst = 0;
         1505  +  struct rule **ppPrev = &pFirst;
         1506  +  while( pA && pB ){
         1507  +    if( pA->iRule<pB->iRule ){
         1508  +      *ppPrev = pA;
         1509  +      ppPrev = &pA->next;
         1510  +      pA = pA->next;
         1511  +    }else{
         1512  +      *ppPrev = pB;
         1513  +      ppPrev = &pB->next;
         1514  +      pB = pB->next;
         1515  +    }
         1516  +  }
         1517  +  if( pA ){
         1518  +    *ppPrev = pA;
         1519  +  }else{
         1520  +    *ppPrev = pB;
         1521  +  }
         1522  +  return pFirst;
         1523  +}
         1524  +
         1525  +/*
         1526  +** Sort a list of rules in order of increasing iRule value
         1527  +*/
         1528  +static struct rule *Rule_sort(struct rule *rp){
         1529  +  int i;
         1530  +  struct rule *pNext;
         1531  +  struct rule *x[32];
         1532  +  memset(x, 0, sizeof(x));
         1533  +  while( rp ){
         1534  +    pNext = rp->next;
         1535  +    rp->next = 0;
         1536  +    for(i=0; i<sizeof(x)/sizeof(x[0]) && x[i]; i++){
         1537  +      rp = Rule_merge(x[i], rp);
         1538  +      x[i] = 0;
         1539  +    }
         1540  +    x[i] = rp;
         1541  +    rp = pNext;
         1542  +  }
         1543  +  rp = 0;
         1544  +  for(i=0; i<sizeof(x)/sizeof(x[0]); i++){
         1545  +    rp = Rule_merge(x[i], rp);
         1546  +  }
         1547  +  return rp;
         1548  +}
  1499   1549   
  1500   1550   /* forward reference */
  1501   1551   static const char *minimum_size_type(int lwr, int upr, int *pnByte);
  1502   1552   
  1503   1553   /* Print a single line of the "Parser Stats" output
  1504   1554   */
  1505   1555   static void stats_line(const char *zLabel, int iValue){
................................................................................
  1541   1591       {OPT_FSTR, "T", (char*)handle_T_option, "Specify a template file."},
  1542   1592       {OPT_FSTR, "W", 0, "Ignored.  (Placeholder for '-W' compiler options.)"},
  1543   1593       {OPT_FLAG,0,0,0}
  1544   1594     };
  1545   1595     int i;
  1546   1596     int exitcode;
  1547   1597     struct lemon lem;
         1598  +  struct rule *rp;
  1548   1599   
  1549   1600     OptInit(argv,options,stderr);
  1550   1601     if( version ){
  1551   1602        printf("Lemon version 1.0\n");
  1552   1603        exit(0); 
  1553   1604     }
  1554   1605     if( OptNArgs()!=1 ){
................................................................................
  1586   1637     qsort(lem.symbols,lem.nsymbol,sizeof(struct symbol*), Symbolcmpp);
  1587   1638     for(i=0; i<lem.nsymbol; i++) lem.symbols[i]->index = i;
  1588   1639     while( lem.symbols[i-1]->type==MULTITERMINAL ){ i--; }
  1589   1640     assert( strcmp(lem.symbols[i-1]->name,"{default}")==0 );
  1590   1641     lem.nsymbol = i - 1;
  1591   1642     for(i=1; ISUPPER(lem.symbols[i]->name[0]); i++);
  1592   1643     lem.nterminal = i;
         1644  +
         1645  +  /* Assign sequential rule numbers */
         1646  +  for(i=0, rp=lem.rule; rp; rp=rp->next){
         1647  +    rp->iRule = rp->code ? i++ : -1;
         1648  +  }
         1649  +  for(rp=lem.rule; rp; rp=rp->next){
         1650  +    if( rp->iRule<0 ) rp->iRule = i++;
         1651  +  }
         1652  +  lem.startRule = lem.rule;
         1653  +  lem.rule = Rule_sort(lem.rule);
  1593   1654   
  1594   1655     /* Generate a reprint of the grammar, if requested on the command line */
  1595   1656     if( rpflag ){
  1596   1657       Reprint(&lem);
  1597   1658     }else{
  1598   1659       /* Initialize the size for all follow and first sets */
  1599   1660       SetSize(lem.nterminal+1);
................................................................................
  3050   3111       case SHIFT: {
  3051   3112         struct state *stp = ap->x.stp;
  3052   3113         fprintf(fp,"%*s shift        %-7d",indent,ap->sp->name,stp->statenum);
  3053   3114         break;
  3054   3115       }
  3055   3116       case REDUCE: {
  3056   3117         struct rule *rp = ap->x.rp;
  3057         -      fprintf(fp,"%*s reduce       %-7d",indent,ap->sp->name,rp->index);
         3118  +      fprintf(fp,"%*s reduce       %-7d",indent,ap->sp->name,rp->iRule);
  3058   3119         RulePrint(fp, rp, -1);
  3059   3120         break;
  3060   3121       }
  3061   3122       case SHIFTREDUCE: {
  3062   3123         struct rule *rp = ap->x.rp;
  3063         -      fprintf(fp,"%*s shift-reduce %-7d",indent,ap->sp->name,rp->index);
         3124  +      fprintf(fp,"%*s shift-reduce %-7d",indent,ap->sp->name,rp->iRule);
  3064   3125         RulePrint(fp, rp, -1);
  3065   3126         break;
  3066   3127       }
  3067   3128       case ACCEPT:
  3068   3129         fprintf(fp,"%*s accept",indent,ap->sp->name);
  3069   3130         break;
  3070   3131       case ERROR:
  3071   3132         fprintf(fp,"%*s error",indent,ap->sp->name);
  3072   3133         break;
  3073   3134       case SRCONFLICT:
  3074   3135       case RRCONFLICT:
  3075   3136         fprintf(fp,"%*s reduce       %-7d ** Parsing conflict **",
  3076         -        indent,ap->sp->name,ap->x.rp->index);
         3137  +        indent,ap->sp->name,ap->x.rp->iRule);
  3077   3138         break;
  3078   3139       case SSCONFLICT:
  3079   3140         fprintf(fp,"%*s shift        %-7d ** Parsing conflict **", 
  3080   3141           indent,ap->sp->name,ap->x.stp->statenum);
  3081   3142         break;
  3082   3143       case SH_RESOLVED:
  3083   3144         if( showPrecedenceConflict ){
................................................................................
  3086   3147         }else{
  3087   3148           result = 0;
  3088   3149         }
  3089   3150         break;
  3090   3151       case RD_RESOLVED:
  3091   3152         if( showPrecedenceConflict ){
  3092   3153           fprintf(fp,"%*s reduce %-7d -- dropped by precedence",
  3093         -                indent,ap->sp->name,ap->x.rp->index);
         3154  +                indent,ap->sp->name,ap->x.rp->iRule);
  3094   3155         }else{
  3095   3156           result = 0;
  3096   3157         }
  3097   3158         break;
  3098   3159       case NOT_USED:
  3099   3160         result = 0;
  3100   3161         break;
................................................................................
  3117   3178       stp = lemp->sorted[i];
  3118   3179       fprintf(fp,"State %d:\n",stp->statenum);
  3119   3180       if( lemp->basisflag ) cfp=stp->bp;
  3120   3181       else                  cfp=stp->cfp;
  3121   3182       while( cfp ){
  3122   3183         char buf[20];
  3123   3184         if( cfp->dot==cfp->rp->nrhs ){
  3124         -        lemon_sprintf(buf,"(%d)",cfp->rp->index);
         3185  +        lemon_sprintf(buf,"(%d)",cfp->rp->iRule);
  3125   3186           fprintf(fp,"    %5s ",buf);
  3126   3187         }else{
  3127   3188           fprintf(fp,"          ");
  3128   3189         }
  3129   3190         ConfigPrint(fp,cfp);
  3130   3191         fprintf(fp,"\n");
  3131   3192   #if 0
................................................................................
  3218   3279   ** Return negative if no action should be generated.
  3219   3280   */
  3220   3281   PRIVATE int compute_action(struct lemon *lemp, struct action *ap)
  3221   3282   {
  3222   3283     int act;
  3223   3284     switch( ap->type ){
  3224   3285       case SHIFT:  act = ap->x.stp->statenum;                        break;
  3225         -    case SHIFTREDUCE: act = ap->x.rp->index + lemp->nstate;        break;
  3226         -    case REDUCE: act = ap->x.rp->index + lemp->nstate+lemp->nrule; break;
         3286  +    case SHIFTREDUCE: act = ap->x.rp->iRule + lemp->nstate;        break;
         3287  +    case REDUCE: act = ap->x.rp->iRule + lemp->nstate+lemp->nrule; break;
  3227   3288       case ERROR:  act = lemp->nstate + lemp->nrule*2;               break;
  3228   3289       case ACCEPT: act = lemp->nstate + lemp->nrule*2 + 1;           break;
  3229   3290       default:     act = -1; break;
  3230   3291     }
  3231   3292     return act;
  3232   3293   }
  3233   3294   
................................................................................
  4237   4298     tplt_xfer(lemp->name,in,out,&lineno);
  4238   4299   
  4239   4300     /* Generate a table containing a text string that describes every
  4240   4301     ** rule in the rule set of the grammar.  This information is used
  4241   4302     ** when tracing REDUCE actions.
  4242   4303     */
  4243   4304     for(i=0, rp=lemp->rule; rp; rp=rp->next, i++){
  4244         -    assert( rp->index==i );
         4305  +    assert( rp->iRule==i );
  4245   4306       fprintf(out," /* %3d */ \"", i);
  4246   4307       writeRuleText(out, rp);
  4247   4308       fprintf(out,"\",\n"); lineno++;
  4248   4309     }
  4249   4310     tplt_xfer(lemp->name,in,out,&lineno);
  4250   4311   
  4251   4312     /* Generate code which executes every time a symbol is popped from
................................................................................
  4333   4394       fprintf(out,"        YYMINORTYPE yylhsminor;\n"); lineno++;
  4334   4395     }
  4335   4396     /* First output rules other than the default: rule */
  4336   4397     for(rp=lemp->rule; rp; rp=rp->next){
  4337   4398       struct rule *rp2;               /* Other rules with the same action */
  4338   4399       if( rp->code==0 ) continue;
  4339   4400       if( rp->code[0]=='\n' && rp->code[1]==0 ) continue; /* Will be default: */
  4340         -    fprintf(out,"      case %d: /* ", rp->index);
         4401  +    fprintf(out,"      case %d: /* ", rp->iRule);
  4341   4402       writeRuleText(out, rp);
  4342   4403       fprintf(out, " */\n"); lineno++;
  4343   4404       for(rp2=rp->next; rp2; rp2=rp2->next){
  4344   4405         if( rp2->code==rp->code ){
  4345         -        fprintf(out,"      case %d: /* ", rp2->index);
         4406  +        fprintf(out,"      case %d: /* ", rp2->iRule);
  4346   4407           writeRuleText(out, rp2);
  4347         -        fprintf(out," */ yytestcase(yyruleno==%d);\n", rp2->index); lineno++;
         4408  +        fprintf(out," */ yytestcase(yyruleno==%d);\n", rp2->iRule); lineno++;
  4348   4409           rp2->code = 0;
  4349   4410         }
  4350   4411       }
  4351   4412       emit_code(out,rp,lemp,&lineno);
  4352   4413       fprintf(out,"        break;\n"); lineno++;
  4353   4414       rp->code = 0;
  4354   4415     }
  4355   4416     /* Finally, output the default: rule.  We choose as the default: all
  4356   4417     ** empty actions. */
  4357   4418     fprintf(out,"      default:\n"); lineno++;
  4358   4419     for(rp=lemp->rule; rp; rp=rp->next){
  4359   4420       if( rp->code==0 ) continue;
  4360   4421       assert( rp->code[0]=='\n' && rp->code[1]==0 );
  4361         -    fprintf(out,"      /* (%d) ", rp->index);
         4422  +    fprintf(out,"      /* (%d) ", rp->iRule);
  4362   4423       writeRuleText(out, rp);
  4363         -    fprintf(out, " */ yytestcase(yyruleno==%d);\n", rp->index); lineno++;
         4424  +    fprintf(out, " */ yytestcase(yyruleno==%d);\n", rp->iRule); lineno++;
  4364   4425     }
  4365   4426     fprintf(out,"        break;\n"); lineno++;
  4366   4427     tplt_xfer(lemp->name,in,out,&lineno);
  4367   4428   
  4368   4429     /* Generate code which executes if a parse fails */
  4369   4430     tplt_print(out,lemp,lemp->failure,&lineno);
  4370   4431     tplt_xfer(lemp->name,in,out,&lineno);

Changes to tool/lempar.c.

   414    414   }
   415    415   #endif
   416    416   
   417    417   /*
   418    418   ** Find the appropriate action for a parser given the terminal
   419    419   ** look-ahead token iLookAhead.
   420    420   */
   421         -static int yy_find_shift_action(
          421  +static unsigned int yy_find_shift_action(
   422    422     yyParser *pParser,        /* The parser */
   423    423     YYCODETYPE iLookAhead     /* The look-ahead token */
   424    424   ){
   425    425     int i;
   426    426     int stateno = pParser->yystack[pParser->yyidx].stateno;
   427    427    
   428    428     if( stateno>=YY_MIN_REDUCE ) return stateno;
................................................................................
   602    602   
   603    603   /*
   604    604   ** Perform a reduce action and the shift that must immediately
   605    605   ** follow the reduce.
   606    606   */
   607    607   static void yy_reduce(
   608    608     yyParser *yypParser,         /* The parser */
   609         -  int yyruleno                 /* Number of the rule by which to reduce */
          609  +  unsigned int yyruleno        /* Number of the rule by which to reduce */
   610    610   ){
   611    611     int yygoto;                     /* The next state */
   612    612     int yyact;                      /* The next action */
   613    613     yyStackEntry *yymsp;            /* The top of the parser's stack */
   614    614     int yysize;                     /* Amount to pop the stack */
   615    615     ParseARG_FETCH;
   616    616     yymsp = &yypParser->yystack[yypParser->yyidx];
   617    617   #ifndef NDEBUG
   618         -  if( yyTraceFILE && yyruleno>=0 
   619         -        && yyruleno<(int)(sizeof(yyRuleName)/sizeof(yyRuleName[0])) ){
          618  +  if( yyTraceFILE && yyruleno<(int)(sizeof(yyRuleName)/sizeof(yyRuleName[0])) ){
   620    619       yysize = yyRuleInfo[yyruleno].nrhs;
   621    620       fprintf(yyTraceFILE, "%sReduce [%s], go to state %d.\n", yyTracePrompt,
   622    621         yyRuleName[yyruleno], yymsp[-yysize].stateno);
   623    622     }
   624    623   #endif /* NDEBUG */
   625    624   
   626    625     /* Check that the stack is large enough to grow by a single entry
................................................................................
   657    656     **  #line <lineno> <thisfile>
   658    657     **     break;
   659    658     */
   660    659   /********** Begin reduce actions **********************************************/
   661    660   %%
   662    661   /********** End reduce actions ************************************************/
   663    662     };
   664         -  assert( yyruleno>=0 && yyruleno<sizeof(yyRuleInfo)/sizeof(yyRuleInfo[0]) );
          663  +  assert( yyruleno<sizeof(yyRuleInfo)/sizeof(yyRuleInfo[0]) );
   665    664     yygoto = yyRuleInfo[yyruleno].lhs;
   666    665     yysize = yyRuleInfo[yyruleno].nrhs;
   667    666     yyact = yy_find_reduce_action(yymsp[-yysize].stateno,(YYCODETYPE)yygoto);
   668    667     if( yyact <= YY_MAX_SHIFTREDUCE ){
   669    668       if( yyact>YY_MAX_SHIFT ) yyact += YY_MIN_REDUCE - YY_MIN_SHIFTREDUCE;
   670    669       yypParser->yyidx -= yysize - 1;
   671    670       yymsp -= yysize-1;
................................................................................
   761    760   void Parse(
   762    761     void *yyp,                   /* The parser */
   763    762     int yymajor,                 /* The major token code number */
   764    763     ParseTOKENTYPE yyminor       /* The value for the token */
   765    764     ParseARG_PDECL               /* Optional %extra_argument parameter */
   766    765   ){
   767    766     YYMINORTYPE yyminorunion;
   768         -  int yyact;            /* The parser action. */
          767  +  unsigned int yyact;   /* The parser action. */
   769    768   #if !defined(YYERRORSYMBOL) && !defined(YYNOERRORRECOVERY)
   770    769     int yyendofinput;     /* True if we are at the end of input */
   771    770   #endif
   772    771   #ifdef YYERRORSYMBOL
   773    772     int yyerrorhit = 0;   /* True if yymajor has invoked an error */
   774    773   #endif
   775    774     yyParser *yypParser;  /* The parser */

Changes to tool/sqldiff.c.

  1240   1240     char **azCol;                   /* NULL terminated array of col names */
  1241   1241     int i;
  1242   1242     int nCol;
  1243   1243     Str ct = {0, 0, 0};             /* The "CREATE TABLE data_xxx" statement */
  1244   1244     Str sql = {0, 0, 0};            /* Query to find differences */
  1245   1245     Str insert = {0, 0, 0};         /* First part of output INSERT statement */
  1246   1246     sqlite3_stmt *pStmt = 0;
         1247  +  int nRow = 0;                   /* Total rows in data_xxx table */
  1247   1248   
  1248   1249     /* --rbu mode must use real primary keys. */
  1249   1250     g.bSchemaPK = 1;
  1250   1251   
  1251   1252     /* Check that the schemas of the two tables match. Exit early otherwise. */
  1252   1253     checkSchemasMatch(zTab);
  1253   1254   
................................................................................
  1285   1286       if( ct.z ){
  1286   1287         fprintf(out, "%s\n", ct.z);
  1287   1288         strFree(&ct);
  1288   1289       }
  1289   1290   
  1290   1291       /* Output the first part of the INSERT statement */
  1291   1292       fprintf(out, "%s", insert.z);
         1293  +    nRow++;
  1292   1294   
  1293   1295       if( sqlite3_column_type(pStmt, nCol)==SQLITE_INTEGER ){
  1294   1296         for(i=0; i<=nCol; i++){
  1295   1297           if( i>0 ) fprintf(out, ", ");
  1296   1298           printQuoted(out, sqlite3_column_value(pStmt, i));
  1297   1299         }
  1298   1300       }else{
................................................................................
  1338   1340       }
  1339   1341   
  1340   1342       /* And the closing bracket of the insert statement */
  1341   1343       fprintf(out, ");\n");
  1342   1344     }
  1343   1345   
  1344   1346     sqlite3_finalize(pStmt);
         1347  +  if( nRow>0 ){
         1348  +    Str cnt = {0, 0, 0};
         1349  +    strPrintf(&cnt, "INSERT INTO rbu_count VALUES('data_%q', %d);", zTab, nRow);
         1350  +    fprintf(out, "%s\n", cnt.z);
         1351  +    strFree(&cnt);
         1352  +  }
  1345   1353   
  1346   1354     strFree(&ct);
  1347   1355     strFree(&sql);
  1348   1356     strFree(&insert);
  1349   1357   }
  1350   1358   
  1351   1359   /*
................................................................................
  1753   1761     int rc;
  1754   1762     char *zErrMsg = 0;
  1755   1763     char *zSql;
  1756   1764     sqlite3_stmt *pStmt;
  1757   1765     char *zTab = 0;
  1758   1766     FILE *out = stdout;
  1759   1767     void (*xDiff)(const char*,FILE*) = diff_one_table;
         1768  +#ifndef SQLITE_OMIT_LOAD_EXTENSION
  1760   1769     int nExt = 0;
  1761   1770     char **azExt = 0;
         1771  +#endif
  1762   1772     int useTransaction = 0;
  1763   1773     int neverUseTransaction = 0;
  1764   1774   
  1765   1775     g.zArgv0 = argv[0];
  1766   1776     sqlite3_config(SQLITE_CONFIG_SINGLETHREAD);
  1767   1777     for(i=1; i<argc; i++){
  1768   1778       const char *z = argv[i];
................................................................................
  1837   1847     sqlite3_enable_load_extension(g.db, 1);
  1838   1848     for(i=0; i<nExt; i++){
  1839   1849       rc = sqlite3_load_extension(g.db, azExt[i], 0, &zErrMsg);
  1840   1850       if( rc || zErrMsg ){
  1841   1851         cmdlineError("error loading %s: %s", azExt[i], zErrMsg);
  1842   1852       }
  1843   1853     }
  1844         -#endif
  1845   1854     free(azExt);
         1855  +#endif
  1846   1856     zSql = sqlite3_mprintf("ATTACH %Q as aux;", zDb2);
  1847   1857     rc = sqlite3_exec(g.db, zSql, 0, 0, &zErrMsg);
  1848   1858     if( rc || zErrMsg ){
  1849   1859       cmdlineError("cannot attach database \"%s\"", zDb2);
  1850   1860     }
  1851   1861     rc = sqlite3_exec(g.db, "SELECT * FROM aux.sqlite_master", 0, 0, &zErrMsg);
  1852   1862     if( rc || zErrMsg ){
  1853   1863       cmdlineError("\"%s\" does not appear to be a valid SQLite database", zDb2);
  1854   1864     }
  1855   1865   
  1856   1866     if( neverUseTransaction ) useTransaction = 0;
  1857         -  if( useTransaction ) printf("BEGIN TRANSACTION;\n");
         1867  +  if( useTransaction ) fprintf(out, "BEGIN TRANSACTION;\n");
         1868  +  if( xDiff==rbudiff_one_table ){
         1869  +    fprintf(out, "CREATE TABLE IF NOT EXISTS rbu_count"
         1870  +           "(tbl TEXT PRIMARY KEY COLLATE NOCASE, cnt INTEGER) "
         1871  +           "WITHOUT ROWID;\n"
         1872  +    );
         1873  +  }
  1858   1874     if( zTab ){
  1859   1875       xDiff(zTab, out);
  1860   1876     }else{
  1861   1877       /* Handle tables one by one */
  1862   1878       pStmt = db_prepare(
  1863   1879         "SELECT name FROM main.sqlite_master\n"
  1864   1880         " WHERE type='table' AND sql NOT LIKE 'CREATE VIRTUAL%%'\n"