SQLite: Check-in [9570b6b4]

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview

Comment:	Minor simplification to the tokenizer. Slightly smaller and faster.
Downloads:	Tarball \| ZIP archive \| SQL archive
Timelines:	family \| ancestors \| descendants \| both \| trunk
Files:	files \| file ages \| folders
SHA1:	9570b6b43df3bc5ce314cded20bca8be9e968efe
User & Date:	drh 2016-02-16 13:04:19

References

2016-02-18
14:49		Avoid a potential buffer overrun if an SQL statement being parsed ends with an illegal "!" token. (This problem was detected by fuzzcheck running under valgrind. The problem was introduced by check-in [9570b6b43df3], two days ago and has not appeared in a release.) (check-in: 2a8d97e7 user: drh tags: trunk)

Context

2016-02-17
11:13		Documentation typo on sqlite3_wal_hook(). No changes to code. (check-in: fc2043eb user: drh tags: trunk)
2016-02-16
21:19		Experimental changes to Lemon for improved parser performance. (check-in: a65d583c user: drh tags: parser-performance)
13:04		Minor simplification to the tokenizer. Slightly smaller and faster. (check-in: 9570b6b4 user: drh tags: trunk)
01:01		Improve the Lemon parser template (lempar.c) so that it avoids unnecessary work when the grammer defines YYNOERRORRECOVERY (as SQLite does). Slightly smaller and faster code results. (check-in: 9235b0cf user: drh tags: trunk)

Changes

Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Changes to src/tokenize.c.

   519    519         }
   520    520         if( tokenType==TK_ILLEGAL ){
   521    521           sqlite3ErrorMsg(pParse, "unrecognized token: \"%T\"",
   522    522                           &pParse->sLastToken);
   523    523           break;
   524    524         }
   525    525       }else{
   526         -      if( tokenType==TK_SEMI ) pParse->zTail = &zSql[i];
   527    526         sqlite3Parser(pEngine, tokenType, pParse->sLastToken, pParse);
   528    527         lastTokenParsed = tokenType;
   529    528         if( pParse->rc!=SQLITE_OK || db->mallocFailed ) break;
   530    529       }
   531    530     }
   532    531     assert( nErr==0 );
          532  +  pParse->zTail = &zSql[i];
   533    533     if( pParse->rc==SQLITE_OK && db->mallocFailed==0 ){
   534    534       assert( zSql[i]==0 );
   535    535       if( lastTokenParsed!=TK_SEMI ){
   536    536         sqlite3Parser(pEngine, TK_SEMI, pParse->sLastToken, pParse);
   537         -      pParse->zTail = &zSql[i];
   538    537       }
   539    538       if( pParse->rc==SQLITE_OK && db->mallocFailed==0 ){
   540    539         sqlite3Parser(pEngine, 0, pParse->sLastToken, pParse);
   541    540       }
   542    541     }
   543    542   #ifdef YYTRACKMAXSTACKDEPTH
   544    543     sqlite3_mutex_enter(sqlite3MallocMutex());