SQLite

Check-in [9570b6b43d]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Minor simplification to the tokenizer. Slightly smaller and faster.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 9570b6b43df3bc5ce314cded20bca8be9e968efe
User & Date: drh 2016-02-16 13:04:19.100
References
2016-02-18
14:49
Avoid a potential buffer overrun if an SQL statement being parsed ends with an illegal "!" token. (This problem was detected by fuzzcheck running under valgrind. The problem was introduced by check-in [9570b6b43df3], two days ago and has not appeared in a release.) (check-in: 2a8d97e7c8 user: drh tags: trunk)
Context
2016-02-17
11:13
Documentation typo on sqlite3_wal_hook(). No changes to code. (check-in: fc2043ebaa user: drh tags: trunk)
2016-02-16
21:19
Experimental changes to Lemon for improved parser performance. (check-in: a65d583ce9 user: drh tags: parser-performance)
13:04
Minor simplification to the tokenizer. Slightly smaller and faster. (check-in: 9570b6b43d user: drh tags: trunk)
01:01
Improve the Lemon parser template (lempar.c) so that it avoids unnecessary work when the grammer defines YYNOERRORRECOVERY (as SQLite does). Slightly smaller and faster code results. (check-in: 9235b0cf6a user: drh tags: trunk)
Changes
Unified Diff Ignore Whitespace Patch
Changes to src/tokenize.c.
︙︙
519
520
521
522
523
524
525
526
527
528
529
530
531
532

533
534
535
536
537
538
539
540
541
542
543
544

      }
      if( tokenType==TK_ILLEGAL ){
        sqlite3ErrorMsg(pParse, "unrecognized token: \"%T\"",
                        &pParse->sLastToken);
        break;
      }
    }else{
      if( tokenType==TK_SEMI ) pParse->zTail = &zSql[i];
      sqlite3Parser(pEngine, tokenType, pParse->sLastToken, pParse);
      lastTokenParsed = tokenType;
      if( pParse->rc!=SQLITE_OK || db->mallocFailed ) break;
    }
  }
  assert( nErr==0 );

  if( pParse->rc==SQLITE_OK && db->mallocFailed==0 ){
    assert( zSql[i]==0 );
    if( lastTokenParsed!=TK_SEMI ){
      sqlite3Parser(pEngine, TK_SEMI, pParse->sLastToken, pParse);
      pParse->zTail = &zSql[i];
    }
    if( pParse->rc==SQLITE_OK && db->mallocFailed==0 ){
      sqlite3Parser(pEngine, 0, pParse->sLastToken, pParse);
    }
  }
#ifdef YYTRACKMAXSTACKDEPTH
  sqlite3_mutex_enter(sqlite3MallocMutex());








<






>




<








519
520
521
522
523
524
525

526
527
528
529
530
531
532
533
534
535
536

537
538
539
540
541
542
543

      }
      if( tokenType==TK_ILLEGAL ){
        sqlite3ErrorMsg(pParse, "unrecognized token: \"%T\"",
                        &pParse->sLastToken);
        break;
      }
    }else{

      sqlite3Parser(pEngine, tokenType, pParse->sLastToken, pParse);
      lastTokenParsed = tokenType;
      if( pParse->rc!=SQLITE_OK || db->mallocFailed ) break;
    }
  }
  assert( nErr==0 );
  pParse->zTail = &zSql[i];
  if( pParse->rc==SQLITE_OK && db->mallocFailed==0 ){
    assert( zSql[i]==0 );
    if( lastTokenParsed!=TK_SEMI ){
      sqlite3Parser(pEngine, TK_SEMI, pParse->sLastToken, pParse);

    }
    if( pParse->rc==SQLITE_OK && db->mallocFailed==0 ){
      sqlite3Parser(pEngine, 0, pParse->sLastToken, pParse);
    }
  }
#ifdef YYTRACKMAXSTACKDEPTH
  sqlite3_mutex_enter(sqlite3MallocMutex());
︙︙