Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
Comment: | Fix a case where during a rollback triggered by an IO or malloc error an unjournalled region of the database could be written to (with it's original data). This was causing an assert in test_journal.c to fail. Add a test case in ioerr2.test to trigger this case. (CVS 6286) |
---|---|
Downloads: | Tarball | ZIP archive |
Timelines: | family | ancestors | descendants | both | trunk |
Files: | files | file ages | folders |
SHA1: |
315a6692f9a03a470871cce4f7456768 |
User & Date: | danielk1977 2009-02-12 09:11:56.000 |
Context
2009-02-12
| ||
09:36 | Modify lock.test to account for "PRAGMA lock_status" returning "unknown" for in-memory databases. (CVS 6287) (check-in: 9a6e558ba6 user: danielk1977 tags: trunk) | |
09:11 | Fix a case where during a rollback triggered by an IO or malloc error an unjournalled region of the database could be written to (with it's original data). This was causing an assert in test_journal.c to fail. Add a test case in ioerr2.test to trigger this case. (CVS 6286) (check-in: 315a6692f9 user: danielk1977 tags: trunk) | |
2009-02-11
| ||
16:06 | Updated for Windows compatibility. Test scripts only. (CVS 6285) (check-in: 2522ad1df3 user: shane tags: trunk) | |
Changes
Changes to src/pager.c.
︙ | ︙ | |||
14 15 16 17 18 19 20 | ** The pager is used to access a database disk file. It implements ** atomic commit and rollback through the use of a journal file that ** is separate from the database file. The pager also implements file ** locking to prevent two processes from writing the same database ** file simultaneously, or one process from reading the database while ** another is writing. ** | | | 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | ** The pager is used to access a database disk file. It implements ** atomic commit and rollback through the use of a journal file that ** is separate from the database file. The pager also implements file ** locking to prevent two processes from writing the same database ** file simultaneously, or one process from reading the database while ** another is writing. ** ** @(#) $Id: pager.c,v 1.567 2009/02/12 09:11:56 danielk1977 Exp $ */ #ifndef SQLITE_OMIT_DISKIO #include "sqliteInt.h" /* ** Macros for troubleshooting. Normally turned off */ |
︙ | ︙ | |||
1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 | ** If this is a savepoint rollback, then memory may have to be dynamically ** allocated by this function. If this is the case and an allocation fails, ** SQLITE_NOMEM is returned. */ static int pager_playback_one_page( Pager *pPager, /* The pager being played back */ int isMainJrnl, /* 1 -> main journal. 0 -> sub-journal. */ i64 *pOffset, /* Offset of record to playback */ int isSavepnt, /* True for a savepoint rollback */ Bitvec *pDone /* Bitvec of pages already played back */ ){ int rc; PgHdr *pPg; /* An existing page in the cache */ Pgno pgno; /* The page number of a page in journal */ | > | 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 | ** If this is a savepoint rollback, then memory may have to be dynamically ** allocated by this function. If this is the case and an allocation fails, ** SQLITE_NOMEM is returned. */ static int pager_playback_one_page( Pager *pPager, /* The pager being played back */ int isMainJrnl, /* 1 -> main journal. 0 -> sub-journal. */ int isUnsync, /* True if reading from unsynced main journal */ i64 *pOffset, /* Offset of record to playback */ int isSavepnt, /* True for a savepoint rollback */ Bitvec *pDone /* Bitvec of pages already played back */ ){ int rc; PgHdr *pPg; /* An existing page in the cache */ Pgno pgno; /* The page number of a page in journal */ |
︙ | ︙ | |||
1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 | PAGERTRACE(("PLAYBACK %d page %d hash(%08x) %s\n", PAGERID(pPager), pgno, pager_datahash(pPager->pageSize, aData), (isMainJrnl?"main-journal":"sub-journal") )); if( (pPager->state>=PAGER_EXCLUSIVE) && (pPg==0 || 0==(pPg->flags&PGHDR_NEED_SYNC)) && isOpen(pPager->fd) ){ i64 ofst = (pgno-1)*(i64)pPager->pageSize; rc = sqlite3OsWrite(pPager->fd, aData, pPager->pageSize, ofst); if( pgno>pPager->dbFileSize ){ pPager->dbFileSize = pgno; } sqlite3BackupUpdate(pPager->pBackup, pgno, aData); | > | 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 | PAGERTRACE(("PLAYBACK %d page %d hash(%08x) %s\n", PAGERID(pPager), pgno, pager_datahash(pPager->pageSize, aData), (isMainJrnl?"main-journal":"sub-journal") )); if( (pPager->state>=PAGER_EXCLUSIVE) && (pPg==0 || 0==(pPg->flags&PGHDR_NEED_SYNC)) && isOpen(pPager->fd) && !isUnsync ){ i64 ofst = (pgno-1)*(i64)pPager->pageSize; rc = sqlite3OsWrite(pPager->fd, aData, pPager->pageSize, ofst); if( pgno>pPager->dbFileSize ){ pPager->dbFileSize = pgno; } sqlite3BackupUpdate(pPager->pBackup, pgno, aData); |
︙ | ︙ | |||
1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 | needPagerReset = isHot; /* This loop terminates either when a readJournalHdr() or ** pager_playback_one_page() call returns SQLITE_DONE or an IO error ** occurs. */ while( 1 ){ /* Read the next journal header from the journal file. If there are ** not enough bytes left in the journal file for a complete header, or ** it is corrupted, then a process must of failed while writing it. ** This indicates nothing more needs to be rolled back. */ rc = readJournalHdr(pPager, szJ, &nRec, &mxPg); | > | 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 | needPagerReset = isHot; /* This loop terminates either when a readJournalHdr() or ** pager_playback_one_page() call returns SQLITE_DONE or an IO error ** occurs. */ while( 1 ){ int isUnsync = 0; /* Read the next journal header from the journal file. If there are ** not enough bytes left in the journal file for a complete header, or ** it is corrupted, then a process must of failed while writing it. ** This indicates nothing more needs to be rolled back. */ rc = readJournalHdr(pPager, szJ, &nRec, &mxPg); |
︙ | ︙ | |||
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 | && pPager->journalHdr+JOURNAL_HDR_SZ(pPager)!=pPager->journalOff && ((szJ - pPager->journalOff) / JOURNAL_PG_SZ(pPager))>0 && pagerNextJournalPageIsValid(pPager) ); if( nRec==0 && !isHot && pPager->journalHdr+JOURNAL_HDR_SZ(pPager)==pPager->journalOff ){ nRec = (int)((szJ - pPager->journalOff) / JOURNAL_PG_SZ(pPager)); } /* If this is the first header read from the journal, truncate the ** database file back to its original size. */ if( pPager->journalOff==JOURNAL_HDR_SZ(pPager) ){ rc = pager_truncate(pPager, mxPg); | > | 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 | && pPager->journalHdr+JOURNAL_HDR_SZ(pPager)!=pPager->journalOff && ((szJ - pPager->journalOff) / JOURNAL_PG_SZ(pPager))>0 && pagerNextJournalPageIsValid(pPager) ); if( nRec==0 && !isHot && pPager->journalHdr+JOURNAL_HDR_SZ(pPager)==pPager->journalOff ){ nRec = (int)((szJ - pPager->journalOff) / JOURNAL_PG_SZ(pPager)); isUnsync = 1; } /* If this is the first header read from the journal, truncate the ** database file back to its original size. */ if( pPager->journalOff==JOURNAL_HDR_SZ(pPager) ){ rc = pager_truncate(pPager, mxPg); |
︙ | ︙ | |||
2014 2015 2016 2017 2018 2019 2020 | ** database file and/or page cache. */ for(u=0; u<nRec; u++){ if( needPagerReset ){ pager_reset(pPager); needPagerReset = 0; } | | | 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 | ** database file and/or page cache. */ for(u=0; u<nRec; u++){ if( needPagerReset ){ pager_reset(pPager); needPagerReset = 0; } rc = pager_playback_one_page(pPager,1,isUnsync,&pPager->journalOff,0,0); if( rc!=SQLITE_OK ){ if( rc==SQLITE_DONE ){ rc = SQLITE_OK; pPager->journalOff = szJ; break; }else{ /* If we are unable to rollback, then the database is probably |
︙ | ︙ | |||
2156 2157 2158 2159 2160 2161 2162 | ** will be skipped automatically. Pages are added to pDone as they ** are played back. */ if( pSavepoint ){ iHdrOff = pSavepoint->iHdrOffset ? pSavepoint->iHdrOffset : szJ; pPager->journalOff = pSavepoint->iOffset; while( rc==SQLITE_OK && pPager->journalOff<iHdrOff ){ | | | 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 | ** will be skipped automatically. Pages are added to pDone as they ** are played back. */ if( pSavepoint ){ iHdrOff = pSavepoint->iHdrOffset ? pSavepoint->iHdrOffset : szJ; pPager->journalOff = pSavepoint->iOffset; while( rc==SQLITE_OK && pPager->journalOff<iHdrOff ){ rc = pager_playback_one_page(pPager, 1, 0, &pPager->journalOff, 1, pDone); } assert( rc!=SQLITE_DONE ); }else{ pPager->journalOff = 0; } /* Continue rolling back records out of the main journal starting at |
︙ | ︙ | |||
2191 2192 2193 2194 2195 2196 2197 | ); if( nJRec==0 && pPager->journalHdr+JOURNAL_HDR_SZ(pPager)==pPager->journalOff ){ nJRec = (u32)((szJ - pPager->journalOff)/JOURNAL_PG_SZ(pPager)); } for(ii=0; rc==SQLITE_OK && ii<nJRec && pPager->journalOff<szJ; ii++){ | | | | 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 | ); if( nJRec==0 && pPager->journalHdr+JOURNAL_HDR_SZ(pPager)==pPager->journalOff ){ nJRec = (u32)((szJ - pPager->journalOff)/JOURNAL_PG_SZ(pPager)); } for(ii=0; rc==SQLITE_OK && ii<nJRec && pPager->journalOff<szJ; ii++){ rc = pager_playback_one_page(pPager, 1, 0, &pPager->journalOff, 1, pDone); } assert( rc!=SQLITE_DONE ); } assert( rc!=SQLITE_OK || pPager->journalOff==szJ ); /* Finally, rollback pages from the sub-journal. Page that were ** previously rolled back out of the main journal (and are hence in pDone) ** will be skipped. Out-of-range pages are also skipped. */ if( pSavepoint ){ u32 ii; /* Loop counter */ i64 offset = pSavepoint->iSubRec*(4+pPager->pageSize); for(ii=pSavepoint->iSubRec; rc==SQLITE_OK && ii<pPager->nSubRec; ii++){ assert( offset==ii*(4+pPager->pageSize) ); rc = pager_playback_one_page(pPager, 0, 0, &offset, 1, pDone); } assert( rc!=SQLITE_DONE ); } sqlite3BitvecDestroy(pDone); if( rc==SQLITE_OK ){ pPager->journalOff = szJ; |
︙ | ︙ |
Changes to src/test_journal.c.
︙ | ︙ | |||
11 12 13 14 15 16 17 | ****************************************************************************** ** ** This file contains code for a VFS layer that acts as a wrapper around ** an existing VFS. The code in this file attempts to verify that SQLite ** correctly populates and syncs a journal file before writing to a ** corresponding database file. ** | | | 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | ****************************************************************************** ** ** This file contains code for a VFS layer that acts as a wrapper around ** an existing VFS. The code in this file attempts to verify that SQLite ** correctly populates and syncs a journal file before writing to a ** corresponding database file. ** ** $Id: test_journal.c,v 1.11 2009/02/12 09:11:56 danielk1977 Exp $ */ #if SQLITE_TEST /* This file is used for testing only */ #include "sqlite3.h" #include "sqliteInt.h" /* |
︙ | ︙ | |||
201 202 203 204 205 206 207 208 209 210 211 212 213 214 | }; struct JtGlobal { sqlite3_vfs *pVfs; /* Parent VFS */ jt_file *pList; /* List of all open files */ }; static struct JtGlobal g = {0, 0}; /* ** The jt_file pointed to by the argument may or may not be a file-handle ** open on a main database file. If it is, and a transaction is currently ** opened on the file, then discard all transaction related data. */ static void closeTransaction(jt_file *p){ | > > > > > > > > > | 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 | }; struct JtGlobal { sqlite3_vfs *pVfs; /* Parent VFS */ jt_file *pList; /* List of all open files */ }; static struct JtGlobal g = {0, 0}; extern int sqlite3_io_error_pending; static void stop_ioerr_simulation(int *piSave){ *piSave = sqlite3_io_error_pending; sqlite3_io_error_pending = -1; } static void start_ioerr_simulation(int iSave){ sqlite3_io_error_pending = iSave; } /* ** The jt_file pointed to by the argument may or may not be a file-handle ** open on a main database file. If it is, and a transaction is currently ** opened on the file, then discard all transaction related data. */ static void closeTransaction(jt_file *p){ |
︙ | ︙ | |||
339 340 341 342 343 344 345 346 347 348 349 350 351 352 | pMain->aCksum = sqlite3_malloc(sizeof(u32) * (pMain->nPage + 1)); pJournal->iMaxOff = 0; if( !pMain->pWritable || !pMain->aCksum || !aData ){ rc = SQLITE_IOERR_NOMEM; }else if( pMain->nPage>0 ){ u32 iTrunk; /* Read the database free-list. Add the page-number for each free-list ** leaf to the jt_file.pWritable bitvec. */ rc = sqlite3OsRead(p, aData, pMain->nPagesize, 0); iTrunk = decodeUint32(&aData[32]); while( rc==SQLITE_OK && iTrunk>0 ){ | > > > | 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 | pMain->aCksum = sqlite3_malloc(sizeof(u32) * (pMain->nPage + 1)); pJournal->iMaxOff = 0; if( !pMain->pWritable || !pMain->aCksum || !aData ){ rc = SQLITE_IOERR_NOMEM; }else if( pMain->nPage>0 ){ u32 iTrunk; int iSave; stop_ioerr_simulation(&iSave); /* Read the database free-list. Add the page-number for each free-list ** leaf to the jt_file.pWritable bitvec. */ rc = sqlite3OsRead(p, aData, pMain->nPagesize, 0); iTrunk = decodeUint32(&aData[32]); while( rc==SQLITE_OK && iTrunk>0 ){ |
︙ | ︙ | |||
368 369 370 371 372 373 374 375 376 377 378 379 380 381 | for(ii=0; rc==SQLITE_OK && ii<pMain->nPage; ii++){ i64 iOff = (i64)(pMain->nPagesize) * (i64)ii; if( iOff==PENDING_BYTE ) continue; rc = sqlite3OsRead(pMain->pReal, aData, pMain->nPagesize, iOff); pMain->aCksum[ii] = genCksum(aData, pMain->nPagesize); } } } sqlite3_free(aData); return rc; } /* | > > | 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 | for(ii=0; rc==SQLITE_OK && ii<pMain->nPage; ii++){ i64 iOff = (i64)(pMain->nPagesize) * (i64)ii; if( iOff==PENDING_BYTE ) continue; rc = sqlite3OsRead(pMain->pReal, aData, pMain->nPagesize, iOff); pMain->aCksum[ii] = genCksum(aData, pMain->nPagesize); } } start_ioerr_simulation(iSave); } sqlite3_free(aData); return rc; } /* |
︙ | ︙ | |||
461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 | static int readJournalFile(jt_file *p, jt_file *pMain){ int rc = SQLITE_OK; unsigned char zBuf[28]; sqlite3_file *pReal = p->pReal; sqlite3_int64 iOff = 0; sqlite3_int64 iSize = p->iMaxOff; unsigned char *aPage; aPage = sqlite3_malloc(pMain->nPagesize); if( !aPage ){ return SQLITE_IOERR_NOMEM; } while( rc==SQLITE_OK && iOff<iSize ){ u32 nRec, nPage, nSector, nPagesize; u32 ii; /* Read and decode the next journal-header from the journal file. */ rc = sqlite3OsRead(pReal, zBuf, 28, iOff); | > > > | 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 | static int readJournalFile(jt_file *p, jt_file *pMain){ int rc = SQLITE_OK; unsigned char zBuf[28]; sqlite3_file *pReal = p->pReal; sqlite3_int64 iOff = 0; sqlite3_int64 iSize = p->iMaxOff; unsigned char *aPage; int iSave; aPage = sqlite3_malloc(pMain->nPagesize); if( !aPage ){ return SQLITE_IOERR_NOMEM; } stop_ioerr_simulation(&iSave); while( rc==SQLITE_OK && iOff<iSize ){ u32 nRec, nPage, nSector, nPagesize; u32 ii; /* Read and decode the next journal-header from the journal file. */ rc = sqlite3OsRead(pReal, zBuf, 28, iOff); |
︙ | ︙ | |||
518 519 520 521 522 523 524 525 526 527 528 529 530 531 | } } iOff = ((iOff + (nSector-1)) / nSector) * nSector; } finish_rjf: sqlite3_free(aPage); if( rc==SQLITE_IOERR_SHORT_READ ){ rc = SQLITE_OK; } return rc; } | > | 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 | } } iOff = ((iOff + (nSector-1)) / nSector) * nSector; } finish_rjf: start_ioerr_simulation(iSave); sqlite3_free(aPage); if( rc==SQLITE_IOERR_SHORT_READ ){ rc = SQLITE_OK; } return rc; } |
︙ | ︙ |
Changes to test/ioerr2.test.
︙ | ︙ | |||
11 12 13 14 15 16 17 | # This file implements regression tests for SQLite library. The # focus of this file is testing for correct handling of I/O errors # such as writes failing because the disk is full. # # The tests in this file use special facilities that are only # available in the SQLite test fixture. # | | | 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | # This file implements regression tests for SQLite library. The # focus of this file is testing for correct handling of I/O errors # such as writes failing because the disk is full. # # The tests in this file use special facilities that are only # available in the SQLite test fixture. # # $Id: ioerr2.test,v 1.11 2009/02/12 09:11:56 danielk1977 Exp $ set testdir [file dirname $argv0] source $testdir/tester.tcl ifcapable !integrityck { finish_test return |
︙ | ︙ | |||
140 141 142 143 144 145 146 147 148 | # do_test ioerr2-6 { set ::sqlite_io_error_hit 0 set ::sqlite_io_error_pending 1 catchsql {PRAGMA temp_store_directory = '/tmp/'} } {1 {not a writable directory}} } finish_test | > > > > > > > > > > > > > > > > > > > > > > > | 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 | # do_test ioerr2-6 { set ::sqlite_io_error_hit 0 set ::sqlite_io_error_pending 1 catchsql {PRAGMA temp_store_directory = '/tmp/'} } {1 {not a writable directory}} } do_ioerr_test ioerr2-7 -persist 0 -sqlprep { PRAGMA cache_size = 10; PRAGMA auto_vacuum = 1; CREATE TABLE ab(a, b); CREATE TABLE de(d, e); INSERT INTO ab VALUES(1, randstr(200,200)); INSERT INTO ab SELECT a+1, randstr(200,200) FROM ab; INSERT INTO ab SELECT a+2, randstr(200,200) FROM ab; INSERT INTO ab SELECT a+4, randstr(200,200) FROM ab; INSERT INTO ab SELECT a+8, randstr(200,200) FROM ab; INSERT INTO ab SELECT a+16, randstr(200,200) FROM ab; INSERT INTO ab SELECT a+32, randstr(200,200) FROM ab; INSERT INTO ab SELECT a+64, randstr(200,200) FROM ab; INSERT INTO de SELECT * FROM ab; } -sqlbody { BEGIN; UPDATE ab SET b = randstr(200,200); UPDATE de SET e = randstr(200,200) WHERE d = (SELECT max(d) FROM de); DELETE FROM ab; COMMIT; } finish_test |