SQLite

Check-in [23ca2389]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:When a sub-transaction is released, if no pages required by containing sub-transactions were journaled, truncate the statement journal. This prevents out of control statement journal growth in some cases. See forum thread e78ffd751185a67e for context.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256: 23ca23894af352ea351c9efcdd7d86b82455f4c81b6001052a6d13aa2d705e84
User & Date: dan 2021-02-23 16:40:47
Original Comment: When a sub-transaction is released, if no pages required by containing sub-transactions were journaled, truncate the statement journal. This prevents out of control statement journal growth in some cases.
References
2022-01-01
19:29
Fix an obscure problem with releasing savepoints stored in an in-memory journal that could cause subsequent savepoint rollback to fail. Problem reported by forum post d7338bf4901f1151, PoC #3. The problem appears to have been introduced at [23ca23894af352ea]. (check-in: 73c2b502 user: dan tags: trunk)
2021-09-13
13:53
Fix an incorrect comment and possible integer overflow in pager resulting from check-in [23ca23894af352ea]. Problem reported by forum post e2ea1a3f61. Also change the datatype of Pager.pageSize to i64 even though page size never exceeds 65536, in order to help prevent future problems of this kind. (check-in: f4a552ed user: drh tags: trunk)
Context
2021-02-25
15:50
Ensure that tests like "rbu.test" and "notify2.test" that are only run by specific configurations during release testing are run for both release and debug versions of the tests. (check-in: 911df43f user: dan tags: trunk)
2021-02-23
16:40
When a sub-transaction is released, if no pages required by containing sub-transactions were journaled, truncate the statement journal. This prevents out of control statement journal growth in some cases. See forum thread e78ffd751185a67e for context. (check-in: 23ca2389 user: dan tags: trunk)
15:53
Allow WHERE terms to be pushed down into sub-queries that contain window functions, provided that the WHERE term is made up of entirely of constants and copies of expressions found in the PARTITION BY clauses of all window functions in the sub-query. (check-in: 20689468 user: dan tags: trunk)
2021-02-22
15:44
When a sub-transaction is released, if no pages required by containing sub-transactions were journaled, truncate the statement journal. This might prevent out-of-control statement journal growth in some cases. (Closed-Leaf check-in: e36327fb user: dan tags: stmt-jrnl-truncate)
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to src/memjournal.c.

66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
** is an instance of this class.
*/
struct MemJournal {
  const sqlite3_io_methods *pMethod; /* Parent class. MUST BE FIRST */
  int nChunkSize;                 /* In-memory chunk-size */

  int nSpill;                     /* Bytes of data before flushing */
  int nSize;                      /* Bytes of data currently in memory */
  FileChunk *pFirst;              /* Head of in-memory chunk-list */
  FilePoint endpoint;             /* Pointer to the end of the file */
  FilePoint readpoint;            /* Pointer to the end of the last xRead() */

  int flags;                      /* xOpen flags */
  sqlite3_vfs *pVfs;              /* The "real" underlying VFS */
  const char *zJournal;           /* Name of the journal file */







<







66
67
68
69
70
71
72

73
74
75
76
77
78
79
** is an instance of this class.
*/
struct MemJournal {
  const sqlite3_io_methods *pMethod; /* Parent class. MUST BE FIRST */
  int nChunkSize;                 /* In-memory chunk-size */

  int nSpill;                     /* Bytes of data before flushing */

  FileChunk *pFirst;              /* Head of in-memory chunk-list */
  FilePoint endpoint;             /* Pointer to the end of the file */
  FilePoint readpoint;            /* Pointer to the end of the last xRead() */

  int flags;                      /* xOpen flags */
  sqlite3_vfs *pVfs;              /* The "real" underlying VFS */
  const char *zJournal;           /* Name of the journal file */
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148

  return SQLITE_OK;
}

/*
** Free the list of FileChunk structures headed at MemJournal.pFirst.
*/
static void memjrnlFreeChunks(MemJournal *p){
  FileChunk *pIter;
  FileChunk *pNext;
  for(pIter=p->pFirst; pIter; pIter=pNext){
    pNext = pIter->pNext;
    sqlite3_free(pIter);
  } 
  p->pFirst = 0;
}

/*
** Flush the contents of memory to a real file on disk.
*/
static int memjrnlCreateFile(MemJournal *p){
  int rc;







|


|



<







126
127
128
129
130
131
132
133
134
135
136
137
138
139

140
141
142
143
144
145
146

  return SQLITE_OK;
}

/*
** Free the list of FileChunk structures headed at MemJournal.pFirst.
*/
static void memjrnlFreeChunks(FileChunk *pFirst){
  FileChunk *pIter;
  FileChunk *pNext;
  for(pIter=pFirst; pIter; pIter=pNext){
    pNext = pIter->pNext;
    sqlite3_free(pIter);
  } 

}

/*
** Flush the contents of memory to a real file on disk.
*/
static int memjrnlCreateFile(MemJournal *p){
  int rc;
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
      }
      rc = sqlite3OsWrite(pReal, (u8*)pIter->zChunk, nChunk, iOff);
      if( rc ) break;
      iOff += nChunk;
    }
    if( rc==SQLITE_OK ){
      /* No error has occurred. Free the in-memory buffers. */
      memjrnlFreeChunks(&copy);
    }
  }
  if( rc!=SQLITE_OK ){
    /* If an error occurred while creating or writing to the file, restore
    ** the original before returning. This way, SQLite uses the in-memory
    ** journal data to roll back changes made to the internal page-cache
    ** before this function was called.  */







|







159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
      }
      rc = sqlite3OsWrite(pReal, (u8*)pIter->zChunk, nChunk, iOff);
      if( rc ) break;
      iOff += nChunk;
    }
    if( rc==SQLITE_OK ){
      /* No error has occurred. Free the in-memory buffers. */
      memjrnlFreeChunks(copy.pFirst);
    }
  }
  if( rc!=SQLITE_OK ){
    /* If an error occurred while creating or writing to the file, restore
    ** the original before returning. This way, SQLite uses the in-memory
    ** journal data to roll back changes made to the internal page-cache
    ** before this function was called.  */
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266


267
268
269











270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
        }

        memcpy((u8*)p->endpoint.pChunk->zChunk + iChunkOffset, zWrite, iSpace);
        zWrite += iSpace;
        nWrite -= iSpace;
        p->endpoint.iOffset += iSpace;
      }
      p->nSize = iAmt + iOfst;
    }
  }

  return SQLITE_OK;
}

/*
** Truncate the file.
**
** If the journal file is already on disk, truncate it there. Or, if it
** is still in main memory but is being truncated to zero bytes in size,
** ignore 
*/
static int memjrnlTruncate(sqlite3_file *pJfd, sqlite_int64 size){
  MemJournal *p = (MemJournal *)pJfd;


  if( ALWAYS(size==0) ){
    memjrnlFreeChunks(p);
    p->nSize = 0;











    p->endpoint.pChunk = 0;
    p->endpoint.iOffset = 0;
    p->readpoint.pChunk = 0;
    p->readpoint.iOffset = 0;
  }
  return SQLITE_OK;
}

/*
** Close the file.
*/
static int memjrnlClose(sqlite3_file *pJfd){
  MemJournal *p = (MemJournal *)pJfd;
  memjrnlFreeChunks(p);
  return SQLITE_OK;
}

/*
** Sync the file.
**
** If the real file has been created, call its xSync method. Otherwise, 







<







|
<
<
<
<



>
>
|
|
|
>
>
>
>
>
>
>
>
>
>
>
|
|
|
|
<








|







242
243
244
245
246
247
248

249
250
251
252
253
254
255
256




257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279

280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
        }

        memcpy((u8*)p->endpoint.pChunk->zChunk + iChunkOffset, zWrite, iSpace);
        zWrite += iSpace;
        nWrite -= iSpace;
        p->endpoint.iOffset += iSpace;
      }

    }
  }

  return SQLITE_OK;
}

/*
** Truncate the in-memory file.




*/
static int memjrnlTruncate(sqlite3_file *pJfd, sqlite_int64 size){
  MemJournal *p = (MemJournal *)pJfd;
  FileChunk *pIter = 0;

  if( size==0 ){
    memjrnlFreeChunks(p->pFirst);
    p->pFirst = 0;
  }else{
    i64 iOff = p->nChunkSize;
    for(pIter=p->pFirst; ALWAYS(pIter) && iOff<=size; pIter=pIter->pNext){
      iOff += p->nChunkSize;
    }
    if( pIter ){
      memjrnlFreeChunks(pIter->pNext);
      pIter->pNext = 0;
    }
  }

  p->endpoint.pChunk = pIter;
  p->endpoint.iOffset = size;
  p->readpoint.pChunk = 0;
  p->readpoint.iOffset = 0;

  return SQLITE_OK;
}

/*
** Close the file.
*/
static int memjrnlClose(sqlite3_file *pJfd){
  MemJournal *p = (MemJournal *)pJfd;
  memjrnlFreeChunks(p->pFirst);
  return SQLITE_OK;
}

/*
** Sync the file.
**
** If the real file has been created, call its xSync method. Otherwise, 

Changes to src/pager.c.

431
432
433
434
435
436
437

438
439
440
441
442
443
444
typedef struct PagerSavepoint PagerSavepoint;
struct PagerSavepoint {
  i64 iOffset;                 /* Starting offset in main journal */
  i64 iHdrOffset;              /* See above */
  Bitvec *pInSavepoint;        /* Set of pages in this savepoint */
  Pgno nOrig;                  /* Original number of pages in file */
  Pgno iSubRec;                /* Index of first record in sub-journal */

#ifndef SQLITE_OMIT_WAL
  u32 aWalData[WAL_SAVEPOINT_NDATA];        /* WAL savepoint context */
#endif
};

/*
** Bits of the Pager.doNotSpill flag.  See further description below.







>







431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
typedef struct PagerSavepoint PagerSavepoint;
struct PagerSavepoint {
  i64 iOffset;                 /* Starting offset in main journal */
  i64 iHdrOffset;              /* See above */
  Bitvec *pInSavepoint;        /* Set of pages in this savepoint */
  Pgno nOrig;                  /* Original number of pages in file */
  Pgno iSubRec;                /* Index of first record in sub-journal */
  int bTruncateOnRelease;      /* If stmt journal may be truncated on RELEASE */
#ifndef SQLITE_OMIT_WAL
  u32 aWalData[WAL_SAVEPOINT_NDATA];        /* WAL savepoint context */
#endif
};

/*
** Bits of the Pager.doNotSpill flag.  See further description below.
1066
1067
1068
1069
1070
1071
1072



1073
1074
1075
1076
1077
1078
1079
  Pager *pPager = pPg->pPager;
  PagerSavepoint *p;
  Pgno pgno = pPg->pgno;
  int i;
  for(i=0; i<pPager->nSavepoint; i++){
    p = &pPager->aSavepoint[i];
    if( p->nOrig>=pgno && 0==sqlite3BitvecTestNotNull(p->pInSavepoint, pgno) ){



      return 1;
    }
  }
  return 0;
}

#ifdef SQLITE_DEBUG







>
>
>







1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
  Pager *pPager = pPg->pPager;
  PagerSavepoint *p;
  Pgno pgno = pPg->pgno;
  int i;
  for(i=0; i<pPager->nSavepoint; i++){
    p = &pPager->aSavepoint[i];
    if( p->nOrig>=pgno && 0==sqlite3BitvecTestNotNull(p->pInSavepoint, pgno) ){
      for(i=i+1; i<pPager->nSavepoint; i++){
        pPager->aSavepoint[i].bTruncateOnRelease = 0;
      }
      return 1;
    }
  }
  return 0;
}

#ifdef SQLITE_DEBUG
6844
6845
6846
6847
6848
6849
6850

6851
6852
6853
6854
6855
6856
6857
    if( isOpen(pPager->jfd) && pPager->journalOff>0 ){
      aNew[ii].iOffset = pPager->journalOff;
    }else{
      aNew[ii].iOffset = JOURNAL_HDR_SZ(pPager);
    }
    aNew[ii].iSubRec = pPager->nSubRec;
    aNew[ii].pInSavepoint = sqlite3BitvecCreate(pPager->dbSize);

    if( !aNew[ii].pInSavepoint ){
      return SQLITE_NOMEM_BKPT;
    }
    if( pagerUseWal(pPager) ){
      sqlite3WalSavepoint(pPager->pWal, aNew[ii].aWalData);
    }
    pPager->nSavepoint = ii+1;







>







6848
6849
6850
6851
6852
6853
6854
6855
6856
6857
6858
6859
6860
6861
6862
    if( isOpen(pPager->jfd) && pPager->journalOff>0 ){
      aNew[ii].iOffset = pPager->journalOff;
    }else{
      aNew[ii].iOffset = JOURNAL_HDR_SZ(pPager);
    }
    aNew[ii].iSubRec = pPager->nSubRec;
    aNew[ii].pInSavepoint = sqlite3BitvecCreate(pPager->dbSize);
    aNew[ii].bTruncateOnRelease = 1;
    if( !aNew[ii].pInSavepoint ){
      return SQLITE_NOMEM_BKPT;
    }
    if( pagerUseWal(pPager) ){
      sqlite3WalSavepoint(pPager->pWal, aNew[ii].aWalData);
    }
    pPager->nSavepoint = ii+1;
6925
6926
6927
6928
6929
6930
6931

6932
6933
6934

6935
6936
6937
6938
6939
6940
6941
6942
6943
6944
6945
      sqlite3BitvecDestroy(pPager->aSavepoint[ii].pInSavepoint);
    }
    pPager->nSavepoint = nNew;

    /* If this is a release of the outermost savepoint, truncate 
    ** the sub-journal to zero bytes in size. */
    if( op==SAVEPOINT_RELEASE ){

      if( nNew==0 && isOpen(pPager->sjfd) ){
        /* Only truncate if it is an in-memory sub-journal. */
        if( sqlite3JournalIsInMemory(pPager->sjfd) ){

          rc = sqlite3OsTruncate(pPager->sjfd, 0);
          assert( rc==SQLITE_OK );
        }
        pPager->nSubRec = 0;
      }
    }
    /* Else this is a rollback operation, playback the specified savepoint.
    ** If this is a temp-file, it is possible that the journal file has
    ** not yet been opened. In this case there have been no changes to
    ** the database file, so the playback operation can be skipped.
    */







>
|


>
|


|







6930
6931
6932
6933
6934
6935
6936
6937
6938
6939
6940
6941
6942
6943
6944
6945
6946
6947
6948
6949
6950
6951
6952
      sqlite3BitvecDestroy(pPager->aSavepoint[ii].pInSavepoint);
    }
    pPager->nSavepoint = nNew;

    /* If this is a release of the outermost savepoint, truncate 
    ** the sub-journal to zero bytes in size. */
    if( op==SAVEPOINT_RELEASE ){
      PagerSavepoint *pRel = &pPager->aSavepoint[nNew];
      if( pRel->bTruncateOnRelease && isOpen(pPager->sjfd) ){
        /* Only truncate if it is an in-memory sub-journal. */
        if( sqlite3JournalIsInMemory(pPager->sjfd) ){
          i64 sz = (pPager->pageSize+4)*pRel->iSubRec;
          rc = sqlite3OsTruncate(pPager->sjfd, sz);
          assert( rc==SQLITE_OK );
        }
        pPager->nSubRec = pRel->iSubRec;
      }
    }
    /* Else this is a rollback operation, playback the specified savepoint.
    ** If this is a temp-file, it is possible that the journal file has
    ** not yet been opened. In this case there have been no changes to
    ** the database file, so the playback operation can be skipped.
    */