/ Check-in [9e511e16]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Simplify the pcache module by only recycling clean pages from 'other' caches. This commit causes errors in test files ioerr5.test and malloc5.test because they test recycling dirty pages from other caches. (CVS 5615)
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1:9e511e161bcb077450d31fca5dd20c2557f103b3
User & Date: danielk1977 2008-08-26 18:05:48
Context
2008-08-26
19:08
Disable some very expensive asserts in pcache.c that are too slow for all.test. (CVS 5616) check-in: 555dad90 user: danielk1977 tags: trunk
18:05
Simplify the pcache module by only recycling clean pages from 'other' caches. This commit causes errors in test files ioerr5.test and malloc5.test because they test recycling dirty pages from other caches. (CVS 5615) check-in: 9e511e16 user: danielk1977 tags: trunk
14:42
Implement a "counter" SQL function that can be used to insert a sequence number each row of a result set. Currently in the test harness only, but a candidate to move into the core. (CVS 5614) check-in: c84d46c7 user: drh tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to src/pager.c.

14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
...
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
....
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
1966
....
1969
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
....
2038
2039
2040
2041
2042
2043
2044
2045
2046
2047
2048
2049
2050
2051
2052
2053
2054
2055
2056
2057
2058
2059
2060
....
2063
2064
2065
2066
2067
2068
2069
2070
2071
2072
2073
2074
2075
2076
2077
2078
2079
2080
2081
2082
2083
2084
2085
2086
2087
2088
2089
2090
....
2097
2098
2099
2100
2101
2102
2103
2104
2105
2106
2107
2108
2109
2110
2111
....
2163
2164
2165
2166
2167
2168
2169
2170
2171
2172
2173
2174
2175
2176
2177
....
2184
2185
2186
2187
2188
2189
2190
2191
2192
2193
2194
2195
2196
2197
2198
....
2203
2204
2205
2206
2207
2208
2209
2210
2211
2212
2213
2214
2215
2216
2217
....
2249
2250
2251
2252
2253
2254
2255
2256
2257
2258
2259
2260
2261
2262
2263
2264
2265
....
2912
2913
2914
2915
2916
2917
2918
2919
2920
2921
2922
2923
2924
2925
2926
2927
2928
....
2935
2936
2937
2938
2939
2940
2941
2942
2943
2944
2945
2946
2947
2948
2949
2950
2951
2952
2953
2954
2955
....
2957
2958
2959
2960
2961
2962
2963
2964
2965
2966
2967
2968
2969
2970
2971
2972
2973
2974
....
2982
2983
2984
2985
2986
2987
2988
2989
2990
2991
2992
2993
2994
2995
2996
2997
2998
....
3074
3075
3076
3077
3078
3079
3080
3081
3082
3083
3084
3085
3086
3087
3088
....
3092
3093
3094
3095
3096
3097
3098
3099
3100
3101
3102
3103
3104
3105
3106
....
3122
3123
3124
3125
3126
3127
3128
3129
3130
3131
3132
3133
3134
3135
3136
....
3358
3359
3360
3361
3362
3363
3364
3365
3366
3367
3368
3369
3370
3371
3372
....
3432
3433
3434
3435
3436
3437
3438
3439
3440
3441
3442
3443
3444
3445
3446
....
3476
3477
3478
3479
3480
3481
3482
3483
3484
3485
3486
3487
3488
3489
3490
....
3498
3499
3500
3501
3502
3503
3504
3505
3506
3507
3508
3509
3510
3511
3512
....
3515
3516
3517
3518
3519
3520
3521
3522
3523
3524
3525
3526
3527
3528
3529
3530
3531
3532
3533
3534
3535
3536
3537
3538
3539
3540
3541
3542
3543
3544
3545
3546
....
3559
3560
3561
3562
3563
3564
3565
3566
3567
3568
3569
3570
3571
3572
3573
....
3612
3613
3614
3615
3616
3617
3618
3619
3620
3621
3622
3623
3624
3625
3626
3627
3628
....
3663
3664
3665
3666
3667
3668
3669
3670
3671
3672
3673
3674
3675
3676
3677
....
3793
3794
3795
3796
3797
3798
3799
3800
3801
3802
3803
3804
3805
3806
3807
....
3820
3821
3822
3823
3824
3825
3826
3827
3828
3829
3830
3831
3832
3833
3834
3835
3836
3837
3838
3839
3840
3841
3842
3843
3844
3845
3846
3847
3848
....
3853
3854
3855
3856
3857
3858
3859
3860
3861
3862
3863
3864
3865
3866
3867
....
3892
3893
3894
3895
3896
3897
3898
3899
3900
3901
3902
3903
3904
3905
3906
....
3917
3918
3919
3920
3921
3922
3923
3924
3925
3926
3927
3928
3929
3930
3931
3932
3933
3934
3935
3936
3937
3938
3939
3940
3941
3942
3943
....
3990
3991
3992
3993
3994
3995
3996
3997
3998
3999
4000
4001
4002
4003
4004
4005
4006
4007
4008
4009
4010
4011
4012
4013
4014
4015
4016
4017
4018
4019
4020
4021
4022
4023
4024
4025
4026
4027
4028
4029
4030
4031
4032
4033
4034
4035
4036
4037
....
4039
4040
4041
4042
4043
4044
4045
4046
4047
4048
4049
4050
4051
4052
4053
....
4135
4136
4137
4138
4139
4140
4141
4142
4143
4144
4145
4146
4147
4148
4149
4150
....
4204
4205
4206
4207
4208
4209
4210
4211
4212
4213
4214
4215
4216
4217
4218
4219
4220
4221
4222
4223
4224
4225
4226
4227
4228
** The pager is used to access a database disk file.  It implements
** atomic commit and rollback through the use of a journal file that
** is separate from the database file.  The pager also implements file
** locking to prevent two processes from writing the same database
** file simultaneously, or one process from reading the database while
** another is writing.
**
** @(#) $Id: pager.c,v 1.479 2008/08/25 17:23:29 drh Exp $
*/
#ifndef SQLITE_OMIT_DISKIO
#include "sqliteInt.h"

/*
** Macros for troubleshooting.  Normally turned off
*/
................................................................................
#define PAGER_MJ_PGNO(x) ((PENDING_BYTE/((x)->pageSize))+1)

/*
** The maximum legal page number is (2^31 - 1).
*/
#define PAGER_MAX_PGNO 2147483647

/*
** The following two macros act as a type of recursive mutex. Their
** only purpose is to provide mutual exclusion between the "normal"
** users of a pager object (the btree.c module) and the user of the
** pagerStress() function (the pcache.c module). While the mutex 
** obtained using pagerEnter() is held, the pcache module guarantees 
** that the pagerStress() callback will not be invoked from a thread
** other than the holder of the mutex.
*/
#define pagerEnter(p) (sqlite3PcacheLock(p->pPCache))
#define pagerLeave(p) (sqlite3PcacheUnlock(p->pPCache))

/*
** Return true if page *pPg has already been written to the statement
** journal (or statement snapshot has been created, if *pPg is part
** of an in-memory database).
*/
static int pageInStatement(PgHdr *pPg){
  Pager *pPager = pPg->pPager;
................................................................................
** inappropriate, then an alternative page size is set to that
** value before returning.
*/
int sqlite3PagerSetPagesize(Pager *pPager, u16 *pPageSize){
  int rc = SQLITE_OK;
  u16 pageSize = *pPageSize;
  assert( pageSize==0 || (pageSize>=512 && pageSize<=SQLITE_MAX_PAGE_SIZE) );
  pagerEnter(pPager);
  if( pageSize && pageSize!=pPager->pageSize 
   && !pPager->memDb && sqlite3PcacheRefCount(pPager->pPCache)==0 
  ){
    char *pNew = (char *)sqlite3PageMalloc(pageSize);
    if( !pNew ){
      rc = SQLITE_NOMEM;
    }else{
................................................................................
      setSectorSize(pPager);
      sqlite3PageFree(pPager->pTmpSpace);
      pPager->pTmpSpace = pNew;
      sqlite3PcacheSetPageSize(pPager->pPCache, pageSize);
    }
  }
  *pPageSize = pPager->pageSize;
  pagerLeave(pPager);
  return rc;
}

/*
** Return a pointer to the "temporary page" buffer held internally
** by the pager.  This is a buffer that is big enough to hold the
** entire content of a database page.  This buffer is used internally
................................................................................
** response is to zero the memory at pDest and continue.  A real IO error 
** will presumably recur and be picked up later (Todo: Think about this).
*/
int sqlite3PagerReadFileheader(Pager *pPager, int N, unsigned char *pDest){
  int rc = SQLITE_OK;
  memset(pDest, 0, N);
  assert(MEMDB||pPager->fd->pMethods||pPager->tempFile);
  pagerEnter(pPager);
  if( pPager->fd->pMethods ){
    IOTRACE(("DBHDR %p 0 %d\n", pPager, N))
    rc = sqlite3OsRead(pPager->fd, pDest, N, 0);
    if( rc==SQLITE_IOERR_SHORT_READ ){
      rc = SQLITE_OK;
    }
  }
  pagerLeave(pPager);
  return rc;
}

/*
** Return the total number of pages in the disk file associated with
** pPager. 
**
................................................................................
** PENDING_BYTE is byte 4096 (the first byte of page 5) and the size of the
** file is 4096 bytes, 5 is returned instead of 4.
*/
int sqlite3PagerPagecount(Pager *pPager, int *pnPage){
  i64 n = 0;
  int rc;
  assert( pPager!=0 );
  pagerEnter(pPager);
  if( pPager->errCode ){
    rc = pPager->errCode;
    pagerLeave(pPager);
    return rc;
  }
  if( pPager->dbSize>=0 ){
    n = pPager->dbSize;
  } else {
    assert(pPager->fd->pMethods||pPager->tempFile);
    if( (pPager->fd->pMethods)
     && (rc = sqlite3OsFileSize(pPager->fd, &n))!=SQLITE_OK ){
      pager_error(pPager, rc);
      pagerLeave(pPager);
      return rc;
    }
    if( n>0 && n<pPager->pageSize ){
      n = 1;
    }else{
      n /= pPager->pageSize;
    }
................................................................................
  }
  if( n>pPager->mxPgno ){
    pPager->mxPgno = n;
  }
  if( pnPage ){
    *pnPage = n;
  }
  pagerLeave(pPager);
  return SQLITE_OK;
}

/*
** Forward declaration
*/
static int syncJournal(Pager*);
................................................................................
/*
** Truncate the file to the number of pages specified.
*/
int sqlite3PagerTruncate(Pager *pPager, Pgno nPage){
  int rc = SQLITE_OK;
  assert( pPager->state>=PAGER_SHARED || MEMDB );

  pagerEnter(pPager);

  sqlite3PagerPagecount(pPager, 0);
  if( pPager->errCode ){
    rc = pPager->errCode;
  }else if( nPage<(unsigned)pPager->dbSize ){
    if( MEMDB ){
      pPager->dbSize = nPage;
................................................................................
      }
      if( rc==SQLITE_OK ){
        rc = pager_truncate(pPager, nPage);
      }
    }
  }

  pagerLeave(pPager);
  return rc;
}

/*
** Shutdown the page cache.  Free all memory and close all files.
**
** If a transaction was in progress when this routine is called, that
................................................................................
**
** This function always succeeds. If a transaction is active an attempt
** is made to roll it back. If an error occurs during the rollback 
** a hot journal may be left in the filesystem but no error is returned
** to the caller.
*/
int sqlite3PagerClose(Pager *pPager){
  pagerEnter(pPager);

  disable_simulated_io_errors();
  sqlite3BeginBenignMalloc();
  pPager->errCode = 0;
  pPager->exclusiveMode = 0;
  pager_reset(pPager);
  pagerUnlockAndRollback(pPager);
................................................................................
#endif

/*
** Increment the reference count for a page.  The input pointer is
** a reference to the page data.
*/
int sqlite3PagerRef(DbPage *pPg){
  pagerEnter(pPg->pPager);
  sqlite3PcacheRef(pPg);
  pagerLeave(pPg->pPager);
  return SQLITE_OK;
}

/*
** Sync the journal.  In other words, make sure all the pages that have
** been written to the journal have actually reached the surface of the
** disk.  It is not safe to modify the original database file until after
................................................................................
int sqlite3PagerAcquire(
  Pager *pPager,      /* The pager open on the database file */
  Pgno pgno,          /* Page number to fetch */
  DbPage **ppPage,    /* Write a pointer to the page here */
  int noContent       /* Do not bother reading content from disk if true */
){
  int rc;
  pagerEnter(pPager);
  rc = pagerAcquire(pPager, pgno, ppPage, noContent);
  pagerLeave(pPager);
  return rc;
}


/*
** Acquire a page if it is already in the in-memory cache.  Do
** not read the page from disk.  Return a pointer to the page,
................................................................................
** has ever happened.
*/
DbPage *sqlite3PagerLookup(Pager *pPager, Pgno pgno){
  PgHdr *pPg = 0;
  assert( pPager!=0 );
  assert( pgno!=0 );

  pagerEnter(pPager);
  if( (pPager->state!=PAGER_UNLOCK)
   && (pPager->errCode==SQLITE_OK || pPager->errCode==SQLITE_FULL)
  ){
    sqlite3PcacheFetch(pPager->pPCache, pgno, 0, &pPg);
  }
  pagerLeave(pPager);

  return pPg;
}

/*
** Release a page.
**
................................................................................
** page is added to the LRU list.  When all references to all pages
** are released, a rollback occurs and the lock on the database is
** removed.
*/
int sqlite3PagerUnref(DbPage *pPg){
  if( pPg ){
    Pager *pPager = pPg->pPager;
    pagerEnter(pPager);
    sqlite3PcacheRelease(pPg);
    pagerUnlockIfUnused(pPager);
    pagerLeave(pPager);
  }
  return SQLITE_OK;
}

/*
** Create a journal file for pPager.  There should already be a RESERVED
** or EXCLUSIVE lock on the database file when this routine is called.
................................................................................

  int rc;
  assert( !MEMDB );
  assert( pPager->state>=PAGER_RESERVED );
  assert( pPager->useJournal );
  assert( pPager->pInJournal==0 );
  sqlite3PagerPagecount(pPager, 0);
  pagerLeave(pPager);
  pPager->pInJournal = sqlite3BitvecCreate(pPager->dbSize);
  pagerEnter(pPager);
  if( pPager->pInJournal==0 ){
    rc = SQLITE_NOMEM;
    goto failed_to_open_journal;
  }

  if( pPager->journalOpen==0 ){
    if( pPager->tempFile ){
................................................................................
** If exFlag is true, go ahead and get an EXCLUSIVE lock on the file
** immediately instead of waiting until we try to flush the cache.  The
** exFlag is ignored if a transaction is already active.
*/
int sqlite3PagerBegin(DbPage *pPg, int exFlag){
  Pager *pPager = pPg->pPager;
  int rc = SQLITE_OK;
  pagerEnter(pPager);
  assert( pPg->nRef>0 );
  assert( pPager->state!=PAGER_UNLOCK );
  if( pPager->state==PAGER_SHARED ){
    assert( pPager->pInJournal==0 );
    sqlite3PcacheAssertFlags(pPager->pPCache, 0, PGHDR_IN_JOURNAL);
    if( MEMDB ){
      pPager->state = PAGER_EXCLUSIVE;
................................................................................
      if( rc==SQLITE_OK ){
        pPager->state = PAGER_RESERVED;
        if( exFlag ){
          rc = pager_wait_on_lock(pPager, EXCLUSIVE_LOCK);
        }
      }
      if( rc!=SQLITE_OK ){
        pagerLeave(pPager);
        return rc;
      }
      pPager->dirtyCache = 0;
      PAGERTRACE2("TRANSACTION %d\n", PAGERID(pPager));
      if( pPager->useJournal && !pPager->tempFile
             && pPager->journalMode!=PAGER_JOURNALMODE_OFF ){
        rc = pager_open_journal(pPager);
................................................................................
      rc = SQLITE_NOMEM;
    }else{
      pPager->origDbSize = pPager->dbSize;
      rc = writeJournalHdr(pPager);
    }
  }
  assert( !pPager->journalOpen || pPager->journalOff>0 || rc!=SQLITE_OK );
  pagerLeave(pPager);
  return rc;
}

/*
** Make a page dirty.  Set its dirty flag and add it to the dirty
** page list.
*/
................................................................................
int sqlite3PagerWrite(DbPage *pDbPage){
  int rc = SQLITE_OK;

  PgHdr *pPg = pDbPage;
  Pager *pPager = pPg->pPager;
  Pgno nPagePerSector = (pPager->sectorSize/pPager->pageSize);

  pagerEnter(pPager);
  if( !MEMDB && nPagePerSector>1 ){
    Pgno nPageCount;          /* Total number of pages in database file */
    Pgno pg1;                 /* First page of the sector pPg is located on. */
    int nPage;                /* Number of pages starting at pg1 to journal */
    int ii;
    int needSync = 0;

................................................................................
    }

    assert( pPager->doNotSync==1 );
    pPager->doNotSync = 0;
  }else{
    rc = pager_write(pDbPage);
  }
  pagerLeave(pPager);
  return rc;
}

/*
** Return TRUE if the page given in the argument was previously passed
** to sqlite3PagerWrite().  In other words, return TRUE if it is ok
** to change the content of the page.
................................................................................
** rolled back in spite of the sqlite3PagerDontRollback() call.
*/
void sqlite3PagerDontWrite(DbPage *pDbPage){
  PgHdr *pPg = pDbPage;
  Pager *pPager = pPg->pPager;

  if( MEMDB ) return;
  pagerEnter(pPager);
  pPg->flags |= PGHDR_ALWAYS_ROLLBACK;
  if( (pPg->flags&PGHDR_DIRTY) && !pPager->stmtInUse ){
    assert( pPager->state>=PAGER_SHARED );
    if( pPager->dbSize==(int)pPg->pgno && pPager->origDbSize<pPager->dbSize ){
      /* If this pages is the last page in the file and the file has grown
      ** during the current transaction, then do NOT mark the page as clean.
      ** When the database file grows, we must make sure that the last page
................................................................................
      IOTRACE(("CLEAN %p %d\n", pPager, pPg->pgno))
      pPg->flags |= PGHDR_DONT_WRITE;
#ifdef SQLITE_CHECK_PAGES
      pPg->pageHash = pager_pagehash(pPg);
#endif
    }
  }
  pagerLeave(pPager);
}

/*
** A call to this routine tells the pager that if a rollback occurs,
** it is not necessary to restore the data on the given page.  This
** means that the pager does not have to record the given page in the
** rollback journal.
................................................................................
** the PgHdr.needRead flag is set) then this routine acts as a promise
** that we will never need to read the page content in the future.
** so the needRead flag can be cleared at this point.
*/
void sqlite3PagerDontRollback(DbPage *pPg){
  Pager *pPager = pPg->pPager;

  pagerEnter(pPager);
  assert( pPager->state>=PAGER_RESERVED );

  /* If the journal file is not open, or DontWrite() has been called on
  ** this page (DontWrite() sets the alwaysRollback flag), then this
  ** function is a no-op.
  */
  if( pPager->journalOpen==0 || (pPg->flags&PGHDR_ALWAYS_ROLLBACK) 
   || pPager->alwaysRollback 
  ){
    pagerLeave(pPager);
    return;
  }
  assert( !MEMDB );    /* For a memdb, pPager->journalOpen is always 0 */

#ifdef SQLITE_SECURE_DELETE
  if( pPg->inJournal || (int)pPg->pgno > pPager->origDbSize ){
    pagerLeave(pPager);
    return;
  }
#endif

  /* If SECURE_DELETE is disabled, then there is no way that this
  ** routine can be called on a page for which sqlite3PagerDontWrite()
  ** has not been previously called during the same transaction.
................................................................................
  pPg->flags &= ~PGHDR_NEED_READ;
  if( pPager->stmtInUse ){
    assert( pPager->stmtSize >= pPager->origDbSize );
    sqlite3BitvecSet(pPager->pInStmt, pPg->pgno);
  }
  PAGERTRACE3("DONT_ROLLBACK page %d of %d\n", pPg->pgno, PAGERID(pPager));
  IOTRACE(("GARBAGE %p %d\n", pPager, pPg->pgno))
  pagerLeave(pPager);
}


/*
** This routine is called to increment the database file change-counter,
** stored at byte 24 of the pager file.
*/
................................................................................
}

/*
** Sync the pager file to disk.
*/
int sqlite3PagerSync(Pager *pPager){
  int rc;
  pagerEnter(pPager);
  rc = sqlite3OsSync(pPager->fd, pPager->sync_flags);
  pagerLeave(pPager);
  return rc;
}

/*
** Sync the database file for the pager pPager. zMaster points to the name
** of a master journal file that should be written into the individual
** journal file. zMaster may be NULL, which is interpreted as no master
................................................................................
          pPager->exclusiveMode!=0) ){
    assert( pPager->dirtyCache==0 || pPager->journalOpen==0 );
    return SQLITE_OK;
  }

  PAGERTRACE4("DATABASE SYNC: File=%s zMaster=%s nTrunc=%d\n", 
      pPager->zFilename, zMaster, nTrunc);
  pagerEnter(pPager);

  /* If this is an in-memory db, or no pages have been written to, or this
  ** function has already been called, it is a no-op.
  */
  if( pPager->state!=PAGER_SYNCED && !MEMDB && pPager->dirtyCache ){
    PgHdr *pPg;

................................................................................
    /* pager_incr_changecounter() may attempt to obtain an exclusive
     * lock to spill the cache and return IOERR_BLOCKED. But since 
     * there is no chance the cache is inconsistent, it is
     * better to return SQLITE_BUSY.
     */
    rc = SQLITE_BUSY;
  }
  pagerLeave(pPager);
  return rc;
}


/*
** Commit all changes to the database and release the write lock.
**
................................................................................
  }
  if( pPager->dbModified==0 &&
        (pPager->journalMode!=PAGER_JOURNALMODE_DELETE ||
          pPager->exclusiveMode!=0) ){
    assert( pPager->dirtyCache==0 || pPager->journalOpen==0 );
    return SQLITE_OK;
  }
  pagerEnter(pPager);
  PAGERTRACE2("COMMIT %d\n", PAGERID(pPager));
  if( MEMDB ){
    sqlite3PcacheCommit(pPager->pPCache, 0);
    sqlite3PcacheCleanAll(pPager->pPCache);
    sqlite3PcacheSetFlags(pPager->pPCache, 
       ~(PGHDR_IN_JOURNAL | PGHDR_NEED_SYNC | PGHDR_ALWAYS_ROLLBACK), 0
    );
    pPager->state = PAGER_SHARED;
  }else{
    assert( pPager->state==PAGER_SYNCED || !pPager->dirtyCache );
    rc = pager_end_transaction(pPager, pPager->setMaster);
    rc = pager_error(pPager, rc);
  }
  pagerLeave(pPager);
  return rc;
}

/*
** Rollback all changes.  The database falls back to PAGER_SHARED mode.
** All in-memory cache pages revert to their original data contents.
** The journal is deleted.
................................................................................
** unless a prior malloc() failed (SQLITE_NOMEM).  Appropriate error
** codes are returned for all these occasions.  Otherwise,
** SQLITE_OK is returned.
*/
int sqlite3PagerRollback(Pager *pPager){
  int rc = SQLITE_OK;
  PAGERTRACE2("ROLLBACK %d\n", PAGERID(pPager));
  pagerEnter(pPager);
  if( MEMDB ){
    sqlite3PcacheRollback(pPager->pPCache, 1);
    sqlite3PcacheRollback(pPager->pPCache, 0);
    sqlite3PcacheCleanAll(pPager->pPCache);
    sqlite3PcacheSetFlags(pPager->pPCache, 
       ~(PGHDR_IN_JOURNAL | PGHDR_NEED_SYNC | PGHDR_ALWAYS_ROLLBACK), 0
    );
................................................................................

    /* If an error occurs during a ROLLBACK, we can no longer trust the pager
    ** cache. So call pager_error() on the way out to make any error 
    ** persistent.
    */
    rc = pager_error(pPager, rc);
  }
  pagerLeave(pPager);
  return rc;
}

/*
** Return TRUE if the database file is opened read-only.  Return FALSE
** if the database is (in theory) writable.
*/
................................................................................

#ifdef SQLITE_TEST
/*
** This routine is used for testing and analysis only.
*/
int *sqlite3PagerStats(Pager *pPager){
  static int a[11];
  pagerEnter(pPager);
  a[0] = sqlite3PcacheRefCount(pPager->pPCache);
  a[1] = sqlite3PcachePagecount(pPager->pPCache);
  a[2] = sqlite3PcacheGetCachesize(pPager->pPCache);
  a[3] = pPager->dbSize;
  a[4] = pPager->state;
  a[5] = pPager->errCode;
  a[6] = pPager->nHit;
  a[7] = pPager->nMiss;
  a[8] = 0;  /* Used to be pPager->nOvfl */
  a[9] = pPager->nRead;
  a[10] = pPager->nWrite;
  pagerLeave(pPager);
  return a;
}
int sqlite3PagerIsMemdb(Pager *pPager){
  return MEMDB;
}
#endif

................................................................................
    sqlite3BitvecDestroy(pPager->pInStmt);
    pPager->pInStmt = 0;
  }
  return rc;
}
int sqlite3PagerStmtBegin(Pager *pPager){
  int rc;
  pagerEnter(pPager);
  rc = pagerStmtBegin(pPager);
  pagerLeave(pPager);
  return rc;
}

/*
** Commit a statement.
*/
int sqlite3PagerStmtCommit(Pager *pPager){
  pagerEnter(pPager);
  if( pPager->stmtInUse ){
    PAGERTRACE2("STMT-COMMIT %d\n", PAGERID(pPager));
    if( !MEMDB ){
      /* sqlite3OsTruncate(pPager->stfd, 0); */
      sqlite3BitvecDestroy(pPager->pInStmt);
      pPager->pInStmt = 0;
    }else{
      sqlite3PcacheCommit(pPager->pPCache, 1);
    }
    pPager->stmtNRec = 0;
    pPager->stmtInUse = 0;
  }
  pPager->stmtAutoopen = 0;
  pagerLeave(pPager);
  return SQLITE_OK;
}

/*
** Rollback a statement.
*/
int sqlite3PagerStmtRollback(Pager *pPager){
  int rc;
  pagerEnter(pPager);
  if( pPager->stmtInUse ){
    PAGERTRACE2("STMT-ROLLBACK %d\n", PAGERID(pPager));
    if( MEMDB ){
      sqlite3PcacheRollback(pPager->pPCache, 1);
      pPager->dbSize = pPager->stmtSize;
      pager_truncate_cache(pPager);
      rc = SQLITE_OK;
................................................................................
      rc = pager_stmt_playback(pPager);
    }
    sqlite3PagerStmtCommit(pPager);
  }else{
    rc = SQLITE_OK;
  }
  pPager->stmtAutoopen = 0;
  pagerLeave(pPager);
  return rc;
}

/*
** Return the full pathname of the database file.
*/
const char *sqlite3PagerFilename(Pager *pPager){
................................................................................

  assert( pPg->nRef>0 );

  PAGERTRACE5("MOVE %d page %d (needSync=%d) moves to %d\n", 
      PAGERID(pPager), pPg->pgno, (pPg->flags&PGHDR_NEED_SYNC)?1:0, pgno);
  IOTRACE(("MOVE %p %d %d\n", pPager, pPg->pgno, pgno))

  pagerEnter(pPager);

  pager_get_content(pPg);

  /* If the journal needs to be sync()ed before page pPg->pgno can
  ** be written to, store pPg->pgno in local variable needSyncPgno.
  **
  ** If the isCommit flag is set, there is no need to remember that
  ** the journal needs to be sync()ed before database page pPg->pgno 
................................................................................
    PgHdr *pPgHdr;
    assert( pPager->needSync );
    rc = sqlite3PagerGet(pPager, needSyncPgno, &pPgHdr);
    if( rc!=SQLITE_OK ){
      if( pPager->pInJournal && (int)needSyncPgno<=pPager->origDbSize ){
        sqlite3BitvecClear(pPager->pInJournal, needSyncPgno);
      }
      pagerLeave(pPager);
      return rc;
    }
    pPager->needSync = 1;
    pPgHdr->flags |= PGHDR_NEED_SYNC;
    pPgHdr->flags |= PGHDR_IN_JOURNAL;
    makeDirty(pPgHdr);
    sqlite3PagerUnref(pPgHdr);
  }

  pagerLeave(pPager);
  return SQLITE_OK;
}
#endif

/*
** Return a pointer to the data for the specified page.
*/







|







 







<
<
<
<
<
<
<
<
<
<
<
<







 







<







 







<







 







<







<







 







<


<









<







 







<







 







<







 







<







 







<







 







<

<







 







<

<







 







<





<







 







<


<







 







<

<







 







<







 







<







 







<







 







<







 







<







 







<







 







<







 







<









<






<







 







<







 







<

<







 







<







 







<







 







<













<







 







<







 







<







 







<











<







 







<

<







<













<








<







 







<







 







<
<







 







<









<







14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
...
298
299
300
301
302
303
304












305
306
307
308
309
310
311
....
1940
1941
1942
1943
1944
1945
1946

1947
1948
1949
1950
1951
1952
1953
....
1956
1957
1958
1959
1960
1961
1962

1963
1964
1965
1966
1967
1968
1969
....
2024
2025
2026
2027
2028
2029
2030

2031
2032
2033
2034
2035
2036
2037

2038
2039
2040
2041
2042
2043
2044
....
2047
2048
2049
2050
2051
2052
2053

2054
2055

2056
2057
2058
2059
2060
2061
2062
2063
2064

2065
2066
2067
2068
2069
2070
2071
....
2078
2079
2080
2081
2082
2083
2084

2085
2086
2087
2088
2089
2090
2091
....
2143
2144
2145
2146
2147
2148
2149

2150
2151
2152
2153
2154
2155
2156
....
2163
2164
2165
2166
2167
2168
2169

2170
2171
2172
2173
2174
2175
2176
....
2181
2182
2183
2184
2185
2186
2187

2188
2189
2190
2191
2192
2193
2194
....
2226
2227
2228
2229
2230
2231
2232

2233

2234
2235
2236
2237
2238
2239
2240
....
2887
2888
2889
2890
2891
2892
2893

2894

2895
2896
2897
2898
2899
2900
2901
....
2908
2909
2910
2911
2912
2913
2914

2915
2916
2917
2918
2919

2920
2921
2922
2923
2924
2925
2926
....
2928
2929
2930
2931
2932
2933
2934

2935
2936

2937
2938
2939
2940
2941
2942
2943
....
2951
2952
2953
2954
2955
2956
2957

2958

2959
2960
2961
2962
2963
2964
2965
....
3041
3042
3043
3044
3045
3046
3047

3048
3049
3050
3051
3052
3053
3054
....
3058
3059
3060
3061
3062
3063
3064

3065
3066
3067
3068
3069
3070
3071
....
3087
3088
3089
3090
3091
3092
3093

3094
3095
3096
3097
3098
3099
3100
....
3322
3323
3324
3325
3326
3327
3328

3329
3330
3331
3332
3333
3334
3335
....
3395
3396
3397
3398
3399
3400
3401

3402
3403
3404
3405
3406
3407
3408
....
3438
3439
3440
3441
3442
3443
3444

3445
3446
3447
3448
3449
3450
3451
....
3459
3460
3461
3462
3463
3464
3465

3466
3467
3468
3469
3470
3471
3472
....
3475
3476
3477
3478
3479
3480
3481

3482
3483
3484
3485
3486
3487
3488
3489
3490

3491
3492
3493
3494
3495
3496

3497
3498
3499
3500
3501
3502
3503
....
3516
3517
3518
3519
3520
3521
3522

3523
3524
3525
3526
3527
3528
3529
....
3568
3569
3570
3571
3572
3573
3574

3575

3576
3577
3578
3579
3580
3581
3582
....
3617
3618
3619
3620
3621
3622
3623

3624
3625
3626
3627
3628
3629
3630
....
3746
3747
3748
3749
3750
3751
3752

3753
3754
3755
3756
3757
3758
3759
....
3772
3773
3774
3775
3776
3777
3778

3779
3780
3781
3782
3783
3784
3785
3786
3787
3788
3789
3790
3791

3792
3793
3794
3795
3796
3797
3798
....
3803
3804
3805
3806
3807
3808
3809

3810
3811
3812
3813
3814
3815
3816
....
3841
3842
3843
3844
3845
3846
3847

3848
3849
3850
3851
3852
3853
3854
....
3865
3866
3867
3868
3869
3870
3871

3872
3873
3874
3875
3876
3877
3878
3879
3880
3881
3882

3883
3884
3885
3886
3887
3888
3889
....
3936
3937
3938
3939
3940
3941
3942

3943

3944
3945
3946
3947
3948
3949
3950

3951
3952
3953
3954
3955
3956
3957
3958
3959
3960
3961
3962
3963

3964
3965
3966
3967
3968
3969
3970
3971

3972
3973
3974
3975
3976
3977
3978
....
3980
3981
3982
3983
3984
3985
3986

3987
3988
3989
3990
3991
3992
3993
....
4075
4076
4077
4078
4079
4080
4081


4082
4083
4084
4085
4086
4087
4088
....
4142
4143
4144
4145
4146
4147
4148

4149
4150
4151
4152
4153
4154
4155
4156
4157

4158
4159
4160
4161
4162
4163
4164
** The pager is used to access a database disk file.  It implements
** atomic commit and rollback through the use of a journal file that
** is separate from the database file.  The pager also implements file
** locking to prevent two processes from writing the same database
** file simultaneously, or one process from reading the database while
** another is writing.
**
** @(#) $Id: pager.c,v 1.480 2008/08/26 18:05:48 danielk1977 Exp $
*/
#ifndef SQLITE_OMIT_DISKIO
#include "sqliteInt.h"

/*
** Macros for troubleshooting.  Normally turned off
*/
................................................................................
#define PAGER_MJ_PGNO(x) ((PENDING_BYTE/((x)->pageSize))+1)

/*
** The maximum legal page number is (2^31 - 1).
*/
#define PAGER_MAX_PGNO 2147483647













/*
** Return true if page *pPg has already been written to the statement
** journal (or statement snapshot has been created, if *pPg is part
** of an in-memory database).
*/
static int pageInStatement(PgHdr *pPg){
  Pager *pPager = pPg->pPager;
................................................................................
** inappropriate, then an alternative page size is set to that
** value before returning.
*/
int sqlite3PagerSetPagesize(Pager *pPager, u16 *pPageSize){
  int rc = SQLITE_OK;
  u16 pageSize = *pPageSize;
  assert( pageSize==0 || (pageSize>=512 && pageSize<=SQLITE_MAX_PAGE_SIZE) );

  if( pageSize && pageSize!=pPager->pageSize 
   && !pPager->memDb && sqlite3PcacheRefCount(pPager->pPCache)==0 
  ){
    char *pNew = (char *)sqlite3PageMalloc(pageSize);
    if( !pNew ){
      rc = SQLITE_NOMEM;
    }else{
................................................................................
      setSectorSize(pPager);
      sqlite3PageFree(pPager->pTmpSpace);
      pPager->pTmpSpace = pNew;
      sqlite3PcacheSetPageSize(pPager->pPCache, pageSize);
    }
  }
  *pPageSize = pPager->pageSize;

  return rc;
}

/*
** Return a pointer to the "temporary page" buffer held internally
** by the pager.  This is a buffer that is big enough to hold the
** entire content of a database page.  This buffer is used internally
................................................................................
** response is to zero the memory at pDest and continue.  A real IO error 
** will presumably recur and be picked up later (Todo: Think about this).
*/
int sqlite3PagerReadFileheader(Pager *pPager, int N, unsigned char *pDest){
  int rc = SQLITE_OK;
  memset(pDest, 0, N);
  assert(MEMDB||pPager->fd->pMethods||pPager->tempFile);

  if( pPager->fd->pMethods ){
    IOTRACE(("DBHDR %p 0 %d\n", pPager, N))
    rc = sqlite3OsRead(pPager->fd, pDest, N, 0);
    if( rc==SQLITE_IOERR_SHORT_READ ){
      rc = SQLITE_OK;
    }
  }

  return rc;
}

/*
** Return the total number of pages in the disk file associated with
** pPager. 
**
................................................................................
** PENDING_BYTE is byte 4096 (the first byte of page 5) and the size of the
** file is 4096 bytes, 5 is returned instead of 4.
*/
int sqlite3PagerPagecount(Pager *pPager, int *pnPage){
  i64 n = 0;
  int rc;
  assert( pPager!=0 );

  if( pPager->errCode ){
    rc = pPager->errCode;

    return rc;
  }
  if( pPager->dbSize>=0 ){
    n = pPager->dbSize;
  } else {
    assert(pPager->fd->pMethods||pPager->tempFile);
    if( (pPager->fd->pMethods)
     && (rc = sqlite3OsFileSize(pPager->fd, &n))!=SQLITE_OK ){
      pager_error(pPager, rc);

      return rc;
    }
    if( n>0 && n<pPager->pageSize ){
      n = 1;
    }else{
      n /= pPager->pageSize;
    }
................................................................................
  }
  if( n>pPager->mxPgno ){
    pPager->mxPgno = n;
  }
  if( pnPage ){
    *pnPage = n;
  }

  return SQLITE_OK;
}

/*
** Forward declaration
*/
static int syncJournal(Pager*);
................................................................................
/*
** Truncate the file to the number of pages specified.
*/
int sqlite3PagerTruncate(Pager *pPager, Pgno nPage){
  int rc = SQLITE_OK;
  assert( pPager->state>=PAGER_SHARED || MEMDB );



  sqlite3PagerPagecount(pPager, 0);
  if( pPager->errCode ){
    rc = pPager->errCode;
  }else if( nPage<(unsigned)pPager->dbSize ){
    if( MEMDB ){
      pPager->dbSize = nPage;
................................................................................
      }
      if( rc==SQLITE_OK ){
        rc = pager_truncate(pPager, nPage);
      }
    }
  }


  return rc;
}

/*
** Shutdown the page cache.  Free all memory and close all files.
**
** If a transaction was in progress when this routine is called, that
................................................................................
**
** This function always succeeds. If a transaction is active an attempt
** is made to roll it back. If an error occurs during the rollback 
** a hot journal may be left in the filesystem but no error is returned
** to the caller.
*/
int sqlite3PagerClose(Pager *pPager){


  disable_simulated_io_errors();
  sqlite3BeginBenignMalloc();
  pPager->errCode = 0;
  pPager->exclusiveMode = 0;
  pager_reset(pPager);
  pagerUnlockAndRollback(pPager);
................................................................................
#endif

/*
** Increment the reference count for a page.  The input pointer is
** a reference to the page data.
*/
int sqlite3PagerRef(DbPage *pPg){

  sqlite3PcacheRef(pPg);

  return SQLITE_OK;
}

/*
** Sync the journal.  In other words, make sure all the pages that have
** been written to the journal have actually reached the surface of the
** disk.  It is not safe to modify the original database file until after
................................................................................
int sqlite3PagerAcquire(
  Pager *pPager,      /* The pager open on the database file */
  Pgno pgno,          /* Page number to fetch */
  DbPage **ppPage,    /* Write a pointer to the page here */
  int noContent       /* Do not bother reading content from disk if true */
){
  int rc;

  rc = pagerAcquire(pPager, pgno, ppPage, noContent);

  return rc;
}


/*
** Acquire a page if it is already in the in-memory cache.  Do
** not read the page from disk.  Return a pointer to the page,
................................................................................
** has ever happened.
*/
DbPage *sqlite3PagerLookup(Pager *pPager, Pgno pgno){
  PgHdr *pPg = 0;
  assert( pPager!=0 );
  assert( pgno!=0 );


  if( (pPager->state!=PAGER_UNLOCK)
   && (pPager->errCode==SQLITE_OK || pPager->errCode==SQLITE_FULL)
  ){
    sqlite3PcacheFetch(pPager->pPCache, pgno, 0, &pPg);
  }


  return pPg;
}

/*
** Release a page.
**
................................................................................
** page is added to the LRU list.  When all references to all pages
** are released, a rollback occurs and the lock on the database is
** removed.
*/
int sqlite3PagerUnref(DbPage *pPg){
  if( pPg ){
    Pager *pPager = pPg->pPager;

    sqlite3PcacheRelease(pPg);
    pagerUnlockIfUnused(pPager);

  }
  return SQLITE_OK;
}

/*
** Create a journal file for pPager.  There should already be a RESERVED
** or EXCLUSIVE lock on the database file when this routine is called.
................................................................................

  int rc;
  assert( !MEMDB );
  assert( pPager->state>=PAGER_RESERVED );
  assert( pPager->useJournal );
  assert( pPager->pInJournal==0 );
  sqlite3PagerPagecount(pPager, 0);

  pPager->pInJournal = sqlite3BitvecCreate(pPager->dbSize);

  if( pPager->pInJournal==0 ){
    rc = SQLITE_NOMEM;
    goto failed_to_open_journal;
  }

  if( pPager->journalOpen==0 ){
    if( pPager->tempFile ){
................................................................................
** If exFlag is true, go ahead and get an EXCLUSIVE lock on the file
** immediately instead of waiting until we try to flush the cache.  The
** exFlag is ignored if a transaction is already active.
*/
int sqlite3PagerBegin(DbPage *pPg, int exFlag){
  Pager *pPager = pPg->pPager;
  int rc = SQLITE_OK;

  assert( pPg->nRef>0 );
  assert( pPager->state!=PAGER_UNLOCK );
  if( pPager->state==PAGER_SHARED ){
    assert( pPager->pInJournal==0 );
    sqlite3PcacheAssertFlags(pPager->pPCache, 0, PGHDR_IN_JOURNAL);
    if( MEMDB ){
      pPager->state = PAGER_EXCLUSIVE;
................................................................................
      if( rc==SQLITE_OK ){
        pPager->state = PAGER_RESERVED;
        if( exFlag ){
          rc = pager_wait_on_lock(pPager, EXCLUSIVE_LOCK);
        }
      }
      if( rc!=SQLITE_OK ){

        return rc;
      }
      pPager->dirtyCache = 0;
      PAGERTRACE2("TRANSACTION %d\n", PAGERID(pPager));
      if( pPager->useJournal && !pPager->tempFile
             && pPager->journalMode!=PAGER_JOURNALMODE_OFF ){
        rc = pager_open_journal(pPager);
................................................................................
      rc = SQLITE_NOMEM;
    }else{
      pPager->origDbSize = pPager->dbSize;
      rc = writeJournalHdr(pPager);
    }
  }
  assert( !pPager->journalOpen || pPager->journalOff>0 || rc!=SQLITE_OK );

  return rc;
}

/*
** Make a page dirty.  Set its dirty flag and add it to the dirty
** page list.
*/
................................................................................
int sqlite3PagerWrite(DbPage *pDbPage){
  int rc = SQLITE_OK;

  PgHdr *pPg = pDbPage;
  Pager *pPager = pPg->pPager;
  Pgno nPagePerSector = (pPager->sectorSize/pPager->pageSize);


  if( !MEMDB && nPagePerSector>1 ){
    Pgno nPageCount;          /* Total number of pages in database file */
    Pgno pg1;                 /* First page of the sector pPg is located on. */
    int nPage;                /* Number of pages starting at pg1 to journal */
    int ii;
    int needSync = 0;

................................................................................
    }

    assert( pPager->doNotSync==1 );
    pPager->doNotSync = 0;
  }else{
    rc = pager_write(pDbPage);
  }

  return rc;
}

/*
** Return TRUE if the page given in the argument was previously passed
** to sqlite3PagerWrite().  In other words, return TRUE if it is ok
** to change the content of the page.
................................................................................
** rolled back in spite of the sqlite3PagerDontRollback() call.
*/
void sqlite3PagerDontWrite(DbPage *pDbPage){
  PgHdr *pPg = pDbPage;
  Pager *pPager = pPg->pPager;

  if( MEMDB ) return;

  pPg->flags |= PGHDR_ALWAYS_ROLLBACK;
  if( (pPg->flags&PGHDR_DIRTY) && !pPager->stmtInUse ){
    assert( pPager->state>=PAGER_SHARED );
    if( pPager->dbSize==(int)pPg->pgno && pPager->origDbSize<pPager->dbSize ){
      /* If this pages is the last page in the file and the file has grown
      ** during the current transaction, then do NOT mark the page as clean.
      ** When the database file grows, we must make sure that the last page
................................................................................
      IOTRACE(("CLEAN %p %d\n", pPager, pPg->pgno))
      pPg->flags |= PGHDR_DONT_WRITE;
#ifdef SQLITE_CHECK_PAGES
      pPg->pageHash = pager_pagehash(pPg);
#endif
    }
  }

}

/*
** A call to this routine tells the pager that if a rollback occurs,
** it is not necessary to restore the data on the given page.  This
** means that the pager does not have to record the given page in the
** rollback journal.
................................................................................
** the PgHdr.needRead flag is set) then this routine acts as a promise
** that we will never need to read the page content in the future.
** so the needRead flag can be cleared at this point.
*/
void sqlite3PagerDontRollback(DbPage *pPg){
  Pager *pPager = pPg->pPager;


  assert( pPager->state>=PAGER_RESERVED );

  /* If the journal file is not open, or DontWrite() has been called on
  ** this page (DontWrite() sets the alwaysRollback flag), then this
  ** function is a no-op.
  */
  if( pPager->journalOpen==0 || (pPg->flags&PGHDR_ALWAYS_ROLLBACK) 
   || pPager->alwaysRollback 
  ){

    return;
  }
  assert( !MEMDB );    /* For a memdb, pPager->journalOpen is always 0 */

#ifdef SQLITE_SECURE_DELETE
  if( pPg->inJournal || (int)pPg->pgno > pPager->origDbSize ){

    return;
  }
#endif

  /* If SECURE_DELETE is disabled, then there is no way that this
  ** routine can be called on a page for which sqlite3PagerDontWrite()
  ** has not been previously called during the same transaction.
................................................................................
  pPg->flags &= ~PGHDR_NEED_READ;
  if( pPager->stmtInUse ){
    assert( pPager->stmtSize >= pPager->origDbSize );
    sqlite3BitvecSet(pPager->pInStmt, pPg->pgno);
  }
  PAGERTRACE3("DONT_ROLLBACK page %d of %d\n", pPg->pgno, PAGERID(pPager));
  IOTRACE(("GARBAGE %p %d\n", pPager, pPg->pgno))

}


/*
** This routine is called to increment the database file change-counter,
** stored at byte 24 of the pager file.
*/
................................................................................
}

/*
** Sync the pager file to disk.
*/
int sqlite3PagerSync(Pager *pPager){
  int rc;

  rc = sqlite3OsSync(pPager->fd, pPager->sync_flags);

  return rc;
}

/*
** Sync the database file for the pager pPager. zMaster points to the name
** of a master journal file that should be written into the individual
** journal file. zMaster may be NULL, which is interpreted as no master
................................................................................
          pPager->exclusiveMode!=0) ){
    assert( pPager->dirtyCache==0 || pPager->journalOpen==0 );
    return SQLITE_OK;
  }

  PAGERTRACE4("DATABASE SYNC: File=%s zMaster=%s nTrunc=%d\n", 
      pPager->zFilename, zMaster, nTrunc);


  /* If this is an in-memory db, or no pages have been written to, or this
  ** function has already been called, it is a no-op.
  */
  if( pPager->state!=PAGER_SYNCED && !MEMDB && pPager->dirtyCache ){
    PgHdr *pPg;

................................................................................
    /* pager_incr_changecounter() may attempt to obtain an exclusive
     * lock to spill the cache and return IOERR_BLOCKED. But since 
     * there is no chance the cache is inconsistent, it is
     * better to return SQLITE_BUSY.
     */
    rc = SQLITE_BUSY;
  }

  return rc;
}


/*
** Commit all changes to the database and release the write lock.
**
................................................................................
  }
  if( pPager->dbModified==0 &&
        (pPager->journalMode!=PAGER_JOURNALMODE_DELETE ||
          pPager->exclusiveMode!=0) ){
    assert( pPager->dirtyCache==0 || pPager->journalOpen==0 );
    return SQLITE_OK;
  }

  PAGERTRACE2("COMMIT %d\n", PAGERID(pPager));
  if( MEMDB ){
    sqlite3PcacheCommit(pPager->pPCache, 0);
    sqlite3PcacheCleanAll(pPager->pPCache);
    sqlite3PcacheSetFlags(pPager->pPCache, 
       ~(PGHDR_IN_JOURNAL | PGHDR_NEED_SYNC | PGHDR_ALWAYS_ROLLBACK), 0
    );
    pPager->state = PAGER_SHARED;
  }else{
    assert( pPager->state==PAGER_SYNCED || !pPager->dirtyCache );
    rc = pager_end_transaction(pPager, pPager->setMaster);
    rc = pager_error(pPager, rc);
  }

  return rc;
}

/*
** Rollback all changes.  The database falls back to PAGER_SHARED mode.
** All in-memory cache pages revert to their original data contents.
** The journal is deleted.
................................................................................
** unless a prior malloc() failed (SQLITE_NOMEM).  Appropriate error
** codes are returned for all these occasions.  Otherwise,
** SQLITE_OK is returned.
*/
int sqlite3PagerRollback(Pager *pPager){
  int rc = SQLITE_OK;
  PAGERTRACE2("ROLLBACK %d\n", PAGERID(pPager));

  if( MEMDB ){
    sqlite3PcacheRollback(pPager->pPCache, 1);
    sqlite3PcacheRollback(pPager->pPCache, 0);
    sqlite3PcacheCleanAll(pPager->pPCache);
    sqlite3PcacheSetFlags(pPager->pPCache, 
       ~(PGHDR_IN_JOURNAL | PGHDR_NEED_SYNC | PGHDR_ALWAYS_ROLLBACK), 0
    );
................................................................................

    /* If an error occurs during a ROLLBACK, we can no longer trust the pager
    ** cache. So call pager_error() on the way out to make any error 
    ** persistent.
    */
    rc = pager_error(pPager, rc);
  }

  return rc;
}

/*
** Return TRUE if the database file is opened read-only.  Return FALSE
** if the database is (in theory) writable.
*/
................................................................................

#ifdef SQLITE_TEST
/*
** This routine is used for testing and analysis only.
*/
int *sqlite3PagerStats(Pager *pPager){
  static int a[11];

  a[0] = sqlite3PcacheRefCount(pPager->pPCache);
  a[1] = sqlite3PcachePagecount(pPager->pPCache);
  a[2] = sqlite3PcacheGetCachesize(pPager->pPCache);
  a[3] = pPager->dbSize;
  a[4] = pPager->state;
  a[5] = pPager->errCode;
  a[6] = pPager->nHit;
  a[7] = pPager->nMiss;
  a[8] = 0;  /* Used to be pPager->nOvfl */
  a[9] = pPager->nRead;
  a[10] = pPager->nWrite;

  return a;
}
int sqlite3PagerIsMemdb(Pager *pPager){
  return MEMDB;
}
#endif

................................................................................
    sqlite3BitvecDestroy(pPager->pInStmt);
    pPager->pInStmt = 0;
  }
  return rc;
}
int sqlite3PagerStmtBegin(Pager *pPager){
  int rc;

  rc = pagerStmtBegin(pPager);

  return rc;
}

/*
** Commit a statement.
*/
int sqlite3PagerStmtCommit(Pager *pPager){

  if( pPager->stmtInUse ){
    PAGERTRACE2("STMT-COMMIT %d\n", PAGERID(pPager));
    if( !MEMDB ){
      /* sqlite3OsTruncate(pPager->stfd, 0); */
      sqlite3BitvecDestroy(pPager->pInStmt);
      pPager->pInStmt = 0;
    }else{
      sqlite3PcacheCommit(pPager->pPCache, 1);
    }
    pPager->stmtNRec = 0;
    pPager->stmtInUse = 0;
  }
  pPager->stmtAutoopen = 0;

  return SQLITE_OK;
}

/*
** Rollback a statement.
*/
int sqlite3PagerStmtRollback(Pager *pPager){
  int rc;

  if( pPager->stmtInUse ){
    PAGERTRACE2("STMT-ROLLBACK %d\n", PAGERID(pPager));
    if( MEMDB ){
      sqlite3PcacheRollback(pPager->pPCache, 1);
      pPager->dbSize = pPager->stmtSize;
      pager_truncate_cache(pPager);
      rc = SQLITE_OK;
................................................................................
      rc = pager_stmt_playback(pPager);
    }
    sqlite3PagerStmtCommit(pPager);
  }else{
    rc = SQLITE_OK;
  }
  pPager->stmtAutoopen = 0;

  return rc;
}

/*
** Return the full pathname of the database file.
*/
const char *sqlite3PagerFilename(Pager *pPager){
................................................................................

  assert( pPg->nRef>0 );

  PAGERTRACE5("MOVE %d page %d (needSync=%d) moves to %d\n", 
      PAGERID(pPager), pPg->pgno, (pPg->flags&PGHDR_NEED_SYNC)?1:0, pgno);
  IOTRACE(("MOVE %p %d %d\n", pPager, pPg->pgno, pgno))



  pager_get_content(pPg);

  /* If the journal needs to be sync()ed before page pPg->pgno can
  ** be written to, store pPg->pgno in local variable needSyncPgno.
  **
  ** If the isCommit flag is set, there is no need to remember that
  ** the journal needs to be sync()ed before database page pPg->pgno 
................................................................................
    PgHdr *pPgHdr;
    assert( pPager->needSync );
    rc = sqlite3PagerGet(pPager, needSyncPgno, &pPgHdr);
    if( rc!=SQLITE_OK ){
      if( pPager->pInJournal && (int)needSyncPgno<=pPager->origDbSize ){
        sqlite3BitvecClear(pPager->pInJournal, needSyncPgno);
      }

      return rc;
    }
    pPager->needSync = 1;
    pPgHdr->flags |= PGHDR_NEED_SYNC;
    pPgHdr->flags |= PGHDR_IN_JOURNAL;
    makeDirty(pPgHdr);
    sqlite3PagerUnref(pPgHdr);
  }


  return SQLITE_OK;
}
#endif

/*
** Return a pointer to the data for the specified page.
*/

Changes to src/pcache.c.

7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

28
29
30
31
32
33
34

35
36
37
38
39
40
41
42
43
44
45
..
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71

72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
...
127
128
129
130
131
132
133


































134
135
136
137
138
139
140
...
201
202
203
204
205
206
207



208
209
210
211
212
213
214
215
216















217
218
219
220
221
222
223



224
225
226
227
228
229











230
231
232
233
234
235
236

237
238
239
240
241
242
243

244
245
246
247
248
249

250
251
252
253
254
255
256
...
257
258
259
260
261
262
263
264

265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
...
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
...
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
...
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
...
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461








462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489

490
491
492
493
494
495
496
...
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520







521
522
523



















524
525
526

527
528
529
530
531
532
533
...
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
...
582
583
584
585
586
587
588

589
590
591
592
593

594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
...
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636

637
638
639

640
641
642
643

644
645
646
647
648
649
650
...
664
665
666
667
668
669
670

671
672
673
674
675
676

677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699





700
701
702
703
704
705
706
...
707
708
709
710
711
712
713
714
715
716
717

718
719
720
721
722
723
724
...
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
...
745
746
747
748
749
750
751
752
753
754
755
756
757

758
759






760
761
762
763
764
765
766
767
768
769
770
771
772
773




774

775

776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
...
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827

828

829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851

852
853
854
855

856
857
858
859
860
861
862
...
866
867
868
869
870
871
872
873
874
875
876
877
878
879

880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
...
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
...
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
...
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
...
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
...
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
....
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
....
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
....
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129

1130
1131


1132
1133
1134
1135
1136
1137
1138
....
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
....
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
**    May you do good and not evil.
**    May you find forgiveness for yourself and forgive others.
**    May you share freely, never taking more than you give.
**
*************************************************************************
** This file implements that page cache.
**
** @(#) $Id: pcache.c,v 1.13 2008/08/25 14:49:42 danielk1977 Exp $
*/
#include "sqliteInt.h"

/*
** A complete page cache is an instance of this structure.
*/
struct PCache {
  PCache *pNextAll, *pPrevAll;        /* List of all page caches */
  int szPage;                         /* Size of every page in this cache */
  int szExtra;                        /* Size of extra space for each page */
  int nHash;                          /* Number of slots in apHash[] */
  int nPage;                          /* Total number of pages in apHash */
  int nMax;                           /* Configured cache size */

  PgHdr **apHash;                     /* Hash table for fast lookup by pgno */
  int bPurgeable;                     /* True if pages are on backing store */
  void (*xDestroy)(PgHdr*);           /* Called when refcnt goes 1->0 */
  int (*xStress)(void*,PgHdr*);       /* Call to try make a page clean */
  void *pStress;                      /* Argument to xStress */
  PgHdr *pClean;                      /* List of clean pages in use */
  PgHdr *pDirty;                      /* List of dirty pages */

  int nRef;                           /* Number of outstanding page refs */

  int iInUseMM;
  int iInUseDB;
};

/*
** Free slots in the page block allocator
*/
typedef struct PgFreeslot PgFreeslot;
struct PgFreeslot {
................................................................................
** cached pages stored to mxPage. In this case mxPagePurgeable is not 
** used.
**
** If mxPage is zero, then the system tries to limit the number of
** pages held by purgable caches to mxPagePurgeable.
**
** The doubly-linked list that runs between pcache.pLruHead and 
** pcache.pLruTail contains all pages in the system with a zero 
** reference count. The pcache.pLruSynced variable points to the last
** (closest to pcache.pLruTail) entry in this list that does not have
** the PGHDR_NEED_SYNC flag set. This is the page that the pcacheRecycle()
** function will try to recycle.
*/
static struct PCacheGlobal {
  int isInit;                         /* True when initialized */
  sqlite3_mutex *mutex_mem2;          /* static mutex MUTEX_STATIC_MEM2 */
  sqlite3_mutex *mutex_lru;           /* static mutex MUTEX_STATIC_LRU */

  PCache *pAll;                       /* list of all page caches */
  int nPage;                          /* Number of pages */
  int nPurgeable;                     /* Number of pages in purgable caches */
  int mxPage;                         /* Globally configured page maximum */
  int mxPagePurgeable;                /* Purgeable page maximum */
  PgHdr *pLruHead, *pLruTail;         /* Global LRU list of unused pages */
  PgHdr *pLruSynced;                  /* Last synced entry in LRU list  */

  /* Variables related to SQLITE_CONFIG_PAGECACHE settings. */
  int szSlot;                         /* Size of each free slot */
  void *pStart, *pEnd;                /* Bounds of pagecache malloc range */
  PgFreeslot *pFree;                  /* Free page blocks */
} pcache = {0};

/*
** All global variables used by this module (most of which are grouped 
** together in global structure "pcache" above) except the list of all
** pager-caches starting with pcache.pAll, are protected by the static 
** SQLITE_MUTEX_STATIC_LRU mutex. A pointer to this mutex is stored in
** variable "pcache.mutex_lru".
**
** Access to the contents of the individual PCache structures is not 
** protected. It is the job of the caller to ensure that these structures
** are accessed in a thread-safe manner. However, this module provides the
** functions sqlite3PcacheLock() and sqlite3PcacheUnlock() that may be used
** by the caller to increment/decrement a lock-count on an individual 
** pager-cache object. This module guarantees that the xStress() callback
** will not be invoked on a pager-cache with a non-zero lock-count except
** from within a call to sqlite3PcacheFetch() on the same pager. A call
** to sqlite3PcacheLock() may block if such an xStress() call is currently 
** underway.
**
** Before the xStress callback of a pager-cache (PCache) is invoked, the
** SQLITE_MUTEX_STATIC_MEM2 mutex is obtained.
**
** Deadlock within the module is avoided by never blocking on the MEM2 
** mutex while the LRU mutex is held.
*/

#define pcacheEnterGlobal() sqlite3_mutex_enter(pcache.mutex_lru)
#define pcacheExitGlobal()  sqlite3_mutex_leave(pcache.mutex_lru)

/********************************** Linked List Management ********************/

#ifndef NDEBUG
/*
** This routine verifies that the number of entries in the hash table
** is pCache->nPage.  This routine is used within assert() statements
................................................................................
    for(p=pCache->apHash[i]; p; p=p->pNextHash){
      nPage++;
    }
  }
  assert( nPage==pCache->nPage );
  return 1;
}


































#endif

/*
** Remove a page from its hash table (PCache.apHash[]).
*/
static void pcacheRemoveFromHash(PgHdr *pPage){
  if( pPage->pPrevHash ){
................................................................................
}

/*
** Remove a page from a linked list that is headed by *ppHead.
** *ppHead is either PCache.pClean or PCache.pDirty.
*/
static void pcacheRemoveFromList(PgHdr **ppHead, PgHdr *pPage){



  if( pPage->pPrev ){
    pPage->pPrev->pNext = pPage->pNext;
  }else{
    assert( *ppHead==pPage );
    *ppHead = pPage->pNext;
  }
  if( pPage->pNext ){
    pPage->pNext->pPrev = pPage->pPrev;
  }















}

/*
** Add a page from a linked list that is headed by *ppHead.
** *ppHead is either PCache.pClean or PCache.pDirty.
*/
static void pcacheAddToList(PgHdr **ppHead, PgHdr *pPage){



  if( (*ppHead) ){
    (*ppHead)->pPrev = pPage;
  }
  pPage->pNext = *ppHead;
  pPage->pPrev = 0;
  *ppHead = pPage;











}

/*
** Remove a page from the global LRU list
*/
static void pcacheRemoveFromLruList(PgHdr *pPage){
  assert( sqlite3_mutex_held(pcache.mutex_lru) );

  if( pPage->pCache->bPurgeable==0 ) return;
  if( pPage==pcache.pLruSynced ){
    PgHdr *p;
    for(p=pPage->pPrevLru; p && (p->flags&PGHDR_NEED_SYNC); p=p->pPrevLru);
    pcache.pLruSynced = p;
  }
  if( pPage->pNextLru ){

    pPage->pNextLru->pPrevLru = pPage->pPrevLru;
  }else{
    assert( pcache.pLruTail==pPage );
    pcache.pLruTail = pPage->pPrevLru;
  }
  if( pPage->pPrevLru ){

    pPage->pPrevLru->pNextLru = pPage->pNextLru;
  }else{
    assert( pcache.pLruHead==pPage );
    pcache.pLruHead = pPage->pNextLru;
  }
}

................................................................................
/*
** Add a page to the global LRU list.  The page is normally added
** to the front of the list so that it will be the last page recycled.
** However, if the PGHDR_REUSE_UNLIKELY bit is set, the page is added
** to the end of the LRU list so that it will be the next to be recycled.
*/
static void pcacheAddToLruList(PgHdr *pPage){
  assert( sqlite3_mutex_held(pcache.mutex_lru) );

  if( pPage->pCache->bPurgeable==0 ) return;
  if( pcache.pLruTail && (pPage->flags & PGHDR_REUSE_UNLIKELY)!=0 ){
    /* If reuse is unlikely.  Put the page at the end of the LRU list
    ** where it will be recycled sooner rather than later. 
    */
    assert( pcache.pLruHead );
    pPage->pNextLru = 0;
    pPage->pPrevLru = pcache.pLruTail;
    pcache.pLruTail->pNextLru = pPage;
    pcache.pLruTail = pPage;
    pPage->flags &= ~PGHDR_REUSE_UNLIKELY;
    if( 0==(pPage->flags&PGHDR_NEED_SYNC) ){
      pcache.pLruSynced = pPage;
    }
  }else{
    /* If reuse is possible. the page goes at the beginning of the LRU
    ** list so that it will be the last to be recycled.
    */
    if( pcache.pLruHead ){
      pcache.pLruHead->pPrevLru = pPage;
    }
    pPage->pNextLru = pcache.pLruHead;
    pcache.pLruHead = pPage;
    pPage->pPrevLru = 0;
    if( pcache.pLruTail==0 ){
      pcache.pLruTail = pPage;
    }
    if( pcache.pLruSynced==0 && 0==(pPage->flags&PGHDR_NEED_SYNC) ){
      pcache.pLruSynced = pPage;
    }
  }
}

/*********************************************** Memory Allocation ***********
**
** Initialize the page cache memory pool.
**
................................................................................

/*
** Allocate a page cache line.  Look in the page cache memory pool first
** and use an element from it first if available.  If nothing is available
** in the page cache memory pool, go to the general purpose memory allocator.
*/
void *pcacheMalloc(int sz, PCache *pCache){
  assert( sqlite3_mutex_held(pcache.mutex_lru) );
  if( sz<=pcache.szSlot && pcache.pFree ){
    PgFreeslot *p = pcache.pFree;
    pcache.pFree = p->pNext;
    sqlite3StatusSet(SQLITE_STATUS_PAGECACHE_SIZE, sz);
    sqlite3StatusAdd(SQLITE_STATUS_PAGECACHE_USED, 1);
    return (void*)p;
  }else{
................................................................................
    ** so that if the attempt to allocate a new buffer causes the the 
    ** configured soft-heap-limit to be breached, it will be possible to
    ** reclaim memory from this pager-cache. Because sqlite3PcacheLock() 
    ** might block on the MEM2 mutex, it has to be called before re-entering
    ** the global LRU mutex.
    */
    pcacheExitGlobal();
    sqlite3PcacheUnlock(pCache);
    p = sqlite3Malloc(sz);
    sqlite3PcacheLock(pCache);
    pcacheEnterGlobal();

    if( p ){
      sz = sqlite3MallocSize(p);
      sqlite3StatusAdd(SQLITE_STATUS_PAGECACHE_OVERFLOW, sz);
    }
    return p;
................................................................................
  return p;
}

/*
** Release a pager memory allocation
*/
void pcacheFree(void *p){
  assert( sqlite3_mutex_held(pcache.mutex_lru) );
  if( p==0 ) return;
  if( p>=pcache.pStart && p<pcache.pEnd ){
    PgFreeslot *pSlot;
    sqlite3StatusAdd(SQLITE_STATUS_PAGECACHE_USED, -1);
    pSlot = (PgFreeslot*)p;
    pSlot->pNext = pcache.pFree;
    pcache.pFree = pSlot;
................................................................................

/*
** Allocate a new page.
*/
static PgHdr *pcachePageAlloc(PCache *pCache){
  PgHdr *p;
  int sz = sizeof(*p) + pCache->szPage + pCache->szExtra;
  assert( sqlite3_mutex_held(pcache.mutex_lru) );
  p = pcacheMalloc(sz, pCache);
  if( p==0 ) return 0;
  memset(p, 0, sizeof(PgHdr));
  p->pData = (void*)&p[1];
  p->pExtra = (void*)&((char*)p->pData)[pCache->szPage];
  pcache.nPage++;
  if( pCache->bPurgeable ){
    pcache.nPurgeable++;
  }

  return p;
}

/*
** Deallocate a page
*/
static void pcachePageFree(PgHdr *p){
  assert( sqlite3_mutex_held(pcache.mutex_lru) );
  pcache.nPage--;
  if( p->pCache->bPurgeable ){
    pcache.nPurgeable--;
  }
  pcacheFree(p->apSave[0]);
  pcacheFree(p->apSave[1]);
  pcacheFree(p);
}

#ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT
/*
** Return the number of bytes that will be returned to the heap when
** the argument is passed to pcachePageFree().
*/
static int pcachePageSize(PgHdr *p){
  assert( sqlite3_mutex_held(pcache.mutex_lru) );
  assert( !pcache.pStart );
  assert( p->apSave[0]==0 );
  assert( p->apSave[1]==0 );
  assert( p && p->pCache );
  return sqlite3MallocSize(p);
}
#endif

static int pcacheRecyclePage(PgHdr *p, PCache *pCache){
  assert( sqlite3_mutex_held(pcache.mutex_lru) );
  assert( sqlite3_mutex_held(pcache.mutex_mem2) );

  PCache *pC = p->pCache;
  assert( pC->iInUseMM==0 );
  pC->iInUseMM = 1;
  if( pC->xStress && (pC->iInUseDB==0 || pC==pCache) ){
    pcacheExitGlobal();
    pC->xStress(pC->pStress, p);
    pcacheEnterGlobal();
  }
  pC->iInUseMM = 0;

  return (p->flags&PGHDR_DIRTY);
}

/*
** Recycle a page from the global LRU list. If no page can be recycled, 
** return NULL. Otherwise, the pointer returned points to a PgHdr 
** object that has been removed from all lists and hash tables in
** which is was referenced. The caller may reuse the allocation directly
** or may pass it to pcachePageFree() to return the memory to the heap
** (or pcache.pFree list).








*/ 
static PgHdr *pcacheRecycle(PCache *pCache){
  PgHdr *p = 0;

  assert( pcache.isInit );
  assert( sqlite3_mutex_held(pcache.mutex_lru) );

  if( SQLITE_OK==sqlite3_mutex_try(pcache.mutex_mem2) ){
    p = pcache.pLruSynced;
    while( p && (p->flags&PGHDR_DIRTY) && pcacheRecyclePage(p, pCache) ){
      do { p = p->pPrevLru; } while( p && (p->flags&PGHDR_NEED_SYNC) );
    }
    if( !p ){
      p = pcache.pLruTail;
      while( p && (p->flags&PGHDR_DIRTY) && pcacheRecyclePage(p, pCache) ){
        do { p = p->pPrevLru; } while( p && 0==(p->flags&PGHDR_NEED_SYNC) );
      }
    }
    sqlite3_mutex_leave(pcache.mutex_mem2);
  }

  if( p ){
    pcacheRemoveFromLruList(p);
    pcacheRemoveFromHash(p);
    pcacheRemoveFromList(&p->pCache->pClean, p);

    /* If the always-rollback flag is set on the page being recycled, set 
    ** the always-rollback flag on the corresponding pager.

    */
    if( p->flags&PGHDR_ALWAYS_ROLLBACK ){
      assert(p->pPager);
      sqlite3PagerAlwaysRollback(p->pPager);
    }
  }

................................................................................
** Return a pointer to the new page, or NULL if an OOM condition occurs.
*/
static PgHdr *pcacheRecycleOrAlloc(PCache *pCache){
  PgHdr *p = 0;

  int szPage = pCache->szPage;
  int szExtra = pCache->szExtra;
  int bPurg = pCache->bPurgeable;

  assert( pcache.isInit );
  assert( sqlite3_mutex_notheld(pcache.mutex_lru) );

  pcacheEnterGlobal();

  if( (pcache.mxPage && pcache.nPage>=pcache.mxPage) 
   || (!pcache.mxPage && bPurg && pcache.nPurgeable>=pcache.mxPagePurgeable)







  ){
    /* If the above test succeeds, then try to obtain a buffer by recycling
    ** an existing page. */



















    p = pcacheRecycle(pCache);
  }


  if( p && (p->pCache->szPage!=szPage || p->pCache->szExtra!=szExtra) ){
    pcachePageFree(p);
    p = 0;
  }

  if( !p ){
    p = pcachePageAlloc(pCache);
................................................................................
** Initialize and shutdown the page cache subsystem. Neither of these 
** functions are threadsafe.
*/
int sqlite3PcacheInitialize(void){
  assert( pcache.isInit==0 );
  memset(&pcache, 0, sizeof(pcache));
  if( sqlite3Config.bCoreMutex ){
    pcache.mutex_lru = sqlite3_mutex_alloc(SQLITE_MUTEX_STATIC_LRU);
    pcache.mutex_mem2 = sqlite3_mutex_alloc(SQLITE_MUTEX_STATIC_MEM2);
    if( pcache.mutex_lru==0 || pcache.mutex_mem2==0 ){
      return SQLITE_NOMEM;
    }
  }
  pcache.isInit = 1;
  return SQLITE_OK;
}
void sqlite3PcacheShutdown(void){
................................................................................
  p->szPage = szPage;
  p->szExtra = szExtra;
  p->bPurgeable = bPurgeable;
  p->xDestroy = xDestroy;
  p->xStress = xStress;
  p->pStress = pStress;
  p->nMax = 100;


  if( bPurgeable ){
    pcacheEnterGlobal();
    pcache.mxPagePurgeable += p->nMax;
    pcacheExitGlobal();

  }

  /* Add the new pager-cache to the list of caches starting at pcache.pAll */
  pcacheEnterGlobal();
  p->pNextAll = pcache.pAll;
  if( pcache.pAll ){
    pcache.pAll->pPrevAll = p;
  }
  p->pPrevAll = 0;
  pcache.pAll = p;
  pcacheExitGlobal();
}

/*
** Change the page size for PCache object.  This can only happen
** when the cache is empty.
*/
................................................................................
  int createFlag,       /* If true, create page if it does not exist already */
  PgHdr **ppPage        /* Write the page here */
){
  PgHdr *pPage;
  assert( pcache.isInit );
  assert( pCache!=0 );
  assert( pgno>0 );
  assert( pCache->iInUseDB || pCache->iInUseMM );

  /* Search the hash table for the requested page. Exit early if it is found. */
  if( pCache->apHash ){
    u32 h = pgno % pCache->nHash;
    for(pPage=pCache->apHash[h]; pPage; pPage=pPage->pNextHash){
      if( pPage->pgno==pgno ){
        if( pPage->nRef==0 /* && (pPage->flags & PGHDR_DIRTY)==0 */ ){

          pcacheEnterGlobal();
          pcacheRemoveFromLruList(pPage);
          pcacheExitGlobal();

        }
        if( (pPage->nRef++)==0 ){
          pCache->nRef++;
        }

        *ppPage = pPage;
        return SQLITE_OK;
      }
    }
  }

  if( createFlag ){
................................................................................
    pPage->pPager = 0;
    pPage->flags = 0;
    pPage->pDirty = 0;
    pPage->pgno = pgno;
    pPage->pCache = pCache;
    pPage->nRef = 1;
    pCache->nRef++;

    pcacheAddToList(&pCache->pClean, pPage);
    pcacheAddToHash(pPage);
  }else{
    *ppPage = 0;
  }


  return SQLITE_OK;
}

/*
** Dereference a page.  When the reference count reaches zero,
** move the page to the LRU list if it is clean.
*/
void sqlite3PcacheRelease(PgHdr *p){
  assert( p->nRef>0 );
  assert( p->pCache->iInUseDB || p->pCache->iInUseMM );
  p->nRef--;
  if( p->nRef==0 ){
    PCache *pCache = p->pCache;
    pCache->nRef--;
    if( p->pCache->xDestroy ){
      p->pCache->xDestroy(p);
    }
#if 0
    if( (p->flags & PGHDR_DIRTY)!=0 ) return;
#endif
    pcacheEnterGlobal();
    pcacheAddToLruList(p);
    pcacheExitGlobal();





  }
}

void sqlite3PcacheRef(PgHdr *p){
  assert(p->nRef>0);
  p->nRef++;
}
................................................................................

/*
** Drop a page from the cache.  This should be the only reference to
** the page.
*/
void sqlite3PcacheDrop(PgHdr *p){
  PCache *pCache;
  assert( p->pCache->iInUseDB );
  assert( p->nRef==1 );
  pCache = p->pCache;
  pCache->nRef--;

  if( p->flags & PGHDR_DIRTY ){
    pcacheRemoveFromList(&pCache->pDirty, p);
  }else{
    pcacheRemoveFromList(&pCache->pClean, p);
  }
  pcacheRemoveFromHash(p);
  pcacheEnterGlobal();
................................................................................

/*
** Make sure the page is marked as dirty.  If it isn't dirty already,
** make it so.
*/
void sqlite3PcacheMakeDirty(PgHdr *p){
  PCache *pCache;
  assert( p->pCache->iInUseDB );
  p->flags &= ~PGHDR_DONT_WRITE;
  if( p->flags & PGHDR_DIRTY ) return;
  assert( (p->flags & PGHDR_DIRTY)==0 );
  assert( p->nRef>0 );
  pCache = p->pCache;
  pcacheRemoveFromList(&pCache->pClean, p);
  pcacheAddToList(&pCache->pDirty, p);
................................................................................

/*
** Make sure the page is marked as clean.  If it isn't clean already,
** make it so.
*/
void sqlite3PcacheMakeClean(PgHdr *p){
  PCache *pCache;
  assert( p->pCache->iInUseDB || p->pCache->iInUseMM );
  if( (p->flags & PGHDR_DIRTY)==0 ) return;
  assert( p->apSave[0]==0 && p->apSave[1]==0 );
  assert( p->flags & PGHDR_DIRTY );
  pCache = p->pCache;
  pcacheRemoveFromList(&pCache->pDirty, p);

  pcacheAddToList(&pCache->pClean, p);
  p->flags &= ~PGHDR_DIRTY;






}

/*
** Make every page in the cache clean.
*/
void sqlite3PcacheCleanAll(PCache *pCache){
  PgHdr *p;
  assert( pCache->iInUseDB );
  pcacheEnterGlobal();
  while( (p = pCache->pDirty)!=0 ){
    assert( p->apSave[0]==0 && p->apSave[1]==0 );
    pcacheRemoveFromList(&pCache->pDirty, p);
    pcacheAddToList(&pCache->pClean, p);
    p->flags &= ~PGHDR_DIRTY;




  }

  sqlite3PcacheAssertFlags(pCache, 0, PGHDR_DIRTY);

  pcacheExitGlobal();
}

/*
** Change the page number of page p to newPgno. If newPgno is 0, then the
** page object is added to the clean-list and the PGHDR_REUSE_UNLIKELY 
** flag set.
*/
void sqlite3PcacheMove(PgHdr *p, Pgno newPgno){
  assert( p->pCache->iInUseDB );
  pcacheRemoveFromHash(p);
  p->pgno = newPgno;
  if( newPgno==0 ){
    p->flags |= PGHDR_REUSE_UNLIKELY;
    pcacheEnterGlobal();
    pcacheFree(p->apSave[0]);
    pcacheFree(p->apSave[1]);
................................................................................
    p->apSave[0] = 0;
    p->apSave[1] = 0;
    sqlite3PcacheMakeClean(p);
  }
  pcacheAddToHash(p);
}

/*
** Set the global maximum number of pages. Return the previous value.
*/
void sqlite3PcacheGlobalMax(int mx){
  pcacheEnterGlobal();
  pcache.mxPage = mx;
  pcacheExitGlobal();
}

/*
** Remove all content from a page cache
*/
void pcacheClear(PCache *pCache){
  PgHdr *p, *pNext;
  assert( sqlite3_mutex_held(pcache.mutex_lru) );
  for(p=pCache->pClean; p; p=pNext){
    pNext = p->pNext;
    pcacheRemoveFromLruList(p);
    pcachePageFree(p);
  }
  for(p=pCache->pDirty; p; p=pNext){
    pNext = p->pNext;
    pcacheRemoveFromLruList(p);
    pcachePageFree(p);
  }
  pCache->pClean = 0;
  pCache->pDirty = 0;

  pCache->nPage = 0;

  memset(pCache->apHash, 0, pCache->nHash*sizeof(pCache->apHash[0]));
}


/*
** Drop every cache entry whose page number is greater than "pgno".
*/
void sqlite3PcacheTruncate(PCache *pCache, Pgno pgno){
  PgHdr *p, *pNext;
  PgHdr *pDirty = pCache->pDirty;
  assert( pCache->iInUseDB );
  pcacheEnterGlobal();
  for(p=pCache->pClean; p||pDirty; p=pNext){
    if( !p ){
      p = pDirty;
      pDirty = 0;
    }
    pNext = p->pNext;
    if( p->pgno>pgno ){
      if( p->nRef==0 ){
        pcacheRemoveFromHash(p);
        if( p->flags&PGHDR_DIRTY ){
          pcacheRemoveFromList(&pCache->pDirty, p);

        }else{
          pcacheRemoveFromList(&pCache->pClean, p);
        }
        pcacheRemoveFromLruList(p);

        pcachePageFree(p);
      }else{
        /* If there are references to the page, it cannot be freed. In this
        ** case, zero the page content instead.
        */
        memset(p->pData, 0, pCache->szPage);
      }
................................................................................
}


/*
** Close a cache.
*/
void sqlite3PcacheClose(PCache *pCache){
  assert( pCache->iInUseDB==1 );
  pcacheEnterGlobal();

  /* Free all the pages used by this pager and remove them from the LRU list. */
  pcacheClear(pCache);
  if( pCache->bPurgeable ){
    pcache.mxPagePurgeable -= pCache->nMax;

  }
  sqlite3_free(pCache->apHash);

  /* Now remove the pager-cache structure itself from the list of
  ** all such structures headed by pcache.pAll. 
  */
  assert(pCache==pcache.pAll || pCache->pPrevAll);
  assert(pCache->pNextAll==0 || pCache->pNextAll->pPrevAll==pCache);
  assert(pCache->pPrevAll==0 || pCache->pPrevAll->pNextAll==pCache);
  if( pCache->pPrevAll ){
    pCache->pPrevAll->pNextAll = pCache->pNextAll;
  }else{
    pcache.pAll = pCache->pNextAll;
  }
  if( pCache->pNextAll ){
    pCache->pNextAll->pPrevAll = pCache->pPrevAll;
  }

  pcacheExitGlobal();
}

/*
** Preserve the content of the page, if it has not been preserved
** already.  If idJournal==0 then this is for the overall transaction.
** If idJournal==1 then this is for the statement journal.
................................................................................
** This routine is used for in-memory databases only.
**
** Return SQLITE_OK or SQLITE_NOMEM if a memory allocation fails.
*/
int sqlite3PcachePreserve(PgHdr *p, int idJournal){
  void *x;
  int sz;
  assert( p->pCache->iInUseDB );
  assert( p->pCache->bPurgeable==0 );
  if( !p->apSave[idJournal] ){
    sz = p->pCache->szPage;
    p->apSave[idJournal] = x = sqlite3PageMalloc( sz );
    if( x==0 ) return SQLITE_NOMEM;
    memcpy(x, p->pData, sz);
  }
................................................................................
}

/*
** Commit a change previously preserved.
*/
void sqlite3PcacheCommit(PCache *pCache, int idJournal){
  PgHdr *p;
  assert( pCache->iInUseDB );
  pcacheEnterGlobal();     /* Mutex is required to call pcacheFree() */
  for(p=pCache->pDirty; p; p=p->pNext){
    if( p->apSave[idJournal] ){
      pcacheFree(p->apSave[idJournal]);
      p->apSave[idJournal] = 0;
    }
  }
................................................................................

/*
** Rollback a change previously preserved.
*/
void sqlite3PcacheRollback(PCache *pCache, int idJournal){
  PgHdr *p;
  int sz;
  assert( pCache->iInUseDB );
  pcacheEnterGlobal();     /* Mutex is required to call pcacheFree() */
  sz = pCache->szPage;
  for(p=pCache->pDirty; p; p=p->pNext){
    if( p->apSave[idJournal] ){
      memcpy(p->pData, p->apSave[idJournal], sz);
      pcacheFree(p->apSave[idJournal]);
      p->apSave[idJournal] = 0;
................................................................................
}

/* 
** Assert flags settings on all pages.  Debugging only.
*/
void sqlite3PcacheAssertFlags(PCache *pCache, int trueMask, int falseMask){
  PgHdr *p;
  assert( pCache->iInUseDB || pCache->iInUseMM );
  for(p=pCache->pDirty; p; p=p->pNext){
    assert( (p->flags&trueMask)==trueMask );
    assert( (p->flags&falseMask)==0 );
  }
  for(p=pCache->pClean; p; p=p->pNext){
    assert( (p->flags&trueMask)==trueMask );
    assert( (p->flags&falseMask)==0 );
................................................................................
  }
}

/* 
** Discard the contents of the cache.
*/
int sqlite3PcacheClear(PCache *pCache){
  assert( pCache->iInUseDB );
  assert(pCache->nRef==0);
  pcacheEnterGlobal();
  pcacheClear(pCache);
  pcacheExitGlobal();
  return SQLITE_OK;
}

................................................................................
}

/*
** Return a list of all dirty pages in the cache, sorted by page number.
*/
PgHdr *sqlite3PcacheDirtyList(PCache *pCache){
  PgHdr *p;
  assert( pCache->iInUseDB );
  for(p=pCache->pDirty; p; p=p->pNext){
    p->pDirty = p->pNext;
  }
  return pcacheSortDirtyList(pCache->pDirty);
}

/* 
................................................................................
  return pCache->nRef;
}

/* 
** Return the total number of pages in the cache.
*/
int sqlite3PcachePagecount(PCache *pCache){
  assert( pCache->iInUseDB || pCache->iInUseMM );
  assert( pCache->nPage>=0 );
  return pCache->nPage;
}

#ifdef SQLITE_CHECK_PAGES
/*
** This function is used by the pager.c module to iterate through all 
** pages in the cache. At present, this is only required if the
** SQLITE_CHECK_PAGES macro (used for debugging) is specified.
*/
void sqlite3PcacheIterate(PCache *pCache, void (*xIter)(PgHdr *)){
  PgHdr *p;
  assert( pCache->iInUseDB || pCache->iInUseMM );
  for(p=pCache->pClean; p; p=p->pNext){
    xIter(p);
  }
  for(p=pCache->pDirty; p; p=p->pNext){
    xIter(p);
  }
}
................................................................................
/* 
** Set flags on all pages in the page cache 
*/
void sqlite3PcacheSetFlags(PCache *pCache, int andMask, int orMask){
  PgHdr *p;

  assert( (orMask&PGHDR_NEED_SYNC)==0 );
  assert( pCache->iInUseDB || pCache->iInUseMM );

  /* Obtain the global mutex before modifying any PgHdr.flags variables 
  ** or traversing the LRU list.
  */ 
  pcacheEnterGlobal();
  assert( sqlite3_mutex_held(pcache.mutex_lru) );

  for(p=pCache->pDirty; p; p=p->pNext){
    p->flags = (p->flags&andMask)|orMask;
  }
  for(p=pCache->pClean; p; p=p->pNext){
    p->flags = (p->flags&andMask)|orMask;
  }

  if( 0==(andMask&PGHDR_NEED_SYNC) ){

    for(p=pcache.pLruTail; p && (p->flags&PGHDR_NEED_SYNC); p=p->pPrevLru);
    pcache.pLruSynced = p;


  }

  pcacheExitGlobal();
}

/*
** Set the suggested cache-size value.
................................................................................
*/
void sqlite3PcacheSetCachesize(PCache *pCache, int mxPage){
  if( mxPage<10 ){
    mxPage = 10;
  }
  if( pCache->bPurgeable ){
    pcacheEnterGlobal();
    pcache.mxPagePurgeable -= pCache->nMax;
    pcache.mxPagePurgeable += mxPage;
    pcacheExitGlobal();
  }
  pCache->nMax = mxPage;
}

/*
** Lock a pager-cache.
*/
void sqlite3PcacheLock(PCache *pCache){
  if( pCache ){
    assert( sqlite3_mutex_notheld(pcache.mutex_lru) );
    pCache->iInUseDB++;
    if( pCache->iInUseMM && pCache->iInUseDB==1 ){
      pCache->iInUseDB = 0;
      sqlite3_mutex_enter(pcache.mutex_mem2);
      assert( pCache->iInUseMM==0 && pCache->iInUseDB==0 );
      pCache->iInUseDB = 1;
      sqlite3_mutex_leave(pcache.mutex_mem2);
    }
  }
}

/*
** Unlock a pager-cache.
*/
void sqlite3PcacheUnlock(PCache *pCache){
  if( pCache ){
    pCache->iInUseDB--;
    assert( pCache->iInUseDB>=0 );
  }
}

#ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT
/*
** This function is called to free superfluous dynamically allocated memory
** held by the pager system. Memory in use by any SQLite pager allocated
** by the current thread may be sqlite3_free()ed.
**
** nReq is the number of bytes of memory required. Once this much has
................................................................................
** of bytes of memory released.
*/
int sqlite3PcacheReleaseMemory(int nReq){
  int nFree = 0;
  if( pcache.pStart==0 ){
    PgHdr *p;
    pcacheEnterGlobal();
    while( (nReq<0 || nFree<nReq) && (p=pcacheRecycle(0)) ){
      nFree += pcachePageSize(p);
      pcachePageFree(p);
    }
    pcacheExitGlobal();
  }
  return nFree;
}
#endif /* SQLITE_ENABLE_MEMORY_MANAGEMENT */







|







<





>






|
>
|
<
|
<







 







|
|
|
<
<



<
|
>
|
|
|
<
<
|
<









|
<

|



|
<
<
<
<
<
<
<
<
<
<
<
<
<


|
|







 







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







 







>
>
>









>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







>
>
>






>
>
>
>
>
>
>
>
>
>
>






|
>

<
<
<
<
<

>






>







 







|
>











<
<
<













<
<
<







 







|







 







<

<







 







|







 







|





<

|

<







|
<

|












|








<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<

|
<
<
<
<
<
>
>
>
>
>
>
>
>
|
|

<
<
|

<
<
<
<
<
<
|
|
<
<
<
<
<
<
<





|
>







 







<


|



|
|
>
>
>
>
>
>
>

<
<
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|


>







 







|
<
|







 







>

<
|
|
|
>


<
<
<
<
<
<
<
<







 







|






|
>
|
|
|
>
|
<


>







 







>






>









<



<



|
|
|
|
|
|
>
>
>
>
>







 







<



>







 







<







 







<





>


>
>
>
>
>
>







<




<

>
>
>
>
|
>

>









|







 







<
<
<
<
<
<
<
<
<





|







<




>

>










<












>


<
|
>







 







<





|
>



<
<
<
<
<
<
<
<
<
<
<
<
<
<
<







 







<







 







<







 







<







 







<







 







<







 







<







 







<












<







 







<





<









>
|
|
>
>







 







|
|





<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<







 







|








7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

37

38
39
40
41
42
43
44
..
54
55
56
57
58
59
60
61
62
63


64
65
66

67
68
69
70
71


72

73
74
75
76
77
78
79
80
81
82

83
84
85
86
87
88













89
90
91
92
93
94
95
96
97
98
99
...
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
...
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284





285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
...
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320



321
322
323
324
325
326
327
328
329
330
331
332
333



334
335
336
337
338
339
340
...
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
...
377
378
379
380
381
382
383

384

385
386
387
388
389
390
391
...
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
...
425
426
427
428
429
430
431
432
433
434
435
436
437

438
439
440

441
442
443
444
445
446
447
448

449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471

















472
473





474
475
476
477
478
479
480
481
482
483
484


485
486






487
488







489
490
491
492
493
494
495
496
497
498
499
500
501
502
...
511
512
513
514
515
516
517

518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533


534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
...
572
573
574
575
576
577
578
579

580
581
582
583
584
585
586
587
...
611
612
613
614
615
616
617
618
619

620
621
622
623
624
625








626
627
628
629
630
631
632
...
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664

665
666
667
668
669
670
671
672
673
674
...
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711

712
713
714

715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
...
736
737
738
739
740
741
742

743
744
745
746
747
748
749
750
751
752
753
...
757
758
759
760
761
762
763

764
765
766
767
768
769
770
...
773
774
775
776
777
778
779

780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800

801
802
803
804

805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
...
832
833
834
835
836
837
838









839
840
841
842
843
844
845
846
847
848
849
850
851

852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868

869
870
871
872
873
874
875
876
877
878
879
880
881
882
883

884
885
886
887
888
889
890
891
892
...
896
897
898
899
900
901
902

903
904
905
906
907
908
909
910
911
912















913
914
915
916
917
918
919
...
921
922
923
924
925
926
927

928
929
930
931
932
933
934
...
936
937
938
939
940
941
942

943
944
945
946
947
948
949
...
952
953
954
955
956
957
958

959
960
961
962
963
964
965
...
969
970
971
972
973
974
975

976
977
978
979
980
981
982
...
983
984
985
986
987
988
989

990
991
992
993
994
995
996
....
1069
1070
1071
1072
1073
1074
1075

1076
1077
1078
1079
1080
1081
1082
....
1086
1087
1088
1089
1090
1091
1092

1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110
1111
....
1114
1115
1116
1117
1118
1119
1120

1121
1122
1123
1124
1125

1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
....
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167



























1168
1169
1170
1171
1172
1173
1174
....
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
**    May you do good and not evil.
**    May you find forgiveness for yourself and forgive others.
**    May you share freely, never taking more than you give.
**
*************************************************************************
** This file implements that page cache.
**
** @(#) $Id: pcache.c,v 1.14 2008/08/26 18:05:48 danielk1977 Exp $
*/
#include "sqliteInt.h"

/*
** A complete page cache is an instance of this structure.
*/
struct PCache {

  int szPage;                         /* Size of every page in this cache */
  int szExtra;                        /* Size of extra space for each page */
  int nHash;                          /* Number of slots in apHash[] */
  int nPage;                          /* Total number of pages in apHash */
  int nMax;                           /* Configured cache size */
  int nMin;                           /* Configured minimum cache size */
  PgHdr **apHash;                     /* Hash table for fast lookup by pgno */
  int bPurgeable;                     /* True if pages are on backing store */
  void (*xDestroy)(PgHdr*);           /* Called when refcnt goes 1->0 */
  int (*xStress)(void*,PgHdr*);       /* Call to try make a page clean */
  void *pStress;                      /* Argument to xStress */
  PgHdr *pClean;                      /* List of clean pages in use */
  PgHdr *pDirty, *pDirtyTail;         /* List of dirty pages in LRU order */
  PgHdr *pSynced;                     /* Last synced page in dirty page list */
  int nRef;                           /* Number of pinned pages */

  int nPinned;                        /* Number of pinned and/or dirty pages */

};

/*
** Free slots in the page block allocator
*/
typedef struct PgFreeslot PgFreeslot;
struct PgFreeslot {
................................................................................
** cached pages stored to mxPage. In this case mxPagePurgeable is not 
** used.
**
** If mxPage is zero, then the system tries to limit the number of
** pages held by purgable caches to mxPagePurgeable.
**
** The doubly-linked list that runs between pcache.pLruHead and 
** pcache.pLruTail contains all clean purgable pages in the system 
** with a zero reference count. pcache.pLruTail is the next page to
** be recycled.


*/
static struct PCacheGlobal {
  int isInit;                         /* True when initialized */

  sqlite3_mutex *mutex;               /* static mutex MUTEX_STATIC_LRU */

  int nMaxPage;                       /* Sum of nMaxPage for purgeable caches */
  int nMinPage;                       /* Sum of nMinPage for purgeable caches */
  int nCurrentPage;                   /* Number of purgeable pages allocated */


  PgHdr *pLruHead, *pLruTail;         /* LRU list of unused clean pgs */


  /* Variables related to SQLITE_CONFIG_PAGECACHE settings. */
  int szSlot;                         /* Size of each free slot */
  void *pStart, *pEnd;                /* Bounds of pagecache malloc range */
  PgFreeslot *pFree;                  /* Free page blocks */
} pcache = {0};

/*
** All global variables used by this module (most of which are grouped 
** together in global structure "pcache" above) are protected by the static 

** SQLITE_MUTEX_STATIC_LRU mutex. A pointer to this mutex is stored in
** variable "pcache.mutex".
**
** Access to the contents of the individual PCache structures is not 
** protected. It is the job of the caller to ensure that these structures
** are accessed in a thread-safe manner.













*/

#define pcacheEnterGlobal() sqlite3_mutex_enter(pcache.mutex)
#define pcacheExitGlobal()  sqlite3_mutex_leave(pcache.mutex)

/********************************** Linked List Management ********************/

#ifndef NDEBUG
/*
** This routine verifies that the number of entries in the hash table
** is pCache->nPage.  This routine is used within assert() statements
................................................................................
    for(p=pCache->apHash[i]; p; p=p->pNextHash){
      nPage++;
    }
  }
  assert( nPage==pCache->nPage );
  return 1;
}

/*
** Based on the current value of PCache.nRef and the contents of the
** PCache.pDirty list, return the expected value of the PCache.nPinned
** counter. This is only used in debugging builds, as follows:
**
**   assert( pCache->nPinned==pcachePinnedCount(pCache) );
*/
static int pcachePinnedCount(PCache *pCache){
  PgHdr *p;
  int nPinned = pCache->nRef;
  for(p=pCache->pDirty; p; p=p->pNext){
    if( p->nRef==0 ){
      nPinned++;
    }
  }
  return nPinned;
}

/*
** Check that the pCache->pSynced variable is set correctly. If it
** is not, either fail an assert or return zero. Otherwise, return
** non-zero. This is only used in debugging builds, as follows:
**
**   assert( pcacheCheckSynced(pCache) );
*/
static int pcacheCheckSynced(PCache *pCache){
  PgHdr *p = pCache->pDirtyTail;
  for(p=pCache->pDirtyTail; p!=pCache->pSynced; p=p->pPrev){
    assert( p->nRef || (p->flags&PGHDR_NEED_SYNC) );
  }
  return (p==0 || p->nRef || (p->flags&PGHDR_NEED_SYNC)==0);
}

#endif

/*
** Remove a page from its hash table (PCache.apHash[]).
*/
static void pcacheRemoveFromHash(PgHdr *pPage){
  if( pPage->pPrevHash ){
................................................................................
}

/*
** Remove a page from a linked list that is headed by *ppHead.
** *ppHead is either PCache.pClean or PCache.pDirty.
*/
static void pcacheRemoveFromList(PgHdr **ppHead, PgHdr *pPage){
  int isDirtyList = (ppHead==&pPage->pCache->pDirty);
  assert( ppHead==&pPage->pCache->pClean || ppHead==&pPage->pCache->pDirty );

  if( pPage->pPrev ){
    pPage->pPrev->pNext = pPage->pNext;
  }else{
    assert( *ppHead==pPage );
    *ppHead = pPage->pNext;
  }
  if( pPage->pNext ){
    pPage->pNext->pPrev = pPage->pPrev;
  }

  if( isDirtyList ){
    PCache *pCache = pPage->pCache;
    assert( pPage->pNext || pCache->pDirtyTail==pPage );
    if( !pPage->pNext ){
      pCache->pDirtyTail = pPage->pPrev;
    }
    if( pCache->pSynced==pPage ){
      PgHdr *pSynced = pPage->pPrev;
      while( pSynced && (pSynced->flags&PGHDR_NEED_SYNC) ){
        pSynced = pSynced->pPrev;
      }
      pCache->pSynced = pSynced;
    }
  }
}

/*
** Add a page from a linked list that is headed by *ppHead.
** *ppHead is either PCache.pClean or PCache.pDirty.
*/
static void pcacheAddToList(PgHdr **ppHead, PgHdr *pPage){
  int isDirtyList = (ppHead==&pPage->pCache->pDirty);
  assert( ppHead==&pPage->pCache->pClean || ppHead==&pPage->pCache->pDirty );

  if( (*ppHead) ){
    (*ppHead)->pPrev = pPage;
  }
  pPage->pNext = *ppHead;
  pPage->pPrev = 0;
  *ppHead = pPage;

  if( isDirtyList ){
    PCache *pCache = pPage->pCache;
    if( !pCache->pDirtyTail ){
      assert( pPage->pNext==0 );
      pCache->pDirtyTail = pPage;
    }
    if( !pCache->pSynced && 0==(pPage->flags&PGHDR_NEED_SYNC) ){
      pCache->pSynced = pPage;
    }
  }
}

/*
** Remove a page from the global LRU list
*/
static void pcacheRemoveFromLruList(PgHdr *pPage){
  assert( sqlite3_mutex_held(pcache.mutex) );
  assert( (pPage->flags&PGHDR_DIRTY)==0 );
  if( pPage->pCache->bPurgeable==0 ) return;





  if( pPage->pNextLru ){
    assert( pcache.pLruTail!=pPage );
    pPage->pNextLru->pPrevLru = pPage->pPrevLru;
  }else{
    assert( pcache.pLruTail==pPage );
    pcache.pLruTail = pPage->pPrevLru;
  }
  if( pPage->pPrevLru ){
    assert( pcache.pLruHead!=pPage );
    pPage->pPrevLru->pNextLru = pPage->pNextLru;
  }else{
    assert( pcache.pLruHead==pPage );
    pcache.pLruHead = pPage->pNextLru;
  }
}

................................................................................
/*
** Add a page to the global LRU list.  The page is normally added
** to the front of the list so that it will be the last page recycled.
** However, if the PGHDR_REUSE_UNLIKELY bit is set, the page is added
** to the end of the LRU list so that it will be the next to be recycled.
*/
static void pcacheAddToLruList(PgHdr *pPage){
  assert( sqlite3_mutex_held(pcache.mutex) );
  assert( (pPage->flags&PGHDR_DIRTY)==0 );
  if( pPage->pCache->bPurgeable==0 ) return;
  if( pcache.pLruTail && (pPage->flags & PGHDR_REUSE_UNLIKELY)!=0 ){
    /* If reuse is unlikely.  Put the page at the end of the LRU list
    ** where it will be recycled sooner rather than later. 
    */
    assert( pcache.pLruHead );
    pPage->pNextLru = 0;
    pPage->pPrevLru = pcache.pLruTail;
    pcache.pLruTail->pNextLru = pPage;
    pcache.pLruTail = pPage;
    pPage->flags &= ~PGHDR_REUSE_UNLIKELY;



  }else{
    /* If reuse is possible. the page goes at the beginning of the LRU
    ** list so that it will be the last to be recycled.
    */
    if( pcache.pLruHead ){
      pcache.pLruHead->pPrevLru = pPage;
    }
    pPage->pNextLru = pcache.pLruHead;
    pcache.pLruHead = pPage;
    pPage->pPrevLru = 0;
    if( pcache.pLruTail==0 ){
      pcache.pLruTail = pPage;
    }



  }
}

/*********************************************** Memory Allocation ***********
**
** Initialize the page cache memory pool.
**
................................................................................

/*
** Allocate a page cache line.  Look in the page cache memory pool first
** and use an element from it first if available.  If nothing is available
** in the page cache memory pool, go to the general purpose memory allocator.
*/
void *pcacheMalloc(int sz, PCache *pCache){
  assert( sqlite3_mutex_held(pcache.mutex) );
  if( sz<=pcache.szSlot && pcache.pFree ){
    PgFreeslot *p = pcache.pFree;
    pcache.pFree = p->pNext;
    sqlite3StatusSet(SQLITE_STATUS_PAGECACHE_SIZE, sz);
    sqlite3StatusAdd(SQLITE_STATUS_PAGECACHE_USED, 1);
    return (void*)p;
  }else{
................................................................................
    ** so that if the attempt to allocate a new buffer causes the the 
    ** configured soft-heap-limit to be breached, it will be possible to
    ** reclaim memory from this pager-cache. Because sqlite3PcacheLock() 
    ** might block on the MEM2 mutex, it has to be called before re-entering
    ** the global LRU mutex.
    */
    pcacheExitGlobal();

    p = sqlite3Malloc(sz);

    pcacheEnterGlobal();

    if( p ){
      sz = sqlite3MallocSize(p);
      sqlite3StatusAdd(SQLITE_STATUS_PAGECACHE_OVERFLOW, sz);
    }
    return p;
................................................................................
  return p;
}

/*
** Release a pager memory allocation
*/
void pcacheFree(void *p){
  assert( sqlite3_mutex_held(pcache.mutex) );
  if( p==0 ) return;
  if( p>=pcache.pStart && p<pcache.pEnd ){
    PgFreeslot *pSlot;
    sqlite3StatusAdd(SQLITE_STATUS_PAGECACHE_USED, -1);
    pSlot = (PgFreeslot*)p;
    pSlot->pNext = pcache.pFree;
    pcache.pFree = pSlot;
................................................................................

/*
** Allocate a new page.
*/
static PgHdr *pcachePageAlloc(PCache *pCache){
  PgHdr *p;
  int sz = sizeof(*p) + pCache->szPage + pCache->szExtra;
  assert( sqlite3_mutex_held(pcache.mutex) );
  p = pcacheMalloc(sz, pCache);
  if( p==0 ) return 0;
  memset(p, 0, sizeof(PgHdr));
  p->pData = (void*)&p[1];
  p->pExtra = (void*)&((char*)p->pData)[pCache->szPage];

  if( pCache->bPurgeable ){
    pcache.nCurrentPage++;
  }

  return p;
}

/*
** Deallocate a page
*/
static void pcachePageFree(PgHdr *p){
  assert( sqlite3_mutex_held(pcache.mutex) );

  if( p->pCache->bPurgeable ){
    pcache.nCurrentPage--;
  }
  pcacheFree(p->apSave[0]);
  pcacheFree(p->apSave[1]);
  pcacheFree(p);
}

#ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT
/*
** Return the number of bytes that will be returned to the heap when
** the argument is passed to pcachePageFree().
*/
static int pcachePageSize(PgHdr *p){
  assert( sqlite3_mutex_held(pcache.mutex) );
  assert( !pcache.pStart );
  assert( p->apSave[0]==0 );
  assert( p->apSave[1]==0 );
  assert( p && p->pCache );
  return sqlite3MallocSize(p);
}
#endif


















/*
** Attempt to 'recycle' a page from the global LRU list. Only clean,





** unreferenced pages from purgeable caches are eligible for recycling.
**
** This function removes page pcache.pLruTail from the global LRU list,
** and from the hash-table and PCache.pClean list of the owner pcache.
** There should be no other references to the page.
**
** A pointer to the recycled page is returned, or NULL if no page is
** eligible for recycling.
*/
static PgHdr *pcacheRecyclePage(){
  PgHdr *p = 0;


  assert( sqlite3_mutex_held(pcache.mutex) );







  if( (p=pcache.pLruTail) ){
    assert( (p->flags&PGHDR_DIRTY)==0 );







    pcacheRemoveFromLruList(p);
    pcacheRemoveFromHash(p);
    pcacheRemoveFromList(&p->pCache->pClean, p);

    /* If the always-rollback flag is set on the page being recycled, set 
    ** the always-rollback flag on the corresponding pager. TODO: This is
    ** a thread-safety problem.
    */
    if( p->flags&PGHDR_ALWAYS_ROLLBACK ){
      assert(p->pPager);
      sqlite3PagerAlwaysRollback(p->pPager);
    }
  }

................................................................................
** Return a pointer to the new page, or NULL if an OOM condition occurs.
*/
static PgHdr *pcacheRecycleOrAlloc(PCache *pCache){
  PgHdr *p = 0;

  int szPage = pCache->szPage;
  int szExtra = pCache->szExtra;


  assert( pcache.isInit );
  assert( sqlite3_mutex_notheld(pcache.mutex) );

  pcacheEnterGlobal();

  /* If we have reached the limit for pinned/dirty pages, and there is at
  ** least one dirty page, invoke the xStress callback to cause a page to
  ** become clean.
  */
  assert( pCache->nPinned==pcachePinnedCount(pCache) );
  assert( pcacheCheckSynced(pCache) );
  if( pCache->xStress
   && pCache->pDirty
   && pCache->nPinned>=(pcache.nMaxPage+pCache->nMin-pcache.nMinPage)
  ){


    PgHdr *pPg;
    assert(pCache->pDirtyTail);

    for(pPg=pCache->pSynced; 
        pPg && (pPg->nRef || (pPg->flags&PGHDR_NEED_SYNC)); 
        pPg=pPg->pPrev
    );
    if( !pPg ){
      for(pPg=pCache->pDirtyTail; pPg && pPg->nRef; pPg=pPg->pPrev);
    }
    if( pPg ){
      pcacheExitGlobal();
      pCache->xStress(pCache->pStress, pPg);
      pcacheEnterGlobal();
    }
  }

  /* If the global page limit has been reached, try to recycle a page. */
  if( pcache.nCurrentPage>=pcache.nMaxPage ){
    p = pcacheRecyclePage();
  }

  /* If a page has been recycled but it is the wrong size, free it. */
  if( p && (p->pCache->szPage!=szPage || p->pCache->szExtra!=szExtra) ){
    pcachePageFree(p);
    p = 0;
  }

  if( !p ){
    p = pcachePageAlloc(pCache);
................................................................................
** Initialize and shutdown the page cache subsystem. Neither of these 
** functions are threadsafe.
*/
int sqlite3PcacheInitialize(void){
  assert( pcache.isInit==0 );
  memset(&pcache, 0, sizeof(pcache));
  if( sqlite3Config.bCoreMutex ){
    pcache.mutex = sqlite3_mutex_alloc(SQLITE_MUTEX_STATIC_LRU);

    if( pcache.mutex==0 ){
      return SQLITE_NOMEM;
    }
  }
  pcache.isInit = 1;
  return SQLITE_OK;
}
void sqlite3PcacheShutdown(void){
................................................................................
  p->szPage = szPage;
  p->szExtra = szExtra;
  p->bPurgeable = bPurgeable;
  p->xDestroy = xDestroy;
  p->xStress = xStress;
  p->pStress = pStress;
  p->nMax = 100;
  p->nMin = 20;


  pcacheEnterGlobal();
  if( bPurgeable ){
    pcache.nMaxPage += p->nMax;
    pcache.nMinPage += p->nMin;
  }









  pcacheExitGlobal();
}

/*
** Change the page size for PCache object.  This can only happen
** when the cache is empty.
*/
................................................................................
  int createFlag,       /* If true, create page if it does not exist already */
  PgHdr **ppPage        /* Write the page here */
){
  PgHdr *pPage;
  assert( pcache.isInit );
  assert( pCache!=0 );
  assert( pgno>0 );
  assert( pCache->nPinned==pcachePinnedCount(pCache) );

  /* Search the hash table for the requested page. Exit early if it is found. */
  if( pCache->apHash ){
    u32 h = pgno % pCache->nHash;
    for(pPage=pCache->apHash[h]; pPage; pPage=pPage->pNextHash){
      if( pPage->pgno==pgno ){
        if( pPage->nRef==0 ){
          if( 0==(pPage->flags&PGHDR_DIRTY) ){
            pcacheEnterGlobal();
            pcacheRemoveFromLruList(pPage);
            pcacheExitGlobal();
            pCache->nPinned++;
          }

          pCache->nRef++;
        }
        pPage->nRef++;
        *ppPage = pPage;
        return SQLITE_OK;
      }
    }
  }

  if( createFlag ){
................................................................................
    pPage->pPager = 0;
    pPage->flags = 0;
    pPage->pDirty = 0;
    pPage->pgno = pgno;
    pPage->pCache = pCache;
    pPage->nRef = 1;
    pCache->nRef++;
    pCache->nPinned++;
    pcacheAddToList(&pCache->pClean, pPage);
    pcacheAddToHash(pPage);
  }else{
    *ppPage = 0;
  }

  assert( pCache->nPinned==pcachePinnedCount(pCache) );
  return SQLITE_OK;
}

/*
** Dereference a page.  When the reference count reaches zero,
** move the page to the LRU list if it is clean.
*/
void sqlite3PcacheRelease(PgHdr *p){
  assert( p->nRef>0 );

  p->nRef--;
  if( p->nRef==0 ){
    PCache *pCache = p->pCache;

    if( p->pCache->xDestroy ){
      p->pCache->xDestroy(p);
    }
    pCache->nRef--;
    if( (p->flags&PGHDR_DIRTY)==0 ){
      pCache->nPinned--;
      pcacheEnterGlobal();
      pcacheAddToLruList(p);
      pcacheExitGlobal();
    }else{
      /* Move the page to the head of the caches dirty list. */
      pcacheRemoveFromList(&pCache->pDirty, p);
      pcacheAddToList(&pCache->pDirty, p);
    }
  }
}

void sqlite3PcacheRef(PgHdr *p){
  assert(p->nRef>0);
  p->nRef++;
}
................................................................................

/*
** Drop a page from the cache.  This should be the only reference to
** the page.
*/
void sqlite3PcacheDrop(PgHdr *p){
  PCache *pCache;

  assert( p->nRef==1 );
  pCache = p->pCache;
  pCache->nRef--;
  pCache->nPinned--;
  if( p->flags & PGHDR_DIRTY ){
    pcacheRemoveFromList(&pCache->pDirty, p);
  }else{
    pcacheRemoveFromList(&pCache->pClean, p);
  }
  pcacheRemoveFromHash(p);
  pcacheEnterGlobal();
................................................................................

/*
** Make sure the page is marked as dirty.  If it isn't dirty already,
** make it so.
*/
void sqlite3PcacheMakeDirty(PgHdr *p){
  PCache *pCache;

  p->flags &= ~PGHDR_DONT_WRITE;
  if( p->flags & PGHDR_DIRTY ) return;
  assert( (p->flags & PGHDR_DIRTY)==0 );
  assert( p->nRef>0 );
  pCache = p->pCache;
  pcacheRemoveFromList(&pCache->pClean, p);
  pcacheAddToList(&pCache->pDirty, p);
................................................................................

/*
** Make sure the page is marked as clean.  If it isn't clean already,
** make it so.
*/
void sqlite3PcacheMakeClean(PgHdr *p){
  PCache *pCache;

  if( (p->flags & PGHDR_DIRTY)==0 ) return;
  assert( p->apSave[0]==0 && p->apSave[1]==0 );
  assert( p->flags & PGHDR_DIRTY );
  pCache = p->pCache;
  pcacheRemoveFromList(&pCache->pDirty, p);
  pcacheEnterGlobal();
  pcacheAddToList(&pCache->pClean, p);
  p->flags &= ~PGHDR_DIRTY;
  if( p->nRef==0 ){
    pcacheAddToLruList(p);
    pCache->nPinned--;
  }
  assert( pCache->nPinned==pcachePinnedCount(pCache) );
  pcacheExitGlobal();
}

/*
** Make every page in the cache clean.
*/
void sqlite3PcacheCleanAll(PCache *pCache){
  PgHdr *p;

  pcacheEnterGlobal();
  while( (p = pCache->pDirty)!=0 ){
    assert( p->apSave[0]==0 && p->apSave[1]==0 );
    pcacheRemoveFromList(&pCache->pDirty, p);

    p->flags &= ~PGHDR_DIRTY;
    pcacheAddToList(&pCache->pClean, p);
    if( p->nRef==0 ){
      pcacheAddToLruList(p);
      pCache->nPinned--;
    }
  }
  sqlite3PcacheAssertFlags(pCache, 0, PGHDR_DIRTY);
  assert( pCache->nPinned==pcachePinnedCount(pCache) );
  pcacheExitGlobal();
}

/*
** Change the page number of page p to newPgno. If newPgno is 0, then the
** page object is added to the clean-list and the PGHDR_REUSE_UNLIKELY 
** flag set.
*/
void sqlite3PcacheMove(PgHdr *p, Pgno newPgno){
  assert( p->nRef>0 );
  pcacheRemoveFromHash(p);
  p->pgno = newPgno;
  if( newPgno==0 ){
    p->flags |= PGHDR_REUSE_UNLIKELY;
    pcacheEnterGlobal();
    pcacheFree(p->apSave[0]);
    pcacheFree(p->apSave[1]);
................................................................................
    p->apSave[0] = 0;
    p->apSave[1] = 0;
    sqlite3PcacheMakeClean(p);
  }
  pcacheAddToHash(p);
}










/*
** Remove all content from a page cache
*/
void pcacheClear(PCache *pCache){
  PgHdr *p, *pNext;
  assert( sqlite3_mutex_held(pcache.mutex) );
  for(p=pCache->pClean; p; p=pNext){
    pNext = p->pNext;
    pcacheRemoveFromLruList(p);
    pcachePageFree(p);
  }
  for(p=pCache->pDirty; p; p=pNext){
    pNext = p->pNext;

    pcachePageFree(p);
  }
  pCache->pClean = 0;
  pCache->pDirty = 0;
  pCache->pDirtyTail = 0;
  pCache->nPage = 0;
  pCache->nPinned = 0;
  memset(pCache->apHash, 0, pCache->nHash*sizeof(pCache->apHash[0]));
}


/*
** Drop every cache entry whose page number is greater than "pgno".
*/
void sqlite3PcacheTruncate(PCache *pCache, Pgno pgno){
  PgHdr *p, *pNext;
  PgHdr *pDirty = pCache->pDirty;

  pcacheEnterGlobal();
  for(p=pCache->pClean; p||pDirty; p=pNext){
    if( !p ){
      p = pDirty;
      pDirty = 0;
    }
    pNext = p->pNext;
    if( p->pgno>pgno ){
      if( p->nRef==0 ){
        pcacheRemoveFromHash(p);
        if( p->flags&PGHDR_DIRTY ){
          pcacheRemoveFromList(&pCache->pDirty, p);
          pCache->nPinned--;
        }else{
          pcacheRemoveFromList(&pCache->pClean, p);

          pcacheRemoveFromLruList(p);
        }
        pcachePageFree(p);
      }else{
        /* If there are references to the page, it cannot be freed. In this
        ** case, zero the page content instead.
        */
        memset(p->pData, 0, pCache->szPage);
      }
................................................................................
}


/*
** Close a cache.
*/
void sqlite3PcacheClose(PCache *pCache){

  pcacheEnterGlobal();

  /* Free all the pages used by this pager and remove them from the LRU list. */
  pcacheClear(pCache);
  if( pCache->bPurgeable ){
    pcache.nMaxPage -= pCache->nMax;
    pcache.nMinPage -= pCache->nMin;
  }
  sqlite3_free(pCache->apHash);
















  pcacheExitGlobal();
}

/*
** Preserve the content of the page, if it has not been preserved
** already.  If idJournal==0 then this is for the overall transaction.
** If idJournal==1 then this is for the statement journal.
................................................................................
** This routine is used for in-memory databases only.
**
** Return SQLITE_OK or SQLITE_NOMEM if a memory allocation fails.
*/
int sqlite3PcachePreserve(PgHdr *p, int idJournal){
  void *x;
  int sz;

  assert( p->pCache->bPurgeable==0 );
  if( !p->apSave[idJournal] ){
    sz = p->pCache->szPage;
    p->apSave[idJournal] = x = sqlite3PageMalloc( sz );
    if( x==0 ) return SQLITE_NOMEM;
    memcpy(x, p->pData, sz);
  }
................................................................................
}

/*
** Commit a change previously preserved.
*/
void sqlite3PcacheCommit(PCache *pCache, int idJournal){
  PgHdr *p;

  pcacheEnterGlobal();     /* Mutex is required to call pcacheFree() */
  for(p=pCache->pDirty; p; p=p->pNext){
    if( p->apSave[idJournal] ){
      pcacheFree(p->apSave[idJournal]);
      p->apSave[idJournal] = 0;
    }
  }
................................................................................

/*
** Rollback a change previously preserved.
*/
void sqlite3PcacheRollback(PCache *pCache, int idJournal){
  PgHdr *p;
  int sz;

  pcacheEnterGlobal();     /* Mutex is required to call pcacheFree() */
  sz = pCache->szPage;
  for(p=pCache->pDirty; p; p=p->pNext){
    if( p->apSave[idJournal] ){
      memcpy(p->pData, p->apSave[idJournal], sz);
      pcacheFree(p->apSave[idJournal]);
      p->apSave[idJournal] = 0;
................................................................................
}

/* 
** Assert flags settings on all pages.  Debugging only.
*/
void sqlite3PcacheAssertFlags(PCache *pCache, int trueMask, int falseMask){
  PgHdr *p;

  for(p=pCache->pDirty; p; p=p->pNext){
    assert( (p->flags&trueMask)==trueMask );
    assert( (p->flags&falseMask)==0 );
  }
  for(p=pCache->pClean; p; p=p->pNext){
    assert( (p->flags&trueMask)==trueMask );
    assert( (p->flags&falseMask)==0 );
................................................................................
  }
}

/* 
** Discard the contents of the cache.
*/
int sqlite3PcacheClear(PCache *pCache){

  assert(pCache->nRef==0);
  pcacheEnterGlobal();
  pcacheClear(pCache);
  pcacheExitGlobal();
  return SQLITE_OK;
}

................................................................................
}

/*
** Return a list of all dirty pages in the cache, sorted by page number.
*/
PgHdr *sqlite3PcacheDirtyList(PCache *pCache){
  PgHdr *p;

  for(p=pCache->pDirty; p; p=p->pNext){
    p->pDirty = p->pNext;
  }
  return pcacheSortDirtyList(pCache->pDirty);
}

/* 
................................................................................
  return pCache->nRef;
}

/* 
** Return the total number of pages in the cache.
*/
int sqlite3PcachePagecount(PCache *pCache){

  assert( pCache->nPage>=0 );
  return pCache->nPage;
}

#ifdef SQLITE_CHECK_PAGES
/*
** This function is used by the pager.c module to iterate through all 
** pages in the cache. At present, this is only required if the
** SQLITE_CHECK_PAGES macro (used for debugging) is specified.
*/
void sqlite3PcacheIterate(PCache *pCache, void (*xIter)(PgHdr *)){
  PgHdr *p;

  for(p=pCache->pClean; p; p=p->pNext){
    xIter(p);
  }
  for(p=pCache->pDirty; p; p=p->pNext){
    xIter(p);
  }
}
................................................................................
/* 
** Set flags on all pages in the page cache 
*/
void sqlite3PcacheSetFlags(PCache *pCache, int andMask, int orMask){
  PgHdr *p;

  assert( (orMask&PGHDR_NEED_SYNC)==0 );


  /* Obtain the global mutex before modifying any PgHdr.flags variables 
  ** or traversing the LRU list.
  */ 
  pcacheEnterGlobal();


  for(p=pCache->pDirty; p; p=p->pNext){
    p->flags = (p->flags&andMask)|orMask;
  }
  for(p=pCache->pClean; p; p=p->pNext){
    p->flags = (p->flags&andMask)|orMask;
  }

  if( 0==(andMask&PGHDR_NEED_SYNC) ){
    PgHdr *pSynced = pCache->pDirtyTail;
    while( pSynced && (pSynced->flags&PGHDR_NEED_SYNC) ){
      pSynced = pSynced->pPrev;
    }
    pCache->pSynced = pSynced;
  }

  pcacheExitGlobal();
}

/*
** Set the suggested cache-size value.
................................................................................
*/
void sqlite3PcacheSetCachesize(PCache *pCache, int mxPage){
  if( mxPage<10 ){
    mxPage = 10;
  }
  if( pCache->bPurgeable ){
    pcacheEnterGlobal();
    pcache.nMaxPage -= pCache->nMax;
    pcache.nMaxPage += mxPage;
    pcacheExitGlobal();
  }
  pCache->nMax = mxPage;
}




























#ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT
/*
** This function is called to free superfluous dynamically allocated memory
** held by the pager system. Memory in use by any SQLite pager allocated
** by the current thread may be sqlite3_free()ed.
**
** nReq is the number of bytes of memory required. Once this much has
................................................................................
** of bytes of memory released.
*/
int sqlite3PcacheReleaseMemory(int nReq){
  int nFree = 0;
  if( pcache.pStart==0 ){
    PgHdr *p;
    pcacheEnterGlobal();
    while( (nReq<0 || nFree<nReq) && (p=pcacheRecyclePage()) ){
      nFree += pcachePageSize(p);
      pcachePageFree(p);
    }
    pcacheExitGlobal();
  }
  return nFree;
}
#endif /* SQLITE_ENABLE_MEMORY_MANAGEMENT */

Changes to src/pcache.h.

8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
..
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
...
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
**    May you find forgiveness for yourself and forgive others.
**    May you share freely, never taking more than you give.
**
*************************************************************************
** This header file defines the interface that the sqlite page cache
** subsystem. 
**
** @(#) $Id: pcache.h,v 1.5 2008/08/23 18:53:08 danielk1977 Exp $
*/

#ifndef _PCACHE_H_

typedef struct PgHdr PgHdr;
typedef struct PCache PCache;

................................................................................
void sqlite3PcacheMakeDirty(PgHdr*);    /* Make sure page is marked dirty */
void sqlite3PcacheMakeClean(PgHdr*);    /* Mark a single page as clean */
void sqlite3PcacheCleanAll(PCache*);    /* Mark all dirty list pages as clean */

/* Change a page number.  Used by incr-vacuum. */
void sqlite3PcacheMove(PgHdr*, Pgno);

/* Set a global maximum page count for all page caches. 
** If the sum of individual cache maxes exceed the global max, the
** individuals are scaled down proportionally. 
*/
void sqlite3PcacheGlobalMax(int N);

/* Remove all pages with pgno>x.  Reset the cache if x==0 */
void sqlite3PcacheTruncate(PCache*, Pgno x);

/* Routines used to implement transactions on memory-only databases. */
int sqlite3PcachePreserve(PgHdr*, int);    /* Preserve current page content */
void sqlite3PcacheCommit(PCache*, int);    /* Drop preserved copy */
void sqlite3PcacheRollback(PCache*, int);  /* Rollback to preserved copy */
................................................................................

/* Iterate through all pages currently stored in the cache. This interface
** is only available if SQLITE_CHECK_PAGES is defined when the library is 
** built.
*/
void sqlite3PcacheIterate(PCache *pCache, void (*xIter)(PgHdr *));

/* Set and get the suggested cache-size for the specified pager-cache. If
** a global maximum on the number of pages cached by the system is 
** configured via the sqlite3PcacheGlobalMax() API, then the suggested
** cache-sizes are not used at all.
**
** If no global maximum is configured, then the system attempts to limit
** the total number of pages cached by purgeable pager-caches to the sum
** of the suggested cache-sizes.
*/
int sqlite3PcacheGetCachesize(PCache *);
void sqlite3PcacheSetCachesize(PCache *, int);

/* Lock and unlock a pager-cache object. The PcacheLock() function may 
** block if the lock is temporarily available. While a pager-cache is locked,
** the system guarantees that any configured xStress() callback will not
** be invoked by any thread other than the one holding the lock.
*/
void sqlite3PcacheLock(PCache *);
void sqlite3PcacheUnlock(PCache *);

int sqlite3PcacheReleaseMemory(int);

#endif /* _PCACHE_H_ */







|







 







<
<
<
<
<
<







 







|
<
<
<








|
<
<
<
<
<
<
<



8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
..
97
98
99
100
101
102
103






104
105
106
107
108
109
110
...
138
139
140
141
142
143
144
145



146
147
148
149
150
151
152
153
154







155
156
157
**    May you find forgiveness for yourself and forgive others.
**    May you share freely, never taking more than you give.
**
*************************************************************************
** This header file defines the interface that the sqlite page cache
** subsystem. 
**
** @(#) $Id: pcache.h,v 1.6 2008/08/26 18:05:48 danielk1977 Exp $
*/

#ifndef _PCACHE_H_

typedef struct PgHdr PgHdr;
typedef struct PCache PCache;

................................................................................
void sqlite3PcacheMakeDirty(PgHdr*);    /* Make sure page is marked dirty */
void sqlite3PcacheMakeClean(PgHdr*);    /* Mark a single page as clean */
void sqlite3PcacheCleanAll(PCache*);    /* Mark all dirty list pages as clean */

/* Change a page number.  Used by incr-vacuum. */
void sqlite3PcacheMove(PgHdr*, Pgno);







/* Remove all pages with pgno>x.  Reset the cache if x==0 */
void sqlite3PcacheTruncate(PCache*, Pgno x);

/* Routines used to implement transactions on memory-only databases. */
int sqlite3PcachePreserve(PgHdr*, int);    /* Preserve current page content */
void sqlite3PcacheCommit(PCache*, int);    /* Drop preserved copy */
void sqlite3PcacheRollback(PCache*, int);  /* Rollback to preserved copy */
................................................................................

/* Iterate through all pages currently stored in the cache. This interface
** is only available if SQLITE_CHECK_PAGES is defined when the library is 
** built.
*/
void sqlite3PcacheIterate(PCache *pCache, void (*xIter)(PgHdr *));

/* Set and get the suggested cache-size for the specified pager-cache.



**
** If no global maximum is configured, then the system attempts to limit
** the total number of pages cached by purgeable pager-caches to the sum
** of the suggested cache-sizes.
*/
int sqlite3PcacheGetCachesize(PCache *);
void sqlite3PcacheSetCachesize(PCache *, int);

/* Try to return memory used by the pcache module to the main memory heap */







int sqlite3PcacheReleaseMemory(int);

#endif /* _PCACHE_H_ */

Changes to src/test2.c.

9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
...
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
...
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
**    May you share freely, never taking more than you give.
**
*************************************************************************
** Code for testing the pager.c module in SQLite.  This code
** is not included in the SQLite library.  It is used for automated
** testing of the SQLite library.
**
** $Id: test2.c,v 1.60 2008/08/20 14:49:25 danielk1977 Exp $
*/
#include "sqliteInt.h"
#include "tcl.h"
#include <stdlib.h>
#include <string.h>
#include <ctype.h>

................................................................................
  if( pPage ){
    sqlite3_snprintf(sizeof(zBuf),zBuf,"%p",pPage);
    Tcl_AppendResult(interp, zBuf, 0);
  }
  return TCL_OK;
}

/*
** Usage:   pcache_global_max NPAGE
*/
static int pcache_global_max(
  void *NotUsed,
  Tcl_Interp *interp,    /* The TCL interpreter that invoked this command */
  int argc,              /* Number of arguments */
  const char **argv      /* Text of each argument */
){
  int nPage;
  if( argc!=2 ){
    Tcl_AppendResult(interp, "wrong # args: should be \"", argv[0],
       " NPAGE\"", 0);
    return TCL_ERROR;
  }
  if( Tcl_GetInt(interp, argv[1], &nPage) ) return TCL_ERROR;
  sqlite3PcacheGlobalMax(nPage);
  return TCL_OK;
}

/*
** Usage:   pager_truncate ID PGNO
*/
static int pager_truncate(
  void *NotUsed,
  Tcl_Interp *interp,    /* The TCL interpreter that invoked this command */
  int argc,              /* Number of arguments */
................................................................................
    { "page_get",                (Tcl_CmdProc*)page_get            },
    { "page_lookup",             (Tcl_CmdProc*)page_lookup         },
    { "page_unref",              (Tcl_CmdProc*)page_unref          },
    { "page_read",               (Tcl_CmdProc*)page_read           },
    { "page_write",              (Tcl_CmdProc*)page_write          },
    { "page_number",             (Tcl_CmdProc*)page_number         },
    { "pager_truncate",          (Tcl_CmdProc*)pager_truncate      },
    { "pcache_global_max",       (Tcl_CmdProc*)pcache_global_max   },
#ifndef SQLITE_OMIT_DISKIO
    { "fake_big_file",           (Tcl_CmdProc*)fake_big_file       },
#endif
    { "sqlite3BitvecBuiltinTest",(Tcl_CmdProc*)testBitvecBuiltinTest},
  };
  int i;
  for(i=0; i<sizeof(aCmd)/sizeof(aCmd[0]); i++){







|







 







<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<







 







<







9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
...
378
379
380
381
382
383
384




















385
386
387
388
389
390
391
...
626
627
628
629
630
631
632

633
634
635
636
637
638
639
**    May you share freely, never taking more than you give.
**
*************************************************************************
** Code for testing the pager.c module in SQLite.  This code
** is not included in the SQLite library.  It is used for automated
** testing of the SQLite library.
**
** $Id: test2.c,v 1.61 2008/08/26 18:05:48 danielk1977 Exp $
*/
#include "sqliteInt.h"
#include "tcl.h"
#include <stdlib.h>
#include <string.h>
#include <ctype.h>

................................................................................
  if( pPage ){
    sqlite3_snprintf(sizeof(zBuf),zBuf,"%p",pPage);
    Tcl_AppendResult(interp, zBuf, 0);
  }
  return TCL_OK;
}





















/*
** Usage:   pager_truncate ID PGNO
*/
static int pager_truncate(
  void *NotUsed,
  Tcl_Interp *interp,    /* The TCL interpreter that invoked this command */
  int argc,              /* Number of arguments */
................................................................................
    { "page_get",                (Tcl_CmdProc*)page_get            },
    { "page_lookup",             (Tcl_CmdProc*)page_lookup         },
    { "page_unref",              (Tcl_CmdProc*)page_unref          },
    { "page_read",               (Tcl_CmdProc*)page_read           },
    { "page_write",              (Tcl_CmdProc*)page_write          },
    { "page_number",             (Tcl_CmdProc*)page_number         },
    { "pager_truncate",          (Tcl_CmdProc*)pager_truncate      },

#ifndef SQLITE_OMIT_DISKIO
    { "fake_big_file",           (Tcl_CmdProc*)fake_big_file       },
#endif
    { "sqlite3BitvecBuiltinTest",(Tcl_CmdProc*)testBitvecBuiltinTest},
  };
  int i;
  for(i=0; i<sizeof(aCmd)/sizeof(aCmd[0]); i++){

Changes to test/memsubsys1.test.

7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
...
224
225
226
227
228
229
230
231
232
233
234


235
236
237
238
239
240
241
...
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
#    May you find forgiveness for yourself and forgive others.
#    May you share freely, never taking more than you give.
#
#***********************************************************************
#
# This file contains tests of the memory allocation subsystem
#
# $Id: memsubsys1.test,v 1.10 2008/08/20 14:49:25 danielk1977 Exp $

set testdir [file dirname $argv0]
source $testdir/tester.tcl
sqlite3_reset_auto_extension

# This procedure constructs a new database in test.db.  It fills
# this database with many small records (enough to force multiple
................................................................................
# that maximum allocation size is small.
#
db close
sqlite3_shutdown
sqlite3_config_pagecache [expr 4096+$xtra_size] 24
sqlite3_config_scratch 25000 1
sqlite3_initialize
pcache_global_max 15
reset_highwater_marks
build_test_db memsubsys1-7 {
  PRAGMA page_size=4096;


}
#show_memstats
do_test memsubsys1-7.3 {
  set pg_used [lindex [sqlite3_status SQLITE_STATUS_PAGECACHE_USED 0] 2]
  expr {$pg_used<24}
} 1
do_test memsubsys1-7.4 {
................................................................................
} 1
do_test memsubsys1-7.6 {
  set s_used [lindex [sqlite3_status SQLITE_STATUS_SCRATCH_USED 0] 2]
} 1
do_test memsubsys1-7.7 {
  set s_ovfl [lindex [sqlite3_status SQLITE_STATUS_SCRATCH_OVERFLOW 0] 2]
} 0
pcache_global_max 0

db close
sqlite3_shutdown
sqlite3_config_pagecache 0 0
sqlite3_config_scratch 0 0
sqlite3_config_lookaside 100 500
sqlite3_initialize
autoinstall_test_functions
finish_test







|







 







<



>
>







 







<









7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
...
224
225
226
227
228
229
230

231
232
233
234
235
236
237
238
239
240
241
242
...
248
249
250
251
252
253
254

255
256
257
258
259
260
261
262
263
#    May you find forgiveness for yourself and forgive others.
#    May you share freely, never taking more than you give.
#
#***********************************************************************
#
# This file contains tests of the memory allocation subsystem
#
# $Id: memsubsys1.test,v 1.11 2008/08/26 18:05:48 danielk1977 Exp $

set testdir [file dirname $argv0]
source $testdir/tester.tcl
sqlite3_reset_auto_extension

# This procedure constructs a new database in test.db.  It fills
# this database with many small records (enough to force multiple
................................................................................
# that maximum allocation size is small.
#
db close
sqlite3_shutdown
sqlite3_config_pagecache [expr 4096+$xtra_size] 24
sqlite3_config_scratch 25000 1
sqlite3_initialize

reset_highwater_marks
build_test_db memsubsys1-7 {
  PRAGMA page_size=4096;
  PRAGMA cache_size=10;
  PRAGMA temp_store=memory;
}
#show_memstats
do_test memsubsys1-7.3 {
  set pg_used [lindex [sqlite3_status SQLITE_STATUS_PAGECACHE_USED 0] 2]
  expr {$pg_used<24}
} 1
do_test memsubsys1-7.4 {
................................................................................
} 1
do_test memsubsys1-7.6 {
  set s_used [lindex [sqlite3_status SQLITE_STATUS_SCRATCH_USED 0] 2]
} 1
do_test memsubsys1-7.7 {
  set s_ovfl [lindex [sqlite3_status SQLITE_STATUS_SCRATCH_OVERFLOW 0] 2]
} 0


db close
sqlite3_shutdown
sqlite3_config_pagecache 0 0
sqlite3_config_scratch 0 0
sqlite3_config_lookaside 100 500
sqlite3_initialize
autoinstall_test_functions
finish_test

Changes to test/tkt2409.test.

12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
..
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
...
219
220
221
222
223
224
225
226
227
228
229
230
231
#
# This file implements tests to verify that ticket #2409 has been
# fixed. More specifically, they verify that if SQLite cannot
# obtain an EXCLUSIVE lock while trying to spill the cache during
# any statement other than a COMMIT, an I/O error is returned instead
# of SQLITE_BUSY.
#
# $Id: tkt2409.test,v 1.4 2008/08/20 14:49:25 danielk1977 Exp $

# Test Outline:
#
#   tkt-2409-1.*: Cause a cache-spill during an INSERT that is within
#       a db transaction but does not start a statement transaction.
#       Verify that the transaction is automatically rolled back
#       and SQLITE_IOERR_BLOCKED is returned
................................................................................
proc unread_lock_db {} {
  if {$::STMT ne ""} {
    sqlite3_finalize $::STMT
    set ::STMT ""
  }
}

pcache_global_max 10

# Open the db handle used by [read_lock_db].
#
sqlite3 db2 test.db
set ::STMT ""

do_test tkt2409-1.1 {
  execsql {
................................................................................
#
integrity_check tkt2409-4.3

do_test tkt2409-4.4 {
  catchsql { ROLLBACK }
} {0 {}}

pcache_global_max 0

unread_lock_db
db2 close
unset -nocomplain t1
finish_test







|







 







<
<







 







<
<




12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
..
75
76
77
78
79
80
81


82
83
84
85
86
87
88
...
217
218
219
220
221
222
223


224
225
226
227
#
# This file implements tests to verify that ticket #2409 has been
# fixed. More specifically, they verify that if SQLite cannot
# obtain an EXCLUSIVE lock while trying to spill the cache during
# any statement other than a COMMIT, an I/O error is returned instead
# of SQLITE_BUSY.
#
# $Id: tkt2409.test,v 1.5 2008/08/26 18:05:48 danielk1977 Exp $

# Test Outline:
#
#   tkt-2409-1.*: Cause a cache-spill during an INSERT that is within
#       a db transaction but does not start a statement transaction.
#       Verify that the transaction is automatically rolled back
#       and SQLITE_IOERR_BLOCKED is returned
................................................................................
proc unread_lock_db {} {
  if {$::STMT ne ""} {
    sqlite3_finalize $::STMT
    set ::STMT ""
  }
}



# Open the db handle used by [read_lock_db].
#
sqlite3 db2 test.db
set ::STMT ""

do_test tkt2409-1.1 {
  execsql {
................................................................................
#
integrity_check tkt2409-4.3

do_test tkt2409-4.4 {
  catchsql { ROLLBACK }
} {0 {}}



unread_lock_db
db2 close
unset -nocomplain t1
finish_test