/ Check-in [a518bc33]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Use exponential buffer size growth in StrAccum, as long as the size does not grow to large, to avoid excess memory allocation resize operations. Also, document the fact that setting scratch memory causes SQLite to try to avoid large memory allocations.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: a518bc3318232d652349eb29303ff250aee40459
User & Date: drh 2014-11-03 14:46:29
Context
2014-11-03
16:56
Add the experimental sqlite3_stmt_scanstatus() API. For comparing the number of rows actually visited by a loop with the estimate used by the query planner. check-in: ab3b0fc5 user: dan tags: trunk
14:46
Use exponential buffer size growth in StrAccum, as long as the size does not grow to large, to avoid excess memory allocation resize operations. Also, document the fact that setting scratch memory causes SQLite to try to avoid large memory allocations. check-in: a518bc33 user: drh tags: trunk
13:24
When enlarging the size of a StrAccum object, use sqlite3DbMallocSize() to record the entire size of the allocation, not just the requested size. check-in: 3dda3c93 user: drh tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to src/printf.c.

766
767
768
769
770
771
772





773
774
775
776
777
778
779
    N = p->nAlloc - p->nChar - 1;
    setStrAccumError(p, STRACCUM_TOOBIG);
    return N;
  }else{
    char *zOld = (p->zText==p->zBase ? 0 : p->zText);
    i64 szNew = p->nChar;
    szNew += N + 1;





    if( szNew > p->mxAlloc ){
      sqlite3StrAccumReset(p);
      setStrAccumError(p, STRACCUM_TOOBIG);
      return 0;
    }else{
      p->nAlloc = (int)szNew;
    }







>
>
>
>
>







766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
    N = p->nAlloc - p->nChar - 1;
    setStrAccumError(p, STRACCUM_TOOBIG);
    return N;
  }else{
    char *zOld = (p->zText==p->zBase ? 0 : p->zText);
    i64 szNew = p->nChar;
    szNew += N + 1;
    if( szNew+p->nChar<=p->mxAlloc ){
      /* Force exponential buffer size growth as long as it does not overflow,
      ** to avoid having to call this routine too often */
      szNew += p->nChar;
    }
    if( szNew > p->mxAlloc ){
      sqlite3StrAccumReset(p);
      setStrAccumError(p, STRACCUM_TOOBIG);
      return 0;
    }else{
      p->nAlloc = (int)szNew;
    }

Changes to src/sqlite.h.in.

1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551






1552
1553
1554
1555
1556
1557
1558
** </dd>
**
** [[SQLITE_CONFIG_SCRATCH]] <dt>SQLITE_CONFIG_SCRATCH</dt>
** <dd> ^This option specifies a static memory buffer that SQLite can use for
** scratch memory.  ^(There are three arguments:  A pointer an 8-byte
** aligned memory buffer from which the scratch allocations will be
** drawn, the size of each scratch allocation (sz),
** and the maximum number of scratch allocations (N).)^  The sz
** argument must be a multiple of 16.
** The first argument must be a pointer to an 8-byte aligned buffer
** of at least sz*N bytes of memory.
** ^SQLite will not use more than two scratch buffers per thread and not
** more than one scratch buffer per thread when not performing
** a [checkpoint] in [WAL mode].
** ^SQLite will never request a scratch buffer that is more than 6
** times the database page size, except when performing a [checkpoint]
** in [WAL mode] when the scratch buffer request size is a small fraction
** of the size of the WAL file.
** ^If SQLite needs needs additional
** scratch memory beyond what is provided by this configuration option, then 
** [sqlite3_malloc()] will be used to obtain the memory needed.</dd>






**
** [[SQLITE_CONFIG_PAGECACHE]] <dt>SQLITE_CONFIG_PAGECACHE</dt>
** <dd> ^This option specifies a static memory buffer that SQLite can use for
** the database page cache with the default page cache implementation.  
** This configuration should not be used if an application-define page
** cache implementation is loaded using the SQLITE_CONFIG_PCACHE2 option.
** There are three arguments to this option: A pointer to 8-byte aligned







|
<











|
>
>
>
>
>
>







1531
1532
1533
1534
1535
1536
1537
1538

1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
** </dd>
**
** [[SQLITE_CONFIG_SCRATCH]] <dt>SQLITE_CONFIG_SCRATCH</dt>
** <dd> ^This option specifies a static memory buffer that SQLite can use for
** scratch memory.  ^(There are three arguments:  A pointer an 8-byte
** aligned memory buffer from which the scratch allocations will be
** drawn, the size of each scratch allocation (sz),
** and the maximum number of scratch allocations (N).)^

** The first argument must be a pointer to an 8-byte aligned buffer
** of at least sz*N bytes of memory.
** ^SQLite will not use more than two scratch buffers per thread and not
** more than one scratch buffer per thread when not performing
** a [checkpoint] in [WAL mode].
** ^SQLite will never request a scratch buffer that is more than 6
** times the database page size, except when performing a [checkpoint]
** in [WAL mode] when the scratch buffer request size is a small fraction
** of the size of the WAL file.
** ^If SQLite needs needs additional
** scratch memory beyond what is provided by this configuration option, then 
** [sqlite3_malloc()] will be used to obtain the memory needed.<p>
** ^When the application provides any amount of scratch memory using
** SQLITE_CONFIG_SCRATCH, SQLite avoids unnecessary large
** [sqlite3_malloc|heap allocations].
** This can help [Robson proof|prevent memory allocation failures] due to heap
** fragmentation in low-memory embedded systems.
** </dd>
**
** [[SQLITE_CONFIG_PAGECACHE]] <dt>SQLITE_CONFIG_PAGECACHE</dt>
** <dd> ^This option specifies a static memory buffer that SQLite can use for
** the database page cache with the default page cache implementation.  
** This configuration should not be used if an application-define page
** cache implementation is loaded using the SQLITE_CONFIG_PCACHE2 option.
** There are three arguments to this option: A pointer to 8-byte aligned

Changes to src/vdbesort.c.

843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861

    if( !sqlite3TempInMemory(db) ){
      pSorter->mnPmaSize = SORTER_MIN_WORKING * pgsz;
      mxCache = db->aDb[0].pSchema->cache_size;
      if( mxCache<SORTER_MIN_WORKING ) mxCache = SORTER_MIN_WORKING;
      pSorter->mxPmaSize = mxCache * pgsz;

      /* If the application has not configure scratch memory using
      ** SQLITE_CONFIG_SCRATCH then we assume it is OK to do large memory
      ** allocations.  If scratch memory has been configured, then assume
      ** large memory allocations should be avoided to prevent heap
      ** fragmentation.
      */
      if( sqlite3GlobalConfig.pScratch==0 ){
        assert( pSorter->iMemory==0 );
        pSorter->nMemory = pgsz;
        pSorter->list.aMemory = (u8*)sqlite3Malloc(pgsz);
        if( !pSorter->list.aMemory ) rc = SQLITE_NOMEM;
      }







|
|
|
<
<







843
844
845
846
847
848
849
850
851
852


853
854
855
856
857
858
859

    if( !sqlite3TempInMemory(db) ){
      pSorter->mnPmaSize = SORTER_MIN_WORKING * pgsz;
      mxCache = db->aDb[0].pSchema->cache_size;
      if( mxCache<SORTER_MIN_WORKING ) mxCache = SORTER_MIN_WORKING;
      pSorter->mxPmaSize = mxCache * pgsz;

      /* EVIDENCE-OF: R-26747-61719 When the application provides any amount of
      ** scratch memory using SQLITE_CONFIG_SCRATCH, SQLite avoids unnecessary
      ** large heap allocations.


      */
      if( sqlite3GlobalConfig.pScratch==0 ){
        assert( pSorter->iMemory==0 );
        pSorter->nMemory = pgsz;
        pSorter->list.aMemory = (u8*)sqlite3Malloc(pgsz);
        if( !pSorter->list.aMemory ) rc = SQLITE_NOMEM;
      }