Documentation Source Text

Check-in [2bc13084bc]

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Comment:Enhancements to malloc.html documentation to address concerns arising out of heartbleed.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 2bc13084bc1c98f1d736c707158757c74f8542f9
User & Date: drh 2014-05-30 17:01:05
Add a section describing the Win32 native memory allocator. (check-in: ef8f1ed845 user: mistachkin tags: trunk)
Enhancements to malloc.html documentation to address concerns arising out of heartbleed. (check-in: 2bc13084bc user: drh tags: trunk)
Change the default maximum POST size in althttpd to 20MB. (check-in: 35fcd6cd5b user: drh tags: trunk)
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Changes to pages/

     4      4   <tcl>hd_keywords {memory allocation}</tcl>
     5      5   
     6      6   <p>SQLite uses dynamic memory allocation to obtain
     7      7   memory for storing various objects
     8      8   (ex: [database connections] and [prepared statements]) and to build
     9      9   a memory cache of the database file and to hold the results of queries.
    10     10   Much effort has gone into making the dynamic memory allocation subsystem
    11         -of SQLite reliable, predictable, robust, and efficient.</p>
           11  +of SQLite reliable, predictable, robust, secure, and efficient.</p>
    12     12   
    13     13   <p>This document provides an overview of dynamic memory allocation within 
    14     14   SQLite.  The target audience is software engineers who are tuning their
    15     15   use of SQLite for peak performance in demanding environments.
    16     16   Nothing in this document is required knowledge for using SQLite.  The
    17     17   default settings and configuration for SQLite will work well in most
    18     18   applications.  However, the information contained in this document may
    55     55   The [sqlite3_soft_heap_limit64()] mechanism allows the application to
    56     56   set a memory usage limit that SQLite strives to stay below.  SQLite
    57     57   will attempt to reuse memory from its caches rather than allocating new
    58     58   memory as it approaches the soft limit.
    59     59   </p></li>
    60     60   
    61     61   <li><p>
    62         -<b>Zero-malloc option</b>
    63         -The application can provide SQLite with several buffers of bulk memory
           62  +<b>Zero-malloc option.</b>
           63  +The application can optionally provide SQLite with several buffers of bulk memory
    64     64   at startup and SQLite will then use those provided buffers for all of
    65     65   its memory allocation needs and never call system malloc() or free().
    66     66   </p></li>
    67     67   
    68     68   <li><p>
    69     69   <b>Application-supplied memory allocators.</b>
    70     70   The application can provide SQLite with pointers to alternative 
    84     84   
    85     85   <li><p>
    86     86   <b>Memory usage statistics.</b>
    87     87   Applications can see how much memory they are using and detect when
    88     88   memory usage is approaching or exceeding design boundaries.
    89     89   </p></li>
    90     90   
           91  +<a name="pwwo"></a>
           92  +<li><p>
           93  +<b>Plays well with memory debuggers.</b>
           94  +Memory allocation in SQLite is structured so that standard
           95  +third-party memory debuggers (such as [ | dmalloc] or 
           96  +[ | valgrind]) can be used to verify correct
           97  +memory allocation behavior.</p>
           98  +
    91     99   <li><p>
    92    100   <b>Minimal calls to the allocator.</b>
    93    101   The system malloc() and free() implementations are inefficient
    94    102   on many systems.  SQLite strives to reduce overall processing time
    95    103   by minimizing its use of malloc() and free().
    96    104   </p></li>
    97    105   
   104    112   </p></li>
   105    113   
   106    114   </ul>
   107    115   
   108    116   <a name="testing"></a>
   109    117   <h2>2.0 Testing</h2>
   110    118   
   111         -<p>Over
   112         -75% of the code in the SQLite source tree is devoted purely to 
          119  +<p>Most of the code in the SQLite source tree is devoted purely to 
   113    120   [testing | testing and verification].  Reliability is important to SQLite.
   114    121   Among the tasks of the test infrastructure is to ensure that
   115    122   SQLite does not misuse dynamically allocated memory, that SQLite
   116    123   does not leak memory, and that SQLite responds
   117    124   correctly to a dynamic memory allocation failure.</p>
   118    125   
   119    126   <p>The test infrastructure verifies that SQLite does not misuse
   188    195   memory leak detector both work over the entire SQLite test suite and
   189    196   the [TCL test suite] provides over 99% statement test coverage and that
   190    197   the [TH3] test harness provides [test coverage | 100% branch test coverage]
   191    198   with no leak leaks. This is
   192    199   strong evidence that dynamic memory allocation is used correctly
   193    200   everywhere within SQLite.</p>
   194    201   
          202  +<a name="allocarray"></a>
          203  +<h3>2.1 Use of reallocarray()</h3>
          204  +
          205  +<p>The reallocarray() interface is a recent innovation (circa 2014)
          206  +from the OpenBSD community that grow out of efforts to prevent the
          207  +next [ | "heartbleed" bug] by avoiding 32-bit integer
          208  +arithmetic overflow on memory allocation size computations.  The
          209  +reallocarray() function has both unit-size and count parameters.
          210  +To allocate memory sufficient to hold an array of N elements each X-bytes
          211  +in size, one calls "reallocarray(0,X,N)".  This is preferred over
          212  +the traditional technique of invoking "malloc(X*N)" as reallocarray()
          213  +eliminates the risk that the X*N multiplication will overflow and
          214  +cause malloc() to return a buffer that is a different size from what
          215  +the application expected.</p>
          216  +
          217  +<p>SQLite does not use reallocarray().  The reason is that reallocarray()
          218  +is not useful to SQLite.  It turns out that SQLite never does memory
          219  +allocations that are the simple product of two integers.  Instead, SQLite
          220  +does allocations of the form "X+C" or "N*X+C" or "M*N*X+C" or
          221  +"N*X+M*Y+C", and so forth.  The reallocarray() interface is not helpful
          222  +in avoiding integer overflow in those cases.</p>
          223  +
          224  +<p>Nevertheless, integer overflow in the computation of memory allocation
          225  +sizes is a concern that SQLite would like to deal with.  To prevent
          226  +problems, all SQLite internal memory allocations occur using thin wrapper
          227  +functions that take a signed 64-bit integer size parameter.  The SQLite 
          228  +source code is audited to ensure that all size computations are carried 
          229  +out using 64-bit signed integers as well.  SQLite will
          230  +refuse to allocate more than about 2GB of memory at one go.  (In common
          231  +use, SQLite seldom ever allocates more than about 8KB of memory at a time
          232  +so a 2GB allocation limit is not a burden.)  So the 64-bit size parameter
          233  +provides lots of headroom for detecting overflows.  The same audit that
          234  +verifies that all size computations are done as 64-bit signed integers
          235  +also verifies that it is impossible to overflow a 64-bit integer
          236  +during the computation.</p>
          237  +
          238  +<p>The code audits used to ensure that memory allocation size computations
          239  +do not overflow in SQLite are repeated prior to every SQLite release.</p>
          240  +
   195    241   <a name="config"></a>
   196    242   <h2>3.0 Configuration</h2>
   197    243   
   198    244   <p>The default memory allocation settings in SQLite are appropriate
   199    245   for most applications.  However, applications with unusual or particularly
   200    246   strict requirements may want to adjust the configuration to more closely
   201    247   align SQLite to their needs.
   936    982   The [memory statistics] interfaces of SQLite provide the application with
   937    983   all the mechanism necessary to complete the monitoring portion of
   938    984   this task.</p>
   939    985   
   940    986   <a name="stability"></a>
   941    987   <h2>5.0 Stability Of Memory Interfaces</h2>
   942    988   
   943         -<p>As of this writing (circa SQLite version 3.6.1) all of the alternative
   944         -memory allocators and mechanisms for manipulating, controlling, and
   945         -measuring memory allocation in SQLite are considered experimental and
   946         -subject to change from one release to the next.  These interfaces are
   947         -in the process of being refined to work on a wide variety of systems
   948         -under a range of constraints.  The SQLite developers need the flexibility
   949         -to change the memory allocator interfaces in order to best meet the
   950         -needs of a wide variety of systems.</p>
   951         -
   952         -<p>One may anticipate that the memory allocator interfaces will
   953         -eventually stabilize.  Appropriate notice will be given when that
   954         -occurs.  In the meantime, applications developers who make use of
   955         -these interfaces need to be prepared to modify their applications
   956         -to accommodate changes in the SQLite interface.</p>
   957         -
   958         -<p><b>Update:</b> As of SQLite version 3.7.0, all of these interfaces
   959         -are considered stable</p>
   960         -
   961         -<a name="summary"></a>
   962         -<h2>6.0 Summary Of Memory Allocator Interfaces</h2>
   963         -
   964         -<p><i>To be completed...</i></p>
          989  +<p><b>Update:</b> As of SQLite version 3.7.0 (2010-07-22), 
          990  +all of SQLite memory allocation interfaces
          991  +are considered stable and will be supported in future releases.</p>