Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
| Comment: | Enhancements to malloc.html documentation to address concerns arising out of heartbleed. |
|---|---|
| Timelines: | family | ancestors | descendants | both | trunk |
| Files: | files | file ages | folders |
| SHA1: |
2bc13084bc1c98f1d736c707158757c7 |
| User & Date: | drh 2014-05-30 17:01:05 |
Context
|
2014-05-30
| ||
| 18:01 | Add a section describing the Win32 native memory allocator. check-in: ef8f1ed845 user: mistachkin tags: trunk | |
| 17:01 | Enhancements to malloc.html documentation to address concerns arising out of heartbleed. check-in: 2bc13084bc user: drh tags: trunk | |
|
2014-05-29
| ||
| 15:46 | Change the default maximum POST size in althttpd to 20MB. check-in: 35fcd6cd5b user: drh tags: trunk | |
Changes
Changes to pages/malloc.in.
4 4 <tcl>hd_keywords {memory allocation}</tcl> 5 5 6 6 <p>SQLite uses dynamic memory allocation to obtain 7 7 memory for storing various objects 8 8 (ex: [database connections] and [prepared statements]) and to build 9 9 a memory cache of the database file and to hold the results of queries. 10 10 Much effort has gone into making the dynamic memory allocation subsystem 11 -of SQLite reliable, predictable, robust, and efficient.</p> 11 +of SQLite reliable, predictable, robust, secure, and efficient.</p> 12 12 13 13 <p>This document provides an overview of dynamic memory allocation within 14 14 SQLite. The target audience is software engineers who are tuning their 15 15 use of SQLite for peak performance in demanding environments. 16 16 Nothing in this document is required knowledge for using SQLite. The 17 17 default settings and configuration for SQLite will work well in most 18 18 applications. However, the information contained in this document may ................................................................................ 55 55 The [sqlite3_soft_heap_limit64()] mechanism allows the application to 56 56 set a memory usage limit that SQLite strives to stay below. SQLite 57 57 will attempt to reuse memory from its caches rather than allocating new 58 58 memory as it approaches the soft limit. 59 59 </p></li> 60 60 61 61 <li><p> 62 -<b>Zero-malloc option</b> 63 -The application can provide SQLite with several buffers of bulk memory 62 +<b>Zero-malloc option.</b> 63 +The application can optionally provide SQLite with several buffers of bulk memory 64 64 at startup and SQLite will then use those provided buffers for all of 65 65 its memory allocation needs and never call system malloc() or free(). 66 66 </p></li> 67 67 68 68 <li><p> 69 69 <b>Application-supplied memory allocators.</b> 70 70 The application can provide SQLite with pointers to alternative ................................................................................ 84 84 85 85 <li><p> 86 86 <b>Memory usage statistics.</b> 87 87 Applications can see how much memory they are using and detect when 88 88 memory usage is approaching or exceeding design boundaries. 89 89 </p></li> 90 90 91 +<a name="pwwo"></a> 92 +<li><p> 93 +<b>Plays well with memory debuggers.</b> 94 +Memory allocation in SQLite is structured so that standard 95 +third-party memory debuggers (such as [http://dmalloc.com | dmalloc] or 96 +[http://valgrind.org | valgrind]) can be used to verify correct 97 +memory allocation behavior.</p> 98 + 91 99 <li><p> 92 100 <b>Minimal calls to the allocator.</b> 93 101 The system malloc() and free() implementations are inefficient 94 102 on many systems. SQLite strives to reduce overall processing time 95 103 by minimizing its use of malloc() and free(). 96 104 </p></li> 97 105 ................................................................................ 104 112 </p></li> 105 113 106 114 </ul> 107 115 108 116 <a name="testing"></a> 109 117 <h2>2.0 Testing</h2> 110 118 111 -<p>Over 112 -75% of the code in the SQLite source tree is devoted purely to 119 +<p>Most of the code in the SQLite source tree is devoted purely to 113 120 [testing | testing and verification]. Reliability is important to SQLite. 114 121 Among the tasks of the test infrastructure is to ensure that 115 122 SQLite does not misuse dynamically allocated memory, that SQLite 116 123 does not leak memory, and that SQLite responds 117 124 correctly to a dynamic memory allocation failure.</p> 118 125 119 126 <p>The test infrastructure verifies that SQLite does not misuse ................................................................................ 188 195 memory leak detector both work over the entire SQLite test suite and 189 196 the [TCL test suite] provides over 99% statement test coverage and that 190 197 the [TH3] test harness provides [test coverage | 100% branch test coverage] 191 198 with no leak leaks. This is 192 199 strong evidence that dynamic memory allocation is used correctly 193 200 everywhere within SQLite.</p> 194 201 202 +<a name="allocarray"></a> 203 +<h3>2.1 Use of reallocarray()</h3> 204 + 205 +<p>The reallocarray() interface is a recent innovation (circa 2014) 206 +from the OpenBSD community that grow out of efforts to prevent the 207 +next [http://heartbleed.com | "heartbleed" bug] by avoiding 32-bit integer 208 +arithmetic overflow on memory allocation size computations. The 209 +reallocarray() function has both unit-size and count parameters. 210 +To allocate memory sufficient to hold an array of N elements each X-bytes 211 +in size, one calls "reallocarray(0,X,N)". This is preferred over 212 +the traditional technique of invoking "malloc(X*N)" as reallocarray() 213 +eliminates the risk that the X*N multiplication will overflow and 214 +cause malloc() to return a buffer that is a different size from what 215 +the application expected.</p> 216 + 217 +<p>SQLite does not use reallocarray(). The reason is that reallocarray() 218 +is not useful to SQLite. It turns out that SQLite never does memory 219 +allocations that are the simple product of two integers. Instead, SQLite 220 +does allocations of the form "X+C" or "N*X+C" or "M*N*X+C" or 221 +"N*X+M*Y+C", and so forth. The reallocarray() interface is not helpful 222 +in avoiding integer overflow in those cases.</p> 223 + 224 +<p>Nevertheless, integer overflow in the computation of memory allocation 225 +sizes is a concern that SQLite would like to deal with. To prevent 226 +problems, all SQLite internal memory allocations occur using thin wrapper 227 +functions that take a signed 64-bit integer size parameter. The SQLite 228 +source code is audited to ensure that all size computations are carried 229 +out using 64-bit signed integers as well. SQLite will 230 +refuse to allocate more than about 2GB of memory at one go. (In common 231 +use, SQLite seldom ever allocates more than about 8KB of memory at a time 232 +so a 2GB allocation limit is not a burden.) So the 64-bit size parameter 233 +provides lots of headroom for detecting overflows. The same audit that 234 +verifies that all size computations are done as 64-bit signed integers 235 +also verifies that it is impossible to overflow a 64-bit integer 236 +during the computation.</p> 237 + 238 +<p>The code audits used to ensure that memory allocation size computations 239 +do not overflow in SQLite are repeated prior to every SQLite release.</p> 240 + 195 241 <a name="config"></a> 196 242 <h2>3.0 Configuration</h2> 197 243 198 244 <p>The default memory allocation settings in SQLite are appropriate 199 245 for most applications. However, applications with unusual or particularly 200 246 strict requirements may want to adjust the configuration to more closely 201 247 align SQLite to their needs. ................................................................................ 936 982 The [memory statistics] interfaces of SQLite provide the application with 937 983 all the mechanism necessary to complete the monitoring portion of 938 984 this task.</p> 939 985 940 986 <a name="stability"></a> 941 987 <h2>5.0 Stability Of Memory Interfaces</h2> 942 988 943 -<p>As of this writing (circa SQLite version 3.6.1) all of the alternative 944 -memory allocators and mechanisms for manipulating, controlling, and 945 -measuring memory allocation in SQLite are considered experimental and 946 -subject to change from one release to the next. These interfaces are 947 -in the process of being refined to work on a wide variety of systems 948 -under a range of constraints. The SQLite developers need the flexibility 949 -to change the memory allocator interfaces in order to best meet the 950 -needs of a wide variety of systems.</p> 951 - 952 -<p>One may anticipate that the memory allocator interfaces will 953 -eventually stabilize. Appropriate notice will be given when that 954 -occurs. In the meantime, applications developers who make use of 955 -these interfaces need to be prepared to modify their applications 956 -to accommodate changes in the SQLite interface.</p> 957 - 958 -<p><b>Update:</b> As of SQLite version 3.7.0, all of these interfaces 959 -are considered stable</p> 960 - 961 -<a name="summary"></a> 962 -<h2>6.0 Summary Of Memory Allocator Interfaces</h2> 963 - 964 -<p><i>To be completed...</i></p> 989 +<p><b>Update:</b> As of SQLite version 3.7.0 (2010-07-22), 990 +all of SQLite memory allocation interfaces 991 +are considered stable and will be supported in future releases.</p>