Documentation Source Text

Check-in [2bc13084bc]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Enhancements to malloc.html documentation to address concerns arising out of heartbleed.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 2bc13084bc1c98f1d736c707158757c74f8542f9
User & Date: drh 2014-05-30 17:01:05.004
Context
2014-05-30
18:01
Add a section describing the Win32 native memory allocator. (check-in: ef8f1ed845 user: mistachkin tags: trunk)
17:01
Enhancements to malloc.html documentation to address concerns arising out of heartbleed. (check-in: 2bc13084bc user: drh tags: trunk)
2014-05-29
15:46
Change the default maximum POST size in althttpd to 20MB. (check-in: 35fcd6cd5b user: drh tags: trunk)
Changes
Side-by-Side Diff Ignore Whitespace Patch
Changes to pages/malloc.in.
1
2
3
4
5
6
7
8
9
10
11

12
13
14
15
16
17
18
1
2
3
4
5
6
7
8
9
10

11
12
13
14
15
16
17
18










-
+







<title>Dynamic Memory Allocation In SQLite</title>

<h1>Dynamic Memory Allocation In SQLite</h1>
<tcl>hd_keywords {memory allocation}</tcl>

<p>SQLite uses dynamic memory allocation to obtain
memory for storing various objects
(ex: [database connections] and [prepared statements]) and to build
a memory cache of the database file and to hold the results of queries.
Much effort has gone into making the dynamic memory allocation subsystem
of SQLite reliable, predictable, robust, and efficient.</p>
of SQLite reliable, predictable, robust, secure, and efficient.</p>

<p>This document provides an overview of dynamic memory allocation within 
SQLite.  The target audience is software engineers who are tuning their
use of SQLite for peak performance in demanding environments.
Nothing in this document is required knowledge for using SQLite.  The
default settings and configuration for SQLite will work well in most
applications.  However, the information contained in this document may
55
56
57
58
59
60
61
62
63


64
65
66
67
68
69
70
55
56
57
58
59
60
61


62
63
64
65
66
67
68
69
70







-
-
+
+







The [sqlite3_soft_heap_limit64()] mechanism allows the application to
set a memory usage limit that SQLite strives to stay below.  SQLite
will attempt to reuse memory from its caches rather than allocating new
memory as it approaches the soft limit.
</p></li>

<li><p>
<b>Zero-malloc option</b>
The application can provide SQLite with several buffers of bulk memory
<b>Zero-malloc option.</b>
The application can optionally provide SQLite with several buffers of bulk memory
at startup and SQLite will then use those provided buffers for all of
its memory allocation needs and never call system malloc() or free().
</p></li>

<li><p>
<b>Application-supplied memory allocators.</b>
The application can provide SQLite with pointers to alternative 
84
85
86
87
88
89
90








91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112

113
114
115
116
117
118
119
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118


119
120
121
122
123
124
125
126







+
+
+
+
+
+
+
+




















-
-
+








<li><p>
<b>Memory usage statistics.</b>
Applications can see how much memory they are using and detect when
memory usage is approaching or exceeding design boundaries.
</p></li>

<a name="pwwo"></a>
<li><p>
<b>Plays well with memory debuggers.</b>
Memory allocation in SQLite is structured so that standard
third-party memory debuggers (such as [http://dmalloc.com | dmalloc] or 
[http://valgrind.org | valgrind]) can be used to verify correct
memory allocation behavior.</p>

<li><p>
<b>Minimal calls to the allocator.</b>
The system malloc() and free() implementations are inefficient
on many systems.  SQLite strives to reduce overall processing time
by minimizing its use of malloc() and free().
</p></li>

<li><p>
<b>Open access.</b>
Pluggable SQLite extensions or even the application itself can 
access to the same underlying memory allocation
routines used by SQLite through the
[sqlite3_malloc()], [sqlite3_realloc()], and [sqlite3_free()] interfaces.
</p></li>

</ul>

<a name="testing"></a>
<h2>2.0 Testing</h2>

<p>Over
75% of the code in the SQLite source tree is devoted purely to 
<p>Most of the code in the SQLite source tree is devoted purely to 
[testing | testing and verification].  Reliability is important to SQLite.
Among the tasks of the test infrastructure is to ensure that
SQLite does not misuse dynamically allocated memory, that SQLite
does not leak memory, and that SQLite responds
correctly to a dynamic memory allocation failure.</p>

<p>The test infrastructure verifies that SQLite does not misuse
188
189
190
191
192
193
194







































195
196
197
198
199
200
201
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247







+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+







memory leak detector both work over the entire SQLite test suite and
the [TCL test suite] provides over 99% statement test coverage and that
the [TH3] test harness provides [test coverage | 100% branch test coverage]
with no leak leaks. This is
strong evidence that dynamic memory allocation is used correctly
everywhere within SQLite.</p>

<a name="allocarray"></a>
<h3>2.1 Use of reallocarray()</h3>

<p>The reallocarray() interface is a recent innovation (circa 2014)
from the OpenBSD community that grow out of efforts to prevent the
next [http://heartbleed.com | "heartbleed" bug] by avoiding 32-bit integer
arithmetic overflow on memory allocation size computations.  The
reallocarray() function has both unit-size and count parameters.
To allocate memory sufficient to hold an array of N elements each X-bytes
in size, one calls "reallocarray(0,X,N)".  This is preferred over
the traditional technique of invoking "malloc(X*N)" as reallocarray()
eliminates the risk that the X*N multiplication will overflow and
cause malloc() to return a buffer that is a different size from what
the application expected.</p>

<p>SQLite does not use reallocarray().  The reason is that reallocarray()
is not useful to SQLite.  It turns out that SQLite never does memory
allocations that are the simple product of two integers.  Instead, SQLite
does allocations of the form "X+C" or "N*X+C" or "M*N*X+C" or
"N*X+M*Y+C", and so forth.  The reallocarray() interface is not helpful
in avoiding integer overflow in those cases.</p>

<p>Nevertheless, integer overflow in the computation of memory allocation
sizes is a concern that SQLite would like to deal with.  To prevent
problems, all SQLite internal memory allocations occur using thin wrapper
functions that take a signed 64-bit integer size parameter.  The SQLite 
source code is audited to ensure that all size computations are carried 
out using 64-bit signed integers as well.  SQLite will
refuse to allocate more than about 2GB of memory at one go.  (In common
use, SQLite seldom ever allocates more than about 8KB of memory at a time
so a 2GB allocation limit is not a burden.)  So the 64-bit size parameter
provides lots of headroom for detecting overflows.  The same audit that
verifies that all size computations are done as 64-bit signed integers
also verifies that it is impossible to overflow a 64-bit integer
during the computation.</p>

<p>The code audits used to ensure that memory allocation size computations
do not overflow in SQLite are repeated prior to every SQLite release.</p>

<a name="config"></a>
<h2>3.0 Configuration</h2>

<p>The default memory allocation settings in SQLite are appropriate
for most applications.  However, applications with unusual or particularly
strict requirements may want to adjust the configuration to more closely
align SQLite to their needs.
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959



960
961
962
963
964
982
983
984
985
986
987
988

















989
990
991












-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
-
-
-
-
-
The [memory statistics] interfaces of SQLite provide the application with
all the mechanism necessary to complete the monitoring portion of
this task.</p>

<a name="stability"></a>
<h2>5.0 Stability Of Memory Interfaces</h2>

<p>As of this writing (circa SQLite version 3.6.1) all of the alternative
memory allocators and mechanisms for manipulating, controlling, and
measuring memory allocation in SQLite are considered experimental and
subject to change from one release to the next.  These interfaces are
in the process of being refined to work on a wide variety of systems
under a range of constraints.  The SQLite developers need the flexibility
to change the memory allocator interfaces in order to best meet the
needs of a wide variety of systems.</p>

<p>One may anticipate that the memory allocator interfaces will
eventually stabilize.  Appropriate notice will be given when that
occurs.  In the meantime, applications developers who make use of
these interfaces need to be prepared to modify their applications
to accommodate changes in the SQLite interface.</p>

<p><b>Update:</b> As of SQLite version 3.7.0, all of these interfaces
are considered stable</p>
<p><b>Update:</b> As of SQLite version 3.7.0 (2010-07-22), 
all of SQLite memory allocation interfaces
are considered stable and will be supported in future releases.</p>

<a name="summary"></a>
<h2>6.0 Summary Of Memory Allocator Interfaces</h2>

<p><i>To be completed...</i></p>