Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
Comment: | Begin adding the document on memory allocation. Update the index and changes documents for the release of version 3.6.1. |
---|---|
Downloads: | Tarball | ZIP archive |
Timelines: | family | ancestors | descendants | both | trunk |
Files: | files | file ages | folders |
SHA1: |
8f269144c39bf02f63f5be0f32f42c07 |
User & Date: | drh 2008-07-31 17:13:58.000 |
Context
2008-08-04
| ||
13:45 | Enhanced markings for experimental and deprecated interfaces. (check-in: afc454f404 user: drh tags: trunk) | |
2008-07-31
| ||
17:13 | Begin adding the document on memory allocation. Update the index and changes documents for the release of version 3.6.1. (check-in: 8f269144c3 user: drh tags: trunk) | |
16:11 | Further incremental improvements to fileio.in. (check-in: 89222751c6 user: dan tags: trunk) | |
Changes
Changes to pages/changes.in.
︙ | ︙ | |||
36 37 38 39 40 41 42 43 44 45 46 47 48 49 | <a href="http://www.sqlite.org/cvstrac/timeline"> http://www.sqlite.org/cvstrac/timeline</a>.</p> } hd_close_aux hd_enable_main 1 } } chng {2008 July 16 (3.6.0 beta)} { <li>Modifications to the [sqlite3_vfs | virtual file system] interface to support a wider range of embedded systems. See [35to36 | 35to36.html] for additional information. <font color="red">*** Potentially incompatible change ***</font></li> <li>All C-preprocessor macros used to control compile-time options | > > > > > > > > > > > > | 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | <a href="http://www.sqlite.org/cvstrac/timeline"> http://www.sqlite.org/cvstrac/timeline</a>.</p> } hd_close_aux hd_enable_main 1 } } chng {2008 Aug 6 (3.6.1)} { <li>Added the "lookaside memory allocator" for a speed improvement in excess of 15% on some workloads. (Your mileage may vary.)</li> <li>Added the [SQLITE_CONFIG_LOOKASIDE] verb to [sqlite3_config()] to control the default lookaside configuration.</li> <li>Added the [sqlite3_db_config()] and [sqlite3_db_status()] interfaces for controlling and monitoring the lookaside allocator separately on each [database connection].</li> <li>Numerious other performance enhancements</li> <li>Miscellaneous minor bug fixes</li> } chng {2008 July 16 (3.6.0 beta)} { <li>Modifications to the [sqlite3_vfs | virtual file system] interface to support a wider range of embedded systems. See [35to36 | 35to36.html] for additional information. <font color="red">*** Potentially incompatible change ***</font></li> <li>All C-preprocessor macros used to control compile-time options |
︙ | ︙ |
Changes to pages/compile.in.
︙ | ︙ | |||
22 23 24 25 26 27 28 29 30 31 32 33 34 35 | <h2>1.1 Options To Set Default Parameter Values</h2> <tcl> proc COMPILE_OPTION {name text} { if {[regexp {SQLITE_([A-Z0-9_]+)} $name all label]} { hd_fragment [string tolower $label] hd_keywords $all } hd_puts <p><b>$name</b></p> regsub -all "\n\\s*\n" $text "</p>\n\n<p>" text hd_resolve <blockquote><p>$text</p></blockquote> } COMPILE_OPTION {SQLITE_DEFAULT_AUTOVACUUM=<i><1 or 0></i>} { | > > > > | 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | <h2>1.1 Options To Set Default Parameter Values</h2> <tcl> proc COMPILE_OPTION {name text} { if {[regexp {SQLITE_([A-Z0-9_]+)} $name all label]} { hd_fragment [string tolower $label] hd_keywords $all } if {[regexp {^YY([A-Z0-9_]+)} $name all label]} { hd_fragment [string tolower $all] hd_keywords $all } hd_puts <p><b>$name</b></p> regsub -all "\n\\s*\n" $text "</p>\n\n<p>" text hd_resolve <blockquote><p>$text</p></blockquote> } COMPILE_OPTION {SQLITE_DEFAULT_AUTOVACUUM=<i><1 or 0></i>} { |
︙ | ︙ | |||
74 75 76 77 78 79 80 81 82 83 84 85 86 87 | COMPILE_OPTION {SQLITE_DEFAULT_TEMP_CACHE_SIZE=<i><pages></i>} { This macro sets the default size of the page-cache for temporary files created by SQLite to store intermediate results, in pages. It does not affect the page-cache for the temp database, where tables created using [CREATE TABLE | CREATE TEMP TABLE] are stored. The default value is 500. } </tcl> <h2>1.2 Options To Set Size Limits</h2> <p>There are compile-time options that will set upper bounds on the sizes of various structures in SQLite. The compile-time options normally set a hard upper bound which can be changed | > > > > > > > > > > | 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | COMPILE_OPTION {SQLITE_DEFAULT_TEMP_CACHE_SIZE=<i><pages></i>} { This macro sets the default size of the page-cache for temporary files created by SQLite to store intermediate results, in pages. It does not affect the page-cache for the temp database, where tables created using [CREATE TABLE | CREATE TEMP TABLE] are stored. The default value is 500. } COMPILE_OPTION {YYSTACKDEPTH=<i><max_depth></i>} { This macro sets the maximum depth of the LALR(1) stack used by the SQL parser within SQLite. The default value is 100. A typical application will use less than about 20 levels of the stack. Developers whose applications contain SQL statements that need more than 100 LALR(1) stack entries should seriously consider refactoring their SQL as it is likely to be well beyond the ability of any human to comprehend. } </tcl> <h2>1.2 Options To Set Size Limits</h2> <p>There are compile-time options that will set upper bounds on the sizes of various structures in SQLite. The compile-time options normally set a hard upper bound which can be changed |
︙ | ︙ | |||
251 252 253 254 255 256 257 258 259 260 261 262 263 264 | subject to certain operating constraints. } COMPILE_OPTION {SQLITE_ENABLE_RTREE} { This option causes SQLite to include support for the [rtree | R*Tree index extension]. } </tcl> <a name="omitfeatures"></a> <h2>1.5 Options To Omit Features</h2> <p>The following options can used to reduce the size of the compiled library by omitting optional features. This is probably only useful | > > > > > > > > > | 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 | subject to certain operating constraints. } COMPILE_OPTION {SQLITE_ENABLE_RTREE} { This option causes SQLite to include support for the [rtree | R*Tree index extension]. } COMPILE_OPTION {YYTRACKMAXSTACKDEPTH} { This option causes the LALR(1) parser stack depth to be tracked and reported using the [sqlite3_status]([SQLITE_STATUS_PARSER_STACK],...) interface. SQLite's LALR(1) parser has a fixed stack depth (determined at compile-time using the [YYSTACKDEPTH] options). This option can be used to help determine if an application is getting close to exceeding the maximum LALR(1) stack depth. } </tcl> <a name="omitfeatures"></a> <h2>1.5 Options To Omit Features</h2> <p>The following options can used to reduce the size of the compiled library by omitting optional features. This is probably only useful |
︙ | ︙ | |||
585 586 587 588 589 590 591 592 | mechanism in SQLite. } COMPILE_OPTION {SQLITE_OMIT_XFER_OPT} { This option omits support for optimizations that help statements of the form "INSERT INTO ... SELECT ..." run faster. } </tcl> | > > > > > > > > > > > > > > > > > > > > > > > > > > | 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 | mechanism in SQLite. } COMPILE_OPTION {SQLITE_OMIT_XFER_OPT} { This option omits support for optimizations that help statements of the form "INSERT INTO ... SELECT ..." run faster. } </tcl> <a name="omitfeatures"></a> <h2>1.6 Analysis and Debugging Options</h2> <tcl> COMPILE_OPTION {SQLITE_DEBUG} { The SQLite source code contains literally thousands of assert() statements used to verify internal assumptions and subroutine preconditions and postconditions. These assert() statements are normally turned off (they generate no code) since turning them on makes SQLite run approximately three times slower. But for testing and analysis, it is useful to turn the assert() statements on. The SQLITE_DEBUG compile-time option does this. SQLITE_DEBUG also turns on some other debugging features. } COMPILE_OPTION {SQLITE_MEMDEBUG} { The SQLITE_MEMDEBUG option causes an instrumented dynamic memory allocator to be used as the default memory allocator within SQLite. The instrumented memory allocator checks for misuse of dynamically allocated memory. Examples of misuse include using memory after it is freed, writing off the ends of a memory allocation, freeing memory not previously obtained from the memory allocator, or failing to initialize newly allocated memory. } </tcl> |
Changes to pages/index.in.
︙ | ︙ | |||
58 59 60 61 62 63 64 | </td> <td width="20"></td><td bgcolor="#80a796" width="1"></td><td width="20"></td> <td valign="top"> <h3>Current Status</h3> <p><ul> | | | 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 | </td> <td width="20"></td><td bgcolor="#80a796" width="1"></td><td width="20"></td> <td valign="top"> <h3>Current Status</h3> <p><ul> <li><a href="releaselog/3_6_1.html">Version 3.6.1</a> of SQLite is recommended for all new development.</li> </ul></p> <h3>Common Links</h3> <p><ul> <li> <a href="features.html">Features</a> </li> |
︙ | ︙ |
Added pages/malloc.in.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 | <h1>Dynamic Memory Allocation In SQLite</h1> <tcl>hd_keywords {memory allocation}</tcl> <p>SQLite makes extensive use of dynamic memory allocation to obtain the memory it needs to store various objects (ex: [database connections] and [prepared statements]) and to build a memory cache of the database file and to hold the results of queries. Much effort has gone into making the dynamic memory allocation subsystem of SQLite reliable, predictable, robust, and efficient.</p> <p>This document provides an overview of dynamic memory allocation within SQLite. The target audience is software engineers who are tuning their use of SQLite for peak performance in demanding environments. Nothing in this document is required knowledge for using SQLite. The default settings and configuration for SQLite will work well in most applications. However, the information contained in this document may be useful to engineers who are tuning SQLite to comply with special requirements or to run under unusual circumstances.</p> <a name="features"></a> <h2>1.0 Features</h2> <p>The SQLite core and its memory allocation subsystem provides the following capabilities:</p> <ul> <li><p> <b>Robust against allocation failures.</b> If a memory allocation ever fails (that is to say, if malloc() or realloc() ever return NULL) then SQLite will recover gracefully. SQLite will first try to find the memory it needs from a backup memory source. (Details are provided below.) Failing that, SQLite will either stop what it is doing and return the [SQLITE_NOMEM] error code back up to the application or it will make due without the requested memory. </p></li> <li><p> <b>No memory leaks.</b> The application must destroy any objects it allocates. (For example, the application must use [sqlite3_finalize()] on every [prepared statement] and [sqlite3_close()] on every [database connection].) But as long as the application cooperates, SQLite will never leak memory. This is true even in the face of memory allocation failures of other system errors. </p></li> <li><p> <b>Memory usage limits.</b> The [sqlite3_soft_heap_limit()] mechanism allows the application to set a memory usage limit that SQLite strives to stay below. SQLite will attempt to reuse memory from its caches rather than allocation new memory as it approaches the soft limit. </p></li> <li><p> <b>Zero-malloc option</b> The application can provide SQLite with several buffers of bulk memory at startup and SQLite will then use those provided buffers for all of its memory allocation needs and never call system malloc() or free(). </p></li> <li><p> <b>Application-supplied memory allocators.</b> The application can provide SQLite with pointers to an alternative memory allocators at start-time. The alternative memory allocator will be used in place of system malloc() and free(). </p></li> <li><p> <b>Proof against breakdown and fragmentation.</b> SQLite can be configured so that, subject to certain usage constraints detailed below, it is guaranteed to never fail a memory allocation or fragment the heap. This property is important to long-running, high-reliability embedded systems where a memory allocation error could contribute to an overall system failure. </p></li> <li><p> <b>Memory usage statistics.</b> Applications can see how much memory they are using and detect when memory usage is approaching or exceeding design boundaries. </p></li> <li><p> <b>Minimal calls to the allocator.</b> The system malloc() and free() implementations are inefficient on many systems. SQLite strives to reduce overall processing time by minimizing its use of malloc() and free(). </p></li> <li><p> <b>Open access.</b> Pluggable SQLite extensions or even the application itself can access to the same underlying memory allocation routines used by SQLite through the [sqlite3_malloc()], [sqlite3_realloc()], and [sqlite3_free()] interfaces. </p></li> </ul> <a name="testing"></a> <h2>2.0 Testing</h2> <p>The source code to SQLite consists mostly of test cases. Over 75% of the code in the SQLite source tree is devoted purely to testing and verification. Reliability is important to SQLite. Among the tasks of the test infrastructure is to insure that SQLite does not misuse dynamically allocated memory, that SQLite does not leak memory, and that SQLite responses correctly to a dynamic memory allocation failure.</p> <p>The test infrastructure verifies that SQLite does not misuse dynamically allocated memory by using a specially instrumented memory allocator. The instrumented memory allocator is enabled at compile-time using the [SQLITE_MEMDEBUG] option. The instrumented memory allocator is much slower than the default memory allocator and so its use is definitely not recommended in production. But when enabled during testing, the instrumented memory allocator performs the following checks:</p> <ul> <li><p><b>Bounds checking.</b> The instrumented memory allocator places sentinal values at both ends of each memory allocation to verify that nothing within SQLite writes outside the bounds of the allocation.</p></li> <li><p><b>Use of memory after freeing.</b> When each block of memory is freed, every byte is overwritten with a nonsense bit pattern. This helps to insure that no memory is ever used after having been freed.</p></li> <li><p><b>Freeing memory not obtained from malloc.</b> Each memory allocation from the instrumented memory allocator contains sentinals used to verify that every allocation freed came from prior malloc.</p></li> <li><p><b>Uninitialized memory.</b> The instrumented memory allocator initializes each memory allocation to a nonsense bit pattern to help insure that the user makes no assumptions about the content of allocation memory.</p></li> </ul> <p>Regardless of whether or not the instrumented memory allocator is used, SQLite keeps track of how much memory is currently checked out. There are hundreds of test scripts used for testing SQLite. At the end of each script, all objects are destroyed and a test is made to insure that all memory has been freed. This is how memory leaks are detected. Notice that memory leak detection is in force at all times, during test builds and during production builds. Whenever one of the developers runs any individual test script, memory leak detection is active. Hence memory leaks that do arise during development are quickly spotted and quinched.</p> <a name="oomtesting"></a> <p>The response of SQLite to out-of-memory (OOM) errors is tested using a specialized memory allocator overlay that can simulate memory failures. The overlay is a layer that is inserted in between the memory allocator and the rest of SQLite. The overlay passes most memory allocation requests straight through to the underlying allocator and passes the results back up to the application. But the overlay can be set to cause the Nth memory allocation to fail. To run an OOM test, the overlay is first set to fail on the first allocation attempt. Then some test script is run and a verification that the allocation was correctly caught and handled is made. Then the overlay is set to fail on the second allocation and the test repeats. The failure point continues to advice one allocation at a time until the entire test procedure runs to completion without hitting a memory allocation error. This whole test sequence run twice. On the first pass, the overlay is set to fail only the Nth allocation. On the second pass, the overlay is set to fail the Nth and all subsequent allocations.</p> <p>Note that the memory leak detection logic continue to work even when the OOM overlay is being used. This verifies that SQLite does not leak memory even when it encounters memory allocation errors. Note also that the OOM overlay can work with any underlying memory allocator, including the instrumented memory allocator that checks for memory allocation misuse. In this way it is verified that OOM errors do not induce other kinds of memory usage errors.</p> <p>Finally, we observe that the instrumented memory allocator and the memory leak detector both work over the entire SQLite test suite and the test suite provides over 99% statement test coverage. This is strong evidence that dynamic memory allocation is used correctly everywhere within SQLite.</p> <a name="config"></a> <h2>3.0 Configuration</h2> <p>The default memory allocation settings in SQLite are appropriate for most applications. However, applications with unusual are particularly strict requirements may want to adjust the configuation to more closely align SQLite to their needs. Both compile-time and start-time configuration options are available.</p> <a name="altalloc"></a> <h3>3.1 Alternative low-level memory allocators</h3> <p>The SQLite source code includes several different memory allocation modules that can be selected at compile-time, or to a limited extent at start-time.</p> <a name="defaultalloc"></a> <h4>3.1.1 The default memory allocator</h4> <p>By default, SQLite uses the malloc(), realloc(), and free() routines from the standard C library for its memory allocation needs. These routines are surrounded by a thin wrapper that also provides a "memsize()" function that will return the size of an existing allocation. The memsize() function is needed to keep an accurate count of the number of bytes of outstanding memory; memsize() determines how many bytes to remove from the outstanding count when an allocation is freed. The default implementation implements memsize() by always allocating 8 extra bytes on each malloc() request and storing the size of the allocating in that 8-byte header.</p> <p>The default memory allocator is recommended for most applications. If you do not have a compelling need to use an alternative memory allocator, then use the default.</p> <a name="memdebug"></a> <h4>3.1.2 The debugging memory allocator</h4> <p>If SQLite is compiled with the [SQLITE_MEMDEBUG] compile-time option, then different heavy wrapper is used around system malloc(), realloc(), and free(). The heavy wrapper allocates around 100 bytes of extra space with each allocation. The extra space is used to places sentinal values at both ends of the allocation returned to the SQLite core. Upon freeing these sentinels are checked to make sure the SQLite core did not overrun the buffer in either direction. When the system library is GLIBC, the heavy wrapper also makes use of the GNU backtrace() function to examine the stack and record the ancestor functions of the malloc() call. When running the SQLite test suite, the heavy wrapper also record the name of the current test case. These latter two features are used to the SQLite developers to track down memory leaks detected by the test suite.</p> <p>The heavy wrapper that is used when [SQLITE_MEMDEBUG] is set also makes sure each new allocation is filled with nonsense data prior to returning the allocation to the caller. And as soon as an allocation is free, it is again filled with nonsense data. These two actions help to insure that the SQLite core does not make assumptions about the state of newly allocated memory and that memory allocations are not used after they have been freed.</p> <p>The heavy wrapper employed by [SQLITE_MEMDEBUG] is intended for use only during testing, analysis, and debugging of SQLite. The heavy wrapper has a significant performance and memory overhead and probably should not be used in production.</p> <a name="memsys5"></a> <h4>3.1.3 Zero-malloc memory allocator</h4> <p>When SQLite is compiled with the [SQLITE_ENABLE_MEMSYS5] option, an alternative memory allocator that does not use malloc() is included in the build. The SQLite developers refer to this alternative memory allocator as "memsys5". Even when it is included ina build, memsys5 is not enabled by default. To enable memsys5, the application must the following SQLite interface at start-time:</p> <blockquote><pre> [sqlite3_config]([SQLITE_CONFIG_HEAP], pBuf, szBuf, mnReq); </pre></blockquote> <p>In the call above, pBuf is a pointer to a large, contiguous chunk of memory space that SQLite will use to satisfy all of its memory allocation needs. pBuf might point to a static array or it might be memory obtained from some other operating-specific mechanism. szBuf is an integer which is the number of bytes of space the pBuf points to. mnReq is another integer which is the minimum size of an allocation. Any call to [sqlite3_malloc(N)] where N is less than mnReq will be rounded up to mnReq. mnReq must be a power of two.</p> <p>The memsys5 allocator is designed for use on embedded systems, though there is nothing to prevent its use on workstations if desired. The szBuf is typically between a few hundred kilobytes up to a few dozen megabytes, depending on system requirements and memory budget.</p> <p>The algorithm used by memsys5 is succinctly described as "power-of-two, first-fit". By this it is meant that the sizes of all memory allocation requests are rounded up to a power of two and that the request is satisfied by the first free slot in pBuf that is large enough. When used appropriately, this algorithm provides mathematical guarantees against fragmentation and breakdown, as described further <a href="#nofrag">below</a>.</p> <a name="memsysx"></a> <h4>3.1.4 Experimental memory allocators</h4> <p>The name "memsys5" used for the zero-malloc memory allocator implies that there are several additional memory allocators available, and indeed there are. The default memory allocator is "memsys1". The debugging memory allocator is "memsys2". Those have already been covered.</p> <p>If SQLite is compiled with [SQLITE_ENABLE_MEMSYS3] than another zero-malloc memory allocator, similar to memsys5, is included in the source tree. The memsys3 allocator, like memsys5, must be activated by a call to [sqlite3_config]([SQLITE_CONFIG_HEAP],...). Memsys3 uses the memory buffer supplied as its source for all memory allocations. The difference between memsys3 and memsys5 is that memsys3 uses a different memory allocation algorithm that seems to work well in practice, but which does not provide an mathematical guarantees against memory fragmentation and breakdown. Memsys3 was a predecessor to memsys5. We now believe that memsys5 is superior to memsys3 and that all applications that need a zero-malloc memory allocator should use memsys5 in preference to memsys3. Memsys3 is considered experiemental and may be removed from the source tree in a future release of SQLite.</p> <p>Code for memsys4 is still in the SQLite source tree (as of this writing - SQLite release 3.6.1), but it has not been maintained for several release cycles and probably does not work. Memsys4 was an attempt to use mmap() to obtain memory for use and then use madvise() to release unused pages back to the operating system so that they could be reused by other processes. The target platform for memsys4 was Google Android. The work on memsys4 has been abandoned and the memsys4 module will likely be removed from the source tree in the near future.</p> <p>Memsys6 uses system malloc() and free() to obtain the memory it needs. Memsys6 serves as an aggregator. Memsys6 only calls malloc() to obtain large allocations. It then services multiple smaller memory allocation requests from the SQLite core using each single large allocation obtained from system malloc(). Memsys6 is intended for use in systems where system malloc() is particularly inefficient. The idea behind memsys6 is to reduce the number of calls to system malloc() by a factor of 10 or more.</p> <p>Memsys6 is made available by compiling SQLite with the SQLITE_ENABLE_MEMSYS6 compile-time option and then at start-time invoking:</p> <blockquote><pre> [sqlite3_config]([SQLITE_CONFIG_CHUNKALLOC]); </pre></blockquote> <p>Memsys6 was added in SQLite version 3.6.1. It is very experimental. Its future is uncertain and it may be removed in a subsequent release.</p> <p>Other experimental memory allocators might be added in future releases of SQLite. One many anticipate that these will be called memsys7, memsys8, and so forth.</p> <a name="appalloc"></a> <h4>3.1.5 Application-defined memory allocators</h4> <p>New memory allocators do not have to be part of the SQLite source tree nor included in the sqlite3.c [amalgamation]. Individual applications can supply their own memory allocators to SQLite at start-time.</p> <p>To cause SQLite to use an new memory allocator, the application simply calls:</p> <blockquote><pre> [sqlite3_config]([SQLITE_CONFIG_MALLOC], pMem); </pre></blockquote> <p>In the call above, pMem is a pointer to an [sqlite3_mem_methods] object that defines the interface to the application-specific memory allocator. The [sqlite3_mem_methods] object is really just a structure containing pointers to functions to implement the various memory allocation primitives. </p> <p>In a multi-threaded application, access to the [sqlite3_mem_methods] is serialized if and only if [SQLITE_CONFIG_MEMSTATUS] is enabled. If [SQLITE_CONFIG_MEMSTATUS] is disabled then the methods in [sqlite3_mem_methods] must take care of their own serialization needs.</p> <a name="overlayalloc"></a> <h4>3.1.6 Memory allocator overlays</h4> <p>An application can insert layers or "overlays" in between the SQLite core and the underlying memory allocator. For example, the <a href="#oomtesting">out-of-memory test logic</a> for SQLite uses an overlay that can simulate memory allocation failures.</p> <p>An overlay can be created by using the</p> <blockquote><pre> [sqlite3_config]([SQLITE_CONFIG_GETMALLOC], pOldMem); </pre></blockquote> <p>interface to obtain pointers to the existing memory allocator. The existing allocator is saved by the overlay and is used as a fallback to do real memory allocation. Then the overlay is insert in place of the existing memory allocator using the [sqlite3_config]([SQLITE_CONFIG_MALLOC],...) as described <a href="#appalloc">above</a>. <a name="scratch"></a> <h3>3.2 Scratch memory</h3> <p>SQLite occasionally needs a large chunk of "scratch" memory to perform some transient calculation. Scratch memory is used, for example, as temporary storage when rebalancing a btree. These scratch memory allocations are typically about 10 kilobytes in size and by definition last only for the duration of a single function call.</p> <p>In older versions of SQLite, the scratch memory was obtained from the process stack. This works great on workstations that have a large stack. But that approach caused problems on embedded systems with a small processor stack (typically 4K or 8K). And so SQLite was modified to allocate scratch memory from the heap.</p> <p>But doing occasional large, transient allocations from the heap can lead to memory fragmentation in embedded systems. To work around this problem, a separate memory allocation system for scratch memory has been created.</p> <p>The scratch memory allocator is set up as follows:</p> <blockquote><pre> [sqlite3_config]([SQLITE_CONFIG_SCRATCH], pBuf, sz, N); </pre></blockquote> <p>The pBuf parameter is a pointer to a contiguous range of bytes that SQLite will use for all scratch memory allocations. The buffer must be at least sz*N bytes in size. The "sz" parameter is the maximum size of each scratch allocation. N is the maximum number of simulataneous scratch allocations. The "sz" parameter should be approximately 6 times the maximum database page size. N should be the number of threads running in the system. No single thread will ever request more than one scratch allocation at a time so if there are never more than N threads, then there will always be enough scratch memory available.</p> <p>If the scratch memory setup does not define enough memory, then SQLite falls back to using the regular memory allocator for its scratch memory allocations. The default setup is sz=0 and N=0 so the use of the regular memory allocator is the default behavior.</p> <a name="pagecache"></a> <h3>3.3 Page cache memory</h3> <p>The database page cache subsystem within SQLite uses more dynamically allocated memory than any other part of SQLite. In fact, the database page cache usually consumes about 10 times more memory than all other parts of SQLite combined. <a name="lookaside"></a> <h3>3.4 Lookaside memory allocator</h3> <a name="memstatus"></a> <h3>3.5 Memory status</h3> <a name="heaplimit"></a> <h3>3.6 Setting memory usage limits</h3> <a name="nofrag"></a> <h2>4.0 Anti-fragmentation Guarantee</h2> <a name="summary"></a> <h2>5.0 Summary Of Memory Allocator Interfaces</h2> |
Changes to pages/news.in.
︙ | ︙ | |||
9 10 11 12 13 14 15 16 17 18 19 20 21 22 | hd_puts "<h3>$date - $title</h3>" regsub -all "\n( *\n)+" $text "</p>\n\n<p>" txt regsub -all {[Tt]icket #(\d+)} $txt \ {<a href="http://www.sqlite.org/cvstrac/tktview?tn=\1">\0</a>} txt hd_resolve "<p>$txt</p>" hd_puts "<hr width=\"50%\">" } newsitem {2008-July-16} {Version 3.6.0 beta} { Version 3.6.0 makes changes to the [sqlite3_vfs | VFS] object in order to make SQLite more easily portable to a wider variety of platforms. There are potential incompatibilities with some legacy applications. See the [35to36 | 35to36.html] document for details. | > > > > > | 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | hd_puts "<h3>$date - $title</h3>" regsub -all "\n( *\n)+" $text "</p>\n\n<p>" txt regsub -all {[Tt]icket #(\d+)} $txt \ {<a href="http://www.sqlite.org/cvstrac/tktview?tn=\1">\0</a>} txt hd_resolve "<p>$txt</p>" hd_puts "<hr width=\"50%\">" } newsitem {2008-Aug-06} {Version 3.6.1} { SQLite version 3.6.1 is a stabilization and performance enhancement release. } newsitem {2008-July-16} {Version 3.6.0 beta} { Version 3.6.0 makes changes to the [sqlite3_vfs | VFS] object in order to make SQLite more easily portable to a wider variety of platforms. There are potential incompatibilities with some legacy applications. See the [35to36 | 35to36.html] document for details. |
︙ | ︙ |
Changes to wrap.tcl.
︙ | ︙ | |||
168 169 170 171 172 173 174 | # Write raw output to both the main file and the auxiliary. Only write # to files that are enabled. # proc hd_puts {text} { global hd if {$hd(enable-main)} { | | | | 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 | # Write raw output to both the main file and the auxiliary. Only write # to files that are enabled. # proc hd_puts {text} { global hd if {$hd(enable-main)} { puts -nonewline $hd(main) $text } if {$hd(enable-aux)} { puts -nonewline $hd(aux) $text } } # Enable or disable the main output file. # proc hd_enable_main {boolean} { global hd |
︙ | ︙ |