Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
Comment: | Remove the scratch memory allocator. |
---|---|
Downloads: | Tarball | ZIP archive |
Timelines: | family | ancestors | descendants | both | trunk |
Files: | files | file ages | folders |
SHA3-256: |
d482cfb5eac713f8d20a364b5ccf2c7e |
User & Date: | drh 2017-08-29 13:19:37.636 |
Context
2017-08-30
| ||
04:46 | Update the CPU usage improvement for 3.21.0. (check-in: 4ea4e27dc5 user: drh tags: trunk) | |
2017-08-29
| ||
13:19 | Remove the scratch memory allocator. (check-in: d482cfb5ea user: drh tags: trunk) | |
2017-08-25
| ||
17:54 | Update the change log for 3.21.0. (check-in: 2d9f1292ff user: drh tags: trunk) | |
Changes
Changes to pages/changes.in.
︙ | ︙ | |||
42 43 44 45 46 47 48 49 50 51 52 53 54 55 | <li> Fewer "stat()" system calls issued by the unix VFS. <li> Query planner enhancements: <ol type="a"> <li> Enhanced the [LIKE optimization] so that it works with an ESCAPE clause. </ol> <li> Enhanced the [CSV virtual table] so that it accepts the last row of input if the final new-line character is missing. <li> Added the swarm virtual table to the existing union virtual table extension. <li> Miscellaneous [microoptimizations] reduce CPU usage by about 1.6%. <li> Bug fixes: <ol type="a"> <li> Fix a faulty assert() statement discovered by OSSFuzz. Ticket [https://sqlite.org/src/info/cb91bf4290c211d|cb91bf4290c211d] <li> Fix an obscure memory leak in [sqlite3_result_pointer()]. | > > > | 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 | <li> Fewer "stat()" system calls issued by the unix VFS. <li> Query planner enhancements: <ol type="a"> <li> Enhanced the [LIKE optimization] so that it works with an ESCAPE clause. </ol> <li> Enhanced the [CSV virtual table] so that it accepts the last row of input if the final new-line character is missing. <li> Remove the rarely-used "scratch" memory allocator. Replace it with the [SQLITE_CONFIG_SMALL_MALLOC] configuration setting that gives SQLite a hint large memory allocations should be avoided when possible. <li> Added the swarm virtual table to the existing union virtual table extension. <li> Miscellaneous [microoptimizations] reduce CPU usage by about 1.6%. <li> Bug fixes: <ol type="a"> <li> Fix a faulty assert() statement discovered by OSSFuzz. Ticket [https://sqlite.org/src/info/cb91bf4290c211d|cb91bf4290c211d] <li> Fix an obscure memory leak in [sqlite3_result_pointer()]. |
︙ | ︙ |
Changes to pages/malloc.in.
︙ | ︙ | |||
449 450 451 452 453 454 455 | on systems that may not have malloc(), free(), or realloc() in their standard library. An application that is compiled with [SQLITE_ZERO_MALLOC] will need to use [sqlite3_config()] together with [SQLITE_CONFIG_MALLOC] or [SQLITE_CONFIG_HEAP] to specify a new alternative memory allocator before beginning to use SQLite.</p> | < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < | 449 450 451 452 453 454 455 456 457 458 459 460 461 462 | on systems that may not have malloc(), free(), or realloc() in their standard library. An application that is compiled with [SQLITE_ZERO_MALLOC] will need to use [sqlite3_config()] together with [SQLITE_CONFIG_MALLOC] or [SQLITE_CONFIG_HEAP] to specify a new alternative memory allocator before beginning to use SQLite.</p> <tcl>hd_fragment pagecache {pagecache memory allocator}</tcl> <h2> Page cache memory</h2> <p>In most applications, the database page cache subsystem within SQLite uses more dynamically allocated memory than all other parts of SQLite combined. It is not unusual to see the database page cache consume over 10 times more memory than the rest of SQLite combined.</p> |
︙ | ︙ | |||
566 567 568 569 570 571 572 | <p>SQLite [version 3.6.1] ([dateof:3.6.1]) introduced the lookaside memory allocator to help reduce the memory allocation load. In the lookaside allocator, each [database connection] preallocates a single large chunk of memory (typically in the range of 60 to 120 kilobytes) and divides that chunk up into small fixed-size "slots" of around 100 to 1000 byte each. This becomes the lookaside memory pool. Thereafter, memory allocations | | | 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 | <p>SQLite [version 3.6.1] ([dateof:3.6.1]) introduced the lookaside memory allocator to help reduce the memory allocation load. In the lookaside allocator, each [database connection] preallocates a single large chunk of memory (typically in the range of 60 to 120 kilobytes) and divides that chunk up into small fixed-size "slots" of around 100 to 1000 byte each. This becomes the lookaside memory pool. Thereafter, memory allocations associated with the [database connection] and that are not too large are satisfied using one of the lookaside pool slots rather than by calling the general-purpose memory allocator. Larger allocations continue to use the general-purpose memory allocator, as do allocations that occur when the lookaside pool slots are all checked out. But in many cases, the memory allocations are small enough and there are few enough outstanding that the new memory requests can be satisfied from the lookaside |
︙ | ︙ | |||
719 720 721 722 723 724 725 | ahead and allocates the extra memory and exceeds its limit. This occurs under the theory that it is better to use additional memory than to fail outright.</p> <p>As of SQLite [version 3.6.1] ([dateof:3.6.1]), the soft heap limit only applies to the general-purpose memory allocator. The soft heap limit does not know | | | | 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 | ahead and allocates the extra memory and exceeds its limit. This occurs under the theory that it is better to use additional memory than to fail outright.</p> <p>As of SQLite [version 3.6.1] ([dateof:3.6.1]), the soft heap limit only applies to the general-purpose memory allocator. The soft heap limit does not know about or interact with the [pagecache memory allocator] or the [lookaside memory allocator]. This deficiency will likely be addressed in a future release.</p> <tcl>hd_fragment nofrag {Robson proof}</tcl> <h1> Mathematical Guarantees Against Memory Allocation Failures</h1> <p>The problem of dynamic memory allocation, and specifically the problem of a memory allocator breakdown, has been studied by |
︙ | ︙ | |||
811 812 813 814 815 816 817 | <h2> Computing and controlling parameters <b>M</b> and <b>n</b></h2> <p>The Robson proof applies separately to each of the memory allocators used by SQLite:</p> <ul> <li>The general-purpose memory allocator ([memsys5]).</li> | < < < < < < < < < | 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 | <h2> Computing and controlling parameters <b>M</b> and <b>n</b></h2> <p>The Robson proof applies separately to each of the memory allocators used by SQLite:</p> <ul> <li>The general-purpose memory allocator ([memsys5]).</li> <li>The [pagecache memory allocator].</li> <li>The [lookaside memory allocator].</li> </ul> <p>For allocators other than [memsys5], all memory allocations are of the same size. Hence, <b>n</b>=1 and therefore <b>N</b>=<b>M</b>. In other words, the memory pool need be no larger than the largest amount of memory in use at any given moment.</p> <p>The usage of pagecache memory is somewhat harder to control in SQLite version 3.6.1, though mechanisms are planned for subsequent releases that will make controlling pagecache memory much easier. Prior to the introduction of these new mechanisms, the only way to control pagecache memory is using the [cache_size pragma].</p> <p>Safety-critical applications will usually want to modify the |
︙ | ︙ | |||
893 894 895 896 897 898 899 | might come from several sources:</p> <ol> <li>SQL table rows that contain large strings or BLOBs.</li> <li>Complex SQL queries that compile down to large [prepared statements].</li> <li>SQL parser objects used internally by [sqlite3_prepare_v2()].</li> <li>Storage space for [database connection] objects.</li> | < < | | | 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 | might come from several sources:</p> <ol> <li>SQL table rows that contain large strings or BLOBs.</li> <li>Complex SQL queries that compile down to large [prepared statements].</li> <li>SQL parser objects used internally by [sqlite3_prepare_v2()].</li> <li>Storage space for [database connection] objects.</li> <li>Page cache memory allocations that overflow into the general-purpose memory allocator.</li> <li>Lookaside buffer allocations for new [database connections].</li> </ol> <p>The last two allocations can be controlled and/or eliminated by configuring the [pagecache memory allocator], and [lookaside memory allocator] appropriately, as described above. The storage space required for [database connection] objects depends to some extent on the length of the filename of the database file, but rarely exceeds 2KB on 32-bit systems. (More space is required on 64-bit systems due to the increased size of pointers.) Each parser object uses about 1.6KB of memory. Thus, elements 3 through 7 above can easily be controlled to keep the maximum memory allocation |
︙ | ︙ | |||
951 952 953 954 955 956 957 | <h2> Ductile failure</h2> <p>If the memory allocation subsystems within SQLite are configured for breakdown-free operation but the actual memory usage exceeds design limits set by the [Robson proof], SQLite will usually continue to operate normally. | | | | 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 | <h2> Ductile failure</h2> <p>If the memory allocation subsystems within SQLite are configured for breakdown-free operation but the actual memory usage exceeds design limits set by the [Robson proof], SQLite will usually continue to operate normally. The the [pagecache memory allocator] and the [lookaside memory allocator] automatically failover to the [memsys5] general-purpose memory allocator. And it is usually the case that the [memsys5] memory allocator will continue to function without fragmentation even if <b>M</b> and/or <b>n</b> exceeds the limits imposed by the [Robson proof]. The [Robson proof] shows that it is possible for a memory allocation to break down and fail in this circumstance, but such a failure requires an especially despicable sequence of allocations and deallocations - a sequence that |
︙ | ︙ |