Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Changes In Branch sahpool-digest Excluding Merge-Ins
This is equivalent to a diff from f786de8d18 to 500f2e6ec7
2025-03-16
| ||
14:05 | Fix a long-standing filename digest computation bug in the OPFS SAHPool VFS which caused all VFS-stored filenames to have a digest value of 0. See /forumpost/042d53c928382021 for full details. (check-in: 493cbe7450 user: stephan tags: trunk) | |
2025-03-14
| ||
12:37 | Fix an internal doc typo reported in forum post e25e581f917. (check-in: fa6f6ccdff user: stephan tags: trunk) | |
11:14 | Minor doc corrections for the sahpool-digest fix and merge in current trunk. (Closed-Leaf check-in: 500f2e6ec7 user: stephan tags: sahpool-digest) | |
09:34 | Cherrypick the [2b582c0097e33] doc addition, which was initially committed to the wrong branch. (check-in: f786de8d18 user: stephan tags: trunk) | |
2025-03-13
| ||
18:51 | Fix the generate_series extension for the case where the termination value is not an even multiple of the step from the start value and there is also a value=NNN constraint in the WHERE clause. Forum post bf2dc8e9 (check-in: 75e72e3b0d user: drh tags: trunk) | |
2025-03-12
| ||
11:41 | Document that sqlite_update_hook() can unset the current hook by passing a NULL callback, to address forum post 652aef4747. (check-in: 2b582c0097 user: stephan tags: cygwin-fixes) | |
2025-02-26
| ||
03:32 | Merge trunk into the sahpool-digest branch. (check-in: fc1eeb7d1f user: stephan tags: sahpool-digest) | |
Changes to ext/wasm/api/sqlite3-vfs-opfs-sahpool.c-pp.js.
︙ | ︙ | |||
75 76 77 78 79 80 81 82 83 84 85 86 87 88 | left in OPFS but are treated as transient by this VFS and they will be cleaned up during VFS init. */ const PERSISTENT_FILE_TYPES = capi.SQLITE_OPEN_MAIN_DB | capi.SQLITE_OPEN_MAIN_JOURNAL | capi.SQLITE_OPEN_SUPER_JOURNAL | capi.SQLITE_OPEN_WAL; /** Subdirectory of the VFS's space where "opaque" (randomly-named) files are stored. Changing this effectively invalidates the data stored under older names (orphaning it), so don't do that. */ const OPAQUE_DIR_NAME = ".opaque"; /** | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 | left in OPFS but are treated as transient by this VFS and they will be cleaned up during VFS init. */ const PERSISTENT_FILE_TYPES = capi.SQLITE_OPEN_MAIN_DB | capi.SQLITE_OPEN_MAIN_JOURNAL | capi.SQLITE_OPEN_SUPER_JOURNAL | capi.SQLITE_OPEN_WAL; const FLAG_COMPUTE_DIGEST_V2 = capi.SQLITE_OPEN_MEMORY /* Part of the fix for https://github.com/sqlite/sqlite-wasm/issues/97 Summary: prior to version 3.50.0 computeDigest() always computes a value of [0,0] due to overflows, so it does not do anything useful. Fixing it invalidates old persistent files, so we instead only fix it for files created or updated since the bug was discovered and fixed. This flag determines whether we use the broken legacy computeDigest() or the v2 variant. We only use this flag for newly-created/overwritten files. Pre-existing files have the broken digest stored in them so need to continue to use that. What this means, in terms of db file compatibility between versions: - DBs created with versions older than this fix (<3.50.0) can be read by post-fix versions. Such DBs which are written to in-place (not replaced) by newer versions can still be read by older versions, as the affected digest is only modified when the SAH slot is assigned to a given filename. - DBs created with post-fix versions will, when read by a pre-fix version, be seen as having a "bad digest" and will be unceremoniously replaced by that pre-fix version. When swapping back to a post-fix version, that version will see that the file entry is missing the FLAG_COMPUTE_DIGEST_V2 bit so will treat it as a legacy file. This flag is stored in the same memory as the various SQLITE_OPEN_... flags and we must be careful here to not use a flag bit which is otherwise relevant for the VFS. SQLITE_OPEN_MEMORY is handled by sqlite3_open_v2() and friends, not the VFS, so we'll repurpose that one. If we take a currently-unused bit and it ends up, at some later point, being used, we would have to invalidate existing VFS files in order to move to another bit. Similarly, if the SQLITE_OPEN_MEMORY bit were ever reassigned (which it won't be!), we'd invalidate all VFS-side files. */; /** Subdirectory of the VFS's space where "opaque" (randomly-named) files are stored. Changing this effectively invalidates the data stored under older names (orphaning it), so don't do that. */ const OPAQUE_DIR_NAME = ".opaque"; /** |
︙ | ︙ | |||
325 326 327 328 329 330 331 332 333 334 335 336 337 338 | } return e ? (e.sqlite3Rc || capi.SQLITE_IOERR) : 0; }, //xSleep is optionally defined below xOpen: function f(pVfs, zName, pFile, flags, pOutFlags){ const pool = getPoolForVfs(pVfs); try{ pool.log(`xOpen ${wasm.cstrToJs(zName)} ${flags}`); // First try to open a path that already exists in the file system. const path = (zName && wasm.peek8(zName)) ? pool.getPath(zName) : getRandomName(); let sah = pool.getSAHForPath(path); if(!sah && (flags & capi.SQLITE_OPEN_CREATE)) { | > | 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 | } return e ? (e.sqlite3Rc || capi.SQLITE_IOERR) : 0; }, //xSleep is optionally defined below xOpen: function f(pVfs, zName, pFile, flags, pOutFlags){ const pool = getPoolForVfs(pVfs); try{ flags &= ~FLAG_COMPUTE_DIGEST_V2; pool.log(`xOpen ${wasm.cstrToJs(zName)} ${flags}`); // First try to open a path that already exists in the file system. const path = (zName && wasm.peek8(zName)) ? pool.getPath(zName) : getRandomName(); let sah = pool.getSAHForPath(path); if(!sah && (flags & capi.SQLITE_OPEN_CREATE)) { |
︙ | ︙ | |||
620 621 622 623 624 625 626 | this.#apBody); this.setAssociatedPath(sah, '', 0); return ''; } const fileDigest = new Uint32Array(HEADER_DIGEST_SIZE / 4); sah.read(fileDigest, {at: HEADER_OFFSET_DIGEST}); | | > | 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 | this.#apBody); this.setAssociatedPath(sah, '', 0); return ''; } const fileDigest = new Uint32Array(HEADER_DIGEST_SIZE / 4); sah.read(fileDigest, {at: HEADER_OFFSET_DIGEST}); const compDigest = this.computeDigest(this.#apBody, flags); //warn("getAssociatedPath() flags",'0x'+flags.toString(16), "compDigest", compDigest); if(fileDigest.every((v,i) => v===compDigest[i])){ // Valid digest const pathBytes = this.#apBody.findIndex((v)=>0===v); if(0===pathBytes){ // This file is unassociated, so truncate it to avoid // leaving stale db data laying around. sah.truncate(HEADER_OFFSET_DATA); |
︙ | ︙ | |||
651 652 653 654 655 656 657 658 659 | in the metadata. */ setAssociatedPath(sah, path, flags){ const enc = textEncoder.encodeInto(path, this.#apBody); if(HEADER_MAX_PATH_SIZE <= enc.written + 1/*NUL byte*/){ toss("Path too long:",path); } this.#apBody.fill(0, enc.written, HEADER_MAX_PATH_SIZE); this.#dvBody.setUint32(HEADER_OFFSET_FLAGS, flags); | > > > > > > > < | > > > | > | | | | | | | > > > > | 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 | in the metadata. */ setAssociatedPath(sah, path, flags){ const enc = textEncoder.encodeInto(path, this.#apBody); if(HEADER_MAX_PATH_SIZE <= enc.written + 1/*NUL byte*/){ toss("Path too long:",path); } if(path && flags){ /* When creating or re-writing files, update their digest, if needed, to v2. We continue to use v1 for the (!path) case (empty files) because there's little reason not to use a digest of 0 for empty entries. */ flags |= FLAG_COMPUTE_DIGEST_V2; } this.#apBody.fill(0, enc.written, HEADER_MAX_PATH_SIZE); this.#dvBody.setUint32(HEADER_OFFSET_FLAGS, flags); const digest = this.computeDigest(this.#apBody, flags); //console.warn("setAssociatedPath(",path,") digest",digest); sah.write(this.#apBody, {at: 0}); sah.write(digest, {at: HEADER_OFFSET_DIGEST}); sah.flush(); if(path){ this.#mapFilenameToSAH.set(path, sah); this.#availableSAH.delete(sah); }else{ // This is not a persistent file, so eliminate the contents. sah.truncate(HEADER_OFFSET_DATA); this.#availableSAH.add(sah); } } /** Computes a digest for the given byte array and returns it as a two-element Uint32Array. This digest gets stored in the metadata for each file as a validation check. Changing this algorithm invalidates all existing databases for this VFS, so don't do that. See the docs for FLAG_COMPUTE_DIGEST_V2 for more details. */ computeDigest(byteArray, fileFlags){ if( fileFlags & FLAG_COMPUTE_DIGEST_V2 ){ let h1 = 0xdeadbeef; let h2 = 0x41c6ce57; for(const v of byteArray){ h1 = Math.imul(h1 ^ v, 2654435761); h2 = Math.imul(h2 ^ v, 104729); } return new Uint32Array([h1>>>0, h2>>>0]); }else{ /* this is what the buggy legacy computation worked out to */ return new Uint32Array([0,0]); } } /** Re-initializes the state of the SAH pool, releasing and re-acquiring all handles. See acquireAccessHandles() for the specifics of the clearFiles |
︙ | ︙ |
Changes to ext/wasm/tester1.c-pp.js.
︙ | ︙ | |||
3502 3503 3504 3505 3506 3507 3508 | ++nAuxReused; } return x; } }); db.exec([ "create table t(a);", | | | 3502 3503 3504 3505 3506 3507 3508 3509 3510 3511 3512 3513 3514 3515 3516 | ++nAuxReused; } return x; } }); db.exec([ "create table t(a);", "insert into t(a) values(1),(2),(1);", "select auxtest(1,a), auxtest(1,a) from t order by a" ]); }finally{ db.close(); wasm.pstack.restore(stack); } T.assert(nAuxSet>0).assert(nAuxReused>0) |
︙ | ︙ |
Added ext/wasm/tests/opfs/sahpool/digest-worker.js.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 | /* 2025-01-31 The author disclaims copyright to this source code. In place of a legal notice, here is a blessing: * May you do good and not evil. * May you find forgiveness for yourself and forgive others. * May you share freely, never taking more than you give. *********************************************************************** This file is part of sahpool-pausing.js's demonstration of the pause/unpause feature of the opfs-sahpool VFS. */ const clog = console.log.bind(console); const wPost = (type,...args)=>postMessage({type, payload:args}); const log = (...args)=>{ clog("Worker:",...args); wPost('log',...args); } const hasOpfs = ()=>{ return globalThis.FileSystemHandle && globalThis.FileSystemDirectoryHandle && globalThis.FileSystemFileHandle && globalThis.FileSystemFileHandle.prototype.createSyncAccessHandle && navigator?.storage?.getDirectory; }; if( !hasOpfs() ){ wPost('error',"OPFS not detected"); throw new Error("OPFS not detected"); } clog("Importing sqlite3..."); const searchParams = new URL(self.location.href).searchParams; importScripts(searchParams.get('sqlite3.dir') + '/sqlite3.js'); const runTests = function(sqlite3, poolUtil){ const fname = '/my.db'; let db = new poolUtil.OpfsSAHPoolDb(fname); let n = (new Date()).valueOf(); try { db.exec([ "create table if not exists t(a);" ]); db.exec({ sql: "insert into t(a) values(?)", bind: n++ }); log(fname,"record count: ",db.selectValue("select count(*) from t")); }finally{ db.close(); } db = new poolUtil.OpfsSAHPoolDb(fname); try { db.exec({ sql: "insert into t(a) values(?)", bind: n++ }); log(fname,"record count: ",db.selectValue("select count(*) from t")); }finally{ db.close(); } const fname2 = '/my2.db'; db = new poolUtil.OpfsSAHPoolDb(fname2); try { db.exec([ "create table if not exists t(a);" ]); db.exec({ sql: "insert into t(a) values(?)", bind: n++ }); log(fname2,"record count: ",db.selectValue("select count(*) from t")); }finally{ db.close(); } }; globalThis.sqlite3InitModule().then(async function(sqlite3){ log("sqlite3 version:",sqlite3.version); const sahPoolConfig = { name: 'opfs-sahpool-digest', clearOnInit: false, initialCapacity: 6 }; return sqlite3.installOpfsSAHPoolVfs(sahPoolConfig).then(poolUtil=>{ log('vfs acquired'); runTests(sqlite3, poolUtil); }); }); |
Added ext/wasm/tests/opfs/sahpool/digest.html.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 | <!doctype html> <html lang="en-us"> <head> <meta charset="utf-8"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <link rel="shortcut icon" href="data:image/x-icon;," type="image/x-icon"> <link rel="stylesheet" href="../../../common/emscripten.css"/> <link rel="stylesheet" href="../../../common/testing.css"/> <title>sqlite3 tester: OpfsSAHPool Digest</title> <style></style> </head> <body><h1 id='color-target'></h1> <p> This is a test app for the digest calculation of the OPFS SAHPool VFS. It requires running it with a new database created using v3.49.0 or older, then running it again with a newer version, then again with 3.49.0 or older. </p> <div class='input-wrapper'> <input type='checkbox' id='cb-log-reverse'> <label for='cb-log-reverse'>Reverse log order?</label> </div> <div id='test-output'></div> <script> /* 2025-02-03 The author disclaims copyright to this source code. In place of a legal notice, here is a blessing: * May you do good and not evil. * May you find forgiveness for yourself and forgive others. * May you share freely, never taking more than you give. *********************************************************************** This is a bugfix test for the OPFS SAHPool VFS. It requires setting up a database created using v3.49.0 or older, then running it again with a newer version. */ (function(){ 'use strict'; document.querySelector('h1').innerHTML = document.querySelector('title').innerHTML; const mapToString = (v)=>{ switch(typeof v){ case 'number': case 'string': case 'boolean': case 'undefined': case 'bigint': return ''+v; default: break; } if(null===v) return 'null'; if(v instanceof Error){ v = { message: v.message, stack: v.stack, errorClass: v.name }; } return JSON.stringify(v,undefined,2); }; const normalizeArgs = (args)=>args.map(mapToString); const logTarget = document.querySelector('#test-output'); const logClass = function(cssClass,...args){ const ln = document.createElement('div'); if(cssClass){ for(const c of (Array.isArray(cssClass) ? cssClass : [cssClass])){ ln.classList.add(c); } } ln.append(document.createTextNode(normalizeArgs(args).join(' '))); logTarget.append(ln); }; const cbReverse = document.querySelector('#cb-log-reverse'); //cbReverse.setAttribute('checked','checked'); const cbReverseKey = 'tester1:cb-log-reverse'; const cbReverseIt = ()=>{ logTarget.classList[cbReverse.checked ? 'add' : 'remove']('reverse'); //localStorage.setItem(cbReverseKey, cbReverse.checked ? 1 : 0); }; cbReverse.addEventListener('change', cbReverseIt, true); /*if(localStorage.getItem(cbReverseKey)){ cbReverse.checked = !!(+localStorage.getItem(cbReverseKey)); }*/ cbReverseIt(); const log = (...args)=>{ //console.log(...args); logClass('',...args); } const warn = (...args)=>{ console.warn(...args); logClass('warning',...args); } const error = (...args)=>{ console.error(...args); logClass('error',...args); }; const toss = (...args)=>{ error(...args); throw new Error(args.join(' ')); }; const endOfWork = (passed=true)=>{ const eH = document.querySelector('#color-target'); const eT = document.querySelector('title'); if(passed){ log("End of work chain. If you made it this far, you win."); eH.innerText = 'PASS: '+eH.innerText; eH.classList.add('tests-pass'); eT.innerText = 'PASS: '+eT.innerText; }else{ eH.innerText = 'FAIL: '+eH.innerText; eH.classList.add('tests-fail'); eT.innerText = 'FAIL: '+eT.innerText; } }; log("Running opfs-sahpool digest tests..."); const W1 = new Worker('digest-worker.js?sqlite3.dir=../../../jswasm'); W1.onmessage = function({data}){ //log("onmessage:",data); switch(data.type){ case 'log': log('worker says:', ...data.payload); break; case 'error': error('worker says:', ...data.payload); endOfWork(false); break; case 'initialized': log(data.workerId, ': Worker initialized',...data.payload); break; } }; })(); </script> </body> </html> |