SQLite

Changes On Branch sahpool-digest
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Changes In Branch sahpool-digest Excluding Merge-Ins

This is equivalent to a diff from f786de8d18 to 500f2e6ec7

2025-03-16
14:05
Fix a long-standing filename digest computation bug in the OPFS SAHPool VFS which caused all VFS-stored filenames to have a digest value of 0. See /forumpost/042d53c928382021 for full details. (check-in: 493cbe7450 user: stephan tags: trunk)
2025-03-14
12:37
Fix an internal doc typo reported in forum post e25e581f917. (check-in: fa6f6ccdff user: stephan tags: trunk)
11:14
Minor doc corrections for the sahpool-digest fix and merge in current trunk. (Closed-Leaf check-in: 500f2e6ec7 user: stephan tags: sahpool-digest)
09:34
Cherrypick the [2b582c0097e33] doc addition, which was initially committed to the wrong branch. (check-in: f786de8d18 user: stephan tags: trunk)
2025-03-13
18:51
Fix the generate_series extension for the case where the termination value is not an even multiple of the step from the start value and there is also a value=NNN constraint in the WHERE clause. Forum post bf2dc8e9 (check-in: 75e72e3b0d user: drh tags: trunk)
2025-03-12
11:41
Document that sqlite_update_hook() can unset the current hook by passing a NULL callback, to address forum post 652aef4747. (check-in: 2b582c0097 user: stephan tags: cygwin-fixes)
2025-02-26
03:32
Merge trunk into the sahpool-digest branch. (check-in: fc1eeb7d1f user: stephan tags: sahpool-digest)

Changes to ext/wasm/api/sqlite3-vfs-opfs-sahpool.c-pp.js.
75
76
77
78
79
80
81










































82
83
84
85
86
87
88
     left in OPFS but are treated as transient by this VFS and
     they will be cleaned up during VFS init. */
  const PERSISTENT_FILE_TYPES =
        capi.SQLITE_OPEN_MAIN_DB |
        capi.SQLITE_OPEN_MAIN_JOURNAL |
        capi.SQLITE_OPEN_SUPER_JOURNAL |
        capi.SQLITE_OPEN_WAL;











































  /** Subdirectory of the VFS's space where "opaque" (randomly-named)
      files are stored. Changing this effectively invalidates the data
      stored under older names (orphaning it), so don't do that. */
  const OPAQUE_DIR_NAME = ".opaque";

  /**







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
     left in OPFS but are treated as transient by this VFS and
     they will be cleaned up during VFS init. */
  const PERSISTENT_FILE_TYPES =
        capi.SQLITE_OPEN_MAIN_DB |
        capi.SQLITE_OPEN_MAIN_JOURNAL |
        capi.SQLITE_OPEN_SUPER_JOURNAL |
        capi.SQLITE_OPEN_WAL;
  const FLAG_COMPUTE_DIGEST_V2 = capi.SQLITE_OPEN_MEMORY
  /* Part of the fix for
     https://github.com/sqlite/sqlite-wasm/issues/97

     Summary: prior to version 3.50.0 computeDigest() always computes
     a value of [0,0] due to overflows, so it does not do anything
     useful.  Fixing it invalidates old persistent files, so we
     instead only fix it for files created or updated since the bug
     was discovered and fixed.

     This flag determines whether we use the broken legacy
     computeDigest() or the v2 variant. We only use this flag for
     newly-created/overwritten files. Pre-existing files have the
     broken digest stored in them so need to continue to use that.

     What this means, in terms of db file compatibility between
     versions:

     - DBs created with versions older than this fix (<3.50.0)
     can be read by post-fix versions. Such DBs which are written
     to in-place (not replaced) by newer versions can still be read
     by older versions, as the affected digest is only modified
     when the SAH slot is assigned to a given filename.

     - DBs created with post-fix versions will, when read by a pre-fix
     version, be seen as having a "bad digest" and will be
     unceremoniously replaced by that pre-fix version. When swapping
     back to a post-fix version, that version will see that the file
     entry is missing the FLAG_COMPUTE_DIGEST_V2 bit so will treat it
     as a legacy file.

     This flag is stored in the same memory as the various
     SQLITE_OPEN_... flags and we must be careful here to not use a
     flag bit which is otherwise relevant for the VFS.
     SQLITE_OPEN_MEMORY is handled by sqlite3_open_v2() and friends,
     not the VFS, so we'll repurpose that one.  If we take a
     currently-unused bit and it ends up, at some later point, being
     used, we would have to invalidate existing VFS files in order to
     move to another bit.  Similarly, if the SQLITE_OPEN_MEMORY bit
     were ever reassigned (which it won't be!), we'd invalidate all
     VFS-side files.
  */;

  /** Subdirectory of the VFS's space where "opaque" (randomly-named)
      files are stored. Changing this effectively invalidates the data
      stored under older names (orphaning it), so don't do that. */
  const OPAQUE_DIR_NAME = ".opaque";

  /**
325
326
327
328
329
330
331

332
333
334
335
336
337
338
      }
      return e ? (e.sqlite3Rc || capi.SQLITE_IOERR) : 0;
    },
    //xSleep is optionally defined below
    xOpen: function f(pVfs, zName, pFile, flags, pOutFlags){
      const pool = getPoolForVfs(pVfs);
      try{

        pool.log(`xOpen ${wasm.cstrToJs(zName)} ${flags}`);
        // First try to open a path that already exists in the file system.
        const path = (zName && wasm.peek8(zName))
              ? pool.getPath(zName)
              : getRandomName();
        let sah = pool.getSAHForPath(path);
        if(!sah && (flags & capi.SQLITE_OPEN_CREATE)) {







>







367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
      }
      return e ? (e.sqlite3Rc || capi.SQLITE_IOERR) : 0;
    },
    //xSleep is optionally defined below
    xOpen: function f(pVfs, zName, pFile, flags, pOutFlags){
      const pool = getPoolForVfs(pVfs);
      try{
        flags &= ~FLAG_COMPUTE_DIGEST_V2;
        pool.log(`xOpen ${wasm.cstrToJs(zName)} ${flags}`);
        // First try to open a path that already exists in the file system.
        const path = (zName && wasm.peek8(zName))
              ? pool.getPath(zName)
              : getRandomName();
        let sah = pool.getSAHForPath(path);
        if(!sah && (flags & capi.SQLITE_OPEN_CREATE)) {
620
621
622
623
624
625
626
627

628
629
630
631
632
633
634
             this.#apBody);
        this.setAssociatedPath(sah, '', 0);
        return '';
      }

      const fileDigest = new Uint32Array(HEADER_DIGEST_SIZE / 4);
      sah.read(fileDigest, {at: HEADER_OFFSET_DIGEST});
      const compDigest = this.computeDigest(this.#apBody);

      if(fileDigest.every((v,i) => v===compDigest[i])){
        // Valid digest
        const pathBytes = this.#apBody.findIndex((v)=>0===v);
        if(0===pathBytes){
          // This file is unassociated, so truncate it to avoid
          // leaving stale db data laying around.
          sah.truncate(HEADER_OFFSET_DATA);







|
>







663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
             this.#apBody);
        this.setAssociatedPath(sah, '', 0);
        return '';
      }

      const fileDigest = new Uint32Array(HEADER_DIGEST_SIZE / 4);
      sah.read(fileDigest, {at: HEADER_OFFSET_DIGEST});
      const compDigest = this.computeDigest(this.#apBody, flags);
      //warn("getAssociatedPath() flags",'0x'+flags.toString(16), "compDigest", compDigest);
      if(fileDigest.every((v,i) => v===compDigest[i])){
        // Valid digest
        const pathBytes = this.#apBody.findIndex((v)=>0===v);
        if(0===pathBytes){
          // This file is unassociated, so truncate it to avoid
          // leaving stale db data laying around.
          sah.truncate(HEADER_OFFSET_DATA);
651
652
653
654
655
656
657







658
659
660
661

662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681


682
683

684
685
686
687
688
689
690




691
692
693
694
695
696
697
       in the metadata.
    */
    setAssociatedPath(sah, path, flags){
      const enc = textEncoder.encodeInto(path, this.#apBody);
      if(HEADER_MAX_PATH_SIZE <= enc.written + 1/*NUL byte*/){
        toss("Path too long:",path);
      }







      this.#apBody.fill(0, enc.written, HEADER_MAX_PATH_SIZE);
      this.#dvBody.setUint32(HEADER_OFFSET_FLAGS, flags);

      const digest = this.computeDigest(this.#apBody);

      sah.write(this.#apBody, {at: 0});
      sah.write(digest, {at: HEADER_OFFSET_DIGEST});
      sah.flush();

      if(path){
        this.#mapFilenameToSAH.set(path, sah);
        this.#availableSAH.delete(sah);
      }else{
        // This is not a persistent file, so eliminate the contents.
        sah.truncate(HEADER_OFFSET_DATA);
        this.#availableSAH.add(sah);
      }
    }

    /**
       Computes a digest for the given byte array and returns it as a
       two-element Uint32Array. This digest gets stored in the
       metadata for each file as a validation check. Changing this
       algorithm invalidates all existing databases for this VFS, so
       don't do that.


    */
    computeDigest(byteArray){

      let h1 = 0xdeadbeef;
      let h2 = 0x41c6ce57;
      for(const v of byteArray){
        h1 = 31 * h1 + (v * 307);
        h2 = 31 * h2 + (v * 307);
      }
      return new Uint32Array([h1>>>0, h2>>>0]);




    }

    /**
       Re-initializes the state of the SAH pool, releasing and
       re-acquiring all handles.

       See acquireAccessHandles() for the specifics of the clearFiles







>
>
>
>
>
>
>


<
|
>




















>
>

|
>
|
|
|
|
|
|
|
>
>
>
>







695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710

711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
       in the metadata.
    */
    setAssociatedPath(sah, path, flags){
      const enc = textEncoder.encodeInto(path, this.#apBody);
      if(HEADER_MAX_PATH_SIZE <= enc.written + 1/*NUL byte*/){
        toss("Path too long:",path);
      }
      if(path && flags){
        /* When creating or re-writing files, update their digest, if
           needed, to v2. We continue to use v1 for the (!path) case
           (empty files) because there's little reason not to use a
           digest of 0 for empty entries. */
        flags |= FLAG_COMPUTE_DIGEST_V2;
      }
      this.#apBody.fill(0, enc.written, HEADER_MAX_PATH_SIZE);
      this.#dvBody.setUint32(HEADER_OFFSET_FLAGS, flags);

      const digest = this.computeDigest(this.#apBody, flags);
      //console.warn("setAssociatedPath(",path,") digest",digest);
      sah.write(this.#apBody, {at: 0});
      sah.write(digest, {at: HEADER_OFFSET_DIGEST});
      sah.flush();

      if(path){
        this.#mapFilenameToSAH.set(path, sah);
        this.#availableSAH.delete(sah);
      }else{
        // This is not a persistent file, so eliminate the contents.
        sah.truncate(HEADER_OFFSET_DATA);
        this.#availableSAH.add(sah);
      }
    }

    /**
       Computes a digest for the given byte array and returns it as a
       two-element Uint32Array. This digest gets stored in the
       metadata for each file as a validation check. Changing this
       algorithm invalidates all existing databases for this VFS, so
       don't do that.

       See the docs for FLAG_COMPUTE_DIGEST_V2 for more details.
    */
    computeDigest(byteArray, fileFlags){
      if( fileFlags & FLAG_COMPUTE_DIGEST_V2 ){
        let h1 = 0xdeadbeef;
        let h2 = 0x41c6ce57;
        for(const v of byteArray){
          h1 = Math.imul(h1 ^ v, 2654435761);
          h2 = Math.imul(h2 ^ v, 104729);
        }
        return new Uint32Array([h1>>>0, h2>>>0]);
      }else{
        /* this is what the buggy legacy computation worked out to */
        return new Uint32Array([0,0]);
      }
    }

    /**
       Re-initializes the state of the SAH pool, releasing and
       re-acquiring all handles.

       See acquireAccessHandles() for the specifics of the clearFiles
Changes to ext/wasm/tester1.c-pp.js.
3502
3503
3504
3505
3506
3507
3508
3509
3510
3511
3512
3513
3514
3515
3516
                ++nAuxReused;
              }
              return x;
            }
          });
          db.exec([
            "create table t(a);",
            "insert into t(a) values(1),(2),(3);",
            "select auxtest(1,a), auxtest(1,a) from t order by a"
          ]);
        }finally{
          db.close();
          wasm.pstack.restore(stack);
        }
        T.assert(nAuxSet>0).assert(nAuxReused>0)







|







3502
3503
3504
3505
3506
3507
3508
3509
3510
3511
3512
3513
3514
3515
3516
                ++nAuxReused;
              }
              return x;
            }
          });
          db.exec([
            "create table t(a);",
            "insert into t(a) values(1),(2),(1);",
            "select auxtest(1,a), auxtest(1,a) from t order by a"
          ]);
        }finally{
          db.close();
          wasm.pstack.restore(stack);
        }
        T.assert(nAuxSet>0).assert(nAuxReused>0)
Added ext/wasm/tests/opfs/sahpool/digest-worker.js.




























































































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
/*
  2025-01-31

  The author disclaims copyright to this source code.  In place of a
  legal notice, here is a blessing:

  *   May you do good and not evil.
  *   May you find forgiveness for yourself and forgive others.
  *   May you share freely, never taking more than you give.

  ***********************************************************************

  This file is part of sahpool-pausing.js's demonstration of the
  pause/unpause feature of the opfs-sahpool VFS.
*/
const clog = console.log.bind(console);
const wPost = (type,...args)=>postMessage({type, payload:args});
const log = (...args)=>{
  clog("Worker:",...args);
  wPost('log',...args);
}

const hasOpfs = ()=>{
  return globalThis.FileSystemHandle
    && globalThis.FileSystemDirectoryHandle
    && globalThis.FileSystemFileHandle
    && globalThis.FileSystemFileHandle.prototype.createSyncAccessHandle
    && navigator?.storage?.getDirectory;
};
if( !hasOpfs() ){
  wPost('error',"OPFS not detected");
  throw new Error("OPFS not detected");
}

clog("Importing sqlite3...");
const searchParams = new URL(self.location.href).searchParams;
importScripts(searchParams.get('sqlite3.dir') + '/sqlite3.js');

const runTests = function(sqlite3, poolUtil){
  const fname = '/my.db';
  let db = new poolUtil.OpfsSAHPoolDb(fname);
  let n = (new Date()).valueOf();
  try {
    db.exec([
      "create table if not exists t(a);"
    ]);
    db.exec({
      sql: "insert into t(a) values(?)",
      bind: n++
    });
    log(fname,"record count: ",db.selectValue("select count(*) from t"));
  }finally{
    db.close();
  }

  db = new poolUtil.OpfsSAHPoolDb(fname);
  try {
    db.exec({
      sql: "insert into t(a) values(?)",
      bind: n++
    });
    log(fname,"record count: ",db.selectValue("select count(*) from t"));
  }finally{
    db.close();
  }

  const fname2 = '/my2.db';
  db = new poolUtil.OpfsSAHPoolDb(fname2);
  try {
    db.exec([
      "create table if not exists t(a);"
    ]);
    db.exec({
      sql: "insert into t(a) values(?)",
      bind: n++
    });
    log(fname2,"record count: ",db.selectValue("select count(*) from t"));
  }finally{
    db.close();
  }
};

globalThis.sqlite3InitModule().then(async function(sqlite3){
  log("sqlite3 version:",sqlite3.version);
  const sahPoolConfig = {
    name: 'opfs-sahpool-digest',
    clearOnInit: false,
    initialCapacity: 6
  };
  return sqlite3.installOpfsSAHPoolVfs(sahPoolConfig).then(poolUtil=>{
    log('vfs acquired');
    runTests(sqlite3, poolUtil);
  });
});
Added ext/wasm/tests/opfs/sahpool/digest.html.


























































































































































































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
<!doctype html>
<html lang="en-us">
  <head>
    <meta charset="utf-8">
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <link rel="shortcut icon" href="data:image/x-icon;," type="image/x-icon">
    <link rel="stylesheet" href="../../../common/emscripten.css"/>
    <link rel="stylesheet" href="../../../common/testing.css"/>
    <title>sqlite3 tester: OpfsSAHPool Digest</title>
    <style></style>
  </head>
  <body><h1 id='color-target'></h1>

    <p>
      This is a test app for the digest calculation of the OPFS
      SAHPool VFS. It requires running it with a new database created using
      v3.49.0 or older, then running it again with a newer version, then
      again with 3.49.0 or older.
    </p>
    <div class='input-wrapper'>
      <input type='checkbox' id='cb-log-reverse'>
      <label for='cb-log-reverse'>Reverse log order?</label>
    </div>
    <div id='test-output'></div>
    <script>
      /*
        2025-02-03

        The author disclaims copyright to this source code.  In place of a
        legal notice, here is a blessing:

        *   May you do good and not evil.
        *   May you find forgiveness for yourself and forgive others.
        *   May you share freely, never taking more than you give.

        ***********************************************************************

        This is a bugfix test for the OPFS SAHPool VFS. It requires setting up
        a database created using v3.49.0 or older, then running it again with
        a newer version.
      */
      (function(){
        'use strict';
        document.querySelector('h1').innerHTML =
        document.querySelector('title').innerHTML;
        const mapToString = (v)=>{
          switch(typeof v){
            case 'number': case 'string': case 'boolean':
            case 'undefined': case 'bigint':
              return ''+v;
            default: break;
          }
          if(null===v) return 'null';
          if(v instanceof Error){
            v = {
              message: v.message,
              stack: v.stack,
              errorClass: v.name
            };
          }
          return JSON.stringify(v,undefined,2);
        };
        const normalizeArgs = (args)=>args.map(mapToString);
        const logTarget = document.querySelector('#test-output');
        const logClass = function(cssClass,...args){
          const ln = document.createElement('div');
          if(cssClass){
            for(const c of (Array.isArray(cssClass) ? cssClass : [cssClass])){
              ln.classList.add(c);
            }
          }
          ln.append(document.createTextNode(normalizeArgs(args).join(' ')));
          logTarget.append(ln);
        };
        const cbReverse = document.querySelector('#cb-log-reverse');
        //cbReverse.setAttribute('checked','checked');
        const cbReverseKey = 'tester1:cb-log-reverse';
        const cbReverseIt = ()=>{
          logTarget.classList[cbReverse.checked ? 'add' : 'remove']('reverse');
          //localStorage.setItem(cbReverseKey, cbReverse.checked ? 1 : 0);
        };
        cbReverse.addEventListener('change', cbReverseIt, true);
        /*if(localStorage.getItem(cbReverseKey)){
          cbReverse.checked = !!(+localStorage.getItem(cbReverseKey));
          }*/
        cbReverseIt();

        const log = (...args)=>{
          //console.log(...args);
          logClass('',...args);
        }
        const warn = (...args)=>{
          console.warn(...args);
          logClass('warning',...args);
        }
        const error = (...args)=>{
          console.error(...args);
          logClass('error',...args);
        };

        const toss = (...args)=>{
          error(...args);
          throw new Error(args.join(' '));
        };

        const endOfWork = (passed=true)=>{
          const eH = document.querySelector('#color-target');
          const eT = document.querySelector('title');
          if(passed){
            log("End of work chain. If you made it this far, you win.");
            eH.innerText = 'PASS: '+eH.innerText;
            eH.classList.add('tests-pass');
            eT.innerText = 'PASS: '+eT.innerText;
          }else{
            eH.innerText = 'FAIL: '+eH.innerText;
            eH.classList.add('tests-fail');
            eT.innerText = 'FAIL: '+eT.innerText;
          }
        };

        log("Running opfs-sahpool digest tests...");
        const W1 = new Worker('digest-worker.js?sqlite3.dir=../../../jswasm');
        W1.onmessage = function({data}){
          //log("onmessage:",data);
          switch(data.type){
            case 'log':
              log('worker says:', ...data.payload);
              break;
            case 'error':
              error('worker says:', ...data.payload);
              endOfWork(false);
              break;
            case 'initialized':
              log(data.workerId, ': Worker initialized',...data.payload);
              break;
          }
        };
      })();
    </script>
  </body>
</html>