/ Check-in [39a4267f]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Have the zonefile extension use binary instead of text keys.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | zonefile
Files: files | file ages | folders
SHA3-256: 39a4267fc9cec77fd8d9be25c73b848e77a68906253cc75d61fe90e549bafa27
User & Date: dan 2018-02-27 14:26:33
Context
2018-02-27
15:42
Adjustments to test numbers for the 'zonefile' extension. check-in: 55de6f14 user: mistachkin tags: zonefile
14:26
Have the zonefile extension use binary instead of text keys. check-in: 39a4267f user: dan tags: zonefile
2018-02-26
07:58
Add extra parameter to zonefileCodecCreate() to indicate whether the new object will be used for mock-encryption or mock-decryption. check-in: 231832c4 user: dan tags: zonefile
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to ext/zonefile/README.md.

54
55
56
57
58
59
60
61




62
63
64
65
66
67
68
69
70



71
72
73
74
75
76
77
..
96
97
98
99
100
101
102
103












104
105
106
107
108
109
110
111
112

<tr valign=top><td>encryptionType<td>"none"
<td>The encryption type to use. At present the only valid values are
"none" (no encryption) and "xor" (an insecure mock encryption method
useful for testing only).

<tr valign=top><td>encryptionKey<td>""
<td>The encryption key (a string) to use. The value of this option is




ignored if <i>encryptionType</i> is set to "none".
</table>

For example, to create a zonefile named "test.zonefile" based on the
contents of database table "test_input" and with a maximum automatic
frame size of 4096 bytes:

>     SELECT zonefile_write('test.zonefile', 'test_input',
>       '{"maxAutoFrameSize":4096}'



>     );

### Using (Reading) Zonefile Files

To create a new zonefile table:

>     CREATE VIRTUAL TABLE z1 USING zonefile;
................................................................................
>     );

Both tables are initially empty. To add a zonefile to the index, insert a
row into the "z1_files" table:

>     INSERT INTO z1_files(filename) VALUES(<filename>);

Currently, any value provided for any column other than "filename" is 












ignored. Files are removed from the index by deleting rows from the
z1_files table:

>     DELETE FROM z1_files WHERE filename = <filename>;

Once zonefile files have been added to the index, their contents are 
visible in table "z1". To retrieve the value associated with a single
key from one of the zonefile files in the index:








|
>
>
>
>
|



|
|


|
>
>
>







 







|
>
>
>
>
>
>
>
>
>
>
>
>
|
|







54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
...
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131

<tr valign=top><td>encryptionType<td>"none"
<td>The encryption type to use. At present the only valid values are
"none" (no encryption) and "xor" (an insecure mock encryption method
useful for testing only).

<tr valign=top><td>encryptionKey<td>""
<td>The encryption key to use. The encryption key must be specified as an
even number of hexadecimal that will be converted to a binary key before
use. It is the responsibility of the caller to specify a key of the optimal
length for each encryption algorithm (e.g. 16 bytes (32 hex digits) for
a 128-bit encryption, or 32 bytes (64 digits) for a 256-bit method).
This option is ignored if <i>encryptionType</i> is set to "none".
</table>

For example, to create a zonefile named "test.zonefile" based on the
contents of database table "test_input", with a maximum automatic
frame size of 4096 bytes and using "xor" encryption with a 128-bit key:

>     SELECT zonefile_write('test.zonefile', 'test_input',
>       '{"maxAutoFrameSize":4096,
>         "encryptionType":"xor",
>         "encryptionKey":"e6e600bc063aad12f6387beab650c48a"
>       }'
>     );

### Using (Reading) Zonefile Files

To create a new zonefile table:

>     CREATE VIRTUAL TABLE z1 USING zonefile;
................................................................................
>     );

Both tables are initially empty. To add a zonefile to the index, insert a
row into the "z1_files" table:

>     INSERT INTO z1_files(filename) VALUES(<filename>);

If the file is an encrypted file, then the encryption key (a blob) must
be inserted into the "ekey" column. Encryption keys are not stored in the
database, they are held in main-memory only. This means that each new
connection must configure encryption key using UPDATE statements before
accessing any encrypted files. For example:

>     -- Add new encrypted file to database:
>     INSERT INTO z1_files(filename, ekey) VALUES(<filename>, <ekey>);
>
>     -- Configure encryption key for existing file after opening database:
>     UPDATE z1_files SET ekey = <ekey> WHERE filename = <filename>;

Currently, values provided for any columns other than "filename" and
"ekey" are ignored. Files are removed from the index by deleting rows 
from the z1_files table:

>     DELETE FROM z1_files WHERE filename = <filename>;

Once zonefile files have been added to the index, their contents are 
visible in table "z1". To retrieve the value associated with a single
key from one of the zonefile files in the index:

Changes to ext/zonefile/zonefile.c.

171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
...
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
...
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263

264
265
266
267
268
269
270
...
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
...
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
...
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
...
733
734
735
736
737
738
739

740
741
742
743
744
745
746
...
844
845
846
847
848
849
850





851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
...
901
902
903
904
905
906
907



908
909
910
911
912
913
914
....
1081
1082
1083
1084
1085
1086
1087




























1088
1089
1090
1091
1092
1093
1094
....
1125
1126
1127
1128
1129
1130
1131


1132
1133
1134

1135
1136
1137
1138
1139
1140
1141
....
1849
1850
1851
1852
1853
1854
1855

1856
1857
1858
1859
1860
1861
1862
1863
1864
1865
1866
1867
1868
1869
1870
1871
....
1886
1887
1888
1889
1890
1891
1892



1893
1894
1895
1896
1897
1898
1899
1900
1901
1902
1903
1904
1905
1906
1907
1908
1909
1910
1911
1912
1913
1914
1915

1916
1917



1918

1919
1920
1921
1922
1923
1924
1925
....
2581
2582
2583
2584
2585
2586
2587
2588
2589
2590
2591
2592
2593
2594
2595
2596
2597
2598
2599
2600
2601
**
** The size of the content will grow by the nonce size.  Hence, the
** buffer must have at least nonce bytes of extra space available at
** the end to accommodate that growth.  When persisting results, be
** sure to include the extra bytes.
*/
static void zonefileCodecEncode(
  ZonefileCodec *pCodec, 
  unsigned char *pIn, int nIn
){
  int i;
  u8 *aNonce = &pIn[nIn];
  assert( pCodec->bEncrypt );
  sqlite3_randomness(16, aNonce);
  for(i=0; i<nIn; i++){
................................................................................
  int nHash;                      /* Size of aHash[] array */
  ZonefileKey **aHash;            /* Hash buckets */
};
struct ZonefileKey {
  const char *zName;              /* Zonefile table name */
  const char *zDb;                /* Database name ("main", "temp" etc.) */
  i64 iFileid;                    /* File id */
  const char *zKey;               /* Key buffer */
  int nKey;                       /* Size of zKey in bytes */
  u32 iHash;                      /* zonefileKeyHash() value */
  ZonefileKey *pHashNext;         /* Next colliding key in hash table */
};

/*
** Return a 32-bit hash value for the three arguments.
................................................................................
  int i;
  for(i=0; zDb[i]; i++) iHash += (iHash<<3) + (u8)zDb[i];
  for(i=0; zTab[i]; i++) iHash += (iHash<<3) + (u8)zTab[i];
  return (iHash ^ (iFileid & 0xFFFFFFFF));
}

/* 
** Store encryption key zKey in the key-store passed as the first argument.
** Return SQLITE_OK if successful, or an SQLite error code (SQLITE_NOMEM)
** otherwise.
*/
static int zonefileKeyStore(
  ZonefileGlobal *pGlobal,
  const char *zDb,                /* Database containing zonefile table */
  const char *zTab,               /* Name of zonefile table */
  i64 iFileid,                    /* File-id to configure key for */
  const char *zKey                /* Key to store */

){
  ZonefileKey **pp;
  u32 iHash = zonefileKeyHash(zDb, zTab, iFileid);

  /* Remove any old entry */
  if( pGlobal->nHash ){
    for(pp=&pGlobal->aHash[iHash%pGlobal->nHash]; *pp; pp=&((*pp)->pHashNext)){
................................................................................
        *pp = pThis->pHashNext;
        sqlite3_free(pThis);
        break;
      }
    }
  }

  if( zKey ){
    int nKey = strlen(zKey);
    int nDb = strlen(zDb);
    int nTab = strlen(zTab);
    ZonefileKey *pNew;

    /* Resize the hash-table, if necessary */
    if( pGlobal->nEntry>=pGlobal->nHash ){
      int i;
................................................................................
    pNew = (ZonefileKey*)sqlite3_malloc(
        sizeof(ZonefileKey) + nKey+1 + nDb+1 + nTab+1
    );
    if( pNew==0 ) return SQLITE_NOMEM;
    memset(pNew, 0, sizeof(ZonefileKey));
    pNew->iFileid = iFileid;
    pNew->iHash = iHash;
    pNew->zKey = (const char*)&pNew[1];
    pNew->nKey = nKey;
    pNew->zDb = &pNew->zKey[nKey+1];
    pNew->zName = &pNew->zDb[nDb+1];
    memcpy((char*)pNew->zKey, zKey, nKey+1);
    memcpy((char*)pNew->zDb, zDb, nDb+1);
    memcpy((char*)pNew->zName, zTab, nTab+1);

    pNew->pHashNext = pGlobal->aHash[iHash % pGlobal->nHash];
    pGlobal->aHash[iHash % pGlobal->nHash] = pNew;
    pGlobal->nEntry++;
  }
................................................................................
** in this case.
*/
static int zonefileKeyFind(
  ZonefileGlobal *pGlobal,
  const char *zDb,                /* Database containing zonefile table */
  const char *zTab,               /* Name of zonefile table */
  i64 iFileid,                    /* File-id to configure key for */
  const char **pzKey              /* OUT: Pointer to key buffer */
){
  if( pGlobal->nHash ){
    ZonefileKey *pKey;
    u32 iHash = zonefileKeyHash(zDb, zTab, iFileid);
    for(pKey=pGlobal->aHash[iHash%pGlobal->nHash]; pKey; pKey=pKey->pHashNext){
      if( pKey->iFileid==iFileid 
       && 0==sqlite3_stricmp(zTab, pKey->zName)
       && 0==sqlite3_stricmp(zDb, pKey->zDb)
      ){
        *pzKey = pKey->zKey;
        return pKey->nKey;
      }
    }
  }

  return 0;
}
................................................................................
typedef struct ZonefileParam ZonefileParam;
struct ZonefileParam {
  ZonefileCompress *pCmpIdx;      /* For compressing the index */
  ZonefileCompress *pCmpData;     /* For compressing each frame */
  int encryptionType;
  int maxAutoFrameSize;
  int debugExtendedHeaderSize;    /* Size of extended header */

  char *encryptionKey;            /* Encryption key */
};

/*
** A structure to store a deserialized zonefile header in.
*/
typedef struct ZonefileHeader ZonefileHeader;
................................................................................

int zonefileIsAutoFrame(sqlite3_value *pFrame){
  return (
      sqlite3_value_type(pFrame)==SQLITE_INTEGER 
   && sqlite3_value_int64(pFrame)==-1
  );
}






static int zonefileEncryption(const char *zName, int *peType, char **pzErr){
  struct Encryption {
    const char *zName;
    int eType;
  } a[] = {
    {"NONE", 0}, 
    {"AES_128_CTR", 1}, 
    {"AES_128_CBC", 2}, 
    {"AES_256_CTR", 3}, 
    {"AES_256_CBC", 4}, 
    {"XOR",         5}, 
  };
  int i;

  for(i=0; i<sizeof(a)/sizeof(a[0]); i++){
    if( 0==sqlite3_stricmp(zName, a[i].zName) ){
      *peType = a[i].eType;
................................................................................
      if( iVal<0 || iVal>255 ){
        zErr = sqlite3_mprintf(
            "debugExtendedHeaderSize value out of range: %d", iVal
        );
        rc = SQLITE_ERROR;
      }
      p->debugExtendedHeaderSize = iVal;



    }else
    if( sqlite3_stricmp("maxAutoFrameSize", zKey)==0 ){
      p->maxAutoFrameSize = iVal;
    }else
    if( sqlite3_stricmp("compressionTypeIndexData", zKey)==0 ){
      const char *zName = (const char*)sqlite3_column_text(pStmt, 1);
      rc = zonefileCompress(zName, &p->pCmpIdx, &zErr);
................................................................................
      int n = MIN(nRem, sizeof(buf));
      if( zonefileFileWrite(pFd, buf, n) ) return SQLITE_ERROR;
      nRem -= n;
    }
  }
  return SQLITE_OK;
}





























/*
** Function:     zonefile_write(F,T[,J])
*/
static void zonefileWriteFunc(
  sqlite3_context *pCtx,       /* Context object */
  int objc,                       /* Number of SQL arguments */
................................................................................
  if( objc==3 ){
    zJson = (const char*)sqlite3_value_text(objv[2]);
  }
  if( zonefileGetParams(pCtx, zJson, &sParam) ) return;

  if( sParam.encryptionType!=0 ){
    int n = strlen(sParam.encryptionKey);


    rc = zonefileCodecCreate(
        sParam.encryptionType, 1, (u8*)sParam.encryptionKey, n, &pCodec, &zErr
    );

    if( rc!=SQLITE_OK ){
      if( zErr ){
        sqlite3_result_error(pCtx, zErr, -1);
      }else{
        sqlite3_result_error_code(pCtx, rc);
      }
      sqlite3_free(zErr);
................................................................................
static int zffUpdate(
  sqlite3_vtab *pVtab, 
  int nVal, 
  sqlite3_value **apVal, 
  sqlite_int64 *pRowid
){
  int rc = SQLITE_OK;

  ZonefileFilesTab *pTab = (ZonefileFilesTab*)pVtab;

  if( sqlite3_value_type(apVal[0])==SQLITE_INTEGER ){
    if( nVal>1 && sqlite3_value_nochange(apVal[2]) ){
      const char *zKey = (const char*)sqlite3_value_text(apVal[3]);
      i64 iFileid = sqlite3_value_int64(apVal[0]);
      return zonefileKeyStore(
          pTab->pGlobal, pTab->zDb, pTab->zBase, iFileid, zKey
      );
    }else{
      if( pTab->pDelete==0 ){
        rc = zonefilePrepare(pTab->db, &pTab->pDelete, &pVtab->zErrMsg,
            "DELETE FROM %Q.'%q_shadow_file' WHERE fileid=?",
            pTab->zDb, pTab->zBase
            );
      }
................................................................................
        sqlite3_step(pTab->pDeleteIdx);
        rc = sqlite3_reset(pTab->pDeleteIdx);
      }
    }
  }
  if( nVal>1 ){
    i64 iFileid = 0;



    const char *zFile = (const char*)sqlite3_value_text(apVal[2]);

    if( pTab->pInsert==0 ){
      rc = zonefilePrepare(pTab->db, &pTab->pInsert, &pVtab->zErrMsg,
          "INSERT INTO %Q.'%q_shadow_file'(filename) VALUES(?)",
          pTab->zDb, pTab->zBase
      );
    }

    /* Add the new entry to the %_shadow_file table. */
    if( rc==SQLITE_OK ){
      sqlite3_bind_text(pTab->pInsert, 1, zFile, -1, SQLITE_TRANSIENT);
      sqlite3_step(pTab->pInsert);
      rc = sqlite3_reset(pTab->pInsert);
    }

    /* Populate the %_shadow_idx table with entries for all keys in
    ** the zonefile just added to %_shadow_file.  */
    if( rc==SQLITE_OK ){
      iFileid = sqlite3_last_insert_rowid(pTab->db);
      rc = zonefilePopulateIndex(pTab, zFile, iFileid);
    }


    if( rc==SQLITE_OK ){
      const char *zKey = (const char*)sqlite3_value_text(apVal[3]);



      rc = zonefileKeyStore(pTab->pGlobal, pTab->zDb, pTab->zBase,iFileid,zKey);

    }
  }

  return rc;
}

/* Each entry in the frame-cache is represented by an instance of the
................................................................................
        rc = pCmpMethod->xOpen(&pCmp, aDict, nDict);
      }
      sqlite3_free(aDict);
    }

    /* Find the encryption method and key. */
    if( rc==SQLITE_OK && hdr.encryptionType ){
      const char *z = 0;
      int n = zonefileKeyFind(pTab->pGlobal, pTab->zDb, pTab->zName, iFile, &z);
      if( n==0 ){
        zErr = sqlite3_mprintf("missing encryption key for file \"%s\"", zFile);
        rc = SQLITE_ERROR;
      }else{
        rc = zonefileCodecCreate(hdr.encryptionType, 0, (u8*)z,n,&pCodec,&zErr);
      }
    }

    /* Read some data into memory. */
    if( rc==SQLITE_OK ){
      int szFrame = sqlite3_column_int(pCsr->pSelect, 3);








|







 







|







 







|








|
>







 







|
<







 







|

|

|







 







|









|







 







>







 







>
>
>
>
>







|
|
|
|







 







>
>
>







 







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







 







>
>
|
|
|
>







 







>




|
<
<
<
<







 







>
>
>
|

|
|
|
|
|
|

|
|
|
|
|
|

|
|
|
|
|
|
|
>

<
>
>
>
|
>







 







|
|




|







171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
...
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
...
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
...
278
279
280
281
282
283
284
285

286
287
288
289
290
291
292
...
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
...
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
...
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
...
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
...
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
....
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
....
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
....
1889
1890
1891
1892
1893
1894
1895
1896
1897
1898
1899
1900
1901




1902
1903
1904
1905
1906
1907
1908
....
1923
1924
1925
1926
1927
1928
1929
1930
1931
1932
1933
1934
1935
1936
1937
1938
1939
1940
1941
1942
1943
1944
1945
1946
1947
1948
1949
1950
1951
1952
1953
1954
1955
1956
1957

1958
1959
1960
1961
1962
1963
1964
1965
1966
1967
1968
1969
....
2625
2626
2627
2628
2629
2630
2631
2632
2633
2634
2635
2636
2637
2638
2639
2640
2641
2642
2643
2644
2645
**
** The size of the content will grow by the nonce size.  Hence, the
** buffer must have at least nonce bytes of extra space available at
** the end to accommodate that growth.  When persisting results, be
** sure to include the extra bytes.
*/
static void zonefileCodecEncode(
  ZonefileCodec *pCodec,
  unsigned char *pIn, int nIn
){
  int i;
  u8 *aNonce = &pIn[nIn];
  assert( pCodec->bEncrypt );
  sqlite3_randomness(16, aNonce);
  for(i=0; i<nIn; i++){
................................................................................
  int nHash;                      /* Size of aHash[] array */
  ZonefileKey **aHash;            /* Hash buckets */
};
struct ZonefileKey {
  const char *zName;              /* Zonefile table name */
  const char *zDb;                /* Database name ("main", "temp" etc.) */
  i64 iFileid;                    /* File id */
  const u8 *aKey;                 /* Key buffer */
  int nKey;                       /* Size of zKey in bytes */
  u32 iHash;                      /* zonefileKeyHash() value */
  ZonefileKey *pHashNext;         /* Next colliding key in hash table */
};

/*
** Return a 32-bit hash value for the three arguments.
................................................................................
  int i;
  for(i=0; zDb[i]; i++) iHash += (iHash<<3) + (u8)zDb[i];
  for(i=0; zTab[i]; i++) iHash += (iHash<<3) + (u8)zTab[i];
  return (iHash ^ (iFileid & 0xFFFFFFFF));
}

/* 
** Store encryption key aKey in the key-store passed as the first argument.
** Return SQLITE_OK if successful, or an SQLite error code (SQLITE_NOMEM)
** otherwise.
*/
static int zonefileKeyStore(
  ZonefileGlobal *pGlobal,
  const char *zDb,                /* Database containing zonefile table */
  const char *zTab,               /* Name of zonefile table */
  i64 iFileid,                    /* File-id to configure key for */
  const u8 *aKey,                 /* Key to store */
  int nKey                        /* Size of aKey[] in bytes */
){
  ZonefileKey **pp;
  u32 iHash = zonefileKeyHash(zDb, zTab, iFileid);

  /* Remove any old entry */
  if( pGlobal->nHash ){
    for(pp=&pGlobal->aHash[iHash%pGlobal->nHash]; *pp; pp=&((*pp)->pHashNext)){
................................................................................
        *pp = pThis->pHashNext;
        sqlite3_free(pThis);
        break;
      }
    }
  }

  if( aKey ){

    int nDb = strlen(zDb);
    int nTab = strlen(zTab);
    ZonefileKey *pNew;

    /* Resize the hash-table, if necessary */
    if( pGlobal->nEntry>=pGlobal->nHash ){
      int i;
................................................................................
    pNew = (ZonefileKey*)sqlite3_malloc(
        sizeof(ZonefileKey) + nKey+1 + nDb+1 + nTab+1
    );
    if( pNew==0 ) return SQLITE_NOMEM;
    memset(pNew, 0, sizeof(ZonefileKey));
    pNew->iFileid = iFileid;
    pNew->iHash = iHash;
    pNew->aKey = (const u8*)&pNew[1];
    pNew->nKey = nKey;
    pNew->zDb = (const char*)&pNew->aKey[nKey+1];
    pNew->zName = &pNew->zDb[nDb+1];
    memcpy((u8*)pNew->aKey, aKey, nKey+1);
    memcpy((char*)pNew->zDb, zDb, nDb+1);
    memcpy((char*)pNew->zName, zTab, nTab+1);

    pNew->pHashNext = pGlobal->aHash[iHash % pGlobal->nHash];
    pGlobal->aHash[iHash % pGlobal->nHash] = pNew;
    pGlobal->nEntry++;
  }
................................................................................
** in this case.
*/
static int zonefileKeyFind(
  ZonefileGlobal *pGlobal,
  const char *zDb,                /* Database containing zonefile table */
  const char *zTab,               /* Name of zonefile table */
  i64 iFileid,                    /* File-id to configure key for */
  const u8 **paKey                /* OUT: Pointer to key buffer */
){
  if( pGlobal->nHash ){
    ZonefileKey *pKey;
    u32 iHash = zonefileKeyHash(zDb, zTab, iFileid);
    for(pKey=pGlobal->aHash[iHash%pGlobal->nHash]; pKey; pKey=pKey->pHashNext){
      if( pKey->iFileid==iFileid 
       && 0==sqlite3_stricmp(zTab, pKey->zName)
       && 0==sqlite3_stricmp(zDb, pKey->zDb)
      ){
        *paKey = pKey->aKey;
        return pKey->nKey;
      }
    }
  }

  return 0;
}
................................................................................
typedef struct ZonefileParam ZonefileParam;
struct ZonefileParam {
  ZonefileCompress *pCmpIdx;      /* For compressing the index */
  ZonefileCompress *pCmpData;     /* For compressing each frame */
  int encryptionType;
  int maxAutoFrameSize;
  int debugExtendedHeaderSize;    /* Size of extended header */
  int debugEncryptionKeyText;     /* True to allow text keys */
  char *encryptionKey;            /* Encryption key */
};

/*
** A structure to store a deserialized zonefile header in.
*/
typedef struct ZonefileHeader ZonefileHeader;
................................................................................

int zonefileIsAutoFrame(sqlite3_value *pFrame){
  return (
      sqlite3_value_type(pFrame)==SQLITE_INTEGER 
   && sqlite3_value_int64(pFrame)==-1
  );
}

#define SQLITE_ZONEFILE_AES_128_CTR 1
#define SQLITE_ZONEFILE_AES_128_CBC 2
#define SQLITE_ZONEFILE_AES_256_CTR 3
#define SQLITE_ZONEFILE_AES_256_CBC 4

static int zonefileEncryption(const char *zName, int *peType, char **pzErr){
  struct Encryption {
    const char *zName;
    int eType;
  } a[] = {
    {"NONE", 0}, 
    {"AES_128_CTR", SQLITE_ZONEFILE_AES_128_CTR}, 
    {"AES_128_CBC", SQLITE_ZONEFILE_AES_128_CBC}, 
    {"AES_256_CTR", SQLITE_ZONEFILE_AES_256_CTR}, 
    {"AES_256_CBC", SQLITE_ZONEFILE_AES_256_CBC}, 
    {"XOR",         5}, 
  };
  int i;

  for(i=0; i<sizeof(a)/sizeof(a[0]); i++){
    if( 0==sqlite3_stricmp(zName, a[i].zName) ){
      *peType = a[i].eType;
................................................................................
      if( iVal<0 || iVal>255 ){
        zErr = sqlite3_mprintf(
            "debugExtendedHeaderSize value out of range: %d", iVal
        );
        rc = SQLITE_ERROR;
      }
      p->debugExtendedHeaderSize = iVal;
    }else
    if( sqlite3_stricmp("debugEncryptionKeyText", zKey)==0 ){
      p->debugEncryptionKeyText = iVal;
    }else
    if( sqlite3_stricmp("maxAutoFrameSize", zKey)==0 ){
      p->maxAutoFrameSize = iVal;
    }else
    if( sqlite3_stricmp("compressionTypeIndexData", zKey)==0 ){
      const char *zName = (const char*)sqlite3_column_text(pStmt, 1);
      rc = zonefileCompress(zName, &p->pCmpIdx, &zErr);
................................................................................
      int n = MIN(nRem, sizeof(buf));
      if( zonefileFileWrite(pFd, buf, n) ) return SQLITE_ERROR;
      nRem -= n;
    }
  }
  return SQLITE_OK;
}

static int zonefileHexChar(char c){
  if( c>='0' && c<='9' ) return c-'0';
  c = c & ~0x20;
  if( c>='A' && c<='F' ) return c-('A'-10);
  return -1;
}

static int zonefileDecodeEncryptionKey(ZonefileParam *p, int *pn, char **pzErr){
  if( p->debugEncryptionKeyText==0 ){
    u8 *z = (u8*)p->encryptionKey;
    int n = *pn;
    int i;
    if( n&0x01 ) goto bad_format;
    for(i=0; i<n; i+=2){
      int a = zonefileHexChar(z[i]);
      int b = zonefileHexChar(z[i+1]);
      if( a<0 || b<0 ) goto bad_format;
      z[i/2] = (u8)(a<<4) + (u8)b;
    }
    *pn = n/2;
  }
  return SQLITE_OK;

 bad_format:
  *pzErr = sqlite3_mprintf("badly formatted hex string");
  return SQLITE_ERROR;
}

/*
** Function:     zonefile_write(F,T[,J])
*/
static void zonefileWriteFunc(
  sqlite3_context *pCtx,       /* Context object */
  int objc,                       /* Number of SQL arguments */
................................................................................
  if( objc==3 ){
    zJson = (const char*)sqlite3_value_text(objv[2]);
  }
  if( zonefileGetParams(pCtx, zJson, &sParam) ) return;

  if( sParam.encryptionType!=0 ){
    int n = strlen(sParam.encryptionKey);
    rc = zonefileDecodeEncryptionKey(&sParam, &n, &zErr);
    if( rc==SQLITE_OK ){
      rc = zonefileCodecCreate(sParam.encryptionType, 
          1, (u8*)sParam.encryptionKey, n, &pCodec, &zErr
      );
    }
    if( rc!=SQLITE_OK ){
      if( zErr ){
        sqlite3_result_error(pCtx, zErr, -1);
      }else{
        sqlite3_result_error_code(pCtx, rc);
      }
      sqlite3_free(zErr);
................................................................................
static int zffUpdate(
  sqlite3_vtab *pVtab, 
  int nVal, 
  sqlite3_value **apVal, 
  sqlite_int64 *pRowid
){
  int rc = SQLITE_OK;
  int bUpdateKey = 0;
  ZonefileFilesTab *pTab = (ZonefileFilesTab*)pVtab;

  if( sqlite3_value_type(apVal[0])==SQLITE_INTEGER ){
    if( nVal>1 && sqlite3_value_nochange(apVal[2]) ){
      bUpdateKey = 1;




    }else{
      if( pTab->pDelete==0 ){
        rc = zonefilePrepare(pTab->db, &pTab->pDelete, &pVtab->zErrMsg,
            "DELETE FROM %Q.'%q_shadow_file' WHERE fileid=?",
            pTab->zDb, pTab->zBase
            );
      }
................................................................................
        sqlite3_step(pTab->pDeleteIdx);
        rc = sqlite3_reset(pTab->pDeleteIdx);
      }
    }
  }
  if( nVal>1 ){
    i64 iFileid = 0;
    if( bUpdateKey ){
      iFileid = sqlite3_value_int64(apVal[0]);
    }else{
      const char *zFile = (const char*)sqlite3_value_text(apVal[2]);

      if( pTab->pInsert==0 ){
        rc = zonefilePrepare(pTab->db, &pTab->pInsert, &pVtab->zErrMsg,
            "INSERT INTO %Q.'%q_shadow_file'(filename) VALUES(?)",
            pTab->zDb, pTab->zBase
        );
      }

      /* Add the new entry to the %_shadow_file table. */
      if( rc==SQLITE_OK ){
        sqlite3_bind_text(pTab->pInsert, 1, zFile, -1, SQLITE_TRANSIENT);
        sqlite3_step(pTab->pInsert);
        rc = sqlite3_reset(pTab->pInsert);
      }

      /* Populate the %_shadow_idx table with entries for all keys in
      ** the zonefile just added to %_shadow_file.  */
      if( rc==SQLITE_OK ){
        iFileid = sqlite3_last_insert_rowid(pTab->db);
        rc = zonefilePopulateIndex(pTab, zFile, iFileid);
      }
    }

    if( rc==SQLITE_OK ){

      int nKey = sqlite3_value_bytes(apVal[3]);
      const u8 *aKey = (const u8*)sqlite3_value_blob(apVal[3]);
      rc = zonefileKeyStore(
          pTab->pGlobal, pTab->zDb, pTab->zBase, iFileid, aKey, nKey
      );
    }
  }

  return rc;
}

/* Each entry in the frame-cache is represented by an instance of the
................................................................................
        rc = pCmpMethod->xOpen(&pCmp, aDict, nDict);
      }
      sqlite3_free(aDict);
    }

    /* Find the encryption method and key. */
    if( rc==SQLITE_OK && hdr.encryptionType ){
      const u8 *a = 0;
      int n = zonefileKeyFind(pTab->pGlobal, pTab->zDb, pTab->zName, iFile, &a);
      if( n==0 ){
        zErr = sqlite3_mprintf("missing encryption key for file \"%s\"", zFile);
        rc = SQLITE_ERROR;
      }else{
        rc = zonefileCodecCreate(hdr.encryptionType, 0, (u8*)a,n,&pCodec,&zErr);
      }
    }

    /* Read some data into memory. */
    if( rc==SQLITE_OK ){
      int szFrame = sqlite3_column_int(pCsr->pSelect, 3);

Changes to ext/zonefile/zonefile1.test.

216
217
218
219
220
221
222

223
224
225
226
227
228
229
...
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
  CREATE TABLE dd(k INTEGER PRIMARY KEY, frame INTEGER, idx INTEGER, v BLOB);
  INSERT INTO dd VALUES(1000, 1, -1, randomblob(44));
  INSERT INTO dd VALUES(1001, 1, -1, randomblob(55));
  INSERT INTO dd VALUES(1002, 2, -1, randomblob(66));
  WITH p(n,v) AS (
      VALUES('maxAutoFrameSize', 2000) UNION ALL
      VALUES('encryptionType', 'xor') UNION ALL

      VALUES('encryptionKey', '0123456789')
  )
  SELECT zonefile_write('test.zonefile', 'dd', json_group_object(n, v)) FROM p;
} {{}}

do_execsql_test 3.1 {
  CREATE VIRTUAL TABLE cc USING zonefile;
................................................................................
load_static_extension db zonefile
do_execsql_test 11.0 {
  CREATE TABLE data(k INTEGER PRIMARY KEY, frame, idx, v BLOB);
  INSERT INTO data VALUES(1, 1, -1, randomblob(200));
  INSERT INTO data VALUES(2, 2, -1, randomblob(200));
  INSERT INTO data VALUES(3, 3, -1, randomblob(200));
  SELECT zonefile_write('test.zonefile', 'data',
      '{"encryptionType":"xor","encryptionKey":"pass"}'
  );

  CREATE VIRTUAL TABLE nm USING zonefile(cachesize=2);
  INSERT INTO nm_files(filename,ekey) VALUES('test.zonefile','pass');
} {{}}

set i 0







>







 







|







216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
...
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
  CREATE TABLE dd(k INTEGER PRIMARY KEY, frame INTEGER, idx INTEGER, v BLOB);
  INSERT INTO dd VALUES(1000, 1, -1, randomblob(44));
  INSERT INTO dd VALUES(1001, 1, -1, randomblob(55));
  INSERT INTO dd VALUES(1002, 2, -1, randomblob(66));
  WITH p(n,v) AS (
      VALUES('maxAutoFrameSize', 2000) UNION ALL
      VALUES('encryptionType', 'xor') UNION ALL
      VALUES('debugEncryptionKeyText', 1) UNION ALL
      VALUES('encryptionKey', '0123456789')
  )
  SELECT zonefile_write('test.zonefile', 'dd', json_group_object(n, v)) FROM p;
} {{}}

do_execsql_test 3.1 {
  CREATE VIRTUAL TABLE cc USING zonefile;
................................................................................
load_static_extension db zonefile
do_execsql_test 11.0 {
  CREATE TABLE data(k INTEGER PRIMARY KEY, frame, idx, v BLOB);
  INSERT INTO data VALUES(1, 1, -1, randomblob(200));
  INSERT INTO data VALUES(2, 2, -1, randomblob(200));
  INSERT INTO data VALUES(3, 3, -1, randomblob(200));
  SELECT zonefile_write('test.zonefile', 'data',
  '{"encryptionType":"xor","encryptionKey":"pass","debugEncryptionKeyText":1}'
  );

  CREATE VIRTUAL TABLE nm USING zonefile(cachesize=2);
  INSERT INTO nm_files(filename,ekey) VALUES('test.zonefile','pass');
} {{}}

set i 0

Changes to ext/zonefile/zonefileenc.test.

13
14
15
16
17
18
19
20
21



22
23
24
25
26

27
























28

29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

46
47
48
49
50
51
52
53
54

55





56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79




80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110

111
112
113
114
115
116
117
...
121
122
123
124
125
126
127

128
129
130
131
132
133
134
135
136
137
138
139
140
141

142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182

183
184
185
186
187
188
189
190
191
192

193
194
195
196
197
198
199
#

if {![info exists testdir]} {
  set testdir [file join [file dirname [info script]] .. .. test]
}
source [file join $testdir tester.tcl]
set testprefix zonefileenc
load_static_extension db zonefile




set K {
  braking bramble brambles brambly
  bran branch branched branches
  branching branchings brand branded
}


























set nFile 100


do_execsql_test 1.0 {
  CREATE TABLE zz(k INTEGER PRIMARY KEY, frame INTEGER, idx INTEGER, v BLOB);
  CREATE TABLE rr(k INTEGER PRIMARY KEY, v);
}
do_test 1.1 {
  for {set i 0} {$i < $nFile} {incr i} {
    set k [lindex $K [expr $i % [llength $K]]]
    execsql {
      DELETE FROM zz;
      INSERT INTO zz VALUES($i*10+1, 1, -1, randomblob(100));
      INSERT INTO zz VALUES($i*10+2, 2, -1, randomblob(100));
      INSERT INTO zz VALUES($i*10+3, 1, -1, randomblob(100));
      INSERT INTO rr SELECT k,v FROM zz;

      WITH p(n,v) AS (
          VALUES('encryptionType', 'xor') UNION ALL

          VALUES('encryptionKey', $k)
      )
      SELECT zonefile_write('test' || $i || '.zonefile', 'zz', 
        json_group_object(n, v)
      ) FROM p;
    }
  }
} {}


proc k {i} { lindex $::K [expr $i % [llength $::K]] }





db func k k

do_execsql_test 1.2 {
  CREATE VIRTUAL TABLE gg USING zonefile;
}
for {set i 0} {$i < $nFile} {incr i} {
  do_execsql_test 1.2.$i { 
    INSERT INTO gg_files(filename, ekey) 
    VALUES('test' || $i || '.zonefile', k($i));
    SELECT count(*) FROM rr JOIN gg USING(k) WHERE rr.v!=gg.v;
  } 0
}

db close
sqlite3 db test.db
load_static_extension db zonefile
db func k k

do_catchsql_test 1.3 {
  SELECT count(*) FROM rr JOIN gg USING(k) WHERE rr.v!=gg.v;
} {1 {missing encryption key for file "test0.zonefile"}}
do_execsql_test 1.4 {
  UPDATE gg_files SET ekey = 'braking' WHERE filename='test0.zonefile';
}




do_catchsql_test 1.5 {
  SELECT count(*) FROM rr JOIN gg USING(k) WHERE rr.v!=gg.v;
} {1 {missing encryption key for file "test1.zonefile"}}

do_execsql_test 1.6 {
  UPDATE gg_files SET ekey = k(rowid-1);
}
do_execsql_test 1.7 {
  SELECT count(*) FROM rr JOIN gg USING(k) WHERE rr.v!=gg.v;
} {0}
do_execsql_test 1.8 {
  SELECT count(*) FROM rr JOIN gg USING(k) WHERE rr.v==gg.v;
} {300}

forcedelete test.db2
do_execsql_test 1.9.1 {
  ATTACH 'test.db2' AS maing;
  CREATE VIRTUAL TABLE maing.g USING zonefile;
  INSERT INTO g_files(filename) SELECT filename FROM gg_files;
}
do_catchsql_test 1.9.2 {
  SELECT count(*) FROM rr JOIN g USING(k) WHERE rr.v!=g.v;
} {1 {missing encryption key for file "test0.zonefile"}}
do_execsql_test 1.9.3 {
  UPDATE g_files SET ekey = k(rowid-1);
  SELECT count(*) FROM rr JOIN g USING(k) WHERE rr.v==g.v;
} {300}

do_execsql_test 1.10 {
  SELECT count(*) FROM rr JOIN gg USING(k) WHERE rr.v==gg.v;
} {300}

#-------------------------------------------------------------------------

reset_db
load_static_extension db zonefile

do_execsql_test 2.0 {
  CREATE TABLE zz(k INTEGER PRIMARY KEY, frame INTEGER, idx INTEGER, v BLOB);
................................................................................
  2 aes_128_cbc 2
  3 AES_256_CTR 3
  4 Aes_256_CBC 4
} {
  do_catchsql_test 2.1.$tn {
    WITH p(n,v) AS (
        VALUES('encryptionType', $alg) UNION ALL

        VALUES('encryptionKey', 'secret')
    )
    SELECT zonefile_write('test' || $i || '.zonefile', 'zz', 
          json_group_object(n, v)
    ) FROM p;
  } "1 {unsupported encryption method: $id}"
}

foreach {tn alg} {
  1 nosuchmethod! 
} {
  do_catchsql_test 2.1.$tn {
    WITH p(n,v) AS (
        VALUES('encryptionType', $alg) UNION ALL

        VALUES('encryptionKey', 'secret')
    )
    SELECT zonefile_write('test' || $i || '.zonefile', 'zz', 
          json_group_object(n, v)
    ) FROM p;
  } "1 {unknown encryption method: $alg}"
}

#-------------------------------------------------------------------------
# Test some hash collisions in the encryption key table.
#

# This is the same hash function used internally to store keys.
#
proc hash {zDb zTab iFile} {
  binary scan $zDb c*  A
  binary scan $zTab c* B
  set h 0
  foreach i $A { set h [expr ($h + ($h << 3) + $i) & 0xFFFFFFFF] }
  foreach i $B { set h [expr ($h + ($h << 3) + $i) & 0xFFFFFFFF] }
  return [expr $h ^ $iFile]
}

do_test 3.0 {
  set h1 [expr [hash main zone 1] % 512]
  for {set i 0} {1} {incr i} {
    set h2 [expr [hash "aux$i" zone 1] % 512]
    if {$h1==$h2} break
  }
  set i
} 52

reset_db
load_static_extension db zonefile
forcedelete test.db2

do_execsql_test 3.1 {
  CREATE TABLE zz(k INTEGER PRIMARY KEY, frame INTEGER, idx INTEGER, v BLOB);
  INSERT INTO zz VALUES(222, -1, -1, randomblob(60));
  WITH p(n,v) AS (
      VALUES('encryptionType', 'xor') UNION ALL

      VALUES('encryptionKey', 'pass')
  )
  SELECT zonefile_write('test1.zonefile', 'zz', 
      json_group_object(n, v)
  ) FROM p;

  DELETE FROM zz;
  INSERT INTO zz VALUES(333, -1, -1, randomblob(80));
  WITH p(n,v) AS (
      VALUES('encryptionType', 'xor') UNION ALL

      VALUES('encryptionKey', 'pass')
  )
  SELECT zonefile_write('test2.zonefile', 'zz', 
      json_group_object(n, v)
  ) FROM p;
} {{} {}}








<

>
>
>
|
|
|
|
|
>
|
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|
>

|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
>
|
|
|
|
|
|
|
|
|
>
|
>
>
>
>
>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
>
>
>
>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
>







 







>



|










>



|












|
|
|






|
|
|
|












>










>







13
14
15
16
17
18
19

20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
...
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
#

if {![info exists testdir]} {
  set testdir [file join [file dirname [info script]] .. .. test]
}
source [file join $testdir tester.tcl]
set testprefix zonefileenc



foreach {tn code} {
  1 { 
    set K {
      braking bramble brambles brambly
      bran branch branched branches
      branching branchings brand branded
    }
    set textkey 1
  }
  2 {
    set K {
      5e008542742ce0442e37cbf2512e9492    c91c26e0573ca3464e037568c51126da
      e90e17489c1aef80ac620c9059271a5a    163338707cbe4c72b18d1058a42c5c78
      5c6b1e7c7c9e8e4a8d8fdc30dfc11bea    ff1012687828ecaac6c9ca86ea0f895e
      a203f25eb11d4c6afa841dfcf7cd0be0    b6c71e38ca914c460926ef90db39dba0
      b38255d031d026c258a0a41a9a75d46a    adccca5e5ffa3a7625144a345713aef0
      cd423b38b73e42ce5894405e6d0e08c0    b460ad2e370a0386726d6ea46e7b0bac
      503b81de72cb3ef87d9346a850040000    369c290a464a6b88bfd9d1c4755afd42
      a8a9343efca528f2bf23a972be49dd66    e366b5226bfe3fd0010fa814aae3b996
      4cad7e80124c2cd447131bae377e60f6    4a0fd2f054e1b08cad0de2dc6aa93246
      8a23c85e3337da2c97d498f806870fa8    8d14e1f055fd9bec7d07cf0e8baae042
      7f6954b0dc373028ab3b030aaf44dd58    d220164c3898435a946de6bcbb478cc4
      566af7ea88ba4ff87fd868e858cf98ea    a5405832235e8f601516f9c49767bdac
      1bd5b4dc6b54e5ca92ba67d20bf65740    59da30e203bf73840e38e108b83ddb82
      e516924c2cdf3114f10f2f0e1bdabbc6    b55dd27222a39764222838007e749984
      190ae9f81b86a5a024e3b97ee2a7121c    469660843a9a9e507d0fb43e92029296
      e6e600bc063aad12f6387beef650c48a    3097be5c3a52a2f00747587add01b550
    }
    set textkey 0
  }
} {
  reset_db
  load_static_extension db zonefile
  set nFile 100
  eval $code

  do_execsql_test 1.$tn.0 {
    CREATE TABLE zz(k INTEGER PRIMARY KEY, frame INTEGER, idx INTEGER, v BLOB);
    CREATE TABLE rr(k INTEGER PRIMARY KEY, v);
  }
  do_test 1.$tn.1 {
    for {set i 0} {$i < $nFile} {incr i} {
      set k [lindex $K [expr $i % [llength $K]]]
      execsql {
        DELETE FROM zz;
        INSERT INTO zz VALUES($i*10+1, 1, -1, randomblob(100));
        INSERT INTO zz VALUES($i*10+2, 2, -1, randomblob(100));
        INSERT INTO zz VALUES($i*10+3, 1, -1, randomblob(100));
        INSERT INTO rr SELECT k,v FROM zz;
  
        WITH p(n,v) AS (
            VALUES('encryptionType', 'xor') UNION ALL
            VALUES('debugEncryptionKeyText', $textkey) UNION ALL
            VALUES('encryptionKey', $k)
        )
        SELECT zonefile_write('test' || $i || '.zonefile', 'zz', 
            json_group_object(n, v)
        ) FROM p;
      }
    }
  } {}
  
  proc k {i} { 
    set val [lindex $::K [expr $i % [llength $::K]]]
    if {$::textkey==0} {
      return [binary decode hex $val]
    }
    return $val
  }
  db func k k
  
  do_execsql_test 1.$tn.2 {
    CREATE VIRTUAL TABLE gg USING zonefile;
  }
  for {set i 0} {$i < $nFile} {incr i} {
    do_execsql_test 1.$tn.2.$i { 
      INSERT INTO gg_files(filename, ekey) 
        VALUES('test' || $i || '.zonefile', k($i));
      SELECT count(*) FROM rr JOIN gg USING(k) WHERE rr.v!=gg.v;
    } 0
  }
  
  db close
  sqlite3 db test.db
  load_static_extension db zonefile
  db func k k
  
  do_catchsql_test 1.$tn.3 {
    SELECT count(*) FROM rr JOIN gg USING(k) WHERE rr.v!=gg.v;
  } {1 {missing encryption key for file "test0.zonefile"}}
  do_execsql_test 1.$tn.4 {
    UPDATE gg_files SET ekey = k(0) WHERE filename='test0.zonefile';
  }
  do_execsql_test 1.$tn.4.2 {
    SELECT count(*) FROM rr JOIN gg USING(k) 
      WHERE rr.v==gg.v AND k IN (1,2,3);
  } {3}
  do_catchsql_test 1.5 {
    SELECT count(*) FROM rr JOIN gg USING(k) WHERE rr.v!=gg.v;
  } {1 {missing encryption key for file "test1.zonefile"}}
  
  do_execsql_test 1.$tn.6 {
    UPDATE gg_files SET ekey = k(rowid-1);
  }
  do_execsql_test 1.$tn.7 {
    SELECT count(*) FROM rr JOIN gg USING(k) WHERE rr.v!=gg.v;
  } {0}
  do_execsql_test 1.$tn.8 {
    SELECT count(*) FROM rr JOIN gg USING(k) WHERE rr.v==gg.v;
  } {300}
  
  forcedelete test.db2
  do_execsql_test 1.$tn.9.1 {
    ATTACH 'test.db2' AS maing;
    CREATE VIRTUAL TABLE maing.g USING zonefile;
    INSERT INTO g_files(filename) SELECT filename FROM gg_files;
  }
  do_catchsql_test 1.$tn.9.2 {
    SELECT count(*) FROM rr JOIN g USING(k) WHERE rr.v!=g.v;
  } {1 {missing encryption key for file "test0.zonefile"}}
  do_execsql_test 1.$tn.9.3 {
    UPDATE g_files SET ekey = k(rowid-1);
    SELECT count(*) FROM rr JOIN g USING(k) WHERE rr.v==g.v;
  } {300}
  
  do_execsql_test 1.$tn.10 {
    SELECT count(*) FROM rr JOIN gg USING(k) WHERE rr.v==gg.v;
  } {300}
}
#-------------------------------------------------------------------------

reset_db
load_static_extension db zonefile

do_execsql_test 2.0 {
  CREATE TABLE zz(k INTEGER PRIMARY KEY, frame INTEGER, idx INTEGER, v BLOB);
................................................................................
  2 aes_128_cbc 2
  3 AES_256_CTR 3
  4 Aes_256_CBC 4
} {
  do_catchsql_test 2.1.$tn {
    WITH p(n,v) AS (
        VALUES('encryptionType', $alg) UNION ALL
        VALUES('debugEncryptionKeyText', 1) UNION ALL
        VALUES('encryptionKey', 'secret')
    )
    SELECT zonefile_write('test' || $i || '.zonefile', 'zz', 
        json_group_object(n, v)
    ) FROM p;
  } "1 {unsupported encryption method: $id}"
}

foreach {tn alg} {
  1 nosuchmethod! 
} {
  do_catchsql_test 2.1.$tn {
    WITH p(n,v) AS (
        VALUES('encryptionType', $alg) UNION ALL
        VALUES('debugEncryptionKeyText', 1) UNION ALL
        VALUES('encryptionKey', 'secret')
    )
    SELECT zonefile_write('test' || $i || '.zonefile', 'zz', 
        json_group_object(n, v)
    ) FROM p;
  } "1 {unknown encryption method: $alg}"
}

#-------------------------------------------------------------------------
# Test some hash collisions in the encryption key table.
#

# This is the same hash function used internally to store keys.
#
proc hash {zDb zTab iFile} {
  binary scan $zDb c*  A
    binary scan $zTab c* B
    set h 0
    foreach i $A { set h [expr ($h + ($h << 3) + $i) & 0xFFFFFFFF] }
  foreach i $B { set h [expr ($h + ($h << 3) + $i) & 0xFFFFFFFF] }
  return [expr $h ^ $iFile]
}

do_test 3.0 {
  set h1 [expr [hash main zone 1] % 512]
    for {set i 0} {1} {incr i} {
      set h2 [expr [hash "aux$i" zone 1] % 512]
        if {$h1==$h2} break
    }
  set i
} 52

reset_db
load_static_extension db zonefile
forcedelete test.db2

do_execsql_test 3.1 {
  CREATE TABLE zz(k INTEGER PRIMARY KEY, frame INTEGER, idx INTEGER, v BLOB);
  INSERT INTO zz VALUES(222, -1, -1, randomblob(60));
  WITH p(n,v) AS (
      VALUES('encryptionType', 'xor') UNION ALL
      VALUES('debugEncryptionKeyText', 1) UNION ALL
      VALUES('encryptionKey', 'pass')
  )
  SELECT zonefile_write('test1.zonefile', 'zz', 
      json_group_object(n, v)
  ) FROM p;

  DELETE FROM zz;
  INSERT INTO zz VALUES(333, -1, -1, randomblob(80));
  WITH p(n,v) AS (
      VALUES('encryptionType', 'xor') UNION ALL
      VALUES('debugEncryptionKeyText', 1) UNION ALL
      VALUES('encryptionKey', 'pass')
  )
  SELECT zonefile_write('test2.zonefile', 'zz', 
      json_group_object(n, v)
  ) FROM p;
} {{} {}}

Changes to ext/zonefile/zonefilefault.test.

30
31
32
33
34
35
36
37


38
39
40
41
42
43
44
...
114
115
116
117
118
119
120
121


122
123
124
125
126
127
128
...
201
202
203
204
205
206
207
208

209
210
211

212



213
214
215
216
217
218
219

do_faultsim_test 1.1 -faults oom* -prep {
  sqlite3 db test.db
  load_static_extension db zonefile
} -body {
  execsql {
    SELECT zonefile_write('test.zonefile', 'tt',
        '{"encryptionType":"xor", "encryptionKey":"secret"}'


    );
  }
} -test {
  faultsim_test_result {0 {{}}}
}

set sql {
................................................................................
  }
}

#-------------------------------------------------------------------------
#
do_execsql_test 2.0 {
  SELECT zonefile_write('test.zonefile', 'tt',
      '{"encryptionType":"xor", "encryptionKey":"secret"}'


  );
  CREATE VIRTUAL TABLE zz USING zonefile;
} {{}}

faultsim_save_and_close
do_faultsim_test 2.1 -faults oom* -prep {
  faultsim_restore_and_reopen
................................................................................
  faultsim_test_result {0 0}
}


if {$HAVE_ZSTD} {
  set params {
      {"encryptionType":"xor","encryptionKey":"pass",
       "compressionTypeContent":"zstd_global_dict"

      }
  }
} else {

  set params { {"encryptionType":"xor","encryptionKey":"pass" } }



}
do_execsql_test 4.2 {
  SELECT zonefile_write('test.zonefile', 'zz', $params);
  CREATE VIRTUAL TABLE zone2 USING zonefile;
  INSERT INTO zone2_files(filename,ekey) VALUES('test.zonefile','pass');
} {{}}








|
>
>







 







|
>
>







 







|
>



>
|
>
>
>







30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
...
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
...
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228

do_faultsim_test 1.1 -faults oom* -prep {
  sqlite3 db test.db
  load_static_extension db zonefile
} -body {
  execsql {
    SELECT zonefile_write('test.zonefile', 'tt',
        '{"encryptionType":"xor", "encryptionKey":"secret",
          "debugEncryptionKeyText":1
        }'
    );
  }
} -test {
  faultsim_test_result {0 {{}}}
}

set sql {
................................................................................
  }
}

#-------------------------------------------------------------------------
#
do_execsql_test 2.0 {
  SELECT zonefile_write('test.zonefile', 'tt',
      '{"encryptionType":"xor", "encryptionKey":"secret",
          "debugEncryptionKeyText":1
      }'
  );
  CREATE VIRTUAL TABLE zz USING zonefile;
} {{}}

faultsim_save_and_close
do_faultsim_test 2.1 -faults oom* -prep {
  faultsim_restore_and_reopen
................................................................................
  faultsim_test_result {0 0}
}


if {$HAVE_ZSTD} {
  set params {
      {"encryptionType":"xor","encryptionKey":"pass",
       "compressionTypeContent":"zstd_global_dict",
          "debugEncryptionKeyText":1
      }
  }
} else {
  set params { 
      {"encryptionType":"xor","encryptionKey":"pass",
          "debugEncryptionKeyText":1
      }
  }
}
do_execsql_test 4.2 {
  SELECT zonefile_write('test.zonefile', 'zz', $params);
  CREATE VIRTUAL TABLE zone2 USING zonefile;
  INSERT INTO zone2_files(filename,ekey) VALUES('test.zonefile','pass');
} {{}}