/ Check-in [2402baa0]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Add tests for another application writing the database while an ota update is ongoing.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | ota-update
Files: files | file ages | folders
SHA1:2402baa0027ca65e9a5106b8b9c4e10f40d2cbc3
User & Date: dan 2014-10-21 18:09:58
Context
2014-10-21
19:35
Test that sqlite3ota_open() works with URI paths. Fix some other issues. check-in: 6fd09854 user: dan tags: ota-update
18:09
Add tests for another application writing the database while an ota update is ongoing. check-in: 2402baa0 user: dan tags: ota-update
2014-10-20
16:34
Merge version-3.8.7 changes with this branch. check-in: d380a648 user: dan tags: ota-update
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to ext/ota/README.txt.

15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

54







55
56
57
58
59
60
61




62
63
64
65
66
67
68

  * The above statements do not check UNIQUE constraints - except those enforced
    by the main b-tree.

  * All non-temporary triggers are disabled.


2) PRAGMA pager_ota_mode:

  This pragma sets a flag on the pager associated with the main database only. In
  a zipvfs system, this pragma is intercepted by zipvfs and the flag is set on
  the lower level pager only.

  The flag can only be set when there is no open transaction and the pager does
  not already have an open WAL file. Attempting to do so is an error.

  Once the flag has been set, it is not possible to open a regular WAL file.
  If, when the next read-transaction is opened, a *-wal file is found or the
  database header flags indicate that it is a wal-mode database,
  SQLITE_CANTOPEN is returned.

  Otherwise, if no WAL file or flags are found, the pager opens the *-oal file
  and uses it as a write-ahead-log with the *-shm data stored in heap-memory.

  The 8-bytes of "salt" at teh start of an *-oal file is a copy of the 8 bytes
  starting at offset 24 of the database file header (the change counter and the
  number of pages in the file). If the *-oal file already exists when it is
  opened, SQLite checks that the salt still matches the database header fields.
  If not, it concludes that the database file has been written by a rollback-mode
  client since the *-oal wa created and an SQLITE_BUSY_SNAPSHOT error is
  returned. No read-transaction can be opened in this case.

  A pager with the pager_ota_mode flag set never runs a checkpoint.

  Other clients see a rollback-mode database on which the pager_ota_mode client
  is holding a SHARED lock. There are no locks to arbitrate between multiple
  pager_ota_mode connections. If two or more such connections attempt to write
  simultaneously, the results are undefined.










3) sqlite3_index_writer()

  This new API function is used to create VMs that can insert or delete entries
  from individual index b-trees within the database. The VMs apply affinities
  and check that UNIQUE constraints are not violated before updating index
  b-trees.






The OTA extension
-----------------

The OTA extension requires that the OTA update be packaged as an SQLite
database. The tables it expects to find are described in sqlite3ota.h.
Essentially, for each table xyz in the target database that the user wishes







|

|
|
|












|



|
|
|








>

>
>
>
>
>
>
>
|






>
>
>
>







15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80

  * The above statements do not check UNIQUE constraints - except those enforced
    by the main b-tree.

  * All non-temporary triggers are disabled.


2) PRAGMA pager_ota_mode=1:

  This pragma sets a flag on the pager associated with the main database only.
  In a zipvfs system, this pragma is intercepted by zipvfs and the flag is set
  on the lower level pager only.

  The flag can only be set when there is no open transaction and the pager does
  not already have an open WAL file. Attempting to do so is an error.

  Once the flag has been set, it is not possible to open a regular WAL file.
  If, when the next read-transaction is opened, a *-wal file is found or the
  database header flags indicate that it is a wal-mode database,
  SQLITE_CANTOPEN is returned.

  Otherwise, if no WAL file or flags are found, the pager opens the *-oal file
  and uses it as a write-ahead-log with the *-shm data stored in heap-memory.

  The 8-bytes of "salt" at the start of an *-oal file is a copy of the 8 bytes
  starting at offset 24 of the database file header (the change counter and the
  number of pages in the file). If the *-oal file already exists when it is
  opened, SQLite checks that the salt still matches the database header fields.
  If not, it concludes that the database file has been written by a
  rollback-mode client since the *-oal wa created and an SQLITE_BUSY_SNAPSHOT
  error is returned. No read-transaction can be opened in this case.

  A pager with the pager_ota_mode flag set never runs a checkpoint.

  Other clients see a rollback-mode database on which the pager_ota_mode client
  is holding a SHARED lock. There are no locks to arbitrate between multiple
  pager_ota_mode connections. If two or more such connections attempt to write
  simultaneously, the results are undefined.

3) PRAGMA pager_ota_mode=2:

  The pager_ota_mode pragma may also be set to 2 if the main database is open 
  in WAL mode. This prevents SQLite from checkpointing the wal file as part
  of sqlite3_close().

  The effects of setting pager_ota_mode=2 if the db is not in WAL mode are
  undefined.

4) sqlite3_index_writer()

  This new API function is used to create VMs that can insert or delete entries
  from individual index b-trees within the database. The VMs apply affinities
  and check that UNIQUE constraints are not violated before updating index
  b-trees.

5) sqlite3_ckpt_open/step/close()

  API for performing (and resuming) incremental checkpoints.


The OTA extension
-----------------

The OTA extension requires that the OTA update be packaged as an SQLite
database. The tables it expects to find are described in sqlite3ota.h.
Essentially, for each table xyz in the target database that the user wishes

Added ext/ota/ota6.test.

































































































































































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
# 2014 October 21
#
# The author disclaims copyright to this source code.  In place of
# a legal notice, here is a blessing:
#
#    May you do good and not evil.
#    May you find forgiveness for yourself and forgive others.
#    May you share freely, never taking more than you give.
#
#***********************************************************************
#
# This file contains tests for the OTA module. Specifically, it tests the
# outcome of some other client writing to the database while an OTA update
# is being applied.

if {![info exists testdir]} {
  set testdir [file join [file dirname [info script]] .. .. test]
}
source $testdir/tester.tcl
set ::testprefix ota6

proc setup_test {} {
  reset_db
  execsql {
    CREATE TABLE t1(a INTEGER PRIMARY KEY, b UNIQUE);
    CREATE TABLE t2(a INTEGER PRIMARY KEY, b UNIQUE);
    CREATE TABLE t3(a INTEGER PRIMARY KEY, b UNIQUE);
  }
  db close

  forcedelete ota.db
  sqlite3 ota ota.db
  ota eval {
    CREATE TABLE data_t1(a, b, ota_control);
    CREATE TABLE data_t2(a, b, ota_control);
    CREATE TABLE data_t3(a, b, ota_control);
    INSERT INTO data_t1 VALUES(1, 't1', 0);
    INSERT INTO data_t2 VALUES(2, 't2', 0);
    INSERT INTO data_t3 VALUES(3, 't3', 0);
  }
  ota close
}

# Test the outcome of some other client writing the db while the *-oal 
# file is being generated. Once this has happened, the update cannot be
# progressed.
#
for {set nStep 1} {$nStep < 7} {incr nStep} {
  do_test 1.$nStep.1 {
    setup_test
    sqlite3ota ota test.db ota.db
    for {set i 0} {$i<$nStep} {incr i} {ota step}
  
    ota close
    sqlite3 db test.db
    execsql { INSERT INTO t1 VALUES(5, 'hello') }
    sqlite3ota ota test.db ota.db
    ota step
  } {SQLITE_BUSY}
  do_test 1.$nStep.2 {
    ota step
  } {SQLITE_BUSY}
  do_test 1.$nStep.3 {
    list [file exists test.db-oal] [file exists test.db-wal]
  } {1 0}
  do_test 1.$nStep.4 {
    list [catch { ota close } msg] $msg
  } {1 {SQLITE_BUSY - database is locked}}
}

for {set nStep 7} {$nStep < 8} {incr nStep} {
  do_test 1.$nStep.1 {
    setup_test
    sqlite3ota ota test.db ota.db
    for {set i 0} {$i<$nStep} {incr i} {ota step}
    ota close
    sqlite3 db test.db
    execsql { INSERT INTO t1 VALUES(5, 'hello') }
    sqlite3ota ota test.db ota.db
    ota step
  } {SQLITE_OK}
  do_test 1.$nStep.2 {
    ota step
  } {SQLITE_OK}
  do_test 1.$nStep.3 {
    list [file exists test.db-oal] [file exists test.db-wal]
  } {0 1}
  do_test 1.$nStep.4 {
    list [catch { ota close } msg] $msg
  } {0 SQLITE_OK}
}


# Test the outcome of some other client writing the db after the *-oal
# file has been copied to the *-wal path. Once this has happened, any
# other client writing to the db causes OTA to consider its job finished.
#
for {set nStep 8} {$nStep < 20} {incr nStep} {
  do_test 1.$nStep.1 {
    setup_test
    sqlite3ota ota test.db ota.db
    for {set i 0} {$i<$nStep} {incr i} {ota step}
    ota close
    sqlite3 db test.db
    execsql { INSERT INTO t1 VALUES(5, 'hello') }
    sqlite3ota ota test.db ota.db
    ota step
  } {SQLITE_DONE}
  do_test 1.$nStep.2 {
    ota step
  } {SQLITE_DONE}
  do_test 1.$nStep.3 {
    file exists test.db-oal
  } {0}
  do_test 1.$nStep.4 {
    list [catch { ota close } msg] $msg
  } {0 SQLITE_DONE}

  do_execsql_test 1.$nStep.5 {
    SELECT * FROM t1;
  } {1 t1 5 hello}
}



finish_test


Changes to ext/ota/sqlite3ota.c.

1299
1300
1301
1302
1303
1304
1305
1306
1307

1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328







1329
1330
1331
1332
1333
1334
1335
          p->eStage = pState->eStage;
        }
        p->nProgress = pState->nProgress;
      }
    }
    assert( p->rc!=SQLITE_OK || p->eStage!=0 );

    if( p->eStage==OTA_STAGE_OAL ){
      if( p->rc==SQLITE_OK ){

        const char *zScript =
          "PRAGMA journal_mode=off;"
          "PRAGMA pager_ota_mode=1;"
          "PRAGMA ota_mode=1;"
          "BEGIN IMMEDIATE;"
          ;
        p->rc = sqlite3_exec(p->db, zScript, 0, 0, &p->zErrmsg);
      }

      /* Point the object iterator at the first object */
      if( p->rc==SQLITE_OK ){
        p->rc = otaObjIterFirst(p, &p->objiter);
      }

      if( p->rc==SQLITE_OK ){
        otaLoadTransactionState(p, pState);
      }
    }else if( p->rc==SQLITE_OK && p->eStage==OTA_STAGE_CKPT ){
      p->rc = sqlite3_ckpt_open(
          p->db, pState->pCkptState, pState->nCkptState, &p->pCkpt
      );







    }

    otaFreeState(pState);
  }

  return p;
}







<
|
>





|

|
<
|
|
|
|
|
|
|
|
|
|
|
|
>
>
>
>
>
>
>







1299
1300
1301
1302
1303
1304
1305

1306
1307
1308
1309
1310
1311
1312
1313
1314
1315

1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
          p->eStage = pState->eStage;
        }
        p->nProgress = pState->nProgress;
      }
    }
    assert( p->rc!=SQLITE_OK || p->eStage!=0 );


    if( p->rc==SQLITE_OK ){
      if( p->eStage==OTA_STAGE_OAL ){
        const char *zScript =
          "PRAGMA journal_mode=off;"
          "PRAGMA pager_ota_mode=1;"
          "PRAGMA ota_mode=1;"
          "BEGIN IMMEDIATE;"
        ;
        p->rc = sqlite3_exec(p->db, zScript, 0, 0, &p->zErrmsg);
  

        /* Point the object iterator at the first object */
        if( p->rc==SQLITE_OK ){
          p->rc = otaObjIterFirst(p, &p->objiter);
        }
  
        if( p->rc==SQLITE_OK ){
          otaLoadTransactionState(p, pState);
        }
      }else if( p->eStage==OTA_STAGE_CKPT ){
        p->rc = sqlite3_ckpt_open(
            p->db, pState->pCkptState, pState->nCkptState, &p->pCkpt
        );
        if( p->rc==SQLITE_MISMATCH ){
          p->eStage = OTA_STAGE_DONE;
          p->rc = SQLITE_DONE;
        }
      }else if( p->eStage==OTA_STAGE_DONE ){
        p->rc = SQLITE_DONE;
      }
    }

    otaFreeState(pState);
  }

  return p;
}