/ Check-in [630fc71f]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Fix a problem in the test scripts for the asynchronous backend. (CVS 4400)
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 630fc71f3df5ab6129ddff9d8184893ecc6cf3c5
User & Date: danielk1977 2007-09-05 11:34:54
Context
2007-09-05
13:56
Remove the unixFile.isOpen variable (no longer in use). (CVS 4401) check-in: 1786e9c8 user: danielk1977 tags: trunk
11:34
Fix a problem in the test scripts for the asynchronous backend. (CVS 4400) check-in: 630fc71f user: danielk1977 tags: trunk
2007-09-04
22:31
Do not use the TryEnterCriticalSection API on windows since it is unavailable on some platforms. (CVS 4399) check-in: bf3d67d1 user: drh tags: trunk
Changes
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Changes to src/test_async.c.

    20     20   ** a database write does not have to wait for (sometimes slow) disk I/O
    21     21   ** to occur.  The write seems to happen very quickly, though in reality
    22     22   ** it is happening at its usual slow pace in the background.
    23     23   **
    24     24   ** Asynchronous I/O appears to give better responsiveness, but at a price.
    25     25   ** You lose the Durable property.  With the default I/O backend of SQLite,
    26     26   ** once a write completes, you know that the information you wrote is
    27         -** safely on disk.  With the asynchronous I/O, this is no the case.  If
    28         -** your program crashes or if you take a power lose after the database
           27  +** safely on disk.  With the asynchronous I/O, this is not the case.  If
           28  +** your program crashes or if a power lose occurs after the database
    29     29   ** write but before the asynchronous write thread has completed, then the
    30     30   ** database change might never make it to disk and the next user of the
    31     31   ** database might not see your change.
    32     32   **
    33     33   ** You lose Durability with asynchronous I/O, but you still retain the
    34     34   ** other parts of ACID:  Atomic,  Consistent, and Isolated.  Many
    35     35   ** appliations get along fine without the Durablity.
    36     36   **
    37     37   ** HOW IT WORKS
    38     38   **
    39         -** Asynchronous I/O works by overloading the OS-layer disk I/O routines
    40         -** with modified versions that store the data to be written in queue of
    41         -** pending write operations.  Look at the asyncEnable() subroutine to see
    42         -** how overloading works.  Six os-layer routines are overloaded:
           39  +** Asynchronous I/O works by creating a special SQLite "vfs" structure
           40  +** and registering it with sqlite3_vfs_register(). When files opened via 
           41  +** this vfs are written to (using sqlite3OsWrite()), the data is not 
           42  +** written directly to disk, but is placed in the "write-queue" to be
           43  +** handled by the background thread.
    43     44   **
    44         -**     sqlite3OsOpenReadWrite;
    45         -**     sqlite3OsOpenReadOnly;
    46         -**     sqlite3OsOpenExclusive;
    47         -**     sqlite3OsDelete;
    48         -**     sqlite3OsFileExists;
    49         -**     sqlite3OsSyncDirectory;
    50         -**
    51         -** The original implementations of these routines are saved and are
    52         -** used by the writer thread to do the real I/O.  The substitute
    53         -** implementations typically put the I/O operation on a queue
    54         -** to be handled later by the writer thread, though read operations
    55         -** must be handled right away, obviously.
    56         -**
    57         -** Asynchronous I/O is disabled by setting the os-layer interface routines
    58         -** back to their original values.
           45  +** The special vfs is registered (and unregistered) by calls to 
           46  +** function asyncEnable() (see below).
    59     47   **
    60     48   ** LIMITATIONS
    61     49   **
    62     50   ** This demonstration code is deliberately kept simple in order to keep
    63     51   ** the main ideas clear and easy to understand.  Real applications that
    64     52   ** want to do asynchronous I/O might want to add additional capabilities.
    65     53   ** For example, in this demonstration if writes are happening at a steady
................................................................................
    69     57   ** the quantity of pending writes and stop accepting new write requests
    70     58   ** when the buffer gets to be too big.
    71     59   */
    72     60   
    73     61   /* 
    74     62   ** If this symbol is defined, then file-system locks are obtained as
    75     63   ** required. This slows things down, but allows multiple processes
    76         -** to access the database concurrently.
           64  +** to access the database concurrently. If this symbol is not defined,
           65  +** then connections from within a single process will respect each
           66  +** others database locks, but external connections will not - leading
           67  +** to database corruption.
    77     68   */
    78     69   #define ENABLE_FILE_LOCKING
    79     70   
    80     71   #include "sqliteInt.h"
    81     72   #include <tcl.h>
    82     73   
    83     74   /*
................................................................................
   121    112   
   122    113   /*
   123    114   ** THREAD SAFETY NOTES
   124    115   **
   125    116   ** Basic rules:
   126    117   **
   127    118   **     * Both read and write access to the global write-op queue must be 
   128         -**       protected by the async.queueMutex.
          119  +**       protected by the async.queueMutex. As are the async.ioError and
          120  +**       async.nFile variables.
          121  +**
          122  +**     * The async.aLock hash-table and all AsyncLock and AsyncFileLock
          123  +**       structures must be protected by teh async.lockMutex mutex.
   129    124   **
   130    125   **     * The file handles from the underlying system are assumed not to 
   131    126   **       be thread safe.
   132    127   **
   133    128   **     * See the last two paragraphs under "The Writer Thread" for
   134    129   **       an assumption to do with file-handle synchronization by the Os.
   135    130   **
................................................................................
   304    299     int op;                      /* One of ASYNC_xxx etc. */
   305    300     i64 iOffset;        /* See above */
   306    301     int nByte;          /* See above */
   307    302     char *zBuf;         /* Data to write to file (or NULL if op!=ASYNC_WRITE) */
   308    303     AsyncWrite *pNext;  /* Next write operation (to any file) */
   309    304   };
   310    305   
          306  +/*
          307  +** An instance of this structure is created for each distinct open file 
          308  +** (i.e. if two handles are opened on the one file, only one of these
          309  +** structures is allocated) and stored in the async.aLock hash table. The
          310  +** keys for async.aLock are the full pathnames of the opened files.
          311  +**
          312  +** AsyncLock.pList points to the head of a linked list of AsyncFileLock
          313  +** structures, one for each handle currently open on the file.
          314  +**
          315  +** If the opened file is not a main-database (the SQLITE_OPEN_MAIN_DB is
          316  +** not passed to the sqlite3OsOpen() call), or if ENABLE_FILE_LOCKING is 
          317  +** not defined at compile time, variables AsyncLock.pFile and 
          318  +** AsyncLock.eLock are never used. Otherwise, pFile is a file handle
          319  +** opened on the file in question and used to obtain the file-system 
          320  +** locks required by database connections within this process.
          321  +**
          322  +** See comments above the asyncLock() function for more details on 
          323  +** the implementation of database locking used by this backend.
          324  +*/
          325  +struct AsyncLock {
          326  +  sqlite3_file *pFile;
          327  +  int eLock;
          328  +  AsyncFileLock *pList;
          329  +};
          330  +
   311    331   /*
   312    332   ** An instance of the following structure is allocated along with each
   313    333   ** AsyncFileData structure (see AsyncFileData.lock), but is only used if the
   314    334   ** file was opened with the SQLITE_OPEN_MAIN_DB.
   315         -**
   316         -** The global async.aLock[] hash table maps from database file-name to a
   317         -** linked-list of AsyncFileLock structures corresponding to handles opened on
   318         -** the file. The AsyncFileLock structures are linked into the list when the
   319         -** file is opened and removed when it is closed. Mutex async.lockMutex must be
   320         -** held before accessing any AsyncFileLock structure or the async.aLock[]
   321         -** table.
   322    335   */
   323    336   struct AsyncFileLock {
   324    337     int eLock;                /* Internally visible lock state (sqlite pov) */
   325    338     int eAsyncLock;           /* Lock-state with write-queue unlock */
   326    339     AsyncFileLock *pNext;
   327    340   };
   328    341   
   329         -struct AsyncLock {
   330         -  sqlite3_file *pFile;
   331         -  int eLock;
   332         -  AsyncFileLock *pList;
   333         -};
   334         -
   335    342   /* 
   336    343   ** The AsyncFile structure is a subclass of sqlite3_file used for 
   337    344   ** asynchronous IO. 
   338    345   **
   339    346   ** All of the actual data for the structure is stored in the structure
   340    347   ** pointed to by AsyncFile.pData, which is allocated as part of the
   341    348   ** sqlite3OsOpen() using sqlite3_malloc(). The reason for this is that the
................................................................................
   607    614     int eRequired = 0;
   608    615   
   609    616     if( pLock->pFile ){
   610    617       for(pIter=pLock->pList; pIter; pIter=pIter->pNext){
   611    618         assert(pIter->eAsyncLock>=pIter->eLock);
   612    619         if( pIter->eAsyncLock>eRequired ){
   613    620           eRequired = pIter->eAsyncLock;
          621  +        assert(eRequired>=0 && eRequired<=SQLITE_LOCK_EXCLUSIVE);
   614    622         }
   615    623       }
   616    624       if( eRequired>pLock->eLock ){
   617    625         rc = sqlite3OsLock(pLock->pFile, eRequired);
   618    626       }else if(eRequired<pLock->eLock){
   619    627         rc = sqlite3OsUnlock(pLock->pFile, eRequired);
   620    628       }
................................................................................
   623    631       }
   624    632     }
   625    633   
   626    634     return rc;
   627    635   }
   628    636   
   629    637   /*
   630         -** No disk locking is performed.  We keep track of locks locally in
   631         -** the async.aLock hash table.  Locking should appear to work the same
   632         -** as with standard (unmodified) SQLite as long as all connections 
   633         -** come from this one process.  Connections from external processes
   634         -** cannot see our internal hash table (obviously) and will thus not
   635         -** honor our locks.
          638  +** The following two methods - asyncLock() and asyncUnlock() - are used
          639  +** to obtain and release locks on database files opened with the
          640  +** asynchronous backend.
   636    641   */
   637    642   static int asyncLock(sqlite3_file *pFile, int eLock){
   638    643     int rc = SQLITE_OK;
   639    644     AsyncFileData *p = ((AsyncFile *)pFile)->pData;
   640    645   
   641    646     pthread_mutex_lock(&async.lockMutex);
   642    647     if( p->lock.eLock<eLock ){
................................................................................
   652    657           (eLock==SQLITE_LOCK_SHARED && pIter->eLock>=SQLITE_LOCK_PENDING)
   653    658         )){
   654    659           rc = SQLITE_BUSY;
   655    660         }
   656    661       }
   657    662       if( rc==SQLITE_OK ){
   658    663         p->lock.eLock = eLock;
   659         -      if( eLock>p->lock.eAsyncLock ){
   660         -        p->lock.eAsyncLock = eLock;
   661         -      }
          664  +      p->lock.eAsyncLock = MAX(p->lock.eAsyncLock, eLock);
   662    665       }
   663    666       assert(p->lock.eAsyncLock>=p->lock.eLock);
   664    667       if( rc==SQLITE_OK ){
   665    668         rc = getFileLock(pLock);
   666    669       }
   667    670     }
   668    671     pthread_mutex_unlock(&async.lockMutex);
................................................................................
   670    673     ASYNC_TRACE(("LOCK %d (%s) rc=%d\n", eLock, p->zName, rc));
   671    674     return rc;
   672    675   }
   673    676   static int asyncUnlock(sqlite3_file *pFile, int eLock){
   674    677     AsyncFileData *p = ((AsyncFile *)pFile)->pData;
   675    678     AsyncFileLock *pLock = &p->lock;
   676    679     pthread_mutex_lock(&async.lockMutex);
   677         -  if( pLock->eLock>eLock ){
   678         -    pLock->eLock = eLock;
   679         -  }
          680  +  pLock->eLock = MIN(pLock->eLock, eLock);
   680    681     pthread_mutex_unlock(&async.lockMutex);
   681    682     return addNewAsyncWrite(p, ASYNC_UNLOCK, 0, eLock, 0);
   682    683   }
   683    684   
   684    685   /*
   685    686   ** This function is called when the pager layer first opens a database file
   686    687   ** and is checking for a hot-journal.
................................................................................
  1006   1007   **
  1007   1008   ** This routine is not even remotely threadsafe.  Do not call
  1008   1009   ** this routine while any SQLite database connections are open.
  1009   1010   */
  1010   1011   static void asyncEnable(int enable){
  1011   1012     if( enable ){
  1012   1013       if( !async_vfs.pAppData ){
         1014  +      static int hashTableInit = 0;
  1013   1015         async_vfs.pAppData = (void *)sqlite3_vfs_find(0);
  1014   1016         async_vfs.mxPathname = ((sqlite3_vfs *)async_vfs.pAppData)->mxPathname;
  1015   1017         sqlite3_vfs_register(&async_vfs, 1);
  1016         -      sqlite3HashInit(&async.aLock, SQLITE_HASH_BINARY, 1);
         1018  +      if( !hashTableInit ){
         1019  +        sqlite3HashInit(&async.aLock, SQLITE_HASH_BINARY, 1);
         1020  +        hashTableInit = 1;
         1021  +      }
  1017   1022       }
  1018   1023     }else{
  1019   1024       if( async_vfs.pAppData ){
  1020   1025         sqlite3_vfs_unregister(&async_vfs);
  1021   1026         async_vfs.pAppData = 0;
  1022         -      sqlite3HashClear(&async.aLock);
  1023   1027       }
  1024   1028     }
  1025   1029   }
  1026   1030   
  1027   1031   /* 
  1028   1032   ** This procedure runs in a separate thread, reading messages off of the
  1029   1033   ** write queue and processing them one by one.  
................................................................................
  1145   1149           ** structures for this file. Obtain the async.lockMutex mutex 
  1146   1150           ** before doing so.
  1147   1151           */
  1148   1152           pthread_mutex_lock(&async.lockMutex);
  1149   1153           pLock = sqlite3HashFind(&async.aLock, pData->zName, pData->nName);
  1150   1154           for(ppIter=&pLock->pList; *ppIter; ppIter=&((*ppIter)->pNext)){
  1151   1155             if( (*ppIter)==&pData->lock ){
  1152         -            *ppIter = (*ppIter)->pNext;
         1156  +            *ppIter = pData->lock.pNext;
  1153   1157               break;
  1154   1158             }
  1155   1159           }
  1156   1160           if( !pLock->pList ){
  1157         -          if( pLock->pFile ) sqlite3OsClose(pLock->pFile);
         1161  +          if( pLock->pFile ){
         1162  +            sqlite3OsClose(pLock->pFile);
         1163  +          }
  1158   1164             sqlite3_free(pLock);
  1159   1165             sqlite3HashInsert(&async.aLock, pData->zName, pData->nName, 0);
         1166  +          if( !sqliteHashFirst(&async.aLock) ){
         1167  +            sqlite3HashClear(&async.aLock);
         1168  +          }
  1160   1169           }else{
  1161   1170             rc = getFileLock(pLock);
  1162   1171           }
  1163   1172           pthread_mutex_unlock(&async.lockMutex);
  1164   1173   
  1165   1174           sqlite3_free(pData);
  1166   1175           break;
................................................................................
  1167   1176         }
  1168   1177   
  1169   1178         case ASYNC_UNLOCK: {
  1170   1179           AsyncLock *pLock;
  1171   1180           AsyncFileData *pData = p->pFileData;
  1172   1181           int eLock = p->nByte;
  1173   1182           pthread_mutex_lock(&async.lockMutex);
  1174         -        if( pData->lock.eAsyncLock>eLock ){
  1175         -          if( pData->lock.eLock>eLock ){
  1176         -            pData->lock.eAsyncLock = pData->lock.eLock;
  1177         -          }else{
  1178         -            pData->lock.eAsyncLock = eLock;
  1179         -          }
  1180         -        }
         1183  +        pData->lock.eAsyncLock = MIN(
         1184  +            pData->lock.eAsyncLock, MAX(pData->lock.eLock, eLock)
         1185  +        );
  1181   1186           assert(pData->lock.eAsyncLock>=pData->lock.eLock);
  1182   1187           pLock = sqlite3HashFind(&async.aLock, pData->zName, pData->nName);
  1183   1188           rc = getFileLock(pLock);
  1184   1189           pthread_mutex_unlock(&async.lockMutex);
  1185   1190           break;
  1186   1191         }
  1187   1192   

Changes to test/async.test.

     2      2   #    May you do good and not evil.
     3      3   #    May you find forgiveness for yourself and forgive others.
     4      4   #    May you share freely, never taking more than you give.
     5      5   #
     6      6   #***********************************************************************
     7      7   # This file runs all tests.
     8      8   #
     9         -# $Id: async.test,v 1.9 2007/09/04 18:28:44 danielk1977 Exp $
            9  +# $Id: async.test,v 1.10 2007/09/05 11:34:54 danielk1977 Exp $
    10     10   
    11     11   
    12     12   if {[catch {sqlite3async_enable}]} {
    13     13     # The async logic is not built into this system
    14     14     return
    15     15   }
    16     16   
    17     17   
    18     18   set testdir [file dirname $argv0]
    19     19   source $testdir/tester.tcl
    20     20   rename finish_test really_finish_test
    21         -proc finish_test {} {}
           21  +proc finish_test {} {
           22  +  catch {db close}
           23  +  catch {db2 close}
           24  +  catch {db3 close}
           25  +}
    22     26   set ISQUICK 1
    23     27   
    24     28   set INCLUDE {
    25     29     select1.test
    26     30     select2.test
    27     31     select3.test
    28     32     select4.test
................................................................................
    30     34     insert2.test
    31     35     insert3.test
    32     36     trans.test
    33     37     lock.test
    34     38     lock3.test
    35     39     lock2.test
    36     40   }
    37         -# set INCLUDE lock4.test
    38     41   
    39     42   # Enable asynchronous IO.
    40     43   sqlite3async_enable 1
    41     44   
    42     45   rename do_test really_do_test
    43     46   proc do_test {name args} {
    44     47     uplevel really_do_test async_io-$name $args
................................................................................
    47     50     sqlite3async_wait
    48     51   }
    49     52   
    50     53   foreach testfile [lsort -dictionary [glob $testdir/*.test]] {
    51     54     set tail [file tail $testfile]
    52     55     if {[lsearch -exact $INCLUDE $tail]<0} continue
    53     56     source $testfile
    54         -  catch {db close}
           57  +
           58  +  # Make sure everything is flushed through. This is because [source]ing 
           59  +  # the next test file will delete the database file on disk (using
           60  +  # [file delete]). If the asynchronous backend still has the file
           61  +  # open, it will become confused.
           62  +  #
           63  +  sqlite3async_halt idle
           64  +  sqlite3async_start
           65  +  sqlite3async_wait
    55     66   }
    56     67   
    57     68   # Flush the write-queue and disable asynchronous IO. This should ensure
    58     69   # all allocated memory is cleaned up.
    59     70   set sqlite3async_trace 1
    60     71   sqlite3async_halt idle
    61     72   sqlite3async_start

Changes to test/async2.test.

     1      1   #
     2      2   #    May you do good and not evil.
     3      3   #    May you find forgiveness for yourself and forgive others.
     4      4   #    May you share freely, never taking more than you give.
     5      5   #
     6      6   #***********************************************************************
     7      7   #
     8         -# $Id: async2.test,v 1.6 2007/08/30 10:49:55 danielk1977 Exp $
            8  +# $Id: async2.test,v 1.7 2007/09/05 11:34:54 danielk1977 Exp $
     9      9   
    10     10   
    11     11   set testdir [file dirname $argv0]
    12     12   source $testdir/tester.tcl
    13     13   
    14     14   if {
    15     15     [info commands sqlite3async_enable]=="" ||
................................................................................
    66     66         ioerr             { set ::sqlite_io_error_pending $n }
    67     67         malloc-persistent { sqlite3_memdebug_fail $n -repeat 1 }
    68     68         malloc-transient  { sqlite3_memdebug_fail $n -repeat 0 }
    69     69       }
    70     70       sqlite3async_halt idle
    71     71       sqlite3async_start
    72     72       sqlite3async_wait
    73         -  
           73  +    sqlite3async_enable 0
           74  +
    74     75       set ::sqlite_io_error_pending 0
    75     76       sqlite3_memdebug_fail -1
    76     77   
    77     78       sqlite3 db test.db
    78     79       set c [db eval {SELECT c FROM counter LIMIT 1}]
    79     80       switch -- $c {
    80     81         1 {
................................................................................
   109    110           } {klmnopqrst and seven}
   110    111         }
   111    112         FIN {
   112    113           set ::go 0
   113    114         }
   114    115       }
   115    116     
   116         -    sqlite3async_enable 0
          117  +    db close
   117    118     }
   118    119   }
   119    120   
   120    121   catch {db close}
   121    122   sqlite3async_halt idle
   122    123   sqlite3async_start
   123    124   sqlite3async_wait
   124    125   
   125    126   finish_test