/ Check-in [7c3492c8]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Add test files for the asynchronous IO concept. (CVS 2853)
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 7c3492c8404c39c808af4429b4fcdb7413539ab3
User & Date: danielk1977 2006-01-03 13:39:26
Context
2006-01-03
15:16
Always case 0 to (char*) on varargs functions. Otherwise there are problems on 64-bit machines. (CVS 2854) check-in: 837dc77f user: drh tags: trunk
13:39
Add test files for the asynchronous IO concept. (CVS 2853) check-in: 7c3492c8 user: danielk1977 tags: trunk
00:33
The sqlite TCL command no longer returns the hex address of the sqlite3* structure. Instead there is a new command in testfixture to find that information. (CVS 2852) check-in: 70b22857 user: drh tags: trunk
Changes
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Added src/test_async.c.

            1  +/*
            2  +** 2005 December 14
            3  +**
            4  +** The author disclaims copyright to this source code.  In place of
            5  +** a legal notice, here is a blessing:
            6  +**
            7  +**    May you do good and not evil.
            8  +**    May you find forgiveness for yourself and forgive others.
            9  +**    May you share freely, never taking more than you give.
           10  +**
           11  +*************************************************************************
           12  +**
           13  +** This file contains an example implementation of an asynchronous IO 
           14  +** backend for SQLite. It is used to test that the concept of asynchronous 
           15  +** IO in SQLite is valid.
           16  +*/
           17  +
           18  +#include "sqliteInt.h"
           19  +#include "os.h"
           20  +
           21  +#define MIN(x,y) ((x)<(y)?(x):(y))
           22  +#define MAX(x,y) ((x)>(y)?(x):(y))
           23  +
           24  +typedef struct AsyncWrite AsyncWrite;
           25  +typedef struct AsyncFile AsyncFile;
           26  +
           27  +/*
           28  +** TODO:
           29  +**     * File locks...
           30  +*/
           31  +
           32  +/*
           33  +** THREAD SAFETY NOTES
           34  +**
           35  +** Basic rules:
           36  +**
           37  +**     * Both read and write access to the global write-op queue must be 
           38  +**       protected by the sqlite3Os mutex functions.
           39  +**     * The file handles from the underlying system are assumed not to 
           40  +**       be thread safe.
           41  +**     * See the last paragraph under "sqlite3_async_flush() Threads" for
           42  +**       an assumption to do with file-handle synchronization by the Os.
           43  +**
           44  +** File system operations (invoked by SQLite thread):
           45  +**
           46  +**     xOpenXXX (three versions)
           47  +**     xDelete
           48  +**     xFileExists
           49  +**
           50  +**     Todo:
           51  +**         xSyncDirectory
           52  +**
           53  +** File handle operations (invoked by SQLite thread):
           54  +**
           55  +**     The following operations add an entry to the global write-op list. They
           56  +**     prepare the entry, aquire the mutex momentarily while list pointers are 
           57  +**     manipulated to insert the new entry, and release the mutex.
           58  +**    
           59  +**         asyncWrite, asyncClose, asyncTruncate, asyncSync, 
           60  +**         asyncSetFullSync, asyncOpenDirectory.
           61  +**    
           62  +**     Read operations. Both of these read from both the underlying file and
           63  +**     the write-op list. So we grab the mutex for the whole call (even 
           64  +**     while performing a blocking read on the file).
           65  +**    
           66  +**         asyncRead, asyncFileSize.
           67  +**    
           68  +**     These locking primitives become no-ops. Files are always opened for 
           69  +**     exclusive access when using this IO backend:
           70  +**    
           71  +**         asyncLock, asyncUnlock, asyncLockState, asyncCheckReservedLock
           72  +**    
           73  +**     The sqlite3OsFileHandle() function is currently only used when 
           74  +**     debugging the pager module. Unless sqlite3OsClose() is called on the
           75  +**     file (shouldn't be possible for other reasons), the underlying 
           76  +**     implementations are safe to call without grabbing any mutex. So we just
           77  +**     go ahead and call it no matter what any other thread is doing.
           78  +**
           79  +**         asyncFileHandle.
           80  +**
           81  +**     Calling this method just manipulates the AsyncFile.iOffset variable. 
           82  +**     Since this variable is never accessed by an sqlite3_async_flush() thread,
           83  +**     this function does not require the mutex. Actual calls to OsSeek() take 
           84  +**     place just before OsWrite() or OsRead(), which are always protected by 
           85  +**     the mutex.
           86  +**    
           87  +**         asyncSeek.
           88  +**
           89  +** sqlite3_async_flush() (any thread):
           90  +**
           91  +**     A pseudo-mutex (a global boolean variable) is used to make sure only 
           92  +**     one thread is inside the sqlite3_async_flush() thread at any one time.
           93  +**     If the variable is set when a thread enters _flush(), then it 
           94  +**     immediately returns SQLITE_BUSY. Otherwise, it sets the variable, 
           95  +**     executes the body of the function, and clears the variable just before
           96  +**     returning. Both read and write access to said global variable 
           97  +**     (sqlite3_asyncIoBusy) is protected by sqlite3Os mutex, of course.
           98  +**
           99  +**     Inside sqlite3_async_flush() is a loop that works like this:
          100  +**
          101  +**         WHILE (write-op list is not empty)
          102  +**             Do IO operation at head of write-op list
          103  +**             Remove entry from head of write-op list
          104  +**         END WHILE
          105  +**
          106  +**     The mutex is always obtained during the <write-op list is not empty>
          107  +**     test, and when the entry is removed from the head of the write-op 
          108  +**     list. Sometimes it is held for the interim period (while the IO is
          109  +**     performed), and sometimes it is relinquished. It is relinquished if
          110  +**     (a) the IO op is an ASYNC_CLOSE or (b) when the file handle was 
          111  +**     opened, two of the underlying systems handles were opened on the
          112  +**     same file-system entry.
          113  +**
          114  +**     If condition (b) above is true, then one file-handle 
          115  +**     (AsyncFile.pBaseRead) is used exclusively by sqlite threads to read the
          116  +**     file, the other (AsyncFile.pBaseWrite) by sqlite3_async_flush() 
          117  +**     threads to perform write() operations. This means that read 
          118  +**     operations are not blocked by asynchronous writes (although 
          119  +**     asynchronous writes may still be blocked by reads).
          120  +**
          121  +**     This assumes that the OS keeps two handles open on the same file
          122  +**     properly in sync. That is, any read operation that starts after a
          123  +**     write operation on the same file system entry has completed returns
          124  +**     data consistent with the write. We also assume that if one thread 
          125  +**     reads a file while another is writing it all bytes other than the
          126  +**     ones actually being written contain valid data.
          127  +**
          128  +**     If the above assumptions are not true, set the preprocessor symbol
          129  +**     SQLITE_ASYNC_TWO_FILEHANDLES to 0.
          130  +*/
          131  +
          132  +#ifndef SQLITE_ASYNC_TWO_FILEHANDLES
          133  +/* #define SQLITE_ASYNC_TWO_FILEHANDLES 0 */
          134  +#define SQLITE_ASYNC_TWO_FILEHANDLES 1
          135  +#endif
          136  +
          137  +/* 
          138  +** First and last elements of the global write-op list. 
          139  +**
          140  +** Whenever an OsWrite(), OsSync(), OsTrunc() or OsClose() operation is
          141  +** requested, instead of performing the file IO immediately, a new AsyncWrite
          142  +** structure is allocated and added to the global linked list that starts at
          143  +** sqlite3_asyncListFirst. The next time to sqlite3_async_flush() is called,
          144  +** all operations are realised and the list elements deleted.
          145  +*/
          146  +static AsyncWrite *sqlite3_asyncListFirst = 0;
          147  +static AsyncWrite *sqlite3_asyncListLast = 0;
          148  +
          149  +/* True after an IO error has occured */
          150  +/* static int *sqlite3_asyncIoError = 0; */
          151  +
          152  +/* True if some thread is currently inside sqlite3_async_flush() */
          153  +static int sqlite3_asyncIoBusy = 0;
          154  +
          155  +/* Possible values of AsyncWrite.op */
          156  +#define ASYNC_WRITE         1
          157  +#define ASYNC_SYNC          2
          158  +#define ASYNC_TRUNCATE      3
          159  +#define ASYNC_CLOSE         4
          160  +#define ASYNC_OPENDIRECTORY 5
          161  +#define ASYNC_SETFULLSYNC   6
          162  +
          163  +#define ASYNC_DELETE        7
          164  +#define ASYNC_OPENEXCLUSIVE 8
          165  +#define ASYNC_SYNCDIRECTORY 9
          166  +
          167  +/*
          168  +** The interpretation of the iOffset and nByte variables varies depending 
          169  +** on the value of AsyncWrite.op:
          170  +**
          171  +** ASYNC_WRITE:
          172  +**     iOffset -> Offset in file to write to.
          173  +**     nByte   -> Number of bytes of data to write (pointed to by zBuf).
          174  +**
          175  +** ASYNC_SYNC:
          176  +**     iOffset -> Unused.
          177  +**     nByte   -> Value of "fullsync" flag to pass to sqlite3OsSync().
          178  +**
          179  +** ASYNC_TRUNCATE:
          180  +**     iOffset -> Size to truncate file to.
          181  +**     nByte   -> Unused.
          182  +**
          183  +** ASYNC_CLOSE:
          184  +**     iOffset -> Unused.
          185  +**     nByte   -> Unused.
          186  +**
          187  +** ASYNC_OPENDIRECTORY:
          188  +**     iOffset -> Unused.
          189  +**     nByte   -> Number of bytes of zBuf points to (directory name).
          190  +**
          191  +** ASYNC_SETFULLSYNC:
          192  +**     iOffset -> Unused.
          193  +**     nByte   -> New value for the full-sync flag.
          194  +**
          195  +**
          196  +** ASYNC_DELETE:
          197  +**     iOffset -> Unused.
          198  +**     nByte   -> Number of bytes of zBuf points to (file name).
          199  +**
          200  +** ASYNC_OPENEXCLUSIVE:
          201  +**     iOffset -> Value of "delflag".
          202  +**     nByte   -> Number of bytes of zBuf points to (file name).
          203  +**
          204  +** For an ASYNC_WRITE operation, zBuf points to the data to write to the file. 
          205  +** This space is sqliteMalloc()d along with the AsyncWrite structure in a
          206  +** single blob, so is deleted when sqliteFree() is called on the parent 
          207  +** structure.
          208  +*/
          209  +struct AsyncWrite {
          210  +  AsyncFile *pFile;   /* File to write data to or sync */
          211  +  int op;             /* One of ASYNC_xxx etc. */
          212  +  i64 iOffset;        /* See above */
          213  +  int nByte;          /* See above */
          214  +  char *zBuf;         /* Data to write to file (or NULL if op!=ASYNC_WRITE) */
          215  +  AsyncWrite *pNext;  /* Next write operation (to any file) */
          216  +};
          217  +
          218  +/* 
          219  +** The AsyncFile structure is a subclass of OsFile used for asynchronous IO.
          220  +*/
          221  +struct AsyncFile {
          222  +  IoMethod *pMethod;   /* Must be first */
          223  +  i64 iOffset;         /* Current seek() offset in file */
          224  +  OsFile *pBaseRead;   /* Read handle to the underlying Os file */
          225  +  OsFile *pBaseWrite;  /* Write handle to the underlying Os file */
          226  +};
          227  +
          228  +/*
          229  +** Add an entry to the end of the global write-op list. pWrite should point 
          230  +** to an AsyncWrite structure allocated using sqliteMalloc(). A future call 
          231  +** to sqlite3_async_flush() is responsible for calling sqliteFree().
          232  +**
          233  +** Once an AsyncWrite structure has been added to the list, it must not be
          234  +** read or modified by the caller (in case another thread calls
          235  +** sqlite3_async_flush() ).
          236  +*/
          237  +static void addAsyncWrite(AsyncWrite *pWrite){
          238  +  sqlite3_os_enter_mutex();
          239  +  assert( !pWrite->pNext );
          240  +  if( sqlite3_asyncListLast ){
          241  +    assert( sqlite3_asyncListFirst );
          242  +    sqlite3_asyncListLast->pNext = pWrite;
          243  +  }else{
          244  +    sqlite3_asyncListFirst = pWrite;
          245  +  }
          246  +  sqlite3_asyncListLast = pWrite;
          247  +  sqlite3_os_leave_mutex();
          248  +}
          249  +
          250  +/*
          251  +** The caller should already hold the mutex when this is called.
          252  +*/
          253  +static void removeAsyncWrite(AsyncWrite *p){
          254  +  assert( p==sqlite3_asyncListFirst );
          255  +  assert( sqlite3_asyncListLast );
          256  +  if( sqlite3_asyncListFirst==sqlite3_asyncListLast ){
          257  +    assert( !sqlite3_asyncListFirst->pNext );
          258  +    sqlite3_asyncListLast = 0;
          259  +  }
          260  +  sqlite3_asyncListFirst = sqlite3_asyncListFirst->pNext;
          261  +}
          262  +
          263  +/*
          264  +** This is a utility function to allocate and populate a new AsyncWrite
          265  +** structure and insert it (via addAsyncWrite() ) into the global list.
          266  +*/
          267  +static int addNewAsyncWrite(
          268  +  AsyncFile *pFile, 
          269  +  int op, 
          270  +  i64 iOffset, 
          271  +  int nByte,
          272  +  const char *zByte
          273  +){
          274  +  AsyncWrite *p = sqlite3Os.xMalloc(sizeof(AsyncWrite) + (zByte?nByte:0));
          275  +  if( !p ){
          276  +    return SQLITE_NOMEM;
          277  +  }
          278  +  p->op = op;
          279  +  p->iOffset = iOffset;
          280  +  p->nByte = nByte;
          281  +  p->pFile = pFile;
          282  +  p->pNext = 0;
          283  +  if( zByte ){
          284  +    p->zBuf = (char *)&p[1];
          285  +    memcpy(p->zBuf, zByte, nByte);
          286  +  }else{
          287  +    p->zBuf = 0;
          288  +  }
          289  +  addAsyncWrite(p);
          290  +  return SQLITE_OK;
          291  +}
          292  +
          293  +/*
          294  +** Close the file. This just adds an entry to the write-op list, the file is
          295  +** not actually closed.
          296  +*/
          297  +static int asyncClose(OsFile **pId){
          298  +  return addNewAsyncWrite((AsyncFile *)*pId, ASYNC_CLOSE, 0, 0, 0);
          299  +}
          300  +
          301  +/*
          302  +** Implementation of sqlite3OsWrite() for asynchronous files. Instead of 
          303  +** writing to the underlying file, this function adds an entry to the end of
          304  +** the global AsyncWrite list. Either SQLITE_OK or SQLITE_NOMEM may be
          305  +** returned.
          306  +*/
          307  +static int asyncWrite(OsFile *id, const void *pBuf, int amt){
          308  +  AsyncFile *pFile = (AsyncFile *)id;
          309  +  int rc = addNewAsyncWrite(pFile, ASYNC_WRITE, pFile->iOffset, amt, pBuf);
          310  +  pFile->iOffset += (i64)amt;
          311  +  return rc;
          312  +}
          313  +
          314  +/*
          315  +** Truncate the file to nByte bytes in length. This just adds an entry to 
          316  +** the write-op list, no IO actually takes place.
          317  +*/
          318  +static int asyncTruncate(OsFile *id, i64 nByte){
          319  +  return addNewAsyncWrite((AsyncFile *)id, ASYNC_TRUNCATE, nByte, 0, 0);
          320  +}
          321  +
          322  +/*
          323  +** Open the directory identified by zName and associate it with the 
          324  +** specified file. This just adds an entry to the write-op list, the 
          325  +** directory is opened later by sqlite3_async_flush().
          326  +*/
          327  +static int asyncOpenDirectory(OsFile *id, const char *zName){
          328  +  AsyncFile *pFile = (AsyncFile *)id;
          329  +  return addNewAsyncWrite(pFile, ASYNC_OPENDIRECTORY, 0, strlen(zName)+1,zName);
          330  +}
          331  +
          332  +/*
          333  +** Sync the file. This just adds an entry to the write-op list, the 
          334  +** sync() is done later by sqlite3_async_flush().
          335  +*/
          336  +static int asyncSync(OsFile *id, int fullsync){
          337  +  return addNewAsyncWrite((AsyncFile *)id, ASYNC_SYNC, 0, fullsync, 0);
          338  +}
          339  +
          340  +/*
          341  +** Set (or clear) the full-sync flag on the underlying file. This operation
          342  +** is queued and performed later by sqlite3_async_flush().
          343  +*/
          344  +static void asyncSetFullSync(OsFile *id, int value){
          345  +  addNewAsyncWrite((AsyncFile *)id, ASYNC_SETFULLSYNC, 0, value, 0);
          346  +}
          347  +
          348  +/*
          349  +** Read data from the file. First we read from the filesystem, then adjust 
          350  +** the contents of the buffer based on ASYNC_WRITE operations in the 
          351  +** write-op queue. Todo: Do we need to think about ASYNC_TRUNCATE in 
          352  +** this method as well?
          353  +**
          354  +** This method holds the mutex from start to finish.
          355  +*/
          356  +static int asyncRead(OsFile *id, void *obuf, int amt){
          357  +  int rc = SQLITE_OK;
          358  +  i64 filesize;
          359  +  int nRead;
          360  +  AsyncFile *pFile = (AsyncFile *)id;
          361  +
          362  +  /* Grab the mutex for the duration of the call */
          363  +  sqlite3_os_enter_mutex();
          364  +
          365  +  if( pFile->pBaseRead ){
          366  +    rc = sqlite3OsFileSize(pFile->pBaseRead, &filesize);
          367  +    if( rc!=SQLITE_OK ){
          368  +      goto asyncread_out;
          369  +    }
          370  +    rc = sqlite3OsSeek(pFile->pBaseRead, pFile->iOffset);
          371  +    if( rc!=SQLITE_OK ){
          372  +      goto asyncread_out;
          373  +    }
          374  +    nRead = MIN(filesize - pFile->iOffset, amt);
          375  +    if( nRead>0 ){
          376  +      rc = sqlite3OsRead(((AsyncFile *)id)->pBaseRead, obuf, nRead);
          377  +    }
          378  +  }
          379  +
          380  +  if( rc==SQLITE_OK ){
          381  +    AsyncWrite *p;
          382  +    i64 iOffset = pFile->iOffset;           /* Current seek offset */
          383  +
          384  +    for(p=sqlite3_asyncListFirst; p; p = p->pNext){
          385  +      if( p->pFile==pFile && p->op==ASYNC_WRITE ){
          386  +        int iBeginIn = (p->iOffset - iOffset);
          387  +        int iBeginOut = (iOffset - p->iOffset);
          388  +        int nCopy;
          389  +
          390  +        if( iBeginIn<0 ) iBeginIn = 0;
          391  +        if( iBeginOut<0 ) iBeginOut = 0;
          392  +        nCopy = MIN(p->nByte-iBeginIn, amt-iBeginOut);
          393  +
          394  +        if( nCopy>0 ){
          395  +          memcpy(&((char *)obuf)[iBeginOut], &p->zBuf[iBeginIn], nCopy);
          396  +        }
          397  +      }
          398  +    }
          399  +
          400  +    pFile->iOffset += (i64)amt;
          401  +  }
          402  +
          403  +asyncread_out:
          404  +  sqlite3_os_leave_mutex();
          405  +  return rc;
          406  +}
          407  +
          408  +/*
          409  +** Seek to the specified offset. This just adjusts the AsyncFile.iOffset 
          410  +** variable - calling seek() on the underlying file is defered until the 
          411  +** next read() or write() operation. 
          412  +*/
          413  +static int asyncSeek(OsFile *id, i64 offset){
          414  +  AsyncFile *pFile = (AsyncFile *)id;
          415  +  pFile->iOffset = offset;
          416  +  return SQLITE_OK;
          417  +}
          418  +
          419  +/*
          420  +** Read the size of the file. First we read the size of the file system 
          421  +** entry, then adjust for any ASYNC_WRITE or ASYNC_TRUNCATE operations 
          422  +** currently in the write-op list. 
          423  +**
          424  +** This method holds the mutex from start to finish.
          425  +*/
          426  +int asyncFileSize(OsFile *id, i64 *pSize){
          427  +  int rc = SQLITE_OK;
          428  +  i64 s = 0;
          429  +  OsFile *pBase;
          430  +  sqlite3_os_enter_mutex();
          431  +
          432  +  /* Read the filesystem size from the base file. If pBaseRead is NULL, this
          433  +  ** means the file hasn't been opened yet. In this case all relevant data 
          434  +  ** must be in the write-op queue anyway, so we can omit reading from the
          435  +  ** file-system.
          436  +  */
          437  +  pBase = ((AsyncFile *)id)->pBaseRead;
          438  +  if( pBase ){
          439  +    rc = sqlite3OsFileSize(pBase, &s);
          440  +  }
          441  +
          442  +  if( rc==SQLITE_OK ){
          443  +    AsyncWrite *p;
          444  +    for(p=sqlite3_asyncListFirst; p; p = p->pNext){
          445  +      if( p->pFile==(AsyncFile *)id ){
          446  +        switch( p->op ){
          447  +          case ASYNC_WRITE:
          448  +            s = MAX(p->iOffset + (i64)(p->nByte), s);
          449  +            break;
          450  +          case ASYNC_TRUNCATE:
          451  +            s = MIN(s, p->nByte);
          452  +            break;
          453  +        }
          454  +      }
          455  +    }
          456  +    *pSize = s;
          457  +  }
          458  +  sqlite3_os_leave_mutex();
          459  +  return rc;
          460  +}
          461  +
          462  +/*
          463  +** Return the operating system file handle. This is only used for debugging 
          464  +** at the moment anyway.
          465  +*/
          466  +static int asyncFileHandle(OsFile *id){
          467  +  return sqlite3OsFileHandle(((AsyncFile *)id)->pBaseRead);
          468  +}
          469  +
          470  +static int asyncLock(OsFile *id, int lockType){
          471  +  return SQLITE_OK;
          472  +}
          473  +static int asyncUnlock(OsFile *id, int lockType){
          474  +  return SQLITE_OK;
          475  +}
          476  +
          477  +/*
          478  +** This function is called when the pager layer first opens a database file
          479  +** and is checking for a hot-journal.
          480  +*/
          481  +static int asyncCheckReservedLock(OsFile *id){
          482  +  return SQLITE_OK;
          483  +}
          484  +
          485  +/* 
          486  +** This is broken. But sqlite3OsLockState() is only used for testing anyway.
          487  +*/
          488  +static int asyncLockState(OsFile *id){
          489  +  return SQLITE_OK;
          490  +}
          491  +
          492  +/*
          493  +** The three file-open functions for the underlying file system layer.
          494  +*/
          495  +static int (*xOrigOpenReadWrite)(const char*, OsFile**, int*) = 0;
          496  +static int (*xOrigOpenExclusive)(const char*, OsFile**, int) = 0;
          497  +static int (*xOrigOpenReadOnly)(const char*, OsFile**) = 0;
          498  +
          499  +/*
          500  +** Pointers to the original versions of other overridden file-system 
          501  +** operations.
          502  +*/
          503  +static int (*xOrigDelete)(const char*) = 0;
          504  +static int (*xOrigFileExists)(const char*) = 0;
          505  +static int (*xOrigSyncDirectory)(const char*) = 0;
          506  +
          507  +static int asyncOpenFile(
          508  +  const char *zName, 
          509  +  OsFile **pFile, 
          510  +  OsFile *pBaseRead,
          511  +  int openSecondFile
          512  +){
          513  +  int rc;
          514  +  AsyncFile *p;
          515  +  OsFile *pBaseWrite = 0;
          516  +
          517  +  static IoMethod iomethod = {
          518  +    asyncClose,
          519  +    asyncOpenDirectory,
          520  +    asyncRead,
          521  +    asyncWrite,
          522  +    asyncSeek,
          523  +    asyncTruncate,
          524  +    asyncSync,
          525  +    asyncSetFullSync,
          526  +    asyncFileHandle,
          527  +    asyncFileSize,
          528  +    asyncLock,
          529  +    asyncUnlock,
          530  +    asyncLockState,
          531  +    asyncCheckReservedLock
          532  +  };
          533  +
          534  +  if( openSecondFile && SQLITE_ASYNC_TWO_FILEHANDLES ){
          535  +    int dummy;
          536  +    rc = xOrigOpenReadWrite(zName, &pBaseWrite, &dummy);
          537  +    if( rc!=SQLITE_OK ){
          538  +      goto error_out;
          539  +    }
          540  +  }
          541  +
          542  +  p = (AsyncFile *)sqlite3Os.xMalloc(sizeof(AsyncFile));
          543  +  if( !p ){
          544  +    rc = SQLITE_NOMEM;
          545  +    goto error_out;
          546  +  }
          547  +  memset(p, 0, sizeof(AsyncFile));
          548  +  
          549  +  p->pMethod = &iomethod;
          550  +  p->pBaseRead = pBaseRead;
          551  +  p->pBaseWrite = pBaseWrite;
          552  +  
          553  +  *pFile = (OsFile *)p;
          554  +  return SQLITE_OK;
          555  +
          556  +error_out:
          557  +  assert(!p);
          558  +  sqlite3OsClose(&pBaseRead);
          559  +  sqlite3OsClose(&pBaseWrite);
          560  +  *pFile = 0;
          561  +  return rc;
          562  +}
          563  +
          564  +/*
          565  +** The async-IO backends implementation of the three functions used to open
          566  +** a file (xOpenExclusive, xOpenReadWrite and xOpenReadOnly). Most of the 
          567  +** work is done in function asyncOpenFile() - see above.
          568  +*/
          569  +static int asyncOpenExclusive(const char *z, OsFile **ppFile, int delFlag){
          570  +  int rc = asyncOpenFile(z, ppFile, 0, 0);
          571  +  if( rc==SQLITE_OK ){
          572  +    AsyncFile *pFile = (AsyncFile *)(*ppFile);
          573  +    int nByte = strlen(z)+1;
          574  +    i64 i = (i64)(delFlag);
          575  +    rc = addNewAsyncWrite(pFile, ASYNC_OPENEXCLUSIVE, i, nByte, z);
          576  +    if( rc!=SQLITE_OK ){
          577  +      sqlite3Os.xFree(pFile);
          578  +      *ppFile = 0;
          579  +    }
          580  +  }
          581  +  return rc;
          582  +}
          583  +static int asyncOpenReadOnly(const char *z, OsFile **ppFile){
          584  +  OsFile *pBase = 0;
          585  +  int rc = xOrigOpenReadOnly(z, &pBase);
          586  +  if( rc==SQLITE_OK ){
          587  +    rc = asyncOpenFile(z, ppFile, pBase, 0);
          588  +  }
          589  +  return rc;
          590  +}
          591  +static int asyncOpenReadWrite(const char *z, OsFile **ppFile, int *pReadOnly){
          592  +  OsFile *pBase = 0;
          593  +  int rc = xOrigOpenReadWrite(z, &pBase, pReadOnly);
          594  +  if( rc==SQLITE_OK ){
          595  +    rc = asyncOpenFile(z, ppFile, pBase, (*pReadOnly ? 0 : 1));
          596  +  }
          597  +  return rc;
          598  +}
          599  +
          600  +/*
          601  +** Implementation of sqlite3Os.xDelete. Add an entry to the end of the 
          602  +** write-op queue to perform the delete.
          603  +*/
          604  +static int asyncDelete(const char *z){
          605  +  return addNewAsyncWrite(0, ASYNC_DELETE, 0, strlen(z)+1, z);
          606  +}
          607  +
          608  +/*
          609  +** Implementation of sqlite3Os.xDelete. Add an entry to the end of the 
          610  +** write-op queue to perform the delete.
          611  +*/
          612  +static int asyncSyncDirectory(const char *z){
          613  +  return addNewAsyncWrite(0, ASYNC_SYNCDIRECTORY, 0, strlen(z)+1, z);
          614  +}
          615  +
          616  +/*
          617  +** Implementation of sqlite3Os.xFileExists. Return true if file 'z' exists
          618  +** in the file system. 
          619  +**
          620  +** This method holds the mutex from start to finish.
          621  +*/
          622  +static int asyncFileExists(const char *z){
          623  +  int ret;
          624  +  AsyncWrite *p;
          625  +  sqlite3_os_enter_mutex();
          626  +
          627  +  /* See if the real file system contains the specified file.  */
          628  +  ret = xOrigFileExists(z);
          629  +  
          630  +  for(p=sqlite3_asyncListFirst; p; p = p->pNext){
          631  +    if( p->op==ASYNC_DELETE && 0==strcmp(p->zBuf, z) ){
          632  +      ret = 0;
          633  +    }else if( p->op==ASYNC_OPENEXCLUSIVE && 0==strcmp(p->zBuf, z) ){
          634  +      ret = 1;
          635  +    }
          636  +  }
          637  +
          638  +  sqlite3_os_leave_mutex();
          639  +  return ret;
          640  +}
          641  +
          642  +/*
          643  +** The following routine is one of two exported symbols in this module (along
          644  +** with sqlite3_async_flush(), see below). This routine should be called
          645  +** once to enable the asynchronous IO features implemented in this file. If 
          646  +** the features are successfully enabled (or if they have already been 
          647  +** enabled) then SQLITE_OK is returned. Otherwise, SQLITE_MISUSE.
          648  +*/
          649  +int sqlite3_async_enable(void){
          650  +  if( xOrigOpenReadWrite==0 ){
          651  +#define ROUTINE(a,b,c) {(void**)&a,SQLITE_OS_ROUTINE_ ## b,(void *)c}
          652  +    struct ReplacementOp {
          653  +      void ** pOldRoutine;
          654  +      int eRoutine;
          655  +      void * pNewRoutine;
          656  +    } aRoutines[] = {
          657  +      ROUTINE(xOrigOpenReadWrite, OPENREADWRITE, asyncOpenReadWrite),
          658  +      ROUTINE(xOrigOpenReadOnly, OPENREADONLY, asyncOpenReadOnly), 
          659  +      ROUTINE(xOrigOpenExclusive, OPENEXCLUSIVE, asyncOpenExclusive), 
          660  +      ROUTINE(xOrigDelete, DELETE, asyncDelete), 
          661  +      ROUTINE(xOrigFileExists, FILEEXISTS, asyncFileExists), 
          662  +      ROUTINE(xOrigSyncDirectory, SYNCDIRECTORY, asyncSyncDirectory)
          663  +    };
          664  +#undef ROUTINE
          665  +    int i;
          666  +
          667  +    sqlite3_os_enter_mutex();
          668  +    for(i=0; i<sizeof(aRoutines)/sizeof(aRoutines[0]); i++){
          669  +      struct ReplacementOp *p = &aRoutines[i];
          670  +      *(p->pOldRoutine) = sqlite3_os_routine_set(p->eRoutine, p->pNewRoutine);
          671  +    }
          672  +    sqlite3_os_leave_mutex();
          673  +  }
          674  +  return SQLITE_OK;
          675  +}
          676  +
          677  +/* 
          678  +** This function is called externally to perform queued write and sync
          679  +** operations. It returns when an IO error occurs or there are no more queued
          680  +** operations to perform.
          681  +*/
          682  +int sqlite3_async_flush(void){
          683  +  AsyncWrite *p = 0;
          684  +  int rc = SQLITE_OK;
          685  +
          686  +  /* Grab the mutex and set the sqlite3_asyncIoBusy flag to make sure this
          687  +  ** is the only thread performing an sqlite3_async_flush() at this time.
          688  +  ** Or, if some other thread is already inside this function, return 
          689  +  ** SQLITE_BUSY to the caller.
          690  +  */
          691  +  sqlite3_os_enter_mutex();
          692  +  if( sqlite3_asyncIoBusy ){
          693  +    sqlite3_os_leave_mutex();
          694  +    return SQLITE_BUSY;
          695  +  }
          696  +  sqlite3_asyncIoBusy = 1;
          697  +
          698  +  while( (p = sqlite3_asyncListFirst) && rc==SQLITE_OK ){
          699  +    int isInsideMutex = 1;
          700  +
          701  +    /* Right now this thread is holding the global mutex. Variable 'p' points
          702  +    ** to the first entry in the write-op queue. In the general case, we
          703  +    ** hold on to the mutex for the entire body of the loop. 
          704  +    **
          705  +    ** However in the cases enumerated below, we relinquish the mutex,
          706  +    ** perform the IO, and then re-request the mutex before removing 'p' from
          707  +    ** the head of the write-op queue. The idea is to increase concurrency with
          708  +    ** sqlite threads.
          709  +    **
          710  +    **     * An ASYNC_CLOSE operation.
          711  +    **     * An ASYNC_OPENEXCLUSIVE operation. For this one, we relinquish 
          712  +    **       the mutex, call the underlying xOpenExclusive() function, then
          713  +    **       re-aquire the mutex before seting the AsyncFile.pBaseRead 
          714  +    **       variable.
          715  +    **     * ASYNC_SYNC and ASYNC_WRITE operations, if 
          716  +    **       SQLITE_ASYNC_TWO_FILEHANDLES was set at compile time and two
          717  +    **       file-handles are open for the particular file being "synced".
          718  +    */
          719  +    OsFile *pBase = 0;
          720  +    if( p->pFile ){
          721  +      pBase = p->pFile->pBaseWrite;
          722  +      if( 
          723  +        p->op==ASYNC_CLOSE || 
          724  +        p->op==ASYNC_OPENEXCLUSIVE ||
          725  +        (pBase && (p->op==ASYNC_SYNC || p->op==ASYNC_WRITE) ) 
          726  +      ){
          727  +        sqlite3_os_leave_mutex();
          728  +        isInsideMutex = 0;
          729  +      }
          730  +      if( !pBase ){
          731  +        pBase = p->pFile->pBaseRead;
          732  +      }
          733  +    }
          734  +
          735  +    switch( p->op ){
          736  +      case ASYNC_WRITE:
          737  +        assert( pBase );
          738  +        rc = sqlite3OsSeek(pBase, p->iOffset);
          739  +        if( rc==SQLITE_OK ){
          740  +          rc = sqlite3OsWrite(pBase, (const void *)(p->zBuf), p->nByte);
          741  +        }
          742  +        break;
          743  +
          744  +      case ASYNC_SYNC:
          745  +        assert( pBase );
          746  +        rc = sqlite3OsSync(pBase, p->nByte);
          747  +        break;
          748  +
          749  +      case ASYNC_TRUNCATE:
          750  +        assert( pBase );
          751  +        rc = sqlite3OsTruncate(pBase, p->nByte);
          752  +        break;
          753  +
          754  +      case ASYNC_CLOSE:
          755  +        sqlite3OsClose(&p->pFile->pBaseRead);
          756  +        sqlite3OsClose(&p->pFile->pBaseWrite);
          757  +        sqlite3Os.xFree(p->pFile);
          758  +        break;
          759  +
          760  +      case ASYNC_OPENDIRECTORY:
          761  +        assert( pBase );
          762  +        sqlite3OsOpenDirectory(pBase, p->zBuf);
          763  +        break;
          764  +
          765  +      case ASYNC_SETFULLSYNC:
          766  +        assert( pBase );
          767  +        sqlite3OsSetFullSync(pBase, p->nByte);
          768  +        break;
          769  +
          770  +      case ASYNC_DELETE:
          771  +        rc = xOrigDelete(p->zBuf);
          772  +        break;
          773  +
          774  +      case ASYNC_SYNCDIRECTORY:
          775  +        rc = xOrigSyncDirectory(p->zBuf);
          776  +        break;
          777  +
          778  +      case ASYNC_OPENEXCLUSIVE: {
          779  +        AsyncFile *pFile = p->pFile;
          780  +        int delFlag = ((p->iOffset)?1:0);
          781  +        OsFile *pBase = 0;
          782  +        rc = xOrigOpenExclusive(p->zBuf, &pBase, delFlag);
          783  +
          784  +        sqlite3_os_enter_mutex();
          785  +        isInsideMutex = 1;
          786  +        if( rc==SQLITE_OK ){
          787  +          pFile->pBaseRead = pBase;
          788  +        }
          789  +        break;
          790  +      }
          791  +
          792  +      default: assert(!"Illegal value for AsyncWrite.op");
          793  +    }
          794  +
          795  +    /* If we didn't hang on to the mutex during the IO op, obtain it now
          796  +    ** so that the AsyncWrite structure can be safely removed from the 
          797  +    ** global write-op queue.
          798  +    */
          799  +    if( !isInsideMutex ){
          800  +      sqlite3_os_enter_mutex();
          801  +    }
          802  +    if( rc==SQLITE_OK ){
          803  +      removeAsyncWrite(p);
          804  +      sqlite3Os.xFree(p);
          805  +    }
          806  +  }
          807  +
          808  +  /* Clear the io-busy flag and exit the mutex */
          809  +  assert( sqlite3_asyncIoBusy );
          810  +  sqlite3_asyncIoBusy = 0;
          811  +  sqlite3_os_leave_mutex();
          812  +
          813  +  return rc;
          814  +}
          815  +
          816  +/*
          817  +** The following code defines a Tcl interface for testing the asynchronous 
          818  +** IO implementation in this file.
          819  +*/
          820  +#if defined(SQLITE_TEST) && defined(TCLSH)
          821  +
          822  +#include <tcl.h>
          823  +
          824  +/*
          825  +** sqlite3_async_enable
          826  +*/
          827  +static int testAsyncEnable(
          828  +  void * clientData,
          829  +  Tcl_Interp *interp,
          830  +  int objc,
          831  +  Tcl_Obj *CONST objv[]
          832  +){
          833  +  if( sqlite3_async_enable() ){
          834  +    Tcl_SetResult(interp, "sqlite3_async_enable() failed", TCL_STATIC);
          835  +    return TCL_ERROR;
          836  +  }
          837  +  return TCL_OK;
          838  +}
          839  +
          840  +/*
          841  +** This is the main proc for a thread spawned by the Tcl command 
          842  +** [sqlite3_async_flush -start]. The client data is a pointer to an integer
          843  +** variable that will be set to non-zero when this thread should exit.
          844  +*/
          845  +static void testAsyncFlushThread(ClientData clientData){
          846  +  int *pStop = (int *)clientData;
          847  +  int rc = 0;
          848  +
          849  +  /* Run in a loop until an IO error occurs or we are told to stop via 
          850  +  ** the *pStop variable. Each iteration of the loop, call 
          851  +  ** sqlite3_async_flush() and then sleep for a tenth of a second.
          852  +  */
          853  +  while( !(*pStop) && !rc ){
          854  +    rc = sqlite3_async_flush();
          855  +    assert( rc==SQLITE_OK );
          856  +    Tcl_Sleep(100);
          857  +  }
          858  +  if( rc==0 ){
          859  +    rc = sqlite3_async_flush();
          860  +  }
          861  +  Tcl_ExitThread(rc);
          862  +}
          863  +
          864  +/*
          865  +** sqlite3_async_flush
          866  +** sqlite3_async_flush -start
          867  +** sqlite3_async_flush -stop
          868  +*/
          869  +static int testAsyncFlush(
          870  +  void * clientData,
          871  +  Tcl_Interp *interp,
          872  +  int objc,
          873  +  Tcl_Obj *CONST objv[]
          874  +){
          875  +  static Tcl_ThreadId thread_id = 0;
          876  +  static int stop = 0;
          877  +
          878  +  assert(stop==0);
          879  +
          880  +  if( objc!=1 && objc!=2 ){
          881  +    Tcl_WrongNumArgs(interp, 1, objv, "?-start | -stop?");
          882  +    return TCL_ERROR;
          883  +  }
          884  +
          885  +  if( objc==2 ){
          886  +    char *zOpt = Tcl_GetString(objv[1]);
          887  +    if( 0==strcmp(zOpt, "-start") ){
          888  +      /* Unless it is already running, kick off the _flush() thread */
          889  +      if( thread_id ){
          890  +        Tcl_AppendResult(interp, "Thread has already started", 0);
          891  +        return TCL_ERROR;
          892  +      }else{
          893  +        int rc = Tcl_CreateThread(
          894  +          &thread_id, 
          895  +          testAsyncFlushThread, 
          896  +          &stop, 
          897  +          TCL_THREAD_STACK_DEFAULT, 
          898  +          TCL_THREAD_JOINABLE
          899  +        );
          900  +        if( rc!=TCL_OK ){
          901  +          Tcl_AppendResult(interp, "Tcl_CreateThread() failed", 0);
          902  +          return TCL_ERROR;
          903  +        }
          904  +      }
          905  +    }else if( 0==strcmp(zOpt, "-stop") ){
          906  +      int dummy;
          907  +      stop = 1;
          908  +      Tcl_JoinThread(thread_id, &dummy);
          909  +      stop = 0;
          910  +      thread_id = 0;
          911  +    }else{
          912  +      Tcl_AppendResult(interp, "Invalid option: \"", zOpt, "\"", 0);
          913  +      return TCL_ERROR;
          914  +    }
          915  +  }else if( sqlite3_async_flush() ){
          916  +    Tcl_SetResult(interp, "sqlite3_async_flush() failed", TCL_STATIC);
          917  +    return TCL_ERROR;
          918  +  }
          919  +  return TCL_OK;
          920  +}
          921  +
          922  +int Sqlitetestasync_Init(Tcl_Interp *interp){
          923  +  Tcl_CreateObjCommand(interp,"sqlite3_async_enable",testAsyncEnable,0,0);
          924  +  Tcl_CreateObjCommand(interp,"sqlite3_async_flush",testAsyncFlush,0,0);
          925  +  return TCL_OK;
          926  +}
          927  +
          928  +#endif
          929  +

Added test/async.test.

            1  +#
            2  +#    May you do good and not evil.
            3  +#    May you find forgiveness for yourself and forgive others.
            4  +#    May you share freely, never taking more than you give.
            5  +#
            6  +#***********************************************************************
            7  +# This file runs all tests.
            8  +#
            9  +# $Id: async.test,v 1.1 2006/01/03 13:39:26 danielk1977 Exp $
           10  +
           11  +set testdir [file dirname $argv0]
           12  +source $testdir/tester.tcl
           13  +rename finish_test really_finish_test
           14  +proc finish_test {} {}
           15  +set ISQUICK 1
           16  +
           17  +if {[info exists ::tcl_platform(threaded)]} {
           18  +  set USE_THREAD 1
           19  +} else {
           20  +  set USE_THREAD 0
           21  +}
           22  +
           23  +set INCLUDE {
           24  +  select1.test
           25  +  select2.test
           26  +  select3.test
           27  +  select4.test
           28  +  insert.test
           29  +  insert2.test
           30  +  insert3.test
           31  +}
           32  +#set INCLUDE [lrange $INCLUDE 0 0]
           33  +
           34  +# Enable asynchronous IO.
           35  +sqlite3_async_enable
           36  +
           37  +# set USE_THREAD 0
           38  +if {$USE_THREAD} {
           39  +  sqlite3_async_flush -start
           40  +} else {
           41  +  rename do_test really_do_test
           42  +  proc do_test {name args} {
           43  +    uplevel really_do_test async_io-$name $args
           44  +    sqlite3_async_flush
           45  +  }
           46  +}
           47  +
           48  +foreach testfile [lsort -dictionary [glob $testdir/*.test]] {
           49  +  set tail [file tail $testfile]
           50  +  if {[lsearch -exact $INCLUDE $tail]<0} continue
           51  +  source $testfile
           52  +  catch {db close}
           53  +#  if {$sqlite_open_file_count>0} {
           54  +#    puts "$tail did not close all files: $sqlite_open_file_count"
           55  +#    incr nErr
           56  +#    lappend ::failList $tail
           57  +#  }
           58  +}
           59  +#source $testdir/misuse.test
           60  +
           61  +
           62  +if {$USE_THREAD} {
           63  +  sqlite3_async_flush -stop
           64  +} else {
           65  +  rename do_test {}
           66  +  rename really_do_test do_test
           67  +  puts "Warning: Tcl library was without --enable-threads."
           68  +  puts "         Cannot test async IO properly."
           69  +  sqlite3_async_flush
           70  +}
           71  +
           72  +set sqlite_open_file_count 0
           73  +really_finish_test
           74  +