/ Check-in [7b51269c]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Have the writefile() function optionally set the modification-time of the files it writes or creates. And many small fixes to the new code on this branch.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | sqlar-shell-support
Files: files | file ages | folders
SHA3-256: 7b51269caebe1492885fe9b965892f49a3f8bdb1d666b0203d594c30f9e83938
User & Date: dan 2017-12-14 19:15:07
Context
2017-12-16
19:11
Do not use the compress() and uncompress() functions in ext/misc/compress.c - they are not quite compatible with the spec. Instead use new functions in ext/misc/sqlar.c. check-in: 7652b3c2 user: dan tags: sqlar-shell-support
2017-12-14
19:15
Have the writefile() function optionally set the modification-time of the files it writes or creates. And many small fixes to the new code on this branch. check-in: 7b51269c user: dan tags: sqlar-shell-support
15:40
Improve error and usage messages output by the shell ".ar" command. check-in: b9d2d5d9 user: dan tags: sqlar-shell-support
Changes
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Changes to ext/misc/fileio.c.

     7      7   **    May you do good and not evil.
     8      8   **    May you find forgiveness for yourself and forgive others.
     9      9   **    May you share freely, never taking more than you give.
    10     10   **
    11     11   ******************************************************************************
    12     12   **
    13     13   ** This SQLite extension implements SQL functions readfile() and
    14         -** writefile().
           14  +** writefile(), and eponymous virtual type "fsdir".
    15     15   **
    16         -** Also, an eponymous virtual table type "fsdir". Used as follows:
           16  +** WRITEFILE(FILE, DATA [, MODE [, MTIME]]):
           17  +**
           18  +**   If neither of the optional arguments is present, then this UDF
           19  +**   function writes blob DATA to file FILE. If successful, the number
           20  +**   of bytes written is returned. If an error occurs, NULL is returned.
           21  +**
           22  +**   If the first option argument - MODE - is present, then it must
           23  +**   be passed an integer value that corresponds to a POSIX mode
           24  +**   value (file type + permissions, as returned in the stat.st_mode
           25  +**   field by the stat() system call). Three types of files may
           26  +**   be written/created:
           27  +**
           28  +**     regular files:  (mode & 0170000)==0100000
           29  +**     symbolic links: (mode & 0170000)==0120000
           30  +**     directories:    (mode & 0170000)==0040000
           31  +**
           32  +**   For a directory, the DATA is ignored. For a symbolic link, it is
           33  +**   interpreted as text and used as the target of the link. For a
           34  +**   regular file, it is interpreted as a blob and written into the
           35  +**   named file. Regardless of the type of file, its permissions are
           36  +**   set to (mode & 0777) before returning.
           37  +**
           38  +**   If the optional MTIME argument is present, then it is interpreted
           39  +**   as an integer - the number of seconds since the unix epoch. The
           40  +**   modification-time of the target file is set to this value before
           41  +**   returning.
           42  +**
           43  +**   If three or more arguments are passed to this function and an
           44  +**   error is encountered, an exception is raised.
           45  +**
           46  +** READFILE(FILE):
           47  +**
           48  +**   Read and return the contents of file FILE (type blob) from disk.
           49  +**
           50  +** FSDIR:
           51  +**
           52  +**   Used as follows:
           53  +**
           54  +**     SELECT * FROM fsdir($path [, $dir]);
           55  +**
           56  +**   Parameter $path is an absolute or relative pathname. If the file that it
           57  +**   refers to does not exist, it is an error. If the path refers to a regular
           58  +**   file or symbolic link, it returns a single row. Or, if the path refers
           59  +**   to a directory, it returns one row for the directory, and one row for each
           60  +**   file within the hierarchy rooted at $path.
    17     61   **
    18         -**   SELECT * FROM fsdir($dirname);
           62  +**   Each row has the following columns:
    19     63   **
    20         -** Returns one row for each entry in the directory $dirname. No row is
    21         -** returned for "." or "..". Row columns are as follows:
           64  +**     name:  Path to file or directory (text value).
           65  +**     mode:  Value of stat.st_mode for directory entry (an integer).
           66  +**     mtime: Value of stat.st_mtime for directory entry (an integer).
           67  +**     data:  For a regular file, a blob containing the file data. For a
           68  +**            symlink, a text value containing the text of the link. For a
           69  +**            directory, NULL.
    22     70   **
    23         -**   name:  Name of directory entry.
    24         -**   mode:  Value of stat.st_mode for directory entry.
    25         -**   mtime: Value of stat.st_mtime for directory entry.
    26         -**   data:  For a regular file, a blob containing the file data. For a
    27         -**          symlink, a text value containing the text of the link. For a
    28         -**          directory, NULL.
           71  +**   If a non-NULL value is specified for the optional $dir parameter and
           72  +**   $path is a relative path, then $path is interpreted relative to $dir. 
           73  +**   And the paths returned in the "name" column of the table are also 
           74  +**   relative to directory $dir.
    29     75   */
    30     76   #include "sqlite3ext.h"
    31     77   SQLITE_EXTENSION_INIT1
    32     78   #include <stdio.h>
    33     79   #include <string.h>
    34     80   #include <assert.h>
    35     81   
................................................................................
    41     87   #include <time.h>
    42     88   #include <utime.h>
    43     89   #include <errno.h>
    44     90   
    45     91   
    46     92   #define FSDIR_SCHEMA "(name,mode,mtime,data,path HIDDEN,dir HIDDEN)"
    47     93   
           94  +/*
           95  +** Set the result stored by context ctx to a blob containing the 
           96  +** contents of file zName.
           97  +*/
    48     98   static void readFileContents(sqlite3_context *ctx, const char *zName){
    49     99     FILE *in;
    50    100     long nIn;
    51    101     void *pBuf;
    52    102   
    53    103     in = fopen(zName, "rb");
    54    104     if( in==0 ) return;
................................................................................
    77    127     const char *zName;
    78    128     (void)(argc);  /* Unused parameter */
    79    129     zName = (const char*)sqlite3_value_text(argv[0]);
    80    130     if( zName==0 ) return;
    81    131     readFileContents(context, zName);
    82    132   }
    83    133   
          134  +/*
          135  +** Set the error message contained in context ctx to the results of
          136  +** vprintf(zFmt, ...).
          137  +*/
    84    138   static void ctxErrorMsg(sqlite3_context *ctx, const char *zFmt, ...){
    85    139     char *zMsg = 0;
    86    140     va_list ap;
    87    141     va_start(ap, zFmt);
    88    142     zMsg = sqlite3_vmprintf(zFmt, ap);
    89    143     sqlite3_result_error(ctx, zMsg, -1);
    90    144     sqlite3_free(zMsg);
................................................................................
   134    188   
   135    189       sqlite3_free(zCopy);
   136    190     }
   137    191   
   138    192     return rc;
   139    193   }
   140    194   
          195  +/*
          196  +** This function does the work for the writefile() UDF. Refer to 
          197  +** header comments at the top of this file for details.
          198  +*/
   141    199   static int writeFile(
   142         -  sqlite3_context *pCtx, 
   143         -  const char *zFile, 
   144         -  mode_t mode, 
   145         -  sqlite3_value *pData
          200  +  sqlite3_context *pCtx,          /* Context to return bytes written in */
          201  +  const char *zFile,              /* File to write */
          202  +  sqlite3_value *pData,           /* Data to write */
          203  +  mode_t mode,                    /* MODE parameter passed to writefile() */
          204  +  sqlite3_int64 mtime             /* MTIME parameter (or -1 to not set time) */
   146    205   ){
   147    206     if( S_ISLNK(mode) ){
   148    207       const char *zTo = (const char*)sqlite3_value_text(pData);
   149    208       if( symlink(zTo, zFile)<0 ) return 1;
   150    209     }else{
   151    210       if( S_ISDIR(mode) ){
   152    211         if( mkdir(zFile, mode) ){
................................................................................
   174    233           sqlite3_int64 n = fwrite(z, 1, sqlite3_value_bytes(pData), out);
   175    234           nWrite = sqlite3_value_bytes(pData);
   176    235           if( nWrite!=n ){
   177    236             rc = 1;
   178    237           }
   179    238         }
   180    239         fclose(out);
   181         -      if( rc==0 && chmod(zFile, mode & 0777) ){
          240  +      if( rc==0 && mode && chmod(zFile, mode & 0777) ){
   182    241           rc = 1;
   183    242         }
   184    243         if( rc ) return 2;
   185    244         sqlite3_result_int64(pCtx, nWrite);
   186    245       }
   187    246     }
          247  +
          248  +  if( mtime>=0 ){
          249  +    struct timespec times[2];
          250  +    times[0].tv_nsec = times[1].tv_nsec = 0;
          251  +    times[0].tv_sec = time(0);
          252  +    times[1].tv_sec = mtime;
          253  +    if( utimensat(AT_FDCWD, zFile, times, AT_SYMLINK_NOFOLLOW) ){
          254  +      return 1;
          255  +    }
          256  +  }
          257  +
   188    258     return 0;
   189    259   }
   190    260   
   191    261   /*
   192         -** Implementation of the "writefile(W,X[,Y]])" SQL function.  
   193         -**
   194         -** The argument X is written into file W.  The number of bytes written is
   195         -** returned. Or NULL is returned if something goes wrong, such as being unable
   196         -** to open file X for writing.
          262  +** Implementation of the "writefile(W,X[,Y[,Z]]])" SQL function.  
          263  +** Refer to header comments at the top of this file for details.
   197    264   */
   198    265   static void writefileFunc(
   199    266     sqlite3_context *context,
   200    267     int argc,
   201    268     sqlite3_value **argv
   202    269   ){
   203    270     const char *zFile;
   204    271     mode_t mode = 0;
   205    272     int res;
          273  +  sqlite3_int64 mtime = -1;
   206    274   
   207         -  if( argc<2 || argc>3 ){
          275  +  if( argc<2 || argc>4 ){
   208    276       sqlite3_result_error(context, 
   209    277           "wrong number of arguments to function writefile()", -1
   210    278       );
   211    279       return;
   212    280     }
   213    281   
   214    282     zFile = (const char*)sqlite3_value_text(argv[0]);
   215    283     if( zFile==0 ) return;
   216    284     if( argc>=3 ){
   217         -    sqlite3_result_int(context, 0);
   218    285       mode = sqlite3_value_int(argv[2]);
   219    286     }
          287  +  if( argc==4 ){
          288  +    mtime = sqlite3_value_int64(argv[3]);
          289  +  }
   220    290   
   221         -  res = writeFile(context, zFile, mode, argv[1]);
          291  +  res = writeFile(context, zFile, argv[1], mode, mtime);
   222    292     if( res==1 && errno==ENOENT ){
   223    293       if( makeDirectory(zFile, mode)==SQLITE_OK ){
   224         -      res = writeFile(context, zFile, mode, argv[1]);
          294  +      res = writeFile(context, zFile, argv[1], mode, mtime);
   225    295       }
   226    296     }
   227    297   
   228         -  if( res!=0 ){
          298  +  if( argc>2 && res!=0 ){
   229    299       if( S_ISLNK(mode) ){
   230    300         ctxErrorMsg(context, "failed to create symlink: %s", zFile);
   231    301       }else if( S_ISDIR(mode) ){
   232    302         ctxErrorMsg(context, "failed to create directory: %s", zFile);
   233    303       }else{
   234    304         ctxErrorMsg(context, "failed to write file: %s", zFile);
   235    305       }
................................................................................
   309    379     if( pCur==0 ) return SQLITE_NOMEM;
   310    380     memset(pCur, 0, sizeof(*pCur));
   311    381     pCur->iLvl = -1;
   312    382     *ppCursor = &pCur->base;
   313    383     return SQLITE_OK;
   314    384   }
   315    385   
          386  +/*
          387  +** Reset a cursor back to the state it was in when first returned
          388  +** by fsdirOpen().
          389  +*/
   316    390   static void fsdirResetCursor(fsdir_cursor *pCur){
   317    391     int i;
   318    392     for(i=0; i<=pCur->iLvl; i++){
   319    393       FsdirLevel *pLvl = &pCur->aLvl[i];
   320    394       if( pLvl->pDir ) closedir(pLvl->pDir);
   321    395       sqlite3_free(pLvl->zDir);
   322    396     }
................................................................................
   337    411   
   338    412     fsdirResetCursor(pCur);
   339    413     sqlite3_free(pCur->aLvl);
   340    414     sqlite3_free(pCur);
   341    415     return SQLITE_OK;
   342    416   }
   343    417   
          418  +/*
          419  +** Set the error message for the virtual table associated with cursor
          420  +** pCur to the results of vprintf(zFmt, ...).
          421  +*/
   344    422   static void fsdirSetErrmsg(fsdir_cursor *pCur, const char *zFmt, ...){
   345    423     va_list ap;
   346    424     va_start(ap, zFmt);
   347    425     pCur->base.pVtab->zErrMsg = sqlite3_vmprintf(zFmt, ap);
   348    426     va_end(ap);
   349    427   }
   350    428   
................................................................................
   582    660         pIdxInfo->estimatedCost = 100.0;
   583    661       }
   584    662     }
   585    663   
   586    664     return SQLITE_OK;
   587    665   }
   588    666   
          667  +/*
          668  +** Register the "fsdir" virtual table.
          669  +*/
   589    670   static int fsdirRegister(sqlite3 *db){
   590    671     static sqlite3_module fsdirModule = {
   591    672       0,                         /* iVersion */
   592    673       0,                         /* xCreate */
   593    674       fsdirConnect,              /* xConnect */
   594    675       fsdirBestIndex,            /* xBestIndex */
   595    676       fsdirDisconnect,           /* xDisconnect */
................................................................................
   607    688       0,                         /* xCommit */
   608    689       0,                         /* xRollback */
   609    690       0,                         /* xFindMethod */
   610    691       0,                         /* xRename */
   611    692     };
   612    693   
   613    694     int rc = sqlite3_create_module(db, "fsdir", &fsdirModule, 0);
   614         -  if( rc==SQLITE_OK ){
   615         -    rc = sqlite3_create_module(db, "fsentry", &fsdirModule, (void*)1);
   616         -  }
   617    695     return rc;
   618    696   }
   619    697   #else         /* SQLITE_OMIT_VIRTUALTABLE */
   620    698   # define fsdirRegister(x) SQLITE_OK
   621    699   #endif
   622    700   
   623    701   #ifdef _WIN32

Changes to src/shell.c.in.

  4088   4088             sqlite3_errmsg(db), sqlite3_errcode(db)
  4089   4089         );
  4090   4090         *pRc = rc;
  4091   4091       }
  4092   4092     }
  4093   4093   }
  4094   4094   
  4095         -static void shellPrepare2(
         4095  +static void shellPreparePrintf(
  4096   4096     sqlite3 *db, 
  4097   4097     int *pRc, 
  4098         -  const char *zSql, 
  4099         -  const char *zTail, 
  4100         -  sqlite3_stmt **ppStmt
         4098  +  sqlite3_stmt **ppStmt,
         4099  +  const char *zFmt, 
         4100  +  ...
  4101   4101   ){
  4102         -  if( *pRc==SQLITE_OK && zTail ){
  4103         -    char *z = sqlite3_mprintf("%s %s", zSql, zTail);
         4102  +  *ppStmt = 0;
         4103  +  if( *pRc==SQLITE_OK ){
         4104  +    va_list ap;
         4105  +    char *z;
         4106  +    va_start(ap, zFmt);
         4107  +    z = sqlite3_vmprintf(zFmt, ap);
  4104   4108       if( z==0 ){
  4105   4109         *pRc = SQLITE_NOMEM;
  4106   4110       }else{
  4107   4111         shellPrepare(db, pRc, z, ppStmt);
  4108   4112         sqlite3_free(z);
  4109   4113       }
  4110         -  }else{
  4111         -    shellPrepare(db, pRc, zSql, ppStmt);
  4112   4114     }
  4113   4115   }
  4114   4116   
  4115   4117   static void shellFinalize(
  4116   4118     int *pRc, 
  4117   4119     sqlite3_stmt *pStmt
  4118   4120   ){
................................................................................
  4425   4427   ** in (*pAr). Leave this WHERE clause in (*pzWhere) before returning.
  4426   4428   ** The caller is responsible for eventually calling sqlite3_free() on
  4427   4429   ** any non-NULL (*pzWhere) value.
  4428   4430   */
  4429   4431   static void arWhereClause(
  4430   4432     int *pRc, 
  4431   4433     ArCommand *pAr, 
  4432         -  char **pzWhere                  /* OUT: New WHERE clause (or NULL) */
         4434  +  char **pzWhere                  /* OUT: New WHERE clause */
  4433   4435   ){
  4434   4436     char *zWhere = 0;
  4435   4437     if( *pRc==SQLITE_OK ){
  4436         -    int i;
  4437         -    const char *zSep = "WHERE ";
  4438         -    for(i=0; i<pAr->nArg; i++){
  4439         -      const char *z = pAr->azArg[i];
  4440         -      zWhere = sqlite3_mprintf(
  4441         -          "%z%s name = '%q' OR name BETWEEN '%q/' AND '%q0'", 
  4442         -          zWhere, zSep, z, z, z
  4443         -      );
  4444         -      if( zWhere==0 ){
  4445         -        *pRc = SQLITE_NOMEM;
  4446         -        break;
         4438  +    if( pAr->nArg==0 ){
         4439  +      zWhere = sqlite3_mprintf("1");
         4440  +    }else{
         4441  +      int i;
         4442  +      const char *zSep = "";
         4443  +      for(i=0; i<pAr->nArg; i++){
         4444  +        const char *z = pAr->azArg[i];
         4445  +        zWhere = sqlite3_mprintf(
         4446  +            "%z%s name = '%q' OR name BETWEEN '%q/' AND '%q0'", 
         4447  +            zWhere, zSep, z, z, z
         4448  +            );
         4449  +        if( zWhere==0 ){
         4450  +          *pRc = SQLITE_NOMEM;
         4451  +          break;
         4452  +        }
         4453  +        zSep = " OR ";
  4447   4454         }
  4448         -      zSep = " OR ";
  4449   4455       }
  4450   4456     }
  4451   4457     *pzWhere = zWhere;
  4452   4458   }
  4453   4459   
  4454   4460   /*
  4455   4461   ** Implementation of .ar "lisT" command. 
  4456   4462   */
  4457   4463   static int arListCommand(ShellState *p, sqlite3 *db, ArCommand *pAr){
  4458         -  const char *zSql = "SELECT name FROM sqlar"; 
         4464  +  const char *zSql = "SELECT name FROM sqlar WHERE %s"; 
  4459   4465     char *zWhere = 0;
  4460   4466     sqlite3_stmt *pSql = 0;
  4461   4467     int rc;
  4462   4468   
  4463   4469     rc = arCheckEntries(db, pAr);
  4464   4470     arWhereClause(&rc, pAr, &zWhere);
  4465   4471   
  4466         -  shellPrepare2(db, &rc, zSql, zWhere, &pSql);
         4472  +  shellPreparePrintf(db, &rc, &pSql, zSql, zWhere);
  4467   4473     while( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pSql) ){
  4468   4474       raw_printf(p->out, "%s\n", sqlite3_column_text(pSql, 0));
  4469   4475     }
  4470   4476     return rc;
  4471   4477   }
  4472   4478   
  4473   4479   
................................................................................
  4478   4484     const char *zSql1 = 
  4479   4485       "SELECT :1 || name, writefile(:1 || name, "
  4480   4486       "CASE WHEN (data AND sz>=0 AND sz!=length(data)) THEN "
  4481   4487       "    uncompress(data) "
  4482   4488       "ELSE"
  4483   4489       "    data "
  4484   4490       "END, "
  4485         -    "mode) FROM sqlar";
  4486         -  const char *zSql2 = "SELECT :1 || name, mtime FROM sqlar"; 
         4491  +    "mode, mtime) FROM sqlar WHERE (%s) AND (data IS NULL OR :2 = 0)";
  4487   4492   
  4488   4493     struct timespec times[2];
  4489   4494     sqlite3_stmt *pSql = 0;
  4490   4495     int rc = SQLITE_OK;
  4491   4496     char *zDir = 0;
  4492   4497     char *zWhere = 0;
         4498  +  int i;
  4493   4499   
  4494   4500     /* If arguments are specified, check that they actually exist within
  4495   4501     ** the archive before proceeding. And formulate a WHERE clause to
  4496   4502     ** match them.  */
  4497   4503     rc = arCheckEntries(db, pAr);
  4498   4504     arWhereClause(&rc, pAr, &zWhere);
  4499   4505   
................................................................................
  4505   4511       }
  4506   4512       if( zDir==0 ) rc = SQLITE_NOMEM;
  4507   4513     }
  4508   4514   
  4509   4515     memset(times, 0, sizeof(times));
  4510   4516     times[0].tv_sec = time(0);
  4511   4517   
  4512         -  shellPrepare2(db, &rc, zSql1, zWhere, &pSql);
         4518  +  shellPreparePrintf(db, &rc, &pSql, zSql1, zWhere);
  4513   4519     if( rc==SQLITE_OK ){
  4514   4520       sqlite3_bind_text(pSql, 1, zDir, -1, SQLITE_STATIC);
         4521  +
         4522  +    /* Run the SELECT statement twice. The first time, writefile() is called
         4523  +    ** for all archive members that should be extracted. The second time,
         4524  +    ** only for the directories. This is because the timestamps for
         4525  +    ** extracted directories must be reset after they are populated (as
         4526  +    ** populating them changes the timestamp).  */
         4527  +    for(i=0; i<2; i++){
         4528  +      sqlite3_bind_int(pSql, 2, i);
         4529  +      while( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pSql) ){
         4530  +        if( i==0 && pAr->bVerbose ){
         4531  +          raw_printf(p->out, "%s\n", sqlite3_column_text(pSql, 0));
         4532  +        }
         4533  +      }
         4534  +      shellReset(&rc, pSql);
         4535  +    }
         4536  +    shellFinalize(&rc, pSql);
  4515   4537     }
  4516         -  while( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pSql) ){
  4517         -    if( pAr->bVerbose ){
  4518         -      raw_printf(p->out, "%s\n", sqlite3_column_text(pSql, 0));
  4519         -    }
  4520         -  }
  4521         -  shellFinalize(&rc, pSql);
  4522         -
  4523         -  shellPrepare2(db, &rc, zSql2, zWhere, &pSql);
  4524         -  if( rc==SQLITE_OK ){
  4525         -    sqlite3_bind_text(pSql, 1, zDir, -1, SQLITE_STATIC);
  4526         -  }
  4527         -  while( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pSql) ){
  4528         -    const char *zPath = (const char*)sqlite3_column_text(pSql, 0);
  4529         -    times[1].tv_sec = (time_t)sqlite3_column_int64(pSql, 1);
  4530         -    if( utimensat(AT_FDCWD, zPath, times, AT_SYMLINK_NOFOLLOW) ){
  4531         -      raw_printf(stderr, "failed to set timestamp for %s\n", zPath);
  4532         -      rc = SQLITE_ERROR;
  4533         -      break;
  4534         -    }
  4535         -  }
  4536         -  shellFinalize(&rc, pSql);
  4537   4538   
  4538   4539     sqlite3_free(zDir);
  4539   4540     sqlite3_free(zWhere);
  4540   4541     return rc;
  4541   4542   }
  4542   4543   
  4543   4544   

Changes to test/shell8.test.

   135    135     do_test 1.$tn.3 {
   136    136       file delete -force ar3
   137    137       file delete -force test_xyz.db
   138    138       catchcmd ":memory:" $c3
   139    139       catchcmd ":memory:" $x3
   140    140       dir_to_list ar3
   141    141     } $expected
          142  +
          143  +  # This is a repeat of test 1.$tn.1, except that there is a 2 second 
          144  +  # pause between creating the archive and extracting its contents.
          145  +  # This is to test that timestamps are set correctly.
          146  +  #
          147  +  # Because it is slow, only do this for $tn==1.
          148  +  if {$tn==1} {
          149  +    do_test 1.$tn.1 {
          150  +      catchcmd test_ar.db $c1
          151  +      file delete -force ar1
          152  +      after 2000
          153  +      catchcmd test_ar.db $x1
          154  +      dir_to_list ar1
          155  +    } $expected
          156  +  }
   142    157   }
   143    158   
   144    159   finish_test
   145    160   
   146    161   
   147    162   
   148    163   finish_test
   149    164