Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
Comment: | Have the writefile() function optionally set the modification-time of the files it writes or creates. And many small fixes to the new code on this branch. |
---|---|
Downloads: | Tarball | ZIP archive |
Timelines: | family | ancestors | descendants | both | sqlar-shell-support |
Files: | files | file ages | folders |
SHA3-256: |
7b51269caebe1492885fe9b965892f49 |
User & Date: | dan 2017-12-14 19:15:07.381 |
Context
2017-12-16
| ||
19:11 | Do not use the compress() and uncompress() functions in ext/misc/compress.c - they are not quite compatible with the spec. Instead use new functions in ext/misc/sqlar.c. (check-in: 7652b3c237 user: dan tags: sqlar-shell-support) | |
2017-12-14
| ||
19:15 | Have the writefile() function optionally set the modification-time of the files it writes or creates. And many small fixes to the new code on this branch. (check-in: 7b51269cae user: dan tags: sqlar-shell-support) | |
15:40 | Improve error and usage messages output by the shell ".ar" command. (check-in: b9d2d5d972 user: dan tags: sqlar-shell-support) | |
Changes
Changes to ext/misc/fileio.c.
1 2 3 4 5 6 7 8 9 10 11 12 13 | /* ** 2014-06-13 ** ** The author disclaims copyright to this source code. In place of ** a legal notice, here is a blessing: ** ** May you do good and not evil. ** May you find forgiveness for yourself and forgive others. ** May you share freely, never taking more than you give. ** ****************************************************************************** ** ** This SQLite extension implements SQL functions readfile() and | | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | | > > > | > > | | | | | | | > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 | /* ** 2014-06-13 ** ** The author disclaims copyright to this source code. In place of ** a legal notice, here is a blessing: ** ** May you do good and not evil. ** May you find forgiveness for yourself and forgive others. ** May you share freely, never taking more than you give. ** ****************************************************************************** ** ** This SQLite extension implements SQL functions readfile() and ** writefile(), and eponymous virtual type "fsdir". ** ** WRITEFILE(FILE, DATA [, MODE [, MTIME]]): ** ** If neither of the optional arguments is present, then this UDF ** function writes blob DATA to file FILE. If successful, the number ** of bytes written is returned. If an error occurs, NULL is returned. ** ** If the first option argument - MODE - is present, then it must ** be passed an integer value that corresponds to a POSIX mode ** value (file type + permissions, as returned in the stat.st_mode ** field by the stat() system call). Three types of files may ** be written/created: ** ** regular files: (mode & 0170000)==0100000 ** symbolic links: (mode & 0170000)==0120000 ** directories: (mode & 0170000)==0040000 ** ** For a directory, the DATA is ignored. For a symbolic link, it is ** interpreted as text and used as the target of the link. For a ** regular file, it is interpreted as a blob and written into the ** named file. Regardless of the type of file, its permissions are ** set to (mode & 0777) before returning. ** ** If the optional MTIME argument is present, then it is interpreted ** as an integer - the number of seconds since the unix epoch. The ** modification-time of the target file is set to this value before ** returning. ** ** If three or more arguments are passed to this function and an ** error is encountered, an exception is raised. ** ** READFILE(FILE): ** ** Read and return the contents of file FILE (type blob) from disk. ** ** FSDIR: ** ** Used as follows: ** ** SELECT * FROM fsdir($path [, $dir]); ** ** Parameter $path is an absolute or relative pathname. If the file that it ** refers to does not exist, it is an error. If the path refers to a regular ** file or symbolic link, it returns a single row. Or, if the path refers ** to a directory, it returns one row for the directory, and one row for each ** file within the hierarchy rooted at $path. ** ** Each row has the following columns: ** ** name: Path to file or directory (text value). ** mode: Value of stat.st_mode for directory entry (an integer). ** mtime: Value of stat.st_mtime for directory entry (an integer). ** data: For a regular file, a blob containing the file data. For a ** symlink, a text value containing the text of the link. For a ** directory, NULL. ** ** If a non-NULL value is specified for the optional $dir parameter and ** $path is a relative path, then $path is interpreted relative to $dir. ** And the paths returned in the "name" column of the table are also ** relative to directory $dir. */ #include "sqlite3ext.h" SQLITE_EXTENSION_INIT1 #include <stdio.h> #include <string.h> #include <assert.h> #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> #include <unistd.h> #include <dirent.h> #include <time.h> #include <utime.h> #include <errno.h> #define FSDIR_SCHEMA "(name,mode,mtime,data,path HIDDEN,dir HIDDEN)" /* ** Set the result stored by context ctx to a blob containing the ** contents of file zName. */ static void readFileContents(sqlite3_context *ctx, const char *zName){ FILE *in; long nIn; void *pBuf; in = fopen(zName, "rb"); if( in==0 ) return; |
︙ | ︙ | |||
77 78 79 80 81 82 83 84 85 86 87 88 89 90 | const char *zName; (void)(argc); /* Unused parameter */ zName = (const char*)sqlite3_value_text(argv[0]); if( zName==0 ) return; readFileContents(context, zName); } static void ctxErrorMsg(sqlite3_context *ctx, const char *zFmt, ...){ char *zMsg = 0; va_list ap; va_start(ap, zFmt); zMsg = sqlite3_vmprintf(zFmt, ap); sqlite3_result_error(ctx, zMsg, -1); sqlite3_free(zMsg); | > > > > | 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 | const char *zName; (void)(argc); /* Unused parameter */ zName = (const char*)sqlite3_value_text(argv[0]); if( zName==0 ) return; readFileContents(context, zName); } /* ** Set the error message contained in context ctx to the results of ** vprintf(zFmt, ...). */ static void ctxErrorMsg(sqlite3_context *ctx, const char *zFmt, ...){ char *zMsg = 0; va_list ap; va_start(ap, zFmt); zMsg = sqlite3_vmprintf(zFmt, ap); sqlite3_result_error(ctx, zMsg, -1); sqlite3_free(zMsg); |
︙ | ︙ | |||
134 135 136 137 138 139 140 141 | sqlite3_free(zCopy); } return rc; } static int writeFile( | > > > > | | < | > > | 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 | sqlite3_free(zCopy); } return rc; } /* ** This function does the work for the writefile() UDF. Refer to ** header comments at the top of this file for details. */ static int writeFile( sqlite3_context *pCtx, /* Context to return bytes written in */ const char *zFile, /* File to write */ sqlite3_value *pData, /* Data to write */ mode_t mode, /* MODE parameter passed to writefile() */ sqlite3_int64 mtime /* MTIME parameter (or -1 to not set time) */ ){ if( S_ISLNK(mode) ){ const char *zTo = (const char*)sqlite3_value_text(pData); if( symlink(zTo, zFile)<0 ) return 1; }else{ if( S_ISDIR(mode) ){ if( mkdir(zFile, mode) ){ |
︙ | ︙ | |||
174 175 176 177 178 179 180 | sqlite3_int64 n = fwrite(z, 1, sqlite3_value_bytes(pData), out); nWrite = sqlite3_value_bytes(pData); if( nWrite!=n ){ rc = 1; } } fclose(out); | | > > > > > > > > > > > | | < < < > | < > > | > | | | | 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 | sqlite3_int64 n = fwrite(z, 1, sqlite3_value_bytes(pData), out); nWrite = sqlite3_value_bytes(pData); if( nWrite!=n ){ rc = 1; } } fclose(out); if( rc==0 && mode && chmod(zFile, mode & 0777) ){ rc = 1; } if( rc ) return 2; sqlite3_result_int64(pCtx, nWrite); } } if( mtime>=0 ){ struct timespec times[2]; times[0].tv_nsec = times[1].tv_nsec = 0; times[0].tv_sec = time(0); times[1].tv_sec = mtime; if( utimensat(AT_FDCWD, zFile, times, AT_SYMLINK_NOFOLLOW) ){ return 1; } } return 0; } /* ** Implementation of the "writefile(W,X[,Y[,Z]]])" SQL function. ** Refer to header comments at the top of this file for details. */ static void writefileFunc( sqlite3_context *context, int argc, sqlite3_value **argv ){ const char *zFile; mode_t mode = 0; int res; sqlite3_int64 mtime = -1; if( argc<2 || argc>4 ){ sqlite3_result_error(context, "wrong number of arguments to function writefile()", -1 ); return; } zFile = (const char*)sqlite3_value_text(argv[0]); if( zFile==0 ) return; if( argc>=3 ){ mode = sqlite3_value_int(argv[2]); } if( argc==4 ){ mtime = sqlite3_value_int64(argv[3]); } res = writeFile(context, zFile, argv[1], mode, mtime); if( res==1 && errno==ENOENT ){ if( makeDirectory(zFile, mode)==SQLITE_OK ){ res = writeFile(context, zFile, argv[1], mode, mtime); } } if( argc>2 && res!=0 ){ if( S_ISLNK(mode) ){ ctxErrorMsg(context, "failed to create symlink: %s", zFile); }else if( S_ISDIR(mode) ){ ctxErrorMsg(context, "failed to create directory: %s", zFile); }else{ ctxErrorMsg(context, "failed to write file: %s", zFile); } |
︙ | ︙ | |||
309 310 311 312 313 314 315 316 317 318 319 320 321 322 | if( pCur==0 ) return SQLITE_NOMEM; memset(pCur, 0, sizeof(*pCur)); pCur->iLvl = -1; *ppCursor = &pCur->base; return SQLITE_OK; } static void fsdirResetCursor(fsdir_cursor *pCur){ int i; for(i=0; i<=pCur->iLvl; i++){ FsdirLevel *pLvl = &pCur->aLvl[i]; if( pLvl->pDir ) closedir(pLvl->pDir); sqlite3_free(pLvl->zDir); } | > > > > | 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 | if( pCur==0 ) return SQLITE_NOMEM; memset(pCur, 0, sizeof(*pCur)); pCur->iLvl = -1; *ppCursor = &pCur->base; return SQLITE_OK; } /* ** Reset a cursor back to the state it was in when first returned ** by fsdirOpen(). */ static void fsdirResetCursor(fsdir_cursor *pCur){ int i; for(i=0; i<=pCur->iLvl; i++){ FsdirLevel *pLvl = &pCur->aLvl[i]; if( pLvl->pDir ) closedir(pLvl->pDir); sqlite3_free(pLvl->zDir); } |
︙ | ︙ | |||
337 338 339 340 341 342 343 344 345 346 347 348 349 350 | fsdirResetCursor(pCur); sqlite3_free(pCur->aLvl); sqlite3_free(pCur); return SQLITE_OK; } static void fsdirSetErrmsg(fsdir_cursor *pCur, const char *zFmt, ...){ va_list ap; va_start(ap, zFmt); pCur->base.pVtab->zErrMsg = sqlite3_vmprintf(zFmt, ap); va_end(ap); } | > > > > | 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 | fsdirResetCursor(pCur); sqlite3_free(pCur->aLvl); sqlite3_free(pCur); return SQLITE_OK; } /* ** Set the error message for the virtual table associated with cursor ** pCur to the results of vprintf(zFmt, ...). */ static void fsdirSetErrmsg(fsdir_cursor *pCur, const char *zFmt, ...){ va_list ap; va_start(ap, zFmt); pCur->base.pVtab->zErrMsg = sqlite3_vmprintf(zFmt, ap); va_end(ap); } |
︙ | ︙ | |||
582 583 584 585 586 587 588 589 590 591 592 593 594 595 | pIdxInfo->estimatedCost = 100.0; } } return SQLITE_OK; } static int fsdirRegister(sqlite3 *db){ static sqlite3_module fsdirModule = { 0, /* iVersion */ 0, /* xCreate */ fsdirConnect, /* xConnect */ fsdirBestIndex, /* xBestIndex */ fsdirDisconnect, /* xDisconnect */ | > > > | 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 | pIdxInfo->estimatedCost = 100.0; } } return SQLITE_OK; } /* ** Register the "fsdir" virtual table. */ static int fsdirRegister(sqlite3 *db){ static sqlite3_module fsdirModule = { 0, /* iVersion */ 0, /* xCreate */ fsdirConnect, /* xConnect */ fsdirBestIndex, /* xBestIndex */ fsdirDisconnect, /* xDisconnect */ |
︙ | ︙ | |||
607 608 609 610 611 612 613 | 0, /* xCommit */ 0, /* xRollback */ 0, /* xFindMethod */ 0, /* xRename */ }; int rc = sqlite3_create_module(db, "fsdir", &fsdirModule, 0); | < < < | 688 689 690 691 692 693 694 695 696 697 698 699 700 701 | 0, /* xCommit */ 0, /* xRollback */ 0, /* xFindMethod */ 0, /* xRename */ }; int rc = sqlite3_create_module(db, "fsdir", &fsdirModule, 0); return rc; } #else /* SQLITE_OMIT_VIRTUALTABLE */ # define fsdirRegister(x) SQLITE_OK #endif #ifdef _WIN32 |
︙ | ︙ |
Changes to src/shell.c.in.
︙ | ︙ | |||
4088 4089 4090 4091 4092 4093 4094 | sqlite3_errmsg(db), sqlite3_errcode(db) ); *pRc = rc; } } } | | < < | > > > | > | > > < < | 4088 4089 4090 4091 4092 4093 4094 4095 4096 4097 4098 4099 4100 4101 4102 4103 4104 4105 4106 4107 4108 4109 4110 4111 4112 4113 4114 4115 4116 4117 4118 4119 4120 | sqlite3_errmsg(db), sqlite3_errcode(db) ); *pRc = rc; } } } static void shellPreparePrintf( sqlite3 *db, int *pRc, sqlite3_stmt **ppStmt, const char *zFmt, ... ){ *ppStmt = 0; if( *pRc==SQLITE_OK ){ va_list ap; char *z; va_start(ap, zFmt); z = sqlite3_vmprintf(zFmt, ap); if( z==0 ){ *pRc = SQLITE_NOMEM; }else{ shellPrepare(db, pRc, z, ppStmt); sqlite3_free(z); } } } static void shellFinalize( int *pRc, sqlite3_stmt *pStmt ){ |
︙ | ︙ | |||
4425 4426 4427 4428 4429 4430 4431 | ** in (*pAr). Leave this WHERE clause in (*pzWhere) before returning. ** The caller is responsible for eventually calling sqlite3_free() on ** any non-NULL (*pzWhere) value. */ static void arWhereClause( int *pRc, ArCommand *pAr, | | > > > | | | | | | | | | | | | | > | | | < > | | > > > > > > > | | | | | | | | < < < < < < < < < < < < | 4427 4428 4429 4430 4431 4432 4433 4434 4435 4436 4437 4438 4439 4440 4441 4442 4443 4444 4445 4446 4447 4448 4449 4450 4451 4452 4453 4454 4455 4456 4457 4458 4459 4460 4461 4462 4463 4464 4465 4466 4467 4468 4469 4470 4471 4472 4473 4474 4475 4476 4477 4478 4479 4480 4481 4482 4483 4484 4485 4486 4487 4488 4489 4490 4491 4492 4493 4494 4495 4496 4497 4498 4499 4500 4501 4502 4503 4504 4505 4506 4507 4508 4509 4510 4511 4512 4513 4514 4515 4516 4517 4518 4519 4520 4521 4522 4523 4524 4525 4526 4527 4528 4529 4530 4531 4532 4533 4534 4535 4536 4537 4538 4539 4540 4541 4542 4543 4544 | ** in (*pAr). Leave this WHERE clause in (*pzWhere) before returning. ** The caller is responsible for eventually calling sqlite3_free() on ** any non-NULL (*pzWhere) value. */ static void arWhereClause( int *pRc, ArCommand *pAr, char **pzWhere /* OUT: New WHERE clause */ ){ char *zWhere = 0; if( *pRc==SQLITE_OK ){ if( pAr->nArg==0 ){ zWhere = sqlite3_mprintf("1"); }else{ int i; const char *zSep = ""; for(i=0; i<pAr->nArg; i++){ const char *z = pAr->azArg[i]; zWhere = sqlite3_mprintf( "%z%s name = '%q' OR name BETWEEN '%q/' AND '%q0'", zWhere, zSep, z, z, z ); if( zWhere==0 ){ *pRc = SQLITE_NOMEM; break; } zSep = " OR "; } } } *pzWhere = zWhere; } /* ** Implementation of .ar "lisT" command. */ static int arListCommand(ShellState *p, sqlite3 *db, ArCommand *pAr){ const char *zSql = "SELECT name FROM sqlar WHERE %s"; char *zWhere = 0; sqlite3_stmt *pSql = 0; int rc; rc = arCheckEntries(db, pAr); arWhereClause(&rc, pAr, &zWhere); shellPreparePrintf(db, &rc, &pSql, zSql, zWhere); while( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pSql) ){ raw_printf(p->out, "%s\n", sqlite3_column_text(pSql, 0)); } return rc; } /* ** Implementation of .ar "eXtract" command. */ static int arExtractCommand(ShellState *p, sqlite3 *db, ArCommand *pAr){ const char *zSql1 = "SELECT :1 || name, writefile(:1 || name, " "CASE WHEN (data AND sz>=0 AND sz!=length(data)) THEN " " uncompress(data) " "ELSE" " data " "END, " "mode, mtime) FROM sqlar WHERE (%s) AND (data IS NULL OR :2 = 0)"; struct timespec times[2]; sqlite3_stmt *pSql = 0; int rc = SQLITE_OK; char *zDir = 0; char *zWhere = 0; int i; /* If arguments are specified, check that they actually exist within ** the archive before proceeding. And formulate a WHERE clause to ** match them. */ rc = arCheckEntries(db, pAr); arWhereClause(&rc, pAr, &zWhere); if( rc==SQLITE_OK ){ if( pAr->zDir ){ zDir = sqlite3_mprintf("%s/", pAr->zDir); }else{ zDir = sqlite3_mprintf(""); } if( zDir==0 ) rc = SQLITE_NOMEM; } memset(times, 0, sizeof(times)); times[0].tv_sec = time(0); shellPreparePrintf(db, &rc, &pSql, zSql1, zWhere); if( rc==SQLITE_OK ){ sqlite3_bind_text(pSql, 1, zDir, -1, SQLITE_STATIC); /* Run the SELECT statement twice. The first time, writefile() is called ** for all archive members that should be extracted. The second time, ** only for the directories. This is because the timestamps for ** extracted directories must be reset after they are populated (as ** populating them changes the timestamp). */ for(i=0; i<2; i++){ sqlite3_bind_int(pSql, 2, i); while( rc==SQLITE_OK && SQLITE_ROW==sqlite3_step(pSql) ){ if( i==0 && pAr->bVerbose ){ raw_printf(p->out, "%s\n", sqlite3_column_text(pSql, 0)); } } shellReset(&rc, pSql); } shellFinalize(&rc, pSql); } sqlite3_free(zDir); sqlite3_free(zWhere); return rc; } |
︙ | ︙ |
Changes to test/shell8.test.
︙ | ︙ | |||
135 136 137 138 139 140 141 142 143 144 145 146 147 148 | do_test 1.$tn.3 { file delete -force ar3 file delete -force test_xyz.db catchcmd ":memory:" $c3 catchcmd ":memory:" $x3 dir_to_list ar3 } $expected } finish_test finish_test | > > > > > > > > > > > > > > > | 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 | do_test 1.$tn.3 { file delete -force ar3 file delete -force test_xyz.db catchcmd ":memory:" $c3 catchcmd ":memory:" $x3 dir_to_list ar3 } $expected # This is a repeat of test 1.$tn.1, except that there is a 2 second # pause between creating the archive and extracting its contents. # This is to test that timestamps are set correctly. # # Because it is slow, only do this for $tn==1. if {$tn==1} { do_test 1.$tn.1 { catchcmd test_ar.db $c1 file delete -force ar1 after 2000 catchcmd test_ar.db $x1 dir_to_list ar1 } $expected } } finish_test finish_test |
︙ | ︙ |