Documentation Source Text

<title>SQLite Backup API</title>


<style>
  pre a:visited, pre a:link { text-decoration: none ; color: #40534b }
  pre {
    background: #F3F3F3;
    float: right;
    padding: 1ex 2ex;
................................................................................

  <li> If a power failure or operating system failure occurs while copying
       the database file the backup database may be corrupted following
       system recovery.
</ul>

<p>
  The <a href="c3ref/backup.html">Online Backup API</a> was created to 
  address these concerns. The online backup API allows the contents of
  one database to be copied into another database, overwriting the 
  original contents of the target database. The copy operation may be 
  done incrementally, in which case the source database does not need
  to be locked for the duration of the copy, only for the brief periods
  of time when it is actually being read from. This allows other database
  users to continue uninterrupted while a backup of an online database is
  made.

<p>
  The online backup API is <a href="c3ref/backup.html">documented here</a>.
  The remainder of this page contains two C language examples illustrating 
  common uses of the API and discussions thereof. Reading these examples
  is no substitute for reading the API documentation!

<h2>Example 1: Loading and Saving In-Memory Databases</h2>

<tcl>
................................................................................
  **
  ** If the operation is successful, SQLITE_OK is returned. Otherwise, if
  ** an error occurs, an SQLite error code is returned.
  */
  int loadOrSaveDb(sqlite3 *pInMemory, const char *zFilename, int isSave){
    int rc;                   /* Function return code */
    sqlite3 *pFile;           /* Database connection opened on zFilename */
    sqlite3_backup *pBackup;  /* Backup handle used to copy data */
    sqlite3 *pTo;             /* Database to copy to (pFile or pInMemory) */
    sqlite3 *pFrom;           /* Database to copy from (pFile or pInMemory) */

    /* Open the database file identified by zFilename. Exit early if this fails
    ** for any reason. */
    rc = sqlite3_open(zFilename, &pFile);
    if( rc!=SQLITE_OK ) goto loaddb_out;

    /* If this is a 'load' operation (isSave==0), then data is copied from the
    ** database file just opened to database pInMemory. Otherwise, if this is a
    ** 'save' operation (isSave==1), then data is copied from pInMemory to 

    ** pFile. Set the variables pFrom and pTo accordingly. */
    pFrom = (isSave ? pInMemory : pFile);
    pTo   = (isSave ? pFile     : pInMemory);

    /* Set up the backup procedure to copy from the "main" database of 
    ** connection pFile to the main database of connection pInMemory. If 
    ** something goes wrong, pBackup will be set to NULL and an error code and 
    ** message left in connection pTo.
    **
    ** If the backup object is successfully created, call backup_step() to copy
    ** data from pFile to pInMemory. Then call backup_finish() to release 
    ** resources associated with the pBackup handle. If an error occured, then 
    ** an error code and message will be left in connection pTo. If no error 

    ** occured, then the error code belonging to pTo is set to SQLITE_OK.
    */
    pBackup = sqlite3_backup_init(pTo, "main", pFrom, "main");
    if( pBackup ){
      (void)sqlite3_backup_step(pBackup, -1);
      (void)sqlite3_backup_finish(pBackup);
    }
    rc = sqlite3_errcode(pTo);


    /* Close the database connection opened on database file zFilename
    ** and return the result of this function. */
   loaddb_out:
    (void)sqlite3_close(pFile);
    return rc;
  }
}
</tcl>

<p>
  The C function to the right is an implementation of one of the simplest,
  and most common, uses of the backup API: loading and saving the contents
  of an in-memory database to a file on disk. The backup API is used as
  follows in this example:

  <ol>
    <li>Function [sqlite3_backup_init] is called to set up an sqlite3_backup
        handle to copy data between the two databases (either from a file and
        into the in-memory database, or vice-versa).
    <li>Function [sqlite3_backup_step] is called with a parameter of -1 to
        copy the entire source database to the destination.
    <li>Function [sqlite3_backup_finish] is called to clean up resources
        allocated by sqlite3_backup_init().
  </ol>

<p><b>Error handling</b>

<p>
  If an error occurs in any of the the three main functions involved in using 
  the backup API, [sqlite3_backup_init](), [sqlite3_backup_step]() or
  [sqlite3_backup_finish](), then the error code and message associated with
  the database connection passed as the first argument to the associated
  [sqlite3_backup_init]() call are set accordingly. Additionally, if
  [sqlite3_backup_step]() encounters an error, then the error code is returned
  by both the [sqlite3_backup_step]() call itself, and by the subsequent call
  to [sqlite3_backup_finish](). So a call to [sqlite3_backup_finish]()
  does not overwrite an error code stored in the destination database handle

  by [sqlite3_backup_step]().

<p>
  This is used in the example code to reduce amount of error handling required.
  The return values of the [sqlite3_backup_step]() and [sqlite3_backup_finish]()
  calls are ignored and the error code indicating the success or failure of
  the copy operation collected from the destination database handle afterward.


<p><b>Possible Enhancements</b>

<p>
  The implementation of this function could be enhanced in at least two ways:

  <ol>
    <li> Failing to obtain the lock on database file zFilename (an SQLITE_BUSY
         error) could be handled, and
    <li> Cases where the page-sizes of database pInMemory and zFilename are
         different could be handled better.
  </ol>

<p>
  Since database zFilename is a file on disk, then it may be accessed 
  externally by another process. This means that when the call to
  sqlite3_backup_step() attempts to read from or write data to it, it may
  fail to obtain the required file lock. If this happens, this implementation
  will fail, returning SQLITE_BUSY immediately. The solution would be to
  register a busy-handler callback or timeout with database connection pFile 
  using SQLite API function [sqlite3_busy_handler]() or [sqlite3_busy_timeout]()

  as soon as it is opened. If it fails to obtain a required lock immediately,
  [sqlite3_backup_step]() uses any registered busy-handler callback or timeout
  in the same way as [sqlite3_step]() or [sqlite3_exec]() does.

<p>
  Usually, it does not matter if the page-sizes of the source database and the
  destination database are different before the contents of the destination
  are overwritten. The page-size of the destination database is simply changed
  as part of the backup operation. The exception is if the destination database
  happens to be an in-memory database. In this case, if the page sizes
................................................................................
  ){
    int rc;                     /* Function return code */
    sqlite3 *pFile;             /* Database connection opened on zFilename */
    sqlite3_backup *pBackup;    /* Backup handle used to copy data */

    /* Open the database file identified by zFilename. */
    rc = sqlite3_open(zFilename, &pFile);
    if( rc!=SQLITE_OK ) goto backupdb_out;


    pBackup = sqlite3_backup_init(pFile, "main", pDb, "main");
    if( pBackup ){

      /* Each iteration of this loop copies 5 database pages from database pDb
      ** to the backup database. If the return value of backup_step() indicates
      ** that there are still further pages to copy, sleep for 250 ms before
      ** repeating. */
      do {
        rc = sqlite3_backup_step(pBackup, 5);
        xProgress(
            sqlite3_backup_remaining(pBackup),
            sqlite3_backup_pagecount(pBackup)
        );
        if( rc==SQLITE_OK || rc==SQLITE_BUSY || rc==SQLITE_LOCKED ){
          sqlite3_vfs *pVfs = sqlite3_vfs_find(0)
          pVfs->xSleep(pVfs, 250);
        }
      } while( rc==SQLITE_OK || rc==SQLITE_BUSY || rc==SQLITE_LOCKED );

      /* Release resources allocated by backup_init(). */
      (void)sqlite3_backup_finish(pBackup);
    }
    rc = sqlite3_errcode(pFile);


    /* Close the database connection opened on database file zFilename
    ** and return the result of this function. */
   backupdb_out:
    (void)sqlite3_close(pFile);
    return rc;
  }
}
</tcl>

<p>
  The function presented in the previous example copies the entire source
  database in one call to [sqlite3_backup_step](). This requires holding a 
  read-lock on the source database file for the duration of the operation,
  preventing any other database user from writing to the database. It also
  holds the mutex associated with database pInMemory throughout the copy,
  preventing any other thread from using it. The C function in this section,
  designed to be called by a background thread or process for creating a 
  backup of an online database, avoids these problems using the following 
  approach:

  <ol>
    <li>Function [sqlite3_backup_init] is called to set up an sqlite3_backup
        handle to copy data from database pDb to the backup database file 
        identified by zFilename.
    <li>Function [sqlite3_backup_step] is called with a parameter of 5 to
        copy 5 pages of database pDb to the backup database (file zFilename).
    <li>If there are still more pages to copy from database pDb, then the
        function sleeps for 250 ms and returns to step 2.

    <li>Function [sqlite3_backup_finish] is called to clean up resources
        allocated by sqlite3_backup_init().
  </ol>

<p><b>File and Database Connection Locking</b>

<p>
  During the 250 ms sleep in step 3 above, no read-lock is held on the database
  file and the mutex associated with pDb is not held. This allows other threads
  to use database handle pDb and other connections to write to the underlying
  database file. 

<p>
  If another thread writes to database connection pDb while this function is
  sleeping, then the backup database (database connection pFile) is
  automatically updated along with pDb. The backup process is continued after 
  the xSleep() call returns as if nothing had happened. If the database file 
  underlying connection pDb is written to by a different process or thread 
  using a different database connection while this function is sleeping, then 
  SQLite detects this within the next call made to sqlite3_backup_step(). 
  In this case the backup process is restarted from the beginning. Either way, 
  the backup database is kept up to date during the backup process so that 
  when the operation is complete the backup database contains a consistent 
  snapshot of the original. However:

  <ul>
    <li> Accounting for writes to the database by an external process or thread
         using a different database connection are significantly more expensive
         than accounting for writes made using pDb.
    <li> If the database is being written using a database handle other than 
         pDb sufficiently often while the backupDb() function is running, it
         may never finish.
  </ul>
 
<p><b>backup_remaining() and backup_pagecount()</b>

<p>
  The backupDb() function uses the sqlite3_backup_remaining() and
  sqlite3_backup_pagecount() functions to report its progress via the
................................................................................
  database is written to by another thread or process after the call to
  sqlite3_backup_step() returns but before the values returned by
  sqlite3_backup_remaining() and sqlite3_backup_pagecount() are used, the 
  values may be technically incorrect. This is not usually a problem.


<div style="clear:both"></div>

<title>SQLite Backup API</title>
<tcl>hd_keywords {Using the SQLite Online Backup API}</tcl>

<style>
  pre a:visited, pre a:link { text-decoration: none ; color: #40534b }
  pre {
    background: #F3F3F3;
    float: right;
    padding: 1ex 2ex;
................................................................................

  <li> If a power failure or operating system failure occurs while copying
       the database file the backup database may be corrupted following
       system recovery.
</ul>

<p>
  The [sqlite3_backup_init | Online Backup API] was created to 
  address these concerns. The online backup API allows the contents of
  one database to be copied into another database, overwriting the 
  original contents of the target database. The copy operation may be 
  done incrementally, in which case the source database does not need
  to be locked for the duration of the copy, only for the brief periods
  of time when it is actually being read from. This allows other database
  users to continue uninterrupted while a backup of an online database is
  made.

<p>
  The online backup API is [sqlite3_backup_init | documented here].
  The remainder of this page contains two C language examples illustrating 
  common uses of the API and discussions thereof. Reading these examples
  is no substitute for reading the API documentation!

<h2>Example 1: Loading and Saving In-Memory Databases</h2>

<tcl>
................................................................................
  **
  ** If the operation is successful, SQLITE_OK is returned. Otherwise, if
  ** an error occurs, an SQLite error code is returned.
  */
  int loadOrSaveDb(sqlite3 *pInMemory, const char *zFilename, int isSave){
    int rc;                   /* Function return code */
    sqlite3 *pFile;           /* Database connection opened on zFilename */
    sqlite3_backup *pBackup;  /* Backup object used to copy data */
    sqlite3 *pTo;             /* Database to copy to (pFile or pInMemory) */
    sqlite3 *pFrom;           /* Database to copy from (pFile or pInMemory) */

    /* Open the database file identified by zFilename. Exit early if this fails
    ** for any reason. */
    rc = sqlite3_open(zFilename, &pFile);
    if( rc==SQLITE_OK ){
  
      /* If this is a 'load' operation (isSave==0), then data is copied
      ** from the database file just opened to database pInMemory. 
      ** Otherwise, if this is a 'save' operation (isSave==1), then data
      ** is copied from pInMemory to pFile.  Set the variables pFrom and
      ** pTo accordingly. */
      pFrom = (isSave ? pInMemory : pFile);
      pTo   = (isSave ? pFile     : pInMemory);
  
      /* Set up the backup procedure to copy from the "main" database of 
      ** connection pFile to the main database of connection pInMemory.
      ** If something goes wrong, pBackup will be set to NULL and an error
      ** code and  message left in connection pTo.
      **
      ** If the backup object is successfully created, call backup_step()
      ** to copy data from pFile to pInMemory. Then call backup_finish()
      ** to release resources associated with the pBackup object.  If an
      ** error occured, then  an error code and message will be left in
      ** connection pTo. If no error occured, then the error code belonging
      ** to pTo is set to SQLITE_OK.
      */
      pBackup = sqlite3_backup_init(pTo, "main", pFrom, "main");
      if( pBackup ){
        (void)sqlite3_backup_step(pBackup, -1);
        (void)sqlite3_backup_finish(pBackup);
      }
      rc = sqlite3_errcode(pTo);
    }

    /* Close the database connection opened on database file zFilename
    ** and return the result of this function. */
   loaddb_out:
    (void)sqlite3_close(pFile);
    return rc;
  }
}
</tcl>

<p>
  The C function to the right demonstrates of one of the simplest,
  and most common, uses of the backup API: loading and saving the contents
  of an in-memory database to a file on disk. The backup API is used as
  follows in this example:

  <ol>
    <li>Function [sqlite3_backup_init()] is called to create an [sqlite3_backup]
        object to copy data between the two databases (either from a file and
        into the in-memory database, or vice-versa).
    <li>Function [sqlite3_backup_step()] is called with a parameter of 
        <tt>-1</tt> to copy the entire source database to the destination.
    <li>Function [sqlite3_backup_finish()] is called to clean up resources
        allocated by [sqlite3_backup_init()].
  </ol>

<p><b>Error handling</b>

<p>
  If an error occurs in any of the the three main backup API routines
  then the [error code] and [sqlite3_errmsg | message] are attached to
  the destination [database connection].

  Additionally, if
  [sqlite3_backup_step()] encounters an error, then the [error code] is returned
  by both the [sqlite3_backup_step()] call itself, and by the subsequent call
  to [sqlite3_backup_finish()]. So a call to [sqlite3_backup_finish()]

  does not overwrite an [error code] stored in the destination
  [database connection] by [sqlite3_backup_step()].  This feature


  is used in the example code to reduce amount of error handling required.
  The return values of the [sqlite3_backup_step()] and [sqlite3_backup_finish()]
  calls are ignored and the error code indicating the success or failure of
  the copy operation collected from the destination [database connection]
  afterward.

<p><b>Possible Enhancements</b>

<p>
  The implementation of this function could be enhanced in at least two ways:

  <ol>
    <li> Failing to obtain the lock on database file zFilename (an [SQLITE_BUSY]
         error) could be handled, and
    <li> Cases where the page-sizes of database pInMemory and zFilename are
         different could be handled better.
  </ol>

<p>
  Since database zFilename is a file on disk, then it may be accessed 
  externally by another process. This means that when the call to
  sqlite3_backup_step() attempts to read from or write data to it, it may
  fail to obtain the required file lock. If this happens, this implementation
  will fail, returning SQLITE_BUSY immediately. The solution would be to
  register a busy-handler callback or 
  timeout with [database connection] pFile 
  using [sqlite3_busy_handler()] or [sqlite3_busy_timeout()]
  as soon as it is opened. If it fails to obtain a required lock immediately,
  [sqlite3_backup_step()] uses any registered busy-handler callback or timeout
  in the same way as [sqlite3_step()] or [sqlite3_exec()] does.

<p>
  Usually, it does not matter if the page-sizes of the source database and the
  destination database are different before the contents of the destination
  are overwritten. The page-size of the destination database is simply changed
  as part of the backup operation. The exception is if the destination database
  happens to be an in-memory database. In this case, if the page sizes
................................................................................
  ){
    int rc;                     /* Function return code */
    sqlite3 *pFile;             /* Database connection opened on zFilename */
    sqlite3_backup *pBackup;    /* Backup handle used to copy data */

    /* Open the database file identified by zFilename. */
    rc = sqlite3_open(zFilename, &pFile);
    if( rc==SQLITE_OK ){

      /* Open the sqlite3_backup object used to accomplish the transfer */
      pBackup = sqlite3_backup_init(pFile, "main", pDb, "main");
      if( pBackup ){
  
        /* Each iteration of this loop copies 5 database pages from database
        ** pDb to the backup database. If the return value of backup_step()
        ** indicates that there are still further pages to copy, sleep for
        ** 250 ms before repeating. */
        do {
          rc = sqlite3_backup_step(pBackup, 5);
          xProgress(
              sqlite3_backup_remaining(pBackup),
              sqlite3_backup_pagecount(pBackup)
          );
          if( rc==SQLITE_OK || rc==SQLITE_BUSY || rc==SQLITE_LOCKED ){

            sqlite3_sleep(250);
          }
        } while( rc==SQLITE_OK || rc==SQLITE_BUSY || rc==SQLITE_LOCKED );
  
        /* Release resources allocated by backup_init(). */
        (void)sqlite3_backup_finish(pBackup);
      }
      rc = sqlite3_errcode(pFile);
    }
    
    /* Close the database connection opened on database file zFilename
    ** and return the result of this function. */

    (void)sqlite3_close(pFile);
    return rc;
  }
}
</tcl>

<p>
  The function presented in the previous example copies the entire source
  database in one call to [sqlite3_backup_step()]. This requires holding a 
  read-lock on the source database file for the duration of the operation,
  preventing any other database user from writing to the database. It also
  holds the mutex associated with database pInMemory throughout the copy,
  preventing any other thread from using it. The C function in this section,
  designed to be called by a background thread or process for creating a 
  backup of an online database, avoids these problems using the following 
  approach:

  <ol>
    <li>Function [sqlite3_backup_init()] is called to create an [sqlite3_backup]
        object to copy data from database pDb to the backup database file 
        identified by zFilename.
    <li>Function [sqlite3_backup_step()] is called with a parameter of 5 to
        copy 5 pages of database pDb to the backup database (file zFilename).
    <li>If there are still more pages to copy from database pDb, then the
        function sleeps for 250 milliseconds (using the [sqlite3_sleep()]
        utility) and then returns to step 2.
    <li>Function [sqlite3_backup_finish()] is called to clean up resources
        allocated by [sqlite3_backup_init()].
  </ol>

<p><b>File and Database Connection Locking</b>

<p>
  During the 250 ms sleep in step 3 above, no read-lock is held on the database
  file and the mutex associated with pDb is not held. This allows other threads
  to use [database connection] pDb and other connections to write to the
  underlying database file. 

<p>
  If another thread writes to database connection pDb while this function is
  sleeping, then the backup database (database connection pFile) is
  automatically updated along with pDb. The backup process is continued after 
  the xSleep() call returns as if nothing had happened. If the database file 
  underlying connection pDb is written to by a different process or thread 
  using a different database connection while this function is sleeping, then 
  SQLite detects this within the next call made to sqlite3_backup_step() and 
  restarts the backup from the beginning. Either way, 
  the backup database is kept up to date during the backup process so that 
  when the operation is complete the backup database contains a consistent 
  snapshot of the original. However:

  <ul>
    <li> Accounting for writes to the database by an external process or thread
         using a different database connection are significantly more expensive
         than accounting for writes made using pDb.
    <li> If the database is being written using a database connection other
         than pDb sufficiently often while the backupDb() function is
         running, it may never finish.
  </ul>
 
<p><b>backup_remaining() and backup_pagecount()</b>

<p>
  The backupDb() function uses the sqlite3_backup_remaining() and
  sqlite3_backup_pagecount() functions to report its progress via the
................................................................................
  database is written to by another thread or process after the call to
  sqlite3_backup_step() returns but before the values returned by
  sqlite3_backup_remaining() and sqlite3_backup_pagecount() are used, the 
  values may be technically incorrect. This is not usually a problem.


<div style="clear:both"></div>

    hd_enable_main 1
  }
}

chng {2009 Feb 18 (3.6.11)} {
<li>Added the [sqlite3_backup_init | hot-backup interface].
<li>Added new commands ".backup" and ".restore" to the [CLI].
<li>Added new methods <a href="tclsqlite.html#backup">backup</a>
    <a href="tclsqlite.html#restore">restore</a> to the TCL interface.
<li>Various minor bug fixes
}

chng {2009 Jan 15 (3.6.10)} {
<li>Fix a cache coherency problem that could lead to database corruption.
    [Ticket #3584].

    hd_enable_main 1
  }
}

chng {2009 Feb 18 (3.6.11)} {
<li>Added the [sqlite3_backup_init | hot-backup interface].
<li>Added new commands ".backup" and ".restore" to the [CLI].
<li>Added new methods <a href="tclsqlite.html#backup">backup</a> and
    <a href="tclsqlite.html#restore">restore</a> to the TCL interface.
<li>Various minor bug fixes
}

chng {2009 Jan 15 (3.6.10)} {
<li>Fix a cache coherency problem that could lead to database corruption.
    [Ticket #3584].

  SQLite.  
}
doc {Version 3 C/C++ API<br>Reference} {c3ref/intro.html} {
  This document describes each API function separately.
}
doc {Frequently Asked Questions} {faq.html} {
  The title of the document says all...






}
doc {Sharing Cache Mode} {sharedcache.html} {
  Version 3.3.0 and later supports the ability for two or more
  database connections to share the same page and schema cache.
  This feature is useful for certain specialized applications.
}
doc {Tcl API} {tclsqlite.html} {

  SQLite.  
}
doc {Version 3 C/C++ API<br>Reference} {c3ref/intro.html} {
  This document describes each API function separately.
}
doc {Frequently Asked Questions} {faq.html} {
  The title of the document says all...
}
doc {Using The Online Backup Interface} {backup.html} {
  The [sqlite3_backup_init | online-backup interface] can be used to
  copy content from a disk file into an in-memory database or vice
  versa and it can make a hot backup of a live database.  This application
  note gives examples of how.
}
doc {Sharing Cache Mode} {sharedcache.html} {
  Version 3.3.0 and later supports the ability for two or more
  database connections to share the same page and schema cache.
  This feature is useful for certain specialized applications.
}
doc {Tcl API} {tclsqlite.html} {

      {<a href="http://www.sqlite.org/cvstrac/tktview?tn=\1">\0</a>} txt
  hd_resolve "<p>$txt</p>"
  hd_puts "<hr width=\"50%\">"
}

newsitem {2009-Feb-18} {Version 3.6.11} {
  SQLite [version 3.6.11] adds support for the
  [sqlite3_backup_init | hot-backup] interface.  This interface can be
  used to create a backup copy of an SQLite database while it is in use.
  The same interface can be used to initialize an in-memory database from
  a persistent disk image or to save an in-memory database into a
  persistent disk image.

}

newsitem {2009-Jan-15} {Version 3.6.10} {
  SQLite [version 3.6.10] fixes a cache coherency bug (Ticket #3584)
  introduced by check-in
  <a href="http://www.sqlite.org/cvstrac/chngview?cn=5864">[5864]</a>
  which was part of [version 3.6.5].  This bug might lead to database

      {<a href="http://www.sqlite.org/cvstrac/tktview?tn=\1">\0</a>} txt
  hd_resolve "<p>$txt</p>"
  hd_puts "<hr width=\"50%\">"
}

newsitem {2009-Feb-18} {Version 3.6.11} {
  SQLite [version 3.6.11] adds support for the
  [sqlite3_backup_init | hot-backup] interface. This interface can be
  used to create a backup copy of an SQLite database while it is in use.
  The same interface can be used to initialize an in-memory database from
  a persistent disk image or to save an in-memory database into a
  persistent disk image.  Usage examples can be found at
  [Using the SQLite Online Backup API].
}

newsitem {2009-Jan-15} {Version 3.6.10} {
  SQLite [version 3.6.10] fixes a cache coherency bug (Ticket #3584)
  introduced by check-in
  <a href="http://www.sqlite.org/cvstrac/chngview?cn=5864">[5864]</a>
  which was part of [version 3.6.5].  This bug might lead to database