Documentation Source Text

    append ret " (C: $::ffreq_children($reqid))"
  } 
  append ret "</p>"
}

set ::Requirements [list]

proc CodeBlock {txt} {
  set txt [string trim $txt "\n"]
  set    out {<div class=codeblock style="margin:0 15ex">}
  append out {<table width=100% style="border:1px solid #80a796;padding:0 1ex;background-color:#EEEEEE"><tr><td><pre style="font-size:1.1em">}

  foreach line [split $txt "\n"] {
    if {![string is space $line]} {
      set nSpace [expr {

    /title { 
      set G(inTitle) 0
      append G(doc) [formattag $tag $details]
    }

    codeblock  { set G(inCodeblock) 1 }
    /codeblock { 
      append G(doc) [CodeBlock $G(codeblock)]
      set G(codeblock) "" 
      set G(inCodeblock) 0 
    }

    table {
      catch {array unset D} 
      array set D $details

<title>The RBU Extension</title>
<tcl>
hd_keywords {RBU} {RBU extension}
proc CODE {text} {
  hd_puts "<blockquote><pre>"
  hd_puts $text
  hd_puts "</pre></blockquote>"
  #hd_puts [CodeBlock $text]
}
</tcl>
<!table_of_contents>
<h1 align='center'>The RBU Extension</h1>

<p>The RBU extension is an add-on for SQLite designed for use with large 

SQLite database files on low-power devices at the edge of a network. RBU
may be used for two separate tasks:

<ul>
  <li> <b>RBU Update operations</b>. An [RBU Update] is a bulk update of a
          database file that may include many insert, update and delete
          operations on one or more tables.
  <li> <b>RBU Vacuum operations</b>. An [RBU Vacuum] optimizes and rebuilds an
          entire database file, with results similar to SQLite's native VACUUM
          command.  
</ul>

<p>The acronym RBU stands for "Resumable Bulk Update".


<p>Both of the RBU functions may be accomplished using SQLite's built-in 
SQL commands - RBU update via a series of [INSERT], [DELETE] and 
[UPDATE] commands within a single transaction, and RBU vacuum by a single
[VACUUM] command.  The RBU module provides the following advantages over
these simpler approaches:


<ol>
<li><b>RBU may be more efficient</b>

<p>The most efficient way to apply changes to a B-Tree (the data structure
that SQLite uses to store each table and index on disk) is to make the
changes in key order. But if an SQL table has one or more indexes, the key
order for each index may be different from the main table and the other
auxiliary indexes. As a result, when executing a series of [INSERT],
[UPDATE] and [DELETE] statements it is not generally possible to order the
operations so that all b-trees are updated in key order. The RBU update
process works around this by applying all changes to the main table in one 

pass, then applying changes to each index in separate passes, ensuring each
B-Tree is updated optimally. For a large database file (one that does not
fit in the OS disk cache) this procedure can result in two orders of
magnitude faster updates.

<p>An RBU Vacuum operation requires less temporary disk space and writes
less data to disk than an SQLite VACUUM. An SQLite VACUUM requires roughly
twice the size of the final database file in temporary disk space to run.
The total amount of data written is around three times the size of the
final database file. By contrast, an RBU Vacuum requires roughly the size
of the final database file in temporary disk space and writes a total of
twice that to disk.

<p>On the other hand, an RBU Vacuum uses more CPU than a regular SQLite
VACUUM - in one test as much as five times as much. For this reason, an RBU
Vaccum is often signicantly slower than an SQLite VACUUM under the same
conditions.

<li><b>RBU runs in the background</b>


<p>An ongoing RBU operation (either an update or a vacuum) does not
interfere with read access to the database file.


<li><b>RBU runs incrementally</b>

<p>RBU operations may be suspended and then later resumed, perhaps with
intervening power outages and/or system resets. For an RBU update, the
original database content remains visible to all database readers until 
the entire update has been applied - even if the update is suspended and
then later resumed.

</ol>

<p>The RBU extension is not enabled by default. To enable it, compile the
[amalgamation] with the [SQLITE_ENABLE_RBU] compile-time option.

<tcl>hd_fragment rbu_updates {RBU Update}</tcl>
<h1>RBU Updates</h1>
<h2>RBU Update Limitations</h2>

<p>The following limitations apply to RBU updates:

<ul>
<li><p>The changes must consist of [INSERT], [UPDATE], and [DELETE]
    operations only.  CREATE and DROP operations are not
    supported.</p></li>
<li><p>[INSERT] statements may not use default values.</p></li>
<li><p>[UPDATE] and [DELETE] statements must identify the target rows
    by rowid or by non-NULL PRIMARY KEY values.</p></li>
<li><p>[UPDATE] statements may not modify PRIMARY KEY or rowid values.
    </p></li>
<li><p>RBU updates cannot be applied to any tables that contain a column
       named "rbu_control".</p></li>
<li><p>The RBU update will not fire any triggers.</p></li>
<li><p>The RBU update will not detect or prevent foreign key or
       CHECK constraint violations.</p></li>
<li><p>All RBU updates use the "OR ROLLBACK" constraint handling mechanism.
    </p></li>
<li><p>The target database may not be in [WAL mode].</p></li>
<li><p>The target database may not contain [indexes on expressions].
<li><p>No other writes may occur on the target database while the
       RBU update is being applied.  A read-lock is held on the target
       database to prevent this.</p></li>
</ul>


<h2>Preparing An RBU Update File</h2>

  <a href=http://fossil-scm.org>Fossil source-code management system</a>, or

  <li> In a custom format defined by the RBU application.
</ul>

<p> The fossil delta format may only be used to update BLOB values. Instead
of storing the new BLOB within the data_% table, the fossil delta is stored
instead. And instead of specifying an 'x' as part of the rbu_control string
for the column to be updated, an 'f' character is stored. When processing
an 'f' update, RBU loads the original BLOB data from disk, applies the fossil
delta to it and stores the results back into the database file. The RBU
databases generated by [sqldiff --rbu] make use of fossil deltas wherever
doing so would save space in the RBU database.

<p> To use a custom delta format, the RBU application must register a

  the data0_ccc_fts view and applied to FTS table ccc_fts. Because
  "data0_ccc_fts" is smaller than "data_ccc", the FTS table will be updated
  first, as required.

<p>
  Cases in which the underlying content table has an explicit INTEGER PRIMARY
  KEY column are slightly more difficult, as the text values stored in the
  rbu_control column are slightly different for the FTS index and its
  underlying content table. For the underlying content table, a character
  must be included in any rbu_control text values for the explicit IPK, but
  for the FTS table itself, which has an implicit rowid, it should not. This
  is inconvenient, but can be solved using a more complicated view, as follows:

<tcl>CODE {
  -- Target database schema
  CREATE TABLE ddd(i INTEGER PRIMARY KEY, k TEXT);
  CREATE VIRTUAL TABLE ddd_fts USING fts4(k, content=ddd);

  [FTS shadow tables | underlying database tables] that they create in order to
  store data within the database are not, and [sqldiff] will include add these
  to any RBU database. For this reason, users attempting to use sqldiff to
  create RBU updates to apply to target databases with one or more virtual
  tables will likely have to run sqldiff using the --table option separately
  for each table to update in the target database.


<h2>RBU Update C/C++ Programming</h2>



<p>The RBU extension interface allows an application to apply an RBU update 
stored in an RBU database to an existing target database.
The procedure is as follows:

<ol>
<li><p>
Open an RBU handle using the sqlite3rbu_open(T,A,S) function.

<p>The T argument is the name of the target database file.
The A argument is the name of the RBU database file.

This allows subsequent processes to automatically
resume the RBU update from where it left off.
If state information is stored in the RBU database, it can be removed
by dropping all tables whose names begin with "rbu_".

<p>For more details, refer to the comments in 
<a href=http://sqlite.org/src/doc/trunk/ext/rbu/sqlite3rbu.h>header file
sqlite3rbu.h</a>.

<tcl>hd_fragment rbu_vacuum {RBU Vacuum}</tcl>
<h1>RBU Vacuum</h1>

<h2>RBU Vacuum Limitations</h2>

<p>When compared with SQLite's built-in VACUUM command, RBU Vacuum has the
following limitations:

<ul>
  <li><p>It may not be used on a database that contains [indexes on expressions].
  <li><p>The database being vacuumed may not be in [WAL mode].
</ul>

<h2>RBU Vacuum C/C++ Programming</h2>

<p> This section provides an overview of and example code demonstrating the
    integration of RBU Vacuum into an application program.  For full details,
    refer to the comments in 
    <a href=http://sqlite.org/src/doc/trunk/ext/rbu/sqlite3rbu.h>header file
    sqlite3rbu.h</a>.

<p> RBU Vacuum applications all implement some variation of the following
procedure:

<ol>
  <li><p> An RBU handle is created by calling sqlite3rbu_vacuum(T, S).

      <p> Argument T is the name of the database file to vacuum. Argument S is
      the name of a database in which the RBU module will save its state if the
      vacuum operation is suspended.

      <p> If state database S does not exist when sqlite3rbu_vacuum() is
      invoked, it is automatically created and populated with the single table
      used to store the state of an RBU vacuum - "rbu_state". If an ongoing RBU
      vacuum is suspended, this table is populated with state data. The next
      time sqlite3rbu_vacuum() is called with the same S parameter, it detects
      this data and attempts to resume the suspended vacuum operation. When
      an RBU vacuum operation is completed or encounters an error, RBU 
      automatically deletes the contents of the rbu_state table. In this case,
      the next call to sqlite3rbu_vacuum() starts an entirely new vacuum
      operation from scratch.

      <p> It is a good idea to establish a convention for determining the RBU
      vacuum state database name based on the target database name. The
      example code below uses "&lt;target&gt;-vacuum", where &lt;target&gt; is
      the name of the database being vacuumed.

  <li><p> Any custom collation sequences used by indexes within the database
      being vacuumed are registered with both of the database handles returned
      by the sqlite3rbu_db() function.

  <li><p> Function sqlite3rbu_step() is called on the RBU handle until either
      the RBU vacuum is finished, an error occurs or the application wishes to
      suspend the RBU vacuum.

      <p> Each call to sqlite3rbu_step() does a small amount of work towards
      completing the vacuum operation. Depending on the size of the database, a
      single vacuum may require thousands of calls to sqlite3rbu_step().
      sqlite3rbu_step() returns SQLITE_DONE if the vacuum operation has
      finished, SQLITE_OK if the vacuum operation has not finished but no error
      has occurred, and an SQLite error code if an error is encountered. If
      an error does occur, all subsequent calls to sqlite3rbu_step() immediately
      return the same error code.

  <li><p> Finally, sqlite3rbu_close() is called to close the RBU handle. If the
      application stopped calling sqlite3rbu_step() before either the vacuum
      finished or an error occurred, the state of the vacuum is saved in the
      state database so that it may be resumed later on.

      <p> Like sqlite3rbu_step(), if the vacuum operation has finished,
      sqlite3rbu_close() returns SQLITE_DONE. If the vacuum has not finished
      but no error has occurred, SQLITE_OK is returned. Or, if an error has
      occurred, an SQLite error code is returned. If an error occurred as part
      of a prior call to sqlite3rbu_step(), sqlite3rbu_close() returns the
      same error code.
</ol>

<p>The following example code illustrates the techniques described above.  

<tcl>CODE {

<i>/*</i>
<i>** Either start a new RBU vacuum or resume a suspended RBU vacuum on </i>
<i>** database zTarget. Return when either an error occurs, the RBU </i>
<i>** vacuum is finished or when the application signals an interrupt</i>
<i>** (code not shown).</i>
<i>**</i>
<i>** If the RBU vacuum is completed successfully, return SQLITE_DONE.</i>
<i>** If an error occurs, return SQLite error code. Or, if the application</i>
<i>** signals an interrupt, suspend the RBU vacuum operation so that it</i>
<i>** may be resumed by a subsequent call to this function and return</i>
<i>** SQLITE_OK.</i>
<i>**</i>
<i>** This function uses the database named "&lt;zTarget&gt;-vacuum" for</i>
<i>** the state database, where &lt;zTarget&gt; is the name of the database </i>
<i>** being vacuumed.</i>
<i>*/</i>
int do_rbu_vacuum(const char *zTarget){
  int rc;
  char *zState;                   <i>/* Name of state database */</i>
  sqlite3rbu *pRbu;               <i>/* RBU vacuum handle */</i>

  zState = sqlite3_mprintf("%s-vacuum", zTarget);
  if( zState==0 ) return SQLITE_NOMEM;
  pRbu = sqlite3rbu_vacuum(zTarget, zState);
  sqlite3_free(zState);

  if( pRbu ){
    sqlite3 *dbTarget = sqlite3rbu_db(pRbu, 0);
    sqlite3 *dbState = sqlite3rbu_db(pRbu, 1);

    <i>/* Any custom collation sequences used by the target database must</i>
    <i>** be registered with both database handles here.  */</i>

    while( sqlite3rbu_step(pRbu)==SQLITE_OK ){
      if( <i>&lt;application has signalled interrupt&gt;</i> ) break;
    }
  }
  rc = sqlite3rbu_close(pRbu);
  return rc;
}

}</tcl>