SQLite Test Harness #3

1.0 Overview

SQLite Test Harness #3 (hereafter "TH3") is a test harness for SQLite designed to meet the following objectives:

TH3 is one of three test harnesses used by the SQLite project. TH3 is designed for validation testing. It is not designed for use during development and debugging. The original TCL-based test suite for SQLite will continue to serve as a platform for development and debug testing. The reason that TH3 exists is that it meets other objectives that are difficult to achieve with the TCL-based tests.

2.0 Operation

TH3 is a test program generator. The output of TH3 is a program written in ANSI-C and intended to be linked against the SQLite library under test. The generated test program is compiled and run on the target platform in order to verify correct operation of SQLite on that platform.

The inputs to TH3 are test modules written in C or SQL and small configuration files that determine how to initialize SQLite. It is intended that the TH3 library eventually will include hundreds or perhaps thousands of test modules and dozens of configuration files. Each time TH3 is run, it reads a subset of the available test modules and configuration files to generate a custom C program the performs all of the specified tests under all configurations. A complete test of SQLite normally involves running TH3 multiple times to generate multiple test programs covering different aspects of SQLite's operation, then linking all test programs against a common SQLite library and running them separately on the target platform. SQLite will be found to work if all test programs pass.

There are no arbitrary limits in TH3. One could generate a single test program that contained all test modules and configuration files. However, such a test program might be too large to deploy on embedded platforms. Hence, TH3 provides the ability to break the library of test modules up into smaller, more easily digested pieces.

Each individual test module might contain dozens, hundreds, or thousands of separate tests. The test modules can be written in C or as scripts of SQL. The test modules written in C are not as cumbersome to write as one might suppose. The TH3 system provides high-level interfaces that simplify test writing. A typical test case in TH3 might contains slightly more syntax that the corresponding TCL-script test, but the difference is not that great. The test modules written as SQL contain special comments that define the boundaries and operation of each test case and the expected results.

Each test module file contains a header which describes the circumstances under which the test is valid. For a particular configuration, only those modules that are compatible with the configuration are run.

2.1 Generating A Test Program

The TH3 program generator is a TCL script named "mkth3.tcl". To generate a test program, one has merely to run this script and supply the names of files containing test modules and test configurations on the command line. TH3 does not force any specific names for test modules or configuration files, but it is customary to use suffixes ".test" and ".cfg". With these conventions, one might generate a test program as follows:

tclsh mkth3.tcl *.test *.cfg >testprog1.c

The output from the mkth3.tcl script is a C program that contains everything needed to run the tests - everything that is except for the SQLite library itself. The generated test program contains implementations for all of the support interfaces used by the test modules and it contains the main() routine that drives the tests. To convert the test program into a working executable, simply compile it against SQLite:

cc -o testprog1 testprog1.c sqlite3.c

The compilation step shown immediately above is merely representative. In a working installation, one would normally want to specify optimization parameters and compile-time switches on the compiler command line.

Once the test program is generated, it is run with no arguments to perform the tests. Progress information as well as error diagnostics appear on standard output. The program returns zero if there are no errors and non-zero if any problems were detected.

2.2 Operating System Interface

The C program generated by TH3 makes minimal demands on the target operating system and standard C library. However, the generated code might still request services that the target environment does not provide. This can be worked around by adding a small interface module and compiling the test program with special compile-time switches. the details on how to do this are described separately

3.0 Test Configurations And Modules

The inputs to TH3 are test configurations and test modules. The TH3 source tree contains approximately 20 prepackaged test configurations and over 600 test modules (as of 2009-09-16 - both numbers continue to grow.)

Each test configuration determines how SQLite set up for a sequence of tests. For example, a test configuration will specify the default text encoding for the database, the default page size, which VFS to use, etc. Test modules exercise some subset of features of SQLite. For each test configuration, the TH3-generated test program runs all applicable modules. Hence, if there are N test configurations and M test modules, then the test program will typically run on the order of N*M separate modules.

However, not all test modules are compatible with every test configuration. For example, many test modules require multiple database connections open on the same database, which is not possible if the test configuration specifies that all databases are to be in-memory databases. (In-memory databases can only be open by a single connection and are automatically deleted when the connection closes.) The TH3-generated test program automatically prunes test modules that are not compatible with a particular test configuration.

3.1 Custom configurations and modules

Licensees of TH3 can write their own test configurations, if desired, that set up TH3 to test SQLite in exactly the same way that they intend to use SQLite. Test configurations consists of a couple dozen lines of plain text and are easy to write based on a template.

Licensees are also free to write their own test modules, though it is less clear why they would want to do this.