Documentation Source Text

Check-in [295c4e35bc]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:First draft of a new quality management plan. Still incomplete.
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256: 295c4e35bca0b90db008008c85ebe85a1bdcaf9e8df3c71b1ac2473117ee7025
User & Date: drh 2018-02-22 18:14:45
Context
2018-02-22
19:34
Remove the NAWK macros from the Makefiles, since they are no longer used. check-in: 76b1d7f1d4 user: drh tags: trunk
18:14
First draft of a new quality management plan. Still incomplete. check-in: 295c4e35bc user: drh tags: trunk
12:39
Merge the 3.22.0 updates into trunk. check-in: c0ef1f2bd8 user: drh tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Added pages/qmplan.in.









































































































































































































































































































































































































































































































































































































































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
<title>Quality Management</title>
<table_of_contents>

<center>
<font color="red" size="5"><b>
This document is a work-in-progress.  It is incomplete and unverified.
</b></font>
</center>

<h1>Overview</h1>

<p>
This is the Quality Management Plan for SQLite.

<p>
Quality management documents tend to expand into
binders full of incomprehensible jargon that nobody
reads.  This document strives to break that pattern by
being concise and useful.

<p>
The inspiration for this document is
[https://en.wikipedia.org/wiki/DO-178B|DO-178B].
Among quality standards, DO-178B seems to have the highest usefulness
to paperwork ratio.  Even so, the amount of documentation needed
for a full-up DO-178B implementation is vast.  SQLite strives to be
nimble and low-ceremony, and to that end, much of the required
DO-178B documentation is omitted.  We retain only those parts that
genuinely improve quality for a open-source software project such
as SQLite.

<p>
The purpose of this document is to brief the reader on how
SQLite development team functions on a daily basis, as they continuely
enhance the SQLite software and work to improve its already high reliability.
The document achieves its purpose if a competent developer can be
assimilated into the development team quickly after perusing this
document.

<h2>About This Document</h2>

<p>
The quality management plan was originally composed by going through
the description of outputs in section 11 of DO-178B (pages 48 through 56)
and writing down those elements that seemed relevant to SQLite.
The text will be subsequent revised to track enhancements to the
SQLite quality process.

<h1>Software Development Plan</h1>

<p>
This section is a coalescense of the Plan For Software Aspects Of
Certification and the Software Development Plan sections of DO-178B.


<p>
See <a href='about.html'>About SQLite</a> for an overview of the
SQLite software and what it does and how it is different.

<h2>Software Life Cycle</h2>

<p>
SQLite uses a continuous integration process.  The software
is under constant enhancement and refinement.  The latest trunk
check-ins are frequently used internally for mission-critical
operations. 

<p>
There is no pre-defined release cycle.  Releases occur
when there is a critical mass of feature enhancements and/or
bug fixes.  Historically, releases have occurred about 5 or 6
times per year.
Users of SQLite pick up new releases from the website on an
as-needed basis.

<h3>Maintenance Releases</h3>

<p>
Routine maintenance releases of SQLite contain feature enhancements,
performance enhancements, and/or fixes for non-critical issues.
The version number for major releases are of the form "3.N.0"
for some integer N.  See the [version numbering conventions] document
for details.

<p>
Upcoming maintenance releases announced on the sqlite-users and
sqlite-dev [mailing lists] about two weeks prior to the anticipated
release.  Approximately one week prior to release, the lead developer
declares "pencils down" after which only bug-fix check-ins are
allowed on trunk.  A new [https://sqlite.org/checklists|release checklist]
is created and updated as needed.  As items of the checklist are 
verified, they are checked off and turn green.  The release occurs
when all elements of the checklist are green.  That process normally
takes about a week.

<h3>Patch Releases</h3>

<p>
Occasionally, a serious problem is found and a small "patch" release
must be made against a regular maintenance release.  Patch are distinct
from maintenance releases in that the number of lines of code changed
from the previous release is very small.  Every effort is made to avoid
patch releases by making sure that maintenance releases are bug free.

<p>
Patch releases may or may not have a release checklist, depending on the
issue.  This is a judgement call by the project leader.

<h2>Release History</h2>

<p>The documentation system automatically maintains a
[chronology] of past releases, as well as a
[complete list of SQLite releases] with change summaries.

<h2>Schedule</h2>

SQLite has a long-range vision.
Planning is done with the assumption that SQLite
will be used and supported through at least the year 2050.
All code is written with the idea that it will one day be read and
maintained by people not yet born.  The code is carefully commented
with an eye toward helping those future developers more easily 
understand the logic and the rationale behind the code.

<h1>Software Development Environment</h1>

<p>
SQLite is written in portable C code.
Development work occurs on a mix of Linux, Mac, and Windows workstations.
The developers use command-line tools and eschew integrated development
environments (IDEs) whenever possible.  All developers are expected to be
fluent with the unix command-line.

<p>
A minimum setup for compiling and testing SQLite from canonical
sources is as follows:

<ul>
<li> A host computer with a 32-bit or 64-bit address space.
     The OS can be Linux, Mac, Windows, *BSD, Solaris, or some other.
<li> A C99 compiler such as GCC (including MinGW variants for Windows),
     Clang, or MSVC
<li> A text editor of the user's choice supporting UTF-8 text.
<li> [https://core.tcl.tk/|Tcl] version 8.6 or later.
<li> The "make" utility, or optionally "nmake" on Windows.
</ul>

<p>
The Tcl script language is used to help translate canonical source code
into the [amalgamation] and to manage testing.  Tcl is not used directly
by SQLite itself (unless requested by a compile-time option).  End users
of the SQLite amalgamation sources do not need Tcl.

<p>
When building the [CLI], it is helpful, but not required, to have
the following third-party libraries on hand:

<ul>
<li> [https://zlib.net/|zLib]
<li> [http://git.savannah.gnu.org/cgit/readline.git?h=devel|readline]
     or [http://thrysoee.dk/editline/|editline]
     or [https://github.com/antirez/linenoise|linenoise] for
     command-line editing.
</ul>

<p>
A complete release-test of SQLite requires additional software,

<ul>
<li> [http://www.valgrind.org/|valgrind]
<li> [https://gcc.gnu.org/onlinedocs/gcc/Gcov.html|gcov]
</ul>

<p>
SQLite is expected to operate the same, and use exactly the same 
[file format|on-disk format],
on all modern operating systems, on all modern computer architectures,
and using all modern C compilers.  The developers are constantly testing
SQLite on as many diverse platforms as they can get their hands on.

<h1>Software Verification Plan</h1>

<p>The testing process for SQLite is described in the [testing] document.
Testing objectives include:

<ul>
<li> 100% MC/DC in an as-delivered configuration
<li> Testing of both source code and object code
<li> Testing on multiple platforms and with multiple compilers
<li> Code change inspection
<li> Dynamic and static analysis of the code
</ul>

<p>The testing process is controlled by the
[release testing checklists].  The checklists succinctly summary
all steps necessary to fully validate SQLite, and they record when
and by whom each validation step was performed.

<p>The set of checklist items for release checklist is potentially
updated for each release.  The content and complete
history of each release checklist are retained for the historical
record.

<h1>Software Configuration Management</h1>

<h2>Version Control</h2>

<p>
SQLite source code is managed using the [https://fossil-scm.org|Fossil]
version control system.  Fossil was written specifically to support
SQLite development.  Fossil provides both distributed version control
and issue tracking.

<h2>Survivability</h2>

<p>
All code is archived on three separate machines:
[https://www.sqlite.org], [https://www2.sqlite.org], [https://www3.sqlite.org].
These machines are located in different cities (Dallas, Newark, and
San Francisco, respectively) and managed by two different hosting
companies ([https://linode.com|Linode] for the first two and
[https://digitalocean.com|Digital Ocean] for the third).
This diversity is intended to avoid a single point of failure.

<p>
The main machine in Dallas [https://www.sqlite.org/] is the primary
server and the one that most people use.  The other two are considered
backups.

<p>
In addition to the official repositories, the developers typically
keep complete clones of all software on their personal machines.
And there are uncountable clones scattered about the internet.

<h2>Repositories</h2>

<p>The SQLite code is broken up into multiple repositories, each described
in a separate section below.

<h3>SQLite Source Code</h3>

<p>The SQLite source code and the [TCL test suite] are stored together
in a single repository.  This one repository is all that is required to
build the SQLite.  The source respository is public and is
readable by anonymous passers by on the internet.

<ul>
<li> Primary location: [https://www.sqlite.org/src]
<li> Backup A: [https://www2.sqlite.org/src]
<li> Backup B: [https://www3.sqlite.org/src]
</ul>

<p>There is an unofficial and unsanctioned Git clone of this repository
at [https://github.com/mackyle/sqlite].

<h3>SQLite Documentation Sources</h3>

<p>The documentation sources include documentation text and images with the
scripts and makefile needed to construct the SQLite website documentation.
This document is contained within the documentation sources.  The
document sources are kept in a separate repository distinct from the
source code.  The documentation sources repository is publicly readable.

<p>The makefiles and scripts used to generate the documentation gather
text from baseline documents in the documentation source repository.
Additional text is extracted from comments in the SQLite source code.
Requirements coverage information is extract from special comments in the
[TCL test suite] which is part of the source repository, and from
comments in the [TH3] test suite which is a separate private repository.

<ul>
<li> Primary location: [https://www.sqlite.org/docsrc]
<li> Backup A: [https://www2.sqlite.org/docsrc]
<li> Backup B: [https://www3.sqlite.org/docsrc]
</ul>

<h3>SQL Logic Test</h3>

<p>
The [SQL Logic Tests] are a set of test cases designed to show that
SQLite behaves the same as other SQL database engines.  These tests
are hosted in a separate code public repository.

<ul>
<li> Primary location: [https://www.sqlite.org/sqllogictest]
<li> Backups on private servers
</ul>

<h3>Test Harness #3</h3>

<p>
The [TH3|Test Harness #3] or [TH3] test suite is a private set of
test cases used to test SQLite to 100% MC/DC in an as-delivered
configuration.  TH3 sources are served on the same servers as the
other SQLite respositories, but differ from the others in being
proprietary.  The TH3 code is only accessible to SQLite developers.


<ul>
<li> Primary location: [https://www.sqlite.org/th3]
<li> Backup A: [https://www2.sqlite.org/th3]
<li> Backup B: [https://www3.sqlite.org/th3]
</ul>

<h3>Test Harness #3 Private Extensions</h3>

<p>
At one point, [TH3] was sometimes licensed to third-parties.
Such licensing no longer occurs.  However, back when it was
occurring, some of the TH3 test cases contained information that
was sensitive and could not be released even to licensees.  This
sensitive information is stored in yet another repository.

<ul>
<li> Primary location: [https://www.sqlite.org/th3private]
<li> Backups on private servers
</ul>

<h2>Software Verification Results</h2>

<p>
Release testing proceeds by [checklist].  The current status and
complete change history for each checklist is stored in a separate
SQLite database file.  These files are not version controlled, but
separate copies are maintained on private backup servers.

<p>The source code to the software that runs the checklists is stored
in its own Fossil repository at [https://www.sqlite.org/checklistapp].

<h1>Software Requirements Standards And Data</h1>

<p><i>TBD...</i>

<h1>Software Design And Coding Standands</h1>

<p><i>TBD...</i>

<h1>Problem Reports</h1>

<p><i>TBD...</i>

Changes to pages/testing.in.

957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
Those test cases will appear to fail when optimizations are disabled.
But the majority of test cases simply check that the correct answer
was obtained, and all of those cases can be run successfully with and
without the optimizations, in order to show that the optimizations do not
cause malfunctions.</p>


<tcl>hd_fragment cklist</tcl>
<h1>Checklists</h1>

<p>The SQLite developers use an on-line checklist to coordinate testing
activity and to verify that all tests pass prior each SQLite release.
<a href="http://www.sqlite.org/checklists/index.html">Past checklists</a>
are retained for historical reference.
(The checklists are read-only for anonymous internet viewers, but







|







957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
Those test cases will appear to fail when optimizations are disabled.
But the majority of test cases simply check that the correct answer
was obtained, and all of those cases can be run successfully with and
without the optimizations, in order to show that the optimizations do not
cause malfunctions.</p>


<tcl>hd_fragment cklist {release testing checklists} {checklist}</tcl>
<h1>Checklists</h1>

<p>The SQLite developers use an on-line checklist to coordinate testing
activity and to verify that all tests pass prior each SQLite release.
<a href="http://www.sqlite.org/checklists/index.html">Past checklists</a>
are retained for historical reference.
(The checklists are read-only for anonymous internet viewers, but