Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
Comment: | First draft of a new quality management plan. Still incomplete. |
---|---|
Downloads: | Tarball | ZIP archive |
Timelines: | family | ancestors | descendants | both | trunk |
Files: | files | file ages | folders |
SHA3-256: |
295c4e35bca0b90db008008c85ebe85a |
User & Date: | drh 2018-02-22 18:14:45.664 |
Context
2018-02-22
| ||
19:34 | Remove the NAWK macros from the Makefiles, since they are no longer used. (check-in: 76b1d7f1d4 user: drh tags: trunk) | |
18:14 | First draft of a new quality management plan. Still incomplete. (check-in: 295c4e35bc user: drh tags: trunk) | |
12:39 | Merge the 3.22.0 updates into trunk. (check-in: c0ef1f2bd8 user: drh tags: trunk) | |
Changes
Added pages/qmplan.in.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 | <title>Quality Management</title> <table_of_contents> <center> <font color="red" size="5"><b> This document is a work-in-progress. It is incomplete and unverified. </b></font> </center> <h1>Overview</h1> <p> This is the Quality Management Plan for SQLite. <p> Quality management documents tend to expand into binders full of incomprehensible jargon that nobody reads. This document strives to break that pattern by being concise and useful. <p> The inspiration for this document is [https://en.wikipedia.org/wiki/DO-178B|DO-178B]. Among quality standards, DO-178B seems to have the highest usefulness to paperwork ratio. Even so, the amount of documentation needed for a full-up DO-178B implementation is vast. SQLite strives to be nimble and low-ceremony, and to that end, much of the required DO-178B documentation is omitted. We retain only those parts that genuinely improve quality for a open-source software project such as SQLite. <p> The purpose of this document is to brief the reader on how SQLite development team functions on a daily basis, as they continuely enhance the SQLite software and work to improve its already high reliability. The document achieves its purpose if a competent developer can be assimilated into the development team quickly after perusing this document. <h2>About This Document</h2> <p> The quality management plan was originally composed by going through the description of outputs in section 11 of DO-178B (pages 48 through 56) and writing down those elements that seemed relevant to SQLite. The text will be subsequent revised to track enhancements to the SQLite quality process. <h1>Software Development Plan</h1> <p> This section is a coalescense of the Plan For Software Aspects Of Certification and the Software Development Plan sections of DO-178B. <p> See <a href='about.html'>About SQLite</a> for an overview of the SQLite software and what it does and how it is different. <h2>Software Life Cycle</h2> <p> SQLite uses a continuous integration process. The software is under constant enhancement and refinement. The latest trunk check-ins are frequently used internally for mission-critical operations. <p> There is no pre-defined release cycle. Releases occur when there is a critical mass of feature enhancements and/or bug fixes. Historically, releases have occurred about 5 or 6 times per year. Users of SQLite pick up new releases from the website on an as-needed basis. <h3>Maintenance Releases</h3> <p> Routine maintenance releases of SQLite contain feature enhancements, performance enhancements, and/or fixes for non-critical issues. The version number for major releases are of the form "3.N.0" for some integer N. See the [version numbering conventions] document for details. <p> Upcoming maintenance releases announced on the sqlite-users and sqlite-dev [mailing lists] about two weeks prior to the anticipated release. Approximately one week prior to release, the lead developer declares "pencils down" after which only bug-fix check-ins are allowed on trunk. A new [https://sqlite.org/checklists|release checklist] is created and updated as needed. As items of the checklist are verified, they are checked off and turn green. The release occurs when all elements of the checklist are green. That process normally takes about a week. <h3>Patch Releases</h3> <p> Occasionally, a serious problem is found and a small "patch" release must be made against a regular maintenance release. Patch are distinct from maintenance releases in that the number of lines of code changed from the previous release is very small. Every effort is made to avoid patch releases by making sure that maintenance releases are bug free. <p> Patch releases may or may not have a release checklist, depending on the issue. This is a judgement call by the project leader. <h2>Release History</h2> <p>The documentation system automatically maintains a [chronology] of past releases, as well as a [complete list of SQLite releases] with change summaries. <h2>Schedule</h2> SQLite has a long-range vision. Planning is done with the assumption that SQLite will be used and supported through at least the year 2050. All code is written with the idea that it will one day be read and maintained by people not yet born. The code is carefully commented with an eye toward helping those future developers more easily understand the logic and the rationale behind the code. <h1>Software Development Environment</h1> <p> SQLite is written in portable C code. Development work occurs on a mix of Linux, Mac, and Windows workstations. The developers use command-line tools and eschew integrated development environments (IDEs) whenever possible. All developers are expected to be fluent with the unix command-line. <p> A minimum setup for compiling and testing SQLite from canonical sources is as follows: <ul> <li> A host computer with a 32-bit or 64-bit address space. The OS can be Linux, Mac, Windows, *BSD, Solaris, or some other. <li> A C99 compiler such as GCC (including MinGW variants for Windows), Clang, or MSVC <li> A text editor of the user's choice supporting UTF-8 text. <li> [https://core.tcl.tk/|Tcl] version 8.6 or later. <li> The "make" utility, or optionally "nmake" on Windows. </ul> <p> The Tcl script language is used to help translate canonical source code into the [amalgamation] and to manage testing. Tcl is not used directly by SQLite itself (unless requested by a compile-time option). End users of the SQLite amalgamation sources do not need Tcl. <p> When building the [CLI], it is helpful, but not required, to have the following third-party libraries on hand: <ul> <li> [https://zlib.net/|zLib] <li> [http://git.savannah.gnu.org/cgit/readline.git?h=devel|readline] or [http://thrysoee.dk/editline/|editline] or [https://github.com/antirez/linenoise|linenoise] for command-line editing. </ul> <p> A complete release-test of SQLite requires additional software, <ul> <li> [http://www.valgrind.org/|valgrind] <li> [https://gcc.gnu.org/onlinedocs/gcc/Gcov.html|gcov] </ul> <p> SQLite is expected to operate the same, and use exactly the same [file format|on-disk format], on all modern operating systems, on all modern computer architectures, and using all modern C compilers. The developers are constantly testing SQLite on as many diverse platforms as they can get their hands on. <h1>Software Verification Plan</h1> <p>The testing process for SQLite is described in the [testing] document. Testing objectives include: <ul> <li> 100% MC/DC in an as-delivered configuration <li> Testing of both source code and object code <li> Testing on multiple platforms and with multiple compilers <li> Code change inspection <li> Dynamic and static analysis of the code </ul> <p>The testing process is controlled by the [release testing checklists]. The checklists succinctly summary all steps necessary to fully validate SQLite, and they record when and by whom each validation step was performed. <p>The set of checklist items for release checklist is potentially updated for each release. The content and complete history of each release checklist are retained for the historical record. <h1>Software Configuration Management</h1> <h2>Version Control</h2> <p> SQLite source code is managed using the [https://fossil-scm.org|Fossil] version control system. Fossil was written specifically to support SQLite development. Fossil provides both distributed version control and issue tracking. <h2>Survivability</h2> <p> All code is archived on three separate machines: [https://www.sqlite.org], [https://www2.sqlite.org], [https://www3.sqlite.org]. These machines are located in different cities (Dallas, Newark, and San Francisco, respectively) and managed by two different hosting companies ([https://linode.com|Linode] for the first two and [https://digitalocean.com|Digital Ocean] for the third). This diversity is intended to avoid a single point of failure. <p> The main machine in Dallas [https://www.sqlite.org/] is the primary server and the one that most people use. The other two are considered backups. <p> In addition to the official repositories, the developers typically keep complete clones of all software on their personal machines. And there are uncountable clones scattered about the internet. <h2>Repositories</h2> <p>The SQLite code is broken up into multiple repositories, each described in a separate section below. <h3>SQLite Source Code</h3> <p>The SQLite source code and the [TCL test suite] are stored together in a single repository. This one repository is all that is required to build the SQLite. The source respository is public and is readable by anonymous passers by on the internet. <ul> <li> Primary location: [https://www.sqlite.org/src] <li> Backup A: [https://www2.sqlite.org/src] <li> Backup B: [https://www3.sqlite.org/src] </ul> <p>There is an unofficial and unsanctioned Git clone of this repository at [https://github.com/mackyle/sqlite]. <h3>SQLite Documentation Sources</h3> <p>The documentation sources include documentation text and images with the scripts and makefile needed to construct the SQLite website documentation. This document is contained within the documentation sources. The document sources are kept in a separate repository distinct from the source code. The documentation sources repository is publicly readable. <p>The makefiles and scripts used to generate the documentation gather text from baseline documents in the documentation source repository. Additional text is extracted from comments in the SQLite source code. Requirements coverage information is extract from special comments in the [TCL test suite] which is part of the source repository, and from comments in the [TH3] test suite which is a separate private repository. <ul> <li> Primary location: [https://www.sqlite.org/docsrc] <li> Backup A: [https://www2.sqlite.org/docsrc] <li> Backup B: [https://www3.sqlite.org/docsrc] </ul> <h3>SQL Logic Test</h3> <p> The [SQL Logic Tests] are a set of test cases designed to show that SQLite behaves the same as other SQL database engines. These tests are hosted in a separate code public repository. <ul> <li> Primary location: [https://www.sqlite.org/sqllogictest] <li> Backups on private servers </ul> <h3>Test Harness #3</h3> <p> The [TH3|Test Harness #3] or [TH3] test suite is a private set of test cases used to test SQLite to 100% MC/DC in an as-delivered configuration. TH3 sources are served on the same servers as the other SQLite respositories, but differ from the others in being proprietary. The TH3 code is only accessible to SQLite developers. <ul> <li> Primary location: [https://www.sqlite.org/th3] <li> Backup A: [https://www2.sqlite.org/th3] <li> Backup B: [https://www3.sqlite.org/th3] </ul> <h3>Test Harness #3 Private Extensions</h3> <p> At one point, [TH3] was sometimes licensed to third-parties. Such licensing no longer occurs. However, back when it was occurring, some of the TH3 test cases contained information that was sensitive and could not be released even to licensees. This sensitive information is stored in yet another repository. <ul> <li> Primary location: [https://www.sqlite.org/th3private] <li> Backups on private servers </ul> <h2>Software Verification Results</h2> <p> Release testing proceeds by [checklist]. The current status and complete change history for each checklist is stored in a separate SQLite database file. These files are not version controlled, but separate copies are maintained on private backup servers. <p>The source code to the software that runs the checklists is stored in its own Fossil repository at [https://www.sqlite.org/checklistapp]. <h1>Software Requirements Standards And Data</h1> <p><i>TBD...</i> <h1>Software Design And Coding Standands</h1> <p><i>TBD...</i> <h1>Problem Reports</h1> <p><i>TBD...</i> |
Changes to pages/testing.in.
︙ | ︙ | |||
957 958 959 960 961 962 963 | Those test cases will appear to fail when optimizations are disabled. But the majority of test cases simply check that the correct answer was obtained, and all of those cases can be run successfully with and without the optimizations, in order to show that the optimizations do not cause malfunctions.</p> | | | 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 | Those test cases will appear to fail when optimizations are disabled. But the majority of test cases simply check that the correct answer was obtained, and all of those cases can be run successfully with and without the optimizations, in order to show that the optimizations do not cause malfunctions.</p> <tcl>hd_fragment cklist {release testing checklists} {checklist}</tcl> <h1>Checklists</h1> <p>The SQLite developers use an on-line checklist to coordinate testing activity and to verify that all tests pass prior each SQLite release. <a href="http://www.sqlite.org/checklists/index.html">Past checklists</a> are retained for historical reference. (The checklists are read-only for anonymous internet viewers, but |
︙ | ︙ |