SQLite

Timeline
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

100 most recent check-ins that include changes to files matching 'ext/fts2/*'

2023-07-06
17:55
Merge all recent trunk enhancements into the wal-shm-exceptions branch. (check-in: e59cea2d user: drh tags: wal-shm-exceptions)
2023-05-08
21:35
WIP, sync with trunk to pickup 22 weeks of shell library enhancements. (check-in: 00d0d5b1 user: larrybr tags: cli_extension)
2023-02-07
15:29
Merge latest trunk changes into this branch. (check-in: 4d21640c user: dan tags: reuse-schema)
2023-02-06
17:35
Merge the latest trunk enhancements into the begin-concurrent branch. (check-in: 46e79167 user: drh tags: begin-concurrent)
2023-02-02
14:27
Merge latest enhancments into this branch from branch wal2. (Leaf check-in: 5a22010c user: dan tags: begin-concurrent-pnu-wal2)
14:21
Merge trunk enhancements into this branch. (check-in: c9b01802 user: dan tags: wal2)
2023-01-21
12:18
Merge trunk into wasi-patches branch. (check-in: 6fc20d75 user: stephan tags: wasi-patches)
2023-01-14
19:53
Omit the long-disused FTS1 and FTS2 implements from the active source tree. The code will persist forever in the source repository, but there is no point in carrying it around in the latest tarballs where it is never used. (check-in: 2bb50d5a user: drh tags: trunk)
2020-07-29
16:18
Dozens and dozens of typo fixes in comments. This change adds no value to the end product and is disruptive, so it is questionable whether or not it will ever land on trunk. (Leaf check-in: a80ae2c9 user: drh tags: typos)
2017-06-02
23:44
Merge all recent trunk enhancements. (check-in: 1d23294d user: drh tags: lsm-vtab)
2017-03-31
11:20
Merge latest trunk into this branch. (check-in: cb721d0b user: dan tags: schemalint)
2016-08-11
19:12
Merge version 3.14 plus some subsequent patches (including the page-cache performance patch) from trunk. (check-in: d9f8918c user: drh tags: begin-concurrent)
2016-08-04
09:13
Merge the ORDER BY LIMIT fix, and other enhancements, from trunk. (check-in: bf5a9542 user: drh tags: apple-osx)
2016-07-30
17:59
Merge latest trunk with this branch. (check-in: 63ae02d0 user: dan tags: rowvalue)
2016-07-28
18:55
Decorate all interfaces with calling convention macros. (check-in: fd784887 user: drh tags: trunk)
14:37
Allow the 'testfixture.exe' target to be compiled with the __stdcall calling convention. (check-in: e8be3dfe user: mistachkin tags: callbackConv)
2014-05-08
23:01
Initial attempt to merge in all trunk changes over the previous 1.5 years. This check-in compiles but there are compiler warnings and "make test" segfaults after only running a few test modules. (check-in: 9411d7dc user: drh tags: apple-osx)
2014-03-13
15:41
Merge latest trunk changes into this branch. (check-in: d17231b6 user: dan tags: threads)
2013-09-12
00:40
Merge updates from trunk. (Closed-Leaf check-in: fca799f0 user: mistachkin tags: vsix2013)
2013-07-20
00:39
Merge updates from trunk. (check-in: dd4b77c8 user: mistachkin tags: toTypeFuncs)
2013-07-09
13:05
Pull in all the latest changes from trunk. (check-in: af3ca4c6 user: drh tags: sessions)
12:36
Modify several extensions to use the new exported function naming. Fix some shared library compilation issues. (check-in: 1e39f850 user: drh tags: trunk)
2013-07-04
23:53
Modify several extensions to use the new exported function naming. Fix some shared library compilation issues. (Closed-Leaf check-in: f2ab8747 user: mistachkin tags: extRefactor)
2013-05-15
08:00
Merge updates from trunk. (check-in: 6fbad957 user: mistachkin tags: toTypeFuncs)
2013-03-28
01:19
Merge the changes for the 3.7.16.1 release candidate into the sessions branch. (check-in: ee35a897 user: drh tags: sessions)
2013-03-25
19:57
Merge all recent trunk changes into the experimental-mmap branch. (check-in: a607d63f user: drh tags: experimental-mmap)
2013-03-21
21:20
Many spelling fixes in comments. No changes to code. (check-in: 6f6e2d50 user: mistachkin tags: trunk)
2012-12-14
15:54
Merge in all the trunk changes that have occurred since this branch was opened. (check-in: 6d507e4d user: drh tags: vtab-IN-opt)
2012-12-08
06:46
Merge updates from trunk. (check-in: e65db42c user: mistachkin tags: configReadOnly)
2012-10-30
21:03
Pull all the latest trunk enhancements into the sessions branch. (check-in: fce667f2 user: drh tags: sessions)
2012-10-19
02:10
Make sure substructure elements have proper alignment in the ICU tokenizers of FTS2 and FTS3. (check-in: aaa2d9b0 user: drh tags: trunk)
2012-08-31
14:23
Update with latest trunk changes. (check-in: 87995dc9 user: dan tags: sessions)
14:10
Merge in latest trunk changes. (check-in: bc9b9cd0 user: dan tags: apple-osx)
2012-08-25
10:01
Fix all known instances of 'repeated the' style typos in comments. No changes to code. (check-in: 9b19b847 user: mistachkin tags: trunk)
2010-11-19
23:50
Merge all the latest changes from the trunk into the apple-osx branch. (check-in: c8bc057c user: drh tags: apple-osx)
2010-09-17
17:10
Remove additional occurrences of <ctype.h> from FTS2. (check-in: ee52589c user: drh tags: trunk)
01:07
Completely remove all trace of ctype.h from FTS2. (check-in: 87684566 user: drh tags: trunk)
2009-11-04
18:51
Fix documentation typos. (check-in: 590d4ac1 user: drh tags: shunning-error)
2009-03-05
04:20
Corrected typos and misspellings. Ticket #3702. (CVS 6336) (check-in: 6404afa0 user: shane tags: trunk)
2009-01-01
12:34
Additional test cases and cleanup of FTS3 parenthesis processing. (CVS 6094) (check-in: afac4293 user: drh tags: trunk)
2008-12-18
05:30
Fix a bug in icuOpen() in fts2. (CVS 6038) (check-in: b9c722bd user: danielk1977 tags: trunk)
2008-07-29
20:38
Backport http://www.sqlite.org/cvstrac/chngview?cn=5489 from fts3. Re-used prepared statement from fts2 cursor. (CVS 5499) (check-in: 02870ed2 user: shess tags: trunk)
2008-07-22
23:54
Be a bit more susicious of invalid results from the tokenizer. Backports check-in (4514) from fts3. (CVS 5459) (check-in: 311aeb9c user: shess tags: trunk)
23:49
Implement optimize() function. Backports check-in (5417) from fts3. (CVS 5458) (check-in: c16900dc user: shess tags: trunk)
23:41
Delete all fts2 index data the table becomes empty. Backports check-in (5413) from fts3. (CVS 5457) (check-in: 4c98179b user: shess tags: trunk)
23:32
fts2 functions for testing scripts. Backports (5340) from fts3. (CVS 5456) (check-in: 4e47394b user: shess tags: trunk)
23:08
Change prefix search from O(N*M) to O(NlogM). Backports (4599) from fts3. (CVS 5455) (check-in: 3f614453 user: shess tags: trunk)
22:57
Changes fts2 to use only sqlite3_malloc() and not system malloc. Backports (4554) and (4555) from fts3. (CVS 5454) (check-in: ecf2dec6 user: shess tags: trunk)
22:20
fts2.c buildTerms() passes -1 for nInput. Backports (4511) from fts3. (CVS 5453) (check-in: d562515e user: shess tags: trunk)
22:15
Cleanup the hash functions in FTS2. Backports (4440) from fts3. (CVS 5452) (check-in: e31d2f87 user: shess tags: trunk)
2008-04-12
13:06
Remove all instances of sprintf() from the FTS modules. Ticket #3049. (CVS 4996) (check-in: 062bf5d4 user: drh tags: trunk)
2007-12-13
21:54
Change all instances of "it's" in comments to either "its" or "it is", as appropriate, in case the comments are ever again read by a pedantic grammarian. Ticket #2840. (CVS 4629) (check-in: 4e91a267 user: drh tags: trunk)
2007-11-23
18:06
Do not require SQLITE_ENABLE_BROKEN_FTS2 if FTS2 is not enabled. The same for FTS1. Ticket #2777. (CVS 4556) (check-in: f94cdcfd user: drh tags: trunk)
2007-11-16
00:23
Don't do anything when input doclists are both empty. Ticket #2774 (CVS 4546) (check-in: 75cb46f8 user: shess tags: trunk)
2007-09-13
18:16
Drop the forced error from fts3.c and add forced errors to fts2.c and fts1.c. (CVS 4427) (check-in: fec6567a user: shess tags: trunk)
2007-08-28
20:36
Fix memory leak of InteriorReader.term. Comes up when doing queries against large segments. (CVS 4315) (check-in: 6c617bd8 user: shess tags: trunk)
2007-08-10
23:47
Convert fts2 to use sqlite3_prepare_v2() to prevent certain logic errors around SQLITE_SCHEMA handling. This also allows sql_step_statement() and sql_step_leaf_statement() to be replaced with sqlite3_step().

Also fix a logic error in flushPendingTerms() which was clearing the term table in case of error. This was wrong in the face of SQLITE_SCHEMA. Even though the change to sqlite3_prepare_v2() should cause us not to see SQLITE_SCHEMA any longer, it was still a logic error... (CVS 4205) (check-in: 16730cb1 user: shess tags: trunk)

2007-08-05
23:52
Fix some compiler warnings. (CVS 4196) (check-in: 6cc15409 user: drh tags: trunk)
2007-07-30
18:55
Fix ticket #2439: the FTS1 and FTS2 extensions use the non-standard, unportable and highly deprecated <malloc.h> header on all platforms except Apple Mac OS X. The <malloc.h> actually is never required on any OS with an at least partly POSIX-conforming API as the malloc(3) & friends functions officially live in <stdlib.h> since over 10 years. Under some platform like FreeBSD the inclusion of <malloc.h> since a few years even causes an "#error" and this way a build failure. So, just get rid of the bad <malloc.h> usage in FTS1 and FTS2 extensions at all and stick with <stdlib.h> there only. (CVS 4191) (check-in: 3f9a6661 user: rse tags: trunk)
2007-07-02
10:16
Modify handling of SQLITE_SCHEMA in fts2 code. An SQLITE_SCHEMA error may cause SQLite to reload the internal schema, deleting and recreating v-table objects. So the sqlite3_vtab structure can be deleted out from under a v-table implementation. (CVS 4151) (check-in: dee1a0fd user: danielk1977 tags: trunk)
2007-06-27
16:26
Implement xRename() for fts2 so that it is possible to rename fts2 tables. (CVS 4143) (check-in: 488474fd user: danielk1977 tags: trunk)
2007-06-26
12:54
Reorganize comments in fts2_tokenizer.h. No code changes. (CVS 4132) (check-in: b331e303 user: danielk1977 tags: trunk)
10:56
Remove the unused EXTSRC variable from the non-configure makefile. (CVS 4129) (check-in: bbdcf372 user: danielk1977 tags: trunk)
10:55
Fix an unitialized variable in fts2. (CVS 4128) (check-in: c349cf94 user: danielk1977 tags: trunk)
2007-06-25
14:28
Modify the non-configure build system to make it easier to build the library with the fts2 or icu extensions linked in. (CVS 4121) (check-in: 02b23c43 user: danielk1977 tags: trunk)
13:50
Put #ifdefs in fts2_tokenizer so that the build works even when FTS2 is omitted. Add the SQLite blessing to the header comments on all FTS2 source files. (CVS 4120) (check-in: c795e6fd user: drh tags: trunk)
12:49
All the use of MySQL-style quoting in the FTS modules. Ticket #2446. (CVS 4119) (check-in: 3be2a6d1 user: drh tags: trunk)
12:05
Add a test that calls fts2_tokenizer() with an argument set via C code. (CVS 4118) (check-in: fbcf2d75 user: danielk1977 tags: trunk)
11:24
Add some tests for the fts2 icu tokenizer. (CVS 4117) (check-in: b79ced3e user: danielk1977 tags: trunk)
09:52
Add some documentation for user-defined fts2 tokenizers. (CVS 4116) (check-in: 5a9eee86 user: danielk1977 tags: trunk)
2007-06-22
15:21
Extend fts2 so that user defined tokenizers may be added. Add a tokenizer that uses the ICU library if available. Documentation and tests to come. (CVS 4108) (check-in: 68677e42 user: danielk1977 tags: trunk)
2007-06-20
06:23
Fix snippet generation when the left-most column of an fts2 table is used in the MATCH clause. Fix for ticket #2429. (CVS 4095) (check-in: fec56ad2 user: danielk1977 tags: trunk)
2007-06-12
18:20
Minor comment edits from my prefix development client. No code changes. (CVS 4058) (check-in: 6953cd09 user: shess tags: trunk)
2007-05-21
21:59
Fix overzealous fts2 assertions WRT rowid 0 or lower. Only check that docids are ascending if there was a prior docid set for the doclist, ignore the initial docid of 0. (CVS 4026) (check-in: ed3a131f user: shess tags: trunk)
2007-05-01
18:25
Enable prefix-search in query-parsing and snippet generation. If the character immediately after the end of a term is '*', that term is marked for prefix matching. Modify term comparison in snippetOffsetsOfColumn() to respect isPrefix. fts2n.test runs prefix searching through some obvious test cases. (CVS 3893) (check-in: 7c4c6592 user: shess tags: trunk)
17:14
Modify loadSegmentLeavesInt() to correctly handle prefix searching. The new function docListUnion() is used to accumulate a union of the hits for the matching terms, which will be merged across segments using docListMerge(). (CVS 3891) (check-in: 72c79630 user: shess tags: trunk)
2007-04-30
22:09
Propagate prefix flag through implementation of doclist query code. Also implement correct prefix-handling for traversal of interior nodes of segment tree. A given prefix can span multiple children of an interior node, and from there the branches need to be followed in parallel. (CVS 3889) (check-in: cae844a0 user: shess tags: trunk)
17:52
Lift docListMerge() call out of loadSegmentLeavesInt() for prefix search. Doclists from multiple prefix matches will need a union merge function, which will have to logically happen across a segment before doclists are merged between segments. (CVS 3887) (check-in: 7ddb8266 user: shess tags: trunk)
2007-04-27
22:02
Break interior-node and leaf-node readers apart in loadSegment(). Previously, the code looped until the block was a leaf node as indicated by a leading NUL. Now the code loops until it finds a block in the range of leaf nodes for this segment, then reads it using LeavesReader. This will make it easier to traverse a range of leaves when doing a prefix search. (CVS 3884) (check-in: 9466367d user: shess tags: trunk)
21:24
Lift code to traverse interior nodes out of loadSegment(). Refactoring towards prefix searching. (CVS 3882) (check-in: 25935db7 user: shess tags: trunk)
21:02
Refactor fts2 loadSegmentLeaf() in preparation for prefix-searching. Prefix-searching will want to accumulate data across multiple leaves in the segment, using LeavesReader instead of LeafReader is the first step in that direction. (CVS 3881) (check-in: 22ffdae4 user: shess tags: trunk)
2007-04-21
16:37
Add the ability to turn the FTS2 module into an amalgamation. (CVS 3864) (check-in: 94374654 user: drh tags: trunk)
2007-04-19
18:36
Fix bug in fts2 handling of OR queries. When one doclist ends before the other, the code potentially tries to read past the end of the doclist. http://www.sqlite.org/cvstrac/tktview?tn=2309 (CVS 3862) (check-in: dfac6082 user: shess tags: trunk)
2007-04-09
20:45
Fix crash in delete when existing row has null fields. Previous code assumed that the row had values in all columns, sigh. Fixes bug http://www.sqlite.org/cvstrac/tktview?tn=2289 . (CVS 3833) (check-in: 81be7290 user: shess tags: trunk)
2007-03-29
18:41
Buffer updates per-transaction rather than per-update. If lots of updates happen within a single transaction, there was a lot of wasted encode/decode overhead due to segment merges. This code buffers updates in memory and writes out larger level-0 segments. It only works when documents are presented in ascending order by docid. Comparing a test set running 100 documents per transaction, the total runtime is cut almost in half. (CVS 3751) (check-in: 0229cba6 user: shess tags: trunk)
16:30
Don't call ctype functions on hi-bit chars. Some platforms raise assertions when this occurs, and it's almost certainly not the right thing to do in the first place. (CVS 3746) (check-in: f6c3abdc user: shess tags: trunk)
2007-03-22
00:14
Refactor PLWriter to remove owned buffer. DLCollector (Document List Collector) now handles the case where PLWriter (Position List Writer) needed a local buffer. Change to using the associated DLWriter (Document List Writer) buffer, which reduces the number of memory copies needed in doclist processing, and brings PLWriter operation in line with DLWriter operation. (CVS 3707) (check-in: d04fa3a1 user: shess tags: trunk)
2007-03-20
23:52
Refactor PLWriter in preparation for buffered-document change. Currently, PLWriter (Position List Writer) creates a locally-owned DataBuffer to write into. This is necessary to support doclist collection during tokenization, where there is no obvious buffer to write output to, but is not necessary for the other users of PLWriter. This change adds a DLCollector (Doc List Collector) structure to handle the tokenization case.

Also fix a potential memory leak in writeZeroSegment(). In case of error from leafWriterStep(), the DataBuffer dl was being leaked. (CVS 3706) (check-in: 1b9918e2 user: shess tags: trunk)

2007-03-16
18:30
Out-of-memory cleanup in tokenizers. Handle NULL return from malloc/calloc/realloc appropriately, and use sizeof(var) instead of sizeof(type) to make certain that we don't get a mismatch between them as the code rots. (CVS 3693) (check-in: fbc53da8 user: shess tags: trunk)
2007-02-07
01:01
http://www.sqlite.org/cvstrac/tktview?tn=2219

When creating fts tables in an attached database, the backing tables are created in database 'main'. This change propagates the appropriate database name to the routines which build sql statements.

Note that I propagate the database name and table name separately. I briefly considered just making the table name be "db.table", but it didn't fit so well in the model used to store the table name and other information, and having the db name passed separately seemed a bit more transparent. (CVS 3631) (check-in: 283385d2 user: shess tags: trunk)

2007-01-19
22:59
http://www.sqlite.org/cvstrac/tktview?tn=2166,35

Calling UPDATE against an fts table in a UTF-16 database inserts corrupted data into the database. The UTF-8 data is being inserted directly. This appears to happen because sqlite3_ value_text() destructively coerces a value to UTF-8, and it's never converted back when updating the table. This works around the problem by rearranging things so that the update happens before the coercion. (CVS 3596) (check-in: 4f2ab4b6 user: shess tags: trunk)

2006-11-29
23:41
Drop a couple variables which are no longer used anywhere. (CVS 3524) (check-in: 08c2cc0e user: shess tags: trunk)
05:17
http://www.sqlite.org/cvstrac/tktview?tn=2046

The virtual table interface allows for a cursor to field multiple xFilter() calls. For instance, if a join is done with a virtual table, there could be a call for each row which potentially matches. Unfortunately, fulltextFilter() assumes that it has a fresh cursor, and overwrites a prepared statement and a malloc'ed pointer, resulting in unfinalized statements and a memory leak.

This change hacks the code to manually clean up offending items in fulltextFilter(), emphasis on "hacks", since it's a fragile fix insofar as future additions to fulltext_cursor could continue to have the problem. (CVS 3521) (check-in: 18142fdb user: shess tags: trunk)

01:02
Delta-encode terms in interior nodes. While experiments have shown that this is of marginal utility when encoding terms resulting from regular English text, it turns out to be very useful when encoding inputs with very large terms. (CVS 3520) (check-in: c8151a99 user: shess tags: trunk)
2006-11-18
00:12
Store minimal terms in interior nodes. Whenever there's a break between leaf nodes, instead of storing the entire leftmost term of the rightmost child, store only that portion of the leftmost term necessary to distinguish it from the rightmost term of the leftmost child. (CVS 3513) (check-in: f6e0b080 user: shess tags: trunk)
2006-11-17
21:12
Refactoring groundwork for coming work on interior nodes. Change LeafWriter to use empty data buffer (instead of empty term) to detect an empty block. Code to validate interior nodes. Moderate revisions to leaf-node and doclist validation. Recast leafWriterStep() in terms of LeafWriterStepMerge(). (CVS 3512) (check-in: f30771d5 user: shess tags: trunk)
2006-11-13
21:09
Delta-encode docids. This is good for around 22% reduction in index size with DL_POSITIONS. It improves performance about 5%-6%. (CVS 3511) (check-in: 9b6d413d user: shess tags: trunk)
21:00
Require a minimum fanout for interior nodes. This prevents cases where excessively large terms keep the tree from finding a single root. A downside is that this could result in large interior nodes in the presence of large terms, which may be prone to fragmentation, though if the nodes were smaller that would translate into more levels in the tree, which would also have that problem. (CVS 3510) (check-in: 64b7e340 user: shess tags: trunk)
20:15
Allow backing tables to be missing on dropping fts table. Fixes http://www.sqlite.org/cvstrac/tktview?tn=1992,35 . (CVS 3509) (check-in: 9628a61a user: shess tags: trunk)
2006-10-31
18:13
Fix a pair of memory leaks. These were turned up by running valgrind memcheck with various 10k doc insert, update, delete, and query tests. (CVS 3497) (check-in: 3cd9b64b user: shess tags: trunk)