/ Timeline
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

100 most recent check-ins that include changes to files matching 'ext/fts2/*'

2017-06-02
23:44
Merge all recent trunk enhancements. check-in: 1d23294d user: drh tags: lsm-vtab
2017-03-31
11:20
Merge latest trunk into this branch. check-in: cb721d0b user: dan tags: schemalint
2016-08-11
19:12
Merge version 3.14 plus some subsequent patches (including the page-cache performance patch) from trunk. check-in: d9f8918c user: drh tags: begin-concurrent
2016-08-04
09:13
Merge the ORDER BY LIMIT fix, and other enhancements, from trunk. check-in: bf5a9542 user: drh tags: apple-osx
2016-07-30
17:59
Merge latest trunk with this branch. check-in: 63ae02d0 user: dan tags: rowvalue
2016-07-28
18:55
Decorate all interfaces with calling convention macros. check-in: fd784887 user: drh tags: trunk
14:37
Allow the 'testfixture.exe' target to be compiled with the __stdcall calling convention. check-in: e8be3dfe user: mistachkin tags: callbackConv
2014-05-08
23:01
Initial attempt to merge in all trunk changes over the previous 1.5 years. This check-in compiles but there are compiler warnings and "make test" segfaults after only running a few test modules. check-in: 9411d7dc user: drh tags: apple-osx
2014-03-13
15:41
Merge latest trunk changes into this branch. check-in: d17231b6 user: dan tags: threads
2013-09-12
00:40
Merge updates from trunk. Closed-Leaf check-in: fca799f0 user: mistachkin tags: vsix2013
2013-07-20
00:39
Merge updates from trunk. check-in: dd4b77c8 user: mistachkin tags: toTypeFuncs
2013-07-09
13:05
Pull in all the latest changes from trunk. check-in: af3ca4c6 user: drh tags: sessions
12:36
Modify several extensions to use the new exported function naming. Fix some shared library compilation issues. check-in: 1e39f850 user: drh tags: trunk
2013-07-04
23:53
Modify several extensions to use the new exported function naming. Fix some shared library compilation issues. Closed-Leaf check-in: f2ab8747 user: mistachkin tags: extRefactor
2013-05-15
08:00
Merge updates from trunk. check-in: 6fbad957 user: mistachkin tags: toTypeFuncs
2013-03-28
01:19
Merge the changes for the 3.7.16.1 release candidate into the sessions branch. check-in: ee35a897 user: drh tags: sessions
2013-03-25
19:57
Merge all recent trunk changes into the experimental-mmap branch. check-in: a607d63f user: drh tags: experimental-mmap
2013-03-21
21:20
Many spelling fixes in comments. No changes to code. check-in: 6f6e2d50 user: mistachkin tags: trunk
2012-12-14
15:54
Merge in all the trunk changes that have occurred since this branch was opened. check-in: 6d507e4d user: drh tags: vtab-IN-opt
2012-12-08
06:46
Merge updates from trunk. check-in: e65db42c user: mistachkin tags: configReadOnly
2012-10-30
21:03
Pull all the latest trunk enhancements into the sessions branch. check-in: fce667f2 user: drh tags: sessions
2012-10-19
02:10
Make sure substructure elements have proper alignment in the ICU tokenizers of FTS2 and FTS3. check-in: aaa2d9b0 user: drh tags: trunk
2012-08-31
14:23
Update with latest trunk changes. check-in: 87995dc9 user: dan tags: sessions
14:10
Merge in latest trunk changes. check-in: bc9b9cd0 user: dan tags: apple-osx
2012-08-25
10:01
Fix all known instances of 'repeated the' style typos in comments. No changes to code. check-in: 9b19b847 user: mistachkin tags: trunk
2010-11-19
23:50
Merge all the latest changes from the trunk into the apple-osx branch. check-in: c8bc057c user: drh tags: apple-osx
2010-09-17
17:10
Remove additional occurrences of <ctype.h> from FTS2. check-in: ee52589c user: drh tags: trunk
01:07
Completely remove all trace of ctype.h from FTS2. check-in: 87684566 user: drh tags: trunk
2009-11-04
18:51
Fix documentation typos. check-in: 590d4ac1 user: drh tags: shunning-error
2009-03-05
04:20
Corrected typos and misspellings. Ticket #3702. (CVS 6336) check-in: 6404afa0 user: shane tags: trunk
2009-01-01
12:34
Additional test cases and cleanup of FTS3 parenthesis processing. (CVS 6094) check-in: afac4293 user: drh tags: trunk
2008-12-18
05:30
Fix a bug in icuOpen() in fts2. (CVS 6038) check-in: b9c722bd user: danielk1977 tags: trunk
2008-07-29
20:38
Backport http://www.sqlite.org/cvstrac/chngview?cn=5489 from fts3. Re-used prepared statement from fts2 cursor. (CVS 5499) check-in: 02870ed2 user: shess tags: trunk
2008-07-22
23:54
Be a bit more susicious of invalid results from the tokenizer. Backports check-in (4514) from fts3. (CVS 5459) check-in: 311aeb9c user: shess tags: trunk
23:49
Implement optimize() function. Backports check-in (5417) from fts3. (CVS 5458) check-in: c16900dc user: shess tags: trunk
23:41
Delete all fts2 index data the table becomes empty. Backports check-in (5413) from fts3. (CVS 5457) check-in: 4c98179b user: shess tags: trunk
23:32
fts2 functions for testing scripts. Backports (5340) from fts3. (CVS 5456) check-in: 4e47394b user: shess tags: trunk
23:08
Change prefix search from O(N*M) to O(NlogM). Backports (4599) from fts3. (CVS 5455) check-in: 3f614453 user: shess tags: trunk
22:57
Changes fts2 to use only sqlite3_malloc() and not system malloc. Backports (4554) and (4555) from fts3. (CVS 5454) check-in: ecf2dec6 user: shess tags: trunk
22:20
fts2.c buildTerms() passes -1 for nInput. Backports (4511) from fts3. (CVS 5453) check-in: d562515e user: shess tags: trunk
22:15
Cleanup the hash functions in FTS2. Backports (4440) from fts3. (CVS 5452) check-in: e31d2f87 user: shess tags: trunk
2008-04-12
13:06
Remove all instances of sprintf() from the FTS modules. Ticket #3049. (CVS 4996) check-in: 062bf5d4 user: drh tags: trunk
2007-12-13
21:54
Change all instances of "it's" in comments to either "its" or "it is", as appropriate, in case the comments are ever again read by a pedantic grammarian. Ticket #2840. (CVS 4629) check-in: 4e91a267 user: drh tags: trunk
2007-11-23
18:06
Do not require SQLITE_ENABLE_BROKEN_FTS2 if FTS2 is not enabled. The same for FTS1. Ticket #2777. (CVS 4556) check-in: f94cdcfd user: drh tags: trunk
2007-11-16
00:23
Don't do anything when input doclists are both empty. Ticket #2774 (CVS 4546) check-in: 75cb46f8 user: shess tags: trunk
2007-09-13
18:16
Drop the forced error from fts3.c and add forced errors to fts2.c and fts1.c. (CVS 4427) check-in: fec6567a user: shess tags: trunk
2007-08-28
20:36
Fix memory leak of InteriorReader.term. Comes up when doing queries against large segments. (CVS 4315) check-in: 6c617bd8 user: shess tags: trunk
2007-08-10
23:47
Convert fts2 to use sqlite3_prepare_v2() to prevent certain logic errors around SQLITE_SCHEMA handling. This also allows sql_step_statement() and sql_step_leaf_statement() to be replaced with sqlite3_step().    Also fix a logic error in flushPendingTerms() which was clearing the term table in case of error. This was wrong in the face of SQLITE_SCHEMA. Even though the change to sqlite3_prepare_v2() should cause us not to see SQLITE_SCHEMA any longer, it was still a logic error... (CVS 4205) check-in: 16730cb1 user: shess tags: trunk
2007-08-05
23:52
Fix some compiler warnings. (CVS 4196) check-in: 6cc15409 user: drh tags: trunk
2007-07-30
18:55
Fix ticket #2439: the FTS1 and FTS2 extensions use the non-standard, unportable and highly deprecated <malloc.h> header on all platforms except Apple Mac OS X. The <malloc.h> actually is never required on any OS with an at least partly POSIX-conforming API as the malloc(3) & friends functions officially live in <stdlib.h> since over 10 years. Under some platform like FreeBSD the inclusion of <malloc.h> since a few years even causes an "#error" and this way a build failure. So, just get rid of the bad <malloc.h> usage in FTS1 and FTS2 extensions at all and stick with <stdlib.h> there only. (CVS 4191) check-in: 3f9a6661 user: rse tags: trunk
2007-07-02
10:16
Modify handling of SQLITE_SCHEMA in fts2 code. An SQLITE_SCHEMA error may cause SQLite to reload the internal schema, deleting and recreating v-table objects. So the sqlite3_vtab structure can be deleted out from under a v-table implementation. (CVS 4151) check-in: dee1a0fd user: danielk1977 tags: trunk
2007-06-27
16:26
Implement xRename() for fts2 so that it is possible to rename fts2 tables. (CVS 4143) check-in: 488474fd user: danielk1977 tags: trunk
2007-06-26
12:54
Reorganize comments in fts2_tokenizer.h. No code changes. (CVS 4132) check-in: b331e303 user: danielk1977 tags: trunk
10:56
Remove the unused EXTSRC variable from the non-configure makefile. (CVS 4129) check-in: bbdcf372 user: danielk1977 tags: trunk
10:55
Fix an unitialized variable in fts2. (CVS 4128) check-in: c349cf94 user: danielk1977 tags: trunk
2007-06-25
14:28
Modify the non-configure build system to make it easier to build the library with the fts2 or icu extensions linked in. (CVS 4121) check-in: 02b23c43 user: danielk1977 tags: trunk
13:50
Put #ifdefs in fts2_tokenizer so that the build works even when FTS2 is omitted. Add the SQLite blessing to the header comments on all FTS2 source files. (CVS 4120) check-in: c795e6fd user: drh tags: trunk
12:49
All the use of MySQL-style quoting in the FTS modules. Ticket #2446. (CVS 4119) check-in: 3be2a6d1 user: drh tags: trunk
12:05
Add a test that calls fts2_tokenizer() with an argument set via C code. (CVS 4118) check-in: fbcf2d75 user: danielk1977 tags: trunk
11:24
Add some tests for the fts2 icu tokenizer. (CVS 4117) check-in: b79ced3e user: danielk1977 tags: trunk
09:52
Add some documentation for user-defined fts2 tokenizers. (CVS 4116) check-in: 5a9eee86 user: danielk1977 tags: trunk
2007-06-22
15:21
Extend fts2 so that user defined tokenizers may be added. Add a tokenizer that uses the ICU library if available. Documentation and tests to come. (CVS 4108) check-in: 68677e42 user: danielk1977 tags: trunk
2007-06-20
06:23
Fix snippet generation when the left-most column of an fts2 table is used in the MATCH clause. Fix for ticket #2429. (CVS 4095) check-in: fec56ad2 user: danielk1977 tags: trunk
2007-06-12
18:20
Minor comment edits from my prefix development client. No code changes. (CVS 4058) check-in: 6953cd09 user: shess tags: trunk
2007-05-21
21:59
Fix overzealous fts2 assertions WRT rowid 0 or lower. Only check that docids are ascending if there was a prior docid set for the doclist, ignore the initial docid of 0. (CVS 4026) check-in: ed3a131f user: shess tags: trunk
2007-05-01
18:25
Enable prefix-search in query-parsing and snippet generation. If the character immediately after the end of a term is '*', that term is marked for prefix matching. Modify term comparison in snippetOffsetsOfColumn() to respect isPrefix. fts2n.test runs prefix searching through some obvious test cases. (CVS 3893) check-in: 7c4c6592 user: shess tags: trunk
17:14
Modify loadSegmentLeavesInt() to correctly handle prefix searching. The new function docListUnion() is used to accumulate a union of the hits for the matching terms, which will be merged across segments using docListMerge(). (CVS 3891) check-in: 72c79630 user: shess tags: trunk
2007-04-30
22:09
Propagate prefix flag through implementation of doclist query code. Also implement correct prefix-handling for traversal of interior nodes of segment tree. A given prefix can span multiple children of an interior node, and from there the branches need to be followed in parallel. (CVS 3889) check-in: cae844a0 user: shess tags: trunk
17:52
Lift docListMerge() call out of loadSegmentLeavesInt() for prefix search. Doclists from multiple prefix matches will need a union merge function, which will have to logically happen across a segment before doclists are merged between segments. (CVS 3887) check-in: 7ddb8266 user: shess tags: trunk
2007-04-27
22:02
Break interior-node and leaf-node readers apart in loadSegment(). Previously, the code looped until the block was a leaf node as indicated by a leading NUL. Now the code loops until it finds a block in the range of leaf nodes for this segment, then reads it using LeavesReader. This will make it easier to traverse a range of leaves when doing a prefix search. (CVS 3884) check-in: 9466367d user: shess tags: trunk
21:24
Lift code to traverse interior nodes out of loadSegment(). Refactoring towards prefix searching. (CVS 3882) check-in: 25935db7 user: shess tags: trunk
21:02
Refactor fts2 loadSegmentLeaf() in preparation for prefix-searching. Prefix-searching will want to accumulate data across multiple leaves in the segment, using LeavesReader instead of LeafReader is the first step in that direction. (CVS 3881) check-in: 22ffdae4 user: shess tags: trunk
2007-04-21
16:37
Add the ability to turn the FTS2 module into an amalgamation. (CVS 3864) check-in: 94374654 user: drh tags: trunk
2007-04-19
18:36
Fix bug in fts2 handling of OR queries. When one doclist ends before the other, the code potentially tries to read past the end of the doclist. http://www.sqlite.org/cvstrac/tktview?tn=2309 (CVS 3862) check-in: dfac6082 user: shess tags: trunk
2007-04-09
20:45
Fix crash in delete when existing row has null fields. Previous code assumed that the row had values in all columns, sigh. Fixes bug http://www.sqlite.org/cvstrac/tktview?tn=2289 . (CVS 3833) check-in: 81be7290 user: shess tags: trunk
2007-03-29
18:41
Buffer updates per-transaction rather than per-update. If lots of updates happen within a single transaction, there was a lot of wasted encode/decode overhead due to segment merges. This code buffers updates in memory and writes out larger level-0 segments. It only works when documents are presented in ascending order by docid. Comparing a test set running 100 documents per transaction, the total runtime is cut almost in half. (CVS 3751) check-in: 0229cba6 user: shess tags: trunk
16:30
Don't call ctype functions on hi-bit chars. Some platforms raise assertions when this occurs, and it's almost certainly not the right thing to do in the first place. (CVS 3746) check-in: f6c3abdc user: shess tags: trunk
2007-03-22
00:14
Refactor PLWriter to remove owned buffer. DLCollector (Document List Collector) now handles the case where PLWriter (Position List Writer) needed a local buffer. Change to using the associated DLWriter (Document List Writer) buffer, which reduces the number of memory copies needed in doclist processing, and brings PLWriter operation in line with DLWriter operation. (CVS 3707) check-in: d04fa3a1 user: shess tags: trunk
2007-03-20
23:52
Refactor PLWriter in preparation for buffered-document change. Currently, PLWriter (Position List Writer) creates a locally-owned DataBuffer to write into. This is necessary to support doclist collection during tokenization, where there is no obvious buffer to write output to, but is not necessary for the other users of PLWriter. This change adds a DLCollector (Doc List Collector) structure to handle the tokenization case.    Also fix a potential memory leak in writeZeroSegment(). In case of error from leafWriterStep(), the DataBuffer dl was being leaked. (CVS 3706) check-in: 1b9918e2 user: shess tags: trunk
2007-03-16
18:30
Out-of-memory cleanup in tokenizers. Handle NULL return from malloc/calloc/realloc appropriately, and use sizeof(var) instead of sizeof(type) to make certain that we don't get a mismatch between them as the code rots. (CVS 3693) check-in: fbc53da8 user: shess tags: trunk
2007-02-07
01:01
http://www.sqlite.org/cvstrac/tktview?tn=2219    When creating fts tables in an attached database, the backing tables are created in database 'main'. This change propagates the appropriate database name to the routines which build sql statements.    Note that I propagate the database name and table name separately. I briefly considered just making the table name be "db.table", but it didn't fit so well in the model used to store the table name and other information, and having the db name passed separately seemed a bit more transparent. (CVS 3631) check-in: 283385d2 user: shess tags: trunk
2007-01-19
22:59
http://www.sqlite.org/cvstrac/tktview?tn=2166,35    Calling UPDATE against an fts table in a UTF-16 database inserts corrupted data into the database. The UTF-8 data is being inserted directly. This appears to happen because sqlite3_ value_text() destructively coerces a value to UTF-8, and it's never converted back when updating the table. This works around the problem by rearranging things so that the update happens before the coercion. (CVS 3596) check-in: 4f2ab4b6 user: shess tags: trunk
2006-11-29
23:41
Drop a couple variables which are no longer used anywhere. (CVS 3524) check-in: 08c2cc0e user: shess tags: trunk
05:17
http://www.sqlite.org/cvstrac/tktview?tn=2046    The virtual table interface allows for a cursor to field multiple xFilter() calls. For instance, if a join is done with a virtual table, there could be a call for each row which potentially matches. Unfortunately, fulltextFilter() assumes that it has a fresh cursor, and overwrites a prepared statement and a malloc'ed pointer, resulting in unfinalized statements and a memory leak.    This change hacks the code to manually clean up offending items in fulltextFilter(), emphasis on "hacks", since it's a fragile fix insofar as future additions to fulltext_cursor could continue to have the problem. (CVS 3521) check-in: 18142fdb user: shess tags: trunk
01:02
Delta-encode terms in interior nodes. While experiments have shown that this is of marginal utility when encoding terms resulting from regular English text, it turns out to be very useful when encoding inputs with very large terms. (CVS 3520) check-in: c8151a99 user: shess tags: trunk
2006-11-18
00:12
Store minimal terms in interior nodes. Whenever there's a break between leaf nodes, instead of storing the entire leftmost term of the rightmost child, store only that portion of the leftmost term necessary to distinguish it from the rightmost term of the leftmost child. (CVS 3513) check-in: f6e0b080 user: shess tags: trunk
2006-11-17
21:12
Refactoring groundwork for coming work on interior nodes. Change LeafWriter to use empty data buffer (instead of empty term) to detect an empty block. Code to validate interior nodes. Moderate revisions to leaf-node and doclist validation. Recast leafWriterStep() in terms of LeafWriterStepMerge(). (CVS 3512) check-in: f30771d5 user: shess tags: trunk
2006-11-13
21:09
Delta-encode docids. This is good for around 22% reduction in index size with DL_POSITIONS. It improves performance about 5%-6%. (CVS 3511) check-in: 9b6d413d user: shess tags: trunk
21:00
Require a minimum fanout for interior nodes. This prevents cases where excessively large terms keep the tree from finding a single root. A downside is that this could result in large interior nodes in the presence of large terms, which may be prone to fragmentation, though if the nodes were smaller that would translate into more levels in the tree, which would also have that problem. (CVS 3510) check-in: 64b7e340 user: shess tags: trunk
20:15
Allow backing tables to be missing on dropping fts table. Fixes http://www.sqlite.org/cvstrac/tktview?tn=1992,35 . (CVS 3509) check-in: 9628a61a user: shess tags: trunk
2006-10-31
18:13
Fix a pair of memory leaks. These were turned up by running valgrind memcheck with various 10k doc insert, update, delete, and query tests. (CVS 3497) check-in: 3cd9b64b user: shess tags: trunk
2006-10-26
00:41
Empty queries should get no results. My recent change ( http://www.sqlite.org/cvstrac/chngview?cn=3486 ) broke test fts2a-5.3. This change should make the expected result more obvious. (CVS 3489) check-in: cde383eb user: shess tags: trunk
00:04
Make memset() uses less error-prone. http://www.sqlite.org/cvstrac/tktview?tn=2036,35 describes some cases where we were passing memset() a length which was the sizeof a pointer, rather than the structure pointed to. Instead, wrap this idiom up in CLEAR() and SCRAMBLE() macros. (CVS 3488) check-in: 5878add0 user: shess tags: trunk
2006-10-25
23:22
Remove unreferenced local variable. (CVS 3487) check-in: 2d3b2219 user: shess tags: trunk
21:00
Replace the DocList and DocListReader structures. The new structures distinguish reading from a static buffer from writing to a dynamic buffer. This allows n-way doclist merging, and in-place merging of segment leaf nodes, which together cut segment merge times in half. (CVS 3486) check-in: af5bfb98 user: shess tags: trunk
05:21
Don't store empty segments. When inserting empty strings, the code was writing out a segment made up of a single leaf node containing the \0 header. LeafReader assumed that leaf nodes always contained at least one term, so assertions would fail.    While it would be possible to support reading and merging empty segments, there's no reason to do so. While this change could have been done in writeZeroSegment(), I put it in leafWriterFlush() so that it would work right if segmentMerge() created an empty segment, which could happen with future changes to how deleted documents are handled. (CVS 3484) check-in: fed79bee user: shess tags: trunk
2006-10-13
11:55
Removing debugging printf from the porter stemmer code. Ticket #2016. (CVS 3475) check-in: 7a08c627 user: drh tags: trunk
2006-10-12
23:15
Convert fts2 to store data in a way which allows for much faster updates. Groups of documents form segments which are encoded in a btree layered over a table of blocks, with various tricks to make merges fast. This performs 20x-25x faster than fts1 when loading the Enron corpus, and is only slightly slower for queries. (CVS 3474) check-in: 85272b2f user: shess tags: trunk
2006-10-10
23:22
Fix leaky symbols. With this change, fts1 and fts2 can both be statically linked. (CVS 3472) check-in: 5e8bbb85 user: shess tags: trunk
17:37
Copy fts1/ to fts2/, changing reference from fts1 to fts2. For future reference, the source versions copied were:    README.txt r1.1 fts1.c r1.37 fts1.h r1.2 fts1_hash.c r1.1 fts1_hash.h r1.1 fts1_porter.c r1.1 fts1_tokenizer.h r1.4 fts1_tokenizer1.c r1.6 (CVS 3471) check-in: d0d1e7cd user: shess tags: trunk