[DBD-SQLite] added support for FTS3

[Dami Laurent (PJ)]

Hi Stefan,

Thanks for the feedback. 

>-----Message d'origine-----
>De : Stefan Evert [mailto:stefanML at collocations.de] 
>Envoyé : mardi, 13. juillet 2010 15:19
>À : DBD::SQLite Mailing List
>Cc : Dami Laurent (PJ)
>Objet : Re: [DBD-SQLite] added support for FTS3
>
>
>> For info : I just committed a number of additions for better 
>support of FTS3 fulltext searches within DBD::SQLite; in 
>particular, adding hooks for tokenizers written in Perl, 
>adding documentation and tests.
>
>That's great!  I'm very interested in playing around with FTS3 
>from Perl -- thanks for the additions.
>
>However, a SVN checkout fails "make test" miserably. After 
>some aimless debugging, I believe this is due to the fact that 
>perl_tokenizer_Open() in dbdimp.c calls newSVpvn_flags() 
>without making sure that the function is properly declared.
>
>Adding
>
>	#define NEED_newSVpvn_flags
>
>at the top of
>
>	dbdimp.c
>
>solves the problem for me, and almost all tests pass.
>

OK, I was not aware of that (probably because I only tested under perl 5.12.1 win32 where this #define is apparently not needed).
I just committed your suggested addition.


>Best,
>Stefan
>
>
>PS: This is where the tests still fail --
>
>> t/rt_44891_strings_look_like_numbers.t ... # requires 
>sqlite3 3.6.23.1 executable for extra tests
>> t/rt_44891_strings_look_like_numbers.t ... Failed 681/829 subtests 
>
>Lots of failures of the form "got: +1.0 expected: +1.0", and 
>then "Abort trap" after 148 subtests.
>

Most probably this has nothing to do with the FTS3 extension. I didn't have that problem because I don't have the sqlite3 executable and therefore those tests are skipped. I'll leave it to other authors to fix.


>PPS: It would be nice if there were an easy way to test FTS3 
>support in DBD::SQLite without having to install unreleased 
>modules (tokenizer and unaccenter) by hand. Will these be 
>available in regular CPAN soon, so one can just do "install 
>Search::Tokenizer"? Or could the tests use a standard 
>tokenizer module instead?
>

Rolling your own tokenizer is just a few lines, as shown in the doc, or in t/43_fts3.c.
But anyway, Search::Tokenizer and Text::Transliterator::Unaccent were just published to CPAN this morning, they should hit your mirror quite soon.

Cheers, Laurent D.