Comments(0) I’d like to share technique I came up with when I needed to index a lot of mp3 podcasts located on remote servers. It would have taken weeks if downloading entire files, count was on tens of thousands.
Example is in Python, but any other programming language could be used to perform these steps.
Read more »
Comments(0) We would like to show you how to create full-text index, thus making your database searchable. MySQL has full-text indexing capabilities built in, but if you want really powerful search which scales you’ll have to do it yourself.
This post is based on real experience of optimizing index and search for 30 million documents database.
By employing these optimization techniques we were able to speed up indexing time from 15 seconds per 1000 rows, to 1 second. After optimizing queries and index structure, searches ran 7-10 times faster as well.
Lucene is great and very flexible search engine, but if not properly used it may not perform very well during indexing and search. Especially when number of rows counts to millions.