[Perlfect-search] Problems with 22000 plus file crawl with large file sizes
Daniel Naber email@example.com
Sat, 8 Nov 2003 13:41:24 +0100
On Saturday 08 November 2003 10:49, James Garrett wrote:
> Setting $LOW_MEMORY_INDEX =1; is just so painfully slow. Found reference
> to $FLUSH_FREQUENCY =100; but couldn't fig how to patch ./indexer.pl
> and don't really know if that will help at all. Also am afraid temp
> files could grow hugh. Such a thing as a 2gig max file size in Linux?
> Perlfect ?
In a nutshell, you will probably want to use a different search engine that
scales better. Try htdig or Lucene (Lucene is only a backend and requires
Java programming). The $FLUSH_FREQUENCY patch probably won't help enough,
the original posting talks about a 33% increase in indexing speed.