Perlfect Solutions

[Perlfect-search] Problems with 22000 plus file crawl with large file sizes

Daniel Naber
Sat, 8 Nov 2003 13:41:24 +0100
On Saturday 08 November 2003 10:49, James Garrett wrote:

> Setting $LOW_MEMORY_INDEX =1; is just so painfully slow. Found reference
> to $FLUSH_FREQUENCY =100;  but couldn't fig how to patch ./
> and don't really know if that will help at all. Also am afraid temp
> files could grow hugh. Such a thing as a 2gig max file size in Linux?
> Perlfect ?

In a nutshell, you will probably want to use a different search engine that 
scales better. Try htdig or Lucene (Lucene is only a backend and requires 
Java programming). The $FLUSH_FREQUENCY patch probably won't help enough, 
the original posting talks about a 33% increase in indexing speed.