Perlfect Solutions
 

[Perlfect-search] Scalability

DaveP davep@hmgcc.gov.uk
Mon, 05 Feb 2001 09:17:26 +0000
Daniel Naber wrote:
> 
> On 2001-02-01 15:49, you wrote:
> 
> > I've been experimenting with Perlfect Search on a subset of our
> > Intranet, with excellent results. However I've now downloaded the latest
> > CVS version (in the hope of clearing some minor issues), and I see there
> > is a hard-coded limit of 65534 documents now enforced.
> 
> You could remove the limit (that's easy), but I doubt that Perlfect Search
> is fast enough for so many documents. Did you try >50,000 documents
> already? If that worked, I will mail you how to remove the limit.

Sadly you are right. I left it indexing the server on Friday evening and
when I returned Monday morning it was still running, so I had to
manually terminate it.

It appears that the bottleneck is in the database update, since the
process was using very little CPU and doing a *lot* of disk I/O. I
suppose the last chance is to use a DB module which is better able to
cope with continual updates than Berkeley hash?

Dave