Perlfect Solutions
 

[Perlfect-search] need support: Response code 500 with HTTP indexing

Adrian Gudas perlfect-search@perlfect.com
Tue, 1 Apr 2003 13:38:44 -0500
Greetings,

Thanks for what seems to be an excellent piece of software, but I am having
trouble getting the HTTP indexing to work. The local directory dig works
fine, but when I run the indexer.pl script, I get the following:

jailshell-2.05$ perl indexer.pl
Using DB_File...
Checking for old temp files...
Building string of special characters...
Loading 'no index' regular expressions:
    - /home/greenpar/public_html/cgi-bin/*
Loading stopwords...371 stopwords loaded.
Starting crawler...
Note: I will not visit more than $HTTP_MAX_PAGES=100 pages.
Loading http://64.141.43.247/robots.txt...
Error: Couldn't get 'http://64.141.43.247/robots.txt': response code 500
Not using any robots.txt.
Error: Couldn't get 'http://localhost/': response code 500

Crawler finished: indexed 0 files, 0 terms (0 different terms).
Ignored 0 files because of conf/no_index.txt
Ignored 0 files because of robots.txt
<snip>


It seems that the Apache logs are not reporting any access, even error 500.
I thought it was a module problem, but again, the local directory indexing
works. I've searched the mailing list with no luck.

I feel stupid asking you directly, but I'm completely stumped. Thanks in
advance.

Adrian Gudas
adrian@fryth.com