Hi All,

I had a perlfect working.  Then moved the site from predomintly .shtml  
pages to .php pages.  Now because they are php pages and I don't really  
want all the "code" to be indexed(highlight matches etc) I need to set  
$HTTP_START_URL Now I am having two problems, potentially related, with  
indexing as follows:

1. Command line indexing causes a permissions error
2. index_form.html solves permissions problem but index zero (0) files
3. modify $START_URL to further up the tree, index html files but not  
PHP

Can any one help?





The long winded version of the problems follows apologies for the  
length but I am at a loss wht to do.

I have tried to index via the following methods
   Index from the command line:  /usr/bin/perl indexer.pl
   Using index_form.html  (action modified to suit local perlfect  
install)

If I use the command line to index, when I search, I get the following  
error regardles if I set the $START_URL (causing it to retrieve dynamic  
pages) or unset causing it to index via the filesystem:

Software error:
Cannot open  
/usr/data/slash/Curtin/units/ec100/cgi_bin/perlfect/search/data/ 
inv_index: Permission denied at  
/usr/apache+ssl/htdocs/www.computing.edu.au/units/ec100/cgi_bin/ 
perlfect/search/search.cgi line 76.



If I index via index_form.html

I do not get the error. I get the following output.

Loading https://www.computing.edu.au/robots.txt...
Error: Couldn't get 'https://www.computing.edu.au/robots.txt': response  
code 501
[Wed Feb 25 22:47:18 2004] indexer.cgi: Not using any robots.txt.
Warning: cannot remove '/usr/data/slash/Curtin/units/ec100/' from  
'https://www.computing.edu.au/units/ec100/'
[Wed Feb 25 22:47:18 2004] indexer.cgi: Use of uninitialized value in  
pattern match (m//) at ./indexer.cgi line 568.
[Wed Feb 25 22:47:18 2004] indexer.cgi: Use of uninitialized value in  
concatenation (.) or string at ./indexer.cgi line 569.
Error: Couldn't get 'https://www.computing.edu.au/units/ec100/':  
response code 500

Crawler finished: indexed 0 files, 0 terms (0 different terms).
etc....

I can now use search_form.html successfully, no error obviously finding  
nothing.  Now I have not changed conf.pl, just the method of indexing.   
When I look in the $PERLFECT_HOME/data I observe all files are 755 and  
owner/group nobody.

Try command line again, change permission and owner to match, still get  
"Software error:...." ummm strange, oh well indexing via cgi, although  
less secure looks more promising.


Umm "index 0 files" and "response code 500".  I suspect this is me  
confusing $DOCUMENT_ROOT, $BASE_URL and $START_URL.   I call the techos  
to have a look at the server logs and they mention something about  
robots.txt (umm sort guessed that). I ask if they had anything  
installed to block spiders/crawler etc, none.

So I comment out the $ROBOT_AGENT (ignoring the comment and "really  
know what you are doing") and get the following error:

Warning: cannot remove '/usr/data/slash/Curtin/units/ec100/' from  
'https://www.computing.edu.au/units/ec100/'
[Wed Feb 25 23:03:47 2004] indexer.cgi: Use of uninitialized value in  
pattern match (m//) at ./indexer.cgi line 568.
[Wed Feb 25 23:03:47 2004] indexer.cgi: Use of uninitialized value in  
concatenation (.) or string at ./indexer.cgi line 569.
Error: Couldn't get 'https://www.computing.edu.au/units/ec100/':  
response code 501

Okay, lets have a play around with the $DOCUMENT_ROOT, $BASE_URL,  
$START_URL.  Now I look after all web pages under the main site.  Our  
home page is:

https://www.computing.edu.au/

I maintain

https://www.computing.edu/units/ec100/

I only want to index pages in my area.  Now after various combinations  
the only one that will index pages is setting the $START_URL to:

http://www.computing.edu.au/

The values of the other variables does not seem to affect the indexer,  
and also notice the protocol has changed from https:// to http://

This still only index html files (regardless of setting in conf.pl). I  
have tried making my start page a html file (rather than a php) still  
no luck.

Getting rather desperate now suspect it might be a missing library  
(response code 501  -- service not provided).  Checking back on the  
perlfect website, and downloading a useful little utility wpm.pl (Where  
is Perl Module) it looks like the LWP::libwww-perl is not installed.   
Did I mention that I do not have root access.  So I install the module  
in $PERLFECT_HOME and add the following line to search.cgi, indexer.cgi  
and indexer_web.cgi  (I also modified the file to require .cgi and not  
.pl file).

use lib 'lib/perl5/site_perl/5.8.1';
print "\@INC is @INC\n";

The path looks okay but still no luck indexing dynamic pages.

Can anyone help?

Michael.
--