Perlfect Solutions
 

[Perlfect-search] How to ignore robots meta tag

perlfect-search@perlfect.com perlfect-search@perlfect.com
Thu, 20 Mar 2003 15:39:38 -0600
How can I tell the indexer to ignore the robots meta tag?  This is what
I have in each of my pages I wish to index:

meta name="ROBOTS" content="NOINDEX, NOFOLLOW"

I saw in the config file where you can tell it to ignore robots.txt and
did so, but it is still honoring the robots meta tag.  I am running the
indexer on the local filesystem of a site which I want to keep out of
public search engines, but I would like site search to be available.

I looked around through the FAQ and archives to find information about
this but came up with nothing, sorry if I missed something.

Thanks,
Rob