|
|
[Perlfect-search] How to ignore robots meta tag
perlfect-search@perlfect.com perlfect-search@perlfect.com
Thu, 20 Mar 2003 15:39:38 -0600
How can I tell the indexer to ignore the robots meta tag? This is what
I have in each of my pages I wish to index:
meta name="ROBOTS" content="NOINDEX, NOFOLLOW"
I saw in the config file where you can tell it to ignore robots.txt and
did so, but it is still honoring the robots meta tag. I am running the
indexer on the local filesystem of a site which I want to keep out of
public search engines, but I would like site search to be available.
I looked around through the FAQ and archives to find information about
this but came up with nothing, sorry if I missed something.
Thanks,
Rob
|
|