[Catalyst] Catalyst and robots.txt

[Bill Moseley]

On Tue, May 16, 2006 at 05:47:30PM +0200, Bernhard Graf wrote:
> On Tuesday 16 May 2006 16:34, Ryan wrote:
> 
> > Anyone have experience with the robots.txt and catalyst, since there
> > is no extension.
> >
> > If I have
> > ------robots.txt-------
> > User-agent: *
> > Disallow: /profile
> >
> > will that tell the robots not to spider urls like
> >
> > http://www.myurl.com/profile?ref=100
> 
> Don't use robots.txt or you'll shoot into your own feet. Some Spambots 
> check for robots.txt to find the interesting pages.

Do you have a reference that supports that claim?

> Better put a <meta> line into the regarding page, e.g.:
> <meta name="robots" content="noindex,nofollow">

If the spambot ignores robots.txt why would it not ignore that?
I'll bet more spiders use robots.txt than use meta tags.
(Considering that meta tags can only be in html pages.)

If the page is not linked anywhere there's no need to place it in
robots.txt.  If it's linked then not using robots.txt doesn't prevent
the bots from finding it.

-- 
Bill Moseley
moseley at hank.org