[Catalyst] Catalyst and robots.txt

[Bernhard Graf]

On Tuesday 16 May 2006 16:34, Ryan wrote:

> Anyone have experience with the robots.txt and catalyst, since there
> is no extension.
>
> If I have
> ------robots.txt-------
> User-agent: *
> Disallow: /profile
>
> will that tell the robots not to spider urls like
>
> http://www.myurl.com/profile?ref=100

Don't use robots.txt or you'll shoot into your own feet. Some Spambots 
check for robots.txt to find the interesting pages.

Better put a <meta> line into the regarding page, e.g.:
<meta name="robots" content="noindex,nofollow">

-- 
Bernhard Graf