[Catalyst] Catalyst and robots.txt

Bernhard Graf catalyst at augensalat.de
Tue May 16 17:47:30 CEST 2006

On Tuesday 16 May 2006 16:34, Ryan wrote:

> Anyone have experience with the robots.txt and catalyst, since there
> is no extension.
> If I have
> ------robots.txt-------
> User-agent: *
> Disallow: /profile
> will that tell the robots not to spider urls like
> http://www.myurl.com/profile?ref=100

Don't use robots.txt or you'll shoot into your own feet. Some Spambots 
check for robots.txt to find the interesting pages.

Better put a <meta> line into the regarding page, e.g.:
<meta name="robots" content="noindex,nofollow">

Bernhard Graf

More information about the Catalyst mailing list