[Catalyst] Catalyst and robots.txt
catalyst at augensalat.de
Tue May 16 17:47:30 CEST 2006
On Tuesday 16 May 2006 16:34, Ryan wrote:
> Anyone have experience with the robots.txt and catalyst, since there
> is no extension.
> If I have
> User-agent: *
> Disallow: /profile
> will that tell the robots not to spider urls like
Don't use robots.txt or you'll shoot into your own feet. Some Spambots
check for robots.txt to find the interesting pages.
Better put a <meta> line into the regarding page, e.g.:
<meta name="robots" content="noindex,nofollow">
More information about the Catalyst