[Catalyst] Catalyst and robots.txt

Ryan god at detz.net
Tue May 16 16:34:32 CEST 2006


Anyone have experience with the robots.txt and catalyst, since there is
no extension.

If I have
------robots.txt-------
User-agent: *
Disallow: /profile

will that tell the robots not to spider urls like

http://www.myurl.com/profile?ref=100




More information about the Catalyst mailing list