Last post Jul 13, 2009 01:06 PM by shapper
Jul 13, 2009 01:40 AM|shapper|LINK
I am trying to create a Robots.txt file for a MVC web site.
How should I create this file? Does anyone has an example?
Shouldn't I block all paths that go to required authenticated actions? For example: CMS.
Jul 13, 2009 07:01 AM|rtpHarry|LINK
You dont need to block hidden paths and you dont really need to block paths that need passwords but it would be cleaner if you did.
You just need to make a list of any pages / directories that you dont want indexing. Put each of them on a separate disallow: line
When you are finished with the site you should go to
Generate your xml site maps (I usually save out the xml, ror and txt formats but the second two are a bit redundant now).
After that go back to your robots.txt and put the full path including domain to your sitemap like:
Jul 13, 2009 12:05 PM|shapper|LINK
For sitemap I have a controller and an action the renders the sitemap on XML.
That way everytime I make a create or delete, for example a product, the sitemap is updated.
Then on Google I place the path of that action: MyDomain.com/Application/Sitemap ...
And that's it.
I will check the Robots.
Jul 13, 2009 01:06 PM|shapper|LINK
I just added the following:
Just as a start. Should I add the paths Contents/, Scripts/ and so on to Disallow?
I suppose this does not make any sense in a MVC web site because of routing.
When the crawler tries to access something out of the routing it goes to the 404 error page.