Last post Feb 20, 2015 10:42 AM by Aswecan
Feb 19, 2015 10:11 AM|ldoodle|LINK
I'm trying to block this as per the recommendation of an SEO check. I used this article http://forums.asp.net/t/1970637.aspx?I+want+to+block+libwww+perl but it doesn't seem to
What am I doing wrong? Is it case-sensitive (Libwww-perl); do the user agents need to go in a specific order?
Feb 20, 2015 07:37 AM|ldoodle|LINK
Doesn't seem possible because even this doesn't seem to block it:
So I'm blocking ever user agent except bingbot, googlebot and msnbot, and the SEO checker still says libwww-perl is not blocked.
That is assuming robots.txt is the correct way of blocking it, of course!
Feb 20, 2015 10:42 AM|Aswecan|LINK
I think your issue is more related with IIS setting. You can post to IIS forum.
Maybe you can get suggestions at there.