Last post Mar 24, 2010 02:11 PM by jprochazka
Mar 24, 2010 10:06 AM|timparis|LINK
I have some calculators that we allow clients to use via an Iframe. I use Request.UrlReferrer to check the clients url against a table of allowed urls. This is working fine. However, the secure calculator pages are coming up in search engine results. When
a search engine user clicks on the link we are denying access because they haven't come from one of our client websites. I'd like to redirect the search engine users to another page on our site.
Does asp.net provide a way to identify if the referring url is a search engine.
Thanks. All help is appreciated.
Mar 24, 2010 10:28 AM|jprochazka|LINK
First off try the following to try and keep pages you do not want in the search engines out of them.
I would start by denying access to the page by search engines using a robots.txt file that should stop search engines that take into consideration the robots.txt file which is most of the big players. To do so take a look at the following page:
Secondly you can try to use the user agent to figure out if it is a robot or not. Of course there are some engines that change their user agent I am sure but nothing is really fool proof when it comes to identifying anything on the Internet.
You can use the following code to try and retieve the visitors user agent.
If BadReferer(Request.UrlReferrer.ToString()) = True Then
private function BadReferer(Byval Referance as String)
Your checking logic...
Mar 24, 2010 12:31 PM|timparis|LINK
Hi, thanks for the all the information. I'm ok with the search engines finding the pages. It's driving traffic to ous sites. I'll give HTTP_USER_AGENT a try to see if that will identify users coming from search engine sites. I want to just "channel" users
away from the iframe pages to pages that we have on our main site.
Mar 24, 2010 01:11 PM|jprochazka|LINK
I think I may have confused you a bit with my post.
HTTP_USER_AGENT will tell you the User Agent or type of browser or web spider is accessing your page.
You will want to use Request.UrlReferrer to get the URL of the site the visitor clicked on to get to your page. A little better example of what to do follows. It is something off the top of my head but should hopefully point you in the right direction.
' Get the referring URL that the visitor followed.
Dim referrer as String = Request.UrlReferrer.ToString()
' Run the function FromSearchEngine()
' If FromSearchEngine() returns True then we will forward them
' to a difrent page then the one they are trying to directly link to.
If FromSearchEngine(referrer) = True Then
' Check that the user was or was not
' referred here via a seach engine link.
Public Function FromSearchEngine(Byval referrer As String) as Boolean
Dim isFromEngine As Boolean = False
' I am sure there are better ways to do this...
' I am just going to look for google.com in the referring URL.
If (referrer.Index.Of("google.com")) <> -1 Then
isFromEngine = True
' Return the result.
Mar 24, 2010 01:46 PM|timparis|LINK
Thanks. The code you provided is how I thought I might have to approach this. I was hoping to avoid keeping a list of search engine providers to match refering urls against, bu the list shouldn't be that long.
Thanks, appreciate the help.
Mar 24, 2010 02:11 PM|jprochazka|LINK
Maybe turn the tables and check if they are referred from your page and redirect anyone else instead of checking if they came from a search engine.
Just a thought.