Hi, I have built a DNN Module for my site that does flight planning. My users put in some info and then my module goes out and gets information from the web and brings it back into one report and performs some logic on it. One of the key pieces of information
my module must get is NOTAMs (Notices to Airmen) for two airfields (whatever the user specifies). So far, I've been accomplishing this by screen scraping (with permission) of another website. I'm not sure how they get the information and I don't think they'd
tell me. What I really want to do is get the information online straight from the source...the FAA. They have online NOTAM access at https://www.notams.faa.gov but I don't know how to get the data programattically. On their website the user puts in the field
ICAO (for example, "KDFW" for Dallas-Fort Worth) and it comes up with the notams. But the URL for the site doesn't change. Does anybody know how to programmatically access a site like this? Is there a way I can programmatically go to that site, put "DFW" in
the input box, programmatically click the "Go get the NOTAMS" button, and then programmatically return the result? I've tried contacting the FAA about getting FTP access or something but have had no luck. I know other sites get the info...just wondering how
to get it myself. Anybody have any ideas? Thanks a ton for any help.
This is actually pretty easy to do thanks to .Net. What you need to do is the following: 1) Get the HTML field name into which you type in the ICAO 2) Get the URL to which this information is posted (i.e. the action attribute of the form) You can then use the
WebRequest or WebClient object to Post the value directly to the URL, capture the result in a string variable and then process/display it to the user. Google WebRequest and you should find plenty of source code for reference. Nik
Couple questions... 1. When I view the source code of the webpage I'm trying to automate...it's very small. It contains the following (which is most of the code): Is that the ICAO Field name you were referring to? Header or Main? I would guess the field I'm
trying to automate would be the Main field. And the action attribute, is that the SRC statement? 2. This site is an HTTPS site and comes up with the security question before the browser will go to the site. Does that throw a wrench in the WebRequest? I'm wading
through articles online...most seem to be written in C and I'm using VB but I think I should be able to figure it out.
Here's what I've got coded to get the information from the site. It's giving me a critical error, "object not set to an instance of an object" at the = "POST"
Dim lcUrl As String = "https://www.thesitetoautomate.com" Dim loHttp As Net.WebRequest loHttp.Create(lcUrl) '// *** Send any POST data Dim lcPostData As String = "Main=" + HttpUtility.UrlEncode("KMGM") loHttp.Method = "POST" 'instance of an object error
generated right here KMGM is a valid ICAO. Maybe I'm getting an error because I'm not putting anything in for the Header field? I'm not sure what would go there...
Even if I take all the Post info out and just try to get the HTML...I get the same error. But it works on other websites. The request is returning the error. Not sure if it's because it's an HTTPS site or what. I read in some posts where they were having problems
accessing SSL sites and they assigned "utf-8" to the encoding property of the streamreader but that doesn't come into play until after the webrequest has been made. So there is something particular with the website. Something about https://www.notams.faa.gov
- but I'm not sure what. When I go to the site manually it gives me the "there is a problem with the site's security certificate" and asks me if I want to proceed. When I proceed the page comes up. I wonder if this message is somehow blocking my request.
None
0 Points
484 Posts
Screen Scraping Challenge...NOTAMs
Jul 09, 2004 12:02 AM|lilryno|LINK
Member
46 Points
1903 Posts
Re: Screen Scraping Challenge...NOTAMs
Aug 09, 2004 09:00 PM|cniknet|LINK
Speerio, Inc.
[DotNetNuke and ASP.Net solutions here]
None
0 Points
484 Posts
Re: Screen Scraping Challenge...NOTAMs
Aug 09, 2004 09:02 PM|lilryno|LINK
None
0 Points
484 Posts
Re: Screen Scraping Challenge...NOTAMs
Aug 09, 2004 09:57 PM|lilryno|LINK
None
0 Points
484 Posts
Re: Screen Scraping Challenge...NOTAMs
Aug 09, 2004 11:53 PM|lilryno|LINK
Dim lcUrl As String = "https://www.thesitetoautomate.com" Dim loHttp As Net.WebRequest loHttp.Create(lcUrl) '// *** Send any POST data Dim lcPostData As String = "Main=" + HttpUtility.UrlEncode("KMGM") loHttp.Method = "POST" 'instance of an object error generated right hereKMGM is a valid ICAO. Maybe I'm getting an error because I'm not putting anything in for the Header field? I'm not sure what would go there...None
0 Points
484 Posts
Re: Screen Scraping Challenge...NOTAMs
Aug 09, 2004 11:58 PM|lilryno|LINK
None
0 Points
484 Posts
Re: Screen Scraping Challenge...NOTAMs
Aug 10, 2004 12:21 AM|lilryno|LINK