Last post Jun 03, 2014 03:50 AM by Kevin Shen - MSFT
May 30, 2014 11:57 AM|SiKni8|LINK
I have a website, http://www.op.nysed.gov/opsearches.htm, for example where the user selects a **Profession** and enters a **Licensee Name** and clicks on the **Search** button which takes them to a new page to display the result.
I looked at scrapy, arachnode and other web crawlers on the web for this purpose but wasn't too convinced that is the right technology for it.
I was told that we have to crawl those search results from the page. Is it something that can be done?
Can crawler crawl as the user does the search?
I asked a question in SO here: http://stackoverflow.com/questions/23958408/how-to-crawl-external-web-search
Jun 03, 2014 03:50 AM|Kevin Shen - MSFT|LINK
For your problem,it seems that you want to crawl the results from page.
And i am not familar with asp.net crawler,so here are just some links which i wish it can help you:
This response contains a reference to a third party World Wide Web site. Microsoft is providing this information as a convenience to you. Microsoft does not control these sites and has not tested any software or information found on these sites; therefore,
Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. There are inherent dangers in the use of any software found on the Internet, and Microsoft cautions you to make sure that you
completely understand the risk before retrieving any software from the Internet.