I'm trying to web scrape a website using HttpWebRequest.
I have confirmed with Fidder, all my headers match and cookies are set.
I am able to login and to get to certain pages along the way while collecting all the viewstates, cookies and other headers.
I need to web scrape information from page (2). This specific information is being calculated by page (1) then using an AJAX Loader transferred to page (2).
When posting directly to page (2) I get an empty page without the calculations that I need (which are being calculated on page (1) probably)
When posting to page (1) I get the following error which I imagine relates to the AJAX on page (1):
179|error|500|The page is performing an async postback but the ScriptManager.SupportsPartialRendering property is set to false. Ensure that the property is set to true during an async postback.|
I need to web scrape information from page (2). This specific information is being calculated by page (1) then using an AJAX Loader transferred to page (2).
Hi,
That is the ajax basic principle. It showing how to transfer data asynchronously. And maybe give your some light.
Please mark the replies as answers if they help or unmark if not.
Feedback to us
I need to manipulate the page some how to think that I'm posting from an AJAX supporting browser. I imagine people have met with this problem in the past. No body knows anything?
I'm sure you got an answer already, but for anyone who finds this thread in the future, I added a User-Agent header to make ASP.NET think the request was coming from an AJAX supporting browser. I just used an IE 7 header:
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
Warren_Nicku...
Member
153 Points
71 Posts
HttpWebRequest to a page with ScriptManager
Jul 19, 2010 02:20 PM|LINK
I'm trying to web scrape a website using HttpWebRequest.
I have confirmed with Fidder, all my headers match and cookies are set.
I am able to login and to get to certain pages along the way while collecting all the viewstates, cookies and other headers.
I need to web scrape information from page (2). This specific information is being calculated by page (1) then using an AJAX Loader transferred to page (2).
When posting directly to page (2) I get an empty page without the calculations that I need (which are being calculated on page (1) probably)
When posting to page (1) I get the following error which I imagine relates to the AJAX on page (1):
179|error|500|The page is performing an async postback but the ScriptManager.SupportsPartialRendering property is set to false. Ensure that the property is set to true during an async postback.|
Can anybody help me?
Nick
Warren_Nicku...
Member
153 Points
71 Posts
Re: HttpWebRequest to a page with ScriptManager
Jul 19, 2010 10:05 PM|LINK
Anybody?
chetan.sarod...
All-Star
66579 Points
11270 Posts
Re: HttpWebRequest to a page with ScriptManager
Jul 20, 2010 03:05 AM|LINK
ScriptManager.SupportsPartialRendering property tells if the curren browser supports the partial rendering feature.
http://schupka.blogspot.com/2007/08/webtest-recorded-using-fiddler-fails.html
Senior Software Engineer,
Approva Systems Pvt Ltd, Pune, India.
Warren_Nicku...
Member
153 Points
71 Posts
Re: HttpWebRequest to a page with ScriptManager
Jul 20, 2010 07:27 PM|LINK
But as I'm trying to scrape the page that property is set to false since I'm not using a real browser.
Is there a way around it? I need to make page (1) process my post request.
Song-Tian - ...
All-Star
43715 Points
4304 Posts
Microsoft
Re: HttpWebRequest to a page with ScriptManager
Jul 21, 2010 06:10 AM|LINK
Hi,
You also could use the basic ajax to achieve the purpose. For the details, please refer to:http://www.w3schools.com/ajax/ajax_aspphp.asp.
Feedback to us
Develop and promote your apps in Windows Store
Warren_Nicku...
Member
153 Points
71 Posts
Re: HttpWebRequest to a page with ScriptManager
Jul 22, 2010 07:51 AM|LINK
I don't understand how this helps me solve the problem?
Song-Tian - ...
All-Star
43715 Points
4304 Posts
Microsoft
Re: HttpWebRequest to a page with ScriptManager
Jul 22, 2010 08:19 AM|LINK
Hi,
That is the ajax basic principle. It showing how to transfer data asynchronously. And maybe give your some light.
Feedback to us
Develop and promote your apps in Windows Store
Warren_Nicku...
Member
153 Points
71 Posts
Re: HttpWebRequest to a page with ScriptManager
Jul 23, 2010 02:03 PM|LINK
I need to manipulate the page some how to think that I'm posting from an AJAX supporting browser. I imagine people have met with this problem in the past. No body knows anything?
Free Spirite...
Member
2 Points
1 Post
Re: HttpWebRequest to a page with ScriptManager
Feb 27, 2012 11:07 PM|LINK
I'm sure you got an answer already, but for anyone who finds this thread in the future, I added a User-Agent header to make ASP.NET think the request was coming from an AJAX supporting browser. I just used an IE 7 header:
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)