I have some problems, I must screen scrape a web page that is an aspx-site where the intresting data is the text in a gridview. I have no problem to get the text from the first page in the gridview.
But how do I get the text from the second page, the third page etc.
The problem is that every page in the gridview has not a specific URL, instead it is a javascript that is executing when I click on the paging linkbutton.
For exempel the javascript for a paging javascript is:
javascript:__doPostBack('ctl00$ctl01$DefaultSiteContentPlaceHolder1$Col1$ucNavBar$rptButtons$ctl02$btnPage','')
How do I simulate this?
This is the code I use to get a webpage for a specific URL:
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(strUrl);
request.Accept = "text/html, application/xhtml+xml, */*";
request.UserAgent = "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)";
request.KeepAlive = true;
using (StreamReader streamReader = new StreamReader(request.GetResponse().GetResponseStream(), Encoding.UTF8, true))
{
return streamReader.ReadToEnd();
}
I cant find information on this, maybe this is impossible?
To make my question more simple; How to simulate from WebClient or HttpWebRequest a click on a button, eg simulate this javascript and get the resulting webpage HTML:
I have found a solution. Used the program Fiddler to be able to see exactly what happens when I clicked on the button. Then I tried to make a POST that looked so simular as the real call as possible.
Marked as answer by Jiggr on Apr 09, 2012 06:18 PM
Jiggr
Member
35 Points
51 Posts
Screenscraping a gridview with paging
Apr 08, 2012 06:48 PM|LINK
I have some problems, I must screen scrape a web page that is an aspx-site where the intresting data is the text in a gridview. I have no problem to get the text from the first page in the gridview.
But how do I get the text from the second page, the third page etc.
The problem is that every page in the gridview has not a specific URL, instead it is a javascript that is executing when I click on the paging linkbutton.
For exempel the javascript for a paging javascript is:
javascript:__doPostBack('ctl00$ctl01$DefaultSiteContentPlaceHolder1$Col1$ucNavBar$rptButtons$ctl02$btnPage','')
How do I simulate this?
This is the code I use to get a webpage for a specific URL:
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(strUrl); request.Accept = "text/html, application/xhtml+xml, */*"; request.UserAgent = "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"; request.KeepAlive = true; using (StreamReader streamReader = new StreamReader(request.GetResponse().GetResponseStream(), Encoding.UTF8, true)) { return streamReader.ReadToEnd(); }Thanks in advance!
Jiggr
Member
35 Points
51 Posts
Re: Screenscraping a gridview with paging
Apr 09, 2012 12:03 PM|LINK
No answer :(
I cant find information on this, maybe this is impossible?
To make my question more simple; How to simulate from WebClient or HttpWebRequest a click on a button, eg simulate this javascript and get the resulting webpage HTML:
javascript:__doPostBack('ctl00$ctl01$DefaultSiteContentPlaceHolder1$Col1$ucNavBar$rptButtons$ctl00$btnPage','')
Jiggr
Member
35 Points
51 Posts
Re: Screenscraping a gridview with paging
Apr 09, 2012 06:18 PM|LINK
I have found a solution. Used the program Fiddler to be able to see exactly what happens when I clicked on the button. Then I tried to make a POST that looked so simular as the real call as possible.