Last post Feb 23, 2020 06:44 AM by Sean Fang
Feb 19, 2020 06:37 PM|icm63|LINK
My source: https://www.wsj.com/market-data/stocks/marketsdiary
I am using asp net 2 - Forms
I can use aspnet 3.5
Note: webrequest only scrapes the source code string for a web page, and it is before rendering.
Feb 20, 2020 12:44 PM|Sean Fang|LINK
You can consider using OpenQA.Selenium.Chrome, (automates browsers) to scrape dynamic content from page and HtmlAgilityPack to parse the element.
Remember to set "headless" argument of the ChromeOptions so that the chrome will run without GUI.
More details, you can refer to below C# code (under .NET Framework 4.0 but should work for 3.5):
<form id="form1" runat="server">
<asp:Button ID="Btn1" runat="server" Text="Click" OnClick="Btn1_Click" />
<div id="scrapeContent" runat="server"></div>
protected void Page_Load(object sender, EventArgs e)
private void getData()
ChromeOptions options = new ChromeOptions();
ChromeDriver driver = new ChromeDriver(options);
HtmlDocument doc = new HtmlDocument();
var tables = doc.DocumentNode.SelectNodes("//table");
foreach (HtmlNode x in tables)
scrapeContent.InnerHtml += "<table>" + x.InnerHtml + "</table>";
protected void Btn1_Click(object sender, EventArgs e)
Hope this can help you!
Feb 20, 2020 06:39 PM|icm63|LINK
Installing , questions on this : OpenQA.Selenium.Chrome
Which one do I install from NuGet in MSVS 2015
Click link to see formats from NuGet
And how to I import the correct one ?
Or what do I download from here :
Please give details ?
Note: Using .net 4.6, vb.net, web forms, win64
Feb 23, 2020 06:44 AM|Sean Fang|LINK
I recommend you installing the package from nuget since it is the easiest way.
Based on the picture you provided, the practical version is the first one, Selenium.WebDriver (18.5M
downloads, v3.141.0), which is suitable for .Net framework 3.5, 4.0, 4.5 without dependency required and .NetStandard 2.0 with a requirement that Newtonsoft.Json should higher than/equal to 10.0.3.
The screenshot of feasible package :
After download completed, you only need to use statement "using OpenQA.Selenium.Chrome;" to import the package.