I want to store the whole data of a webpage in database. Can anyone tell me how i can do that?
Whole data means whole page including all scripts, static data and images.
Please Help..
What is it that you are trying to achieve? From what I can read you want to create a template and that all your pages call apon that for their scripts, etc. This is actually easily achieved with WebMatrix and Razor syntax.
This webpage will guide you through the process of creating a template - without having the need to call about a database continually for your scripts, etc.
I want to store the whole data of a webpage in database. Can anyone tell me how i can do that?
Whole data means whole page including all scripts, static data and images.
Please Help..
What is it that you are trying to achieve? From what I can read you want to create a template and that all your pages call apon that for their scripts, etc. This is actually easily achieved with WebMatrix and Razor syntax.
This webpage will guide you through the process of creating a template - without having the need to call about a database continually for your scripts, etc.
Put this in a file called Helpers.cshtml in App_Code:
@functions {
public static string GetHtmlPage(string url){
WebRequest request = HttpWebRequest.Create(url);
WebResponse response = request.GetResponse();
using (StreamReader sr = new StreamReader(response.GetResponseStream())){
return sr.ReadToEnd();
}
}
}
Then in your page, you can do this:
var db = Database.Open("MyDb)";
var sql = "INSERT INTO table (myField) VALUES (@0);
var page = Helpers.GetHtmlPage("http://www.google.com");
db.Execute(sql, page);
Put this in a file called Helpers.cshtml in App_Code:
@functions {
public static string GetHtmlPage(string url){
WebRequest request = HttpWebRequest.Create(url);
WebResponse response = request.GetResponse();
using (StreamReader sr = new StreamReader(response.GetResponseStream())){
return sr.ReadToEnd();
}
}
}
Then in your page, you can do this:
var db = Database.Open("MyDb)";
var sql = "INSERT INTO table (myField) VALUES (@0);
var page = Helpers.GetHtmlPage("http://www.google.com");
db.Execute(sql, page);
Thank You Mike.
But in my table, i has a colume "webpageStore". It is in varchar with max limit but when i am running Your code it is giving error.
0 : String truncation: max=4000, len=28951
length is greater than my row size.
Ho can now remove this problem?
Mike after fetching page can't we divide it in parts, so that it can be store in database?
Mike thanks for all help. Mike can you tell me a way like now everything in the page is storing but if i only wants page data not html tags, javascripts.
nikunj25
Member
288 Points
160 Posts
Question related to webpage capture and Database???
Jun 23, 2011 06:53 AM|LINK
Hi,
I want to store the whole data of a webpage in database. Can anyone tell me how i can do that?
Whole data means whole page data.
Like in google, i want to store all the data of that page.
Please Help..
a-rad
Member
444 Points
156 Posts
Re: Question related to webpage capture and Database???
Jun 23, 2011 07:04 AM|LINK
What is it that you are trying to achieve? From what I can read you want to create a template and that all your pages call apon that for their scripts, etc. This is actually easily achieved with WebMatrix and Razor syntax.
This webpage will guide you through the process of creating a template - without having the need to call about a database continually for your scripts, etc.
http://www.asp.net/webmatrix/tutorials/3-creating-a-consistent-look
Does this assist you? If not, please elaborate or provide an example and we/I will try to help :)
nikunj25
Member
288 Points
160 Posts
Re: Question related to webpage capture and Database???
Jun 23, 2011 07:07 AM|LINK
No, take google for an instance, i want to store all data present on google page in my database...
How can i do that?
ayyappan.CNN
Participant
870 Points
326 Posts
Re: Question related to webpage capture and Database???
Jun 23, 2011 07:07 AM|LINK
Hey,
I think less possiblity to do that. but you can do one, simple convert whole page to PDF or you can download whole site.
Try this links, http://www.httrack.com/ or http://forum.esoft.in/applications/5489-metaproducts-offline-explorer-enterprise-5-0-2752-a.html
I hope this helps U
Cheers !
HAND IN HAND IND., Asst. Project Director, Admin and IT,Systems, TN,India
Mark post(s) as "Answer" that helped you
nikunj25
Member
288 Points
160 Posts
Re: Question related to webpage capture and Database???
Jun 23, 2011 07:10 AM|LINK
I want to write a code which will do that automatically..
Mikesdotnett...
All-Star
154901 Points
19864 Posts
Moderator
MVP
Re: Question related to webpage capture and Database???
Jun 24, 2011 04:49 AM|LINK
Put this in a file called Helpers.cshtml in App_Code:
@functions { public static string GetHtmlPage(string url){ WebRequest request = HttpWebRequest.Create(url); WebResponse response = request.GetResponse(); using (StreamReader sr = new StreamReader(response.GetResponseStream())){ return sr.ReadToEnd(); } } }Then in your page, you can do this:
var db = Database.Open("MyDb)"; var sql = "INSERT INTO table (myField) VALUES (@0); var page = Helpers.GetHtmlPage("http://www.google.com"); db.Execute(sql, page);Beginning ASP.NET Web Pages with WebMatrix | My Site | Twitter
nikunj25
Member
288 Points
160 Posts
Re: Question related to webpage capture and Database???
Jun 24, 2011 06:06 AM|LINK
Thank You Mike.
But in my table, i has a colume "webpageStore". It is in varchar with max limit but when i am running Your code it is giving error.
0 : String truncation: max=4000, len=28951
length is greater than my row size.
Ho can now remove this problem?
Mike after fetching page can't we divide it in parts, so that it can be store in database?
Please help...
Mikesdotnett...
All-Star
154901 Points
19864 Posts
Moderator
MVP
Re: Question related to webpage capture and Database???
Jun 24, 2011 06:19 AM|LINK
Use the nText data type.
Beginning ASP.NET Web Pages with WebMatrix | My Site | Twitter
nikunj25
Member
288 Points
160 Posts
Re: Question related to webpage capture and Database???
Jun 24, 2011 06:32 AM|LINK
Mike thanks for all help. Mike can you tell me a way like now everything in the page is storing but if i only wants page data not html tags, javascripts.
Then it is posible to do that?
Mikesdotnett...
All-Star
154901 Points
19864 Posts
Moderator
MVP
Re: Question related to webpage capture and Database???
Jun 24, 2011 09:45 AM|LINK
@using System.Text.RegularExpressions; @{ var page = Helpers.GetHtmlPage("http://www.google.com"); page = Regex.Replace(page, @"<script.*?</script>", string.Empty, RegexOptions.Singleline); page = Regex.Replace(page, @"<style.*?</style>", string.Empty,RegexOptions.Singleline); page = Regex.Replace(page, @"<(.|\n)*?>", string.Empty); var db = Database.Open("MyDb)"; var sql = "INSERT INTO table (myField) VALUES (@0); db.Execute(sql, cleaned); }Beginning ASP.NET Web Pages with WebMatrix | My Site | Twitter