Question related to webpage capture and Database???

Hi,

I want to store the whole data of a webpage in database. Can anyone tell me how i can do that?

Whole data means whole page data.

Like in google, i want to store all the data of that page.

Please Help..

Re: Question related to webpage capture and Database???

nikunj25

Hi,

I want to store the whole data of a webpage in database. Can anyone tell me how i can do that?

Whole data means whole page including all scripts, static data and images.

Please Help..

What is it that you are trying to achieve? From what I can read you want to create a template and that all your pages call apon that for their scripts, etc. This is actually easily achieved with WebMatrix and Razor syntax.

This webpage will guide you through the process of creating a template - without having the need to call about a database continually for your scripts, etc.

http://www.asp.net/webmatrix/tutorials/3-creating-a-consistent-look

Does this assist you? If not, please elaborate or provide an example and we/I will try to help :)

Re: Question related to webpage capture and Database???

a-rad

nikunj25

Hi,

I want to store the whole data of a webpage in database. Can anyone tell me how i can do that?

Whole data means whole page including all scripts, static data and images.

Please Help..

What is it that you are trying to achieve? From what I can read you want to create a template and that all your pages call apon that for their scripts, etc. This is actually easily achieved with WebMatrix and Razor syntax.

This webpage will guide you through the process of creating a template - without having the need to call about a database continually for your scripts, etc.

http://www.asp.net/webmatrix/tutorials/3-creating-a-consistent-look

Does this assist you? If not, please elaborate or provide an example and we/I will try to help :)

No, take google for an instance, i want to store all data present on google page in my database...

How can i do that?

Re: Question related to webpage capture and Database???

Hey,

I think less possiblity to do that. but you can do one, simple convert whole page to PDF or you can download whole site.

Try this links, http://www.httrack.com/ or http://forum.esoft.in/applications/5489-metaproducts-offline-explorer-enterprise-5-0-2752-a.html

I hope this helps U

Cheers !

Re: Question related to webpage capture and Database???

ayyappan.CNN

Hey,

I think less possiblity to do that. but you can do one, simple convert whole page to PDF or you can download whole site.

Try this links, http://www.httrack.com/ or http://forum.esoft.in/applications/5489-metaproducts-offline-explorer-enterprise-5-0-2752-a.html

I hope this helps U

Cheers !

I want to write a code which will do that automatically..

Re: Question related to webpage capture and Database???

Put this in a file called Helpers.cshtml in App_Code:

@functions {
    public static string GetHtmlPage(string url){
    
        WebRequest request = HttpWebRequest.Create(url);
        WebResponse response = request.GetResponse();
        using (StreamReader sr = new StreamReader(response.GetResponseStream())){
            return sr.ReadToEnd();
        }
    }
}

Then in your page, you can do this:

var db = Database.Open("MyDb)";
var sql = "INSERT INTO table (myField) VALUES (@0);
var page = Helpers.GetHtmlPage("http://www.google.com");
db.Execute(sql, page);

Re: Question related to webpage capture and Database???

Mikesdotnetting

Put this in a file called Helpers.cshtml in App_Code:

@functions {
    public static string GetHtmlPage(string url){
    
        WebRequest request = HttpWebRequest.Create(url);
        WebResponse response = request.GetResponse();
        using (StreamReader sr = new StreamReader(response.GetResponseStream())){
            return sr.ReadToEnd();
        }
    }
}

Then in your page, you can do this:

var db = Database.Open("MyDb)";
var sql = "INSERT INTO table (myField) VALUES (@0);
var page = Helpers.GetHtmlPage("http://www.google.com");
db.Execute(sql, page);

Thank You Mike.

But in my table, i has a colume "webpageStore". It is in varchar with max limit but when i am running Your code it is giving error.

0 : String truncation: max=4000, len=28951

length is greater than my row size.

Ho can now remove this problem?

Mike after fetching page can't we divide it in parts, so that it can be store in database?

Please help...

Re: Question related to webpage capture and Database???

Use the nText data type.

Re: Question related to webpage capture and Database???

Mikesdotnetting

Use the nText data type.

Mike thanks for all help. Mike can you tell me a way like now everything in the page is storing but if i only wants page data not html tags, javascripts.

Then it is posible to do that?

Re: Question related to webpage capture and Database???

@using System.Text.RegularExpressions;
@{
    var page = Helpers.GetHtmlPage("http://www.google.com");
    page = Regex.Replace(page, @"<script.*?</script>", string.Empty, RegexOptions.Singleline);
    page = Regex.Replace(page, @"<style.*?</style>", string.Empty,RegexOptions.Singleline);
    page = Regex.Replace(page, @"<(.|\n)*?>", string.Empty);
    var db = Database.Open("MyDb)";
    var sql = "INSERT INTO table (myField) VALUES (@0);
    db.Execute(sql, cleaned);
}