I have been a developer for several years, but I'm just now getting my feet wet in web development and I need some advice on how best to handle a problem in a web environment.
I have a website with a SQL server back end... the database that drives this website is less then optimum... but it is what it is... I'll have to address that later... what I am needing is to pull data from the database and then generate scores for various
metrics based on the data that I get back for each individual.. I have managed to fix the database enough to materialize the data for each individual in a reasonable amount of time... however I need to also rank individuals based on their total score acoss
all metrics when compared to everyone else (about 3000 other people). Which means I need to pull all of the data for each of the 3000 individuals and score them as well.
What I am thinking (and not real sure if this is entirely possible or if it is the best method) is that I have some sort of process on the server side that periodically collects the data for each of the 3000 people and creates a collection of objects that
contains the data and scores for each individual and maintain that collection in memory (in the Application state?) on the server that is accessible to all sessions for this application. Then as each individual logs into the portal the data to drive all of
their pages is fed from their object in the collection stored in the application state. But my I will need some method of periodically going back to the datatabase and collecting the data for each individual again and scoring it all again so that I can keep
the rankings as up to date as possible.
My question(s), does this sound feasible? If so, is the Application State the best place to store this type of data? What type of method do you suggest for kicking off a secondary process to go and update this collection of objects that are stored in the
Application State?
I have done something like this before. I would only advise it if the flow of information is one way, meaning that you are only pulling from the DB to update Application State (not the other way around). This will reduce overhead on your SQL Server, but
increase overhead on your Web Server as it needs to maintain all of those objects in memory.
Look at Global.asax file, specifically within the Appliation_Start method. This will ensure that the data gets populated every time the application gets restarted (which might happen unexpectedly for a number of reasons).
In addtion to that (or even alternatively), if you have a central BasePage class that all pages inherit from, you can perform a check on your Application State. I would check to see if it is null or if it has a timestamp older than whatever period you'd like
to refresh on. In either case, do a database pull, store the objects in a list, store that list in your Application State then store the dateTime stamp. This way, ever time a page inheriting from BasePage is requested, it checks to see if the Application
State is "old", if so, it updates the data, if not, it skips that step.
"Dream as if you'll live forever, live as if you'll die today." --James Dean
Marked as answer by Amy Peng - MSFT on Jan 25, 2013 04:45 AM
For sure your a not on an easy track ;) From my personal point of view and if you cannot create a kind of datawarehouse to pre-compute your scores, I will go for an in memory way. At the exception of using the app session. You have to go for the .Net Cache
namespace and decouple your app in a multi-tier architecture. This will lead to provide you with a separated library that can be later re-used for different UI.
I second the nomination about doing something on the database side. Whether it's a data warehouse or a summary table in the existing database depends on the overall requirement.
bmturney
0 Points
1 Post
New to web dev and need some advice
Jan 18, 2013 06:48 PM|LINK
I have been a developer for several years, but I'm just now getting my feet wet in web development and I need some advice on how best to handle a problem in a web environment.
I have a website with a SQL server back end... the database that drives this website is less then optimum... but it is what it is... I'll have to address that later... what I am needing is to pull data from the database and then generate scores for various metrics based on the data that I get back for each individual.. I have managed to fix the database enough to materialize the data for each individual in a reasonable amount of time... however I need to also rank individuals based on their total score acoss all metrics when compared to everyone else (about 3000 other people). Which means I need to pull all of the data for each of the 3000 individuals and score them as well.
What I am thinking (and not real sure if this is entirely possible or if it is the best method) is that I have some sort of process on the server side that periodically collects the data for each of the 3000 people and creates a collection of objects that contains the data and scores for each individual and maintain that collection in memory (in the Application state?) on the server that is accessible to all sessions for this application. Then as each individual logs into the portal the data to drive all of their pages is fed from their object in the collection stored in the application state. But my I will need some method of periodically going back to the datatabase and collecting the data for each individual again and scoring it all again so that I can keep the rankings as up to date as possible.
My question(s), does this sound feasible? If so, is the Application State the best place to store this type of data? What type of method do you suggest for kicking off a secondary process to go and update this collection of objects that are stored in the Application State?
AceCorban
Star
12318 Points
2269 Posts
Re: New to web dev and need some advice
Jan 18, 2013 06:58 PM|LINK
I have done something like this before. I would only advise it if the flow of information is one way, meaning that you are only pulling from the DB to update Application State (not the other way around). This will reduce overhead on your SQL Server, but increase overhead on your Web Server as it needs to maintain all of those objects in memory.
Look at Global.asax file, specifically within the Appliation_Start method. This will ensure that the data gets populated every time the application gets restarted (which might happen unexpectedly for a number of reasons).
In addtion to that (or even alternatively), if you have a central BasePage class that all pages inherit from, you can perform a check on your Application State. I would check to see if it is null or if it has a timestamp older than whatever period you'd like to refresh on. In either case, do a database pull, store the objects in a list, store that list in your Application State then store the dateTime stamp. This way, ever time a page inheriting from BasePage is requested, it checks to see if the Application State is "old", if so, it updates the data, if not, it skips that step.
Laflak
Member
28 Points
7 Posts
Re: New to web dev and need some advice
Jan 18, 2013 07:09 PM|LINK
Dan Bracuk
Contributor
3970 Points
1096 Posts
Re: New to web dev and need some advice
Jan 18, 2013 07:26 PM|LINK
I second the nomination about doing something on the database side. Whether it's a data warehouse or a summary table in the existing database depends on the overall requirement.