Last post Jul 01, 2011 02:46 PM by AceCorban
Jun 30, 2011 01:26 PM|sephiroth100|LINK
Always like to ask the experts when I stretch my meagre knowledge so I thought I'd ask about this scenario.
I have some user data, quite small which I stick in session. However, I'm now developing a new feature which means that each user could potentailly have a list of say 500 unique selections stored against their profile. It doesn't take a mathematician to
work out that a few users = lots of data.
The issue is that a user might access this data many times during their visit so I don't really think going back to the database each time is perhaps the best thing to do. What I'm thinking of is storing it in session, perhaps even storing all their selections
as a single string then parsing it when I store and retrieve.
If this is the case I'll change my session state from inproc to sqlserver.
So I have a few questions:
1) does this seem a reasonable way of handling this kind of data?
2) is sqlserver mode going to cause me more problems than it solves? I don't have a lot else in session.
3) are there any better ways of handling this data?
Any advice always appreciated.
Jun 30, 2011 02:19 PM|AceCorban|LINK
If you are storing session data on the sql server, you will have to hit the database every time you wish to access the session anyway.
I would honestly not worry about the users hitting the database frequently. Most applications these days make a visit to the database per user per page (some even more than that). Depending on which data you need to access and how, this could be done different
ways (either through AJAX calls, or simply handling it in Page_Load for any given page).
In the Page_Load example, a nice thing to consider doing is to implement a Base Page for all things that happen commonly across all other pages. So, for instance, if you need some user data against a userId you are storing in session, you could have the
Page_Load of the BasePage make those database calls, and have each other page Inherit from BasePage, and call it's Page_Load in their pageLoad:
public class MyPage : BasePage
protected override void Page_Load(object sender, EventArgs e)
//then do whatever you need for that specific page.
Of course, this still depends on what type of data we are talking about. Care to shed a little more light?
Jun 30, 2011 02:45 PM|sephiroth100|LINK
Thanks for the reply Ace...
At the moment i store a users basics in session, but i want to give users an opportunity to create lists of things - favourite stores, products etc and not have to keep strict limits on the number of items.
If you do some quick sums, 100,000 users who have 100 items = 10m records, hence the not wanting to retrieve the data in a raw state each time.
If I store the data as a single string this brings the search down to say 100k. My thoughts on storing it in session would be that the session state database might only contain data for 1000 sessions at any one time and therefore make up for the impact
of not being inproc which i believe is about 25% quicker.
Does is sound more plausable now you know more or am I still overthinking the problem?
Jun 30, 2011 03:22 PM|AceCorban|LINK
Are you trying to save trips to the database, or the size/number of records? Database operations are a little trickier to itemize like this. If you have 100,000 users, even with all of them using your site at the exact same time, you might find that you
only have a few "concurrent users" at any moment (this sorta depends on how long you expect your users to be viewing the site, rather than actively requesting data from it). While I used to baby my database as much as possible as well, I later decided that
this is what they are made to do, just like a web server is meant to deliver html output to any number of concurrent users it experiences.
Storing this data in session as you suggest introduces a few challenges.
First, storing as a string may reduce the number of "records" you have to keep alive at any given moment, but you now have to parse that information at any moment into meaningful chunks of data that can be processed. Performing a split, for instance, will
drastically hurt your performance (especially if you are dealing with that many items).
Second, you will have synching issues. If, at any point, their session data is modified (preferences added, removed, modified, etc), you will have to decide whether to update the session and backsynch to the database or update the database and upsynch the
session data. Either way will end up nullifying any performance equity you might gain from such an approach. Depending on how often these types of updates could happen, this could end up being considerable.
Third, storing in memory is simply not an option (which you've already realized), so forcing the session data onto SQL versus InProc eliminates any reduction in trips to the DB you might have gained with an inProc session since every request for their session
data is still a database trip. Additionally, you now add the overhead of parsing the data either on the server or the client, making it even less efficient.
This is certainly an important decision, and you are showing some wisdom to try to go outside traditional bounds to find room for optimization. However, I think in this regard, you might not experience a significant performance increase at all. My guess
is that you want to show their preferences somehow on every page. If not, you have even less to worry about because you simply do not need to access all the information all the time, and simply draw the information that is relevant for the user's request
at that particular moment.
Jun 30, 2011 04:06 PM|sephiroth100|LINK
Thanks a lot for your advice - it is very much appreciated.
I suppose the reason i'm being so cautious is that on some pages I will have a list of things that can be manipulated a lot. As an example I might have 5000 stores onsite that don't change all the time. These could be cached, then filtered at page level
using linq for narrowing the selection and paging.
I'm now wanting to integrate user data into these lists by highlighting favourite stores or maybe recording a saved product. A single page might be re-displayed 10 or 20 times, as the user narrows and traverses, saves and removes, so their data will need
reapplying each time.
I need to consider all you have said so far and try to understand what will give a benefit and what won't.
Jun 30, 2011 06:00 PM|AceCorban|LINK
For something like that, I'd probably use an AJAX solution, so the page itself isn't constantly being reloaded. Within the AJAX call, have the backend webservice first do one database call to return an array of store IDs that the user prefers. Then, perform
a 2nd call to list all stores. As you render out your response stream (whether it is XML or JSON), you can conditionally place a flag on the result data based on if the user has a preference toward it or not by seeing if the primary key is contained within
the first array. This solution only really does 2 database calls, which isn't so bad.
Jul 01, 2011 03:07 AM|sephiroth100|LINK
Jul 01, 2011 02:46 PM|AceCorban|LINK
Well, it's just one guy's opinion.
But I like where your head is at, exploring options to try to make your application the best it can be.