Im making a website, and currently its so data intensive, that it takes 20-30 seconds to read and write from database on avarage pageload. Everything needs to be realtime and synchronised, so i truly cant avoid using cashe. using cashe
and indexing techniques ive brought it back to milliseconds, but having almost 2/3 of the data in cashe feels scarey to me if the website process might crash or lose data, and im wondering how i can avoid feeling scared. or what kind of alternatives or safety
features i could use.
The use for the database has become useless as everything is now working from cashe, and i decided to go with serialisation instead, and load it on server startup, and save it in background processes way after the page has been returned. as this makes the
website startup faster than loading it over TCP trough the databaseconnection.
since everything is cashed and serialised from xml to classes, programming has become magically powerfull, allowing all kinds of operations that no programmer would advice over SQL connections. The website runs smooth, solid, and more powerfull than anything
ive seen before, and i almost feel like im outsmarting the world by doing this, but i expect it to be a really bad design, because it seems too good to be true. what are the opinions of you guys?
XML files was never meant a replacement for db. As you told, you'll likkely lose data if the app is recycled. What about consistent backup, db integrity, concurrency etc...
You tried to understand first why your database was slow? It's IMO bad to just try something else because something that should work doesn't rather than trying first to understand why something that should work doesn't. 20 to 30 seconds is clearly not the
kind of response time you have from a db (you were trying to read thousands of rows each time ???)
You have a whole lot of database flavor depending on what you are trying to do such as in memory databases or document oriented databases. Relational engines may have also optimization features such as "in memory tables" etc...
Im making a website, and currently its so data intensive, that it takes 20-30 seconds to read and write from database on avarage pageload. Everything needs to be realtime and synchronised, so i truly cant avoid using cashe. using cashe and indexing
techniques ive brought it back to milliseconds, but having almost 2/3 of the data in cashe feels scarey to me if the website process might crash or lose data, and im wondering how i can avoid feeling scared. or what kind of alternatives or safety features
i could use.
Most applications take advantage of indexes and caching. Caching should not cause data loss. I recommend testing your design to see what happens if there is an application restart or fatal exception.
fazioliamboina
The use for the database has become useless as everything is now working from cashe, and i decided to go with serialisation instead, and load it on server startup, and save it in background processes way after the page has been returned. as this makes the website
startup faster than loading it over TCP trough the databaseconnection.
This design sounds like an in-memory data store not cache. Cache is temporary storage that refreshes during run-time. Often cache will expire which causes a reload. Other times cache is refreshed due to a change in the back end data store.
fazioliamboina
since everything is cashed and serialised from xml to classes, programming has become magically powerfull, allowing all kinds of operations that no programmer would advice over SQL connections. The website runs smooth, solid, and more powerfull than anything
ive seen before, and i almost feel like im outsmarting the world by doing this, but i expect it to be a really bad design, because it seems too good to be true. what are the opinions of you guys?
In my opinion this design is atypical and not an option for most web applications. If this design works for your application than fantastic.
None
0 Points
20 Posts
xml as database alternative too good to be true?
Nov 18, 2019 12:04 PM|fazioliamboina|LINK
hello people of all sorts,
Im making a website, and currently its so data intensive, that it takes 20-30 seconds to read and write from database on avarage pageload. Everything needs to be realtime and synchronised, so i truly cant avoid using cashe. using cashe and indexing techniques ive brought it back to milliseconds, but having almost 2/3 of the data in cashe feels scarey to me if the website process might crash or lose data, and im wondering how i can avoid feeling scared. or what kind of alternatives or safety features i could use.
The use for the database has become useless as everything is now working from cashe, and i decided to go with serialisation instead, and load it on server startup, and save it in background processes way after the page has been returned. as this makes the website startup faster than loading it over TCP trough the databaseconnection.
since everything is cashed and serialised from xml to classes, programming has become magically powerfull, allowing all kinds of operations that no programmer would advice over SQL connections. The website runs smooth, solid, and more powerfull than anything ive seen before, and i almost feel like im outsmarting the world by doing this, but i expect it to be a really bad design, because it seems too good to be true. what are the opinions of you guys?
All-Star
46080 Points
16780 Posts
Re: xml as database alternative too good to be true?
Nov 18, 2019 12:27 PM|PatriceSc|LINK
Hi,
XML files was never meant a replacement for db. As you told, you'll likkely lose data if the app is recycled. What about consistent backup, db integrity, concurrency etc...
You tried to understand first why your database was slow? It's IMO bad to just try something else because something that should work doesn't rather than trying first to understand why something that should work doesn't. 20 to 30 seconds is clearly not the kind of response time you have from a db (you were trying to read thousands of rows each time ???)
You have a whole lot of database flavor depending on what you are trying to do such as in memory databases or document oriented databases. Relational engines may have also optimization features such as "in memory tables" etc...
Edit: for example maybe transitioning to a https://en.wikipedia.org/wiki/Document-oriented_database. The very first step would be likely to check what you need from https://en.wikipedia.org/wiki/ACID (you likely want Durability ?)
All-Star
44391 Points
19183 Posts
Re: xml as database alternative too good to be true?
Nov 18, 2019 12:37 PM|mgebhard|LINK
Most applications take advantage of indexes and caching. Caching should not cause data loss. I recommend testing your design to see what happens if there is an application restart or fatal exception.
This design sounds like an in-memory data store not cache. Cache is temporary storage that refreshes during run-time. Often cache will expire which causes a reload. Other times cache is refreshed due to a change in the back end data store.
In my opinion this design is atypical and not an option for most web applications. If this design works for your application than fantastic.