in my asp.net website, i need to keep a 100MB file in memory. putting it in a application variable at application_start makes the website boot up slow.. and if my website crashes the data is lost.
is there a out of process way to do this? i need actual local memory, fast fast fast memory. cant afford any kind of pipe or tcp/sql connection when it comes to speed. even a virtual ram diskdrive doesnt cut it for me.
i need the data in memory, as if it was a actual variable in the application. without any serialisation or whatever, to keep acces incredibly fast.
ive tried pipes trough an external process, it takes a few sets of minutes to transfer 100mb. tcp is significantly worse.
the application state variable was fast enough. however, not reliable, and needs a spinup delay to load up everytime the application recycles.
any ideas?
i tried a random acces file, however it doesnt seem to be fast at all, when it comes to searching for something other than the ID of the row, all data still has to be read. or am i doing something wrong?
external database services are NO go. The problem is, that unlike usual use of SQL, i always need all data for comparisons, and litterally all columns and rows are always used in the maths and analyses. which makes non-local memory acces incompetent.
I can't tell if you are a crackpot or trying to be disruptive on purpose. This is an ASP.NET forum, business apps, and you're asking about systems design,
fazioliamboina
in my asp.net website, i need to keep a 100MB file in memory. putting it in a application variable at application_start makes the website boot up slow.. and if my website crashes the data is lost.
This is a fact. It takes time to load data in memory and volatile memory is lost when an app recycles, This is not a new concept in hosted apps.
fazioliamboina
is there a out of process way to do this? i need actual local memory, fast fast fast memory. cant afford any kind of pipe or tcp/sql connection when it comes to speed. even a virtual ram diskdrive doesnt cut it for me.
This is a great example of disruptive behavior. The question infers inadequacies without any proof or reproducible measurements. Its all hyperbole.
fazioliamboina
i need the data in memory, as if it was a actual variable in the application. without any serialisation or whatever, to keep acces incredibly fast.
What is "incredibly fast" and how is "incredibly fast" measured?
fazioliamboina
ive tried pipes trough an external process, it takes a few sets of minutes to transfer 100mb. tcp is significantly worse.
I agree 1+ minutes to send 100mb (megabits) is very slow. Even 100 Mb is very slow. Is there anyway you can post the tests and and source code?
fazioliamboina
the application state variable was fast enough. however, not reliable, and needs a spinup delay to load up everytime the application recycles.
ASP.NET application startup is well documented and there are solutions to alleviate the startup costs. Can you elaborate on reliable? What does not reliable mean?
thank you for your answer. i thought it was still an ASP.NET matter, since im talking about a alternative to application variables, which is a asp.net thing.
with not reliable, i meant, the app recycles and the data is lost. an out of process solution with its own memory that doesn't recycle will be reliable, but any type communication is not fast enough.
definition of fast = like looping trough an in-process array variable (which application state variables could provide) (10 ms)
definition of too slow = requesting data trough tcp, waiting for response, serializing it to classes, and then looping trough the array of classes (2,2 seconds)
it needs to stay under 10 ms, so i dont think is of any help to discuss my code for out of process solutions.
all i could think of to make it more reliable and to stop the spin up time, is to set the application to always running. but if the application crashes, it will lose all data.
As stated many many times over several of your treads. Use a database. I would put the database behind a service as it provides a clear separation.
Next, I would design a caching pattern to increase performance. Using cache is common practice and a suggestion made several times. The concept is very simple. The API, that you must write, checks if the requested data is cached. If the data is not
cached, the data is fetched and cached. There is a one time cost for fetching data. How, when, and how much data is fetched is up to you. It can be small chunks on demand or the entire file at startup.
Saving data is a little more complex but not rocket science. Add a timer to the API which checks for changes in the cache. The API does an async request to persist the changes.
The API - that you must write - is the middle ground and handles all the heavy lifting. You web application has no idea what happens behind the API. Then in the future, if you find a better database or whatever you can swap it out without affecting the
app. Also you can tweak the API without changing web app code.
as I said, a database would be too slow. and databases are outdated. except for some really weird risky and still imperfect projects, that dont even have a fancy name. and require learning to use, and still nowhere near as fast. (im talking about delayless
fast)
im about to finish my project. The perfect .NET friendly, inmemory in-process Modeldatabase with categorytree support and submodels. including an model editor. and my website is faster than 99+% of the internet. Models can be provided as classes. and the
classes are automatically extended with Database record properties, such as ID's owners, creationtime etc. changing all models in the database does not even take more than 20 ms.
good bye SQL, goodbye entity framework, and good bye to cashing.
please look at my latest thread, where I ask about getting the variables from a class, to make my object editor.
ive based the inmemorycashe on the application variable anyhow, and added a background thread to save the data when requested. When any really important data is written that shouldn't be abruptively lost, it is written to a backup file in case the app doesnt
make it to the next save-tick. if you still have an idea for an out of process solution that is just as fast as a local variable, id like to know. how fast are memory mapped files? in a test ive done, it was too slow, but also it was a really bad demo project
ive downloaded about it. there was a 300 ms delay before the data was received. but I need it to be absolutely unmeasurably ~ 0.
as I said, a database would be too slow. and databases are outdated. except for some really weird risky and still imperfect projects, that dont even have a fancy name. and require learning to use, and still nowhere near as fast. (im talking about delayless
fast)
im about to finish my project. The perfect .NET friendly, inmemory in-process Modeldatabase with categorytree support and submodels. including an model editor. and my website is faster than 99+% of the internet. Models can be provided as classes. and the
classes are automatically extended with Database record properties, such as ID's owners, creationtime etc. changing all models in the database does not even take more than 20 ms.
good bye SQL, goodbye entity framework, and good bye to cashing.
please look at my latest thread, where I ask about getting the variables from a class, to make my object editor.
ive based the inmemorycashe on the application variable anyhow, and added a background thread to save the data when requested. When any really important data is written that shouldn't be abruptively lost, it is written to a backup file in case the app doesnt
make it to the next save-tick. if you still have an idea for an out of process solution that is just as fast as a local variable, id like to know. how fast are memory mapped files? in a test ive done, it was too slow, but also it was a really bad demo project
ive downloaded about it. there was a 300 ms delay before the data was received. but I need it to be absolutely unmeasurably ~ 0.
Enlighten the community and post your test code. I'd love to see how you solved this problem.
Member
7 Points
236 Posts
alternative to application state variables
Jun 14, 2018 11:15 PM|uid390594|LINK
in my asp.net website, i need to keep a 100MB file in memory. putting it in a application variable at application_start makes the website boot up slow.. and if my website crashes the data is lost.
is there a out of process way to do this? i need actual local memory, fast fast fast memory. cant afford any kind of pipe or tcp/sql connection when it comes to speed. even a virtual ram diskdrive doesnt cut it for me.
i need the data in memory, as if it was a actual variable in the application. without any serialisation or whatever, to keep acces incredibly fast.
ive tried pipes trough an external process, it takes a few sets of minutes to transfer 100mb. tcp is significantly worse.
the application state variable was fast enough. however, not reliable, and needs a spinup delay to load up everytime the application recycles.
any ideas?
i tried a random acces file, however it doesnt seem to be fast at all, when it comes to searching for something other than the ID of the row, all data still has to be read. or am i doing something wrong?
external database services are NO go. The problem is, that unlike usual use of SQL, i always need all data for comparisons, and litterally all columns and rows are always used in the maths and analyses. which makes non-local memory acces incompetent.
All-Star
35391 Points
14068 Posts
Re: alternative to application state variables
Jun 15, 2018 12:49 AM|mgebhard|LINK
I can't tell if you are a crackpot or trying to be disruptive on purpose. This is an ASP.NET forum, business apps, and you're asking about systems design,
This is a fact. It takes time to load data in memory and volatile memory is lost when an app recycles, This is not a new concept in hosted apps.
This is a great example of disruptive behavior. The question infers inadequacies without any proof or reproducible measurements. Its all hyperbole.
What is "incredibly fast" and how is "incredibly fast" measured?
I agree 1+ minutes to send 100mb (megabits) is very slow. Even 100 Mb is very slow. Is there anyway you can post the tests and and source code?
ASP.NET application startup is well documented and there are solutions to alleviate the startup costs. Can you elaborate on reliable? What does not reliable mean?
Member
7 Points
236 Posts
Re: alternative to application state variables
Jun 15, 2018 10:27 AM|uid390594|LINK
hi,
thank you for your answer. i thought it was still an ASP.NET matter, since im talking about a alternative to application variables, which is a asp.net thing.
with not reliable, i meant, the app recycles and the data is lost. an out of process solution with its own memory that doesn't recycle will be reliable, but any type communication is not fast enough.
definition of fast = like looping trough an in-process array variable (which application state variables could provide) (10 ms)
definition of too slow = requesting data trough tcp, waiting for response, serializing it to classes, and then looping trough the array of classes (2,2 seconds)
it needs to stay under 10 ms, so i dont think is of any help to discuss my code for out of process solutions.
all i could think of to make it more reliable and to stop the spin up time, is to set the application to always running. but if the application crashes, it will lose all data.
All-Star
35391 Points
14068 Posts
Re: alternative to application state variables
Jun 15, 2018 11:53 AM|mgebhard|LINK
As stated many many times over several of your treads. Use a database. I would put the database behind a service as it provides a clear separation.
Next, I would design a caching pattern to increase performance. Using cache is common practice and a suggestion made several times. The concept is very simple. The API, that you must write, checks if the requested data is cached. If the data is not cached, the data is fetched and cached. There is a one time cost for fetching data. How, when, and how much data is fetched is up to you. It can be small chunks on demand or the entire file at startup.
Saving data is a little more complex but not rocket science. Add a timer to the API which checks for changes in the cache. The API does an async request to persist the changes.
The API - that you must write - is the middle ground and handles all the heavy lifting. You web application has no idea what happens behind the API. Then in the future, if you find a better database or whatever you can swap it out without affecting the app. Also you can tweak the API without changing web app code.
Member
7 Points
236 Posts
Re: alternative to application state variables
Jun 16, 2018 04:31 PM|uid390594|LINK
as I said, a database would be too slow. and databases are outdated. except for some really weird risky and still imperfect projects, that dont even have a fancy name. and require learning to use, and still nowhere near as fast. (im talking about delayless fast)
im about to finish my project. The perfect .NET friendly, inmemory in-process Modeldatabase with categorytree support and submodels. including an model editor. and my website is faster than 99+% of the internet. Models can be provided as classes. and the classes are automatically extended with Database record properties, such as ID's owners, creationtime etc. changing all models in the database does not even take more than 20 ms.
good bye SQL, goodbye entity framework, and good bye to cashing.
please look at my latest thread, where I ask about getting the variables from a class, to make my object editor.
ive based the inmemorycashe on the application variable anyhow, and added a background thread to save the data when requested. When any really important data is written that shouldn't be abruptively lost, it is written to a backup file in case the app doesnt make it to the next save-tick. if you still have an idea for an out of process solution that is just as fast as a local variable, id like to know. how fast are memory mapped files? in a test ive done, it was too slow, but also it was a really bad demo project ive downloaded about it. there was a 300 ms delay before the data was received. but I need it to be absolutely unmeasurably ~ 0.
All-Star
35391 Points
14068 Posts
Re: alternative to application state variables
Jun 16, 2018 06:14 PM|mgebhard|LINK
Enlighten the community and post your test code. I'd love to see how you solved this problem.