We are excited to announce that the ASP.NET Forums are moving to the new Microsoft Q&A experience. Learn more >
How to store huge data in List<string>[Answered]
13 replies
Last post Apr 26, 2018 10:48 AM by chandradev1
Last post Apr 26, 2018 10:48 AM by chandradev1
Participant
952 Points
489 Posts
How to store huge data in List<string>
Apr 24, 2018 06:03 PM|chandradev1|LINK
Hi All,
Could you please help me to optimize the code which is written to store huge data in list<string> like this
List<string> msg = new List<string>();
for (int i = 0; i < 100000000; i++)
{
string m = "I am string message";
msg.Add(m);
Response.Write(msg[0].ToString());
}
Currently this is throwing the out of memory exception.
Contributor
2853 Points
1372 Posts
Re: How to store huge data in List<string>
Apr 24, 2018 06:32 PM|adamturner34|LINK
That's because List<> has capacity of Int32.
Your code is impractical any way. There is nothing to optimize.
All-Star
48710 Points
18172 Posts
Re: How to store huge data in List<string>
Apr 24, 2018 06:37 PM|PatriceSc|LINK
Hi,
Try perhaps https://docs.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/runtime/gcallowverylargeobjects-element and make sure to force 64 bit mode.
It's around 3.5 Gb of data if I'm not mistaken. Beyond that "brute force" approach, it highly depends on what you are trying to do (for example you could store each unique value and a usage count or use a db etc...).
Participant
1660 Points
952 Posts
Re: How to store huge data in List<string>
Apr 24, 2018 11:34 PM|PaulTheSmith|LINK
What are you really trying to do? What business problem are you trying to solve?
Let's do some small calculations. Your string is nineteen character long and you are appending it to the Response 100 million times. That's 1.9 gigabytes being returned to the client page. At, say, 2 megabits per second internet connection speed that will take about 10,000 seconds.
Roughly three hours to load the page. Hmmm, doesn't seem like a real requirement.
Participant
952 Points
489 Posts
Re: How to store huge data in List<string>
Apr 25, 2018 12:43 AM|chandradev1|LINK
Hi PaulTheSmith,
I am fixing the bugs of one of real life application. There is the logic to generate possible combination on basis of reg-expression. That is generating more than 10000 records, then it being stored in list<string> data type. Then they are doing some string filtration. The exception is throwing while storing the huge data in list<string>.
I have to ask with business, is the input is really correct ? But i hope they will tell, my input could be any things.
All-Star
52803 Points
15764 Posts
Re: How to store huge data in List<string>
Apr 25, 2018 01:29 AM|oned_gk|LINK
The bug is storing hug records from database into web server memmory
Maybe you can find another way to not download 100000000 records to web server
Maybe find way to move filteration in the database
Suwandi - Non Graduate Programmer
All-Star
48710 Points
18172 Posts
Re: How to store huge data in List<string>
Apr 25, 2018 07:40 AM|PatriceSc|LINK
And it worked previously ? It depends on data found somewhere maybe in a database ? This is not something like a full text search ?
Participant
952 Points
489 Posts
Re: How to store huge data in List<string>
Apr 25, 2018 08:01 AM|chandradev1|LINK
There is no requirement for store in database or fetch from database. They will the give the input parameter, on basis of input there is one method to generate the possible combination on basis of input, Then that possible input then will hold in List<string> and it will display in UI.
It is working for small input. But when they are giving 30 to 40 input in this scenario it will take more memory. Then it is throwing exception.
Star
11464 Points
2439 Posts
Re: How to store huge data in List<string>
Apr 25, 2018 08:27 AM|Yohann Lu|LINK
Hi chandradev,
You are generating huge data and store in List<string>, And then filter it in this collection. It is easy to call the out of memory exception.
As far as I know, there is not a good way to avoid your issue. I suggest you can refer the following ideas.
1: limit the number of input parameters.
2: Save the huge data in the database. Then, show the data from the Data Controls(Paging display).
Best Regards,
Yong Lu
All-Star
52803 Points
15764 Posts
Re: How to store huge data in List<string>
Apr 25, 2018 08:37 AM|oned_gk|LINK
Before filter the list string, maybe you can also filter the data from database to limit the records.
For exampe you want filter the list based "abc123", then you can limit the records based "abc"
Suwandi - Non Graduate Programmer
All-Star
48710 Points
18172 Posts
Re: How to store huge data in List<string>
Apr 25, 2018 09:59 AM|PatriceSc|LINK
And you can't filter as you go ? You are are doing multiple combinations between all those inputs to generate a huge list and then only you filter ? It's is not possible to filter this list earlier ?
A higher level view could help (looks like a "full text search", "spell checking" or making a search for "similarities" to deduplicate data ????) but if I you really have no way to optimize your approach by using less data at some point using some form of "db" could be helpfull.
It could be writing to a file and then readback the file to load only what you need from what you generated... Also I would compute the size I need for the worst case scenario to see what would fit (to avoid trying to solve the case for 3.5 Gb and then later end up in needing 10 Gb etc...)
Participant
1660 Points
952 Posts
Re: How to store huge data in List<string>
Apr 26, 2018 02:24 AM|PaulTheSmith|LINK
Generate one combination, if it does not pass filter then discard otherwise keep. Repeat until all combinations are generated.
All-Star
52803 Points
15764 Posts
Re: How to store huge data in List<string>
Apr 26, 2018 03:55 AM|oned_gk|LINK
maybe you limit the list by split it into several lists or using collection
Suwandi - Non Graduate Programmer
Participant
952 Points
489 Posts
Re: How to store huge data in List<string>
Apr 26, 2018 10:48 AM|chandradev1|LINK
Hi All,
Thanks for reply. I discussed with business people, I told them, If you will provide the huge input to application, it will generate huge data in memory and we can only store upto 2 GB data. Even though it will take more than 20 min to read data.
Actually that input is not like real input, they are thinking on it. They are giving the combination of 4 to 5 test case in single statement.