Last post Nov 01, 2013 01:03 PM by RNG
Mar 24, 2013 09:41 AM|Roy Gourgi|LINK
I am having difficulty with arrays because of the limitation of the size of the arrays that I can declare. I need sizes that are very large and windows or C# does not allow me to create bigger than the following for example:
public static int gaPoints = new int[20000000000000000000 + 1];
I get an error message saying that "integral size is too large". I need to be able to declare sizes that are even much larger than the abovementioned size. How can I circumvent this problem?
Do I have to use something other than arrays?
Mar 24, 2013 09:59 AM|Rion Williams|LINK
You could consider using another type of collection that would be dynamically sized, such as a List<int> :
List<int> gaPoints = new List<int>();
as the value that you mentioned far exceeds the limitations that a single integer can hold.
Although, if you are seriously storing that amount of values or more you are likely going to run out of memory. You really should consider what you are using all of these values for and determining if there is a better method of handling this.
If you explained what you are trying to accomplish, we could likely point you in the direction of some alternatives other than storing trillions of integers. (Is there a formula that you could use to actually generate the values dynamically as opposed
to storing them all?)
(Read more here)
Mar 24, 2013 10:27 AM|Roy Gourgi|LINK
I am working on the Graph Isomorphism Problem and as such large graphs have many points and vertices that have to be accessed quickly. In order to avoid redundancy and as such speed issues, I store the information of points, vertices, etc... in arrays for
quick setting and retrieval.
For example, a very small graph with 10 vertices and row sizes of 3 would have 120 points (10 * 9 * 8) / (3 * 2 * 1).
So in this case my array gaPoints = new int;
But as the number of vertices gets larger the numbers explode exponentially and I soon run out of memory!
In order speed setting and retrieval I keep a tab of the values stored in the gaPoints array but only while the program is running. Once the program is finished running I do not have to store the information in the gaPoints array.
Should I be using a database such as SQL or use flat files to do this?
Mar 24, 2013 10:32 AM|Rion Williams|LINK
A file or database will likely be preferred if you are going to be storing as many values as you mentioned.
In-memory storage really isn't ideal when the number of values being stored begins to exceed the limitations of some of the more simple numerical data-types such as integers.
Mar 24, 2013 10:40 AM|Roy Gourgi|LINK
If as you say I use a file or database to do it, would I then be able to store these very large sizes in either a file or database?
For example, if rather than storing the information in an array I store it in a database, would I then be able to store ANY amount in the database and set it and retrieve it?
Does a database or file not have limits also? I used to work with Visual FoxPro and there was a limit of 2 billion records in the datase.
Mar 24, 2013 06:02 PM|Paul Linton|LINK
You may need to rethink this a little bit. Your sample array (I love the +1 in the size by the way) is 80 ExaBytes in size (plus 4 bytes). The ram for this computer will cost you about 20 billion dollars (I'm assuming that you get a large discount on a
purchase of this size). I don't think your computer will have enough slots to hold the memory cards! If you can read one int every nanosecond it will take about 600 years to iterate through the array.
As I say, you might want to rethink your approach to this problem.
Mar 24, 2013 07:35 PM|Rion Williams|LINK
As I mentioned earlier and Paul also touched on,
There are certainly better ways of handling this issue rather than resorting to using high-end super-computers to handle all of the storage and processing (which still could struggle with all the information).
I would strongly consider determining if there is a way to generate some of the values that you are storing to avoid storing them (or at least as many) in the first-place if that is possible. It may add some additional processing time to your problem, but
at least it will be solvable and not restrained by hardware and storage limitations.
Nov 01, 2013 01:03 PM|RNG|LINK
I am emphatic to announce that I have resolved the arrays size is too large memory issue for the Graph Isomorphism Problem! :):):)
If anyone remembers I needed to create arrays such as this just for starters.
I have since resolved that problem and now I am using a fraction of that amount! :)
Although it added a few extra steps of processing to my code, the searches are now done on a fraction of the number of records. I can't quantify it yet as to the net effect, but it seems that the few extra steps of processing are easily offset by the much
faster searches on much fewer records! :)
What is most important ofcourse is that now I am able to continue!!! :)
Thanks everybody I have a few more questions that I will post later!