Last post Apr 30, 2014 05:58 AM by Heinz09
Apr 29, 2014 03:27 AM|Heinz09|LINK
Hi I am strugling to speed up an import application. The web form uploads the contents of a csv file to a db, i tried this both on the network server and local server, though in both cases I have very poor performance. It takes about 15 to 25 minutes to
load the 30 000 records, which I do not believe should take that long for the speed of the server or machine doing the uploading.
I am writing a small importing application, it takes a CSV file, it just take a look if the code should be imported or not, wether or not I take a list of a 100 000 people or only 30 000 the time it takes stays exactly the same.
I did a performance analysis on it, to see if I can find the bottleneck, I thought it could be my code that manipulates the string before it gets added and saved, I was utterly wrong!
This is what happened after I pulled the analysis. It seems the .Add is where to problem occurs, does anyone have another approach or idea how to speed this up?
foreach (string sContent in arrContents)
if (HeaderCount % 2 == 0)
if (tmpString == "Active")
streamContents += tmpDec.ToString() + ":" + tmpString+";";
IntermediaryStatu tmpIS = new IntermediaryStatu();
string tmpS = tmpDec.ToString();
tmpS = "0000000000000" + tmpS;
tmpS = tmpS.Substring(tmpS.Length - 13);
tmpIS.thirteendigitconscode = tmpS;
db.IntermediaryStatus.Add(tmpIS); // Takes more than 98%
db.SaveChanges(); //takes about 0.2%
Apr 29, 2014 04:53 AM|ignatandrei|LINK
Apr 29, 2014 07:48 AM|Heinz09|LINK
I am looking around for a tutorial, this one kinda points you the right direction, there is just one part that I don't get, where does the member function BulkInsert come from?
So far no success in finding it.
Apr 29, 2014 02:15 PM|ignatandrei|LINK
Apr 30, 2014 05:58 AM|Heinz09|LINK
Thank you very much, it was a few small things to change to get it up and running but once I got it working the average time of about 20 minutes for the import dropped to about 2.
I populated a datatable and passed it to the Bulk SQL insertion and it hardly uses any processing power, the most processing power now goes into string concatenation.