May 24, 2016 12:43 AM|PaulTheSmith|LINK
Look at the numbers.
Reading a line from a file takes about 2 microseconds. See my code.
The web service calls that you make for each line take 430,000 microseconds. (You said that a 100 line file takes 43 seconds)
So, what does you program do? It does web service calls. That's all. The file reading part is 0.0004% of the work, that's zero to humans. forget it.
What is a web service call? It is shouting a question at someone on the other side of the planet and waiting for them to notice your question, then get around to answering you and then shout the answer back at you. If you really need to yell hundreds of
thousands of questions to some remote server then it is never, ever going to be fast. You design is screwed.
You need to go back to the drawing board and re-architecture the whole thing.
Maybe you need to upload your file to a slightly different web service and ask it to process the whole file for you in one transaction which can happen locally (no remote calls) and then send the results back to you.
Maybe you should not wait for a file to be created but instead get each line of information as it is generated and do the web service calls immediately. So, while the other system is working out how to create the next line in the file you can process the
previous line. If your web service processing is less than the time it takes for the generating system to create the next line then you processing will be complete half a second after you get the last line.
Maybe you should not process every line in the input file. Is someone really going to look at all 100,000 lines of output? Can you filter the input file?
Is the file similar to the previous file? Maybe you just need to work out the 'delta' and process that.
Set expectations. Maybe you should start processing after everyone goes home and give them the results in the morning. Does it matter if it takes 10 seconds or 10 hours if no one is waiting?
Look carefully at the business need. Do you really, really, really need to do all those calls for every single line? Is each line really that different? Maybe the whole thing needs to be interactive. Let user pick the line they are interested in and
only do the web service call for that line when they ask for it. (They will not do that 100,000 times).
Does the information that the web service provide change very often? Can you download all the remote information once a day/week/month/hour in the background and use the local copy to do your processing.
So many possibilities.