I would like to use the built-in file upload handling features of ASP.NET, however I have found that even if you increase the maxRequestLength and executionTimeout values, after 180 seconds, the aspnet_wp is recycled because it thinks it is deadlocked.

In addition, there is the issue that there is no feedback to the client of whether the upload is proceeding or encountered an error (the globe just sits and spins).

What would be ideal would be to asynchronously intercept the incoming request in an HttpModule or HttpHandler, and since the content-length is first, be able to set some current bytes/total bytes value in the application. Then with the help of a little client-side javascript that opens a self-refreshing "progress indicator" modelessdialogbox that could read these values every few seconds, not only would the deadlock issue be solved, but the client would also get some visual indication that the upload is proceeding.

After scouring everything I could find in the way of documentation and newsgroups, I have not been able to locate any information on how to just "count the bytes" of a request as they come in, but it seems like the async StartBeginRequest event may be the key.

This sounds like a real common scenario, can anyone help me?

Thanks in advance!

Somebody in this forum most have dealt with this issue, and I could really use some advice. Perhaps my last post was not worded correctly, here's a simplification:

The built-in file upload handling features of ASP.NET load the entire request into server memory before you can save the file. If the file is large, this can cause execution timeouts, or at the very least, consume *huge* amounts of server memory. What I am looking for is a way to intercept an incoming request (asynchronously, I suppose) so that I can process the incoming bytes in smaller chucks, saving each chunk to the file as it is received, rather than waiting for the entire request to get loaded into server memory before processing. On the client side, I am using the standard <input type=file> tag, so the uploaded file(s) are sent in a multipart/form-encoded POST to the server.

Has anybody dealt with processing incoming request bodies as they arrive, rather than after the entire request is received?

Is there a way to use a custom HttpHandler to do this? Perhaps with the asynchronous BeginProcessRequest() function? Has anyone here done asynchronous processing?

Any help would be *greatly* appreciated!

See:
http://www.asp.net/ControlGallery/ControlDetail.aspx?Control=222&tabindex=2

They've figured it out. This control is nice and should do all that you need to.

D

Did you ever resolve this problem?

We have been the same problem - we use modem access and occasionally upload files of around 2MB into a SQL2000 DB. Often this fails (not due to the internet connection), but due to some timeout or other.

We thought it may be the session timeout, but it happens before the session expires. I have also increased the scripttimeout and execution timeout properties without success....

If you have found a solution I would be very interested to hear about it.

James

To micklemj:

Increasing the script timeout and execution timeout won't help,
it's the aspnet_wp.exe process that thinks it's deadlocked after
180 seconds and "recycles" itself for you (how nice of it!).

I have found a partial solution though. As long as ASP.NET is
serving some requests, aspnet_wp won't timeout. I have used
this solution for uploads that take up to an hour with success.
The way to make it work is to spawn a new window from the
page before starting the upload. The code in the spawned
page just refreshes itself every few seconds until the upload
is finished, then it closes itself.

However, this does not fix my main problem, which is that
the *entire* uploaded file is kept in server memory during
the upload. My uploads can be hundreds of megabytes, so
that is not feasible.

That's why I was asking about anyone who had worked with
asynchronous handlers... I thought maybe I could grab every
4000 bytes or so and dump it to a temp file, rather than wait
for the entire request to get there.

I know that there are 3rd party utilities for this, but that's not
an option for me, I have to write the code and no client-side
ActiveX controls are allowed (other than what comes with
Internet Explorer)

Hope this helps...

I briefly researched that problem and I found two possible solutions (other than using ActiveX, Java or .NET DLLs on the client):

- open a second upload window (or use an IFRAME) that does the file POST. write a httphandler that handles the upload process on the server. in the while() loop for reading in the file from the stream, write the current number of received bytes into a static variable. use XMLHTTP or a simple page refresh every 5s or so in the first window to call back to the server. have the server read that static variable from the httphandler and return it to the client after the post-back.

- use ADO on the client to read in the file and upload it via an IFRAME. I forgot how that exactly worked, but you should be able to find a link on google if you look for 'file upload ADO object' or something similar.

hope this help!!

Marc

To Marc Hoeppner:

Yes, as I mentioned in my previous post, I am using the "second window"
method to keep the upload alive (actually I use a modeless dialog box that
contains aframeset that contains an aspx page that contains client script that
refreshes every few seconds, but that's not my issue)

My issue is that the post from the main page uses the <input type=file>
control, so the *entire* file (or files) are sent in one big http request.
While this works, the aspx page receiving this huge post (which can be
hundreds of megabytes for my application) stores the *entire* request
in server memory before I get a chance to begin handling it.

Obviously, this will only marginally work with a single user, and if several
users try to upload simultaneously, the server will run out of memory.

What I need to do is intercept the request *AS IT COMES IN*, not after
it is complete.

That's why I was asking about asynchronous httprequest handlers.

Do you (or anyone else reading this) have any experience with these?

Can they be used to read an incoming request in chunks, so that the
server doesn't have to buffer the *entire* posted file in memory?

Any pointers greatly appreciated!

J B. Podolak

>Do you (or anyone else reading this) have any experience with these?
>Can they be used to read an incoming request in chunks, so that the
>server doesn't have to buffer the *entire* posted file in memory?

Without having done exactly that as of yet, from my experience with HttpHandlers the answer is yes. You should be able to get the BeginRequest event and take over from there. The standard ASPX handler certainly does something similar for the native ASP.NET file upload. Anyway, it certainly is possible with an ISAPI filter/module, so that should work with either HttpModule or HttpHandler.

I took a quick look for you at my link list that I luckily still have from that research, so you may want to take a look at these links for an alternative ADO/XMLHTTP way of uploading files:

http://www.15seconds.com/issue/010522.htm
http://www.pstruh.cz/tips/detpg_uploadvbsie.htm

If you control the environment totally, you may want to think about a .NET DLL that is loaded via the object tag right inside the browser. You then can do anything like you would in an ordinary Windows Form application. The client needs .NET framework 1.1 installed in his machine, so this may be a show-stopper at the current time.

One last thought is to try something completly different: for a CMS-like application that we are currently writting, we have a similar problem with large files. Obviously the main problem with large files is the browser timeout that is hard to control. We are probably going to change the process totally, so that the user first gets fills out a form with all the meta data (name, description, author, keywords, categories, etc.) and then gets a ticket number from the system. He then mails the file to an email service on the same server. A windows service on the machine constantly checks for incoming email with a valid ticket number and then finishes the 'upload' process by adding both the meta data and the file to a database. The really nice thing about this is that the upload works asynchronously without the user having to wait in front of his browser or forcing him to deal with timeouts :)

Hope this helps!

Best regards,

Marc

sounds interesting, anyone willing to post some sample code?

Hi

I've been trying to crack this nut for the last 4 months. This is what I've got so far:

1. It can easily be done, if you set up a TcpListener, and direct the upload to a port other than 80. Via the TcpListener, you get the raw request stream, and you call all the shots yourself. No reading into server memory. I got this solution up and running. Tradeoff is, that not all networks are open to other ports than 80. Not all clients will be able to upload. So I took it down again... :-(

2. Studying the disassembly from abcupload.net, they only tap into the BeginRequest event via a HttpModule when implementing the progress bar. In other words: It is possible to split the http request somehow, without using either a HttpModule or HttpHandler. This must be first priority.

Turn this point around, and you get this:

They DO need to use a httpmodule to tap into the upload process when they implement the progress bar. That must mean, that their upload class does not fully split the upload, even though they do enable saving of the request in chunks of 8 kb. Otherwise, they would not have bothered with the HttpModule. I only make this point in the hope that it will point us in the right direction. I think it means, that a bufferedreader must be introduces somewhere...

Right now I am looking at the following two hopes:

1. The HttpContext.Request.Filter allows you to set a stream through which the request must pass. Maybe we can block the request in the filter...? Hmm...not counting on it though!

2. You can assign a new HttpApplication to a HttpContext, and set it to a class of your own. Remember, HttpApplication is the class "serving" HttpContext. I think this is the class that does the request reading. So if we could make our own implementation of this class, we could override the default reading method....somehow...hmm...sounds difficult!

3. Yes, async Http request handlers might also work. But don't they need to be registered in web.config? Based on the fact that abcupload.net does not need registration of handlers, I find it unlikely (Although they might create the async handler via a HttpHandlerFactory, thus not needing alterations in web.config.....hmm...is that possible?)

This problem MUST be solved, and abcupload.net shows that it can! We need a site dedicated to solving this problem. Or maybe a newsgroup. A place where we can meet and crack this nut. Anyone got any suggestions?

If anyone makes any progress concearning this problem, please email me at rh@hghardware.dk

Kind regards,
Rasmus
Denmark

I can provide a website for development and testing. I could even be talked into hosting a source control system of some type if there was enough interest.

Regards,

Doug Wilson
Fridge Productions

To Rasmus,

I looked into asynch httphandlers, but it would seem that these
are for when your custom processing needs to do asynch work.

ASP.NET just calls you once, then you call ASP.NET back when
you are done with your processing.

From what I have found in many, many web searches for docs,
the first thing that happens when a request (posted file) comes
in is that IIS mapping determines that it should go to the ASP.NET
dll, and then IIS "forwards" the request to whatever BeginRequest
handler is currently associated (built-in or your custom handler).

But, if I understand this process correctly, by the time BeginRequest
has been called, IIS has *already* received (and buffered) the
*entire* request in memory, so even though a custom BeginRequest
handler could read the request in, say, 8000 byte chunks and save
it to a temp file, the stream that it is reading is *already* completely
loaded into IIS memory, so it wouldn't help.

However, as you point out, AspUpload.Net claims that their GigUpload
technology keeps server memory load really low (from their docs, it
looks like the "chunksize" is all that is kept in memory).

So either they are only being *partially* truthful in their marketing
(ASP.NET only uses, say, 8k, but IIS uses the full file size), or I am
missing something.

I know that the interception of an incoming request "as it comes in"
can be done in an ISAPI extension or ISAPI filter, but everything
that I have read states that by the time *any* ASP.NET code sees
the request, IIS has already received the *entire* request.

Any Microsoft folks want to clarify this for me?

Given all that, I really don't know how AspUpload.Net does it without
using a client side control or a real (unmanaged) custom ISAPI filter.

There is one other way to make this work. If you can count on the
browser being Internet Explorer, you can use an ADO objects (the
XMLHttp and Stream objects) along with client-side javascript to
post the upload in small chunks. The problem with this is that the
security settings for Internet Explorer have to be relaxed to allow
reading of local files, and I'm pretty sure that most people won't
want to do this...

J B. Podolak

I too am working on this problem for an application and here's my thoughts so far:

In the Application_BeginRequest event handler you do have control before the entire stream is buffered into memory. Just set a breakpoint in that method somewhere and you'll see that it fires immediately upon clicking your submit button. So that means that there is the ability to do some pre-processing on the request before the upload begins a-la ABC Upload. What the pre-processing is that we need to do is a mystery....

Here's what I have done:

I've trapped the request for the upload in Application_BeginRequest and cached it in the Cache object. I then launch a popup window which reads the cached Request object to try and access the Request.TotalBytes property to get the length of the input stream. The problem is that the input stream is so busy doing the upload that it never responds to the line of code trying to get the TotalBytes property. I know this because if I hit the Stop button on the main browser window (the one doing the upload) my popup immediately displays the amount of bytes uploaded to that point.

I feel so close, yet I think my approach may be too simplistic....

Jay

To J B. Podolak and ITJ

Sounds like async handlers are out of the question. Too bad - they looked promising :-).

Regarding IIS buffering the request:
My experience is, that it is possible to intercept the event before the buffering occurs - as ITJ points out. However, as soon as a reference to the request stream is made, the *entire* request is read into memory, as J.B. points out.

I think we need to override the method supplying the request to HttpContext. I am not sure how to do this. However, one approach would be to implement your own hosting scenario. Look for System.Web.Hosting in the documentation, and it states, that you are able to host your own virtual address, thus CIRCUMVENTING IIS.

Basically it requires you to implement the SimpleWorkerRequest interface, which apparently will be responsible for serving the HttpContext object.

To me, this sounds like good news. However, I haven't been down that road yet, and are reluctant to go there, because it doesn't seem to be what abcupload.net is doing. However, after my exams (18/12), I'll probably try this.

I can only support your request for a microsoft developer to look at this. In my opinion, we need an optimal solution to this problem, as the current upload method clearly has shortcomings or even a memory leak, as indicated by the 4. post in this thread:

http://www.asp.net/Forums/ShowPost.aspx?tabindex=1&PostID=95255

I've been doing some more digging in System.Web.Hosting assisted by the ms opensource Cassini web server documentation

I really believe we could get somewhere by giving this a shot. It is the backdoor into the top of the http request pipeline that we need.

When IIS receives a request for an aspx resource, it instantiates a HttpRuntime object, and calls ProcessRequest(HttpWorkerRequest hwr)

The HttpRuntime object is TOP of the http pipeline, as shown here:
http://msdn.microsoft.com/msdnmag/issues/02/09/HTTPPipelines/default.aspx

The hwr argument passed to the HttpRuntime object is the toolbox that we want to alter. Inside HttpWorkerRequest are all the methods that HttpRuntime uses when it wants the request processed. HttpWorkerRequest is an abstract class, so we can implement our own version, and override the methods responsible for reading the request.

There is even a class called SimpleWorkerRequest, which is already implementing HttpWorkerRequest. So in fact we only need to implement our own SimpleWorkerRequest.

By creating our own hosting environment, we do the job of IIS: We pass an instance of our SimpleWorkerRequest-derived class to HttpRuntime.ProcessRequest, to make it the new toolbox that HttpRuntime must use, when it wants the request processed.

How to setup a separate hosting environment? This articles explains how - and apparently, it seams quite possible:
http://www.iunknown.com/Weblog/HostingASP.NETinaWinForms.html

However, I find it slightly annoying that we have to go so far to theoretically have a chance of achieving the goal. I've found indications in MSDN about a possibility of changing the HttpWorkerRequest object, even without creating your own hosting environment, and I'll be looking at that next.

Hmm...once again my exam preparation was cancelled in favour of .net. Terrible. Only 4 days left. I'll be seeing you guys on the other side of the exams.