Rasmus said: Congrats on the house! Whereat in the world, may I ask? Kelowna, BC. Great spring, summer and fall. Lousy winters unless you're a skier, which I'm not.
any thoughts as to how we can "most cleanly" access the statistics variables (bytes read, file length, etc.)? My first thought is that there should be some way to store them in the user's session. Then it would simply be a matter of sending them back
to the thermometer, whatever form that takes. Actually, it might be better to store them at the application level, then you could build a global upload montoring page!
could really use a smoothly increasing progress bar, without the continous refreshing in the background. Well, I can't think of any way to get away from the continuous refreshing, but you could certainly reduce the appearance of it by using your #2 suggestion.
A couple years ago I coded the server side for a
Penguin Chat that was more or less real time. We worked only with Flash 4 and basically form POSTs, no XML Sockets. It worked very smoothly and with a half decent Flash artist who can handle the action script, it should be no problem. As far as I know,
all major browsers ship with at least Flash 4. Doug
Cyberfloatie wrote: My first thought is that there should be some way to store them in the user's session My first choice too. But remember, at the time we are processing the files, the session hasn't been setup yet. We're working too high up in the
http pipeline to use session data. Application level is - as you say - possible. However, not very nice in my opinion. Here is an ambitious plan on how we could get to the session data: 1. In the httpModule, we only read the "dredman vars" and then set them
to 0. Then we let the page load as normal...it should ignore the posted data(?). 2. In the normal Page_OnLoad event, we reset the dredman vars to their original values, grab the HttpWorkerRequest, and do the upload processing. That way we get the session data,
and move the whole processing to a "standard" level in the pipeline - where all code is normally executed. How impossible is that plan?
We worked only with Flash 4 and basically form POSTs, no XML Sockets Ah, posting in Flash...thats another possibility. Also, one could make a function that approximates the bytes uploaded, given only a few time-value pairs. That greatly reduces the need
for postbacks...and makes solution 3 a real possibility. Sounds like new skies will be a hit for christmas :-)
Can anyone give us clueless a few tips on how to parse the stream? I know i can take the byte[] and create a memory stream from that, and pass that stream to a streamreader. Right now i just read every line and check it for the content boundary, im sure there
is a faster and more efficent way to do this. Thanks for your help.
Cyberfloatie/ups101, Have you (or anyone else) been able to prevent uploading of files larger than a certain size? Before I buffer the incoming entity body, I do a check on the "content_length" similar to the code that Bjorn.B provided on the 20th of December.
If the content_length is greater than a certain size, the idea is to stop to remainder of the incoming request and redirect the user off to a page that informs them that the file they are uploading is too big. Whenever I attempt to do the redirection to another
page or URL (using HttpApplication.Context.Response.Redirect(...), HttpApplication.Response.Redirect(...) or even something like ....Response.End()) I receive a DNS error in the browser. Sure enough, the file upload is ceased, however I would like to be able
to alert the user to the fact they are attempting to upload a file that is too big. I have thought about HttpHandlers, however I would tend to think that by the time the Handler processes the request, the request has already been fully uploaded. I have a feeling
that because the HTTPModule is attempting to redirect the request while the request is still being sent by the user, that it gets confused and "spits the dummy". Has anybody had a similar problem to this, or have any suggestions as to what direction I should
be heading? The reason I need to do this is to prevent unnecessary data being transferred to our server (and hence any related download charges). Thanks, Paul.
Hi Paul, regarding redirectI have a feeling that because the HTTPModule is attempting to redirect the request while the request is still being sent by the user, that it gets confused and "spits the dummy". I think you're right. Do you correct
"the dredman vars" before leaving the request? (see earlier posts for explanation).
I have thought about HttpHandlers, however I would tend to think that by the time the Handler processes the request, the request has already been fully uploaded I remember experiementing with this, a long time ago. I am certain that the upload first
will take place when you touch on any part of the request object, including checking the content length. So I sugest you try something like this: 1. Post your upload to a "standard" http handler (ie. one not using session state - this is default, as I recall).
2. Get the content length in a httpmodule. If size > max allowed, either leave immediately, or first reset the dredman vars (not sure if necessary). Let the request proceed to the http handler, maybe after appending the request size as a request string to
the request url (see rewriting request urls elsewhere in this forum). 3. In the http handler, check the request url for the inserted request string. If existing, alert the user appropriately. Otherwise - well, whatever else. The idea being, that you can avoid
standard upload processing by posting to a http handler - and you can grab the request size in a http module, without triggering the upload process.
X0nic, regarding byte searchRight now i just read every line and check it for the content boundary, im sure there is a faster and more efficent way to do this I am using a stupid way myself - converting to a string, then searching that string,
in the hope that asp.net has a native string pattern matching algorithm that outperforms whatever I implement anyway. Clearly this is not an advisable solution - but I'm lazy :-). However, it is actually a classical problem, since it is so hard to do right
- so you should be able to pick up an algorithm somewhere on the internet. Search for the following (top 2 are most recommended: Boyer-Moore Matcher (linear running time, recommended for larger searches - complex) Knuth-Morris-Pratt pattern matching (linear
running time, fast on smaller searches - simple) Rabin-Karp string matching (normally linear running time, performs well on average - simple) Pseudo-code for each algorithm can be found in "Introduction to Algorithms" by Cormen, Leiserson and Rivest. Kind
regards, Rasmus
It has been a while since I have checked this thread. However, it appears some of you are still wrestling with the code I originally posted. As some of you noticed, there were some mistyped variables in that code. Here is another snippet of code for those of
you still working on this. I'm sorry but I cannot post all my code however, this code snippet should be a huge help for most of you. using System; using System.Web; using System.IO; using System.Reflection; using System.Text; using System.Threading; using
System.Collections; namespace NETXUpload { public class ProgressModule : IHttpModule { public string ModuleName { get { return "ProgressModule"; } } public void Dispose() { } public void Init(HttpApplication context) { context.BeginRequest += new EventHandler(this.OnBeginRequest);
} private void OnBeginRequest(object src, EventArgs e) { HttpApplication httpApplication = (HttpApplication)src; GetEntireRawContent(httpApplication); } private void GetEntireRawContent(HttpApplication httpApplication) { ArrayList alData; int nCount; int nBytesReadTotal;
byte[] arrTemp; int nTempSize; int nBytesRead; int nBufferSize; //Get ContentType string sContentType = httpApplication.Request.ContentType.ToLower(); //Check to see if this is a multipart upload if ( sContentType.IndexOf("multipart/form-data") == -1 ) return;
//Get the multipart boundary byte[] arrBoundary = GetMultipartBoundary(sContentType); //Get the HttpWorkerRequest HttpWorkerRequest _wr = GetHttpWorkerRequest(httpApplication); byte[] _rawContent = null; //Create the progress monitor ProgressData progress;
//Get the upload id string UploadID = httpApplication.Request.QueryString["UploadID"]; //Create and load the progress object for this upload progress = ProgressModule.GetProgressObject(UploadID); //Getting Size int ContentLength = httpApplication.Request.ContentLength;
//Checking Size int nMaxRequestLength = GetMaxRequestLength(httpApplication); if (ContentLength > nMaxRequestLength) { return; } //Preloading progress.UpdateProgressData(0,ContentLength); byte[] arrBuffer = _wr.GetPreloadedEntityBody(); if (arrBuffer == null)
arrBuffer = new Byte[0]; DataReader byteReader = new DataReader(arrBoundary ,httpApplication.Request.ContentEncoding, ContentLength, progress); byteReader.ReadBuffer(arrBuffer, arrBuffer.Length); nBufferSize = (int) arrBuffer.Length; progress.UpdateProgressData(nBufferSize,ContentLength);
if (!(_wr.IsEntireEntityBodyIsPreloaded())) { if (ContentLength <= 0) return; int nBytesRemaining = ContentLength - (int) arrBuffer.Length; alData = new ArrayList(); nCount = 0; nBytesReadTotal = arrBuffer.Length; //Transferring while (nBytesRemaining > 0)
{ arrTemp = new Byte[nBufferSize]; nTempSize = (int) arrTemp.Length; if (nTempSize > nBytesRemaining) nTempSize = nBytesRemaining; nBytesRead =_wr.ReadEntityBody(arrTemp, nTempSize); if ( nBytesRead < nTempSize ) { //Upload stopped unexpectantly return; }
if (nBytesRead > 0) { nBytesRemaining = nBytesRemaining - nBytesRead; alData.Add(nBytesRead); alData.Add(arrTemp); nCount++; nBytesReadTotal += nBytesRead; progress.UpdateProgressData(nBytesReadTotal, ContentLength); byteReader.ReadBuffer(arrTemp, nBytesRead);
} } progress.Note = Configuration.GetConfig.GetNotesConfig.Finishing; } _rawContent = byteReader.InputStream; //_rawContent = arrBuffer; PopulateRequestData(_wr, _rawContent); //PopulateRequestData(_wr, byteReader.InputStream); } private static HttpWorkerRequest
GetHttpWorkerRequest(HttpApplication inApp) { BindingFlags bindingFlags = BindingFlags.Instance | BindingFlags.NonPublic; return (HttpWorkerRequest)inApp.Context.GetType().GetProperty("WorkerRequest", bindingFlags).GetValue(inApp.Context, null); } private
static int GetMaxRequestLength(HttpApplication inApp) { object local = inApp.Context.GetConfig("system.web/httpRuntime"); Type type = local.GetType(); BindingFlags bindingFlags = BindingFlags.Instance | BindingFlags.NonPublic; return (int)type.GetProperty("MaxRequestLength",
bindingFlags).GetValue(local, null); } private static byte[] PopulateRequestData(HttpWorkerRequest inWR, byte[] inData) { BindingFlags bindingFlags = BindingFlags.Instance | BindingFlags.NonPublic; Type type = inWR.GetType().BaseType; int i = (int)inData.Length;
type.GetField("_contentAvailLength", bindingFlags).SetValue(inWR, i); type.GetField("_contentTotalLength", bindingFlags).SetValue(inWR, i); type.GetField("_preloadedContent", bindingFlags).SetValue(inWR, inData); type.GetField("_preloadedContentRead", bindingFlags).SetValue(inWR,
true); return inData; } public static ProgressData GetProgressObject(string inID) { int i = 0; try { i = Convert.ToInt32(inID); } catch { i = ProgressMonitor.GetNewID(); } return ProgressMonitor.AddProgress(i); } private byte[] GetMultipartBoundary(string
ContentType) { int nPos = ContentType.IndexOf("boundary="); string sTemp = ContentType.Substring(nPos+9); string sBoundary = sTemp = "--" + sTemp; return Encoding.ASCII.GetBytes(sTemp.ToCharArray()); } } }
I will be getting just a tad off the track of the original post, but it is an issue that need to address in a current project that the original post brings to mind: I would presume that downloads are pretty much the same, in that they load the file into memory
as it is downloaded from the website. Is there any way to chunk and stream a download in a similar manner as to minimize memory usage of the webserver?
Hi Richard, I also find it reasonable to address this issue here. The solution is to use Response.IsClientConnected. This call will return true as long as the client is connected - and also stall the process until the current chunk has been tranferred to the
client, thus minimizing mem usage to a buffer size array. However, I've found that this call in fact cannot detect if a user cancels a download in progress, before the users CLOSES the browser (not just navigates away). Therefore, one should be careful not
to leave anything open between calls of IsClientConnected - as the answer might be a long time underway. This is part of what I'm using for downloads (implement in HttpHandler or HttpModule):
Hi All, I want to thank you all for this great thread, a really hard work that help me to build my upload module, many of us got a headache when try to parse the file I will post mi parser, you will need to modify the code a little bit, This is my file header
-----------------------------7d240305e4 Content-Disposition: form-data; name="__VIEWSTATE" dDwxNzc3NzMyODIxOzs+zvmqaKYoxiiqDx5fTsr37EOyJXE= -----------------------------7d240305e4 Content-Disposition: form-data; name="UploadPath" D:\temp\Uploads\ -----------------------------7d240305e4
Content-Disposition: form-data; name="UserID" Mariano -----------------------------7d240305e4 Content-Disposition: form-data; name="file"; filename="C:\archivo.asx" Content-Type: text/html <---- (first body bytes) you can see 4 fields, file, UploadPath, UserID
(i need it), and one generated by the asp.net web form, the UploadPath and UserID are generated by hidden fields on the form, so the header maybe will change on your upload code. The parser work seeking line breaks (byte value of '13') a total of 16 in my
header, the same structure always if I dont modify the upload form, the first block of bytes you receive got all you need to parse all the stream, the code is a class with a kind properties for that, thats not all my friend, you need to write the code to parse
the file, remember that if you parse after the file is uploaded you will need to rewrite all the file again, imagine that with 40Mb file, so find a way to do that on stream fly. Here you got, Mariano Public Class HeadBodyParser Private _HeadBytes As Byte()
Private _BodyBytes As Byte() Private _BoundaryBytes As Byte() Private _BoundaryTotalBytes As Integer Private _UploadPath As String Private _NombreArchivo As String Private _UserID As String Public ReadOnly Property UploadPath() As String Get Return _UploadPath
End Get End Property Public ReadOnly Property UserID() As String Get Return _UserID End Get End Property Public ReadOnly Property NombreArchivo() As String Get Return _NombreArchivo End Get End Property Public ReadOnly Property HeadBytes() As Byte() Get Return
_HeadBytes End Get End Property Public ReadOnly Property BodyBytes() As Byte() Get Return _BodyBytes End Get End Property Public ReadOnly Property BoundaryTotalBytes() As Integer Get Return _BoundaryTotalBytes End Get End Property Public ReadOnly Property
BoundaryBytes() As Byte() Get Return _BoundaryBytes End Get End Property Public Sub New(ByVal Bytes As Byte()) Dim UploadPathInicio As Integer Dim UploadPathFin As Integer Dim UserIDInicio As Integer Dim UserIDFin As Integer Dim ArchivoInicio As Integer Dim
ArchivoFin As Integer Dim x As Integer Dim Acumulador As Byte = 0 For x = 0 To Bytes.Length - 1 If Bytes.GetValue(x) = 13 Then Acumulador = Acumulador + 1 If Acumulador = 1 Then _BoundaryTotalBytes = x + 6 Dim BoundaryBytes As Byte() = New Byte(_BoundaryTotalBytes)
{} Dim z As Integer For z = 0 To _BoundaryTotalBytes - 1 BoundaryBytes(z) = Bytes(z) Next End If If Acumulador = 7 Then UploadPathInicio = x + 2 End If If Acumulador = 8 Then UploadPathFin = x - 1 Dim UploadPath As String Dim u As Integer For u = 0 To UploadPathFin
- UploadPathInicio UploadPath = UploadPath & Chr(Bytes(UploadPathInicio + u)) Next _UploadPath = Trim(UploadPath) End If If Acumulador = 11 Then UserIDInicio = x + 2 End If If Acumulador = 12 Then UserIDFin = x - 1 Dim UserID As String Dim u As Integer For
u = 0 To UserIDFin - UserIDInicio UserID = UserID & Chr(Bytes(UserIDInicio + u)) Next _UserID = Trim(UserID) End If If Acumulador = 13 Then ArchivoInicio = x + 2 End If If Acumulador = 14 Then ArchivoFin = x - 1 Dim Archivo As String Dim u As Integer For u
= 0 To ArchivoFin - ArchivoInicio Archivo = Archivo & Chr(Bytes(ArchivoInicio + u)) Next Dim arrayArchivo As String() arrayArchivo = Split(Archivo, ";") Archivo = Trim(arrayArchivo(UBound(arrayArchivo))) Dim arrayArchivoFinal As String() arrayArchivoFinal
= Split(Archivo, Chr(34)) Archivo = Trim(arrayArchivoFinal(UBound(arrayArchivoFinal) - 1)) Archivo = Path.GetFileName(Archivo) _NombreArchivo = Trim(Archivo) End If End If If Acumulador >= 16 Then Dim y As Integer Dim LHead As Integer = x + 3 Dim LBody As
Integer = Bytes.Length - LHead Dim HeadBytes As Byte() = New Byte(LHead) {} Dim BodyBytes As Byte() = New Byte(LBody) {} For y = 0 To HeadBytes.Length - 1 - 2 HeadBytes(y) = Bytes(y) Next _HeadBytes = HeadBytes For y = 0 To BodyBytes.Length - 1 BodyBytes(y)
= Bytes(LHead + y - 1) Next _BodyBytes = BodyBytes Exit Sub End If Next End Sub End Class
I must say that this has been one outstanding thread, and I appreciate everyone's posts thus far. I have one question about parsing the raw request. What happens if the end of one buffer contains only part of a boundary? For example, you are reading the request
in 5k chunks and one buffer ends with "-----------------------------7d32" and the next one begins with "b51510414". When searching the buffer for the boundary, it will not be found in either of these chunks. Parsing the raw http request is not difficult when
you have the entire request, but i'm a little confused about how to do it when it comes in chunks. I know that this is not a likely scenerio, but none the less, it could happen. Any direction would be greatly appreciated. Thanks.
Cyberfloatie
Member
102 Points
29 Posts
Re: HttpHandler or HttpModule for file upload, large files, progress indicator?
May 17, 2003 12:45 AM|LINK
ups101
Member
205 Points
41 Posts
Re: HttpHandler or HttpModule for file upload, large files, progress indicator?
May 17, 2003 10:31 AM|LINK
X0nic
Member
10 Points
2 Posts
Re: HttpHandler or HttpModule for file upload, large files, progress indicator?
May 25, 2003 08:31 PM|LINK
pfs1138
Member
10 Points
2 Posts
Re: HttpHandler or HttpModule for file upload, large files, progress indicator?
May 30, 2003 12:09 PM|LINK
ups101
Member
205 Points
41 Posts
Re: HttpHandler or HttpModule for file upload, large files, progress indicator?
Jun 04, 2003 08:39 PM|LINK
dredman
Member
10 Points
2 Posts
Re: HttpHandler or HttpModule for file upload, large files, progress indicator?
Jun 10, 2003 08:23 PM|LINK
robrichard
Member
418 Points
88 Posts
Re: HttpHandler or HttpModule for file upload, large files, progress indicator?
Jun 10, 2003 08:40 PM|LINK
www.robrichard.com
ups101
Member
205 Points
41 Posts
Re: HttpHandler or HttpModule for file upload, large files, progress indicator?
Jun 11, 2003 02:37 AM|LINK
string FullFileName = folder.FullPath + "\\" + file.Name; if (!System.IO.File.Exists(FullFileName)) {Entry ("Alert : " + FullFileName + " does not exist!");return;} ctx.Response.ContentType = "application/octet-stream"; ctx.Response.AddHeader("Content-Disposition", "attachment;filename=" + file.Name); ctx.Response.AddHeader("Content-Length", new FileInfo(FullFileName).Length.ToString()); long CurPos = 0; byte[] Buffer = new byte[524288]; int BytesRead = 0; FileStream FS = System.IO.File.Open(FullFileName,FileMode.Open,FileAccess.Read,FileShare.Read); FS.Position = CurPos; BytesRead = FS.Read(Buffer,0,Buffer.Length); CurPos = FS.Position; FS.Close(); //updating algorithm 18-05-2003 while (BytesRead > 0 && ctx.Response.IsClientConnected) { ctx.Response.OutputStream.Write(Buffer,0,BytesRead); ctx.Response.Flush(); FS = System.IO.File.OpenRead(FullFileName); FS.Position = CurPos; BytesRead = FS.Read(Buffer,0,Buffer.Length); CurPos = FS.Position; FS.Close(); }BR, RasmusM4ri4no
Member
5 Points
1 Post
Re: HttpHandler or HttpModule for file upload, large files, progress indicator?
Jun 25, 2003 11:59 AM|LINK
a_dude
Member
10 Points
1 Post
Re: HttpHandler or HttpModule for file upload, large files, progress indicator?
Jul 15, 2003 02:21 PM|LINK