EDIT: I have isolated the problem to HttpClient, repro program at
When I upload a 500MB file using a StreamContent, my memory footprint increases instantly to ~600MB as the file is POSTed - this should be streaming and not buffer in RAM, right?
Here is the code I am using
var httpCLient = new HttpClient();
var fileStream = File.OpenRead(@"C:\halfgigfile.zip");
var request = new HttpRequestMessage(HttpMethod.Post, "http://localhost:3000/home/upload")
{
Content = new StreamContent(fileStream)
};
httpCLient.SendAsync(request).ContinueWith(s => { Console.WriteLine("sent"); fileStream.Close(); });
Console.WriteLine("sending");
It seems to buffer the moment the upload starts (rather than incrementing as the ContentStream is read)
Is this expected/desireable behaviour or a bug? How can I stream send an Http Request?
To reproduce
create a file c:\halfgigfile.zip (which should be ~500MB)
set multiple startup projects Api and HttpClientMemoryTest, and start them. It will display it's memory footprint in the console window
press 'Enter' and it will try to send the halfgigfile.zip to the Api web site. The footprint will increase
I have written a very simple Http Proxy using WebApi, which passes the incoming request to HttpClient.SendAsync(), . This is Self Hosted with TransferMode.Streamed enabled.
No where in my code does the request get buffered or accessed syncronously, yet if I POST a 500MB file via my proxy, the memory consumption grows as the file is uploaded to the WebApi.
Am I doing something wrong which is causing buffering, or is it a problem with WebApi? I really hope this can work, as streaming pass-through proxies are one of the things that I'm most envious of node.js land!
My controller (and repro project) is at: https://github.com/mcintyre321/WebAPI-Proxy/blob/master/WebApiProxy/ProxyController.cs
If I remember correctly HttpClient naturally buffers the content if you try and read something out of the Content property. In client code you can get around it by using the Content.CopyToAsync. However in yoru case you need to wrap up the reading of request
Content in a new HttpContent without actually reading the stream. I think you would be able to do this with Henrik's ActionofStreamContent class. http://blogs.msdn.com/b/henrikn/archive/2012/02/17/push-and-pull-streams-using-httpclient.aspx
In the lambda you pass to the ActionOfStreamContent constructor you would do a CopyAsync from the request.Content into the new request.Content that you send to the origin server.
In my code, I think all access to the .Content (except the .Headers) is done via async methods - I pass the HttpWebRequest that comes in via WebAPI straight into HttpClient without consuming the data.
On .Net 4.5 the default buffering of all client requests is turned off so this scenario will work but in .Net 4 it does buffer. The issue is that HttpWebRequest (which lies under HttpClient) buffers the request on .Net 4 due to the default setting on HttpWebRequest.
There is a way to control the setting in HttpWebReqeust using the AllowWriteStreamBuffering property but that is for variety of reasons not exposed to HttpClient.
Thanks Henrik - this explains a lot! I haven't got the 4.5 CTP installed in case I mess up my development machine, but I will definitely be testing this out when it is gold. The ability to write an HttpProxy in 10 lines of code is a big thing I am really
looking forwards to!
mcintyre321
0 Points
12 Posts
HttpClient buffering when POSTing data
Apr 17, 2012 04:32 PM|LINK
EDIT: I have isolated the problem to HttpClient, repro program at
When I upload a 500MB file using a StreamContent, my memory footprint increases instantly to ~600MB as the file is POSTed - this should be streaming and not buffer in RAM, right?
Here is the code I am using
var httpCLient = new HttpClient(); var fileStream = File.OpenRead(@"C:\halfgigfile.zip"); var request = new HttpRequestMessage(HttpMethod.Post, "http://localhost:3000/home/upload") { Content = new StreamContent(fileStream) }; httpCLient.SendAsync(request).ContinueWith(s => { Console.WriteLine("sent"); fileStream.Close(); }); Console.WriteLine("sending");To reproduce
I have written a very simple Http Proxy using WebApi, which passes the incoming request to HttpClient.SendAsync(), . This is Self Hosted with TransferMode.Streamed enabled.
No where in my code does the request get buffered or accessed syncronously, yet if I POST a 500MB file via my proxy, the memory consumption grows as the file is uploaded to the WebApi.
Am I doing something wrong which is causing buffering, or is it a problem with WebApi? I really hope this can work, as streaming pass-through proxies are one of the things that I'm most envious of node.js land!
My controller (and repro project) is at: https://github.com/mcintyre321/WebAPI-Proxy/blob/master/WebApiProxy/ProxyController.cs
Darrel Mille...
Member
176 Points
56 Posts
Re: HttpClient buffering when POSTing data
Apr 17, 2012 05:21 PM|LINK
If I remember correctly HttpClient naturally buffers the content if you try and read something out of the Content property. In client code you can get around it by using the Content.CopyToAsync. However in yoru case you need to wrap up the reading of request Content in a new HttpContent without actually reading the stream. I think you would be able to do this with Henrik's ActionofStreamContent class. http://blogs.msdn.com/b/henrikn/archive/2012/02/17/push-and-pull-streams-using-httpclient.aspx
In the lambda you pass to the ActionOfStreamContent constructor you would do a CopyAsync from the request.Content into the new request.Content that you send to the origin server.
mcintyre321
0 Points
12 Posts
Re: HttpClient buffering when POSTing data
Apr 23, 2012 08:15 AM|LINK
In my code, I think all access to the .Content (except the .Headers) is done via async methods - I pass the HttpWebRequest that comes in via WebAPI straight into HttpClient without consuming the data.
Henrik Fryst...
Member
96 Points
13 Posts
Microsoft
Re: HttpClient buffering when POSTing data
Apr 27, 2012 01:23 PM|LINK
On .Net 4.5 the default buffering of all client requests is turned off so this scenario will work but in .Net 4 it does buffer. The issue is that HttpWebRequest (which lies under HttpClient) buffers the request on .Net 4 due to the default setting on HttpWebRequest. There is a way to control the setting in HttpWebReqeust using the AllowWriteStreamBuffering property but that is for variety of reasons not exposed to HttpClient.
Hope this helps,
Henrik
[1] http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.allowwritestreambuffering.aspx
mcintyre321
0 Points
12 Posts
Re: HttpClient buffering when POSTing data
Apr 27, 2012 01:52 PM|LINK
Thanks Henrik - this explains a lot! I haven't got the 4.5 CTP installed in case I mess up my development machine, but I will definitely be testing this out when it is gold. The ability to write an HttpProxy in 10 lines of code is a big thing I am really looking forwards to!
mcintyre321
0 Points
12 Posts
Re: HttpClient buffering when POSTing data
Apr 27, 2012 01:54 PM|LINK
Do you have some documentation on this? If I look at the page you linked to, and switch to 4.5, it still says the default is to buffer...
manfred.stey...
Member
35 Points
57 Posts
Re: HttpClient buffering when POSTing data
Aug 24, 2012 05:22 PM|LINK
Hi,
I have the same problem but I'm using .NET 4.5 and fetched the current version of web-api with NuGet.
Can you provide us with an sample that shows how to use streaming?
Wishes,
Manfred