Last post Apr 26, 2018 06:41 PM by rlangham___
Dec 11, 2017 09:08 AM|mennoh|LINK
I have created a small asp.net core API, which has one endpoint that resolves OpenGraph tags and sends back its results.
You can get the full source code here:
The problem I am facing:
If I run this asp.net core api locally (not using docker) and I run the attached stress tester, all works great. The stress tester ends correctly and memory usage is constant.
However, when I use docker locally and I run the attached stresstester, the docker container will freeze. (as if there is a deadlock happening)
If I run this docker container on Rancher, it will endlessly eat up all memory (as if no garbage collection is being done). (at least it does not freeze)
What I have done so far:
Tried to set:
Did not change anything
Tried to set a max memory on the docker instance so that the garbage collector knows it is not allowed to use up all memory (which had a bug in asp.net core which was fixed in 2.0.3), but this does not change anything (it will simply crash the container
as it will be OOM).
(also did numerous unrelated things, that did not resolve this isssue)
Related issues, that did not resolve my issue:
To run the stresstester:
dotnet .\StressTester.dll "http://localhost:xxxx" where xxxx is the port number assigned to your instance
Any ideas? I am left with none =(
Win 10, dev machine:
Output of `dotnet --info`
.NET Command Line Tools (2.1.2)
Commit SHA-1 hash: 5695315371
OS Name: Windows
OS Version: 10.0.16299
OS Platform: Windows
Base Path: C:\Program Files\dotnet\sdk\2.1.2\
Microsoft .NET Core Shared Framework Host
Version : 2.0.3
Build : a9190d4a75f4a982ae4b4fa8d1a24526566c69df
Dec 11, 2017 02:59 PM|Radomir|LINK
Linux container? I'd start with simple stuff first.. like 1) eliminate container(s) and if Linux just stand up a VM or use a physical box to see if there's any difference (I kinda don't expect much but worth trying).. 2) try Win container (if it repros
there it'd be easier to troubleshoot)...
just a suggestion based on some prior experiences - eliminate all logging
for testing purposes and particularly NLog (if you're using it)... then test again...
Dec 11, 2017 04:21 PM|mennoh|LINK
Thank you for your suggestions. The results are as follows:
1) Linux (Ubuntu 16.04.3 LTS) with .net Core (2.0.2) running in a VM => Same behaviour as on rancher (looks as if GC is not doing anything, linear increase of memory consumption)
2) Windows Container => runs correctly, just like running on windows nativly
Regarding NLog or any other loggin framework, this is disabled/not part of this app.
Dec 11, 2017 05:11 PM|Radomir|LINK
Ouch that'll be tough to track down... technically we'd do a memory dump and go through it to see what's going on.. not "fun" (nor easy) for sure.. You can start with
https://github.com/Microsoft/ProcDump-for-Linux and see if you can make any headway...
Dec 11, 2017 05:23 PM|mennoh|LINK
Is it safe to say that the code is not the problem?
If this is so, where can I file a bug for the asp.net core team? (as they already worked on the other referenced items which are related to asp.net core running in a linux container and having OOM issues and not triggering the GC)
Already analyzed memory usage on a windows host and there wasn't anything to find.
Dec 11, 2017 05:54 PM|Radomir|LINK
Definitely file an issue on GitHub (for asp.net).
IMHO, at this point nothing is "safe to say" :) but I do not think that should prevent you from filing an issue there... so I'd say go for it definitely.
If you can reproduce outside Docker (as you seem to be able to), then I'd suggest not to frame this as docker-specific when you go file an issue on GitHub.. it may confuse things. I'd say something like "on Linux, both inside a docker container as well
as on standalone VM...."
PS. do you repro this with Hello World app? that'd be interesting to see (and trivial to check).
Apr 26, 2018 06:41 PM|rlangham___|LINK
Any update on this? I believe I am seeing similar issue. I am doing load testing and my current api deployed using .net 4.5, Windows, IIS beats the heck out of my async .net core api on linux ec2 or linux dockere. It seems to get stuck in some lock state
or contention where my api requests are not being handled responsively at all. Even to the point where my aws health check fails and it terminates the instance. My api endpoints are simply pass thru to a sql server database and I have fully async code
from the controller to the database.