Using Web API Beta - we've recently been seeing appdomain recycling (and on one occasion IIS Rapid Fail Protection shutting down the app pool) due to unhandled exceptions. I have seen this in a couple of flavours - both have the raising of an HttpException
with "The remote host closed the connection". Sometimes I get this on it's own, and other times inside an AggregateException - leading to a process termination.
Has anyone seen this and know of any ways to address it? Is it a known issue? Could it be somethine I am doing wrong? Any suggestions greatly appreciated! This is a bit of a road-block for us. If this is as it appears, that clients connections dropping causes
Web API to crash - then that's pretty bad!
Typical event log entry for the AggregateException:
An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/1/ROOT/{Censored}
Process ID: 8884
Exception: System.AggregateException
Message: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread.
StackTrace: at System.Threading.Tasks.TaskExceptionHolder.Finalize()
InnerException: System.Web.HttpException
Message: The remote host closed the connection. The error code is 0x80070057.
StackTrace: at System.Web.Http.WebHost.HttpControllerHandler.EndProcessRequest(IAsyncResult result)
at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)
From the exception, I understood that it is because the Web API is not responding in a given period of time and probably that is causing the issue. Don't you have any reTry Logic such that if there is any exception then it should be connecting back and get
the required data. Even if it fails after few retries then you have some problem with the Web API itself. So, an Exception has to be catched in your application, I am unsure where that could be but mostly it is the place where you are calling the API and the
moment if there is any Exception, let the Logic should be in such a way that it should reconnect. I am sure it's a problem with the Exception handling somewhere in your code.
@Sri - Are you saying I should have retry logic in my client? This code is for a public API, I have no guarantee of control over the client code - and in any case I don't think a server side crash is acceptable. We have a start-up cost that isn't neglible.
This has to be handled on the server. The stack trace suggests (to me) the exception is being raised (and unhandled) in web api code.
I am running debug-diag to see if I can catch it in the act...
I didn't try Application_Error! I don't know why. Doh! For some reason I assumed it wouldn't apply in the case of Web API. I'll wire it up and see if it catches it.
@Sri - Are you saying I should have retry logic in my client? This code is for a public API, I have no guarantee of control over the client code - and in any case I don't think a server side crash is acceptable.
REST is modeled after the web and so clients are naïve if they don't anticipate failures. That's why there's a 500 status code :)
It's the logical equivalent of a human getting an error in a web page and hitting refresh and if there's still an error just trying again later.
This is correct that an unobserved exception in TPL will cause the take down of application domain in .NET 4.0. I think james you are correct. This seems to be
bug.
"And whoever is removed away from the Fire and admitted to Paradise, he indeed is successful." (The Holy Quran)
Excellent Windows VPS Hosting Imran Baloch MVP, MVB, MCP, MCTS, MCPD
@Sri - Are you saying I should have retry logic in my client? This code is for a public API, I have no guarantee of control over the client code - and in any case I don't think a server side crash is acceptable.
REST is modeled after the web and so clients are naïve if they don't anticipate failures. That's why there's a 500 status code :)
It's the logical equivalent of a human getting an error in a web page and hitting refresh and if there's still an error just trying again later.
I think you've missed the point... the problem is that the client disconnecting shouldn't cause an app pool recycle... :) I've confirm this is exactly what happens. I put a break-point in a handler in the return phase of the pipeline (i.e. after the action
has generated a response) and crucially after the exception filter handling. I then hit this in the debugger by making a request from fiddler. Once hit, I killed fiddler and pressed F5 - and bang! it took down the app domain.
I think you've missed the point... the problem is that the client disconnecting shouldn't cause an app pool recycle... :) I've confirm this is exactly what happens. I put a break-point in a handler in the return phase of the pipeline (i.e. after the action
has generated a response) and crucially after the exception filter handling. I then hit this in the debugger by making a request from fiddler. Once hit, I killed fiddler and pressed F5 - and bang! it took down the app domain.
Pretty sure this is a web-api bug.
Yea, I was speaking more generally and wasn't suggesting the client should be changed to accomodate a specific bug on the server.
And for certain -- client calls triggering a crash in the server such that the server goes down is a bug :)
james-world
Member
103 Points
50 Posts
Unhandled exceptions in Web API
May 29, 2012 03:06 PM|LINK
Using Web API Beta - we've recently been seeing appdomain recycling (and on one occasion IIS Rapid Fail Protection shutting down the app pool) due to unhandled exceptions. I have seen this in a couple of flavours - both have the raising of an HttpException with "The remote host closed the connection". Sometimes I get this on it's own, and other times inside an AggregateException - leading to a process termination.
Has anyone seen this and know of any ways to address it? Is it a known issue? Could it be somethine I am doing wrong? Any suggestions greatly appreciated! This is a bit of a road-block for us. If this is as it appears, that clients connections dropping causes Web API to crash - then that's pretty bad!
Typical event log entry for the AggregateException:
An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/1/ROOT/{Censored}
Process ID: 8884
Exception: System.AggregateException
Message: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread.
StackTrace: at System.Threading.Tasks.TaskExceptionHolder.Finalize()
InnerException: System.Web.HttpException
Message: The remote host closed the connection. The error code is 0x80070057.
StackTrace: at System.Web.Http.WebHost.HttpControllerHandler.EndProcessRequest(IAsyncResult result)
at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)
mcirkven
Member
2 Points
1 Post
Re: Unhandled exceptions in Web API
May 30, 2012 07:40 AM|LINK
Did you try to handle Application_Error in Global.asax?
protected void Application_Error(object sender, EventArgs e) { Exception ex = Server.GetLastError(); logger.Fatal(ex); Server.ClearError(); }nadellas
Participant
1218 Points
299 Posts
Re: Unhandled exceptions in Web API
May 30, 2012 01:34 PM|LINK
Hi
From the exception, I understood that it is because the Web API is not responding in a given period of time and probably that is causing the issue. Don't you have any reTry Logic such that if there is any exception then it should be connecting back and get the required data. Even if it fails after few retries then you have some problem with the Web API itself. So, an Exception has to be catched in your application, I am unsure where that could be but mostly it is the place where you are calling the API and the moment if there is any Exception, let the Logic should be in such a way that it should reconnect. I am sure it's a problem with the Exception handling somewhere in your code.
It's just a rough guess.
thanks
Srini
james-world
Member
103 Points
50 Posts
Re: Unhandled exceptions in Web API
May 30, 2012 02:08 PM|LINK
@Sri - Are you saying I should have retry logic in my client? This code is for a public API, I have no guarantee of control over the client code - and in any case I don't think a server side crash is acceptable. We have a start-up cost that isn't neglible. This has to be handled on the server. The stack trace suggests (to me) the exception is being raised (and unhandled) in web api code.
I am running debug-diag to see if I can catch it in the act...
james-world
Member
103 Points
50 Posts
Re: Unhandled exceptions in Web API
May 30, 2012 02:09 PM|LINK
@mcirkven
I didn't try Application_Error! I don't know why. Doh! For some reason I assumed it wouldn't apply in the case of Web API. I'll wire it up and see if it catches it.
BrockAllen
All-Star
27432 Points
4890 Posts
MVP
Re: Unhandled exceptions in Web API
May 30, 2012 02:16 PM|LINK
REST is modeled after the web and so clients are naïve if they don't anticipate failures. That's why there's a 500 status code :)
It's the logical equivalent of a human getting an error in a web page and hitting refresh and if there's still an error just trying again later.
DevelopMentor | http://www.develop.com
thinktecture | http://www.thinktecture.com/
imran_ku07
All-Star
45785 Points
7698 Posts
MVP
Re: Unhandled exceptions in Web API
May 30, 2012 03:40 PM|LINK
This is correct that an unobserved exception in TPL will cause the take down of application domain in .NET 4.0. I think james you are correct. This seems to be bug.
Excellent Windows VPS Hosting
Imran Baloch MVP, MVB, MCP, MCTS, MCPD
james-world
Member
103 Points
50 Posts
Re: Unhandled exceptions in Web API
May 30, 2012 03:52 PM|LINK
I think you've missed the point... the problem is that the client disconnecting shouldn't cause an app pool recycle... :) I've confirm this is exactly what happens. I put a break-point in a handler in the return phase of the pipeline (i.e. after the action has generated a response) and crucially after the exception filter handling. I then hit this in the debugger by making a request from fiddler. Once hit, I killed fiddler and pressed F5 - and bang! it took down the app domain.
Pretty sure this is a web-api bug.
BrockAllen
All-Star
27432 Points
4890 Posts
MVP
Re: Unhandled exceptions in Web API
May 30, 2012 04:00 PM|LINK
Yea, I was speaking more generally and wasn't suggesting the client should be changed to accomodate a specific bug on the server.
And for certain -- client calls triggering a crash in the server such that the server goes down is a bug :)
DevelopMentor | http://www.develop.com
thinktecture | http://www.thinktecture.com/
hpinsley
Member
17 Points
19 Posts
Re: Unhandled exceptions in Web API
May 30, 2012 06:53 PM|LINK
Just confirming that I'm seeing the same sort of behavior as James.