Last post Mar 01, 2012 04:27 PM by johnathan398
Sep 01, 2009 05:02 PM|AmrineA|LINK
We have a web service that we've been running for months. It process reports and emails for us. It hasn't changed much over the last couple of months. However, just within the last week, we've had issues where it stops responding or runs really slowly for
requests starting outside of the server the web service is hosted on. For example, one of the web applications tries to make a request to it and it just doesn't respond or responds slowly. We have a page that we added to the web service so that we can monitor
it and it won't load up remotely for us either.
Here are some additional details:
If anyone has any idea or things to check, I will respond quickly. Today alone, it has stopped responding to outside request 5 times. Usually we run for weeks without any issues.
Sep 02, 2009 12:04 AM|mendhak|LINK
If these requests are originating outside the network, then check with your admin guys whether there's a load balancer in between or some sort of a Cisco switch that's causing all this grief.
I'd also run a trace and hit the web service a few times to see where it's slowing down. (Have you tried recycling the app pool as well?)
To run a trace, add this to the config file:
<source name="System.ServiceModel.MessageLogging" switchValue="Warning, ActivityTracing" >
<source name="System.ServiceModel" switchValue="Verbose,ActivityTracing" >
<source name="System.Runtime.Serialization" switchValue="Verbose,ActivityTracing">
type="System.Diagnostics.XmlWriterTraceListener, System, Version=126.96.36.199, Culture=neutral, PublicKeyToken=b77a5c561934e089"
Sep 02, 2009 12:18 AM|AmrineA|LINK
I'm sorry, I forgot to mention that in order for it to start responding again, we usually do an IISRESET. All of the requests are coming from within the local network and are not going through a firewall. This leads me to believe it's not network related
because after we do an IISRESET, it starts responding again for a while.
What does the tracing do? Where does it log the data to? How badly will it affect performance if we left it in there a few days?
Sep 02, 2009 09:57 AM|mendhak|LINK
The tracing creates an svclog file and it grows fast, so only do it for a few hits, not a few days!
That you have to perform an IISRESET indicates that it could be either an exception or a memory leak somewhere that's slowing down the application response.
Have a look at the performance and memory usage of the running process. You can run perfmon as well to see how it's doing. If it spikes, then there's definitely a resource problem.
Sep 02, 2009 01:41 PM|AmrineA|LINK
I'm pretty sure it's not a memory issue. The server has 32GB of memory and 8 processor cores. When this happened, the process was only consuming 3 gigs of memory and it has safely used over 20GB before. The application continue to responds quickly as if
there is no issue from the server itself. It's only outside requests that are incredibly slow or timeout. We've been monitoring perfmon and haven't seen anything out of the ordinary.
Feb 23, 2010 02:51 PM|AmrineA|LINK
Unfortunately, we're still having issues. The problem seems to be a little different than last time. Requests to the web service seem to slow down to the point where the application calling the web service ends up timing out. The slowness is even showing
up in the server itself, which was not originally the case. We have increased the executionTimeout in the web.config of the web service to 240 seconds. However, yesterday we started getting operation timed out errors. I checked and it was responding pretty
well for me when I hit our status page that we built into the web service. About 2-3 hours later though, they started coming more frequently. Eventually, we had to IISRESET the box to make it responsive again. It appears that the request to the web service
are being completed, since the data is appearing in the database. However, it's just taking the web service a long time to send a response back to the application saying that it's done. This is odd, because normally during the day, the process is fast. It
just decides to get a to a point where everything starts to slow down and the responses start timing out.
If there is anything I should be looking for, please let me know. I've added some additional logging in our web service to hopefully find something, but haven't been able to dig anything up yet. I would imagine that this machine should be able to easily
handle the number of requests were asking it to do, but I could be wrong.
Mar 01, 2012 04:27 PM|johnathan398|LINK
I am wondering if you were able to resolve this issue as I am experiencing a similar problem. Any leads you could provide would be helpful. Thanks.