Web Service stops responding to outside requests

We have a web service that we've been running for months. It process reports and emails for us. It hasn't changed much over the last couple of months. However, just within the last week, we've had issues where it stops responding or runs really slowly for requests starting outside of the server the web service is hosted on. For example, one of the web applications tries to make a request to it and it just doesn't respond or responds slowly. We have a page that we added to the web service so that we can monitor it and it won't load up remotely for us either.

Here are some additional details:

If we log into the web service box and attempt to hit our status page, it responds quickly as if there is no issue.
There are no errors in the event log that lead us to any conclusion as to the problem we are having.
We originally had the maxconnection property set to 10 in the web.config, removed it (supposedly it will set to the correct value automatically because we have the processModel of autoConfig=true), and we set it to 96. In has had the issue with all of these settings. We currently have it at 10 since that seems to have the fewest number of issue.

If anyone has any idea or things to check, I will respond quickly. Today alone, it has stopped responding to outside request 5 times. Usually we run for weeks without any issues.

Thanks,

Adam

Re: Web Service stops responding to outside requests

If these requests are originating outside the network, then check with your admin guys whether there's a load balancer in between or some sort of a Cisco switch that's causing all this grief.

I'd also run a trace and hit the web service a few times to see where it's slowing down. (Have you tried recycling the app pool as well?)

To run a trace, add this to the config file:

<system.diagnostics>
   <sources>
      <source name="System.ServiceModel.MessageLogging" switchValue="Warning, ActivityTracing" >
         <listeners>
            <add name="ServiceModelTraceListener"/>
         </listeners>
      </source>


      <source name="System.ServiceModel" switchValue="Verbose,ActivityTracing"                >
         <listeners>
            <add name="ServiceModelTraceListener"/>
         </listeners>
         </source>
         <source name="System.Runtime.Serialization" switchValue="Verbose,ActivityTracing">
            <listeners>
               <add name="ServiceModelTraceListener"/>
            </listeners>
         </source>
   </sources>
   <sharedListeners>
      <add initializeData="App_tracelog.svclog"
         type="System.Diagnostics.XmlWriterTraceListener, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
         name="ServiceModelTraceListener" traceOutputOptions="Timestamp"/>
   </sharedListeners>
</system.diagnostics>

Re: Web Service stops responding to outside requests

I'm sorry, I forgot to mention that in order for it to start responding again, we usually do an IISRESET. All of the requests are coming from within the local network and are not going through a firewall. This leads me to believe it's not network related because after we do an IISRESET, it starts responding again for a while.

What does the tracing do? Where does it log the data to? How badly will it affect performance if we left it in there a few days?

Thanks,

Adam

Re: Web Service stops responding to outside requests

The tracing creates an svclog file and it grows fast, so only do it for a few hits, not a few days!

That you have to perform an IISRESET indicates that it could be either an exception or a memory leak somewhere that's slowing down the application response.

Have a look at the performance and memory usage of the running process. You can run perfmon as well to see how it's doing. If it spikes, then there's definitely a resource problem.

Re: Web Service stops responding to outside requests

I'm pretty sure it's not a memory issue. The server has 32GB of memory and 8 processor cores. When this happened, the process was only consuming 3 gigs of memory and it has safely used over 20GB before. The application continue to responds quickly as if there is no issue from the server itself. It's only outside requests that are incredibly slow or timeout. We've been monitoring perfmon and haven't seen anything out of the ordinary.

Re: Web Service stops responding to outside requests

Unfortunately, we're still having issues. The problem seems to be a little different than last time. Requests to the web service seem to slow down to the point where the application calling the web service ends up timing out. The slowness is even showing up in the server itself, which was not originally the case. We have increased the executionTimeout in the web.config of the web service to 240 seconds. However, yesterday we started getting operation timed out errors. I checked and it was responding pretty well for me when I hit our status page that we built into the web service. About 2-3 hours later though, they started coming more frequently. Eventually, we had to IISRESET the box to make it responsive again. It appears that the request to the web service are being completed, since the data is appearing in the database. However, it's just taking the web service a long time to send a response back to the application saying that it's done. This is odd, because normally during the day, the process is fast. It just decides to get a to a point where everything starts to slow down and the responses start timing out.

If there is anything I should be looking for, please let me know. I've added some additional logging in our web service to hopefully find something, but haven't been able to dig anything up yet. I would imagine that this machine should be able to easily handle the number of requests were asking it to do, but I could be wrong.

Re: Web Service stops responding to outside requests

I am wondering if you were able to resolve this issue as I am experiencing a similar problem. Any leads you could provide would be helpful. Thanks.