Last post Dec 07, 2018 04:12 PM by Koenemoes
Sep 20, 2018 12:59 PM|shahiid.iqbal|LINK
We are experiencing an error "The state server has closed an expired TCP/IP connection. The IP address of the client is 127.0.0.1. The expired Read operation began at 09/14/2018 09:01:31." randomly on random servers.
Currently, the problem we are facing on the server OS: Windows 2008 R2 and IIS 7.5
Following are the settings in web.config for stateserver.
<sessionState mode="StateServer" stateNetworkTimeout="120" timeout="120"/>
Same code is running perfectly on all servers except few of them.
Can you please help me to figure this issue out?
Sep 20, 2018 01:35 PM|mgebhard|LINK
Probably related to your other thread COM+ applications where removed from a few servers.
Sep 21, 2018 02:11 AM|Brando ZWZ|LINK
As far as I know, if either the Web server or the state server is under very high CPU utilization (close to 100 percent), a TCP/IP operation can sometimes take more than 120 seconds and thus is cancelled before it has the chance to finish. As a result, one
of the above-mentioned events is logged, and the originating client request fails.
So I suggest you could firstly check your server to make sure there are something which will is under very high CPU utilization (close to 100 percent) by using task manager.
More details about how to use task managerm, you could refer to below article.
Dec 07, 2018 04:12 PM|Koenemoes|LINK
We have had this issue for a couple of months at seemingly random times and levels of severity. Last week I finally got the extra hint from a seemingly unrelated event.
We also saw a 100% CPU spike on at least 1 IIS app pool which brings down the whole server after a while.
IISRESET was our only solution and this would work.
Long story short: it happened during WSUS updates of our dc's. The AD resync after rebooting them took longer than the configured time window of 1 hour.
So the situation could arise that both DNS servers were offline -> so also no AD
State server was configured as "localhost", 127.0.0.1 would not have had this issue. I check by disassembling webengine4.dll. It returns something like "connection closed by remote host" HRESULT 72746.
But system.web interprets this as the session being locked by a concurrent query and keeps retrying at full speed. This created an infinite loop of failing socket connections PER SESSION!
And that is why cpu got to 100%. It fills up the request queue, still waiting for the session data and then fajils completely with http 503 for all appools.
Our logs also showed connection issues to our sql because ad was down.
We were so focussed on the 100% cpu being the cause, but it was only a symptom. The fact that this error logged in event viewer was very misleading in my case.
In the end it was a quick 1 minute fix of the DC. -> no more emergency wake up calls for me :)
This was a perfect storm of timing events and small config mistakes which led to misleading logs and cpu spike by a asp.net bug...? Why keep retrying to read the session data if it won't work due to dns. Just end the webrequest there and log sometinhg like:
NameResolution Failed instead of connection timeout