Categories

Fix SharePoint 2013 Distributed Cache Timeouts

Fix SharePoint 2013 Distributed Cache Timeouts

SharePoint 2013 uses the AppFabric Caching Service to centralize all caches that are used by SharePoint. The distributed cache is used by many features including Authentication, Newsfeed, Security, Page Loading, Workflows, and more. While Microsoft claims that SharePoint’s use of the Distributed Cache greatly increases performance, the service is also quite unreliable and difficult to troubleshoot. Hopefully they will write more documentation and improve the product with the release of SP1.

There is a good TechNet guide with Powershell commands that you might find useful if you ever need to change any of the AppFabric configuration. The article does not cover increasing Request Timeouts so I will review that in this blog post.

A few weeks ago we ran into a problem where one of our web parts stopped working when users were not on the local network. After investigating ULS logs we tracked down the issue to Distributed Cache timeouts.

The web part, a timer application, was developed with heavy use of the View State. As a result, as the View State Cache continued to grow, we began seeing intermittent Timeout errors in the logs.

Unexpected error occurred in method 'Put' , usage 'SPViewStateCache' - Exception 'Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode:SubStatus:The request timed out.. Additional Information : The client was trying to communicate with the server : net.tcp://SP2013Dev.DMC.local:22233 at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody, RequestBody reqBody) at Microsoft.ApplicationServer.Caching.DataCache.InternalPut(String key, Object value, DataCacheItemVersion oldVersion, TimeSpan timeout, DataCacheTag[] tags, String region, IMonitoringListener listener) at Microsoft.ApplicationServer.Caching.DataCache.<>c__DisplayClass25. b__24() at Microsoft.ApplicationServer.Caching.DataCache.Put(String key, Object value, TimeSpan timeout) at Microsoft.SharePoint.DistributedCaching.SPDistributedCache.Put(String key, Object value)'. Unexpected error occurred in method 'Put' , usage 'SPViewStateCache' - Exception 'Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode:SubStatus:The connection was terminated, possibly due to server or network problems or serialized Object size is greater than MaxBufferSize on server. Result of the request is unknown. ---> System.TimeoutException: The socket was aborted because an asynchronous receive from the socket did not complete within the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.IO.IOException: The read operation failed, see inner exception. ---> System.TimeoutException: The socket was aborted because an asynchronous receive from the socket did not complete within the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.Sockets.SocketException: The I/O operation has been aborted because of either a thread exit or an application request at System.ServiceModel.Channels.SocketConnection.HandleReceiveAsyncCompleted() at System.ServiceModel.Channels.SocketConnection.OnReceiveAsync(Object sender, SocketAsyncEventArgs eventArgs) -

We also had a problem with starting SP 2013 workflows which we were able to trace down to Distributed Logon Token Cache timeouts.

Unexpected error occurred in method 'GetObject' , usage 'Distributed Logon Token Cache' - Exception 'Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode:SubStatus:The request timed out.. Additional Information : The client was trying to communicate with the server : net.tcp://SP2013Dev.DMC.local:22233 at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody, RequestBody reqBody) at Microsoft.ApplicationServer.Caching.DataCache.InternalGet(String key, DataCacheItemVersion& version, String region, IMonitoringListener listener) at Microsoft.ApplicationServer.Caching.DataCache.<>c__DisplayClass49.b__48() at Microsoft.SharePoint.DistributedCaching.SPDistributedCache.GetObject(String key)'.

To fix these errors, you need to increase the Request Timeout limit for the relevant cache. You can do so using the following commands.

$LogonTokenCache = Get-SPDistributedCacheClientSetting -ContainerType DistributedLogonTokenCache $LogonTokenCache.RequestTimeout = 300 Set-SPDistributedCacheClientSetting -ContainerType DistributedLogonTokenCache -DistributedCacheClientSettings $LogonTokenCache 

$ViewStateCache = Get-SPDistributedCacheClientSetting -ContainerType DistributedViewStateCache $ViewStateCache.RequestTimeout = 300 Set-SPDistributedCacheClientSetting -ContainerType DistributedViewStateCache -DistributedCacheClientSettings $ViewStateCache

By running Get-SPDistributedCacheClientSetting -ContainerType CacheName you are able to see what the current request timeout is set to. By default, Microsoft sets the value of the RequestTimeout to 20 milliseconds. As a result, if the server is under any significant stress you will start seeing a lot of misses on calls to the cache.

PS C:\Users\user> $viewstatecache ChannelInitializationTimeout : 60000 ConnectionBufferSize : 131072 MaxBufferPoolSize : 268435456 MaxBufferSize : 8388608 MaxOutputDelay : 2 ReceiveTimeout : 60000 ChannelOpenTimeOut : 20 RequestTimeout : 300 MaxConnectionsToServer : 99

Learn more about DMC's SharePoint services.

Comments

chris
# chris
Since AppFabric CU7 didn't fix my problems, I opened a case with Microsoft Support's SharePoint Admin team in December 2015 on this. The ErrorCode:SubStatus problems were solved by setting MaxConnectionsToServer to 1 for all caches. Our token cache had it set to 100. Not good! Reference: https://technet.microsoft.com/en-us/library/jj219613.aspx?f=255&MSPPError=-2147217396#finetune
Rockie
Hi Jerry!

Thank You very much.
Have this problem, it hangs my server, Your method does not resolve it totally, but reduces the problem so I can return to work. Thank You very much!

Best regards,
Gennady
Daniel
# Daniel
Hi


Good post will test this out - I am having some weird issues on my sp2013 dev server.. REST services not working on my my appsdev webapp nor is the people picker. I VS2013 cannot connect to the server on project create time...

Unexpected error occurred in method 'GetObject' , usage 'Distributed Logon Token Cache' - Exception 'Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode:SubStatus:The request timed out.. Additional Information : The client was trying to communicate with the server : net.tcp://SharePointDev.datelgroup.local:22233

Post a comment

Name (required)

Email (required)

Website

CAPTCHA image
Enter the code shown above:

Categories

Related Blog Posts