Hey,
We’re running Umbraco 15.4.3 and experiencing intermittent issues where the site stops responding. The logs contain several references to “An existing connection was forcibly closed by the remote host.” Has anyone encountered this before? HybridCache?
I can’t see any indications that the resource capacity is insufficient. The site is relatively small, with just over 700 pages.
This is the Azure resource setup:
- Windows webapp, Standard S1
- Azure SQL, Standard S2 (50DTUs)
This is the exception that keeps appearing in the logs whenever the issue occurs:
Microsoft.Data.SqlClient.SqlException (0x80131904): Snix_PreLoginBeforeSuccessfulWrite (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)
---> System.ComponentModel.Win32Exception (10054): An existing connection was forcibly closed by the remote host.
at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, SqlCommand command, Boolean callerHasConnectionLock, Boolean asyncClose)
at Microsoft.Data.SqlClient.TdsParserStateObject.SNIWritePacket(PacketHandle packet, UInt32& sniError, Boolean canAccumulate, Boolean callerHasConnectionLock, Boolean asyncClose)
at Microsoft.Data.SqlClient.TdsParserStateObject.WriteSni(Boolean canAccumulate)
at Microsoft.Data.SqlClient.TdsParserStateObject.WritePacket(Byte flushMode, Boolean canAccumulate)
at Microsoft.Data.SqlClient.TdsParser.SendPreLoginHandshake(Byte[] instanceName, SqlConnectionEncryptOption encrypt, Boolean integratedSecurity, String serverCertificateFilename)
at Microsoft.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, TimeoutTimer timeout, SqlConnectionString connectionOptions, Boolean withFailover)
at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, TimeoutTimer timeout, Boolean withFailover)
at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool, Func`3 accessTokenCallback)
at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
at NPoco.Database.OpenSharedConnectionImp(Boolean isInternal)
at Umbraco.Cms.Infrastructure.Scoping.Scope.get_Database()
at Umbraco.Cms.Infrastructure.HybridCache.Persistence.DatabaseCacheRepository.GetContentSourceAsync(Guid key, Boolean preview)
at Umbraco.Cms.Infrastructure.HybridCache.Services.DocumentCacheService.<>c__DisplayClass18_0.<<GetNodeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.Extensions.Caching.Hybrid.Internal.DefaultHybridCache.StampedeState`2.BackgroundFetchAsync()
at Microsoft.Extensions.Caching.Hybrid.Internal.DefaultHybridCache.StampedeState`2.<UnwrapReservedAsync>g__AwaitedAsync|18_0(ILogger log, Task`1 task)
at Umbraco.Cms.Infrastructure.HybridCache.Services.DocumentCacheService.GetNodeAsync(Guid key, Boolean preview)
at Umbraco.Cms.Infrastructure.HybridCache.Services.DocumentCacheService.GetByIdAsync(Int32 id, Nullable`1 preview)
at Umbraco.Cms.Infrastructure.HybridCache.DocumentCache.GetByIdAsync(Int32 id, Nullable`1 preview)
at Umbraco.Cms.Core.Routing.PublishedRouter.<>c__DisplayClass24_0.<FindDomain>g__IsPublishedContentDomain|0(Domain domain)
at System.Linq.Enumerable.ArrayWhereIterator`1.ToList(ReadOnlySpan`1 source, Func`2 predicate)
at System.Linq.Enumerable.ListWhereIterator`1.ToList()
at Umbraco.Cms.Core.Routing.PublishedRouter.FindAndSetDomain(IPublishedRequestBuilder request)
at Umbraco.Cms.Core.Routing.PublishedRouter.RouteRequestAsync(IPublishedRequestBuilder builder, RouteRequestOptions options)
at Umbraco.Cms.Web.Website.Routing.UmbracoRouteValueTransformer.RouteRequestAsync(IUmbracoContext umbracoContext)
at Umbraco.Cms.Web.Website.Routing.UmbracoRouteValueTransformer.TransformAsync(HttpContext httpContext, RouteValueDictionary values)
at Microsoft.AspNetCore.Mvc.Routing.DynamicControllerEndpointMatcherPolicy.ApplyAsync(HttpContext httpContext, CandidateSet candidates)
at Microsoft.AspNetCore.Routing.Matching.DfaMatcher.SelectEndpointWithPoliciesAsync(HttpContext httpContext, IEndpointSelectorPolicy[] policies, CandidateSet candidateSet)
at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.<Invoke>g__AwaitMatch|10_1(EndpointRoutingMiddleware middleware, HttpContext httpContext, Task matchTask)
at StackExchange.Profiling.MiniProfilerMiddleware.Invoke(HttpContext context) in C:\projects\dotnet\src\MiniProfiler.AspNetCore\MiniProfilerMiddleware.cs:line 112
at Umbraco.Cms.Web.Common.Middleware.UmbracoRequestMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Umbraco.Cms.Web.Common.Middleware.UmbracoRequestMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Umbraco.Cms.Web.Common.Middleware.PreviewAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Umbraco.Cms.Web.Common.Middleware.UmbracoRequestLoggingMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at SixLabors.ImageSharp.Web.Middleware.ImageSharpMiddleware.Invoke(HttpContext httpContext, Boolean retry)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at SixLabors.ImageSharp.Web.Middleware.ImageSharpMiddleware.Invoke(HttpContext httpContext, Boolean retry)
at Umbraco.Cms.Api.Management.Middleware.BackOfficeAuthorizationInitializationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()
at Umbraco.Cms.Api.Management.Middleware.BackOfficeAuthorizationInitializationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()
ClientConnectionId:11712f6b-a782-407c-8264-123e3f5277f9
Error Number:10054,State:0,Class:20
ClientConnectionId before routing:7bec5b2a-1319-4fd9-a1a6-e5db09e358ee
Routing Destination:fd5c2b60a892.tr51805.westeurope1-a.worker.database.windows.net,11038