I’m suddenly receiving “Recursive Locks” error since my solution upgraded to Umbraco 13.
This was not the case in Umbraco 10.
I have several background tasks running, while content editors work in Umbraco tree as well.
I have already tried to add scopes in my recurring tasks but it doesn’t seem to help.
Anyone encountered this before?
Thanks!
I also have this issue.
I update one content node per background task with Hangfire.
When I first got this error I started searching and I found an article about distributed locks.
I wrapped my save operation in this scope, but I still get this exception:
My code looks something like:
private void UpdateItem<T>(IPublishedContent? jobConfig)
{
using var scope = scopeProvider.CreateScope();
scope.WriteLock(Constants.Locks.ContentTree);
try
{
// get job configuration as IContent from IPuiblishedContent or create new one
IContent?jobConfigContentItem = contentService.GetById(jobConfig.Id);
// assign new value to config
jobConfigContentItem.SetValue("PropertyName"), "udi://.....");
contentService.SaveAndPublish(jobConfigContentItem);
}
finally
{
scope.Complete();
}
}
Try removing this line:
contentService.SaveAndPublish() calls this internally.
Now I get this exception:
Transaction (Process ID 65) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
Or the same one as before…
I’d really like to come to the bottom of this issue…
Should I create a custom semaphore for that operation?
We are seeing the same issue in our 13.7.1 instance. We have an endpoint that creates and/or updates some nodes. If that is requested while an editor is busy (or it’s requested twice at the same time) it errors out with the ‘Recursive locks not allowed’ message. Adding an explicit core scope does not solve the issue, but seems to change the error message to either the deadlock error mentioned by @gregor-tusar-sowa, or a write lock timeout. The timeout makes sense and we can circumvent that with retries, but I’d like to avoid retries on the other errors if it’s a symptom of us doing something wrong.
Online there is also inconsistent information on when it’s needed to use explicit scopes when using the ContentService. For example: the docs don’t mention scopes, but this comment mentions to always use scopes in write operations.
Any luck so far @gregor-tusar-sowa @Neirijnck?
No, we couldn’t fix this yet.
Hi,
We have multiple scheduled tasks running that modify and save content.
Here are two improvements I made:
.SaveAndPublish calls. These scopes are kept as small as possible and do not wrap large blocks of code.Additionally, I fixed a bug that seemed to trigger the exception (I was able to reproduce it).
We have a strategy in Umbraco that executes during content publishing. This strategy was running code that sent an email asynchronously. I changed the email sending process to synchronous execution, which resolved the issue.
I’m not entirely sure if these changes fully resolve the recursive lock error, but I haven’t encountered it since.
It’s still possible that this issue might occur while editors are actively working on content. That’s why I believe it’s crucial to keep the scope limited strictly to the .SaveAndPublish operations to minimize the chance of overlapping actions.
Hi All,
We are seeing a similar problem recently which may be related and seems to be triggered on publishing a node.
In the logs we can see the node is published with the message:
We then see two info logs saying:
After that we get two errors:
System.InvalidOperationException: Recursive locks not allowed
at Umbraco.Cms.Infrastructure.PublishedCache.ContentStore.Lock(WriteLockInfo lockInfo, Boolean forceGen)
at Umbraco.Cms.Infrastructure.PublishedCache.ContentStore.ReleaseLocalDb()
at Umbraco.Cms.Infrastructure.PublishedCache.PublishedSnapshotService.MainDomRelease()
at Umbraco.Cms.Core.Runtime.MainDom.OnSignal(String source)
We then see a load of info logs showing the shutdown process e.g.
Then starting back up e.g.
We then see 100’s of errors along the lines of :
System.IO.IOException: The process cannot access the file 'C:\local\Temp\UmbracoData\5d9d3c0e7934256b96a5d4d33d746bc25e4d7955\NuCache\NuCache.Media.db' because it is being used by another process.
at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
at CSharpTest.Net.IO.TransactedCompoundFile..ctor(Options options)
at CSharpTest.Net.Storage.BTreeFileStoreV2..ctor(Options options)
at CSharpTest.Net.Collections.BPlusTree`2.OptionsV2.CreateStorage()
at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheBase..ctor(BPlusTreeOptions`2 options)
at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheNone..ctor(BPlusTreeOptions`2 options)
at CSharpTest.Net.Collections.BPlusTree`2..ctor(BPlusTreeOptions`2 ioptions)
at CSharpTest.Net.Collections.BPlusTree`2..ctor(OptionsV2 optionsV2)
at Umbraco.Cms.Infrastructure.PublishedCache.DataSource.BTree.GetTree(String filepath, Boolean exists, NuCacheSettings settings, ContentDataSerializer contentDataSerializer)
at Umbraco.Cms.Infrastructure.PublishedCache.PublishedSnapshotService.MainDomRegister()
at Umbraco.Cms.Core.Runtime.MainDom.Register(Action install, Action release, Int32 weight)
at Umbraco.Cms.Infrastructure.PublishedCache.PublishedSnapshotService.<EnsureCaches>b__56_0()
at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory)
at System.Threading.LazyInitializer.EnsureInitialized[T](T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory)
at Umbraco.Cms.Infrastructure.PublishedCache.PublishedSnapshotService.EnsureCaches()
at Umbraco.Cms.Infrastructure.PublishedCache.PublishedSnapshotService.CreatePublishedSnapshot(String previewToken)
at Umbraco.Cms.Web.Common.UmbracoContext.UmbracoContext.<>c__DisplayClass11_0.<.ctor>b__0()
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
at System.Lazy`1.CreateValue()
at System.Lazy`1.get_Value()
at Umbraco.Cms.Web.Common.UmbracoContext.UmbracoContext.get_PublishedSnapshot()
at Umbraco.Cms.Web.Common.UmbracoContext.UmbracoContext.get_Content()
at Umbraco.Cms.Web.Website.Routing.UmbracoRouteValueTransformer.TransformAsync(HttpContext httpContext, RouteValueDictionary values)
at Microsoft.AspNetCore.Mvc.Routing.DynamicControllerEndpointMatcherPolicy.ApplyAsync(HttpContext httpContext, CandidateSet candidates)
at Microsoft.AspNetCore.Routing.Matching.DfaMatcher.SelectEndpointWithPoliciesAsync(HttpContext httpContext, IEndpointSelectorPolicy[] policies, CandidateSet candidateSet)
at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.<Invoke>g__AwaitMatch|10_1(EndpointRoutingMiddleware middleware, HttpContext httpContext, Task matchTask)
at StackExchange.Profiling.MiniProfilerMiddleware.Invoke(HttpContext context) in C:\projects\dotnet\src\MiniProfiler.AspNetCore\MiniProfilerMiddleware.cs:line 112
at Umbraco.Cms.Web.Common.Middleware.UmbracoRequestMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Umbraco.Cms.Web.Common.Middleware.UmbracoRequestMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Umbraco.Cms.Web.Common.Middleware.PreviewAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Umbraco.Cms.Web.Common.Middleware.UmbracoRequestLoggingMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at SeoToolkit.Umbraco.RobotsTxt.Core.Middleware.RobotsTxtMiddleware.Invoke(HttpContext context)
at SixLabors.ImageSharp.Web.Middleware.ImageSharpMiddleware.Invoke(HttpContext httpContext, Boolean retry)
at Umbraco.Forms.Web.HttpModules.ProtectFormUploadRequestsMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at MiddlewareInitializer.<>c.<<ConfigureMiddleware>b__0_0>d.MoveNext() in D:\a\1\s\sitename\Site\Startup\MiddlewareInitializer.cs:line 88
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
Lastly we noticed that when this happens the we see the log:
Acquired MainDom.
Run twice at the same time which may be the reason for the nuCache locking?
We have no scheduled tasks but quite a few editors working simultaneously.
Hi Craig,
For the recursive locks there is a fix set to release in 13.10.0: When multiple editors are adding content at the same time, a "Recursive locks not allowed" error is thrown · Issue #19338 · umbraco/Umbraco-CMS · GitHub.
For the other issues, it’s difficult to pinpoint where the underlying issue is. The locked file errors on NuCache do point to some configuration issues, are you running on Azure and do you have the right configuration set for that?
As for the MainDom logs, those seem normal. One log says ‘Acquiring’ and the other ‘Acquired’, so it’s not actually the same log twice.
Thanks Rowan,
The sites are running on Azure and do have the 3 recommended setting on production:
"Global": { "MainDomLock": "FileSystemMainDomLock" },
"Hosting": {
"LocalTempStorageLocation": "EnvironmentTemp",
"Debug": false
},
"Examine": { "LuceneDirectoryFactory": "TempFileSystemDirectoryFactory" }
We have multiple sites which have all been running well for a very long time until this latest umbraco update. Although the update was live for around a week before we seen any problems, then two sites went down on the same day. We do quite a bit of custom indexing on publish so that could potentially be slowing things down a little making it more likely to trigger.
Thanks for your help, we have been googling and testing for a few days now to try and find a solution but never came across that article.