Limits on concurrent API requests for Umbraco 13

Questions
1

Hi,

We’re using Umbraco 13.5.3 in our web application, in which we’re exposing an endpoint through a Surface Controller. We noticed and verified, that receiving 50 or more concurrent requests through this API (even though the API does not do anything and returns right away) these requests bring the site down and make it unloadable.

Is this a known issue in Umbraco 13? If so, is there a patch that we can apply?
Are there any known limits on concurrent requests for Umbraco 13?
Is there a specific configuration that can help improve performance under load?
What troubleshooting steps can be taken to further investigate this issue?

Any suggestions or advice to tackle this issue is very appreciated.

Thank you,
Sam

2

I don’t know if it makes a difference, but you can start to update Umbraco to 13.8.1. That will also fix a number of vulnerabilities.

3

As Luuk notes, you should be on the very latest version at least. This is not a known bug though. I would suspect your server setup is probably not able to handle the load.

  • Is this a known issue in Umbraco 13? If so, is there a patch that we can apply?
    • Not a known issue, but upgrade anyway
  • Are there any known limits on concurrent requests for Umbraco 13?
    • It completely depends on how you set up your infrastructure
  • Is there a specific configuration that can help improve performance under load?
    • Nothing specific, you need to load test, identify bottle necks and tweak accordingly, for both Umbraco and general .NET apps - best practices documentation for Umbraco could help: Good practice and defaults | Umbraco CMS
  • What troubleshooting steps can be taken to further investigate this issue?
    • See previous answer
4

Thank you Luuk and Sebastiaan for your replies. We’ll try your suggestions and will post an update later.

5

I have updated to umbraco 13.8.1 and that did not solve the issues.

We tested the limitation on our production servers, then we created a separate VM that had the site/db on the same machine. This still did not resolve the issue.

Here is the code we use to create members and login. Anything that touches the database from umbraco’s side seems to be a chokepoint and causing large amount of deadlocks.

All ADGUmbracoMember.Init is doing is setting properties on the model.

public IADGUmbracoMember CreateMember(string username, string email, string name, string type)
{
IMember? member = _IMemberService.CreateMemberWithIdentity(username, email, name, type, true);

 if (member != null)
 {
     member.CreateDate = TimeZoneHelper.CurrentTimeZone(_AppSettings.UmbracoConnection);
 }
 return ADGUmbracoMember.Init(member, _IMemberService, _IMemberManager);

}

Here is the login code. We use PasswordSignInAsync. We also have to update LastLoginDate and perform a .Save because there is a bug in Umbraco that will not update the LastLoginDate when logging in.

public bool Login(IADGUmbracoMember member, string password)
{
if (member != null)
{
var signInStatus = _IMemberSignInManager.PasswordSignInAsync(member.Username, password, isPersistent: true, lockoutOnFailure: true).Result;
if (signInStatus.Succeeded)
{
SetLastActive();

         member.LastLoginDate = TimeZoneHelper.CurrentTimeZone(_AppSettings.UmbracoConnection);
         member.Save();
         return true;
     }
     else
     {
         return false;
     }
 }
 else
 {
     return false;
 }

}

6

May not solve the problem, but I’d suggest using async patterns throughout the call chain. If you’re calling this on an API endpoint you can start the async chain there and work up.

7

There are some performance issues with member logins: Slow performance and deadlocks in SignInAsync during high load (v.10–13–?) · Issue #13768 · umbraco/Umbraco-CMS

8

Does anyone know if this issue (SignIn deadlock) is solved with Umbraco 15? Or should we expect the same issue after a potential upgrade from 13 to 15?

9

Just an update. We updated to make it more barebones and made it asynchronous as well.

Here is the create member code now. It hit this immediately and creates the member/logs them in. This is causing massive deadlocks on the database still when attempting with even 100 concurrent logins.

private async Task<IdentityResult> CreateMemberAsync(string netId, string firstName, string lastName, string email, string integratedPassword, IEnumerable<FeatureSettingModel> memberProperties, IEnumerable<FeatureSettingModel> memberGroups, MemberTypes memberType)
{
    using ICoreScope scope = _coreScopeProvider.CreateCoreScope(autoComplete: true);

    var identityUser = MemberIdentityUser.CreateNew(netId, email, memberType.ToString(), true, firstName);
    IdentityResult identityResult = await _memberManager.CreateAsync(identityUser,integratedPassword);

    if (identityResult.Succeeded)
    {
        IMember? member = _memberService.GetByKey(identityUser.Key);
        if (member == null)
        {
            throw new InvalidOperationException($"Could not find a member with key: {member?.Key}.");
        }

        _memberService.Save(member);
        await _memberSignInManager.SignInAsync(identityUser, false);
    }

    return identityResult;
}
10

I have applied the 13.9 updates and this did not fix the issue. It’s still deadlocking constantly when a large amount of accounts are attempted to be created at once. The site is unusable when these accounts are being created, it will not load. Below is a small glimpse at what our logs look like when I sent 100 requests to make accounts around the same time. This new update is actually bringing (on local) new unhandled errors, before it was not showing these errors in debug mode.

image
image1920×502 269 KB

11

Reading through the issue you referenced on GitHub the patch worked on login performance, not creation performance.

This definitely sounds like an issue with Umbraco. I’d suggest opening a new issue in GitHub with some code (or a GitHub repo) to be easily replicable by the Umbraco team.

12

Hi @SamM-0000

Not sure if you ever got this resolved for yourself.

I went through a similar painful journey a few years ago… I describe it in the last comment here:

I am not an expert on the Umbraco source code, so I am happy to be corrected, but my findings might point you in a helpful direction.

I belive that when you perform CRUD operations on a member record, the member content tree is locked for the duration of the operation. This is also true for content.

In a nutshell, you can only update one member at a time or one content node at a time.

The lock occurs at the database level.

So although the web app itself can handler 100s of concurrent requests, (and i find it is very performant at serving static data) the database locking mechanism will only allow one member/node to be updated oncurrently. All other requests have to wait. Given enough requests, you start getting timeouts waiting for the database lock to release. The more updates, the slower the response time, the longer the lock is held for and… you get the idea.

Again, I am not an expert on the source code, just sharing my findings. So all this could be nonsense.

But take a look at the MemberService.cs Save() methods. (Last time I checked was V13.something and this was still thre). You will see calls to

scope.WriteLock(Constants.Locks.MemberTree);

From what I understand, this only allows you updaate one member at a time.

But it gets worse if you use any notification handlers.

The lock is enabled BEFORE notification handlers are called and released AFTER they all finish. So if you make any downstream API calls from within a notification handler (for example, sending an email, creating a PDF), the lock is maintained for the duration of those calls too, making the issue even worse.

Again…. these are only my own observations. I like Umbraco and find it very performant when serving data and the backoffice has been a great tool for us. But concurrent updates has been a constant issue for us. Enough for us to move all members to a custom identity server implementation and we are considering moving large portions of our content nodes to custom apis in the future.

1 Like