We upgraded a clients website from v13 to v16 at the start of last month, the site is not relly that big, around 300 pages and 1200 media items.
The performance has been absolutely terrible, the back office is slow/sluggish, assuming you have been able to log in as sometimes you just get stuck on the login screen and have to close the browser and try again.
Editing the website has become a major undertaking as saving and publishing a task that was almost instant before takes a very noticable amount of time, 20+ seconds from my count just now. We have also experienced today when making multiple edits throughout the morning, that the entire website, including back office, would become non responsive and just hang.
We are hosting the site on Microsoft Azure and neither the App Service or Database is showing anywhere near max resource usage, so it’s not even a case of we just need to ramp up the hosting abit.
I am really hoping that between v13 and v16 I have missed some magic config setting that will just fix everything and make the site as responsive as it was back in v13. Either that or if there was a change between these versions that means a feature or method we are using is now bad for performance.
We do have the site running in production mode, and the health check confirms this, but I did notice the template editing screen no longer shows the warning message, but I assume this is an oversight and will be fixed.
Really appreciate any advice, I cannot share the site name here, but if anyone is able to help I am open to discussion outside of the forum.
Any custom code that might be adding some complexity (maybe around caching if it’s being used)?
I would also check for changes in database schema or indexes. Some users upgrading from earlier versions reported lock errors and slow publishing due to schema changes.
Run SQL Profiler or similar tools to identify slow queries.
Any chance you could install a newly installed version with minimal setup to see what happens?
You could try and use the network tab in your browser to see if anything is highlighted with slow endpoints.
Is the same occurring locally?
Finally, (if you can login to the backoffice) you could try to republish the entire site and see if that makes it any better.
Really REALLY check for browser plugins! We’ve had this discussion on Discord and I’ve had a collegue that were all complaining about very slow backoffices and in both cases it was browser plugins that really messed things up.
When you say browser plugins, I assume you mean extensions like ad blockers or video downloaders that kind of thing?
I am running some browser extensions but that wouldnt explain the front end of the website becoming unresponsive completely while the client was making edits in the back office. Users (clients customers) of the website also made comments of how slow it was performing yesterday morning while content editing was taking place, even to the point of hanging/crashing on them while they were browsing pages not even being edited.
I will try a browser without extensions though to see if that helps the back office experience.
Ah, that’s different. Browser plugins caused in our case very slow and unresponsive backoffices, but that did not impact the frontend.
In that case, it sounds more like a threading or database connection pooling issue. Can you check if you have MultipleActiveResultsSets set to false in the database connection string. In my experience, setting it to true can cause serious issues with the connection pool.
Yep we do have that set to false as agree we have experienced issues with it in the past also.
Frustratingly the site is performing well this morning… but no editing is currently taking place, but will be later this morning.
The only way I can describe what I am experiencing is sometimes, not everytime during editing, its like the website is rebooting, like you have just published new DLL files or manually restarted it. Front end pages will just keep spinning in the browsers for 30-60 seconds and then the site works like normal again front end wise. The back office will usually crash when this happens aswell as again it feels like the site has undergone a reboot, so you get a wall of the red error popups as you try to navigate or refresh and then hope your login session is still active after the refresh, or you have to close the browser window and try login again from fresh as if you get pinged back to the login screen it never works, you just get a blank white screen after entering credentials.
It’s not necessarily as straightforward as that. There are other bottlenecks that Umbraco often needs scaling for that don’t show up in the obvious metrics. But the fact that you’re on Azure makes this easier to start debugging.
Do you have Application Insights enabled in Azure? If not, it’s time to turn it on.
The best thing you can do is profile the site to see what code is actually taking so long to execute.
If you set up your web app with all the defaults then you will likely already have Application Insights set up and maybe even some profiling sessions that you can dig into to see where the slowdowns are. I suggest starting a session, then doing whatever it is that causes the site to slow down.
Since it’s also a problem locally, you can profile with dotTrace too, but that’s a bit more complex to get up and running, and understanding the results, if you’re not used to profiling stuff.
Still trying to do some of the other suggestions, but a common issue just happened again right now.
My back office login had timed out so I was on the screen with the photo to the left and the sign into Umbraco button on the right, past experience tells me clicking that button has a high probability of not working, so I edited the url to remove the /logout from the end.
When I then hit enter to try and access the back office, which should then represent the login screen. At this point the browser tab started just spinning, at the same time the front end of the site also stopped working and started spinning if you tried to navigate to any pages. Only once the back office login page had loaded, did the front end become responsive again and the page loaded, it took around 30-40 seconds for this to happen.
They have multiple staff using the back office throughout the day, so if this is happening to them I can see why we get random hangs with no clear reason.
Have not managed to get to the bottom of why that is happening, but its very consistant from my experience.
Wow, yeah, that is indeed a bad experience. I’m sorry about that.
The “stuck on login” or “button not working” is at least being addressed in 16.3, of which there is a Release Candidate on NuGet available for: GitHub · Where software is built
What you otherwise experienced, with the 30-40 seconds spinning, is likely the database being locked for one reason or another. It could have been the infamous “Failed to acquire write lock” that has unfortunately haunted us for a while.
More tidbits around the write lock can be found here - you might be lucky that the very long thread holds some valuable info for you:
In your case, I would honestly wait and see if the caching regression I mentioned before opens up for some of the performance again. It didn’t cut the first Release Candidate, but we aim to have it ready for RC2, possibly tomorrow.
I hope you are able to follow Jason’s advice and enable Application Insights as well.
Just to be sure, did you run the v13 site on the exact same setup? Hardware etc? With the same database etc.?
From my research v16 will be a lot more chatty with the database because of how HybridCache works, if the connection between you webapp and database is very slow that might cause issues.
It would be interesting to know if you see any differences on your frontend when using a higher number for the cache seed.
Yep its exactly the same Azure App Service and Database level, both the resources are showing nowhere near at capacity with things like CPU, Memory or DTUs.
I did see the cache settings before and currently both of them are set to 5000, as I did think this would help.
The confusing part is the front end is perfectly fine unless someone is interacting with the back office, as like I mentioned things like saving and publishing or even trying to login can randomly cause massive hangs. I am pretty confident they are related as when the back office finally loads or logs in the front end recovers and is back to normal.
Thanks for all advice and things to try so far, will keep trying them and hopefully find something that stands out.
Is the slow experience the same on localhost as on the hosted site?
I once experienced an incredibly slow backoffice after upgrading from v9 to v13 because I forgot to disable uSync in production. Since uSync was monitoring all content changes, it really slowed everything down. I’m not sure how much it affected the frontend though.
Yes it is happening in localhost aswell, but since it feels random when it happens its proving hard to debug.
Previously we always kept uSync fully enabled in production, aka export on saving and with v13 that was not an issue.
I did disable this last week on the v16 site and well its hard to tell if its had much impact, since there has been very minimal editing on the site since i turned it off typically!
I think the front end issues come as a result of the back office causing some locks or hangs to happen to Umbraco as a whole, so I dont think there is a direct issue with the front end potentially, its just what the client sees of course when it happens!
Just to share that we have found a fairly significant performance regression that comes to light if you have a lot of media pickers being validated - which can add up if there are many blocks and or languages.
This is the PR and we’ll include in 16.3.0-rc4 which we’ll aim to push out today. Would be worth seeing if that helps with the issue you are experiencing.
We are running similar issues with one of our clients that was upgrading from CMS 13 to CMS 16, but the performance degradation only occurs on large block grid items in the Umbraco Cloud environments, not locally.
The RC4-fix was suppose to address this issue but we still experience publishing times in the backoffice between 10-15 seconds on some pages.
The errand is currently ongoing with the Umbraco black ops team and hopefully more information regarding this can be found.
We have been looking through the solution locally with no finds regarding anything suspicous, so far!
Hopefully this info finds someone that are in a similiar situation as we are.
