(Hangfire.Server.ServerWatchdog) 1 servers were removed due to timeout

deanleigh · February 26, 2025, 8:31am

For no apparent reason after a build a v13 partially loaded then stopped.
I was running it in IIS Express so tried dotnet run and followed the errors:

2025-02-25 07:21:37 [INFO] (Hangfire.SqlServer.SqlServerObjectsInstaller) Start installing Hangfire SQL objects...

2025-02-25 07:21:37 [INFO] (Hangfire.SqlServer.SqlServerObjectsInstaller) Hangfire SQL objects installed.

2025-02-25 07:21:38 [INFO] (Hangfire.BackgroundJobServer) Starting Hangfire Server using job storage: 'SQL Server (custom)'

2025-02-25 07:21:38 [INFO] (Hangfire.BackgroundJobServer) Using the following options for SQL Server job storage: Queue poll interval: 00:00:00.

2025-02-25 07:21:38 [INFO] (Hangfire.BackgroundJobServer) Using the following options for Hangfire Server:

Worker count: 20

Listening queues: 'default'

Shutdown timeout: 00:00:15

Schedule polling interval: 00:00:15

Any Idea what could be causing this?

I have also tried:

clearing the Hangfire tables so they rebuild
setting the hangfire timeout to 10 minutes

sebastiaan · February 26, 2025, 8:38am

This all looks perfect! So did you not get any more output on the console after this?

Can you say more about what happened before this? Did it ever work, or is this the first attempt at using Hangfire?

Just to make sure, you’re using Cultiv.Hangfire version 3 with Umbraco 13, right?

deanleigh · February 26, 2025, 9:10am

Hangfire 3.1.2

It’s a project I have been working on every day for 8 months with very few changes.

Full output

2025-02-26 09:06:49 [INFO]  (Hangfire.SqlServer.SqlServerObjectsInstaller) Start installing Hangfire SQL objects...
2025-02-26 09:06:49 [INFO]  (Hangfire.SqlServer.SqlServerObjectsInstaller) Hangfire SQL objects installed.
2025-02-26 09:06:50 [INFO]  (Hangfire.BackgroundJobServer) Starting Hangfire Server using job storage: 'SQL Server (custom)'
2025-02-26 09:06:50 [INFO]  (Hangfire.BackgroundJobServer) Using the following options for SQL Server job storage: Queue poll interval: 00:00:00.
2025-02-26 09:06:50 [INFO]  (Hangfire.BackgroundJobServer) Using the following options for Hangfire Server:                                                                                            anager, CountersAggregator, SqlServerHeartbeatProcess, Worker, DelayedJobScheduler, RecurringJobScheduler...
2025-02-26 09:06:50 [INFO]  (Hangfire.Server.ServerWatchdog) 1 servers were removed due to timeout
2025-02-26 09:06:50 [INFO]  (Hangfire.Server.BackgroundServerProcess) Server d-desktop:21768:6caf0fc3 all the dispatchers started
2025-02-26 09:09:15 [INFO]  (Hangfire.Server.BackgroundServerProcess) Server d-desktop:21768:6caf0fc3 caught stopping signal...
2025-02-26 09:09:15 [INFO]  (Hangfire.Server.BackgroundServerProcess) Server d-desktop:21768:6caf0fc3 All dispatchers stoppedin 7.3152 ms
2025-02-26 09:09:15 [INFO]  (Hangfire.Server.BackgroundServerProcess) Server d-desktop:21768:6caf0fc3 successfully reported itself as stopped in 3.2378 msog, ServerJobCancellationWatcher, ExpirationMa
2025-02-26 09:09:15 [INFO]  (Hangfire.Server.BackgroundServerProcess) Server d-desktop:21768:6caf0fc3 has been stopped in total 5.1981 ms

sebastiaan · February 26, 2025, 9:33am

There’s nothing I can see wrong here though, clearly d-desktop became unavailable and has been removed.

Let’s start over, without looking at the logs, what is the actual problem you’re experiencing?

deanleigh · February 26, 2025, 9:39am

Thanks @sebastiaan on a standard build the site is partially rendered.
If refreshed I get a 503.0 error message.

sebastiaan · February 26, 2025, 9:42am

Hmmm… I’m stumped, it doesn’t sounds like anything I’ve seen before!

Could you have a look at the Windows event viewer to see what’s happening in administrative events? Maybe there’s more clues there.

Obviously if you’ve added or changed any Hangfire jobs recently then I would immediately suspect those. Or any other recent changes that might affect your jobs.

deanleigh · February 26, 2025, 3:38pm

I’m not sure if this helps but it seems to be stopping here

//Needed for Hangfire|
var sqlStorage = new SqlServerStorage(_config.GetConnectionString(umbracoDbDSN)); <=====
JobStorage.Current = sqlStorage;|

sebastiaan · February 26, 2025, 3:40pm

Gotcha! On local, remove the cache parameter from the SQLite connection string and you should be good to go again!

deanleigh · February 26, 2025, 3:42pm

So I don’t have Cache=Private in my appsettings.Local.json connection string but it is in appsettings.json.
But the other devs have the same but don’t get this issue?

sebastiaan · February 26, 2025, 3:44pm

I don’t know, try to remove it and see what happens for you first!

deanleigh · February 26, 2025, 3:53pm

OK I removed it and it still behaves the same.
It stops the server.

But I also spotted

Error loading schema: appsettings-schema.json
There are problems with this document’s schema impacting one or more items in the document. Please report this issue to schema owner.

This line

"$schema": "http://json-schema.org/draft-04/schema#",

And it’s excluded from git so I wonder if this is part of the issue?

deanleigh · February 26, 2025, 4:07pm

OK I have removed Cache=Private;Foreign Keys=True; and it has moved onto the hangfire composer so I will go back to the team.
Thanks for helping so far.

sebastiaan · February 26, 2025, 4:20pm

Ah don’t worry about the Schema that’s just metadata and doesn’t affect anything apart from autocomplete in your appsettings.*.json files.

Yes, good point, it doesn’t support anything separated by ; so the Foreign Keys also needed to go. Though I will say on v15 it does seem to work, so maybe there’s something that I can update on the v13 package to make it work.

Anyway, you’re not using SQLite in production I hope, and locally these changes make no difference so you can live without them no problem!

deanleigh · March 6, 2025, 3:28pm

Just an update @sebastiaan I have it running in Kestrel just not IISExpress.
I don’t suppose that gives any clues?

sebastiaan · March 6, 2025, 3:47pm

Possibly, I haven’t had IIS(Express) installed in years, so I have not tested it there

Make sure to create an issue for it on the issue tracker, other people might be able to help - I won’t have the time to dig in for the next few weeks I’m afraid!

deanleigh · March 11, 2025, 3:46pm

I have been persisting in testing differences with the environments where it is working using IISExpress. My colleagues only have .Net 8 installed whereas I have .Net 9

This in itself should not cause an issue but I found a post that suggests that multitargeted packages with automatically use version that match the .Net version e.g. a package that targets .Net 8 for v13 and .Net 9 for v15 will server the v15 package.

Does this sound familiar?

deanleigh · March 17, 2025, 4:26pm

Further update.
We had an issue with deployment in Umbraco Cloud and HQ has attempted to recreate it locally.

They replied:

I got an update from the developer.>
He has been trying to get live running locally.>
And he has been unable to get it to run.>
The thing that is stopping me is the hangfire jobs starts, and after 15-20 mins timeouts and site fails to boot.

I have instructed them to try in Kestrel but does this help identify the issue?

sebastiaan · March 17, 2025, 6:06pm

Doubt there’s any difference, but worth a try! Do report back

deanleigh · March 17, 2025, 6:29pm

Oh it definitely works with Kestrel as I mentioned in previous posts.
Is it possible it’s because I have .Net 9 SDK installed?
Is Hangfire multitargeted?

sebastiaan · March 17, 2025, 6:59pm

The only difference is the web server, so it will need investigating what it errors out with etc.