Distributed background jobs error

Questions
1

I’m seeing the following error in the Umbraco logs roughly every 10 minutes. The site is running on local IIS without any load balancing.

An exception occurred while attempting to ensure distributed background jobs on startup.

System.ArgumentException: An item with the same key has already been added. Key: WebhookFiring
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Linq.Enumerable.SpanToDictionary[TSource,TKey,TElement](ReadOnlySpan`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
   at Umbraco.Cms.Infrastructure.Services.Implement.DistributedJobService.EnsureJobsAsync()
   at Umbraco.Cms.Infrastructure.BackgroundJobs.DistributedBackgroundJobHostedService.ExecuteAsync(CancellationToken stoppingToken)

Does anyone have any idea where this is coming from?

2

Hi @creativesuspects

That’s a duplicate registration of the WebhookFiring recurring job - the startup routine builds a lookup keyed on job name and throws an error when there is a duplicate. The 10-minutes is due to the hosted service retrying, so it’ll keep going until the duplicate’s removed.

Can you check you aren’t registering webhooks twice, perhaps via a manual AddWebhooks call on top of the default core one, or a package doing its own thing. I’d have a search through your composers and App_Plugins to see what’s registering it.

Justin

3

@justin-nevitech No, I’m not doing anything with webhooks. So no manual AddWebhooks() call, or anything webhook related in my composers or App_Plugins. As a matter of fact I’ve disabled the webhooks through the Umbraco:CMS:Webhook settings. All the umbracoWebhook* tables are empty. Is there anywhere else I can look?

4

Hi @creativesuspects

What version of Umbraco 17 are you using and have you got any packages installed that may be doing anything with webhooks (regardless of whether you are using them or not)? The error you have is during registration, so the background jobs may still run even when disabled (but not do anything).

Justin

5 
<PackageReference Include="Umbraco.Cms" Version="17.5.3" />
<PackageReference Include="Umbraco.Community.Contentment" Version="6.2.1" />
<PackageReference Include="Umbraco.Community.Icoover" Version="0.1.1" />
<PackageReference Include="uSync" Version="17.3.6" />

And a few bespoke packages for custom property editors, not involving webhooks.

6

Never mind, I found the culprit. I had implemented an IUmbracoBuilder extension method called AddBackgroundJobs() to inject my custom background jobs. I call this method inside an IComposer, but apparently Umbraco.Cms.Infrastructure.DependencyInjection.UmbracoBuilderExtensions also has a method with that same name, and I was accidentally using that method instead of my own. So this caused the duplicate injection of the WebhookFiring job. :see_no_evil_monkey:

I previously hadn’t noticed that my background jobs weren’t triggered at all, because they were disabled (by an app setting) on the development and staging environments. So the background jobs not running actually was expected behaviour.