I’d gently push back on what your AI agent is telling you — I think it has cause and effect reversed.
cmsContentNu is the output of the rebuild process, not the input. The rebuild populates it by reading from umbracoNode, umbracoContent, umbracoContentVersion, umbracoMediaVersion and umbracoPropertyData. So if cmsContentNu is incomplete, that’s because the rebuild is failing — not the cause of the failure.
Your stack trace is telling you exactly why it’s failing:
System.InvalidOperationException: No data for media 1612
at NuCacheContentRepository.CreateMediaNodeKit
at NuCacheContentRepository.GetAllMediaSources
...
at PublishedSnapshotService.LoadMediaFromDatabaseLocked
The rebuild is crashing during the media load phase because media node 1612 has a row in umbracoNode but is missing one of its dependent rows (most likely in umbracoMediaVersion, or its umbracoContentVersion row doesn’t have current = 1). Until that source data corruption is fixed, every rebuild attempt will throw the same Panic and abort — which is why cmsContentNu only has 334 rows and isn’t progressing.
Truncating cmsContentNu and triggering another rebuild won’t help — it’ll just hit the same exception on 1612 again and stop in the same place.
To diagnose, can you run these queries and share the results?
SELECT id, nodeObjectType, parentId, [text]
FROM umbracoNode WHERE id = 1612;
SELECT nodeId, contentTypeId
FROM umbracoContent WHERE nodeId = 1612;
SELECT id, nodeId, [current], versionDate
FROM umbracoContentVersion WHERE nodeId = 1612;
SELECT mv.id, mv.versionDate
FROM umbracoMediaVersion mv
INNER JOIN umbracoContentVersion cv ON cv.id = mv.id
WHERE cv.nodeId = 1612;
Whichever query returns no rows (or returns rows with current = 0 for all of them) is your culprit. Once we know exactly what’s missing, we can fix the underlying data and then the rebuild should complete.
Also worth a thought — fair warning that 1612 might not be the only corrupt media item. The Panic fires on the first one it hits, so fixing this one may expose the next. Once it’s resolved, watch the logs for the next “No data for media XXXX” and repeat.
One more question — does anything come to mind about what happened around 5 May? That’s when NuCache was last successfully serialized, so whatever broke this likely happened around then (failed deployment, DB restore, bulk media operation, etc).
Around the 5th of May, we restored some umbraco property data rows from an old backup, some images had been removed because some fields where switching between being culture invariants and culture variated
Could restoring old property values have caused this if that was done directly on the database? If you’ve bypassed the API, then you may be missing data or versions required elsewhere that completes the picture.
Are you able to double check what data was restored and how it was achieved to see if there are any gaps?
Going back to a database before the 5th May will most likely resolve your issue but you may not be able to do that as you will lose any changes since then.
If you really need to fix this, running the database locally against the Umbraco code base may be your best bet as you can step through the code and see the exact issue. Fixing the data may be harder though depending on what is missing or invalid…
Still working on getting the database backup on my machine.
The properties that we “copied” from the old database were just some image pickers, the actual images are still on the prod machine in the media library.
If we take an error log in isolation, like this one:
System.InvalidOperationException: No data for media 15379
Glad to try and help! Let me know once you’ve got a local copy of the database.
I don’t mind taking a look at the database too if you are able to share? I realise that may not be possible due to the data it stores and client confidentiality, etc.
Without the data in the NuCache tables, the front-end won’t render as it uses the published cache. The actual content is still in the database but the IIS reset has obviously got you in a worse state than before.
Are you able to go back to a pre 5th May database in the interim to get the site up and running whilst you look at the issue locally?
I don’t know how critical this site is or what implications that may have…
So, after some very sweaty and high-stress hours our site is finally back up! I’ll attempt to summarize:
After many attempts to try and get Umbraco to rebuild it’s cmsContentNu cache, we cloned the source code from Github and checked out the version tag we’re running (v13.11) to attempt to figure out what was causing the issues.
We set the Serilog log-level to just show Errors and saw a lot of “No data for media” exceptions, which led us to this line:
Since our site was very large we were throwing thousands of exceptions which caused the entire site to crash. So we patched this dll (Umbraco.PublishedCache.NuCache.dll) to instead of throwing exceptions, to simply log and move on. This patch made it possible to boot the site, but the pages looked weird, stuff was missing and it just seemed way off. But hey, it was up, which was progress!
So deeper into the rabbit hole we went, and we ended up writing another patch to the same dll, because when we attempted to click the “Rebuild” button on the database cache it just ticked for maybe 2 seconds then stopped, no errors, no logs, no nothing. so we ended up looking at this file:
And if you follow that you will see that it just swallows any exceptions that might be thrown, so our next patch was to catch these exceptions and log them.
From that log message we could create an SQL query that compiled of list of “Corrupt/Broken” documents in the database:
IF OBJECT_ID('tempdb..#CorruptDocs') IS NOT NULL DROP TABLE #CorruptDocs;
SELECT d.nodeId, 'missing_umbracoContent' AS reason
INTO #CorruptDocs
FROM umbracoDocument d
LEFT JOIN umbracoContent c ON c.nodeId = d.nodeId
WHERE c.nodeId IS NULL
UNION ALL
SELECT d.nodeId, 'missing_current_content_version'
FROM umbracoDocument d
LEFT JOIN umbracoContentVersion cv ON cv.nodeId = d.nodeId AND cv.[current] = 1
WHERE cv.id IS NULL
UNION ALL
SELECT d.nodeId, 'missing_document_version_for_current'
FROM umbracoDocument d
JOIN umbracoContentVersion cv ON cv.nodeId = d.nodeId AND cv.[current] = 1
LEFT JOIN umbracoDocumentVersion dv ON dv.id = cv.id
WHERE dv.id IS NULL
UNION ALL
SELECT d.nodeId, 'published_without_published_version'
FROM umbracoDocument d
WHERE d.published = 1
AND NOT EXISTS (
SELECT 1
FROM umbracoContentVersion cv
JOIN umbracoDocumentVersion dv ON dv.id = cv.id AND dv.published = 1
WHERE cv.nodeId = d.nodeId
);
SELECT reason, COUNT(*) AS cnt
FROM #CorruptDocs
GROUP BY reason
ORDER BY reason;
SELECT *
FROM #CorruptDocs
ORDER BY nodeId, reason;
This yieled a database table with around 45 broken nodes. Where there’s been a miss-match of various published statuses.
The fix was to fix these affected nodes with the following SQL script:
BEGIN TRAN;
DECLARE @Broken TABLE (nodeId INT PRIMARY KEY);
INSERT INTO @Broken (nodeId) VALUES
(4186),(4350),(5068),(5229),(6600),(9358),(10247),(13662),(14249),(14437),
(14442),(14913),(15182),(15192),(15197),(15198),(15234),(15605),(15664),(15816),
(16694),(16715),(17673),(17944),(18001),(18077),(20501),(27739),(28130),(28245),
(29461),(29463),(29574),(29863),(29872),(30479),(30489),(30578),(31565),(31583),
(31978),(31984),(32367),(32404),(32485);
-- 1) Ensure current-version has a row in umbracoDocumentVersion
-- (if missing, create it with published=1)
INSERT INTO umbracoDocumentVersion (id, templateId, published)
SELECT cv.id, NULL, 1
FROM umbracoContentVersion cv
JOIN @Broken b ON b.nodeId = cv.nodeId
LEFT JOIN umbracoDocumentVersion dv ON dv.id = cv.id
WHERE cv.[current] = 1
AND dv.id IS NULL;
-- 2) Clear published-flag on all variants of this node
UPDATE dv
SET dv.published = 0
FROM umbracoDocumentVersion dv
JOIN umbracoContentVersion cv ON cv.id = dv.id
JOIN @Broken b ON b.nodeId = cv.nodeId;
-- 3) Set published=1 on current-version
UPDATE dv
SET dv.published = 1
FROM umbracoDocumentVersion dv
JOIN umbracoContentVersion cv ON cv.id = dv.id
JOIN @Broken b ON b.nodeId = cv.nodeId
WHERE cv.[current] = 1;
-- 4) Ensure document is published
UPDATE d
SET d.published = 1
FROM umbracoDocument d
JOIN @Broken b ON b.nodeId = d.nodeId;
-- 5) Verify, should yield 0 rows
SELECT d.nodeId
FROM umbracoDocument d
JOIN @Broken b ON b.nodeId = d.nodeId
WHERE d.published = 1
AND NOT EXISTS (
SELECT 1
FROM umbracoContentVersion cv
JOIN umbracoDocumentVersion dv ON dv.id = cv.id AND dv.published = 1
WHERE cv.nodeId = d.nodeId
);
-- COMMIT;
-- ROLLBACK;
Our verification step yielded 0 rows, which was expected. We could then stop the app-pool, clear the NuCache folders and restart everything back up. Now the entire site booted correctly and we could also click the Rebuild button on the Published Status screen and we could successfully rebuild the cmsContentNu cache!
(I know the forum post is old, but this client hasn’t wanted us to do any work on this site up until now, so that’s why we hadn’t taken any action for so long)
