Richtext blocks migration

Good evening everyone,

I have recently setup a new Umbraco16 installation and imported my former items via uSync into the new instance.

Most of the import went fine, but today we realized that the “inserted blocks” in the richtext editor are missing. I checked the database and they’re still there, but they’re not shown anymore. When inserting a NEW block everything works fine.

So I compared the old and new version in the database and saw that the placeholder changed its structure…

Before:

<umb-rte-block data-content-udi="umb://element/fbb93c5f63fe40a394df57f4c7334865"><!--Umbraco-Block--></umb-rte-block>

After:

<umb-rte-block data-content-key="d99e600b-1aef-4290-8c64-55d862ede0cd"></umb-rte-block>

So I thought about just looping over the 70 instances we have in the database and replace the attribute, BUT the remaining structure in the database field changed as well:

Before

{
   "markup":"<Placeholder data-content-udi="umb://element/fbb93c5f63fe40a394df57f4c7334865" />",
   "blocks":{
      "contentData":[
         {
            "contentTypeKey":"eee672af-223f-47a8-a702-bee77cff0123",
            "udi":"umb://element/fbb93c5f63fe40a394df57f4c7334865",
            "accordionItems":"<dataformyblock>"
         }
      ],
      "settingsData":[
         
      ],
      "Layout":{
         "Umbraco.TinyMCE":[
            {
               "contentUdi":"umb://element/fbb93c5f63fe40a394df57f4c7334865",
               "settingsUdi":null
            }
         ]
      }
   }
}

After

{
   "markup":"\u003Cumb-rte-block data-content-key=\u002210ed0a08-8e9f-42e8-9875-34cf1b7fe4da\u0022\u003E\u003C/umb-rte-block\u003E\u003Cp\u003E\u003C/p\u003E",
   "blocks":{
      "contentData":[
         {
            "contentTypeKey":"eee672af-223f-47a8-a702-bee77cff0123",
            "udi":null,
            "key":"10ed0a08-8e9f-42e8-9875-34cf1b7fe4da",
            "values":[
               {
                  "editorAlias":"Umbraco.BlockList",
                  "culture":null,
                  "segment":null,
                  "alias":"accordionItems",
                  "value":"{\u0022contentData\u0022:[{\u0022contentTypeKey\u0022:\u0022e36978ca-9624-4477-9e4f-e5d73f4d948b\u0022,\u0022udi\u0022:null,\u0022key\u0022:\u0022841dc436-03e2-44d5-a09e-4932975e43d9\u0022,\u0022values\u0022:[{\u0022editorAlias\u0022:\u0022Umbraco.TextBox\u0022,\u0022culture\u0022:null,\u0022segment\u0022:null,\u0022alias\u0022:\u0022title\u0022,\u0022value\u0022:\u0022asdfasf\u0022},{\u0022editorAlias\u0022:\u0022Umbraco.RichText\u0022,\u0022culture\u0022:null,\u0022segment\u0022:null,\u0022alias\u0022:\u0022content\u0022,\u0022value\u0022:\u0022{\\u0022markup\\u0022:\\u0022\\\\u003Cp\\\\u003Easdfasas\\\\u003C/p\\\\u003E\\u0022,\\u0022blocks\\u0022:{\\u0022contentData\\u0022:[],\\u0022settingsData\\u0022:[],\\u0022expose\\u0022:[],\\u0022Layout\\u0022:{}}}\u0022}]}],\u0022settingsData\u0022:[],\u0022expose\u0022:[{\u0022contentKey\u0022:\u0022841dc436-03e2-44d5-a09e-4932975e43d9\u0022,\u0022culture\u0022:null,\u0022segment\u0022:null}],\u0022Layout\u0022:{\u0022Umbraco.BlockList\u0022:[{\u0022contentUdi\u0022:null,\u0022settingsUdi\u0022:null,\u0022contentKey\u0022:\u0022841dc436-03e2-44d5-a09e-4932975e43d9\u0022,\u0022settingsKey\u0022:null}]}}"
               }
            ]
         }
      ],
      "settingsData":[
         
      ],
      "expose":[
         {
            "contentKey":"10ed0a08-8e9f-42e8-9875-34cf1b7fe4da",
            "culture":null,
            "segment":null
         }
      ],
      "Layout":{
         "Umbraco.RichText":[
            {
               "contentUdi":null,
               "settingsUdi":null,
               "contentKey":"10ed0a08-8e9f-42e8-9875-34cf1b7fe4da",
               "settingsKey":null
            }
         ]
      }
   }
}

So obviously I would have to:

  • Parse out the old uid attribute
  • Add a key property in the blocks.contentData object
  • Create a blocklist item in the values property of contentData
  • Add an expose-property
  • Update the key from Umbraco.TinyMCE to Umbraco.RichText
  • Add contentKey also below Umbraco.RichText

I’m just curious because this seems like an operation that should be done automatically and wonder if I’m just to blind to figure out the built-in functionality.

Obviously I did something wrong … I upgraded using the nuget package and because I ran into issues I setup the new instance and imported there to prevent issues …

But as I can see there’s migration code built into v15 … so obviously I missed something:

So I thought I could be smart and just call the migration steps manually:

            From(string.Empty)
                .To<ConvertBlockListEditorProperties>("convert-blocklist-editor-properties")
                .To<ConvertBlockGridEditorProperties>("convert-blockgrid-editor-properties")
                .To<ConvertRichTextEditorProperties>("convert-richtext-editor-properties")
                .To<ConvertLocalLinks>("convert-local-links")
                .To<FixConvertLocalLinks>("fix-convert-local-links");

It fixed the attributes from uid to key … BUT … it didnt migrate the contentData properties … so I have now correct placeholders but lost the content of it … backup-restore incoming :smiley:

Anyone has an idea? I was going for manual conversion of that field but it feels sketchy …

I’m thinking that usync 13 may export files in a different format than 16? I’m not sure why the data wouldn’t be imported correctly.

I haven’t tackled this issue specifically but I have used c# scripts to manipulate block list / grid data multiple times. Typically I’ll create a c# controller and execute it. I prefer using a querystring parameter in my controller (?previewOnly=true) and then I can check the parameter in my logic, and return an array of string messages on what it would do without actually doing any modification. When I’m happy the logic is correct, I can run it with ?previewOnly=false

I always prefer calling the c# umbraco content APIs instead of modifying the database directly. This reduces the risk of damaging data. The content api allows you to manipulate block list / grid field data as a string.

You are on the right path, in general:

  1. Get the desired format data (which you have done)
  2. Write logic to translate from the source to the target
  3. Update the content field value

I typically go the whole nine yards and create c# models that match the desired target structure, build the content structures, then serialize it and save the serialized value.

There are many ways of modifying data, but this is my preferred approach.

Here is a sample controller shell that will manipulate content node content:

using Microsoft.Extensions.Logging;
using Umbraco.Cms.Core.Services;
using Umbraco.Cms.Core.Web;

namespace Core.SiteSearch.Controllers
{
	/// <summary>
	/// Ex. /MyImport/Import/?previewOnly=true
	/// </summary>
	[ApiController]
	[Route("[controller]/[action]")]
	public class MyImportController : ControllerBase
    {
        private readonly ILogger<MyImportController> _logger;
        private IUmbracoContextAccessor _umbracoContextAccessor { get; set; }
        private IContentService _contentService { get; set; }
        public MyImportController(ILogger<MyImportController> logger, IUmbracoContextAccessor umbracoContextAccessor,
            IContentService contentService)
        {
            _logger = logger;
            _umbracoContextAccessor = umbracoContextAccessor;
            _contentService = contentService;
        }


        // /MyImport/Import/
        [HttpGet]
        public List<string> Import(bool previewOnly = true)
        {
            List<string> result = new List<string>();
            var contentNodes = _contentService.GetByIds(new List<int> { 1, 2 });

            foreach (var node in contentNodes)
            {
                var currentValue = node.GetValue<string>("blockContent");

                //TODO: build new structure and serialize it here...
                var newValue = "";
                //ENDTODO

                if (previewOnly)
                {
                    result.Add($"I would be saving {node.Id} {node.Name}: {newValue}");
                }
                else
                {
                    node.SetValue("blockContent", newValue);
                    _contentService.Save(node);
                    _contentService.Publish(node, Array.Empty<string>());
                    result.Add($"Saved {node.Id} {node.Name}: {newValue}");
                }
            }
            return result;
        }
    }
}

Also make sure that this controller code never gets into the production branch, you don’t want any code on a prod environment that could unwittingly manipulate the data.

Hey Aaron. Thanks for replying.

I’d prefer that way to use the APIs as well … but it would still be the same issue replicating myself what somewhere in the Update-Scripts already exists, wouldnt it?

Potentially, yes. I’m not sure what detection and import scripts are built into umbraco migrations.

You are already using uSync,
have you looked into uSync Migrations?
I don’t know if they have what you are looking for but this would be something very useful to the community as a whole.

I fed the migration classes from the Umbraco.Infrastructure repository into Claude Opus. But I asked it for my specific intent, as I knew the richtext properties in the DB that I wanted to update, I asked it specifically to migrate the old RTE style into the new one … it worked like 90% … Had to save the RTEs again after in the backoffice my blocks reappeared, which fixed obviously the object sufficiently that my frontend now works again.

The generated class from Claude (ran it once and then discarded it :smiley: )

Summary

public static class RtePropertyConverter
{
private static readonly Regex RteBlockPattern = new(
@“<umb-rte-block.(?data-content-udi)=“”(?[^“”])”“.*”,
RegexOptions.IgnoreCase | RegexOptions.Singleline);

private static readonly Regex LegacyLocalLinkPattern = new(
    @"/{localLink:([a-fA-F0-9-]+)}", 
    RegexOptions.IgnoreCase);

private static readonly Regex FaultyHrefPattern = new(
    @"<a\s+(?<faultyHref>href=['""]?\s*(?<typeAttribute>type=['""][^'""]*?['""]?)?(?<localLink>/{localLink:[a-fA-F0-9-]+}['""]?)).*?>",
    RegexOptions.IgnoreCase | RegexOptions.Singleline);

made
public static ConversionResult ConvertRtePropertyData(string originalJson)
{
var result = new ConversionResult
{
OriginalJson = originalJson,
UpdatedJson = originalJson,
HasChanges = false,
ChangesSummary = new List()
};

    if (string.IsNullOrWhiteSpace(originalJson))
    {
        result.ChangesSummary.Add("Skipped: Empty or null JSON");
        return result;
    }

    try
    {
        using var jsonDoc = JsonDocument.Parse(originalJson);
        
        if (!jsonDoc.RootElement.TryGetProperty("markup", out var markupElement))
        {
            result.ChangesSummary.Add("Skipped: No 'markup' property found");
            return result;
        }

        string originalMarkup = markupElement.GetString() ?? "";
        
        var convertedJson = ConvertJsonStructure(jsonDoc, result.ChangesSummary);
        
        if (convertedJson != originalJson)
        {
            result.UpdatedJson = convertedJson;
            result.HasChanges = true;
            result.ChangesSummary.Add("Successfully converted JSON structure to V15 format");
        }
        else
        {
            result.ChangesSummary.Add("No structural changes needed");
        }

        return result;
    }
    catch (JsonException ex)
    {
        result.HasError = true;
        result.ErrorMessage = $"Invalid JSON: {ex.Message}";
        result.ChangesSummary.Add($"Error: {ex.Message}");
        return result;
    }
    catch (Exception ex)
    {
        result.HasError = true;
        result.ErrorMessage = $"Conversion error: {ex.Message}";
        result.ChangesSummary.Add($"Error: {ex.Message}");
        return result;
    }
}

public static Dictionary<int, ConversionResult> ConvertMultipleRteProperties(Dictionary<int, string> propertyDataEntries)
{
    var results = new Dictionary<int, ConversionResult>();

    foreach (var entry in propertyDataEntries)
    {
        results[entry.Key] = ConvertRtePropertyData(entry.Value);
    }

    return results;
}

private static string ConvertJsonStructure(JsonDocument originalDoc, List<string> changesSummary)
{
    // First pass: Build UDI to Key mapping from blocks
    var udiToKeyMap = new Dictionary<string, string>();
    if (originalDoc.RootElement.TryGetProperty("blocks", out var blocksElement) &&
        blocksElement.TryGetProperty("contentData", out var contentDataElement))
    {
        foreach (var contentItem in contentDataElement.EnumerateArray())
        {
            if (contentItem.TryGetProperty("udi", out var udiProp))
            {
                var udiValue = udiProp.GetString();
                if (!string.IsNullOrEmpty(udiValue))
                {
                    if (TryParseGuidFromUdi(udiValue, out var existingGuid))
                    {
                        udiToKeyMap[udiValue] = existingGuid.ToString("D");
                    }
                    else
                    {
                        udiToKeyMap[udiValue] = Guid.NewGuid().ToString("D");
                    }
                }
            }
            else
            {
                var newKey = Guid.NewGuid().ToString("D");
            }
        }
    }

    var options = new JsonWriterOptions { Indented = false };
    using var stream = new MemoryStream();
    using var writer = new Utf8JsonWriter(stream, options);

    writer.WriteStartObject();

    foreach (var property in originalDoc.RootElement.EnumerateObject())
    {
        switch (property.Name.ToLower())
        {
            case "markup":
                var convertedMarkup = ConvertMarkup(property.Value.GetString() ?? "", udiToKeyMap, changesSummary);
                writer.WriteString("markup", convertedMarkup);
                break;

            case "blocks":
                ConvertBlocksStructure(writer, property.Value, udiToKeyMap, changesSummary);
                break;

            default:
                property.WriteTo(writer);
                break;
        }
    }

    writer.WriteEndObject();
    writer.Flush();

    return System.Text.Encoding.UTF8.GetString(stream.ToArray());
}

private static string ConvertMarkup(string markup, Dictionary<string, string> udiToKeyMap, List<string> changesSummary)
{
    int rteBlockConversions = 0;

    var convertedMarkup = RteBlockPattern.Replace(markup, match =>
    {
        var udiValue = match.Groups["udi"].Value;
        
        string newKey = null;
        
        if (udiToKeyMap.TryGetValue(udiValue, out var mappedKey))
        {
            newKey = mappedKey;
        }
        else if (TryParseGuidFromUdi(udiValue, out var guid))
        {
            newKey = guid.ToString("D");
        }
        
        if (newKey != null)
        {
            rteBlockConversions++;
            return match.Value
                .Replace(match.Groups["attribute"].Value, "data-content-key")
                .Replace(match.Groups["udi"].Value, newKey);
        }
        
        return match.Value;
    });

    convertedMarkup = ConvertLegacyLocalLinks(convertedMarkup, changesSummary);

    convertedMarkup = FixFaultyHrefAttributes(convertedMarkup, changesSummary);

    if (rteBlockConversions > 0)
    {
        changesSummary.Add($"Converted {rteBlockConversions} RTE block UDI references to content keys");
    }

    return convertedMarkup;
}

private static void ConvertBlocksStructure(Utf8JsonWriter writer, JsonElement blocksElement, Dictionary<string, string> udiToKeyMap, List<string> changesSummary)
{
    writer.WritePropertyName("blocks");
    writer.WriteStartObject();

    var contentDataConverted = 0;
    var exposeEntries = new List<string>();

    foreach (var blocksProperty in blocksElement.EnumerateObject())
    {
        switch (blocksProperty.Name.ToLower())
        {
            case "contentdata":
                writer.WritePropertyName("contentData");
                writer.WriteStartArray();
                
                foreach (var contentItem in blocksProperty.Value.EnumerateArray())
                {
                    var contentKey = ConvertContentDataItem(writer, contentItem, udiToKeyMap, changesSummary);
                    if (!string.IsNullOrEmpty(contentKey))
                    {
                        exposeEntries.Add(contentKey);
                    }
                    contentDataConverted++;
                }
                
                writer.WriteEndArray();
                break;

            case "settingsdata":
                writer.WritePropertyName("settingsData");
                writer.WriteStartArray();
                
                foreach (var settingsItem in blocksProperty.Value.EnumerateArray())
                {
                    ConvertSettingsDataItem(writer, settingsItem, udiToKeyMap);
                }
                
                writer.WriteEndArray();
                break;

            case "layout":
                ConvertLayoutStructure(writer, blocksProperty.Value, udiToKeyMap, changesSummary);
                break;

            default:
                blocksProperty.WriteTo(writer);
                break;
        }
    }

    writer.WritePropertyName("expose");
    writer.WriteStartArray();
    
    foreach (var contentKey in exposeEntries)
    {
        writer.WriteStartObject();
        writer.WriteString("contentKey", contentKey);
        writer.WriteNull("culture");
        writer.WriteNull("segment");
        writer.WriteEndObject();
    }
    
    writer.WriteEndArray();
    
    if (exposeEntries.Any())
    {
        changesSummary.Add($"Added expose array with {exposeEntries.Count} entries");
    }

    writer.WriteEndObject();

    if (contentDataConverted > 0)
    {
        changesSummary.Add($"Converted {contentDataConverted} content data items to V15 structure");
    }
}

private static string ConvertContentDataItem(Utf8JsonWriter writer, JsonElement contentItem, Dictionary<string, string> udiToKeyMap, List<string> changesSummary)
{
    writer.WriteStartObject();

    string? contentKey = null;
    var valuesProperties = new List<(string name, JsonElement value)>();

    foreach (var property in contentItem.EnumerateObject())
    {
        switch (property.Name.ToLower())
        {
            case "udi":
                var udiValue = property.Value.GetString();
                if (!string.IsNullOrEmpty(udiValue))
                {
                    if (udiToKeyMap.TryGetValue(udiValue, out var mappedKey))
                    {
                        contentKey = mappedKey;
                    }
                    else if (TryParseGuidFromUdi(udiValue, out var guid))
                    {
                        contentKey = guid.ToString("D");
                        udiToKeyMap[udiValue] = contentKey; // Update mapping
                    }
                    else
                    {
                        contentKey = Guid.NewGuid().ToString("D");
                        udiToKeyMap[udiValue] = contentKey;
                    }
                    
                    writer.WriteString("key", contentKey);
                    writer.WriteNull("udi"); // Set udi to null in V15+
                }
                else
                {
                    contentKey = Guid.NewGuid().ToString("D");
                    writer.WriteString("key", contentKey);
                    writer.WriteNull("udi");
                }
                break;

            case "contenttypekey":
                property.WriteTo(writer);
                break;

            default:
                valuesProperties.Add((property.Name, property.Value));
                break;
        }
    }

    if (contentKey == null)
    {
        contentKey = Guid.NewGuid().ToString("D");
        writer.WriteString("key", contentKey);
        writer.WriteNull("udi");
    }

    if (valuesProperties.Any())
    {
        writer.WritePropertyName("values");
        writer.WriteStartArray();

        foreach (var (name, value) in valuesProperties)
        {
            writer.WriteStartObject();
            
            var editorAlias = DetermineEditorAlias(name, value);
            writer.WriteString("editorAlias", editorAlias);
            writer.WriteNull("culture");
            writer.WriteNull("segment");
            writer.WriteString("alias", name);
            
            var convertedValue = ConvertPropertyValue(value, editorAlias);
            writer.WriteString("value", convertedValue);
            
            writer.WriteEndObject();
        }

        writer.WriteEndArray();
    }

    writer.WriteEndObject();
    
    return contentKey ?? "";
}

private static void ConvertSettingsDataItem(Utf8JsonWriter writer, JsonElement settingsItem, Dictionary<string, string> udiToKeyMap)
{
    writer.WriteStartObject();

    foreach (var property in settingsItem.EnumerateObject())
    {
        if (property.Name.ToLower() == "udi")
        {
            var udiValue = property.Value.GetString();
            if (!string.IsNullOrEmpty(udiValue))
            {
                if (udiToKeyMap.TryGetValue(udiValue, out var mappedKey))
                {
                    writer.WriteString("key", mappedKey);
                }
                else if (TryParseGuidFromUdi(udiValue, out var guid))
                {
                    var key = guid.ToString("D");
                    writer.WriteString("key", key);
                    udiToKeyMap[udiValue] = key;
                }
                else
                {
                    var newKey = Guid.NewGuid().ToString("D");
                    writer.WriteString("key", newKey);
                    udiToKeyMap[udiValue] = newKey;
                }
                writer.WriteNull("udi");
            }
            else
            {
                writer.WriteString("key", Guid.NewGuid().ToString("D"));
                writer.WriteNull("udi");
            }
        }
        else
        {
            property.WriteTo(writer);
        }
    }

    writer.WriteEndObject();
}

private static void ConvertLayoutStructure(Utf8JsonWriter writer, JsonElement layoutElement, Dictionary<string, string> udiToKeyMap, List<string> changesSummary)
{
    writer.WritePropertyName("Layout");
    writer.WriteStartObject();

    foreach (var layoutProperty in layoutElement.EnumerateObject())
    {
        var propertyName = layoutProperty.Name == "Umbraco.TinyMCE" ? "Umbraco.RichText" : layoutProperty.Name;
        
        writer.WritePropertyName(propertyName);
        writer.WriteStartArray();

        foreach (var layoutItem in layoutProperty.Value.EnumerateArray())
        {
            writer.WriteStartObject();

            foreach (var itemProperty in layoutItem.EnumerateObject())
            {
                switch (itemProperty.Name.ToLower())
                {
                    case "contentudi":
                        var udiValue = itemProperty.Value.GetString();
                        if (!string.IsNullOrEmpty(udiValue))
                        {
                            if (udiToKeyMap.TryGetValue(udiValue, out var mappedKey))
                            {
                                writer.WriteString("contentKey", mappedKey);
                            }
                            else if (TryParseGuidFromUdi(udiValue, out var guid))
                            {
                                var key = guid.ToString("D");
                                writer.WriteString("contentKey", key);
                                udiToKeyMap[udiValue] = key;
                            }
                            else
                            {
                                writer.WriteNull("contentKey");
                            }
                        }
                        else
                        {
                            writer.WriteNull("contentKey");
                        }
                        writer.WriteNull("contentUdi");
                        break;

                    case "settingsudi":
                        var settingsUdiValue = itemProperty.Value.GetString();
                        if (!string.IsNullOrEmpty(settingsUdiValue))
                        {
                            if (udiToKeyMap.TryGetValue(settingsUdiValue, out var mappedSettingsKey))
                            {
                                writer.WriteString("settingsKey", mappedSettingsKey);
                            }
                            else if (TryParseGuidFromUdi(settingsUdiValue, out var settingsGuid))
                            {
                                var settingsKey = settingsGuid.ToString("D");
                                writer.WriteString("settingsKey", settingsKey);
                                udiToKeyMap[settingsUdiValue] = settingsKey;
                            }
                            else
                            {
                                writer.WriteNull("settingsKey");
                            }
                        }
                        else
                        {
                            writer.WriteNull("settingsKey");
                        }
                        writer.WriteNull("settingsUdi");
                        break;

                    default:
                        itemProperty.WriteTo(writer);
                        break;
                }
            }

            writer.WriteEndObject();
        }

        writer.WriteEndArray();
    }

    writer.WriteEndObject();
    changesSummary.Add("Converted layout structure to use keys instead of UDIs");
}

private static string DetermineEditorAlias(string propertyName, JsonElement value)
{
    switch (propertyName.ToLower())
    {
        case "content" when IsRichTextValue(value):
            return "Umbraco.RichText";
        
        case "title":
        case "headline":
        case "name":
            return "Umbraco.TextBox";
            
        case var name when name.Contains("items") && IsBlockListValue(value):
            return "Umbraco.BlockList";
            
        case var name when name.Contains("grid") && IsBlockGridValue(value):
            return "Umbraco.BlockGrid";
            
        default:
            // Try to infer from value structure
            if (IsRichTextValue(value))
                return "Umbraco.RichText";
            if (IsBlockListValue(value))
                return "Umbraco.BlockList";
            if (IsBlockGridValue(value))
                return "Umbraco.BlockGrid";
            
            return "Umbraco.TextBox"; // Default fallback
    }
}

private static bool IsRichTextValue(JsonElement value)
{
    if (value.ValueKind != JsonValueKind.String)
        return false;
        
    var stringValue = value.GetString() ?? "";
    if (string.IsNullOrEmpty(stringValue))
        return false;

    try
    {
        using var doc = JsonDocument.Parse(stringValue);
        return doc.RootElement.TryGetProperty("markup", out _);
    }
    catch
    {
        return false;
    }
}

private static bool IsBlockListValue(JsonElement value)
{
    if (value.ValueKind != JsonValueKind.String)
        return false;
        
    var stringValue = value.GetString() ?? "";
    if (string.IsNullOrEmpty(stringValue))
        return false;

    try
    {
        using var doc = JsonDocument.Parse(stringValue);
        return doc.RootElement.TryGetProperty("contentData", out _) &&
               doc.RootElement.TryGetProperty("Layout", out var layout) &&
               layout.TryGetProperty("Umbraco.BlockList", out _);
    }
    catch
    {
        return false;
    }
}

private static bool IsBlockGridValue(JsonElement value)
{
    if (value.ValueKind != JsonValueKind.String)
        return false;
        
    var stringValue = value.GetString() ?? "";
    if (string.IsNullOrEmpty(stringValue))
        return false;

    try
    {
        using var doc = JsonDocument.Parse(stringValue);
        return doc.RootElement.TryGetProperty("contentData", out _) &&
               doc.RootElement.TryGetProperty("Layout", out var layout) &&
               layout.TryGetProperty("Umbraco.BlockGrid", out _);
    }
    catch
    {
        return false;
    }
}

private static string ConvertPropertyValue(JsonElement value, string editorAlias)
{
    if (value.ValueKind != JsonValueKind.String)
        return value.ToString() ?? "";

    var stringValue = value.GetString() ?? "";
    
    if (editorAlias == "Umbraco.BlockList" && !string.IsNullOrEmpty(stringValue))
    {
        return ConvertNestedBlockStructure(stringValue);
    }
    
    if (editorAlias == "Umbraco.RichText" && IsRichTextValue(value))
    {
        var nestedResult = ConvertRtePropertyData(stringValue);
        return nestedResult.HasChanges ? nestedResult.UpdatedJson : stringValue;
    }
    
    if (editorAlias == "Umbraco.BlockGrid" && IsBlockGridValue(value))
    {
        return ConvertNestedBlockStructure(stringValue);
    }

    return stringValue;
}

private static string ConvertNestedBlockStructure(string blockJson)
{
    try
    {
        using var doc = JsonDocument.Parse(blockJson);
        
        var nestedUdiToKeyMap = new Dictionary<string, string>();
        if (doc.RootElement.TryGetProperty("contentData", out var contentData))
        {
            foreach (var item in contentData.EnumerateArray())
            {
                if (item.TryGetProperty("udi", out var udiProp))
                {
                    var udiValue = udiProp.GetString();
                    if (!string.IsNullOrEmpty(udiValue))
                    {
                        if (TryParseGuidFromUdi(udiValue, out var guid))
                        {
                            nestedUdiToKeyMap[udiValue] = guid.ToString("D");
                        }
                        else
                        {
                            nestedUdiToKeyMap[udiValue] = Guid.NewGuid().ToString("D");
                        }
                    }
                }
            }
        }
        
        var changesSummary = new List<string>();
        var convertedJson = ConvertNestedJsonStructure(doc, nestedUdiToKeyMap, changesSummary);
        
        return convertedJson;
    }
    catch (JsonException)
    {
        return blockJson;
    }
}

private static string ConvertNestedJsonStructure(JsonDocument originalDoc, Dictionary<string, string> udiToKeyMap, List<string> changesSummary)
{
    var options = new JsonWriterOptions { Indented = false };
    using var stream = new MemoryStream();
    using var writer = new Utf8JsonWriter(stream, options);

    writer.WriteStartObject();

    foreach (var property in originalDoc.RootElement.EnumerateObject())
    {
        switch (property.Name.ToLower())
        {
            case "contentdata":
                writer.WritePropertyName("contentData");
                writer.WriteStartArray();
                
                var exposeEntries = new List<string>();
                foreach (var contentItem in property.Value.EnumerateArray())
                {
                    var contentKey = ConvertContentDataItem(writer, contentItem, udiToKeyMap, changesSummary);
                    if (!string.IsNullOrEmpty(contentKey))
                    {
                        exposeEntries.Add(contentKey);
                    }
                }
                
                writer.WriteEndArray();
                
                if (!originalDoc.RootElement.TryGetProperty("expose", out _))
                {
                    foreach (var key in exposeEntries)
                    {
                        udiToKeyMap[$"expose_{key}"] = key; // Store for expose section
                    }
                }
                break;

            case "settingsdata":
                writer.WritePropertyName("settingsData");
                writer.WriteStartArray();
                
                foreach (var settingsItem in property.Value.EnumerateArray())
                {
                    ConvertSettingsDataItem(writer, settingsItem, udiToKeyMap);
                }
                
                writer.WriteEndArray();
                break;

            case "layout":
                ConvertNestedLayoutStructure(writer, property.Value, udiToKeyMap);
                break;

            case "expose":
                property.WriteTo(writer);
                break;

            default:
                property.WriteTo(writer);
                break;
        }
    }

    if (!originalDoc.RootElement.TryGetProperty("expose", out _))
    {
        writer.WritePropertyName("expose");
        writer.WriteStartArray();
        
        var exposeKeys = udiToKeyMap.Where(kvp => kvp.Key.StartsWith("expose_")).Select(kvp => kvp.Value);
        foreach (var key in exposeKeys)
        {
            writer.WriteStartObject();
            writer.WriteString("contentKey", key);
            writer.WriteNull("culture");
            writer.WriteNull("segment");
            writer.WriteEndObject();
        }
        
        writer.WriteEndArray();
    }

    writer.WriteEndObject();
    writer.Flush();

    return System.Text.Encoding.UTF8.GetString(stream.ToArray());
}

private static void ConvertNestedLayoutStructure(Utf8JsonWriter writer, JsonElement layoutElement, Dictionary<string, string> udiToKeyMap)
{
    writer.WritePropertyName("Layout");
    writer.WriteStartObject();

    foreach (var layoutProperty in layoutElement.EnumerateObject())
    {
        writer.WritePropertyName(layoutProperty.Name);
        writer.WriteStartArray();

        foreach (var layoutItem in layoutProperty.Value.EnumerateArray())
        {
            writer.WriteStartObject();

            foreach (var itemProperty in layoutItem.EnumerateObject())
            {
                switch (itemProperty.Name.ToLower())
                {
                    case "contentudi":
                        var udiValue = itemProperty.Value.GetString();
                        if (!string.IsNullOrEmpty(udiValue) && udiToKeyMap.TryGetValue(udiValue, out var mappedKey))
                        {
                            writer.WriteString("contentKey", mappedKey);
                        }
                        else
                        {
                            writer.WriteNull("contentKey");
                        }
                        writer.WriteNull("contentUdi");
                        break;

                    case "settingsudi":
                        var settingsUdiValue = itemProperty.Value.GetString();
                        if (!string.IsNullOrEmpty(settingsUdiValue) && udiToKeyMap.TryGetValue(settingsUdiValue, out var mappedSettingsKey))
                        {
                            writer.WriteString("settingsKey", mappedSettingsKey);
                        }
                        else
                        {
                            writer.WriteNull("settingsKey");
                        }
                        writer.WriteNull("settingsUdi");
                        break;

                    default:
                        itemProperty.WriteTo(writer);
                        break;
                }
            }

            writer.WriteEndObject();
        }

        writer.WriteEndArray();
    }

    writer.WriteEndObject();
}

private static string ConvertLegacyLocalLinks(string markup, List<string> changesSummary)
{
    int conversions = 0;

    var result = LegacyLocalLinkPattern.Replace(markup, match =>
    {
        var guidValue = match.Groups[1].Value;
        
        if (Guid.TryParse(guidValue, out var guid))
        {
            conversions++;
            return $"/{{{guid}}}\" type=\"Document";
        }
        
        return match.Value;
    });

    return result;
}

private static string FixFaultyHrefAttributes(string markup, List<string> changesSummary)
{
    int fixes = 0;

    var result = FaultyHrefPattern.Replace(markup, match =>
    {
        var typeAttribute = match.Groups["typeAttribute"].Value;
        var localLink = match.Groups["localLink"].Value;

        if (!string.IsNullOrEmpty(typeAttribute) && !string.IsNullOrEmpty(localLink))
        {
            fixes++;
            var correctedHref = $"href=\"{localLink.Trim('\"')}\" {typeAttribute.Trim()}";
            return match.Value.Replace(match.Groups["faultyHref"].Value, correctedHref);
        }
        
        return match.Value;
    });

    return result;
}

private static bool TryParseGuidFromUdi(string udiString, out Guid guid)
{
    guid = Guid.Empty;
    
    if (udiString.StartsWith("umb://"))
    {
        var parts = udiString.Split('/');
        if (parts.Length >= 3 && Guid.TryParse(parts[2], out guid))
        {
            return true;
        }
    }
    else if (Guid.TryParse(udiString, out guid))
    {
        return true;
    }
    
    return false;
}

public class ConversionResult
{
    public string OriginalJson { get; set; } = "";
    public string UpdatedJson { get; set; } = "";
    public bool HasChanges { get; set; }
    public bool HasError { get; set; }
    public string? ErrorMessage { get; set; }
    public List<string> ChangesSummary { get; set; } = new();

1 Like

Hi,

Yes we missed a change there between the values in v13 and those in v16

the migrations in umbraco fix this if you upgrade (because they run on the db) but only once, and if you already have a site and drop the uSync file in then it’s down to us [uSync] to run something to fix the migration.

this is something we should support and so it’s something we are going to fix.

uSync issue

PR

Just need to test this and release the update, then this should be fine.

2 Likes

Fixed in uSync v16.0.1.

When “upgrading uSync files” with this version - it should do two things.

  1. it converts the RTE 's to use TipTap (unless the blocking config is in, which i think happens if you install the TinyMCE package).
  2. It updates the block attributes, (just like the migration).

There is other stuff going on and we are still working on migrations (e.g dumping v13 uSync folders into v16… sites). the aim is to make this seamless, but there will always bit little bits like missing packages, or format changes that might not get 100% mapped. where we can we are basically folding a very simple version of uSync.Migrations into the process. we will never directly support v8 to v16+ though that will require something more full on migrationy . which will probably have to live in the separate uSync.Migrations package.

1 Like

Sounds wonderful