uSync Complete Pull/Sync Pack fails with large amount of Media

Questions
, ,
1

Hi all, this is my first time using uSync Complete, so apologies if I’m missing something simple here, but I’m having an issue where Pulling and Sync Pack generation seems to fail on the Zip Folder step.

I installed uSync Complete onto my client’s project, and to test it I am trying to sync between our Development deployment and my Localhost.

For this test, the Development environment is an Azure Web App that is using an anonymized copy of the prod database, and my localhost is using a blank database.

The uSync Publisher interface shows a configuration for pushing and pulling content between Umbraco installations, with a local server highlighted and options to add new servers for synchronization. (Captioned by AI)
The uSync Publisher interface shows a configuration for pushing and pulling content between Umbraco installations, with a local server highlighted and options to add new servers for synchronization. (Captioned by AI)1339×514 37.9 KB

I first tried to Pull the entire content tree (from Dev to Local), but that failed due to the Dev azure web app restarting… I tried this twice, and both attempts ended with the Dev web app restarting. The last thing the UI showed was the Media step with media item “logo-2.png”.

So next I tried generating a Sync Pack, and that ended in the same way as the Pull attempts above.

A "Create a sync pack" modal window shows the progress of calculating, exporting, and getting media, with "Exporting...Logo 2" currently in progress. (Captioned by AI)
A "Create a sync pack" modal window shows the progress of calculating, exporting, and getting media, with "Exporting...Logo 2" currently in progress. (Captioned by AI)776×530 71.4 KB

Looking through the logs on the Dev Web App, there is no errors logged.

The image shows a log display with timestamps, information or warning indicators, and messages related to an application shutting down and releasing resources. (Captioned by AI)
The image shows a log display with timestamps, information or warning indicators, and messages related to an application shutting down and releasing resources. (Captioned by AI)1399×426 37.2 KB

So next step, I ran my localhost against the anonymized prod db copy, and tried to generate a Sync Pack on my laptop, with Debug logging enabled on the uSync namespace.

It failed on the same step as the Dev deployment, (the UI showed the Media step with the logo-2.png), but interestingly, the last entry in my Debug logs was the ZipFolder step:

The image shows a debugging log from a computer program, displaying timestamps, levels (Information and Debug), and messages related to file paths, media copying, and task completions. (Captioned by AI)
The image shows a debugging log from a computer program, displaying timestamps, levels (Information and Debug), and messages related to file paths, media copying, and task completions. (Captioned by AI)1334×700 78.8 KB

image

Checking my AppData/Local folder, I can see that it has created the extract folder, and it is around 5.4GB (media is around 5.3GB of that)

A screenshot of a Windows folder explorer window showing the properties of a folder named "21fa4021-c4de-47d6-b79f-812d9d25f7c5", which is located in the Temp directory and contains 28,099 files and 4,634 folders. (Captioned by AI)
A screenshot of a Windows folder explorer window showing the properties of a folder named "21fa4021-c4de-47d6-b79f-812d9d25f7c5", which is located in the Temp directory and contains 28,099 files and 4,634 folders. (Captioned by AI)1052×603 37.3 KB

I tried zipping with explorer, it took almost 20 minutes, but it worked.

The site has 13,779 content nodes and 17,031 media nodes.

Could this be a RAM issue? The Dev web app is a 64bit, on P0v3 plan with 4GB of RAM that’s shared between a few other web apps, and my laptop has 16GB (which regularly runs at 80-90%).
I’ve just bumped it up to a P1v3 and triggered a new sync pack creation, and will update with the reuslts.

Has anyone else used uSync Complete on a site with large amounts of content/media? Has anyone encountered this issue before?

Any help would be greatly appreciated :slight_smile:

2

Hi,

If you have really massive media libraries, you might want to tweak some of the pageSize and mediaPageSize settings in uSync.

by default uSync will do things in 25 items at a time and if there are lots then that can result in large zip files and things hitting the default size limits

also for publisher you can look at changing the max request size stuff so the big zip files can make it between servers.

*these are the publisher settings, there is also a page size setting for exporter - but i think you where just debugging with that ? ( uSync.Exporter | Jumoo docs )

1 Like
3

Cheers Kevin,

I’ll have a play around with the page sizes and request limits and report back :slight_smile:

4

It worked!

5 and half hours later and the Pull completed! :partying_face:

The image shows a screen indicating that a development process labeled "Dev Complete" has updated 11,929 items, with an indication that the content is fully updated. (Captioned by AI)
The image shows a screen indicating that a development process labeled "Dev Complete" has updated 11,929 items, with an indication that the content is fully updated. (Captioned by AI)498×294 11.9 KB

I followed your suggestions, and changed the maxRequestLength and maxAllowedContentLength, on my LocalHost project, to be the same as in your docs.

Although I did have to set maxRequestLength via the program.cs as I couldn’t get the <location> element to work, the /umbraco/uSyncReceive URLs all 404’d when I tried to use it, so I used a UseWhen this instead: (which seemed to work)

Program.cs

  app.UseWhen(context => context.Request.Path.StartsWithSegments("/umbraco/uSyncReceive"),
      appBuilder =>
      {
          appBuilder.Use(async (context, next) =>
          {
              var bodyControlFeature = context.Features.Get<Microsoft.AspNetCore.Http.Features.IHttpMaxRequestBodySizeFeature>();
              if (bodyControlFeature is { IsReadOnly: false })
              {
                  bodyControlFeature.MaxRequestBodySize = 524_288_000; // 500MB
              }
              await next();
          });
      });

web.config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
	<system.webServer>
		<security>
			<requestFiltering>
				<!-- 500MB in bytes -->
				<requestLimits maxAllowedContentLength="524288000" />
			</requestFiltering>
		</security>
	</system.webServer>
</configuration>

I also added the page size values on both my local and remote dev:

"PageSize": 50,
"MediaPageSize": 5

And finally, I also upped the App Service plan to P1v3.

Unsure which of those changes got it to complete, but it has :smiley:

Thanks for the advice Kevin! :umbraco-heart: