Azure hosted Umbraco site occassionally fails

Questions
1

I receive azure montioring alerts that my umbraco site is failing. Looking at the logs in Umbraco I see this

System.IO.IOException: Unexpected end of Stream, the content may have already been read by another component.
at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
at System.IO.StreamReader.ReadBufferAsync(CancellationToken cancellationToken)
at System.IO.StreamReader.ReadToEndAsyncInternal(CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.MultipartSectionStreamExtensions.ReadAsStringAsync(MultipartSection section, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Http.Features.FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Http.Features.FormFeature.ReadForm()
at Umbraco.Cms.Core.Routing.PublishedRouter.FindPublishedContent(IPublishedRequestBuilder request)
at Umbraco.Cms.Core.Routing.PublishedRouter.RouteRequestInternalAsync(IPublishedRequestBuilder builder, Boolean skipContentFinders)
at Umbraco.Cms.Core.Routing.PublishedRouter.RouteRequestAsync(IPublishedRequestBuilder builder, RouteRequestOptions options)
at Umbraco.Cms.Web.Website.Routing.UmbracoRouteValueTransformer.RouteRequestAsync(IUmbracoContext umbracoContext)
at Umbraco.Cms.Web.Website.Routing.UmbracoRouteValueTransformer.TransformAsync(HttpContext httpContext, RouteValueDictionary values)
at Microsoft.AspNetCore.Mvc.Routing.DynamicControllerEndpointMatcherPolicy.ApplyAsync(HttpContext httpContext, CandidateSet candidates)
at Microsoft.AspNetCore.Routing.Matching.DfaMatcher.SelectEndpointWithPoliciesAsync(HttpContext httpContext, IEndpointSelectorPolicy[] policies, CandidateSet candidateSet)
at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.<Invoke>g__AwaitMatch|10_1(EndpointRoutingMiddleware middleware, HttpContext httpContext, Task matchTask)
at Umbraco.Cms.Web.Common.Middleware.ProtectRecycleBinMediaMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at StackExchange.Profiling.MiniProfilerMiddleware.Invoke(HttpContext context) in C:\projects\dotnet\src\MiniProfiler.AspNetCore\MiniProfilerMiddleware.cs:line 112
at Umbraco.Cms.Web.Common.Middleware.UmbracoRequestMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Umbraco.Cms.Web.Common.Middleware.UmbracoRequestMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Umbraco.Cms.Web.Common.Middleware.PreviewAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Umbraco.Cms.Web.Common.Middleware.BootFailedMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Umbraco.Cms.Web.Common.Middleware.UmbracoRequestLoggingMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at SixLabors.ImageSharp.Web.Middleware.ImageSharpMiddleware.Invoke(HttpContext httpContext, Boolean retry)
at Umbraco.Cms.Api.Management.Middleware.BackOfficeAuthorizationInitializationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Umbraco.SITENAME.Middleware.XFrameOptionsMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in E:\PrivateRepos\Umbraco.SITENAME\Umbraco.SITENAME\Middleware\XFrameOptionsMiddleware.cs:line 8
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Umbraco.SITENAME.Middleware.NoSniffMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in E:\PrivateRepos\Umbraco.SITENAME\Umbraco.SITENAME\Middleware\NoSniffMiddleware.cs:line 8
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()

Umbraco system information
--------------------------------
Server Troubleshooting: 
Server OS: Microsoft Windows 10.0.20348
Server Framework: .NET 10.0.8
Default Language: en
Umbraco Version: 17.4.0
Current Culture: en-US
Current UI Culture: en-US
Current Webserver: IIS
Models Builder Mode: InMemoryAuto
Runtime Mode: BackofficeDevelopment
Debug Mode: False
Database Provider: Microsoft.Data.SqlClient
Current Server Role: Single

Server Information: 
Umbraco build version: 17.4.0+3aa87fe
Umbraco assembly version: 17.4.0
Server time offset: 00:00:00
Runtime mode: BackofficeDevelopment

Client Information: 
Umbraco client version: 17.4.0

Current user: 
User is admin: Yes
User sections: Umb.Section.Content, Umb.Section.Forms, Umb.Section.Media, Umb.Section.Members, Umb.Section.Packages, Umb.Section.Settings, Umb.Section.Translation, Umb.Section.Users
User culture: en-US
User languages: All
User document start nodes: 

Temporary file configuration: 
Max allowed file size: Not set (unlimited)
Allowed file types: 
Disallowed file types: ashx, aspx, ascx, config, cshtml, vbhtml, asmx, air, axd, xamlx
Image file types: jpeg, jpg, gif, bmp, png, tiff, tif, webp

Browser Troubleshooting: 
Browser (user agent): Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36
Browser language: en
Browser location: https://SITENAME.com/umbraco/section/settings/workspace/logviewer/view/search/?lq=Has(%40Exception)
2

Hi @magiczombie

Welcome to the forum!

Are you able to see the logs to find out which URLs cause this and what user agents as it could be the result of a bot attack or probing. Do you use a CDN or firewall on your site?

Justin

3

Hi @magiczombie ,

I was checking why and how it could have happened. Found few things which might be helpful: (have used AI for research on this)

The System.IO.IOException: Unexpected end of Stream error usually happens when ASP.NET Core tries to read an incoming HTTP request body (like a form POST), but the data stream cuts off abruptly before the transfer is complete.

Because the request appears to have a body, Umbraco’s PublishedRouter attempts to parse it using FormFeature.ReadForm. When it encounters an empty or broken stream, it throws this exception.

As @justin-nevitech mentioned look for URLs causing this and what agents, yes it could be a bot attack if the URLs are not pointing to the root as Bot tries some random URLs. As they purposely sends half of a payload and intentionally leaves off the closing boundary to see if your server crashes and leaks sensitive information in the error screen. If that’s the case your site is ignoring them and throwing this error.

There could be an issue on the Azure itself specifically on the Azure Monitor. When Azure checks your site’s health, they opens a connection and sends the HTTP request headers. To save resources, they perform a “hit-and-run.” If the probe is configured to send a POST request, it tells the server to expect a data payload. However, the millisecond the probe receives the initial HTTP response headers (like 200 OK), it forcefully drops the TCP connection before actually sending the body. Umbraco tries to read that missing body and crashes.

Your stack trace flags two custom files right before the crash:

  • XFrameOptionsMiddleware.cs (Line 8)
  • NoSniffMiddleware.cs (Line 8)

By default, the HTTP request body stream is forward-only and can only be read once. If here’s a logging or validation inside these middlewares that inspects context.Request.Body or context.Request.Form, they consume the stream. When the request passes down the pipeline to Umbraco, the stream is already empty, triggering the crash.

These are some solutions for these issues:

Solution 1: Use a Dedicated Health Endpoint (Recommended)

Do not point Azure Monitor at your root URL. Hitting the homepage forces the heavy Umbraco CMS router to run on every ping. Instead, use ASP.NET Core’s native health checks. This creates a bypass lane that immediately returns a 200 OK and stops processing before the request ever reaches Umbraco.

In your Program.cs:

// 1. Register the services
builder.Services.AddHealthChecks();

var app = builder.Build();

// 2. Map the endpoint BEFORE Umbraco's routing middleware
app.MapHealthChecks("/health");

app.UseUmbraco();
// ... rest of pipeline

Once deployed, update your Azure Monitor probe to ping [https://SITENAME.com/health](https://SITENAME.com/health).

Solution 2: Change the Azure Probe to GET

If the Azure probe is currently set to POST, change it to an HTTP GET request in the Azure Portal. A GET request has no body payload. Because there is no body, Umbraco will not trigger the MultipartReaderStream parser, completely neutralizing the error even if Azure drops the connection early.

Solution 3: Enable Buffering in Custom Middleware

If your custom XFrameOptionsMiddleware or NoSniffMiddleware actually needs to read the incoming request body, you must explicitly enable buffering and reset the stream position so Umbraco can read it afterward.

Update the middleware code:

public async Task InvokeAsync(HttpContext context, RequestDelegate next)
{
    // 1. Enable buffering at the start
    context.Request.EnableBuffering();

    // 2. Your custom logic reading the body stream goes here
    
    // 3. Reset the position to 0 BEFORE calling next()
    context.Request.Body.Position = 0;
    
    await next(context);
}

Resources used:

Hope it helps!

4

Thank you for quick and detailed response.

I’ve added additional logging on the app service to see what is going on at the time things starts to fail.

Also implemented solution 1 with the health endpoint, good suggestion!

When it comes to the middleware the 2 only append to the response header, like this

 public class XFrameOptionsMiddleware : IMiddleware
 {
     public async Task InvokeAsync(HttpContext context, RequestDelegate next)
     {
         context.Response.Headers.Append("X-Frame-Options", "SAMEORIGIN");
         await next(context);
     }
 }

Since the error is intermittent, but only observed since I upgraded to v17, I will have to wait it out a bit.

2 Likes