[SEO Checker 13.5.0/13.5.1] Custom routed robots.txt results in 404

Questions
1

Hey all,

I’m currently facing a small issue where I’m trying to serve a custom robots.txt using SEO Checker 13.5.0 (and 13.5.1) by Richard Soeteman on Umbraco 13.7.2, but I am getting a 404 error.

I cannot use the built-in functionality because we need a specially crafted robots.txt (as always), which requires a custom implementation.

I want to serve a custom robots.txt via a RoutingComposer and a Controller:

 endpoints.MapControllerRoute(
     "Robots txt",
     "/robots.txt",
     new
     {
         controller = "Sitemap",
         action = "Robots"
     });

The problem is that the SEO Checker route is prioritized over my custom route, even though the option “Enable dynamic robots.txt file” in the settings tab is disabled.

In the database config JSON, I have:

...
"RobotsTxt":
    {
        "Enabled": false
    },
...

Logs:

3 candidate(s) found for the request path ‘“/robots.txt”’

Endpoint ‘“SEOChecker.Common.Controllers.SEOCheckerRobotsTxtController.Index (SEOChecker.Common)”’ with route pattern ‘“robots.txt”’ is valid for the request path ‘“/robots.txt”’

Endpoint ‘“MyProject.Controllers.MVC.SitemapController.Robots (MyProject)”’ with route pattern ‘“/robots.txt”’ is valid for the request path ‘“/robots.txt”’

Endpoint ‘“/{**umbracoSlug}”’ with route pattern ‘“/{**umbracoSlug}”’ is valid for the request path ‘“/robots.txt”’

Request matched endpoint ‘“SEOChecker.Common.Controllers.SEOCheckerRobotsTxtController.Index (SEOChecker.Common)”’

Executing endpoint ‘“SEOChecker.Common.Controllers.SEOCheckerRobotsTxtController.Index (SEOChecker.Common)”’

Executed endpoint ‘“SEOChecker.Common.Controllers.SEOCheckerRobotsTxtController.Index (SEOChecker.Common)”’

1 candidate(s) found for the request path ‘“/404-page-not-found/”’

As a workaround, I implemented a custom EndpointSelector which works but feels somewhat dirty:

public override Task SelectAsync(HttpContext httpContext, CandidateSet candidates)
{
    var state = candidates[0];

    if (httpContext.Request.Path.StartsWithSegments("/robots.txt"))
    {
        state = candidates[1];                
    }

    httpContext.SetEndpoint(state.Endpoint);
    httpContext.Request.RouteValues = state.Values!;

    return Task.CompletedTask;
}

Is this a bug or am I doing something wrong?

Thanks in advance!

All the best,
Silvano

1 Like