Repository navigation
Serve the site with a per-request file middleware instead of MapStaticAssets - #2178
Conversation
…cAssets MapStaticAssets needs a manifest of every file of the web root. With the site's ~70,500 files (8.4 GB), building it dominates every StaticHost build and publish, and each instance loads it at startup into one route per file. StaticHost now turns static web assets off and serves the web root with SiteFileMiddleware: - It looks the file up per request, prefers the brotli copy next to it when the client accepts br, and answers single byte ranges. - It computes the ETag (base64 SHA-256, the same value MapStaticAssets sent) the first time a file is served and caches it in memory. - It keeps the existing caching: immutable for /_astro/, the SDK's one-hour default for images, video and fonts, no-cache otherwise. - It steps aside when an endpoint matched, so API, health and download endpoints keep precedence. Site.targets publishes the frontend's output to wwwroot, as the static web assets pipeline did: it asks frontend.esproj for JavaScriptOutputGroup (which runs its BuildCommand) and writes a brotli copy (Optimal, the level used before) of each text file, reusing copies newer than their source. It replaces Caching.targets. The Frontend Build job no longer copies dist into wwwroot and rebuilds StaticHost to regenerate the manifest. Measured on a 4-core VM against the real production build of the site, current setup vs this change: - dotnet publish (Astro build excluded): 213 s -> 55 s (22 s when the brotli copies are reused) - CI-style build with dist in wwwroot: 86 s -> 9 s - Endpoints manifest: 142 MB -> none - First response after start: 13-18 s -> 2.3-5.8 s (the rest is startup both share) - Memory after warm load: 2.1-2.2 GB -> 280 MB RSS - Each of the 33k pages once: 1,450 -> 3,900 req/s, p99 56 -> 9 ms - Warm load: 5,700-5,850 -> 10,100-10,800 req/s, p99 21 -> 9 ms Comparing both servers on every page and file, with and without br (207,786 responses), Content-Type, Cache-Control, ETag, Vary and Content-Length match. The differences: Last-Modified on brotli responses is now the file's own, 4 tiny files no longer get a brotli copy that is not smaller, and StaticHost's own wwwroot (.gitignore, the dev-only Scalar theme) is no longer published. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01871HTuV3jb825jyAUQ3L7d
Frontend HTML artifact readyThe latest frontend build uploaded the This comment updates automatically when a new frontend build artifact is uploaded. |
Krish Khinchi (krishkhinchi)
left a comment
There was a problem hiding this comment.
The middleware approach looks promising, especially the reduction in startup memory and build time. However, there are a few compatibility concerns worth addressing before merging, particularly around dotnet run no longer serving src/frontend/dist, the changed POST behavior for static file paths, and the Last-Modified difference for Brotli responses. The inline MSBuild Brotli task also adds some complexity and would benefit from a more maintainable implementation. Additionally, the container build, App Service deployment, and GitHub Actions workflow still need validation. Recommendation: Request Changes until these deployment and compatibility edge cases are verified.
- Write the brotli copies with src/statichost/compress-site.cs, a file-based app that Site.targets runs, instead of an inline MSBuild task that loaded BrotliStream by reflection (which fails under .NET Framework MSBuild). - Reuse a brotli copy only while it carries its source's exact timestamp. A source replaced by new content with an older timestamp (a cache restore or an archive extraction) kept the copy of the old content. - Answer If-Match and If-Unmodified-Since with 412, as MapStaticAssets did. - Apply a date in If-Range only to uncompressed responses: Last-Modified is the original file's, so it does not identify the bytes of its brotli copy. - Key cached ETags by the file's path in the web root, not its physical path or bare name, and parse the ETag once per request.
- Compute ETags with XxHash128 instead of SHA-256: an ETag only has to tell versions of a file apart, which needs no cryptographic hash. It still hashes the content, so unchanged files keep their ETag across deployments. - Say why the brotli copy is looked up even for clients that do not accept brotli (the Vary header both versions need), and why cached ETags are keyed by ordinal path.
Request paths that differ only by empty or dot segments (/a/b.json, /a//b.json, /a/./b.json) name one file, and the file provider resolves them to the same physical path. Keyed by request path, each spelling added an entry and hashed the file again, so a client could grow the cache without bound and force a full read of any file on every request. Keyed by physical path, the cache holds at most one entry per file of the web root. Concurrent first requests for a file may still both hash it; that happens once per file per instance, and serving the file reads it anyway.
The static asset endpoints allowed only GET and HEAD, so routing matched a file's path and answered 405 for other methods. The middleware stepped aside for them instead, and a POST to a file fell through to the 404 page. It now answers 405 with Allow: GET, HEAD; a path with no file still gets 404.
There was a problem hiding this comment.
🟡 Changes recommended
Range processing incorrectly applies to HEAD requests, producing partial-content or 416 responses instead of the full representation’s headers required by RFC 9110.
1 open issue
What changed in this PR
Replaces StaticHost’s manifest-based MapStaticAssets serving with per-request file middleware to reduce build, startup, and memory costs for the documentation site.
Changes:
- Adds file serving with Brotli negotiation, cached content-based ETags, conditional responses, and byte ranges.
- Replaces the static web assets publishing pipeline with explicit site publishing and reusable Brotli compression.
- Removes the manifest-generation rebuild from CI and adds middleware tests.
| File | Description |
|---|---|
| tests/StaticHost.Tests/SiteFileTests.cs | Adds file-serving and HTTP behavior tests. |
| tests/StaticHost.Tests/AgentReadinessTestServer.cs | Uses the new middleware in the test server. |
| src/statichost/StaticHost/StaticHost.csproj | Disables static web assets and imports site publishing targets. |
| src/statichost/StaticHost/SiteFiles/SiteFileMiddleware.cs | Implements per-request static file serving. |
| src/statichost/StaticHost/SiteFiles/SiteFileExtensions.cs | Adds service and middleware registration helpers. |
| src/statichost/StaticHost/SiteFiles/SiteFileETags.cs | Caches content hashes by file path and metadata. |
| src/statichost/StaticHost/Site.targets | Publishes frontend output and compressed copies. |
| src/statichost/StaticHost/Program.cs | Registers file serving in place of MapStaticAssets. |
| src/statichost/StaticHost/GlobalUsings.cs | Imports the site-file namespace. |
| src/statichost/StaticHost/Caching.targets | Removes manifest-based caching configuration. |
| src/statichost/StaticHost/AgentReadiness/MarkdownNegotiationMiddleware.cs | Updates comments to reflect the new file-serving middleware. |
| src/statichost/compress-site.cs | Generates and reuses Brotli copies for publishing. |
| src/frontend/playwright.config.mjs | Updates comments describing CI hosting. |
| .github/workflows/frontend-build.yml | Removes the site-copy and manifest-rebuild step. |
Review effort: Balanced
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Ignore Range on HEAD (RFC 9110, 14.2): HEAD gets the whole file's headers. - Choose brotli or identity by Accept-Encoding quality values; identity when neither is acceptable. - Hash the file again for requests with If-Match, If-None-Match or If-Range, so a file replaced with the same length and time cannot get a stale 304. - Reuse a brotli copy only when its source's SHA-256 matches, not its time. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Summary
Draft for feedback: StaticHost serves the site with a small per-request file middleware instead of
MapStaticAssets. Eric Erhardt (@eerhardt) David Pine (@IEvangelist), I'd value your take on the approach before I take it further.Why.
MapStaticAssetsneeds a manifest of every file of the web root. With the site's ~70,500 files (8.4 GB), building it dominates every StaticHost build and publish (the SDK's static web assets targets), and every instance loads it at startup and builds one route per file. Front Door doesn't help: the route has no caching (x-cache: CONFIG_NOCACHEon every response), so the origin handles all compression, ETags and 304s.What changes.
SiteFileMiddlewarereplacesMapStaticAssets. Per request, it looks the file up in the web root (asUseStaticFilesdoes), serves the.brcopy next to it when the client accepts brotli, and answers single byte ranges (Safari needs them for the.mp4on the AI coding agents page). It runs afterUseRoutingand steps aside when an endpoint matched, so the API, health and download endpoints keep precedence.SiteFileETags). A content hash stays the same across deployments for files that didn't change. The cache is keyed by physical path, so it holds at most one entry per file, about 55 MB if all 126k files and brotli copies are requested on one instance. The values differ from the SHA-256MapStaticAssetssent, so each browser downloads each file once more after the switch.MapStaticAssetsdid: If-Match and If-Unmodified-Since answer 412, If-None-Match and If-Modified-Since 304. A date in If-Range applies only to uncompressed responses, since Last-Modified is the original file's.immutablefor/_astro/, the SDK'smax-age=3600, must-revalidatefor images, video and fonts,no-cacheotherwise.StaticWebAssetsEnabled=false.Site.targets(replacingCaching.targets) publishes the site towwwrootas the static web assets pipeline did: it asksfrontend.esprojforJavaScriptOutputGroup(which runs itsBuildCommand), then runs a file-based app,src/statichost/compress-site.cs, that writes a brotli copy of each text file atOptimal, the level used before. A copy is reused while it carries its source's exact timestamp.frontend-build.yml: the Frontend Build job no longer copiesdistintowwwrootand rebuilds StaticHost just to regenerate the manifest.Measured on a 4-core VM against a real
build:productionof the site, current setup vs this PR:dotnet publish(Astro build excluded)distinwwwrootThe publish times were measured with an earlier inline MSBuild task, which the file-based app replaced; it does the same work.
Behavior compared by running both servers side by side over every page and file, with and without
Accept-Encoding: br(207,786 responses): Content-Type, Cache-Control, Vary and Content-Length match, and so did ETag before the switch to XxHash128. The only differences:.brfile's.wwwrootis no longer published. Production currently serveshttps://aspire.dev/.gitignoreand the dev-only Scalar theme.Open questions.
dotnet runno longer includessrc/frontend/distin what it serves (the static web assets runtime manifest did that); it serves only its ownwwwroot, whichbuild:statichostfills. Does anything rely on the old behavior?/docs//page/, now return the page. Endpoint routing likely answered 404 (not compared). Should they 404, or redirect to the canonical path?Third-party links and affiliations
None.
Validation
tests/StaticHost.Tests: 46 new tests inSiteFileTests(brotli negotiation, ETags and 304s, If-Match and If-Unmodified-Since, If-Modified-Since, ranges and If-Range, cache headers, unknown types, HEAD, endpoint precedence, one cache entry per file). The existing test server now uses the new middleware. All 412 tests pass; the Redis integration tests were not run (no Docker daemon in my environment).distand compared the output: the same 70,531 site files and 1,604 MB of brotli copies. With the file-based app, a publish of a local build (68,388 files) writes 57,615 brotli copies, skipping 4 that aren't smaller, and recompresses only files whose timestamp changed.🤖 Generated with Claude Code
https://claude.ai/code/session_01871HTuV3jb825jyAUQ3L7d