Skip to content

Serve the site with a per-request file middleware instead of MapStaticAssets - #2178

Merged
David Pine (IEvangelist) merged 7 commits into
mainfrom
sebros/optimistic-babbage-j11aeu
Nov 20, 2026
Merged

David Pine (IEvangelist) merged 7 commits into
mainfrom
sebros/optimistic-babbage-j11aeu

Conversation

@sebastienros

@sebastienros Sébastien Ros (sebastienros) commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Draft for feedback: StaticHost serves the site with a small per-request file middleware instead of MapStaticAssets. Eric Erhardt (@eerhardt) David Pine (@IEvangelist), I'd value your take on the approach before I take it further.

Why. MapStaticAssets needs a manifest of every file of the web root. With the site's ~70,500 files (8.4 GB), building it dominates every StaticHost build and publish (the SDK's static web assets targets), and every instance loads it at startup and builds one route per file. Front Door doesn't help: the route has no caching (x-cache: CONFIG_NOCACHE on every response), so the origin handles all compression, ETags and 304s.

What changes.

  • SiteFileMiddleware replaces MapStaticAssets. Per request, it looks the file up in the web root (as UseStaticFiles does), serves the .br copy next to it when the client accepts brotli, and answers single byte ranges (Safari needs them for the .mp4 on the AI coding agents page). It runs after UseRouting and steps aside when an endpoint matched, so the API, health and download endpoints keep precedence.
  • ETags are a hash of the file's content (XxHash128), computed the first time a file is served and kept in memory (SiteFileETags). A content hash stays the same across deployments for files that didn't change. The cache is keyed by physical path, so it holds at most one entry per file, about 55 MB if all 126k files and brotli copies are requested on one instance. The values differ from the SHA-256 MapStaticAssets sent, so each browser downloads each file once more after the switch.
  • Preconditions follow RFC 9110, as MapStaticAssets did: If-Match and If-Unmodified-Since answer 412, If-None-Match and If-Modified-Since 304. A date in If-Range applies only to uncompressed responses, since Last-Modified is the original file's.
  • Caching headers are unchanged: immutable for /_astro/, the SDK's max-age=3600, must-revalidate for images, video and fonts, no-cache otherwise.
  • The project sets StaticWebAssetsEnabled=false. Site.targets (replacing Caching.targets) publishes the site to wwwroot as the static web assets pipeline did: it asks frontend.esproj for JavaScriptOutputGroup (which runs its BuildCommand), then runs a file-based app, src/statichost/compress-site.cs, that writes a brotli copy of each text file at Optimal, the level used before. A copy is reused while it carries its source's exact timestamp.
  • frontend-build.yml: the Frontend Build job no longer copies dist into wwwroot and rebuilds StaticHost just to regenerate the manifest.

Measured on a 4-core VM against a real build:production of the site, current setup vs this PR:

Current This PR
dotnet publish (Astro build excluded) 213 s 55 s (22 s when the brotli copies are reused)
CI-style build with dist in wwwroot 86 s 9 s
Endpoints manifest 142 MB none
First response after start 13–18 s 2.3–5.8 s (the rest is startup both share)
Memory (RSS) after warm load 2.1–2.2 GB 280 MB
Each of the 33k pages once 1,450 req/s, p99 56 ms 3,900 req/s, p99 9 ms
Warm load (20 s, 32 connections) ~5,800 req/s, p99 21 ms ~10,400 req/s, p99 9 ms

The publish times were measured with an earlier inline MSBuild task, which the file-based app replaced; it does the same work.

Behavior compared by running both servers side by side over every page and file, with and without Accept-Encoding: br (207,786 responses): Content-Type, Cache-Control, Vary and Content-Length match, and so did ETag before the switch to XxHash128. The only differences:

  • Last-Modified on brotli responses is now the original file's, not the .br file's.
  • 4 tiny files no longer get a brotli copy that isn't smaller than the file.
  • ETags are XxHash128 instead of SHA-256 (above).
  • StaticHost's own wwwroot is no longer published. Production currently serves https://aspire.dev/.gitignore and the dev-only Scalar theme.
  • POST to a file path now gets 404. The static asset endpoints only allowed GET and HEAD, so they likely answered 405 (not compared).

Open questions.

  • Running StaticHost with dotnet run no longer includes src/frontend/dist in what it serves (the static web assets runtime manifest did that); it serves only its own wwwroot, which build:statichost fills. Does anything rely on the old behavior?
  • URLs with empty segments, such as /docs//page/, now return the page. Endpoint routing likely answered 404 (not compared). Should they 404, or redirect to the canonical path?

Third-party links and affiliations

None.

Validation

  • tests/StaticHost.Tests: 46 new tests in SiteFileTests (brotli negotiation, ETags and 304s, If-Match and If-Unmodified-Since, If-Modified-Since, ranges and If-Range, cache headers, unknown types, HEAD, endpoint precedence, one cache entry per file). The existing test server now uses the new middleware. All 412 tests pass; the Redis integration tests were not run (no Docker daemon in my environment).
  • Published both setups from the same real dist and compared the output: the same 70,531 site files and 1,604 MB of brotli copies. With the file-based app, a publish of a local build (68,388 files) writes 57,615 brotli copies, skipping 4 that aren't smaller, and recompresses only files whose timestamp changed.
  • Header comparison and benchmarks above, plus spot checks of 304s, Markdown negotiation, canonical redirects, the 404 page and video ranges.
  • Not yet validated: this workflow in Actions, the container build and App Service.

🤖 Generated with Claude Code

https://claude.ai/code/session_01871HTuV3jb825jyAUQ3L7d

…cAssets

MapStaticAssets needs a manifest of every file of the web root. With the
site's ~70,500 files (8.4 GB), building it dominates every StaticHost
build and publish, and each instance loads it at startup into one route
per file.

StaticHost now turns static web assets off and serves the web root with
SiteFileMiddleware:

- It looks the file up per request, prefers the brotli copy next to it
  when the client accepts br, and answers single byte ranges.
- It computes the ETag (base64 SHA-256, the same value MapStaticAssets
  sent) the first time a file is served and caches it in memory.
- It keeps the existing caching: immutable for /_astro/, the SDK's
  one-hour default for images, video and fonts, no-cache otherwise.
- It steps aside when an endpoint matched, so API, health and download
  endpoints keep precedence.

Site.targets publishes the frontend's output to wwwroot, as the static
web assets pipeline did: it asks frontend.esproj for JavaScriptOutputGroup
(which runs its BuildCommand) and writes a brotli copy (Optimal, the level
used before) of each text file, reusing copies newer than their source.
It replaces Caching.targets.

The Frontend Build job no longer copies dist into wwwroot and rebuilds
StaticHost to regenerate the manifest.

Measured on a 4-core VM against the real production build of the site,
current setup vs this change:

- dotnet publish (Astro build excluded): 213 s -> 55 s (22 s when the
  brotli copies are reused)
- CI-style build with dist in wwwroot: 86 s -> 9 s
- Endpoints manifest: 142 MB -> none
- First response after start: 13-18 s -> 2.3-5.8 s (the rest is
  startup both share)
- Memory after warm load: 2.1-2.2 GB -> 280 MB RSS
- Each of the 33k pages once: 1,450 -> 3,900 req/s, p99 56 -> 9 ms
- Warm load: 5,700-5,850 -> 10,100-10,800 req/s, p99 21 -> 9 ms

Comparing both servers on every page and file, with and without br
(207,786 responses), Content-Type, Cache-Control, ETag, Vary and
Content-Length match. The differences: Last-Modified on brotli responses
is now the file's own, 4 tiny files no longer get a brotli copy that is
not smaller, and StaticHost's own wwwroot (.gitignore, the dev-only
Scalar theme) is no longer published.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01871HTuV3jb825jyAUQ3L7d
@aspire-repo-bot

Copy link
Copy Markdown
Contributor

Frontend HTML artifact ready

The latest frontend build uploaded the frontend-dist artifact for PR #2178. Use the VS Code button below to open this PR with GitHub Artifacts Explorer and browse the built HTML locally.

VS Code: Open PR #2178 artifacts

This comment updates automatically when a new frontend build artifact is uploaded.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The middleware approach looks promising, especially the reduction in startup memory and build time. However, there are a few compatibility concerns worth addressing before merging, particularly around dotnet run no longer serving src/frontend/dist, the changed POST behavior for static file paths, and the Last-Modified difference for Brotli responses. The inline MSBuild Brotli task also adds some complexity and would benefit from a more maintainable implementation. Additionally, the container build, App Service deployment, and GitHub Actions workflow still need validation. Recommendation: Request Changes until these deployment and compatibility edge cases are verified.

krishkhinchi

This comment was marked as off-topic.

- Write the brotli copies with src/statichost/compress-site.cs, a file-based
  app that Site.targets runs, instead of an inline MSBuild task that loaded
  BrotliStream by reflection (which fails under .NET Framework MSBuild).
- Reuse a brotli copy only while it carries its source's exact timestamp. A
  source replaced by new content with an older timestamp (a cache restore or
  an archive extraction) kept the copy of the old content.
- Answer If-Match and If-Unmodified-Since with 412, as MapStaticAssets did.
- Apply a date in If-Range only to uncompressed responses: Last-Modified is
  the original file's, so it does not identify the bytes of its brotli copy.
- Key cached ETags by the file's path in the web root, not its physical path
  or bare name, and parse the ETag once per request.
- Compute ETags with XxHash128 instead of SHA-256: an ETag only has to tell
  versions of a file apart, which needs no cryptographic hash. It still
  hashes the content, so unchanged files keep their ETag across deployments.
- Say why the brotli copy is looked up even for clients that do not accept
  brotli (the Vary header both versions need), and why cached ETags are
  keyed by ordinal path.
Request paths that differ only by empty or dot segments (/a/b.json,
/a//b.json, /a/./b.json) name one file, and the file provider resolves them
to the same physical path. Keyed by request path, each spelling added an
entry and hashed the file again, so a client could grow the cache without
bound and force a full read of any file on every request. Keyed by physical
path, the cache holds at most one entry per file of the web root.

Concurrent first requests for a file may still both hash it; that happens
once per file per instance, and serving the file reads it anyway.
The static asset endpoints allowed only GET and HEAD, so routing matched a
file's path and answered 405 for other methods. The middleware stepped aside
for them instead, and a POST to a file fell through to the 404 page. It now
answers 405 with Allow: GET, HEAD; a path with no file still gets 404.
@sebastienros
Sébastien Ros (sebastienros) marked this pull request as ready for review November 18, 2026 18:39
Copilot AI balanced review requested due to automatic review settings November 18, 2026 18:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Range processing incorrectly applies to HEAD requests, producing partial-content or 416 responses instead of the full representation’s headers required by RFC 9110.

1 open issue
What changed in this PR

Replaces StaticHost’s manifest-based MapStaticAssets serving with per-request file middleware to reduce build, startup, and memory costs for the documentation site.

Changes:

  • Adds file serving with Brotli negotiation, cached content-based ETags, conditional responses, and byte ranges.
  • Replaces the static web assets publishing pipeline with explicit site publishing and reusable Brotli compression.
  • Removes the manifest-generation rebuild from CI and adds middleware tests.
File Description
tests/​StaticHost.Tests/​SiteFileTests.cs Adds file-serving and HTTP behavior tests.
tests/​StaticHost.Tests/​AgentReadinessTestServer.cs Uses the new middleware in the test server.
src/​statichost/​StaticHost/​StaticHost.csproj Disables static web assets and imports site publishing targets.
src/​statichost/​StaticHost/​SiteFiles/​SiteFileMiddleware.cs Implements per-request static file serving.
src/​statichost/​StaticHost/​SiteFiles/​SiteFileExtensions.cs Adds service and middleware registration helpers.
src/​statichost/​StaticHost/​SiteFiles/​SiteFileETags.cs Caches content hashes by file path and metadata.
src/​statichost/​StaticHost/​Site.targets Publishes frontend output and compressed copies.
src/​statichost/​StaticHost/​Program.cs Registers file serving in place of MapStaticAssets.
src/​statichost/​StaticHost/​GlobalUsings.cs Imports the site-file namespace.
src/​statichost/​StaticHost/​Caching.targets Removes manifest-based caching configuration.
src/​statichost/​StaticHost/​AgentReadiness/​MarkdownNegotiationMiddleware.cs Updates comments to reflect the new file-serving middleware.
src/​statichost/​compress-site.cs Generates and reuses Brotli copies for publishing.
src/​frontend/​playwright.config.mjs Updates comments describing CI hosting.
.github/​workflows/​frontend-build.yml Removes the site-copy and manifest-rebuild step.

Review effort: Balanced

Comment thread src/statichost/compress-site.cs Resolved
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Ignore Range on HEAD (RFC 9110, 14.2): HEAD gets the whole file's headers.
- Choose brotli or identity by Accept-Encoding quality values; identity when
  neither is acceptable.
- Hash the file again for requests with If-Match, If-None-Match or If-Range,
  so a file replaced with the same length and time cannot get a stale 304.
- Reuse a brotli copy only when its source's SHA-256 matches, not its time.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@IEvangelist
David Pine (IEvangelist) merged commit 67f4c2c into main Nov 20, 2026
21 checks passed
@IEvangelist
David Pine (IEvangelist) deleted the sebros/optimistic-babbage-j11aeu branch November 20, 2026 18:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants