Drain guest responses before scale-to-zero

[tnsardesai]

problem

An HTTP handler can return while response bytes remain unacknowledged in the guest TCP send queue. Re-enabling scale-to-zero at that point can suspend the guest before the peer receives the response tail.

change

This implements the fix entirely inside the guest image. It requires no caller or control-plane protocol changes.

• track every non-loopback HTTP response on the API, DevTools, and ChromeDriver listeners
• retain the request's scale-to-zero hold until the handler returns and the response reaches a safe terminal state
• poll Linux TIOCOUTQ after net/http finalizes explicit response framing
• use adaptive asynchronous polling so keep-alive requests do not wait on a fixed polling interval
• cancel an older idle drain when the next request starts, carrying its hold forward until the connection next drains
• transfer older request holds only after the next request acquires its hold, bounding retained keep-alive state to one request without creating a scale-to-zero gap
• apply rolling five-minute write deadlines so blocked writes terminate without limiting healthy long-lived streams
• preserve rolling deadlines through io.ReaderFrom/sendfile transfers by advancing them once per 1 MiB chunk
• release hijacked HTTP connections from response-tail tracking while retaining the handler-lifetime hold, preserving prompt WebSocket cancellation and close behavior
• preserve close-delimited HTTP semantics by duplicating the socket, sending FIN with shutdown(SHUT_WR), closing the Go connection promptly, and retaining the hold until TCP acknowledges the close
• abort stalled or uninspectable sockets with SO_LINGER=0
• recover terminalization failures through bounded abort retries or duplicated-socket monitoring with TCP_USER_TIMEOUT; if no safe terminal state can be established, terminate the whole guest without releasing the hold
• terminate failed-disable requests with http.ErrAbortHandler, preventing net/http from synthesizing a successful response for an operation that never ran
• expose drain outcome counters plus active and fail-closed hold gauges

The release invariant is: response data and required close framing were acknowledged outside the guest, or the connection was abortively terminated. A timeout, inspection failure, or cleanup failure alone never releases the hold.

tests

• go build ./...
• go vet ./...
• go test -race ./lib/scaletozero ./lib/metrics ./cmd/api/api
• go test -race $(go list ./... | grep -v '/e2e$')
• 50 repeated race-enabled lifecycle, WebSocket, shutdown, framing, keep-alive, slow-reader, timeout, and terminal-recovery tests
• a progressive sendfile transfer runs longer than its configured timeout while refreshing the deadline for every chunk
• 1,000 pipelined requests on one connection retain one pending hold, never physically re-enable between requests, complete every response, and return all gauges to baseline after the final drain
• persistent abort, socket duplication, linger, inspection, and TCP_USER_TIMEOUT failures reach bounded guest termination without releasing the hold
• Darwin amd64 and Linux arm64 compile checks for the scale-to-zero package
• production-runtime A/B with a 16 MiB response and approximately 200 KiB/s reader: the unchanged image stalled 106,496 bytes short, while patched fresh guests repeatedly delivered the complete checksum without a caller-side keepalive or wake fallback
• a patched guest resumed from standby, completed the same slow transfer, and returned to standby afterward