Skip to content

Fix concurrent same-content transfers, disk-full handling and small-frame writes - #45

Draft
TeoSlayer wants to merge 1 commit into
mainfrom
hosted-node-fixes
Draft

TeoSlayer wants to merge 1 commit into
mainfrom
hosted-node-fixes

Conversation

@TeoSlayer

Copy link
Copy Markdown
Contributor

Summary

Four fixes to file transfer and framing, found by running a node in a 1 GB container under load.

Changes

  • Several transfers of the same content at once no longer fail. The .partial is named after the content hash so a retry can resume it. Two peers sending the same file (or one sender on two connections) therefore wrote the same file, and all but the first failed at the final rename. The first transfer keeps the resumable .partial; a concurrent one gets a private file, removed if it does not finish.
  • A file that cannot fit on the receiver's disk is refused at INIT, keeping 16 MiB in reserve. The byte quota is a fixed number and can be larger than the disk: a transfer ran until the disk was full, failed, and left its bytes in .partial, where they kept the disk full and blocked incoming messages. If a write still hits ENOSPC, that .partial is deleted.
  • A refusal at INIT is reported as a refusal. The sender took any non-INIT_ACK reply for a receiver without streamed-transfer support, so the caller fell back to pushing the whole file in one frame at a peer that had just declined it.
  • WriteFrame sends a frame of up to 64 KiB as one write. Header and payload as two writes made the payload wait on Nagle and the peer's delayed ACK.

Wire format is unchanged. The free-space check uses statfs on unix and is skipped elsewhere.

Test Plan

  • go build ./..., go vet ./...
  • go test ./... -count=1 and -race
  • New tests: four concurrent transfers of one payload all succeed and leave no .partial; a transfer is refused when free space is one byte short and accepted when it fits; a disk-full write removes the .partial while other write errors keep it; small frames are one write and round-trip.
  • In a two-container lab: four concurrent transfers of the same 20 MB file went from 3 of 4 failing to 0 of 4.

🤖 Generated with Claude Code

…rame writes

Found by running a node in a 1 GB container under load.

- Several transfers of the same content at once failed: the .partial is
  named after the content hash, so they shared one file and all but the
  first failed at the final rename. The first transfer keeps the resumable
  .partial; a concurrent one gets a private file, removed if it does not
  finish.
- A file that cannot fit on the receiver's disk is refused at INIT (16 MiB
  kept in reserve) instead of running until the disk is full. If a write
  still hits ENOSPC the .partial is deleted rather than left holding the
  disk full.
- A refusal at INIT is reported as a refusal. The sender took it for a
  receiver without streamed-transfer support, and the caller fell back to
  pushing the whole file in one frame.
- WriteFrame sends a frame of up to 64 KiB as one write, so the payload is
  not held behind the header by Nagle and the peer's delayed ACK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@TeoSlayer

Copy link
Copy Markdown
Contributor Author

Converted to draft after review; two things to fix before this merges (nothing ships until a stable tag, but both are regressions against main).

  1. Resume regression (filestream.go around lines 642-651). A retry that arrives while the stalled connection is still open on the receiver (idle timeout is 2 minutes) gets a private .partial and restarts at offset 0; main resumes mid-file in the same scenario. The stale .partial is then orphaned and counts against the quota, and with a quota set the retry is refused outright ("receiver disk quota exceeded") where main accepts it.
  2. Build break on some unix targets (filestream_disk_unix.go:3). //go:build unix no longer compiles on openbsd, netbsd, solaris and illumos; main does. linux, darwin, windows, freebsd and android are fine.

The rest reviewed well: wire bytes are unchanged, pilotctl already handles res.OK == false, and the new concurrency test fails on main and passes here. The handleChunk ENOSPC path is not covered by the PR's own tests (it does work against /dev/full).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant