Skip to content

libcontainer: count fds with readlinkat into a reused buffer - #3956

Open
lollllcat wants to merge 1 commit into
google:masterfrom
lollllcat:fd-readlinkat
Open

lollllcat wants to merge 1 commit into
google:masterfrom
lollllcat:fd-readlinkat

Conversation

@lollllcat

@lollllcat lollllcat commented Oct 26, 2026 •

Copy link
Copy Markdown

What

processStatsFromProcs walks /proc/<pid>/fd for every pid in the cgroup on every housekeeping tick and resolves each link to count sockets. Per fd, the current loop pays for:

  • os.ReadDir, which sorts the entries and allocates a DirEntry per fd;
  • path.Join(dirPath, fd.Name()), which runs path.Clean on every call;
  • os.Readlink, which allocates a heap string per link target.

On hosts exporting a few hundred containers with socket-heavy processes this function dominated the exporter's CPU profile, and most of the cost was this bookkeeping rather than the getdents64 / readlinkat syscalls themselves.

This PR reads the directory with Readdirnames(-1) (unsorted, names only) and resolves links with unix.Readlinkat into one reused 64-byte buffer, comparing the "socket" prefix in place. Counts, error handling and log lines are unchanged.

Testing

  • TestProcessStatsFromProcsSynthetic builds a synthetic /proc tree and asserts process / fd / socket counts.
  • BenchmarkProcessStatsFromProcs against the same tree (identical syscall sequence to a real host, minus procfs kernel cost). master vs this branch on the same machine:
shape ns/op B/op allocs/op
1 pid / 128 fds 299 µs → 146 µs 65 KB → 7 KB 911 → 272
8 pids / 512 fds 10.6 ms → 4.9 ms 2.1 MB → 185 KB 28.7k → 8.2k
32 pids / 2048 fds 187 ms → 99 ms 36 MB → 3.9 MB 459k → 131k

On real hosts the syscalls are a larger share than on tmpfs, so the wall-clock win is smaller than the benchmark's ~50%, but the ~90% drop in allocations also removes GC pressure.

go test -race ./lib/container/libcontainer/... passes.

@google-cla

google-cla Bot commented Oct 26, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

processStatsFromProcs walks /proc/<pid>/fd for every pid in the cgroup on
every housekeeping tick and resolves each link to count sockets. Per fd it
paid for os.ReadDir (sorts entries, allocates a DirEntry each), path.Join
(path.Clean per call) and os.Readlink (a heap string per link target). On a
host exporting a few hundred containers this function dominated the
exporter's CPU profile, and most of the cost was this bookkeeping rather
than the getdents64/readlinkat syscalls themselves.

Read the directory with Readdirnames(-1) (unsorted, names only) and resolve
links with unix.Readlinkat into one reused 64-byte buffer, comparing the
"socket" prefix in place. Counts, error handling and log lines are
unchanged; a synthetic /proc tree test asserts process/fd/socket counts.

BenchmarkProcessStatsFromProcs on a synthetic tree (same syscall sequence
as a real host, minus procfs kernel cost), master vs this change:

  shape                 ns/op            B/op              allocs/op
  pids=1  / fds=128     299 us -> 146 us  65 KB -> 7 KB     911 -> 272
  pids=8  / fds=512     10.6 ms -> 4.9 ms 2.1 MB -> 185 KB  28.7k -> 8.2k
  pids=32 / fds=2048    187 ms -> 99 ms   36 MB -> 3.9 MB   459k -> 131k
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants