Install a 4 KiB-page jemalloc on every architecture so arm64 TaskManagers stop getting OOMKilled

[velo]

Problem

Since prod moved to arm64 (Graviton r6gd) on 2026-10-14, the metrics2 TaskManagers get OOMKilled (exit 137) every 1 to 4.5 hours. On amd64 m5ad, the same job ran for days without a kill.

The cause is jemalloc. The Flink base image installs Ubuntu's libjemalloc2, and our entrypoint preloads it. On arm64 that package is compiled for 64 KiB pages so one binary also runs on 64K-page kernels. Our nodes run 4 KiB-page kernels:

	amd64 (m5ad.2xlarge)	arm64 (r6gd.xlarge)
jemalloc 5.3.0 Page size	4096	65536
kernel getconf PAGESIZE	4096	4096

With 64 KiB pages, every jemalloc slab and extent is rounded to 64 KiB, and dirty memory is purged in 64 KiB units. A TaskManager with ~290 RocksDB instances and ~3.5k threads pays for that in resident memory that Flink's memory model doesn't account for.

Same job, image and config, measured from /proc/1/smaps: the malloc'd region was ~3.3 GiB on ARM against 2.5–2.8 GiB on x86 during its first 40 minutes. That's roughly 500–800 MiB more on ARM, against a JVM overhead budget of 799 MiB. Thread stacks were identical (~380 MiB on both), so the 2 MiB default stack size on aarch64 doesn't matter.

Mate found the same 64 KiB page size independently.

Change

• The Dockerfile has a new jemalloc-4k builder stage. It builds jemalloc 5.3.0 with --with-lg-page=12 and verifies the tarball's SHA-256.
• The final stage installs that library over /usr/lib/<arch>-linux-gnu/libjemalloc.so.2 on every architecture. The Flink operator starts pods with the upstream /docker-entrypoint.sh, not our /entrypoint.sh, so choosing a library in a script would be skipped in prod. Replacing the file both entrypoints preload makes the choice independent of the entrypoint, and amd64 and arm64 run the same allocator.
• The build fails unless the installed library reports Version: "5.3.0-…" and Page size: 4096 when preloaded.
• adr/fix-jemalloc-arm64-page-size.md documents the ARM issue, the evidence, the decisions, and how to verify a running TaskManager.

DISABLE_JEMALLOC=true still works as an escape hatch, and it's required on 64K-page kernels, where a 4 KiB-page jemalloc won't start.

Testing

• amd64, built locally: the final stage's page-size check passed, and the upstream /docker-entrypoint.sh preloads the replaced 4 KiB library (confirmed in /proc/self/maps).
• arm64: the builder stage compiles under QEMU. The final-stage page-size check runs in CI's multi-arch build.
• mvn test for the flink-sql-runner module passes.
• Not yet verified in prod. Next step is to run the PR image on the retired metrics2 deployment 32c6a0744 on arm64 and compare its memory with the current image.

[velo]

Not proceeding: we'll handle the extra native memory in cloud-compilation sizing (drop Flink's JVM share from 90% to 80% when the CPU limit is more than 2x the request) instead of shipping a custom jemalloc.