Skip to content

Disable OPcache JIT by default (fixes php-fpm workers spinning at 100% CPU) - #545

Open
mauro2306 wants to merge 2 commits into
linuxserver:masterfrom
mauro2306:disable-opcache-jit
Open

Disable OPcache JIT by default (fixes php-fpm workers spinning at 100% CPU)#545
mauro2306 wants to merge 2 commits into
linuxserver:masterfrom
mauro2306:disable-opcache-jit

Conversation

@mauro2306

@mauro2306 mauro2306 commented Sep 1, 2026

Copy link
Copy Markdown
  • I have read the contributing guideline and understand that I have made the correct modifications

Description:

Sets opcache.jit=disable and opcache.jit_buffer_size=0 in 00_opcache.ini, instead of
opcache.jit=1255 / opcache.jit_buffer_size=128M. Same two lines in Dockerfile and
Dockerfile.aarch64, plus a changelog entry in readme-vars.yml.

Benefits of this PR and context:

Closes #539 (and very likely #536, which is the same failure with a different trigger).

Under sustained traffic the tracing JIT can emit machine code that never makes forward progress.
I caught an instance in the failed state and attached gdb to the stuck workers before restarting it:
the PHP VM instruction pointer (EG.current_execute_data->opline) is frozen at the same address in
every worker while their CPU time keeps climbing, and the CPU instruction pointer is inside the
shared opcache JIT buffer. The PHP call chain the workers are frozen in contains no loop at all, so
the emitted machine code isn't doing what the PHP says. Full write-up with the captures is
here.

Because the JIT buffer is shared memory, once bad code is emitted every worker reaching that path
hangs, including workers forked afterwards. That's why only a full restart clears it, and why it
comes back at random days later.

Worth noting what we currently ship: 1255 is not a documented preset. The named tracing mode is
1254; 1255 is the same tracing JIT at optimisation level 5 ("optimise whole script"). PHP itself
ships with the JIT off (on 8.4 that is opcache.jit=disable), and Nextcloud's own server tuning
docs recommend tracing with an 8M buffer, noting most instances use under 2 MiB. We allocate
128M. Nextcloud is dominated by database I/O rather than computation, so the JIT buys very little
here. OPcache itself is untouched and keeps doing the useful work.

How Has This Been Tested?

PRs no longer auto-build, so I could not test a built image. Instead I ran the equivalent override
at runtime (/config/php/php-local.ini, read after conf.d, so it wins) across the four instances
I administer. All four run 34.0.3-ls448 and Nextcloud 34.0.3, so the JIT setting is the only
variable between them.

Three patched on 16 Aug, one left unpatched by accident:

3 patched 1 unpatched
opcache.jit disable 1255
executable JIT mapping in a worker absent present
upstream timed out in nginx 0 / 0 / 0 5182
pm.max_children 30 / 60 / 120 20

The mapping check is the objective part, from /proc/<worker>/maps. Unpatched:

557157800000-55715f800000 rw-s 00000000 ... /dev/zero (deleted)   <- opcache data, 128M
55715f800000-557167800000 r-xs 08000000 ... /dev/zero (deleted)   <- JIT code, 128M, shared+executable

Patched: the r-xs line is simply gone, only the data segments remain. So the failure mode is
removed structurally, not just reset the way a plain restart does.

On the patched instance that used to fail most often, php-fpm has restarted 8 times since 16 Aug,
including the weekly logrotate restarts on 23 and 30 Aug (s6-svc -t in
/etc/logrotate.d/php-fpm, a full pool restart) and the image update to ls448 on 2 Sep. Those
restarts are exactly what used to set the failure off, and there has been no recurrence. Before the
change it was roughly one incident every two or three restarts. The unpatched one failed again on
26 Aug and 2 Sep, the latter about seven hours after Watchtower restarted its pool.

Two other users in #539 report the same result independently after applying the same override.

I set both values explicitly rather than deleting the two lines, because it is the combination that
has been running for three weeks, and because the PHP default has already moved once (8.3 → 8.4
changed from jit_buffer_size=0 to opcache.jit=disable), so an explicit value will not silently
change meaning again.

Source / References:

Strictly the miscompilation is a PHP bug and the real fix belongs upstream, but nobody in #539 has
managed to reproduce it on demand (it needs the JIT to decide the path is hot, which takes hours of
real traffic). In the meantime this stops opting users into a non-default, undocumented JIT level.

The tracing JIT this image enables (opcache.jit=1255, 128M buffer) can emit
machine code that loops forever, wedging every php-fpm worker at 100% CPU
until the container is restarted. See linuxserver#539.

PHP ships with the JIT off by default; Nextcloud's own tuning docs recommend
tracing with an 8M buffer. 1255 is not a documented preset (tracing is 1254).

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for opening this pull request! Be sure to follow the pull request template!

Added changelog entry for disabling OPcache JIT due to performance issues.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

[BUG] High CPU usage at random moments

2 participants