Disable OPcache JIT by default (fixes php-fpm workers spinning at 100% CPU) - #545
Open
mauro2306 wants to merge 2 commits into
Open
Disable OPcache JIT by default (fixes php-fpm workers spinning at 100% CPU)#545mauro2306 wants to merge 2 commits into
mauro2306 wants to merge 2 commits into
Conversation
The tracing JIT this image enables (opcache.jit=1255, 128M buffer) can emit machine code that loops forever, wedging every php-fpm worker at 100% CPU until the container is restarted. See linuxserver#539. PHP ships with the JIT off by default; Nextcloud's own tuning docs recommend tracing with an 8M buffer. 1255 is not a documented preset (tracing is 1254).
There was a problem hiding this comment.
Thanks for opening this pull request! Be sure to follow the pull request template!
1 task
Added changelog entry for disabling OPcache JIT due to performance issues.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Sets
opcache.jit=disableandopcache.jit_buffer_size=0in00_opcache.ini, instead ofopcache.jit=1255/opcache.jit_buffer_size=128M. Same two lines inDockerfileandDockerfile.aarch64, plus a changelog entry inreadme-vars.yml.Benefits of this PR and context:
Closes #539 (and very likely #536, which is the same failure with a different trigger).
Under sustained traffic the tracing JIT can emit machine code that never makes forward progress.
I caught an instance in the failed state and attached gdb to the stuck workers before restarting it:
the PHP VM instruction pointer (
EG.current_execute_data->opline) is frozen at the same address inevery worker while their CPU time keeps climbing, and the CPU instruction pointer is inside the
shared opcache JIT buffer. The PHP call chain the workers are frozen in contains no loop at all, so
the emitted machine code isn't doing what the PHP says. Full write-up with the captures is
here.
Because the JIT buffer is shared memory, once bad code is emitted every worker reaching that path
hangs, including workers forked afterwards. That's why only a full restart clears it, and why it
comes back at random days later.
Worth noting what we currently ship:
1255is not a documented preset. The namedtracingmode is1254;1255is the same tracing JIT at optimisation level 5 ("optimise whole script"). PHP itselfships with the JIT off (on 8.4 that is
opcache.jit=disable), and Nextcloud's own server tuningdocs recommend
tracingwith an 8M buffer, noting most instances use under 2 MiB. We allocate128M. Nextcloud is dominated by database I/O rather than computation, so the JIT buys very little
here. OPcache itself is untouched and keeps doing the useful work.
How Has This Been Tested?
PRs no longer auto-build, so I could not test a built image. Instead I ran the equivalent override
at runtime (
/config/php/php-local.ini, read afterconf.d, so it wins) across the four instancesI administer. All four run
34.0.3-ls448and Nextcloud 34.0.3, so the JIT setting is the onlyvariable between them.
Three patched on 16 Aug, one left unpatched by accident:
opcache.jitdisable1255upstream timed outin nginxpm.max_childrenThe mapping check is the objective part, from
/proc/<worker>/maps. Unpatched:Patched: the
r-xsline is simply gone, only the data segments remain. So the failure mode isremoved structurally, not just reset the way a plain restart does.
On the patched instance that used to fail most often, php-fpm has restarted 8 times since 16 Aug,
including the weekly logrotate restarts on 23 and 30 Aug (
s6-svc -tin/etc/logrotate.d/php-fpm, a full pool restart) and the image update tols448on 2 Sep. Thoserestarts are exactly what used to set the failure off, and there has been no recurrence. Before the
change it was roughly one incident every two or three restarts. The unpatched one failed again on
26 Aug and 2 Sep, the latter about seven hours after Watchtower restarted its pool.
Two other users in #539 report the same result independently after applying the same override.
I set both values explicitly rather than deleting the two lines, because it is the combination that
has been running for three weeks, and because the PHP default has already moved once (8.3 → 8.4
changed from
jit_buffer_size=0toopcache.jit=disable), so an explicit value will not silentlychange meaning again.
Source / References:
this comment)
request_terminate_timeout. Complementary and worth merging too, but it is a seatbeltrather than a fix: it stops one runaway worker from taking the whole pool down, it does not
prevent the miscompilation.
for the
CRTOvalues and the named presetsfor the recommended JIT settings
Strictly the miscompilation is a PHP bug and the real fix belongs upstream, but nobody in #539 has
managed to reproduce it on demand (it needs the JIT to decide the path is hot, which takes hours of
real traffic). In the meantime this stops opting users into a non-default, undocumented JIT level.