Conversation
k-morozov
self-requested a review
September 16, 2026 12:49
Port of the budget selection from upstream (1fd4b6e, 7918dfd, 5e152c3), reduced to the part that chooses the per-PID timeout: - `LLDB_TIMEOUT` (30 s) for release, `LLDB_SLOW_BUILD_TIMEOUT` (120 s) for debug, sanitizer and coverage servers, read from `args.build_flags`, or from the binary via `clickhouse local` (`is_slow_build_binary`) when the server never started and the flags were not collected. - `print_c_stacktraces` takes `per_pid_timeout` and `total_timeout`, clamps each attach to what is left of the aggregate ceiling and names the pids it skipped. The per-test timeout handler, which runs inside a fired one-shot alarm, keeps the tight pair (30 s per pid, 60 s total); the abort paths take the flavor budget. Not ported: the stacktrace log files, the truncation marker and `keep_output_on_error` (a separate upstream change), and the removal of the on-demand lldb install, which the fork's images still need. Tests mirror upstream's, run without a server. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
… JIT breakpoint out of the server `get_stacktraces_from_lldb` killed lldb on the budget and assumed the server was untouched. It is not, though not for the reason first suspected. On Linux the ptrace tracer is `lldb-server`, launched in its own session; it exits once its client is gone, the kernel restarts the stopped threads by itself, and neither lldb nor `lldb-server` removes the breakpoints lldb planted in the inferior on the way out. The JIT loader plants one on `__jit_debug_register_code` at attach; in the clickhouse binary the linker folds that empty function with other trivial ones, so ordinary server threads run through it. Each of them is delivered SIGTRAP, which the server treats as fatal: the handler waits 300 s for a stack dump that is never written because `SignalListener` and `AsyncLogger` are trapped as well, the rest of the server piles up behind them, and the process shows state S while ignoring SIGTERM and every client. Reproduced with a release binary in the stateless-test image (lldb 14): lldb killed 5 s into `thread backtrace all` leaves nine threads in `signalHandler(SIGTRAP)` at that address and the server unresponsive; killed with the JIT loader disabled, or interrupted with SIGINT so it runs the queued `detach`, the server answers right afterwards. This is the MSan CAS-S3 shard freeze (six attaches on the budget, then `system flush logs` and `clickhouse stop` timing out); the watchdog's `stopped by signal 19` line is the attach itself, not a lasting group-stop, and SIGCONT would not have helped. - `_lldb_command` turns `plugin.jit-loader.gdb.enable` off and attaches with an explicit `process attach`, so a killed lldb leaves no breakpoint in server code. The dynamic-loader breakpoint in `ld.so` remains; it is only reached through `dlopen`. - On the budget the collector sends SIGINT and gives lldb `LLDB_INTERRUPT_GRACE` (15 s) to abandon the backtrace, detach and quit, keeping the partial output. Only then is it killed. - No process-group or SIGCONT handling. Tests: the command order, SIGINT with the partial output kept, SIGKILL after the grace period. The two live-server tests still exercise the real command. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
force-pushed
the
fix/antalya-26.6/lldb-budget-and-sigcont
branch
from
September 17, 2026 16:10
d29211a to
3885c5c
Compare
filimonov
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes to
tests/clickhouse-test's lldb stack collection, each its own commit:1fd4b6eea445,7918dfd548d4,5e152c370237, reduced to the timeout selection): 30 s per pid for release, 120 s for debug/sanitizer/coverage, read fromargs.build_flagsor from the binary viaclickhouse localwhen the server never started;print_c_stacktracesgets an aggregate deadline and names the pids it skips; the per-test timeout handler keeps the tight 30 s / 60 s pair. Not ported: the stacktrace log files and truncation marker, and the removal of the on-demand lldb install (the fork's images still need it).lldb-server, which exits on its own once the client is gone, the kernel restarts the threads, and nobody removes the breakpoints lldb planted. Its JIT loader puts one on__jit_debug_register_code, which the linker folds with other trivial functions in the clickhouse binary, so ordinary threads hit the staleint3, getSIGTRAP, and land in the fatal-signal handler;SignalListenerandAsyncLoggerare among them, so the 300 s wait for the stack dump never ends and the server stays in stateSwhile ignoringSIGTERMand every client. Reproduced with a release binary and lldb 14 in the stateless-test image (nine threads insignalHandler(SIGTRAP)at that address); with the JIT loader disabled, or with lldb interrupted bySIGINTso it runs the queueddetach, the server answers right afterwards. Now_lldb_commandturnsplugin.jit-loader.gdb.enableoff and attaches with an explicitprocess attach; on the budget the collector sendsSIGINT, gives lldb 15 s to abandon the backtrace, detach and quit (keeping the partial output), and only then kills it. An earlier revision of this PR ran lldb in its own process group and sentSIGCONTto the server; that never reachedlldb-serverand would not have helped, so it is gone. Intended to be proposed upstream as well.Tests in
ci/tests/test_print_stacktraces.py: six ported budget tests, plus the command order,SIGINTwith the partial output kept, andSIGKILLafter the grace period. The nine server-independent tests pass locally; the real command was exercised end to end against a live server in the stateless-test image with lldb 14 on theSIGINT,SIGINT-during-attach,SIGKILL-after-grace andSIGKILL-mid-backtrace paths, with the server healthy after each.Related: #2298
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
CI: the test runner gives lldb 120 s instead of 30 s to collect stacks from debug and sanitizer servers, and a timed-out lldb no longer leaves the server wedged: it is interrupted so it detaches, and its JIT breakpoint is kept out of the server.
Documentation entry for user-facing changes
CI/CD Options
Exclude tests:
Regression jobs to run:
🤖 Generated with Claude Code
https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx