Repository navigation
Don't block Connection.close() on the final telemetry flush - #1798
vuanhphung wants to merge 6 commits into
Conversation
When close() removed the last connection to a host, it waited for the host's final telemetry push: a new TCP+TLS connection plus the POST, about 100 ms on top of CloseSession, and up to TelemetrySocketTimeout when /telemetry-ext throttles. close() now updates every shared map synchronously (telemetry holder, configurator, feature-flag context, HTTP clients) and hands only the bounded final flush, followed by closing the telemetry HTTP client and configurator it uses, to a small non-daemon executor. It runs inline when that executor is saturated. Also rebind a host's shared telemetry client when the connection it pushes through closes while others stay open (its pushes were silently skipped until the host drained), and stop re-creating a telemetry collector for an already-closed connection on every final flush. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
Verdict: 2 Low
The change looks sound overall, with 2 low-severity concerns. Moving the final flush off the caller's thread keeps the ordering correct: shared maps are updated synchronously, the telemetry HTTP client and configurator are closed only after the flush, and work falls back to running inline when all workers are busy. The rebind fix only covers batches flushed after the rebind, and the background error log drops the stack trace.
A batch that flush() queued before a rebind kept the old push client, which still resolved the closing connection's (now closed) telemetry HTTP client, so the batch was skipped. Resolve the push client when the task runs. Also log background close failures with their stack trace; inline, the same failure propagates from close(). Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Low
Looks good overall: one low-severity concern. Moving the final flush off the caller's thread is carefully done. Shared maps are updated synchronously, the TELEMETRY HTTP client and configurator close after the flush, and the fallback when workers are busy runs the work inline. The remaining concern is a narrow race where a push already in flight when the bound connection closes can still be dropped silently.
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
Verdict: 2 Low · 1 Nit
Looks good overall. Moving the final flush off the caller's thread is done carefully: every shared map is updated before close() returns, and the TELEMETRY HTTP client and configurator close only after the flush. I have three low-severity concerns, about the rebind fallback, exceptions thrown from inside the compute lambda, and a test that can flake because it depends on shared static state.
Other findings
- ⚪ Nit —
testCloseFinishesInlineWhenBackgroundClosesAreSaturatedassumes the JVM-wide staticCLOSE_EXECUTORhas no busy workers before it starts. If another test leaves a worker running, for exampleDatabricksConnectionCloseTest(it waits up to 10 s per flush) or any parallel test execution, fewer than 32 of this test's submissions run in the background.backgroundStarted.awaitthen times out and the test flakes. Consider waiting forbackgroundCloseThreadCount() == 0at the start of the test.
Catch failures building the replacement push client inside the holder update, so they cannot skip the unauthenticated detach or fail close(). Also start the saturation test only once the JVM-wide close executor is idle. Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
|
Nit on |
There was a problem hiding this comment.
Verdict: 1 Low · 1 Nit
Looks good: two low-severity issues. Moving the final flush to the background is well structured. Every shared map is updated synchronously, the TELEMETRY HTTP client and configurator closes are deferred, and close falls back to running inline when all workers are busy. Rebinding under computeIfPresent uses only non-creating lookups (getConfiguratorOnlyIfExists) and the per-host circuit breaker, so it doesn't bring back the #1442 stall.
Other findings
- ⚪ Nit —
CLOSE_EXECUTORis shared across the whole JVM, butawaitNoBackgroundCloseThreads()here waits only 2 s and then asserts the pool size is exactly 0.DatabricksConnectionCloseTestuses its own copy with a 10 s timeout. Two things can make these asserts flaky: - On a slow CI box, background flushes started by other tests can still be draining.
- If surefire runs test classes in parallel, another class can be using the executor at the same time.
The saturation test is the most exposed: it assumes it can fill all 32 slots itself. Consider a longer, shared timeout, or a test-only hook that injects or resets the executor, so the tests don't depend on global pool state.
|
Nit on the shared |
Resolve the main changes by merging (no force-push). Signed-off-by: Vu Anh Phung <vu.phung@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Nit
Looks good, with no blocking issues. I traced the close ordering: the telemetry holder is detached and rebound before HTTP clients and configurators are released, and the TELEMETRY client and configurator are closed only after the deferred flush. The rebind and lazy push-client read look correct. A minor follow-up: with TelemetrySocketTimeout=0, a hung final flush now keeps a non-daemon close worker alive at JVM exit; before this change it blocked the caller's thread instead.
Other findings
- ⚪ Nit —
DatabricksConnectionCloseTest.setUpgives theHttpServeranewCachedThreadPool()executor, andtearDownnever shuts it down (server.stop(0)leaves the executor running). The idle non-daemon threads then stay alive in the surefire fork for up to 60 s. Keep a reference to the executor and callshutdownNow()intearDown.
Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
|
Review-body items on
|
Description
When
Connection.close()removes the last connection to a host, as connection-per-query apps do, it waits for the host's final telemetry push. JFR on a real warehouse (~24 ms RTT) shows that push opening a new TCP+TLS connection on the TELEMETRY client and POSTing to/telemetry-ext. That costs about 100–120 ms on top of the ~85 ms CloseSession RPC, so a sequential close took ~190 ms (Simba 2.7.3: ~80 ms). When/telemetry-extanswers 429, the push retries at ~1 s intervals, and every such close took 1.2–5 s, up toTelemetrySocketTimeout. A probe saw 429s in 17–83% of samples during the runs below. With 200 concurrent threads, close is only the CloseSession RPC.This revisits #1772's choice to wait for the final flush in
close(). TheTelemetrySocketTimeoutcap stays, but the wait moves off the caller's thread.close()still sends CloseSession. It now also updates every shared map synchronously (telemetry holder, auth configurator, feature-flag context, HTTP clients), so a new connection to the same host never observes the closing one. Only the final flush runs on a small executor, followed by closing the telemetry HTTP client and configurator that the flush pushes through. The executor has up to 32 non-daemon threads, and they exit as soon as they are idle. If all workers are busy, the work runs inline as before. TELEMETRY HTTP clients no longer hold the shared async client, which they never used.What can now be dropped, and when: as before, a batch still pushing after
TelemetrySocketTimeout. Because closes no longer wait, an endpoint that stays slower than the close rate now queues final flushes in the 10-thread push pool. Flushes queued past that timeout, or rejected by the telemetry circuit breaker, are dropped. Against a local endpoint taking 4.5 s per push, 10 of 60 final batches arrived;maindelivered all of them but blocked every close for 4.5 s. Also new: the final batch is dropped when the JVM callsSystem.exit()or halts right afterclose(), or whenclose()runs inside a shutdown hook. A normal JVM exit still waits for a pending flush, bounded by the same timeout. With nothing pending it exits as before.Two fixes in the same path. First, a host's shared telemetry client pushed through the HTTP client of the connection that created it. Once that connection closed, telemetry from the host's other open connections was silently skipped until they had all closed. The client now moves to another open connection, as the feature-flag context already does, and queued batches use the new binding. This stays in this PR because its race safety needs it: a connection that reopens to the host and joins the closing one's telemetry client must keep delivering. Second,
TelemetryClient.close()re-created an empty collector for the already-closed connection, which leaked one per close (20 after 20 sequential closes on main). That fix is also in #1800, with an identical hunk, so it can land on its own. It stays here because the close-race test checks that no per-connection state is left behind.Testing
Close latency on a real warehouse (~24 ms RTT), in ms as p50 / p95 / p99, pooled over interleaved fresh-JVM runs. Sequential means one thread doing 30 s of connect, optional query, then close. Most main and branch runs overlapped 429 throttling. Without throttling, main's sequential p50 was 185–198 ms.
SELECT 1→ close (5 runs)SELECT 1→ close (5 runs)The
SELECT 1rows ran an earlier build with a 10-worker cap, which those runs never reached. Simba'sSELECT 1runs failed on JDK 17 (Arrow), so they are omitted. Two outliers did not reproduce in JFR-instrumented reruns: one 9.1 s sequential close on this branch, and one 200-thread run where ~100 connects and ~100 closes stalled together for ~22 s. In the reruns, the only multi-second waits were socket reads for server responses.Unit tests cover a 50-iteration race where the last close overlaps a reopen to the same host. Half the iterations make the new connection join the closing connection's telemetry client first. The test uses the real factories and a local telemetry server, and asserts that both connections' events arrive, that the new connection's flags, telemetry and HTTP clients stay live, and that nothing is left after both connections close. Further tests cover push-before-close ordering, a close issued from a close worker, the saturation fallback, and idle workers exiting. Each of these fails on a mutant without the rebind, without the deferred telemetry-client close, or with the work always inline. The full
jdbc-coresuite has one failure,ArrowBufferAllocatorTest.testCreateDatabricksBufferAllocator, which also fails onmain. A dry-run merge with #1783 is clean, and the race and feature-flag tests pass on the merge.Exit behavior, from a process-level check with a local telemetry server:
close()takes ~4 ms. The JVM exits 325 ms aftermainreturns with nothing pending (same onmain), 567 ms with a 200 ms pending flush, and 2.3 s with an 8 s endpoint andTelemetrySocketTimeout=2.System.exit()afterclose(), andclose()inside a shutdown hook, return within ~330 ms and drop the batch. On the real warehouse, Thrift and SEA delivered 3 of 3 final batches with no errors.Telemetry Errors
DatabricksDriverErrorCodewhere appropriate, and anynew code is uniquely numbered and tested.
requested because the author cannot access the classification.
This pull request and its description were written by Isaac.