Skip to content

Don't block Connection.close() on the final telemetry flush - #1798

Open
vuanhphung wants to merge 6 commits into
mainfrom
perf/close
Open

vuanhphung wants to merge 6 commits into
mainfrom
perf/close

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Description

When Connection.close() removes the last connection to a host, as connection-per-query apps do, it waits for the host's final telemetry push. JFR on a real warehouse (~24 ms RTT) shows that push opening a new TCP+TLS connection on the TELEMETRY client and POSTing to /telemetry-ext. That costs about 100–120 ms on top of the ~85 ms CloseSession RPC, so a sequential close took ~190 ms (Simba 2.7.3: ~80 ms). When /telemetry-ext answers 429, the push retries at ~1 s intervals, and every such close took 1.2–5 s, up to TelemetrySocketTimeout. A probe saw 429s in 17–83% of samples during the runs below. With 200 concurrent threads, close is only the CloseSession RPC.

This revisits #1772's choice to wait for the final flush in close(). The TelemetrySocketTimeout cap stays, but the wait moves off the caller's thread. close() still sends CloseSession. It now also updates every shared map synchronously (telemetry holder, auth configurator, feature-flag context, HTTP clients), so a new connection to the same host never observes the closing one. Only the final flush runs on a small executor, followed by closing the telemetry HTTP client and configurator that the flush pushes through. The executor has up to 32 non-daemon threads, and they exit as soon as they are idle. If all workers are busy, the work runs inline as before. TELEMETRY HTTP clients no longer hold the shared async client, which they never used.

What can now be dropped, and when: as before, a batch still pushing after TelemetrySocketTimeout. Because closes no longer wait, an endpoint that stays slower than the close rate now queues final flushes in the 10-thread push pool. Flushes queued past that timeout, or rejected by the telemetry circuit breaker, are dropped. Against a local endpoint taking 4.5 s per push, 10 of 60 final batches arrived; main delivered all of them but blocked every close for 4.5 s. Also new: the final batch is dropped when the JVM calls System.exit() or halts right after close(), or when close() runs inside a shutdown hook. A normal JVM exit still waits for a pending flush, bounded by the same timeout. With nothing pending it exits as before.

Two fixes in the same path. First, a host's shared telemetry client pushed through the HTTP client of the connection that created it. Once that connection closed, telemetry from the host's other open connections was silently skipped until they had all closed. The client now moves to another open connection, as the feature-flag context already does, and queued batches use the new binding. This stays in this PR because its race safety needs it: a connection that reopens to the host and joins the closing one's telemetry client must keep delivering. Second, TelemetryClient.close() re-created an empty collector for the already-closed connection, which leaked one per close (20 after 20 sequential closes on main). That fix is also in #1800, with an identical hunk, so it can land on its own. It stays here because the close-race test checks that no per-connection state is left behind.

Testing

Close latency on a real warehouse (~24 ms RTT), in ms as p50 / p95 / p99, pooled over interleaved fresh-JVM runs. Sequential means one thread doing 30 s of connect, optional query, then close. Most main and branch runs overlapped 429 throttling. Without throttling, main's sequential p50 was 185–198 ms.

Scenario main this PR Simba 2.7.3
Sequential, Thrift, open → close (4 runs) 1218 / 3338 / 5099 85 / 119 / 233 80 / 102 / 143
Sequential, SEA, open → close (4 runs) 315 / 3292 / 3355 96 / 122 / 167 –
Sequential, Thrift, open → SELECT 1 → close (5 runs) 206 / 2280 / 3339 82 / 97 / 127 –
Sequential, SEA, open → SELECT 1 → close (5 runs) 1237 / 3414 / 5192 90 / 112 / 155 –
200 threads × 2 min, Thrift (5 runs) 86 / 110 / 156 86 / 112 / 162 88 / 112 / 163
200 threads × 2 min, SEA (2 runs) 95 / 116 / 149 96 / 121 / 176 –

The SELECT 1 rows ran an earlier build with a 10-worker cap, which those runs never reached. Simba's SELECT 1 runs failed on JDK 17 (Arrow), so they are omitted. Two outliers did not reproduce in JFR-instrumented reruns: one 9.1 s sequential close on this branch, and one 200-thread run where ~100 connects and ~100 closes stalled together for ~22 s. In the reruns, the only multi-second waits were socket reads for server responses.

Unit tests cover a 50-iteration race where the last close overlaps a reopen to the same host. Half the iterations make the new connection join the closing connection's telemetry client first. The test uses the real factories and a local telemetry server, and asserts that both connections' events arrive, that the new connection's flags, telemetry and HTTP clients stay live, and that nothing is left after both connections close. Further tests cover push-before-close ordering, a close issued from a close worker, the saturation fallback, and idle workers exiting. Each of these fails on a mutant without the rebind, without the deferred telemetry-client close, or with the work always inline. The full jdbc-core suite has one failure, ArrowBufferAllocatorTest.testCreateDatabricksBufferAllocator, which also fails on main. A dry-run merge with #1783 is clean, and the race and feature-flag tests pass on the merge.

Exit behavior, from a process-level check with a local telemetry server: close() takes ~4 ms. The JVM exits 325 ms after main returns with nothing pending (same on main), 567 ms with a 200 ms pending flush, and 2.3 s with an 8 s endpoint and TelemetrySocketTimeout=2. System.exit() after close(), and close() inside a shutdown hook, return within ~330 ms and drop the batch. On the real warehouse, Thrift and SEA delivered 3 of 3 final batches with no errors.

Telemetry Errors

  • Not applicable — this PR does not add or change a telemetry-visible error.
  • Applicable — the error uses DatabricksDriverErrorCode where appropriate, and any
    new code is uniquely numbered and tested.
  • Applicable — its driver/server/user classification is linked, or maintainer help is
    requested because the author cannot access the classification.

This pull request and its description were written by Isaac.

When close() removed the last connection to a host, it waited for the
host's final telemetry push: a new TCP+TLS connection plus the POST,
about 100 ms on top of CloseSession, and up to TelemetrySocketTimeout
when /telemetry-ext throttles. close() now updates every shared map
synchronously (telemetry holder, configurator, feature-flag context,
HTTP clients) and hands only the bounded final flush, followed by
closing the telemetry HTTP client and configurator it uses, to a small
non-daemon executor. It runs inline when that executor is saturated.

Also rebind a host's shared telemetry client when the connection it
pushes through closes while others stay open (its pushes were silently
skipped until the host drained), and stop re-creating a telemetry
collector for an already-closed connection on every final flush.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 2 Low

The change looks sound overall, with 2 low-severity concerns. Moving the final flush off the caller's thread keeps the ordering correct: shared maps are updated synchronously, the telemetry HTTP client and configurator are closed only after the flush, and work falls back to running inline when all workers are busy. The rebind fix only covers batches flushed after the rebind, and the background error log drops the stack trace.

A batch that flush() queued before a rebind kept the old push client,
which still resolved the closing connection's (now closed) telemetry
HTTP client, so the batch was skipped. Resolve the push client when the
task runs. Also log background close failures with their stack trace;
inline, the same failure propagates from close().

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good overall: one low-severity concern. Moving the final flush off the caller's thread is carefully done. Shared maps are updated synchronously, the TELEMETRY HTTP client and configurator close after the flush, and the fallback when workers are busy runs the work inline. The remaining concern is a narrow race where a push already in flight when the bound connection closes can still be dropped silently.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 2 Low · 1 Nit

Looks good overall. Moving the final flush off the caller's thread is done carefully: every shared map is updated before close() returns, and the TELEMETRY HTTP client and configurator close only after the flush. I have three low-severity concerns, about the rebind fallback, exceptions thrown from inside the compute lambda, and a test that can flake because it depends on shared static state.

Other findings

  • ⚪ Nit — testCloseFinishesInlineWhenBackgroundClosesAreSaturated assumes the JVM-wide static CLOSE_EXECUTOR has no busy workers before it starts. If another test leaves a worker running, for example DatabricksConnectionCloseTest (it waits up to 10 s per flush) or any parallel test execution, fewer than 32 of this test's submissions run in the background. backgroundStarted.await then times out and the test flakes. Consider waiting for backgroundCloseThreadCount() == 0 at the start of the test.

Catch failures building the replacement push client inside the holder
update, so they cannot skip the unauthenticated detach or fail close().
Also start the saturation test only once the JVM-wide close executor is
idle.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung

Copy link
Copy Markdown
Collaborator Author

Nit on testCloseFinishesInlineWhenBackgroundClosesAreSaturated (review body): fixed in 22f69c8; the test now waits for backgroundCloseThreadCount() == 0 before it starts.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low · 1 Nit

Looks good: two low-severity issues. Moving the final flush to the background is well structured. Every shared map is updated synchronously, the TELEMETRY HTTP client and configurator closes are deferred, and close falls back to running inline when all workers are busy. Rebinding under computeIfPresent uses only non-creating lookups (getConfiguratorOnlyIfExists) and the per-host circuit breaker, so it doesn't bring back the #1442 stall.

Other findings

  • ⚪ Nit — CLOSE_EXECUTOR is shared across the whole JVM, but awaitNoBackgroundCloseThreads() here waits only 2 s and then asserts the pool size is exactly 0. DatabricksConnectionCloseTest uses its own copy with a 10 s timeout. Two things can make these asserts flaky:
  • On a slow CI box, background flushes started by other tests can still be draining.
  • If surefire runs test classes in parallel, another class can be using the executor at the same time.

The saturation test is the most exposed: it assumes it can fill all 32 slots itself. Consider a longer, shared timeout, or a test-only hook that injects or resets the executor, so the tests don't depend on global pool state.

@vuanhphung

Copy link
Copy Markdown
Collaborator Author

Nit on the shared CLOSE_EXECUTOR in tests (review body): leaving as is. This repo's surefire config has no parallel execution (no parallel, forkCount, or threadCount), so test classes run one at a time. Only DatabricksConnectionTest and DatabricksConnectionCloseTest create background closes, and every test that does waits for the pool to drain before it ends. Workers have zero keep-alive and exit as soon as they are idle, so the 2 s wait only has to cover tasks in DatabricksConnectionTest, which finish in milliseconds.

Resolve the main changes by merging (no force-push).

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Nit

Looks good, with no blocking issues. I traced the close ordering: the telemetry holder is detached and rebound before HTTP clients and configurators are released, and the TELEMETRY client and configurator are closed only after the deferred flush. The rebind and lazy push-client read look correct. A minor follow-up: with TelemetrySocketTimeout=0, a hung final flush now keeps a non-daemon close worker alive at JVM exit; before this change it blocked the caller's thread instead.

Other findings

  • ⚪ Nit — DatabricksConnectionCloseTest.setUp gives the HttpServer a newCachedThreadPool() executor, and tearDown never shuts it down (server.stop(0) leaves the executor running). The idle non-daemon threads then stay alive in the surefire fork for up to 60 s. Keep a reference to the executor and call shutdownNow() in tearDown.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
@vuanhphung

Copy link
Copy Markdown
Collaborator Author

Review-body items on 72c5cab9:

  • Nit on the HttpServer executor in DatabricksConnectionCloseTest: fixed in 4a0bc74. tearDown now calls shutdownNow() on it.
  • TelemetrySocketTimeout=0 with a hung final flush: leaving as is. On main, that same flush blocks the caller's close() forever. Here close() returns, and only JVM exit waits, under the same timeout, as the description says. A daemon worker would drop the final batch at a normal exit, which is the delivery trade-off this PR deliberately avoids.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant