fix(ENGHLP-1703): keep TLS when DIODE_SKIP_TLS_VERIFY is set

[jajeffries]

Summary

• Make DIODE_SKIP_TLS_VERIFY / skip_tls_verify behave like Go's InsecureSkipVerify: the connection stays on TLS and only certificate validation is skipped.
• Fixes deployments behind Traefik or other proxies that present self-signed certificates, where the previous behaviour downgraded to plaintext (h2c) and surfaced as 404 or protocol errors.
• Replaces the earlier pin-and-rename approach. grpcio has no way to turn verification off, so pinning the server leaf with grpc.ssl_target_name_override could not match Go (see below).

What changed

• New netboxlabs/diode/sdk/_skip_verify_tunnel.py. For a secure target with verification off, gRPC speaks plaintext to a private local listener (a Unix socket in a 0700 directory, or loopback where Unix sockets are unavailable). The tunnel forwards each connection to the server over TLS with CERT_NONE, minimum TLS 1.2 and ALPN h2. The server sees the real host as SNI and :authority, and nothing is pinned.
• HTTP CONNECT proxies are handled by the tunnel, including Basic auth from the proxy URL. https:// proxy URLs raise DiodeConfigError in this mode, as gRPC does not support them either.
• skip_tls_verify is added to DiodeClient and DiodeOTLPClient (it was documented but not implemented). HTTPS_PROXY is now chosen for secure targets whether or not verification is on.
• Channel creation is shared between the two clients in _open_grpc_channel. The tunnel closes with close(), on leaving a with block, and on garbage collection.
• The cryptography runtime dependency is removed. It is now a test-only extra.
• README documents the behaviour and the local listener.

Why not pinning

A lab that runs the Go and Python SDKs against a Diode ingester behind nginx found six ways the pinning approach differed from Go: expired and not-yet-valid certificates, multi-label wildcards, Host-based routing (the name override rewrites :authority), SNI-selected certificates, load-balanced backends with different leaves, and certificate rotation under a live client. Each earlier review round fixed one symptom of that design.

How tested

• pytest tests on Python 3.10, 3.13 and 3.14: 235 passed. The new tests use real TLS servers (expired certificate with the wrong name, SNI and ALPN, rotation, a 1024-bit key, proxy auth and a 407, a real gRPC server for both clients, socket cleanup) instead of mocking the TLS layer. The old test that asserted the plaintext downgrade is removed.
• ruff check netboxlabs tests: clean.
• The full lab matrix against this branch, plus a stress run: an 8 MB payload, 800 concurrent ingests, reconnect after a server restart, cleanup on garbage collection, and a clean exit with no close().

Linear

• ENGHLP-1703

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
/opt/hostedtoolcache/Python/3.11.16/x64/lib/python3.11/site-packages/netboxlabs/diode/sdk
_skip_verify_tunnel.py	236	12	95%	92, 95, 116–118, 160, 212, 219, 265–267, 326
chunking.py	33	1	97%	112
client.py	597	41	93%	195–196, 215–216, 235–237, 240–243, 333, 573, 641, 646, 650, 738–740, 769–771, 801, 833, 855, 857, 931, 962, 970, 1032–1033, 1042, 1075, 1102, 1107–1108, 1137, 1142–1145
exceptions.py	44	3	93%	69, 82–83
TOTAL	925	57	94%

Tests	Skipped	Failures	Errors	Time
257	0 💤	0 ❌	0 🔥	5.953s ⏱️

[nbl-ai-review]

AI Code Review — risk tier: Full · dimensions run: correctness, security

Advisory only. A human owns the merge decision.

Summary — Net positive. No new correctness or security issues at this head. The previously open loopback-listener access-control gap is now genuinely resolved: the unauthenticated TCP fallback is off by default and requires an explicit opt-in (with a one-off warning), falling closed otherwise.

Resolved since last review

• netboxlabs/diode/sdk/_skip_verify_tunnel.py — Loopback fallback listener access-control gap is fixed: the unauthenticated loopback listener is now disabled by default, raising DiodeConfigError unless DIODE_SKIP_TLS_VERIFY_ALLOW_LOOPBACK=true is explicitly set (with a one-off warning). Unix-socket path (Linux/macOS) is unaffected.

Nits

• The cleartext proxy CONNECT behavior for https:// proxy URLs (credentials sent unencrypted) remains a standing, human-waived item from earlier in this PR's history (matches grpc-go's own behavior, documented in the README) — not re-raised.

_{🤖 AI Code Review · run · prompts: code-review/v1 · models: claude-sonnet-5/medium · context: description · 23 comment(s) · 1 prior AI review(s) · 6/8 thread(s) resolved · usage: $2.25 · 3866k in (95% cached) · 30k out}

Previous review · 2026-10-06 · 3170683

AI Code Review — risk tier: Full · dimensions run: correctness, security

Advisory only. A human owns the merge decision.

Summary — Net positive: the shutdown race flagged in the last round is genuinely fixed (connect and proxy CONNECT now use cancellable loop.sock_connect/sock_sendall/sock_recv instead of a non-cancellable executor thread), and no new correctness issues surfaced. The loopback-listener access-control gap remains open — the author has acknowledged it and is awaiting a design decision, so no new ground to cover there this round.

Resolved since last review

• netboxlabs/diode/sdk/_skip_verify_tunnel.py — The shutdown race that could abandon an in-flight connect (leaking the event loop/thread) is fixed: connect/CONNECT now run on cancellable socket primitives instead of a blocking executor call, with a regression test covering the previously-hanging case.

Previously raised — still open

• netboxlabs/diode/sdk/_skip_verify_tunnel.py:199 — [Medium] [security] Loopback fallback listener (used when Unix sockets are unavailable, e.g. Windows) still has no per-connection access control — any local user/process can relay traffic through it using the deployer's proxy credentials. Narrowed in scope (Unix-socket path now used more often, with a /tmp retry) and documented with a warning, but the underlying gap on the TCP fallback path itself is unchanged. The author has noted this is awaiting a design decision.

Nits

• The cleartext proxy CONNECT behavior for https:// proxy URLs (credentials sent unencrypted regardless of scheme) was raised and explicitly waived earlier in this PR's history as matching grpc-go's own behavior and documented in the README; not re-raised here since nothing has changed in that area.

_{🤖 AI Code Review · run · prompts: code-review/v1 · models: claude-sonnet-5/medium · context: description · 21 comment(s) · 1 prior AI review(s) · 6/8 thread(s) resolved · usage: $2.25 · 3141k in (93% cached) · 35.9k out}

Previous review · 2026-10-06 · 90d6ae4

AI Code Review — risk tier: Full · dimensions run: correctness, security

Advisory only. A human owns the merge decision.

Summary — Net positive: the https:// proxy-URL crash is fixed with test coverage, and the tunnel design itself remains sound. One new medium-severity correctness issue surfaces in the delta (a shutdown race that can abandon an in-flight connection), and the previously-flagged loopback-listener access-control gap is narrowed but not resolved.

Findings

• [Medium] [correctness] Tunnel shutdown can abandon an in-flight connect, leaking the event loop/thread — netboxlabs/diode/sdk/_skip_verify_tunnel.py:308 — _shutdown() cancels self._tasks and awaits asyncio.gather(...), but a _handle() task blocked inside _dial() → run_in_executor(self._connect_blocking) (up to _CONNECT_TIMEOUT_S=10s) can't be interrupted by task.cancel() since a running concurrent.futures.Future ignores cancellation. close() (~line 329) only waits _SHUTDOWN_TIMEOUT_S=5.0 on run_coroutine_threadsafe(...).result(), swallows the resulting TimeoutError, and calls self._loop.call_soon_threadsafe(self._loop.stop) regardless. If the blocking connect is still outstanding past 5s, the loop stops and the thread exits before _shutdown() finishes gathering, so the in-flight connection's finally cleanup (closing writer/upstream_writer) never runs, and loop.close() is called with a still-pending task — producing "Task was destroyed but it is pending" warnings and relying on GC rather than the deterministic close() the README promises. Suggest tracking the raw executor futures (not just the wrapping tasks) and waiting on those before stopping the loop, or making the blocking connect respond to cancellation (e.g. closing the raw socket from another thread to unblock it).

• [Medium] [security] Loopback fallback listener still has no local-access control — netboxlabs/diode/sdk/_skip_verify_tunnel.py:252 — When Unix sockets are unavailable (or the path still exceeds _UNIX_PATH_LIMIT even after the new /tmp retry), _listen() falls back to an unauthenticated asyncio.start_server(self._handle, "127.0.0.1", 0). Any other local user on the host can connect and have _handle() dial out over unverified TLS — and through the configured proxy, using the deployer's proxy credentials — on their behalf, for the client's lifetime. This PR narrows how often the fallback triggers and adds a warning log plus a README callout, but per-connection authentication on the TCP fallback path is still absent. Suggest a lightweight per-connection shared-secret check in _handle(), or a platform-appropriate ACL'd alternative (e.g. a Windows named pipe) instead of a bare loopback TCP port.

Resolved since last review

• netboxlabs/diode/sdk/_skip_verify_tunnel.py — The https:// HTTPS_PROXY constructor crash is fixed: the tunnel now sends a plain CONNECT regardless of the proxy URL's scheme (matching grpc-go), with a new unit test covering it.

Previously raised — still open

• netboxlabs/diode/sdk/_skip_verify_tunnel.py:252 — [Medium] [security] Loopback fallback listener has no local-access control. The author notes a per-connection secret isn't feasible here since the tunnel carries gRPC's own bytes, and has mitigated by preferring the Unix-socket path more often plus a warning/README callout — but the underlying gap on the fallback path itself is unchanged.

_{🤖 AI Code Review · run · prompts: code-review/v1 · models: claude-sonnet-5/medium · context: description · 19 comment(s) · 1 prior AI review(s) · 6/8 thread(s) resolved · usage: $1.69 · 2133k in (91% cached) · 29.4k out}

Previous review · 2026-10-06 · 58e0ff2

AI Code Review — risk tier: Full · dimensions run: correctness, security

Advisory only. A human owns the merge decision.

Summary — Net positive: replacing the probe-and-pin approach with a local plaintext-to-TLS tunnel is a much sounder design, and it resolves every previously open finding tied to the old pinning code (that code is gone entirely). Two new issues surface in the delta, both edge cases of the new tunnel rather than problems with the core approach.

Findings

• [Medium] [correctness] https:// HTTPS_PROXY crashes client construction only when skip-verify is on — netboxlabs/diode/sdk/_skip_verify_tunnel.py:116-120 — SkipVerifyTunnel.__init__ raises DiodeConfigError unconditionally for any proxy scheme other than http, and nothing in DiodeClient.__init__/DiodeOTLPClient.__init__ catches it. The non-skip-verify path (client.py:354, via grpc.http_proxy) accepts HTTPS_PROXY=https://... today, since the URL scheme is advisory for CONNECT proxies (same convention curl and Go's ProxyFromEnvironment use) and the proxy connection itself is always plain TCP + CONNECT regardless. A deployment that already works with an https:// proxy URL and TLS verification on breaks hard — an unhandled constructor exception, no fallback — the moment DIODE_SKIP_TLS_VERIFY=true is turned on. Suggest normalizing/accepting the https:// scheme the same way the non-tunnel path does, or warn-and-fall-back instead of raising out of the constructor.
• [Medium] [security] Loopback fallback listener has no local-access control, unlike the Unix-socket path — netboxlabs/diode/sdk/_skip_verify_tunnel.py:143-156 — When start_unix_server isn't available, or the temp socket path exceeds the 100-byte limit, or it raises OSError, _listen() falls back to asyncio.start_server(self._handle, "127.0.0.1", 0) with no authentication. The Unix-socket path is protected by a 0700 tempfile.mkdtemp() directory, restricting it to the owning OS user; the loopback TCP fallback accepts a connection from any local process/user on the box. Each accepted connection re-runs the configured proxy CONNECT (using the deployer's proxy credentials) and dials out over unverified TLS, so a second local user could ride this open relay to reach the configured target through the deployer's credentials/trust without knowing them. Suggest a per-connection shared-secret check in _handle() for the fallback path, or at minimum documenting the reduced isolation alongside the existing Unix-socket description in the README.

Resolved since last review

• netboxlabs/diode/sdk/client.py — Multi-probe skip-verify pinning (trusting the union of all probed leaf certs) is gone entirely; the probe-and-pin design has been replaced by the plaintext-to-TLS tunnel.
• netboxlabs/diode/sdk/client.py — The IPv6-literal HTTPS_PROXY port-mangling bug in _proxy_url_for_grpc no longer applies; that function and the code path it served have been removed.

Pre-existing / out-of-scope

• None identified outside this change's diff.

_{🤖 AI Code Review · run · prompts: code-review/v1 · models: claude-sonnet-5/medium · context: description · 17 comment(s) · 1 prior AI review(s) · 6/8 thread(s) resolved · usage: $1.98 · 2788k in (92% cached) · 26.4k out}

[jajeffries]

AI review follow-up (pushed d621ee3)

• Peer-cert probe vs gRPC channel: _skip_verify_channel_credentials now runs up to three probe handshakes and passes a combined root_certificates bundle (all distinct leaf PEMs) to gRPC, so a load-balanced peer that rotates certs between connections is more likely to match the real channel. When probe runs return different SNI names, grpc.ssl_target_name_override falls back to the authority host.
• Unwrapped network/TLS errors: _connect_socket and _fetch_peer_leaf_certificate wrap OSError / ssl.SSLError in DiodeConfigError with context (proxy CONNECT failures unchanged).
• Private CPython cert decoder: removed ssl._ssl._test_decode_cert; server name comes from SSLSocket.getpeercert() during the probe handshake.
• CodeQL (TLS 1.0/1.1 on probe): probe uses SSLContext(PROTOCOL_TLS_CLIENT) with minimum_version = TLSv1_2 instead of create_default_context().

Verification: pytest tests/test_client.py -q (136 passed), ruff check netboxlabs/diode/sdk/client.py tests/test_client.py.

[jajeffries]

/ai-review

[jajeffries]

AI review follow-up (open finding: unwrapped TLS probe errors)

Pushed fix for the remaining Medium item on _fetch_peer_leaf_certificate: the TLS handshake block now catches (ssl.SSLError, OSError) and wraps both in DiodeConfigError, matching _connect_socket.

Verification: pytest tests/test_client.py -q (136 passed), ruff check netboxlabs/diode/sdk/client.py tests/test_client.py.

/ai-review

[jajeffries]

AI review follow-up (low findings on 0c40e2ee)

• IPv6 literals: _authority_host_port strips [/] so grpcs://[::1]:port skip-verify probes connect correctly.
• HTTPS proxy: _connect_socket wraps the proxy TCP socket in TLS when the proxy URL uses https:// before issuing CONNECT.

Verification: pytest tests/test_client.py -q (139 passed), ruff check netboxlabs/diode/sdk/client.py tests/test_client.py.

/ai-review

[jajeffries]

AI review follow-up (nested TLS on 46c67dd)

Reverted TLS-wrapping the proxy socket before CONNECT. The skip-verify probe now mirrors grpc.http_proxy: plain TCP to the proxy, CONNECT, then a single TLS handshake to the target on the tunneled socket.

Waiver (low security, HTTPS proxy CONNECT): We are not adding a separate TLS session to the proxy for this probe path. That matches how gRPC uses HTTPS_PROXY for the real channel; a TLS-to-proxy CONNECT tunnel would also require TLS-in-TLS through the probe, which breaks when combined with the target handshake (see AI review on 46c67dd).

Verification: pytest tests/test_client.py -q (138 passed), ruff check netboxlabs/diode/sdk/client.py tests/test_client.py.

/ai-review

[leoparente]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-06T11:50:49.148980Z	674ee1f	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 197c0b75a5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[leoparente]

P1: skip_tls_verify still appears to enforce hostname verification

I don't think this fully matches Go's InsecureSkipVerify semantics yet.

In _fetch_peer_leaf_certificate() we do:

context.check_hostname = False
context.verify_mode = ssl.CERT_NONE

tls_sock = context.wrap_socket(raw_sock, server_hostname=host)

der_cert = tls_sock.getpeercert(binary_form=True)
server_name = _tls_server_name_from_peercert(
    tls_sock.getpeercert(),
    host,
)

The problem is that Python returns an empty decoded dict from getpeercert() when the certificate was not validated (CERT_NONE). So in the real skip-verify path _tls_server_name_from_peercert() will normally fall back to host.

We then pin the peer leaf cert as a trusted root and set:

("grpc.ssl_target_name_override", host)

which means gRPC can still reject the connection if the certificate SAN/CN doesn't match the target hostname/IP.

For example:

target: grpcs://10.0.0.20:443
cert SAN: diode.internal
DIODE_SKIP_TLS_VERIFY=true

Go's InsecureSkipVerify: true should accept this, while this implementation looks like it would still fail hostname validation.

Can we add an integration test with a real self-signed TLS server where the target hostname deliberately does not match the cert SAN, and make sure skip_tls_verify=True succeeds? We probably need to derive the SAN/CN from the DER certificate directly (or otherwise truly disable gRPC hostname verification) rather than relying on the decoded getpeercert() result.

[leoparente]

P1: skip_tls_verify still rejects expired/not-yet-valid certificates

The DER decoding fix removes certificate validation from the probe path, but the actual gRPC channel still builds normal TLS credentials from the pinned leaf:

credentials = grpc.ssl_channel_credentials(
root_certificates=pem
)

That means gRPC/BoringSSL still performs certificate validity checks during the real channel handshake.

So an expired or not-yet-valid self-signed certificate can still fail with CERTIFICATE_VERIFY_FAILED even when DIODE_SKIP_TLS_VERIFY=true.

This does not match Go's InsecureSkipVerify: true, which skips both chain/hostname validation and certificate validity checks.

Could we add an integration test using an expired certificate and verify that skip_tls_verify=True can actually establish the gRPC channel? The current implementation appears to fix the probe-side validation but still reapplies verification when the real gRPC channel is created.

[jajeffries]

/ai-review

[jajeffries]

AI review follow-up (pushed 90d6ae4)

• https:// proxy URL no longer raises (medium, correctness). The tunnel now sends a plain CONNECT whatever the proxy URL scheme says, as grpc-go does. The default proxy port is 443 for https:// URLs. A new unit test covers it. Worth noting from the lab: with an https:// proxy URL pointing at a plain-HTTP proxy, the OAuth token request already fails in every SDK (Go included), because it does TLS to the proxy. That is independent of skip-verify, so this configuration cannot have worked before.
• Loopback listener isolation (medium, security). The Unix socket (a 0700 directory) is now used in more cases: if TMPDIR makes the socket path too long, the SDK retries in /tmp. The unauthenticated loopback listener is therefore only used where Unix sockets are unavailable, for example Windows. In that case a warning is logged once, and the README states the reduced isolation. A per-connection secret is not possible, because the tunnel carries gRPC's own bytes and the client cannot add a handshake. Tests cover the /tmp retry (mutation-checked) and the loopback fallback with the warning.

Verification: pytest tests (237 passed), ruff check. Re-run against the TLS lab: 171 scenario/mode cells, 1 differs from Go (Ed25519 in verifying DIODE_CERT_FILE mode, a gRPC core limit that the released SDK shares). The new https://-proxy scenarios behave identically in Go and Python.

[jajeffries]

/ai-review

[jajeffries]

AI review follow-up (pushed 3170683)

• Shutdown race (medium, correctness): fixed. Connect and proxy CONNECT now use loop.sock_connect / sock_sendall / sock_recv instead of an executor thread, so every step is cancellable and close() completes promptly with no pending task. The new test holds a CONNECT open on a proxy that never answers and fails against the previous implementation (it took 5s and left the task pending). Re-checked with the TLS lab (171 cells, still only the Ed25519 cafile gRPC-core gap differs from Go), an 8 MB payload, 800 concurrent ingests, reconnect after a server restart, and pytest on 3.10 and 3.14 (238 passed).
• Loopback listener access control (medium, security): open, needs a decision. Open. Awaiting a design decision from the PR author.

[jajeffries]

/ai-review

[jajeffries]

AI review follow-up (pushed 36441ce)

• Loopback listener access control (medium, security): resolved by failing closed. A per-connection secret is not possible, because the tunnel carries gRPC's own bytes and the client cannot add a handshake. Instead the unauthenticated loopback listener is now off by default. Where Unix sockets are unavailable (mainly Windows), the client raises a DiodeConfigError that names the setting, and DIODE_SKIP_TLS_VERIFY_ALLOW_LOOPBACK=true is required to opt in (with a one-off warning). Linux and macOS use the 0700 Unix socket and are unaffected. Two tests cover the default refusal (no thread left behind) and the opt-in path. The README documents it.

Verification: pytest tests (239 passed, also on 3.10 and 3.14 for the previous commit), ruff check, and a re-run of the TLS lab subset: 0 cells differ from Go.

[jajeffries]

/ai-review

[nbl-ai-review]

🔁 AI Code Review updated for 36441ce1 — see the review · 1 resolved · 0 open (run)

[leoparente]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 36441ce183

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[leoparente]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 674ee1f93a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

🎉 This PR is included in version 1.14.2 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀