Skip to content

Complete native Volume transfers for ExecContext - #560

Open
aakash-saravanan-db wants to merge 20 commits into
mainfrom
aakash-saravanan_data/feat/volume-restart-20261113
Open

aakash-saravanan-db wants to merge 20 commits into
mainfrom
aakash-saravanan_data/feat/volume-restart-20261113

Conversation

@aakash-saravanan-db

@aakash-saravanan-db aakash-saravanan-db commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Feature or gap

Kernel-backed Go connections need to complete Volume operations with existing staging authorization and error handling.

What this PR addresses

Delegate transfers to shared native completion and preserve cancellation causes. Apply the existing fatal-session policy to SQL instruction errors while preserving sessions on storage HTTP/body failures, including errors racing context cancellation, without returning ErrBadConn or replaying the statement.

Dependencies

Builds select databricks/databricks-sql-kernel#465 at d9fcf834d36dc7f3de1621225db716200151fa0f and its matching C header. Merge and release require compatible binding versions, updated modules/sums, and clean installed-package validation. Published 1.4.0 archives lack the new exports.

Tests and validation

Retained validation against native archive cec365e6 passes 48 focused native-enabled checks, including failure-origin/cancellation and closed-connection/telemetry controls. The retained source-built named file lifecycle matches Thrift on DBSQL CURRENT: one matching workflow and zero differences, with initial bytes, changed overwrite, storage-origin removal and owned cleanup verified. The recorded executable links the audited kernel archive; the instruction-reader error-telemetry path is verified through focused shared-core tests. Retained default/full-suite checks, the invalidation-ordering negative control and performance comparisons keep their recorded build qualifications. Clean installed-package and scheduled activation remain open.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
@aakash-saravanan-db aakash-saravanan-db added the integration-test-kernel Preview the full passthrough kernel integration suite (live warehouse) label Nov 14, 2026
@github-actions

Copy link
Copy Markdown

Go integration tests triggered (sea / passthrough). View workflow runs.

@aakash-saravanan-db
aakash-saravanan-db marked this pull request as ready for review November 14, 2026 02:18

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low · 1 Nit

Mostly sound, with two low-severity items. The native staging dispatch, cancellation watcher, and defer teardown order look correct. REMOVE not-found=success matches Thrift's 404 handling. The kernel-side behavior depends on the pinned kernel revision, which isn't in this checkout, so I couldn't check it here. Nit: the error strings "Volume transfer: ..." and "Volume operation requires..." start with a capital letter, which is against Go convention (staticcheck ST1005).

Other findings

  • ⚪ Nit — nativeStagingStub has an err field, but no table case ever sets it. The dispatch path that wraps a CompleteStaging failure in NewDriverError(..., "error performing staging operation", err) is never tested. No test checks that the returned error is non-nil in that case, or that the cause can still be reached with errors.Is/errors.As (e.g. context.Canceled, *os.PathError). That cause chain is the point of the new KernelError.Unwrap and the ctx-wrapping in CompleteStaging. A case with err: context.Canceled that asserts errors.Is(err, context.Canceled) would cover it.

C.kernel_result_stream_canceller_cancel(canceller)
case <-done:
}
}()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — CompleteStaging returns toStatementError(err) for failures from get_result_stream, canceller_new, and complete_volume_operation, but it never calls o.backend.evictIfSessionFatal(err). Both Results (L373-382) and execute make that call on the same kinds of kernel errors. Suppose the token expires or the session is dropped while a Volume transfer completes. The error still reaches the caller, but the conn stays marked valid, so the pool keeps handing out a dead connection. The non-staging read path doesn't have this problem. IsStaging's error path has the same gap. Calling o.backend.evictIfSessionFatal(err) before returning in these branches would match Results.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the reachable error sources before adding eviction. kernel_executed_statement_get_result_stream only validates/takes the existing synchronous stream (src/c_abi/statement.rs:733–772); marker and detached-canceller creation are local ABI operations. They cannot observe an expired token or dropped remote session.

Completion can encounter both result transport and presigned-storage failures. Their shared kernel error codes do not establish that the SQL session is dead; applying evictIfSessionFatal indiscriminately could discard a healthy SQL connection on a storage authentication/permission failure. I have not found a demonstrated session-origin failure requiring a change here. No message-prefix classifier or new error-origin framework was added for this hypothetical case.

The native source pin is updated to 22f96fac for the required shared retries. Native-enabled package tests and the final named two-backend Volume workflow pass with exact bytes/deletion proof. A concrete session-origin reproduction would be useful evidence to reassess selective eviction.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right about get_result_stream and the canceller: they're local ABI operations, so they can't report a dead session, and that part of my finding was wrong. The complete_volume_operation branch still looks inconsistent to me, though. isSessionFatal only matches Unauthenticated, Unavailable and NetworkError. Results/rows.go already call evictIfSessionFatal on those codes even when the error comes from CloudFetch downloads to presigned storage, which carry the same "storage, not session" ambiguity you describe. So the trade-off you're avoiding here is one the read path already makes. Accepting it in one place and not the other is reasonable, but it's a call for a maintainer. The worst case is a dead conn going back to the pool after a Volume completion fails with an auth or transport error. That's low severity, so I'll leave the thread open for a human to decide instead of pushing further.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
@github-actions github-actions Bot removed the integration-test-kernel Preview the full passthrough kernel integration suite (live warehouse) label Nov 14, 2026
@github-actions

Copy link
Copy Markdown

Integration test approval reset.

New commits were pushed to this PR. Label(s) integration-test-kernel were removed for security.

A maintainer must re-review and re-add a label to preview tests again. (The real gate runs in the merge queue.)

Latest commit: 26f142a

@aakash-saravanan-db aakash-saravanan-db added the integration-test-kernel Preview the full passthrough kernel integration suite (live warehouse) label Nov 14, 2026

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 2 Low

The dispatch is mostly sound, with one medium lint concern in the new kernel CompleteStaging path. ExecContext now routes kernel Volume operations to native completion, and Thrift keeps its existing row-based path. The rest are low-severity issues: missing session-fatal eviction and thin test coverage.

Comment thread internal/backend/kernel/operation.go Resolved
return toStatementError(err)
}
defer C.kernel_result_stream_close(stream)
var canceller *C.kernel_result_stream_canceller_t

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — CompleteStaging converts kernel failures with toStatementError but never calls o.backend.evictIfSessionFatal(err). That applies to get_result_stream, canceller_new and complete_volume_operation. Every other kernel path that runs after execute does call it, including Results on the same get_result_stream call and every failure branch in execute. A completion that runs long enough for the token to expire, or for the session to be dropped, will return the error but leave the conn marked valid in the pool. The next statement on that conn then fails as well. Call o.backend.evictIfSessionFatal(err) on these failure branches, the same way Results does.

Comment thread connection_test.go Resolved
@github-actions

Copy link
Copy Markdown

Go integration tests triggered (sea / passthrough). View workflow runs.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
@github-actions github-actions Bot removed the integration-test-kernel Preview the full passthrough kernel integration suite (live warehouse) label Nov 14, 2026
@github-actions

Copy link
Copy Markdown

Integration test approval reset.

New commits were pushed to this PR. Label(s) integration-test-kernel were removed for security.

A maintainer must re-review and re-add a label to preview tests again. (The real gate runs in the merge queue.)

Latest commit: f0c6f41

@aakash-saravanan-db aakash-saravanan-db added the integration-test-kernel Preview the full passthrough kernel integration suite (live warehouse) label Nov 14, 2026

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Nit

Mostly sound. One medium issue: when native staging completion fails with a session-fatal error, the session is not marked dead, unlike every other kernel result path. Kernel-side test coverage for the new paths is also thin. Nit: the new CHANGELOG entry is followed by a stray blank line inside the bullet list.

Other findings

  • ⚪ Nit — None of the new kernel-side behaviour has unit coverage. Untested paths:
  • kernelOp.IsStaging with a nil exec.
  • The CompleteStaging outcome mapping (outcome != 2 returns an error).
  • Wrapping of the cancelled-context error.
  • The new KernelError.Unwrap / cause path, where errors.Is(err, fs.ErrNotExist) should work through a *KernelError carrying an os.PathError.

The only new test (TestConnNativeStagingDispatch) uses a Go stub at the conn layer. The removed TestExecuteRejectsStaging has no replacement, and driver_e2e_test.go has no kernel staging test. Unwrap could be pinned in the non-cgo errors_classify_test.go. kernel_test.go could cover IsStaging with a nil exec and the outcome mapping.

Comment thread internal/backend/kernel/operation.go Resolved
@github-actions

Copy link
Copy Markdown

Go integration tests triggered (sea / passthrough). View workflow runs.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
@github-actions github-actions Bot removed the integration-test-kernel Preview the full passthrough kernel integration suite (live warehouse) label Nov 14, 2026
@github-actions

Copy link
Copy Markdown

Integration test approval reset.

New commits were pushed to this PR. Label(s) integration-test-kernel were removed for security.

A maintainer must re-review and re-add a label to preview tests again. (The real gate runs in the merge queue.)

Latest commit: 3629843

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 2 Low

The PR looks correct overall. There is one medium concern: CompleteStaging skips evicting a dead session when a session-fatal error races context cancellation, which execute() deliberately avoids. There are also two low gaps: no eviction on stream-acquisition failure, and no test for the cancellation path.

Comment thread internal/backend/kernel/operation.go Resolved
Comment thread internal/backend/kernel/operation.go Resolved
Comment thread staging_kernel_test.go Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 2 Low

The dispatch and cancellation design look sound, with two low-severity concerns: two early-failure paths in CompleteStaging skip session-fatal eviction, and the native path drops two telemetry tags. As the PR description already says, the pinned KERNEL_REV and header point at an unmerged kernel commit and need exports that the published binding modules don't have, so databricks_kernel-tagged builds against published bindings will fail to link until those modules and the module/sum updates ship.

Comment thread internal/backend/kernel/operation.go Resolved
Comment thread connection.go Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Low · 1 Nit

Mostly solid, with one medium gap: the session-fatal eviction this PR adds is skipped on the early get_result_stream failure path in CompleteStaging. Also worth noting for release: KERNEL_REV pins an unmerged kernel PR commit, and the new C symbols need a kernel release later than 1.4.0 before this can ship. The cancellation test relies on the exact number of ctx.Err() calls (errCalls == 2), which is fragile (nit).

Other findings

  • 🔵 Low — Removing the isStagingStatement pre-check means staging detection now relies only on the server manifest's is_volume_operation flag, reported through kernel_executed_statement_is_volume_operation. If a warehouse or result shape leaves that flag out (or the kernel misses it), IsStaging returns false, and ExecContext on a PUT/GET reports success with 0 rows while no file moves. That is the silent-data-loss case the removed guard existed to block. Consider keeping the SQL-text detector as a fallback: if the text looks like PUT/GET/REMOVE but the operation is not marked, return an error instead of success. Alternatively, record that the server is guaranteed to always set the flag.
  • ⚪ Nit — The native branch returns before the telemetryUpdate callback is wired, so kernel Volume transfers never set TagChunkCount / TagBytesDownloaded. The Thrift staging path does set them. This is a minor telemetry gap compared with Thrift. If the C ABI can't report byte counts, a short comment noting the omission would help.

Comment thread internal/backend/kernel/operation.go Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low · 2 Nit

Mostly sound, with 3 low concerns: the session-eviction policy is applied inconsistently on the early error paths of CompleteStaging, one error message is misleading for status 0, and the new cancellation watcher has no test. Follow-up outside this PR: per the description, the published 1.4.0 binding archives lack the new kernel_get_last_error_* and volume exports, so a -tags databricks_kernel build against the published bindings will fail to link until a matching bindings release ships.

Other findings

  • ⚪ Nit — The outcome != 2 branch handles two different cases: status 0 ("ordinary result": the stream turned out not to be a Volume operation) and status 1 ("input bytes required"). Both get the same error, volume operation requires a local file instruction, and that message only makes sense for status 1. A status-0 result after IsStaging returned true points to a kernel or server inconsistency, and this message would send whoever debugs it in the wrong direction. Neither case is tested. staging_kernel_test.go only covers error returns from completion, and TestConnNativeStagingDispatch stubs out CompleteStaging. Consider using a distinct message for each status (including the numeric status helps) and adding a test for it.
  • ⚪ Nit — The *Cancelled subtests do not exercise the cancellation path this PR adds. cancelOnCompletionErrorContext cancels on the second Err() call, which comes after kernel_result_stream_complete_volume_operation has already returned an error. So the watcher goroutine never calls kernel_result_stream_canceller_cancel while a transfer is running. These cases only check how the error is wrapped after the fact. They also depend on how many times CompleteStaging happens to call ctx.Err(), so adding or removing one check would silently change what they test. A test that cancels while the storage request is in flight (for example, a handler that blocks until the ctx is cancelled) would cover the documented promise that "context cancellation" interrupts the transfer.

Comment thread internal/backend/kernel/operation.go Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low · 1 Nit

Looks good overall, with 2 low concerns: some kernel CompleteStaging setup failures skip the session-eviction policy, and nothing tests that cancellation actually interrupts an in-flight transfer. Follow-up (not caused by this PR): the native path doesn't emit the chunk-count/bytes telemetry tags that the Thrift staging path does.

Other findings

  • ⚪ Nit — Nothing tests the result-stream canceller watcher (case <-ctx.Done(): C.kernel_result_stream_canceller_cancel(canceller)). The two *Cancelled subtests in staging_kernel_test.go cancel the context from inside Err() only after native completion has already returned. They exercise the error-wrapping branch, not an interrupt of an in-flight transfer. If the watcher were dropped or never fired, a cancelled ExecContext would block until the storage transfer finished and every test would still pass. Consider adding a case where the /object handler blocks until the request context is cancelled, then assert that ExecContext returns context.Canceled promptly.

Comment thread internal/backend/kernel/operation.go Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good overall, with one low concern. Native Volume completion now goes through CompleteStaging. The canceller and deferred cleanup run in the correct LIFO order, and storage failures leave the session valid. When the instruction stream fails to open, CompleteStaging doesn't apply the session-eviction policy that Results applies to the same call. Nit: the VOLUME_STORAGE_* error-code strings are inline literals and could be named constants.

Comment thread internal/backend/kernel/operation.go Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium

Looks mostly correct, with 1 medium issue: the new CompleteStaging doesn't evict a dead session when getting the result stream or the canceller fails. Otherwise the dispatch, cancellation wiring, defer ordering, and the storage-vs-instruction eviction split look right. Follow-up: the native path no longer emits the chunk-count and bytes-downloaded telemetry tags that the Thrift staging path records.

Comment thread internal/backend/kernel/operation.go Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

The change looks sound, with one low-severity concern: the kernel example still describes staging as rejected on the kernel backend. CompleteStaging cleans up in a safe order (stops the watcher goroutine, then frees the canceller, then closes the stream). It only evicts the session for non-storage failures, and it never returns ErrBadConn. Nits: the cancellation test depends on the exact number of ctx.Err() calls (errCalls==2), so a harmless refactor could break it. The native completion path also skips the chunk-count/bytes telemetry tags that the Thrift staging path records.

Other findings

  • 🔵 Low — This PR removes the up-front staging rejection from Execute, but examples/kernel/main.go still says staging is "rejected at execute". It also runs a real PUT '/tmp/local.csv' INTO '/Volumes/main/default/vol/f.csv' OVERWRITE and expects ErrNotSupportedByKernel.

After this change, that statement goes to the server. With no staging context set, it fails with "staging ctx must be provided." instead of the documented rejection. If someone copies the example and adds staging roots, the PUT really runs and overwrites the Volume file. The example and its comment should be updated or the staging block removed, along with the doc.go/CHANGELOG edits.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
@aakash-saravanan-db

Copy link
Copy Markdown
Collaborator Author

The obsolete staging-rejection example is removed in 6218f29. The example retains backend selection, connection handling and SELECT; it no longer submits the hard-coded PUT. go test ./examples/kernel compiles successfully without executing remote actions.

The cancellation fault-injection test's dependence on the ctx.Err() boundary remains an acknowledged maintenance tradeoff; it checks the real post-completion cancellation/session-error race. The native path's missing Thrift chunk/byte telemetry tags remain an optional observability limitation, with no telemetry-parity claim.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks mostly good, with one low concern. Native completion goes through ExecContext cleanly, and the storage-vs-instruction eviction split is tested against the real native library, including races with cancellation. Possible follow-up: the native staging path no longer records chunk/byte telemetry tags the way the Results-based path did.

Comment thread internal/backend/kernel/operation.go Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium

Looks good overall, with one medium gap. On two early setup failures in CompleteStaging, the session-fatal eviction that Results() applies is skipped. Not blocking but worth a follow-up: the native path drops the chunk/bytes telemetry tags that the Thrift staging path sets, and no tagged test covers a successful native completion or the outcome != 2 branch.

Comment thread internal/backend/kernel/operation.go Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good overall. I found one low-severity gap: the early error paths in CompleteStaging skip the session-eviction policy. The dispatch, cancellation teardown order, ErrBadConn avoidance and REMOVE-404 parity with Thrift all check out. A possible follow-up: the native path records no chunk/bytes telemetry tags, while the Thrift staging path does.

Comment thread internal/backend/kernel/operation.go Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good overall, with one low concern: CompleteStaging doesn't apply the fatal-session eviction when it fails to get the result stream or create the canceller. The dispatch in execStagingOperation, the cancellation wrapping and the storage-vs-instruction eviction split look correct. The new error-code values (VOLUME_STORAGE_HTTP_ERROR and the others) are typed in as string literals; named constants would be a nice touch (nit).

Comment thread internal/backend/kernel/operation.go Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good overall, with one low-severity gap. When opening the result stream or its canceller fails inside CompleteStaging, the session is not evicted, unlike in Results. As a nit, the native path no longer adds the chunk-count/bytes telemetry tags that the Thrift staging path adds, which could be follow-up work.

Comment thread internal/backend/kernel/operation.go Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Low

Mostly sound, with one medium test-coverage gap and one low inconsistency in session eviction. The dispatch, cancellation watcher, defer ordering and storage/instruction error split look correct. As follow-up: the native path does not record the chunk-count and bytes-downloaded telemetry tags that the Thrift staging path reports, and the storage exemption depends on two hard-coded kernel error-code strings.

Comment thread staging_kernel_test.go
return c.Context.Err()
}

func TestKernelVolumeFailureOriginControlsSessionValidity(t *testing.T) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The only in-repo test of the native path covers failures: instruction 401, storage 401, a short body, and the two cancelled variants. Nothing tests the main feature the CHANGELOG and doc.go describe:

  • a GET or PUT through ExecContext that succeeds and writes or uploads the bytes, then reports zero rows affected;
  • a REMOVE that succeeds, including a 404 treated as success because of remove_not_found_is_success=true;
  • the new rule that rejects a symlink whose resolved target leaves the staging roots.

TestConnNativeStagingDispatch stubs CompleteStaging, so it never reaches kernelOp.CompleteStaging or the C ABI. The httptest harness already in this file can do this. Add a success mode that checks the destination's new contents and IsValid()==true. Add a mode where localFile is a symlink inside directory that points outside it, and assert an error with no storage request. Without these, a kernel bump that breaks the success outcome (outcome != 2) or the symlink check would pass CI.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Go-local native test does cover failure/cancellation rather than successful transfer or escaped symlinks; expanding it would be useful optional regression coverage. Required behavior is covered by distinct groups: current shared-core tests cover resolved-root rejection and permissive REMOVE404; the C ABI maps Completed directly to outcome2; and the actual source-built Go ExecContext lifecycle verifies initial/changed PUT, GET bytes and REMOVE against Thrift with one match and zero differences. That retained comparison used Go513f8e73 and native archivecec365e6; the current executable Go adapter is byte-identical, with only its source pin changed. The retained native predates the later common-core reader/empty-batch corrections, which have fresh shared-core checks. The live run does not prove Go-local symlink or REMOVE404 cases. A duplicate connector matrix is not needed for this correction; package/install and named scheduled execution gates remain open.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that's a fair summary of where coverage lives. Shared-core tests and the outcome-2 mapping in the C ABI do cover the logic itself. But those tests run in the kernel's CI, and the source-built live comparison was a one-off check against a pinned archive, not a gate in this repo. You also note that it doesn't cover the Go-local symlink or REMOVE-404 cases. So the risk I raised is still there. If a future kernel bump or adapter edit breaks how kernelOp.CompleteStaging handles a successful outcome, CI here would still pass. One success mode in the existing httptest harness would close most of that gap: a GET that checks the destination bytes, IsValid()==true and zero rows affected. A second mode would cover the symlink: a symlink that escapes the staging roots, asserting an error and no storage request. Leaving this open so a human can decide whether to add those tests now or track them as follow-up.

Comment thread internal/backend/kernel/operation.go Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant