Skip to content

SK-3139: Implement query, get_tokens, upload_files, delete_files in flowvault - #283

Merged
saileshwar-skyflow merged 4 commits into
flowvault-release/26.10.0from
saileshwar/SK-3139-implement-query-gettokens
Oct 6, 2026
Merged

saileshwar-skyflow merged 4 commits into
flowvault-release/26.10.0from
saileshwar/SK-3139-implement-query-gettokens

Conversation

@saileshwar-skyflow

@saileshwar-skyflow saileshwar-skyflow commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds four new unary operations to the Python flowvault SDK (skyflow-flowvault-python), mirroring the design doc and the existing get/detokenize patterns:

Operation Method Notes
Query vault().query() Read-only SQL SELECT (≤25 rows, page via OFFSET). No per-record results — any failure is thrown; request_id always populated.
GetTokens vault().get_tokens() Value → token lookup for deterministic token groups. Per-record results (200/207).
UploadFiles vault().upload_files() Orchestrated: fetch a signed URL per column (POST /v2/files/upload), then PUT the bytes. Accepts file_path / base64 / file_object; per-column UPLOADED/FAILED/SKIPPED. The signed URL is used internally and not returned.
DeleteFiles vault().delete_files() Delete files from file columns by skyflow_id or unique_values (POST /v2/files/delete).

What's included

  • Data classes under flowvault/skyflow/vault/data/ for each operation (request/record/column/options/response) plus base classes in common/vault/data/ (mirroring the Java layout).
  • Validators (validate_query_request, validate_get_tokens_request, validate_upload_files_request, validate_delete_files_request) and message-catalog entries (Error / Info / ErrorLogs).
  • Controller methods + helpers in _vault.py; VaultClient accessors get_query_api(), get_files_api(), and put_signed_url().
  • Generated files wire client (generated/rest/files/) and file request/response types.
  • Samples (query_records.py, get_tokens.py, upload_files.py, delete_files.py) and README / samples-README updates.
  • New FileUploadStatus enum.

Testing

  • Compilation: clean across common, flowvault, skyvault.
  • Unit tests: common 88 passed, flowvault 255 passed, skyvault 451 passed (2 pre-existing collection errors in skyvault require a credentials.json test fixture not present in this checkout — unrelated to this change).
  • Builds: wheels + sdists produced for all three packages; the flowvault wheel bundles common and the new files API.

Notes

  • upload_files is sequential in v1; the async virus-scan fileStatus is read later via get().
  • Design docs updated separately (Python + Java) to drop signed_url from the per-column upload result.

…lowvault

Add four unary operations to the Python flowvault SDK, mirroring the design
doc and the existing get/detokenize patterns:

- query: read-only SQL SELECT (QueryRequest/QueryResponse); failures are thrown.
- get_tokens: value -> token lookup for deterministic token groups, with
  per-record results.
- upload_files: orchestrated two-phase upload (fetch a signed URL per column,
  then PUT the bytes), accepting file_path/base64/file_object input and
  returning a per-column upload status. The signed URL is not surfaced.
- delete_files: delete files from file columns by skyflow_id or unique_values.

Includes base classes in common, validators, message-catalog entries, the
generated files wire client, VaultClient accessors (get_query_api,
get_files_api, put_signed_url), data-class exports, runnable samples, and
README updates.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

✅ Gitleaks Findings: No secrets detected. Safe to proceed!

…eline

- version.py: the Fern regeneration had overwritten the repo customization with
  metadata.version("skyflow.generated.rest"), which has no installed distribution
  and raised PackageNotFoundError -- breaking `unittest discover` (and all imports
  of skyflow.generated.rest) in CI. Restore the lookup of "skyflow-flowvault-python"
  with the PackageNotFoundError -> "0.0.0" fallback.
- api-report/skyflow.api.json: refresh the public-API contract baseline to include
  the new query / get_tokens / upload_files / delete_files surface (additive only;
  no members removed).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

✅ Gitleaks Findings: No secrets detected. Safe to proceed!

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

Public API contract change (flowvault)

This PR changes flowvault/api-report/skyflow.api.json (the approved public API contract for skyflow). Review the surface change below:

diff --git a/flowvault/api-report/skyflow.api.json b/flowvault/api-report/skyflow.api.json
index 984c4b3..96fe8e2 100644
--- a/flowvault/api-report/skyflow.api.json
+++ b/flowvault/api-report/skyflow.api.json
@@ -34,16 +34,24 @@
   "skyflow.utils.enums.EnvUrls.PROD": "attr = 'skyvault.skyflowapis.com'",
   "skyflow.utils.enums.EnvUrls.SANDBOX": "attr = 'skyvault.skyflowapis-preview.com'",
   "skyflow.utils.enums.EnvUrls.STAGE": "attr = 'skyvault.skyflowapis.tech'",
+  "skyflow.utils.enums.FileUploadStatus": "class (Enum)",
+  "skyflow.utils.enums.FileUploadStatus.FAILED": "attr = 'FAILED'",
+  "skyflow.utils.enums.FileUploadStatus.SKIPPED": "attr = 'SKIPPED'",
+  "skyflow.utils.enums.FileUploadStatus.UPLOADED": "attr = 'UPLOADED'",
   "skyflow.utils.enums.UpsertType": "class (Enum)",
   "skyflow.utils.enums.UpsertType.REPLACE": "attr = 'REPLACE'",
   "skyflow.utils.enums.UpsertType.UPDATE": "attr = 'UPDATE'",
   "skyflow.vault.controller.VaultController": "class (BaseVaultController)",
   "skyflow.vault.controller.VaultController.__init__": "def (self, vault_client)",
   "skyflow.vault.controller.VaultController.delete": "def (self, request: DeleteRequest, options: DeleteOptions = None) -> DeleteResponse",
+  "skyflow.vault.controller.VaultController.delete_files": "def (self, request: DeleteFilesRequest, options: DeleteFilesOptions = None) -> DeleteFilesResponse",
   "skyflow.vault.controller.VaultController.detokenize": "def (self, request: DetokenizeRequest, options: DetokenizeOptions = None) -> DetokenizeResponse",
   "skyflow.vault.controller.VaultController.get": "def (self, request: GetRequest, options: GetOptions = None) -> GetResponse",
+  "skyflow.vault.controller.VaultController.get_tokens": "def (self, request: GetTokensRequest, options: GetTokensOptions = None) -> GetTokensResponse",
   "skyflow.vault.controller.VaultController.insert": "def (self, request: InsertRequest, options: InsertOptions = None) -> InsertResponse",
+  "skyflow.vault.controller.VaultController.query": "def (self, request: QueryRequest, options: QueryOptions = None) -> QueryResponse",
   "skyflow.vault.controller.VaultController.update": "def (self, request: UpdateRequest, options: UpdateOptions = None) -> UpdateResponse",
+  "skyflow.vault.controller.VaultController.upload_files": "def (self, request: UploadFilesRequest, options: UploadFilesOptions = None) -> UploadFilesResponse",
   "skyflow.vault.data.ColumnRedactions": "class ()",
   "skyflow.vault.data.ColumnRedactions.__init__": "def (self, column_name: str, redaction: str = None)",
   "skyflow.vault.data.ColumnRedactions.column_name": "attr",
@@ -52,6 +60,31 @@
   "skyflow.vault.data.CustomHeaderKey.REQUEST_ID_HEADER": "attr = 'x-request-id'",
   "skyflow.vault.data.CustomHeaderKey.SKYFLOW_ACCOUNT_ID": "attr = 'x-skyflow-account-id'",
   "skyflow.vault.data.CustomHeaderKey.SKYFLOW_ACCOUNT_NAME": "attr = 'x-skyflow-account-name'",
+  "skyflow.vault.data.DeleteFilesColumnResult": "class ()",
+  "skyflow.vault.data.DeleteFilesColumnResult.__init__": "def (self, column: str, status: str = None)",
+  "skyflow.vault.data.DeleteFilesColumnResult.column": "attr",
+  "skyflow.vault.data.DeleteFilesColumnResult.status": "attr",
+  "skyflow.vault.data.DeleteFilesOptions": "class ()",
+  "skyflow.vault.data.DeleteFilesOptions.__init__": "def (self, interceptor: Optional[Callable[[RequestContext], None]] = None)",
+  "skyflow.vault.data.DeleteFilesOptions.interceptor": "attr",
+  "skyflow.vault.data.DeleteFilesRequest": "class (BaseDeleteFilesRequest)",
+  "skyflow.vault.data.DeleteFilesRequest.__init__": "def (self, records: List[DeleteFilesRequestRecord])",
+  "skyflow.vault.data.DeleteFilesRequestRecord": "class ()",
+  "skyflow.vault.data.DeleteFilesRequestRecord.__init__": "def (self, table_name: str, columns: List[str], skyflow_id: str = None, unique_values: List[dict] = None)",
+  "skyflow.vault.data.DeleteFilesRequestRecord.columns": "attr",
+  "skyflow.vault.data.DeleteFilesRequestRecord.skyflow_id": "attr",
+  "skyflow.vault.data.DeleteFilesRequestRecord.table_name": "attr",
+  "skyflow.vault.data.DeleteFilesRequestRecord.unique_values": "attr",
+  "skyflow.vault.data.DeleteFilesResponse": "class (BaseDeleteFilesResponse)",
+  "skyflow.vault.data.DeleteFilesResponse.__init__": "def (self, records = None)",
+  "skyflow.vault.data.DeleteFilesResponseRecord": "class ()",
+  "skyflow.vault.data.DeleteFilesResponseRecord.__init__": "def (self, skyflow_id: str = None, table_name: str = None, columns = None, http_code: int = None, error: str = None, request_id: str = None)",
+  "skyflow.vault.data.DeleteFilesResponseRecord.columns": "attr",
+  "skyflow.vault.data.DeleteFilesResponseRecord.error": "attr",
+  "skyflow.vault.data.DeleteFilesResponseRecord.http_code": "attr",
+  "skyflow.vault.data.DeleteFilesResponseRecord.request_id": "attr",
+  "skyflow.vault.data.DeleteFilesResponseRecord.skyflow_id": "attr",
+  "skyflow.vault.data.DeleteFilesResponseRecord.table_name": "attr",
   "skyflow.vault.data.DeleteOptions": "class ()",
   "skyflow.vault.data.DeleteOptions.__init__": "def (self, interceptor: Optional[Callable[[RequestContext], None]] = None)",
   "skyflow.vault.data.DeleteOptions.interceptor": "attr",
@@ -125,6 +158,25 @@
   "skyflow.vault.data.GetResponseRecord.skyflow_id": "attr",
   "skyflow.vault.data.GetResponseRecord.table_name": "attr",
   "skyflow.vault.data.GetResponseRecord.tokens": "attr",
+  "skyflow.vault.data.GetTokensOptions": "class ()",
+  "skyflow.vault.data.GetTokensOptions.__init__": "def (self, interceptor: Optional[Callable[[RequestContext], None]] = None)",
+  "skyflow.vault.data.GetTokensOptions.interceptor": "attr",
+  "skyflow.vault.data.GetTokensRequest": "class (BaseGetTokensRequest)",
+  "skyflow.vault.data.GetTokensRequest.__init__": "def (self, records: List[GetTokensRequestRecord])",
+  "skyflow.vault.data.GetTokensRequestRecord": "class ()",
+  "skyflow.vault.data.GetTokensRequestRecord.__init__": "def (self, value, token_group_name: str)",
+  "skyflow.vault.data.GetTokensRequestRecord.token_group_name": "attr",
+  "skyflow.vault.data.GetTokensRequestRecord.value": "attr",
+  "skyflow.vault.data.GetTokensResponse": "class (BaseGetTokensResponse)",
+  "skyflow.vault.data.GetTokensResponse.__init__": "def (self, records = None)",
+  "skyflow.vault.data.GetTokensResponseRecord": "class ()",
+  "skyflow.vault.data.GetTokensResponseRecord.__init__": "def (self, value = None, token_group_name = None, token = None, http_code = None, error = None, request_id = None)",
+  "skyflow.vault.data.GetTokensResponseRecord.error": "attr",
+  "skyflow.vault.data.GetTokensResponseRecord.http_code": "attr",
+  "skyflow.vault.data.GetTokensResponseRecord.request_id": "attr",
+  "skyflow.vault.data.GetTokensResponseRecord.token": "attr",
+  "skyflow.vault.data.GetTokensResponseRecord.token_group_name": "attr",
+  "skyflow.vault.data.GetTokensResponseRecord.value": "attr",
   "skyflow.vault.data.InsertOptions": "class ()",
   "skyflow.vault.data.InsertOptions.__init__": "def (self, interceptor: Optional[Callable[[RequestContext], None]] = None)",
   "skyflow.vault.data.InsertOptions.interceptor": "attr",
@@ -152,6 +204,21 @@
   "skyflow.vault.data.InsertResponseRecord.skyflow_id": "attr",
   "skyflow.vault.data.InsertResponseRecord.table_name": "attr",
   "skyflow.vault.data.InsertResponseRecord.tokens": "attr",
+  "skyflow.vault.data.QueryOptions": "class ()",
+  "skyflow.vault.data.QueryOptions.__init__": "def (self, interceptor: Optional[Callable[[RequestContext], None]] = None)",
+  "skyflow.vault.data.QueryOptions.interceptor": "attr",
+  "skyflow.vault.data.QueryRequest": "class (BaseQueryRequest)",
+  "skyflow.vault.data.QueryRequest.__init__": "def (self, query: str)",
+  "skyflow.vault.data.QueryResponse": "class (BaseQueryResponse)",
+  "skyflow.vault.data.QueryResponse.__init__": "def (self, records = None, metadata = None, request_id = None)",
+  "skyflow.vault.data.QueryResponse.metadata": "attr",
+  "skyflow.vault.data.QueryResponse.request_id": "attr",
+  "skyflow.vault.data.QueryResponseMetadata": "class ()",
+  "skyflow.vault.data.QueryResponseMetadata.__init__": "def (self, columns: list = None)",
+  "skyflow.vault.data.QueryResponseMetadata.columns": "attr",
+  "skyflow.vault.data.QueryResponseRecord": "class ()",
+  "skyflow.vault.data.QueryResponseRecord.__init__": "def (self, data: dict = None)",
+  "skyflow.vault.data.QueryResponseRecord.data": "attr",
   "skyflow.vault.data.RequestContext": "class ()",
   "skyflow.vault.data.RequestContext.__init__": "def (self, operation, batch_index = NOT_BATCHED, total_batches = NOT_BATCHED)",
   "skyflow.vault.data.RequestContext.add_header": "def (self, key, value)",
@@ -195,6 +262,40 @@
   "skyflow.vault.data.UpdateResponseRecord.skyflow_id": "attr",
   "skyflow.vault.data.UpdateResponseRecord.table_name": "attr",
   "skyflow.vault.data.UpdateResponseRecord.tokens": "attr",
+  "skyflow.vault.data.UploadFilesColumnResult": "class ()",
+  "skyflow.vault.data.UploadFilesColumnResult.__init__": "def (self, column: str, file_name: str = None, upload_status: str = None, error: str = None)",
+  "skyflow.vault.data.UploadFilesColumnResult.column": "attr",
+  "skyflow.vault.data.UploadFilesColumnResult.error": "attr",
+  "skyflow.vault.data.UploadFilesColumnResult.file_name": "attr",
+  "skyflow.vault.data.UploadFilesColumnResult.upload_status": "attr",
+  "skyflow.vault.data.UploadFilesOptions": "class ()",
+  "skyflow.vault.data.UploadFilesOptions.__init__": "def (self, interceptor: Optional[Callable[[RequestContext], None]] = None)",
+  "skyflow.vault.data.UploadFilesOptions.interceptor": "attr",
+  "skyflow.vault.data.UploadFilesRequest": "class (BaseUploadFilesRequest)",
+  "skyflow.vault.data.UploadFilesRequest.__init__": "def (self, records: List[UploadFilesRequestRecord])",
+  "skyflow.vault.data.UploadFilesRequestColumn": "class ()",
+  "skyflow.vault.data.UploadFilesRequestColumn.__init__": "def (self, column: str, file_path: str = None, base64: str = None, file_object: BinaryIO = None, file_name: str = None, content_type: str = None)",
+  "skyflow.vault.data.UploadFilesRequestColumn.base64": "attr",
+  "skyflow.vault.data.UploadFilesRequestColumn.column": "attr",
+  "skyflow.vault.data.UploadFilesRequestColumn.content_type": "attr",
+  "skyflow.vault.data.UploadFilesRequestColumn.file_name": "attr",
+  "skyflow.vault.data.UploadFilesRequestColumn.file_object": "attr",
+  "skyflow.vault.data.UploadFilesRequestColumn.file_path": "attr",
+  "skyflow.vault.data.UploadFilesRequestRecord": "class ()",
+  "skyflow.vault.data.UploadFilesRequestRecord.__init__": "def (self, table_name: str, columns: List[UploadFilesRequestColumn], skyflow_id: str = None)",
+  "skyflow.vault.data.UploadFilesRequestRecord.columns": "attr",
+  "skyflow.vault.data.UploadFilesRequestRecord.skyflow_id": "attr",
+  "skyflow.vault.data.UploadFilesRequestRecord.table_name": "attr",
+  "skyflow.vault.data.UploadFilesResponse": "class (BaseUploadFilesResponse)",
+  "skyflow.vault.data.UploadFilesResponse.__init__": "def (self, records = None)",
+  "skyflow.vault.data.UploadFilesResponseRecord": "class ()",
+  "skyflow.vault.data.UploadFilesResponseRecord.__init__": "def (self, skyflow_id: str = None, table_name: str = None, columns = None, http_code: int = None, error: str = None, request_id: str = None)",
+  "skyflow.vault.data.UploadFilesResponseRecord.columns": "attr",
+  "skyflow.vault.data.UploadFilesResponseRecord.error": "attr",
+  "skyflow.vault.data.UploadFilesResponseRecord.http_code": "attr",
+  "skyflow.vault.data.UploadFilesResponseRecord.request_id": "attr",
+  "skyflow.vault.data.UploadFilesResponseRecord.skyflow_id": "attr",
+  "skyflow.vault.data.UploadFilesResponseRecord.table_name": "attr",
   "skyflow.vault.data.UpsertOptions": "class ()",
   "skyflow.vault.data.UpsertOptions.__init__": "def (self, unique_columns: list = None, update_type: UpsertType = None)",
   "skyflow.vault.data.UpsertOptions.unique_columns": "attr",

Replace the magic 200/300 in upload_files' per-column PUT status check with
named HTTP_STATUS_OK / HTTP_STATUS_MULTIPLE_CHOICES constants, matching the
repo's no-magic-values lint rule.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

✅ Gitleaks Findings: No secrets detected. Safe to proceed!

…iles

Cover the four new operations across controller, validations, data classes,
and client accessors (85 new tests). Exercises happy paths, per-record error
handling, the two-phase upload orchestration (file_path/base64/file_object,
PUT success/failure/skip, content-type inference), and request/response mapping.
New source is ~98% covered, satisfying the codecov patch gate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

✅ Gitleaks Findings: No secrets detected. Safe to proceed!

@saileshwar-skyflow
saileshwar-skyflow changed the base branch from main to flowvault-release/26.10.0 October 6, 2026 10:25
@saileshwar-skyflow
saileshwar-skyflow merged commit e38eae2 into flowvault-release/26.10.0 Oct 6, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants