Skip to content

feat(sdmx): add SdmxEndpoint for SDMX 3.0 data and availability queries - #293

Open
dwnoble wants to merge 5 commits into
datacommonsorg:masterfrom
dwnoble:feat/sdmx-endpoint
Open

dwnoble wants to merge 5 commits into
datacommonsorg:masterfrom
dwnoble:feat/sdmx-endpoint

Conversation

@dwnoble

@dwnoble dwnoble commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Problem

The Data Commons API now serves SDMX 3.0 data and availability endpoints, but datacommons-client only covers the V2 endpoints (node, observation, resolve). We also need a way to pass custom request headers (such as Authorization bearer tokens) when connecting to private Data Commons instances.

Solution

  • Add SdmxEndpoint (client.sdmx): Adds fetch_data and fetch_availability methods to query /sdmx/v3/data/dataflow/DC/DF_OBS/1.0.0/* and /sdmx/v3/availability/dataflow/DC/DF_OBS/1.0.0/<component_id>, handling both /sdmx/v3 (public Data Commons) and /core/api/sdmx/v3 (custom DCP instances).
  • Support custom headers and optional instance validation: Adds headers and validate_instance parameters to DataCommonsClient and API so callers can pass pre-resolved URLs and custom auth headers directly.
  • Add SDMX exceptions: Adds SdmxClientError and SdmxAPIError (inheriting from APIError / DataCommonsError), along with parse_filters and build_query_params helpers.
  • Update documentation: Adds SDMX usage examples to datacommons_client/README.md.

Verification

  • All 235 unit tests pass (pytest datacommons_client/tests/).
  • Style and import checks pass (./run_test.sh -l).
  • Verified live SDMX fetch_data and fetch_availability queries against https://api.datacommons.org.
  • Verified live V2 node and SDMX fetch_data / fetch_availability queries against testbed-1 (datcom-dcp) using IAM OIDC bearer headers.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the library to version 2.2.0, introducing the SdmxEndpoint to query the Data Commons SDMX 3.0 REST APIs, and adding support for custom HTTP headers and optional instance validation skipping. The review feedback highlights two key issues: first, the api_key is not passed to resolve_instance_url when custom headers are absent, which can cause authenticated instance validation to fail; second, the constraint builder checks for Sequence but misses set types, leading to incorrect stringification of set-based constraints.

Comment thread datacommons_client/endpoints/base.py Outdated
Comment on lines +76 to +77
else:
self.base_url = resolve_instance_url(dc_instance)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

When headers is None but api_key is provided, the api_key is not passed to resolve_instance_url. This causes the custom instance validation to run without authentication, which will fail if the custom instance requires an API key. We should pass api_key to resolve_instance_url in the else block as well.

Suggested change
else:
self.base_url = resolve_instance_url(dc_instance)
else:
self.base_url = resolve_instance_url(dc_instance, api_key=api_key)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Passed api_key=api_key to resolve_instance_url in the else branch as well.

Comment thread datacommons_client/endpoints/sdmx.py Outdated
Comment on lines +85 to +88
if isinstance(value, str):
values = [value]
elif isinstance(value, Sequence):
values = list(value)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using isinstance(value, Sequence) to check for collection types will not match set (since set is not a subclass of Sequence in collections.abc). If a user passes a set of constraint values (e.g., {"country/USA", "country/CAN"}), it will fall back to the else block and be stringified as a single string "{'country/USA', 'country/CAN'}" instead of being joined by commas. We should explicitly support set alongside Sequence.

Suggested change
if isinstance(value, str):
values = [value]
elif isinstance(value, Sequence):
values = list(value)
if isinstance(value, str):
values = [value]
elif isinstance(value, (Sequence, set)):
values = list(value)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Added set support alongside Sequence in build_query_params.

@dwnoble
dwnoble marked this pull request as ready for review September 22, 2026 04:36
@dwnoble
dwnoble requested a review from clincoln8 September 22, 2026 04:36

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant