feat: P2 enhancements — filtering, column selection, untyped data, performance

docs/examples/exploring-cbs-data.md

@@ -85,7 +85,7 @@ List the available columns:
 ```python
 for col in meta.properties[:10]:
     unit = f" ({col.unit})" if col.unit else ""
-    print(f"  {col.display_name}{unit}")
+    print(f"  {col.name}{unit}")
 ```
 
 ```

docs/examples/period-codes.md

@@ -9,6 +9,7 @@ CBS encodes time periods as strings with a specific format:
 | Pattern | Example | Decoded |
 |---------|---------|---------|
 | `YYYYJJnn` | `2023JJ00` | `2023` (yearly) |
+| `YYYYHJnn` | `2023HJ01` | `2023 H1` (half-yearly) |
 | `YYYYKWnn` | `2023KW01` | `2023 Q1` (quarterly) |
 | `YYYYMMnn` | `2023MM03` | `2023 March` (monthly) |
 
@@ -25,6 +26,7 @@ _MONTHS = [
     "July", "August", "September", "October", "November", "December",
 ]
 _YEARLY = re.compile(r"^(\d{4})JJ00$")
+_HALFYEARLY = re.compile(r"^(\d{4})HJ0([12])$")
 _QUARTERLY = re.compile(r"^(\d{4})KW0([1-4])$")
 _MONTHLY = re.compile(r"^(\d{4})MM(\d{2})$")
 
@@ -33,6 +35,8 @@ def decode_period(raw: str) -> str:
     s = raw.strip()
     if m := _YEARLY.match(s):
         return m.group(1)
+    if m := _HALFYEARLY.match(s):
+        return f"{m.group(1)} H{m.group(2)}"
     if m := _QUARTERLY.match(s):
         return f"{m.group(1)} Q{m.group(2)}"
     if m := _MONTHLY.match(s):
@@ -54,6 +58,7 @@ mo.md(f"**Input:** `{period_input.value}` **→** **Output:** `{decoded}`")
 Try these examples:
 
 - `2023JJ00` (yearly)
+- `2023HJ01` (half-yearly -- H1)
 - `2023KW03` (quarterly -- Q3)
 - `2023MM12` (monthly -- December)
 - `1990JJ00` (older data)
@@ -65,6 +70,7 @@ Here is a batch of period codes and their decoded values:
 ```python {marimo}
 examples = [
     "2023JJ00", "2022JJ00", "2021JJ00",
+    "2023HJ01", "2023HJ02",
     "2023KW01", "2023KW02", "2023KW03", "2023KW04",
     "2023MM01", "2023MM06", "2023MM12",
     "unknown_format",
@@ -78,8 +84,9 @@ mo.ui.table(rows, selection=None)
 cbspy uses three regular expressions to match period codes:
 
 1. **Yearly:** `^\d{4}JJ00$` -- matches codes like `2023JJ00`. Extracts the four-digit year.
-2. **Quarterly:** `^\d{4}KW0[1-4]$` -- matches codes like `2023KW01`. Extracts year and quarter number.
-3. **Monthly:** `^\d{4}MM\d{2}$` -- matches codes like `2023MM03`. Extracts year and month number, then maps to the month name.
+2. **Half-yearly:** `^\d{4}HJ0[12]$` -- matches codes like `2023HJ01`. Extracts year and half number.
+3. **Quarterly:** `^\d{4}KW0[1-4]$` -- matches codes like `2023KW01`. Extracts year and quarter number.
+4. **Monthly:** `^\d{4}MM\d{2}$` -- matches codes like `2023MM03`. Extracts year and month number, then maps to the month name.
 
 If none of the patterns match, the original string is returned unchanged. This ensures that unexpected formats pass through without raising errors.
 

docs/examples/population-data.md

@@ -104,6 +104,7 @@ CBS uses specific period code formats. Pass them directly to `periods`:
 | Format | Example | Meaning |
 |--------|---------|---------|
 | Yearly | `2023JJ00` | Year 2023 |
+| Half-yearly | `2023HJ01` | 2023 H1 |
 | Quarterly | `2023KW01` | 2023 Q1 |
 | Monthly | `2023MM03` | 2023 March |
 

docs/getting-started.md

@@ -56,7 +56,7 @@ print(f"Period: {meta.period}")
 print()
 
 for col in meta.properties:
-    print(f"  {col.display_name} ({col.unit})")
+    print(f"  {col.name} ({col.unit})")
 ```
 
 ```
@@ -70,7 +70,7 @@ Period: 1950 - 2023
   ...
 ```
 
-Each `Column` object has `id`, `name`, `dutch_name`, `unit`, `datatype`, `description`, and a `display_name` computed property that prefers the English name.
+Each `Column` object has `id`, `name`, `unit`, `datatype`, and `description`.
 
 ## Fetching data
 
@@ -95,6 +95,46 @@ shape: (2, 28)
 
 Period codes like `2022JJ00` are automatically decoded to `2022`. Quarterly periods (`2023KW01`) become `2023 Q1`, and monthly periods (`2023MM03`) become `2023 March`.
 
+## Filtering by dimension
+
+By default, `get_data()` downloads the entire table. Use `periods` to filter by time, or `filters` to filter on any dimension:
+
+```python
+# Filter by region and period
+df = client.get_data(
+    "71450ned",
+    filters={"RegioS": ["GM0363", "GM0599"]},
+    periods=["2023JJ00"],
+)
+```
+
+You can also pass a raw OData filter string:
+
+```python
+df = client.get_data("71450ned", filters="RegioS eq 'GM0363'")
+```
+
+## Selecting columns
+
+Many CBS tables have dozens of columns. Use `columns` to fetch only what you need:
+
+```python
+df = client.get_data(
+    "37296eng",
+    columns=["Periods", "Total population", "Males"],
+)
+```
+
+You can use either human-readable column names or CBS internal keys.
+
+## Preserving statistical symbols
+
+By default, CBS replaces missing-data symbols with `null`. To preserve the original symbols (`.` = not applicable, `x` = suppressed, `-` = nil), pass `typed=False`:
+
+```python
+df = client.get_data("37296eng", typed=False)
+```
+
 ## Error handling
 
 cbspy raises specific exceptions for common problems:

reports/audit-2026-02-21.md

@@ -51,44 +51,37 @@ The largest opportunity: cbspy currently exposes only raw dimension codes (e.g.
 
 ## Priority 2 — High-Value Data Science Enhancements
 
-### 2.1 Dimension code resolution (labels for region, gender, age, etc.)
-
-| | |
-|---|---|
-| **Why it matters** | This is the single biggest usability gap. When a user calls `get_data("71450ned")`, dimension columns contain opaque codes like `"T001038"` (gender) or `"GM0363    "` (municipality). To get "Amsterdam" or "Male", users must manually discover dimension endpoints, fetch them, strip trailing spaces from keys, and join. The existing `cbsodata` library does this automatically; cbspy does not. Every data scientist hitting CBS data hits this wall. |
-| **Action** | (a) Add `Client.get_dimension(table_id, dimension_name) -> pl.DataFrame` to fetch and expose dimension code→label mappings. (b) Add a `resolve_labels=True` parameter to `get_data()` that auto-fetches all dimension endpoints and replaces codes with titles. (c) Strip trailing whitespace from dimension keys (CBS pads them to fixed width). |
-
-### 2.2 Filtering on any dimension, not just periods
+### 2.1 Filtering on any dimension, not just periods
 
 | | |
 |---|---|
 | **File** | `src/cbspy/client.py:76-95` |
 | **Why it matters** | `get_data()` only accepts a `periods` filter. Data scientists routinely need to filter by region (`RegioS eq 'GM0363'`), gender, age group, etc. Without server-side filtering, users must download entire tables (potentially millions of rows) and filter client-side. |
 | **Action** | Add a `filters: dict[str, list[str]] | str | None` parameter. Accept either a raw OData `$filter` string or a dict like `{"RegioS": ["GM0363", "GM0599"], "Perioden": ["2023JJ00"]}` that gets compiled to OData syntax. |
 
-### 2.3 Column selection (`$select`)
+### 2.2 Column selection (`$select`)
 
 | | |
 |---|---|
 | **Why it matters** | Many CBS tables have 50+ topic columns. A population researcher may only need 3. Without `$select`, the full payload is transferred and parsed. For large tables this wastes bandwidth and memory. |
 | **Action** | Add a `columns: list[str] | None` parameter to `get_data()`. Map human-readable names back to CBS keys, then pass as `$select`. |
 
-### 2.4 Switch from ODataApi to ODataFeed
+### 2.3 Switch from ODataApi to ODataFeed
 
 | | |
 |---|---|
 | **File** | `src/cbspy/_odata.py:9` |
 | **Why it matters** | cbspy uses `/ODataApi/odata/` which returns max 10,000 rows per page. CBS also provides `/ODataFeed/OData/` with no page-size limit — same endpoints, same params, fewer round-trips. For a 500K-row table, ODataApi requires 50+ HTTP requests; ODataFeed may need 1-5. `cbsodata` uses ODataFeed by default. |
 | **Action** | Change `_ODATA_API = "/ODataFeed/OData"`. This is a one-line change with large performance impact. Add an option to fall back to ODataApi if needed. |
 
-### 2.5 `UntypedDataSet` support
+### 2.4 `UntypedDataSet` support
 
 | | |
 |---|---|
 | **Why it matters** | `TypedDataSet` replaces CBS statistical symbols with `null`, losing information about *why* data is missing (`.` = not applicable, `x` = suppressed, `-` = nil). Economists doing data quality analysis or imputation need these symbols. |
 | **Action** | Add `typed: bool = True` parameter to `get_data()`. When `False`, fetch `UntypedDataSet` and return string columns preserving symbols. |
 
-### 2.6 Period decoder missing half-yearly format
+### 2.5 Period decoder missing half-yearly format
 
 | | |
 |---|---|
@@ -239,12 +232,11 @@ The largest opportunity: cbspy currently exposes only raw dimension codes (e.g.
 | 1.2 | Add context manager to `Client` | P1 | Bug | Small |
 | 1.3 | Fix or remove broken Dockerfile | P1 | Bug | Trivial |
 | 1.4 | Implement retry logic for 5xx/network errors | P1 | Bug | Medium |
-| 2.1 | Dimension code→label resolution | P2 | Enhancement | Large |
-| 2.2 | Filter on any dimension | P2 | Enhancement | Medium |
-| 2.3 | `$select` column subsetting | P2 | Enhancement | Small |
-| 2.4 | Switch to ODataFeed for larger page size | P2 | Enhancement | Trivial |
-| 2.5 | UntypedDataSet support | P2 | Enhancement | Medium |
-| 2.6 | Half-yearly period format | P2 | Bug | Trivial |
+| 2.1 | Filter on any dimension | P2 | Enhancement | Medium |
+| 2.2 | `$select` column subsetting | P2 | Enhancement | Small |
+| 2.3 | Switch to ODataFeed for larger page size | P2 | Enhancement | Trivial |
+| 2.4 | UntypedDataSet support | P2 | Enhancement | Medium |
+| 2.5 | Half-yearly period format | P2 | Bug | Trivial |
 | 3.1 | PyPI trusted publishing (OIDC) | P3 | CI/CD | Small |
 | 3.2 | Dynamic versioning with hatch-vcs | P3 | CI/CD | Medium |
 | 3.3 | Add Dependabot config | P3 | CI/CD | Trivial |

src/cbspy/_odata.py

@@ -7,7 +7,7 @@
 
 from cbspy.exceptions import APIError, TableNotFoundError
 
-_ODATA_API = "/ODataApi/odata"
+_ODATA_API = "/ODataFeed/OData"
 _CATALOG = "/ODataCatalog/Tables"
 
 _MAX_RETRIES = 1

src/cbspy/_periods.py

@@ -16,6 +16,7 @@
 ]
 
 _YEARLY = re.compile(r"^(\d{4})JJ00$")
+_HALFYEARLY = re.compile(r"^(\d{4})HJ0([12])$")
 _QUARTERLY = re.compile(r"^(\d{4})KW0([1-4])$")
 _MONTHLY = re.compile(r"^(\d{4})MM(\d{2})$")
 
@@ -25,6 +26,7 @@ def decode_period(raw: str) -> str:
 
     Examples:
         2023JJ00 -> "2023"
+        2023HJ01 -> "2023 H1"
         2023KW01 -> "2023 Q1"
         2023MM03 -> "2023 March"
     """
@@ -33,6 +35,9 @@ def decode_period(raw: str) -> str:
     if m := _YEARLY.match(s):
         return m.group(1)
 
+    if m := _HALFYEARLY.match(s):
+        return f"{m.group(1)} H{m.group(2)}"
+
     if m := _QUARTERLY.match(s):
         return f"{m.group(1)} Q{m.group(2)}"
 

src/cbspy/client.py

@@ -85,12 +85,28 @@ def get_metadata(self, table_id: str) -> TableMetadata:
             properties=columns,
         )
 
-    def get_data(self, table_id: str, periods: list[str] | None = None) -> pl.DataFrame:
+    def get_data(
+        self,
+        table_id: str,
+        periods: list[str] | None = None,
+        filters: dict[str, list[str]] | str | None = None,
+        columns: list[str] | None = None,
+        typed: bool = True,
+    ) -> pl.DataFrame:
         """Fetch dataset as a Polars DataFrame with human-readable column names.
 
         Args:
             table_id: CBS table identifier (e.g. "37296eng").
             periods: Optional list of CBS period codes to filter by.
+            filters: Optional dimension filters. Either a raw OData $filter string
+                or a dict mapping dimension keys to value lists, e.g.
+                ``{"RegioS": ["GM0363", "GM0599"]}``.
+            columns: Optional list of columns to retrieve. Accepts human-readable
+                names (e.g. "Total population") or CBS keys (e.g. "TotalPopulation_1").
+                Fetches all columns if None.
+            typed: If True (default), fetch TypedDataSet where CBS replaces statistical
+                symbols with null. If False, fetch UntypedDataSet preserving symbols
+                like "." (not applicable), "x" (suppressed), "-" (nil).
 
         Returns:
             Polars DataFrame with resolved column names and decoded periods.
@@ -99,16 +115,22 @@ def get_data(self, table_id: str, periods: list[str] | None = None) -> pl.DataFr
         column_map = {p["Key"]: p.get("Title", p["Key"]) for p in prop_rows}
         period_keys = {p["Key"] for p in prop_rows if p.get("Type") == "TimeDimension"}
 
-        params: dict[str, str] | None = None
-        if periods:
-            period_filter = " or ".join(f"Periods eq '{p}'" for p in periods)
-            params = {"$filter": period_filter}
+        filter_str = self._build_filter(periods, filters, period_keys)
+        params: dict[str, str] = {}
+        if filter_str:
+            params["$filter"] = filter_str
+
+        if columns is not None:
+            title_to_key = {v: k for k, v in column_map.items()}
+            select_keys = [title_to_key.get(c, c) for c in columns]
+            params["$select"] = ",".join(select_keys)
 
-        data_rows = self._odata.get_json(table_id, "TypedDataSet", params=params)
+        resource = "TypedDataSet" if typed else "UntypedDataSet"
+        data_rows = self._odata.get_json(table_id, resource, params=params or None)
 
         if not data_rows:
-            columns = {column_map.get(k, k): [] for k in column_map}
-            return pl.DataFrame(columns)
+            empty_cols = {column_map.get(k, k): [] for k in column_map}
+            return pl.DataFrame(empty_cols)
 
         renamed_rows = []
         for row in data_rows:
@@ -124,6 +146,29 @@ def get_data(self, table_id: str, periods: list[str] | None = None) -> pl.DataFr
 
         return pl.DataFrame(renamed_rows)
 
+    @staticmethod
+    def _build_filter(
+        periods: list[str] | None,
+        filters: dict[str, list[str]] | str | None,
+        period_keys: set[str],
+    ) -> str:
+        """Compile periods and filters into an OData $filter string."""
+        parts: list[str] = []
+
+        if periods and period_keys:
+            period_key = next(iter(period_keys))
+            clause = " or ".join(f"{period_key} eq '{p}'" for p in periods)
+            parts.append(f"({clause})")
+
+        if isinstance(filters, str):
+            parts.append(filters)
+        elif isinstance(filters, dict):
+            for dim, values in filters.items():
+                clause = " or ".join(f"{dim} eq '{v}'" for v in values)
+                parts.append(f"({clause})")
+
+        return " and ".join(parts)
+
     @staticmethod
     def _parse_column(prop: dict[str, Any]) -> Column:
         """Convert a raw DataProperties entry to a Column model."""