Skip to content

fix(overview_metrics): opt-in strict counts (ISSUE-123) #448

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dwolfson merged 1 commit into from
Oct 13, 2026
Merged

fix(overview_metrics): opt-in strict counts (ISSUE-123) #448

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 54 additions & 39 deletions PYEGERIA_ISSUES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,15 +148,14 @@ content was changed. The numbered section list above describes the

# Open Issues

**At a glance** — 6 open pyegeria items, 7 awaiting re-test, 6 open Egeria Server issues.
**At a glance** — 5 open pyegeria items, 7 awaiting re-test, 6 open Egeria Server issues.

**Open pyegeria items**
- ISSUE-140: The executor charts any dict with two or more top-level numbers, regardless of the result's state
- ISSUE-139: The executor renders no result envelope, and the Vega bar chart has no subtitle, so a not-established result has nowhere to say so on a chart
- ISSUE-138: Analytic params are JSON-only, and only parameters named `mgr`/`ce`/`expert`/`client` are bound as clients, so a routine taking a reader object can't be named by its own dotted path
- ISSUE-132: `DeleteElementRequestBody` serializes the cascade flag as `cascadeDelete`; Egeria's field is `cascadedDelete` and unknown fields are ignored, so no pyegeria cascade delete has ever cascaded
- ISSUE-127: `CollectionManager.delete_collection(cascade=True)` silently never sends `cascadeDelete` -- same shape as ISSUE-62
- ISSUE-123: `overview_metrics.count_elements` (and `_find`/`_element_count`) return `0`/`[]` on any failure — a failed measurement is indistinguishable from a real zero

**Awaiting re-test**
- ISSUE-125: `deleteEnumDef` answers 500 "unknown TypeDef" for an enum the server lists -- an enum added through the API cannot be removed
Expand Down Expand Up @@ -303,43 +302,6 @@ Agreement.

---

### ISSUE-123: `overview_metrics.count_elements` (and `_find`/`_element_count`) return `0`/`[]` on any failure — a failed measurement is indistinguishable from a real zero

**Layer:** Pyegeria · **Status:** open · **Found:** 2026-10-04, while
making the portal's Egeria Overview stop showing confident numbers during
an outage · **GitHub:** odpi/egeria-python#424

**Root cause, from reading `pyegeria/view/overview_metrics.py` (6.1.28):**

- `_find(mgr, body, page_size)` wraps `find_metadata_elements` in a bare
`except Exception` and returns `[]`, logging only at `debug`.
- `_element_count(mgr, body, as_of)` tries the native count; if it is
unsupported *or raises*, it falls back to `len(_find(...))` — so a failed
query becomes `0`.
- `count_elements(...)`'s docstring states "0 on any failure (never
raises)". The callers `people_counts`, `usage_context_counts` and
`counts_by_type` inherit the same behaviour.

**Effect:** a dashboard cannot tell "there are 0 elements" from "Egeria
timed out / returned 401 / 500". An outage or auth failure renders as a
confident `0`. A fallback count that returns exactly the page size is also
only a lower bound, not a total, and is not flagged.

**Proposed fix:** return `None` on failure (or add a strict variant, e.g.
`count_elements(..., strict=True)` / `try_count_elements`, that raises or
returns `None`), keeping today's behaviour behind an explicit opt-in if
existing callers rely on it. Apply the same to `_find`/`_element_count` and
propagate `None` through `people_counts`, `usage_context_counts` and
`counts_by_type`. Log failures above `debug`. Flag a capped fallback count
as a lower bound.

**Workaround in egeria-workspaces:** the portal's Overview handler calls
`find_metadata_elements` directly and returns `None` for failed or capped
queries, rendering "Not measured" instead of `0` (odpi/egeria-workspaces
#601, #607).

---

### ISSUE-125: `deleteEnumDef` answers 500 "unknown TypeDef" for an enum the server lists -- an enum added through the API cannot be removed

**Layer:** Egeria Server (not pyegeria) · **Status:** open; reported to the Egeria team (response 2026-10-05); Egeria fix merged to Egeria `main` 2026-10-06 (odpi/egeria#9363), not yet re-tested on a platform built from it · **Found:**
Expand Down Expand Up @@ -1739,6 +1701,59 @@ pm.delete_collection(guid, cascade=True)

---

### ISSUE-123: `overview_metrics.count_elements` (and `_find`/`_element_count`) return `0`/`[]` on any failure — a failed measurement is indistinguishable from a real zero

**Layer:** Pyegeria · **Status:** fixed 2026-10-12 (opt-in `strict=True`) · **Found:** 2026-10-04, while
making the portal's Egeria Overview stop showing confident numbers during
an outage · **GitHub:** odpi/egeria-python#424

**Root cause, from reading `pyegeria/view/overview_metrics.py` (6.1.28):**

- `_find(mgr, body, page_size)` wraps `find_metadata_elements` in a bare
`except Exception` and returns `[]`, logging only at `debug`.
- `_element_count(mgr, body, as_of)` tries the native count; if it is
unsupported *or raises*, it falls back to `len(_find(...))` — so a failed
query becomes `0`.
- `count_elements(...)`'s docstring states "0 on any failure (never
raises)". The callers `people_counts`, `usage_context_counts` and
`counts_by_type` inherit the same behaviour.

**Effect:** a dashboard cannot tell "there are 0 elements" from "Egeria
timed out / returned 401 / 500". An outage or auth failure renders as a
confident `0`. A fallback count that returns exactly the page size is also
only a lower bound, not a total, and is not flagged.

**Proposed fix:** return `None` on failure (or add a strict variant, e.g.
`count_elements(..., strict=True)` / `try_count_elements`, that raises or
returns `None`), keeping today's behaviour behind an explicit opt-in if
existing callers rely on it. Apply the same to `_find`/`_element_count` and
propagate `None` through `people_counts`, `usage_context_counts` and
`counts_by_type`. Log failures above `debug`. Flag a capped fallback count
as a lower bound.

**Workaround in egeria-workspaces:** the portal's Overview handler calls
`find_metadata_elements` directly and returns `None` for failed or capped
queries, rendering "Not measured" instead of `0` (odpi/egeria-workspaces
#601, #607).

**Fixed 2026-10-12 — opt-in strict variant (owner's choice):** the default
is unchanged, so existing callers still get `0`. Pass `strict=True` to
`count_elements`, `count_elements_by_property`, `counts_by_type`,
`people_counts` or `usage_context_counts` to get `None` whenever the count is
not known:
- the fallback `find_metadata_elements` query raised, or answered with
something that is neither a list nor pyegeria's miss string;
- the fallback returned a full page (`DEFAULT_CAP`), which is only a lower
bound.

A native count is trusted as is, even above the cap. A real zero (an empty
list, or the "No elements found" string) is still `0`. Each `None` logs a
warning saying why. `strict` is also declared on those five analytic-registry
entries, so a report spec can pass it. Test:
`tests/micro-tests/test_issue_123_strict_counts.py`.

---

### ISSUE-122: `AssetMaker.get_catalog_targets` / `get_catalog_target` send `metadataElementTypeName="CatalogTarget"` (a relationship type) — server rejects with OMAG-COMMON-400-019, surfaced as SERVER_ERROR_500

**Status: FIXED and released.** Merged in #421 (`f2532bbd`, 2026-10-04) and shipped in **pyegeria 6.1.28** (and 6.1.29); present in the portal image's installed copy. Re-verified read-only on 2026-10-05 against the rebuilt platform: `get_catalog_targets` on the JDBC cataloguer returns "No elements found" in JSON, DICT and MD with no server error (it always answered `SERVER_ERROR_500` before the fix). The non-empty case was verified live on 2026-10-04, including the second defect found then (a blank DICT/MD record for the single get) and the `CatalogTargetProperties` field additions.
Expand Down
13 changes: 11 additions & 2 deletions pyegeria/view/analytic_registry.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,12 @@ def _as_of(description: str = "ISO-8601 asOfTime for a point-in-time value (None
return AnalyticParam(name="as_of", type="str", description=description)


def _strict() -> AnalyticParam:
return AnalyticParam(name="strict", type="bool", default=False,
description="Return None instead of 0 when a count could not be measured "
"(query failed, or the fallback hit the page-size cap). ISSUE-123.")


# ── Built-ins -- every analytic function already proven live via the Overview
# dashboard's /api/overview/* endpoints (overview_handler.py), just not yet
# reachable from a Dr.Egeria report spec. ──────────────────────────────────
Expand All @@ -154,6 +160,7 @@ def _as_of(description: str = "ISO-8601 asOfTime for a point-in-time value (None
AnalyticParam(name="type_name", type="str",
description="Open metadata type to count, e.g. GlossaryTerm, DigitalProduct. Omit for any type."),
_as_of(),
_strict(),
AnalyticParam(name="classifications", type="list[str]",
description="Match ANY of these classification names."),
],
Expand All @@ -169,6 +176,7 @@ def _as_of(description: str = "ISO-8601 asOfTime for a point-in-time value (None
AnalyticParam(name="type_map", type="list[tuple[str, str]]", required=True,
description="(label, type_name) pairs to count, e.g. [('Data Stores', 'DataStore'), ...]."),
_as_of(),
_strict(),
],
),
"governed_coverage": AnalyticFunctionSpec(
Expand Down Expand Up @@ -225,7 +233,7 @@ def _as_of(description: str = "ISO-8601 asOfTime for a point-in-time value (None
returns="dict (persons, teams, organizations, itProfiles, communities)",
generic=False,
binding_note="Fixed to Person/Team/Organization/ITProfile/Community -- not a parameter.",
params=[_as_of()],
params=[_as_of(), _strict()],
),
"feedback_summary": AnalyticFunctionSpec(
name="feedback_summary",
Expand All @@ -246,7 +254,7 @@ def _as_of(description: str = "ISO-8601 asOfTime for a point-in-time value (None
returns="dict (informationSupplyChains, blueprints)",
generic=False,
binding_note="Fixed to InformationSupplyChain and SolutionBlueprint -- not a parameter.",
params=[_as_of()],
params=[_as_of(), _strict()],
),
"semantic_grounding": AnalyticFunctionSpec(
name="semantic_grounding",
Expand Down Expand Up @@ -371,6 +379,7 @@ def _as_of(description: str = "ISO-8601 asOfTime for a point-in-time value (None
AnalyticParam(name="property_value", type="str", required=True,
description="Value to match the property against, e.g. ACTIVE."),
_as_of(),
_strict(),
],
),
"contextualised_coverage": AnalyticFunctionSpec(
Expand Down
Loading
Loading