Skip to content

fix(common-utils): return timestamps for min/max on date and time columns - #3722

Merged
kodiakhq[bot] merged 5 commits into
mainfrom
brandon/mcp-timestamp-minmax
Nov 19, 2026
Merged

kodiakhq[bot] merged 5 commits into
mainfrom
brandon/mcp-timestamp-minmax

Conversation

@brandon-pereira

@brandon-pereira brandon-pereira commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Summary

min and max on a Date, DateTime, or DateTime64 column (e.g. Timestamp) returned 0 typed Float64. The renderer casts every aggregate's value with toFloat64OrDefault(toString(expr)), and a DateTime's string form doesn't parse as a float. MCP agents hit this with clickstack_table and worked around it by putting raw SQL in aggFn: "none", one of the paths that pushes them toward clickstack_sql.

min/max now skip the cast when the value is a bare column whose type, from the cached column metadata, is Date/DateTime/DateTime64. The result keeps the column type, and MCP serializes it as an ISO string like "2026-11-18T10:00:02.123456789Z".

The cast is kept everywhere else:

Case Behaviour
min/max of a date/time column in a table, number tile, or MCP table/number Fixed: returns the timestamp
Numeric columns, string attributes like SpanAttributes['x'], expressions, unknown columns Unchanged (cast)
Other aggregates (avg, sum, quantile, …) on a date column Unchanged (cast)
Line, bar, pie, heatmap, and any time-bucketed chart Unchanged (cast) on purpose
Ratios, formulas, and number tiles with several series Unchanged (cast) on purpose

Charts keep the cast because they need a numeric value: pie/bar drop non-numeric columns and would throw "No value columns found", and a time-bucketed result's chart formatter treats the first date column as the time bucket. They still show 0 for min/max of a timestamp, as on main. Ratios and formulas do arithmetic on the value, which ClickHouse rejects for dates, and a multi-series number tile shows the first numeric column, so a date series would be silently skipped.

Screenshots or video

Before After
Screenshot 2026-11-18 at 1 53 46 PM Screenshot 2026-11-18 at 1 54 13 PM

How to reproduce

Starting point: yarn app:dev:local connected to the demo server, or yarn dev.

On main:

  1. Open Chart Explorer (/chart), pick a Traces source, display type Table.
  2. Series 1: Minimum of Timestamp. Add series 2: Maximum of Timestamp. Group by ServiceName.
  3. Both columns show 0.

On this branch:

  1. Same steps: both columns show the first and last timestamp per service.
  2. Display type Number with Maximum of Timestamp shows the latest timestamp.
  3. Display type Line or Pie with the same series renders as on main (no error).

Link for ease

How to test on Vercel preview

Preview routes: /chart

Steps:

  1. Select a Traces source and set the display type to Table.
  2. Set the first series aggregation to Minimum with column Timestamp, add a second series with Maximum of Timestamp, and run the query.
  3. Verify both columns show date-time values, not 0.

Tests

New min/max on date and time columns block in renderChartConfig.test.ts: DateTime64, DateTime, Nullable(Date), backtick-quoted names, -If with an aggCondition, and sample-weighted rendering return the uncast column (these fail on main); numeric, map-attribute, unknown-column, avg, pie, bar, and time-bucketed cases keep the cast. Existing numeric min/max tests and snapshots are unchanged.

References

@brandon-pereira
brandon-pereira requested a review from a team as a code owner November 18, 2026 19:55
@vercel

vercel Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 19, 2026 4:10pm UTC
hyperdx-storybook Ready Ready Preview Nov 19, 2026 4:10pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 3c45a2b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the review/tier-2 Low risk — AI review + quick human skim label Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

🔵 Tier 2 — Low Risk

Small, isolated change with no API route or data model modifications.

Why this tier:

  • Standard feature/fix — introduces new logic or modifies core functionality

Additional context: touches the query rendering engine lightly (70 lines, under the 150-line bar for Tier 4)

Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns.
SLA: Resolve within 4 business hours.

Stats
  • Production files changed: 1
  • Production lines changed: 70 (+ 491 in test files, excluded from tier calculation)
  • Branch: brandon/mcp-timestamp-minmax
  • Author: brandon-pereira

To override this classification, remove the review/tier-2 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 107 in total (1.12%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 P0/P1 in code this PR adds or changes: fix before asking for review.
  • 🟡 P2 in code this PR changes: your call. Fix if small, otherwise reply.
  • 🔵 P3 is collapsed at the bottom: nobody needs to act on it before merge.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    priority. Reply with one sentence; if it is P0, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

2 finding(s): 🔴 0 P0/P1 · 🟡 2 P2 · 🔵 0 P3

2 posted as inline comment(s) on the changed lines.


Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

The change is small and well-scoped: a new isMinMaxOnDateTimeColumn() guard plus a keepValueType flag on aggFnExpr that skips the toFloat64OrDefault(toString(...)) cast for min/max on a bare Date/DateTime/DateTime64 column in Table / single-series Number / MCP paths, returning the real timestamp. The guard short-circuits cheaply before its async metadata.getColumn lookup, the lookup reuses an already-warmed, deduped column cache (no added ClickHouse round-trips in the common path), and the cast is preserved for charts, ratios, formulas, multi-series tiles, granularity, and CTEs.

✅ No critical issues found.

🟡 P2 — recommended

  • packages/common-utils/src/__tests__/minMaxDateTime.int.test.ts:40 — the new integration suite connects to a live ClickHouse (CLICKHOUSE_HOST, defaulting to http://localhost:8123) in beforeAll, so if it is not excluded from the default unit-test run it will fail wherever no ClickHouse is reachable.
    • Fix: confirm .int.test.ts is matched by a ClickHouse-gated job and ignored by the default yarn test path (e.g. a testPathIgnorePatterns / project split), and document the required CLICKHOUSE_* env for local runs.
  • packages/common-utils/src/core/renderChartConfig.ts:566 — on any metadata.getColumn failure the helper logs and returns false, silently falling back to the Float64 cast (date renders as 0) with no test covering that branch.
    • Fix: add a unit test that stubs getColumn to throw and asserts the cast is kept, locking in the intended safe-fallback behavior.
    • maintainability
🔵 P3 nitpicks (2)
  • packages/common-utils/src/core/renderChartConfig.ts:550 — the ten-condition || guard folds unrelated concerns (aggFn, displayType, seriesCount, seriesReturnType, formulas, granularity, CTE, db/table) into one boolean, forcing the reader to hold all cross-module semantics at once.
    • Fix: extract intent-named predicates (e.g. isDateDisplayCapable, doesArithmeticOnValue, hasResolvableBaseTable) so the guard reads as a few labeled checks.
    • maintainability
  • packages/common-utils/src/core/renderChartConfig.ts:559 — the hasSubqueryCte(chartConfig.with) || !databaseName || !tableName skip restates the "no resolvable base table" concept that appears with subtly different &&/|| shapes at three other sites (≈806, 1034, 1388), inviting drift.
    • Fix: introduce a single hasResolvableBaseTable(chartConfig) helper and reuse it, noting where the differing boolean shape is intentional.
    • maintainability

Reviewers (3): correctness/orchestrator analysis, maintainability, performance.

Testing gaps:

  • No test exercises the catch → console.warn → cast-fallback branch in isMinMaxOnDateTimeColumn.
  • No test independently locks the seriesCount > 1 Number-tile, granularity, and CTE skip branches against future refactors.
  • No test asserts that N min/max series on one table issue a single DESCRIBE (cache/pendingQueries dedup) rather than N.

Note: synthesized from the reviewers that had reported when output was required; the correctness, testing, api-contract, adversarial, security, TypeScript, previous-comments, agent-native, and learnings reviewers were dispatched but had not yet returned, so their findings are not reflected above.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 479 passed • 1 skipped • 1079s

Status Count
✅ Passed 479
❌ Failed 0
⚠️ Flaky 3
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@pulpdrew

Copy link
Copy Markdown
Contributor

Related to #3543?

pulpdrew
pulpdrew previously approved these changes Nov 18, 2026

@pulpdrew pulpdrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Nice tests! IMO they'd be even nicer if they were .int integration tests that hit clickhouse for real, instead

@brandon-pereira

Copy link
Copy Markdown
Member Author

@pulpdrew updated to link to issue, and added an integration test

const unsafeExpr = {
UNSAFE_RAW_SQL:
isAny || isNone ? `${expr}` : `toFloat64OrDefault(toString(${expr}))`,
isAny || isNone || keepValueType

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — Uncast min/max over zero matching rows returns 1970-01-01 epoch instead of an empty value

When keepValueType is set, render the -OrNull form (minOrNull/maxOrNull, minIfOrNull/maxIfOrNull) so no matching rows gives NULL. This applies to the plain, -If and sample-weighted min/max branches. Also add an integration case for a group with no rows matching the aggCondition. Right now minIf(Timestamp, cond)/max(Timestamp) over no rows returns the type default, 1970-01-01 00:00:00. That is easy to read as a real timestamp, where the old 0 was clearly empty. It is common in MCP: mergeWhereIntoSelectItems (packages/api/src/mcp/tools/query/table.ts:327) puts the top-level where into every select's aggCondition. So clickstack_table with groupBy ServiceName, where StatusCode:Error and max(Timestamp) reports "last error 1970-01-01T00:00:00Z" for services that have no errors. A number tile over an empty range shows the epoch too.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

);
} catch (e) {
console.warn(
`Column ${valueExpression} lookup failed while choosing min/max cast`,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — MV-optimized query with a SimpleAggregateFunction(min|max, DateTime) column still casts to 0

Unwrap SimpleAggregateFunction(<fn>, T) before calling classifyTimestampType, or have the MV rewrite carry the source column's type. Otherwise the same tile gives a timestamp on the base table and 0 when MV optimization applies. For SimpleAggregateFunction columns, getAggregateMergeFunction (packages/common-utils/src/core/materializedViews.ts:266-269) keeps aggFn as min/max and swaps valueExpression to the MV column. The column type is then SimpleAggregateFunction(max, DateTime64(9)), which stripTypeWrappers doesn't unwrap and classifyTimestampType rejects, so the cast comes back.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@kodiakhq
kodiakhq Bot merged commit e8d1d9a into main Nov 19, 2026
27 of 28 checks passed
@kodiakhq
kodiakhq Bot deleted the brandon/mcp-timestamp-minmax branch November 19, 2026 16:14

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 3c45a2bf Deployed Nov 19, 2026 by vercel[bot]
Preview – hyperdx-storybook — 3c45a2bf Deployed Nov 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge review/tier-2 Low risk — AI review + quick human skim

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants