Skip to content

Fix logs --since and /proxy/logs?since= to filter by date and time - #1996

Open
waldekmastykarz wants to merge 8 commits into
dotnet:mainfrom
waldekmastykarz:symmetrical-tribble
Open

waldekmastykarz wants to merge 8 commits into
dotnet:mainfrom
waldekmastykarz:symmetrical-tribble

Conversation

@waldekmastykarz

@waldekmastykarz waldekmastykarz commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #1961

Problem

devproxy logs --since and GET /proxy/logs?since= assumed log lines start with [hh:mm:ss.fff] and combined that with DateTime.Today. Detached log lines don't have that prefix (e.g. info …, req ╭ GET …). Any line that didn't match was included, so since filtered nothing. For example, --since 2030-01-01T00:00:00 returned the whole log. Even when it did match, the date was ignored.

Fix

New LogSinceFilter (shared by the CLI command and the API controller) filters full timestamp instants:

  • Before the first timestamp in legacy logs, lines use the selected process's actual start time, which remains stable across hot reloads.
  • Request blocks (req ╭ … ╰, including the preceding blank line) use the full ISO timestamp from their time │ line.
  • Detached regular text messages include full UTC timestamps at every surviving log level, including Warning/Error-only logging. Buffered regular messages retain their emission timestamps.
  • Legacy debug/trace [HH:mm:ss.fffffff] prefixes (and [HH:mm:ss.fff]) are reconstructed against the previous timestamp, accounting for midnight, small out-of-order writes, and the repeated daylight-saving hour.
  • JSON log lines retain their timestamp instants.
  • Other lines use the most recent preceding timestamp.
  • Detached logs always retain timestamp metadata, even with showTimestamps: false. Interactive output continues to honor that setting and keeps its existing formatting.
  • Shared CLI/API cutoff parsing and comparisons use DateTimeOffset, preserving explicit offsets and correct ordering across both occurrences of the repeated daylight-saving hour.

CLI --follow and API SSE streaming share an incremental reader that filters by event time, retains timestamp reconstruction state, and buffers partial lines and incomplete request blocks across reads. Requests that start before the cutoff remain excluded even if they finish afterward.

Validation

  • All 491 solution tests pass, including regression coverage for process start after hot reload, buffered requests across reads, partial lines, JSON/text/request timestamps, midnight rollover, daylight-saving rollback and instant ordering, elapsed-time cutoff parsing, Warning-level timestamp metadata, and unchanged interactive output.
  • All 38 log-filtering, CLI log, and logging-configuration tests also pass with TZ=America/New_York.
  • Solution build succeeds without warnings or errors after integrating upstream main 94a9579 (Count intercepted requests for mock nth #1987).
  • Detached end-to-end checks with showTimestamps: false confirm CLI/API snapshot and follow filtering at Information and Warning levels. Newly emitted Error records remain visible after a post-start cutoff at Warning level; older records and future-cutoff output are excluded.

…loses dotnet#1961

Detached log lines don't start with a [hh:mm:ss.fff] timestamp, so the
previous filter matched no lines, included all of them and ignored the
date. Reconstruct each line's full date and time from the instance start
time, request blocks' 'time' lines, debug/trace time-of-day prefixes (with
midnight rollover) and JSON 'timestamp' properties, and filter on that.
The log line format is unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3b2ff54c-c6ab-4c98-a477-c522a6c02ac2
Copilot AI balanced review requested due to automatic review settings October 21, 2026 14:04
@waldekmastykarz
waldekmastykarz requested a review from a team as a code owner October 21, 2026 14:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Filtering remains incorrect after hot reload, with timestamps disabled, and for buffered requests in follow mode.

3 open issues
What changed in this PR

Addresses #1961 by sharing full-date log filtering between the CLI and management API without changing the log format.

Changes:

  • Reconstructs timestamps from request blocks, debug/trace prefixes, and JSON entries.
  • Integrates shared filtering into both log readers.
  • Adds tests for multi-day filtering, midnight rollover, and timestamp fallback.
File Description
DevProxy/​Logging/​LogSinceFilter.cs Implements shared timestamp reconstruction and filtering.
DevProxy/​Commands/​LogsCommand.cs Integrates filtering and changes follow-mode handling.
DevProxy/​ApiControllers/​ProxyController.cs Uses shared filtering with process start time.
DevProxy.Tests/​LogSinceFilterTests.cs Covers timestamp formats and date boundaries.

Review effort: Balanced

Comment thread DevProxy/Commands/LogsCommand.cs Outdated
Comment thread DevProxy/Logging/LogSinceFilter.cs
Comment thread DevProxy/Commands/LogsCommand.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Daylight-saving clock rollback can incorrectly date logs one day ahead and include them beyond the requested cutoff.

4 open issues

Review effort: Balanced

Comment thread DevProxy/Logging/LogSinceFilter.cs Outdated
Preserve detached timestamps, use process start across hot reloads, share event-time filtering for CLI/API follow mode, and handle DST rollback. Add regression coverage and document detached timestamp retention.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate upstream main c9a4a0b while preserving the reviewed log filtering fixes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Repeated-hour comparisons and missing timestamps in warning/error-only text logs can still produce incorrect filtering results.

2 open issues
4 resolved since last review

Review effort: Balanced

Comment thread DevProxy/Extensions/ILoggingBuilderExtensions.cs
Comment thread DevProxy/Logging/LogSinceFilter.cs
waldekmastykarz and others added 2 commits October 26, 2026 11:48
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Include full UTC timestamps on detached regular records at every log level, preserve DateTimeOffset instants through snapshot/follow filtering, and share offset-aware CLI/API cutoff parsing. Add repeated-hour and Warning-level regressions while retaining interactive formatting.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate upstream main b9865a6; complete solution build and all 494 tests pass.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Blank-label request terminators are not recognized, delaying filtered streaming output and misdating subsequent snapshot messages.

1 open issue
2 resolved since last review

Review effort: Balanced

Comment thread DevProxy/Logging/LogSinceFilter.cs Outdated
waldekmastykarz and others added 2 commits October 26, 2026 12:05
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Accept Normal request entries with an empty label before the closing corner. Add real-formatter regressions for snapshot timestamps and immediate plain/ANSI follow completion.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No unresolved blocking issues remain, and the supplied regression coverage addresses the timestamp and streaming failures reported previously.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

logs --since and /proxy/logs?since= compare only the time of day

2 participants