Skip to content

docs(debug-journal): compact the journal from 730KB to 126KB and tighten what entries contain - #5988

Merged
springfall2008 merged 1 commit into
mainfrom
docs/compact-debug-journal
Nov 5, 2026
Merged

springfall2008 merged 1 commit into
mainfrom
docs/compact-debug-journal

Conversation

@springfall2008

Copy link
Copy Markdown
Owner

Why

Every triage, PR-review and journal-update run reads tools/debug-journal.md end to end before it starts work, and the file had grown to 730KB. Most of that sat in table cells of 5–62KB, each on a single line, and the Read tool cuts a line off at 2000 characters. Runs therefore paged through the file with awk and grep:

  • On 2026-11-04 the nightly flush spent 80 of its 201 tool calls just reading the journal.
  • The flush pushed nothing on 2026-11-04 or 2026-11-05, and 99 findings are now queued.

#5981 raised the turn limits to buy time. This PR fixes the cause.

What changed

The journal (730KB, 379 lines → 126KB, 893 lines; longest line 460 characters):

  • Per-integration notes: each table row is now a ### heading, followed by one durable fact or "look here first" pointer per bullet.
  • Symptom → first place to look: grouped under eight topic headings. Where a symptom belongs to one integration, the bullet points to that area's notes instead of repeating them. Log lines are kept verbatim so they can still be searched.
  • Traps and the opening technique sections: kept, with the stories around them removed.
  • Dropped: per-issue narrative (what was tried, rejected fixes, probe transcripts, reporter-thread detail), every file.py:NNN citation, and bugs fixed before 2026-09-24 that are no longer traps. Fixes from after that date are cut to one line ("fixed in PR #N (date)") for anyone holding an older log.
  • Headings: every ## heading other links rely on (Symptom → first place to look, Traps when investigating, and so on) is unchanged.

The last uncompacted version is git show ea6ba38d:tools/debug-journal.md, and the journal's header says so.

Claims re-checked while compacting: every "open", "still live", "unmerged" or "draft" claim that could be checked cheaply was checked with gh or a grep of main, and updated. For example:

Where a claim couldn't be checked cheaply, it stays marked as unverified, with its date. Nothing marked suspected has been upgraded to fact, and no new claims were added.

Guidance, so the file stays small:

  • The journal's rules: an entry is now a symptom, the fact or mechanism behind it, and where to start (a function or config item). There are no file paths or line numbers. Naming a file is still fine when the file is the subject (apps.yaml, a template, a test to run). The rules also give size limits: a few hundred characters per bullet, and consolidate an area once it passes about 3KB.
  • JOURNAL_CAPTURE_PROMPT in tools/triage_daemon.py: this prompt tells every bot run what to put in the queue, and now asks for the same shape. A new test pins this.
  • .claude/skills/journal-update/SKILL.md: findings now go in as bullets under the right heading, any file and line citations a finding carries are removed, bugs fixed more than about six weeks ago are retired, and the file is consolidated once it passes about 150KB.

How it was done

The original was split into nine slices. Each was distilled in parallel against the journal's own rules, with the state checks above. A final pass over the whole file then removed duplication across slices, which no single slice could see. After that I restored the Sigenergy 1201 throttle, HTTP 403 and ratedEnergy units facts, which the cuts had lost and a symptom bullet pointed to; I re-checked them against sigenergy.py.

Detail lost on purpose: to reach the size target, some secondary detail went beyond pure duplicates, such as test-fixture asides, a few pre-fix log signatures, and some Octopus URL notes. The PRs and issues they cite still hold that detail.

Testing

  • python3 tools/test_triage_daemon.py: 341 tests pass, including the new prompt test.
  • Pre-commit passes on all four files (cspell, markdownlint, black, ruff).

Merge notes

🤖 Generated with Claude Code

…ten what entries contain

Every triage, review and journal-update run reads the journal end to end. At
730KB, with table cells of up to 62KB on a single line (the Read tool cuts
lines off at 2000 characters), it was costing those runs most of their turns,
and the nightly flush pushed nothing on 2026-11-04 or 2026-11-05.

- Per-integration notes: one ### heading per area, one durable fact per bullet.
- Symptoms: grouped under eight topic headings, pointing to the area notes
  rather than repeating them.
- Dropped: per-issue narrative, file:line citations, and bugs fixed before
  2026-09-24 that are no longer traps. Recent fixes are cut to one line.
- Claims checked against main and GitHub while compacting: merged or closed
  PRs and issues updated, and a stale constant corrected.

Guidance: an entry is a symptom, the fact behind it and where to start, with
no file paths or line numbers. JOURNAL_CAPTURE_PROMPT asks for the same
shape, so queued candidates arrive that way. The journal-update skill now
folds candidates in as bullets and consolidates when the file passes about
150KB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the BOT_REVIEW Trigger an autotriage label Nov 5, 2026

@springfall2008 springfall2008 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated comment from the triage bot.

Reviewed at d35f836e2990e731e4bc2e1ecd00440c0c4208a5. No issues found — this is a read-only editorial change (docs + one prompt string + one pinning test), and every claim I could check cheaply verified.

What I checked:

  • Base pointer. tools/debug-journal.md is byte-identical between ea6ba38d and main's tip 4f602104 (this PR's parent), so the header's git show ea6ba38d:tools/debug-journal.md escape hatch points at the true last uncompacted version.
  • State claims sampled against git/GitHub. Merged-PR citations #5629/#5625/#5801/#5736/#5864/#5977/#5838/#5841/#5844/#5845/#5867 all appear in main's history; #5670/#4645/#5636/#5758 are CLOSED with no merge commit; "open" citations #5591/#5363/#5909/#5767/#5729/#5666/#5677/#5698/#5704/#5358 are all OPEN. All match the new text.
  • Code claims sampled against this tree (~20). low_power_pv_threshold_w is the current name with default 150 (config.py), LOW_POWER_PV_THRESHOLD is gone; set_battery_charging_time() really has no production caller; the execute.py hold gate soc_percent >= inv_target_soc_percent has no tolerance and the - 1.0 hysteresis nested inside it is dead as stated; calculate_yesterday() really does fake minutes_now = 0 and rewind midnight_utc on the shared instance; the set_arg(key, None) clobber is still on main for pause_mode/pause_start_time/pause_end_time/discharge_target_soc/givtcp_rest; KRAKEN_REST_RATES_UNAVAILABLE_STATUSES is 400/404/410 with standing-charge fallback on 404/410 only; get_state(refresh=True) only re-reads when not self.ha_key; a sensor.* control write goes through set_state_wrapper() (no integration write); transient_ok covers only car-charging items; Sigenergy's restored facts (1201 throttle, 403 → not onboarded, ratedEnergy kWh-vs-Ah threshold) all match sigenergy.py, as does fetch_controls() running only under if first:; gateway.py returns before _inject_entities() on an empty inverter list unless gateway_inverter_serial is set; web_annual.py's _arg() passes no required_unit; is_alive()'s staleness is 60 minutes; test_time_loop_guard.py tests only ha_interface is None; Fox errno 42015/44096 marking and the IOG-SMB-only six-hour cap both verify.
  • Retention of dropped entries. The rewrite dropped 227 citation numbers; I sampled 13 of the recent ones and each was either retained in substance elsewhere (e.g. the power_ceiling ratchet, the held-dispatch bridge, the CPython-3.x CI trap, the recorder default-deny trap, interpolate=False), a legitimately retired pre-cutoff fix, or a PR-review cross-reference. Everything from 2026-10-28 onward is retained; the dropped set tops out at #5833 (closed, its general lesson is kept under "fixture encodes the bug").
  • Consistency of the three guidance surfaces. The journal's own rules, journal-update/SKILL.md and JOURNAL_CAPTURE_PROMPT now all describe the same entry shape and retire/size policy; the new test pins both prompt sentences and would fail on revert; longest new line is 460 characters, inside the 2000-character limit the skill now states; the skill's "read the journal end to end" step stays feasible at 126KB / 893 lines.
  • Merge note. All nine PRs named as journal-conflicting (#5905, #5900, #5715, #5632, #5620, #5498, #5363, #5287, #5272) do touch tools/debug-journal.md (verified via the API file lists), so the note is accurate.

@springfall2008

Copy link
Copy Markdown
Owner Author

Automated review failed to complete for this PR. Reason: Claude ran out of turns before finishing (--max-turns) (exit status 1). Failed at: triage_daemon.py:1962 in review_pr() (daemon 063549db). Log: pr-5988-review.log in the bot's logs directory. Not retrying automatically; remove BOT_FAILED and re-add BOT_REVIEW to try again.

@springfall2008 springfall2008 added BOT_FAILED An automated triage/PR attempt failed; see the comment for details and removed BOT_REVIEW Trigger an autotriage labels Nov 5, 2026
@springfall2008
springfall2008 enabled auto-merge (squash) November 5, 2026 19:00
@springfall2008
springfall2008 merged commit ff6eccf into main Nov 5, 2026
9 checks passed
@springfall2008
springfall2008 deleted the docs/compact-debug-journal branch November 5, 2026 19:00
springfall2008 pushed a commit that referenced this pull request Nov 5, 2026
Brings #5591 up to date. The PR does not touch the debug journal, so main's compacted format from #5988 is taken unchanged.
No conflicts: apps/predbat/fetch.py auto-merged; the new replay lines still read rate_*_saving_minutes, rate_*_pre_saving and io_adjusted, which main keeps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
springfall2008 pushed a commit to thewhale21/batpred that referenced this pull request Nov 5, 2026
Brings springfall2008#5632 up to date. The debug journal entry is re-applied in the compacted format from springfall2008#5988.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
springfall2008 pushed a commit that referenced this pull request Nov 5, 2026
Brings #5715 up to date. The debug journal entry is re-applied in the compacted format from #5988.
apps/predbat/tests/test_hanchu_api.py: import conflict, kept main's ComponentBase import (for the real set_arg_auto in MockHanchu) and the PR's battery_health_lines/encrypt_payload imports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
springfall2008 pushed a commit that referenced this pull request Nov 5, 2026
Brings #5287 up to date. The debug journal entry is re-applied in the compacted format from #5988.
apps/predbat/components.py: kept both new registry entries, main's hanchu and this PR's sungrow (already registered by "sungrow.SungrowAPI" path, so no change needed for load_component_class()).
apps/predbat/unit_test.py: kept both test registrations, main's hanchu_* and this PR's sungrow_*.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
springfall2008 pushed a commit that referenced this pull request Nov 5, 2026
Brings #5900 up to date. The debug journal entry is re-applied in the compacted format from #5988.
apps/predbat/wallbox.py: run() keeps the PR's poll-every-cycle-while-a-force-hold-is-busy condition and main's refresh_discovery() on every cycle.
apps/predbat/tests/test_wallbox.py: both sides' new tests kept (force hold, then discovery), and all of them registered in test_wallbox().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
springfall2008 pushed a commit to dragon2611/batpred that referenced this pull request Nov 5, 2026
Brings springfall2008#5272 up to date. The debug journal entry is re-applied in the compacted format from springfall2008#5988.
docs/components.md: kept the PR's plain-language AlphaESS discharge-floor bullet and folded in main's 10% batUseCap floor and set_reserve_min advice (springfall2008#5838).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
springfall2008 pushed a commit that referenced this pull request Nov 5, 2026
Brings #5620 up to date. The debug journal entry is re-applied in the compacted format from #5988.
apps/predbat/gateway.py: kept the PR's inverter_data_error()/is_alive(); main's only change there was a comment on the old offline-is-alive branch, which the PR replaces.
apps/predbat/tests/test_gateway.py: imported TestGatewayFreshness alongside main's TestTokenRefreshProvider; stubbed inverter_data_error in main's new TestCheckCommandedSerials; gave TestGatewayOnlineTopic a fresh battery and made its silent-hub test expect an offline hub to be not alive, while still checking that housekeeping records no success.
apps/predbat/tests/test_gateway_freshness.py, predbat.py: the freshness check now sits in main's plan_once(), so the main-cycle test binds plan_once and the early exit returns None.
apps/predbat/tests/test_solis_handover.py: binds check_inverter_data_freshness on main's executor stub, which calls execute_plan().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
springfall2008 pushed a commit to arturonaredo/batpred that referenced this pull request Nov 5, 2026
Brings springfall2008#5498 up to date. The debug journal entry is re-applied in the compacted format from springfall2008#5988.
.cspell/custom-dictionary-workspace.txt: union of both sides (rebasable, rebooked), sorted.
apps/predbat/unit_test.py: kept both sides' imports and TEST_REGISTRY entries (car_manual_soc_planning and import_limit from the PR; save_restore_options_per_engine, export_threshold_per_step and export_gate_nominal from main).
apps/predbat/tests/test_solis_handover.py: main's new executor fixture gains args={}, because the PR's execute_plan() now reads self.args via execute_car_charging_plan().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
springfall2008 pushed a commit that referenced this pull request Nov 5, 2026
Brings #5905 up to date. The debug journal entry is re-applied in the compacted format from #5988.
apps/predbat/prediction.py: both kept - the car cancel revert runs first, then main's confirmed-run dispatch_risk (#5896); a reverted step is no longer a dispatch that can go away, so it carries no dispatch_risk.
apps/predbat/prediction_kernel.cpp: the same combination mirrored in pk_run_one(); PK_PARITY_REVISION bumped to 21 (branch 20, main 19), ABI stays 12.
apps/predbat/prediction_kernel.py: KERNEL_ABI_VERSION 12, KERNEL_PARITY_REVISION bumped to 21 to match.
apps/predbat/prediction_kernel_lib_*.so: taken from the upstream side; CI's kernel-binaries job rebuilds them for parity 21.
apps/predbat/tests/test_random_scenarios.py: the branch's slot line kept; it already copies each slot dict, which subsumes main's copy fix (#5977's re-apply test passes).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
springfall2008 pushed a commit that referenced this pull request Nov 5, 2026
Brings #5363 up to date. The debug journal entry is re-applied in the compacted format from #5988.
.cspell/custom-dictionary-workspace.txt: union of both sides, sorted.
apps/predbat/tests/test_single_debug.py: kept main's rebuild_battery_value_rates() (#5872) and its --redo call ahead of the PR's extracted rescan_rate_windows(); restore_debug_state() keeps main's rate_export_min_base reset.
docs/developing.md: took main's restructured guide and put the PR's "Replaying a log forwards" and "The dummy inverter" sections under Testing, after "Replaying a debug dump".
apps/predbat/tests/replay_forward.py (no textual conflict): main's rate_scan() no longer builds rate_min_forward, so rescan_rate_stats() now calls rebuild_battery_value_rates(); test_rescan_rate_stats covers a free session left out of the forward minimum.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

BOT_FAILED An automated triage/PR attempt failed; see the comment for details

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant