From bd5081d00c12fbaf8ffd550946b5447067b913f8 Mon Sep 17 00:00:00 2001 From: Adrianno Esnarriaga Sereno Date: Sun, 18 Oct 2026 11:24:31 -0300 Subject: [PATCH 1/5] fix(agent): stub oversized history only on an empty summary Compaction treated a truncated stub as the whole message. When earlier lines had already filled the 64 KiB summary source, a later message that cannot fit was cut to the leftover sliver and then dropped. Stub that line only on an empty source so this pass summarizes the prefix and the next pass keeps about 64 KiB of the oversized line. Consolidated from cursor/critical-bug-identification-d77c onto the post-promote development base: the only-empty rule replaces the half-source stub threshold from 0d4e380 (same if/branch), and the later-line regression now drives the oversized turn through several turns with a 50 KiB predecessor to assert the defer-then-full-stub sequence instead of a one-shot sliver stub. Supersedes #132. --- CHANGELOG.md | 1 + src/core/agent.c | 30 ++++++++--------- tests/test_agent.c | 82 +++++++++++++++++++++++++++++++--------------- 3 files changed, 70 insertions(+), 43 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4876cb1..e0d55fe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,7 @@ All notable changes to ShellClaw are documented here. Format follows [Keep a Cha - Session compaction keeps a message that does not fit the 64 KiB summary source, and everything after it, instead of skipping the long line and deleting it anyway. - Session compaction no longer persists the summarized transcript before the user turn finishes: a failed follow-up provider call leaves the raw stored session unchanged, and when the kept tail already starts with a user turn the summary is folded into that turn so consecutive user roles cannot break the next Anthropic call (supersedes #126). - Session compaction summarizes a message larger than the summary source as a truncated `[truncated]` stub, so a single oversized line can no longer abort the pass with an empty summary source or keep the session above `max_context_messages` forever. +- Session compaction stubs that oversized line only when the summary source is still empty. A later oversized message is left in the session until the next pass, instead of being cut to whatever sliver the earlier lines left and then deleted. - WebChat `ws_send_to` drops a send whose session has no live `wsi` (still returning 0) and `ws_unregister_conn` drops frames still queued for the closing session, so ~64 abandoned replies cannot pin the shared outgoing queue at `MAX_QUEUE` (64) and stall every live client until restart. - `read_file` reads that do not fit the result buffer now fail with a JSON error instead of returning a clipped prefix, so a later `write_file` cannot persist a silently truncated file. diff --git a/src/core/agent.c b/src/core/agent.c index 124c8f4..47d34fe 100644 --- a/src/core/agent.c +++ b/src/core/agent.c @@ -257,23 +257,19 @@ static int compact_session_via_llm(const char *session_id, char *session_buf, si if (cJSON_IsString(r)) role = r->valuestring; if (cJSON_IsString(c)) content = c->valuestring; if (!summary_source_append(source_buf, SUMMARY_SOURCE_MAX, &used, role, content)) { - /* A line that can never fit whole must be stubbed when the - * source is still (mostly) empty: the first line has to be - * stubbed or the source stays empty and compaction returns - * -1 on every turn. Once earlier lines already consumed the - * buffer, a stub would see only the bytes left over and - * destroy the line in the summary, so leave it raw via - * next_unsummarized; the next pass starts it near the top of - * a fresh source and gives it (almost) the whole buffer. A - * line that merely lacks room here is left raw the same way. */ - int stubbed = summary_line_never_fits(role, content) && - used <= SUMMARY_SOURCE_MAX / 2 && - summary_source_append_truncated(source_buf, SUMMARY_SOURCE_MAX, - &used, role, content); - if (!stubbed) { - next_unsummarized = i; - break; - } + /* Stub a never-fit line only against an empty source. Leftover + * room after earlier lines is a sliver: the stub still counts as + * the whole message, and the unread tail is deleted. Defer so + * this pass summarizes the prefix and the next pass stubs the + * line at the full 64 KiB. A never-fit first line still has to + * be stubbed here, or used==0 aborts compaction and the session + * never shrinks. Lines that fit a fresh source stay raw. */ + if (used == 0 && + summary_line_never_fits(role, content) && + summary_source_append_truncated(source_buf, SUMMARY_SOURCE_MAX, &used, role, content)) + continue; + next_unsummarized = i; + break; } } if (used == 0) { diff --git a/tests/test_agent.c b/tests/test_agent.c index fbdd800..881bc68 100644 --- a/tests/test_agent.c +++ b/tests/test_agent.c @@ -758,21 +758,31 @@ static int test_compaction_progresses_when_first_message_is_oversized(void) return failed; } -/* An oversized LATER line used to be kept raw with everything after it, so - * new_count stayed above max_ctx forever. The truncated stub now consumes it - * and this pass brings the session back down to max_ctx. */ +/* A later line that cannot fit a fresh 64 KiB source must not be stubbed into + * whatever sliver the earlier lines left. That stub is treated as the whole + * message, so the unread tail is deleted. The prefix is summarized first; a + * later pass stubs the oversized line against an empty source, and the + * session still shrinks back to max_ctx. */ static int test_compaction_progresses_when_later_message_is_oversized(void) { - enum { OVERSIZE = 65 * 1024, JSON_CAP = 96 * 1024, LOAD_CAP = 128 * 1024 }; + enum { + PRE_SIZE = 50 * 1024, + OVERSIZE = 65 * 1024, + JSON_CAP = 192 * 1024, + LOAD_CAP = 128 * 1024 + }; int failed = 1; + int shrunk = 0; config_t *cfg = NULL; char *session_json = NULL; char *loaded = NULL; + char *pre = NULL; char *big = NULL; const char *db_path = "build/test_agent_compact_stall_later.db"; const char *config_path = "build/test_agent_compact_stall_later.toml"; size_t off; int i; + int nmsg; char response_buf[4096]; char errbuf[256]; FILE *cf; @@ -782,17 +792,21 @@ static int test_compaction_progresses_when_later_message_is_oversized(void) if (memory_init(db_path) != 0) goto cleanup_early; session_json = malloc(JSON_CAP); + pre = malloc((size_t)PRE_SIZE + 1); big = malloc((size_t)OVERSIZE + 1); loaded = malloc(LOAD_CAP); - if (!session_json || !big || !loaded) + if (!session_json || !pre || !big || !loaded) goto cleanup; - memset(big, 'A', (size_t)OVERSIZE); + memset(pre, 'P', (size_t)PRE_SIZE); + memcpy(pre + PRE_SIZE - 7, "PRE_END", 7); + pre[PRE_SIZE] = '\0'; + memset(big, 'B', (size_t)OVERSIZE); memcpy(big + OVERSIZE - 11, "STALL_B_END", 11); big[OVERSIZE] = '\0'; off = 0; off += (size_t)snprintf(session_json + off, JSON_CAP - off, "["); off += (size_t)snprintf(session_json + off, JSON_CAP - off, - "{\"role\":\"user\",\"content\":\"alpha_small\"}"); + "{\"role\":\"user\",\"content\":\"%s\"}", pre); off += (size_t)snprintf(session_json + off, JSON_CAP - off, ",{\"role\":\"user\",\"content\":\"%s\"}", big); for (i = 2; i < 6; i++) { @@ -813,26 +827,41 @@ static int test_compaction_progresses_when_later_message_is_oversized(void) if (config_load(config_path, &cfg, errbuf, sizeof(errbuf)) != 0) goto cleanup; response_buf[0] = '\0'; - if (agent_run(cfg, "cli:stalllater", "hi", &compaction_provider, NULL, 0, - response_buf, sizeof(response_buf)) != 0) - goto cleanup; - if (!compaction_source_truncated) { - fprintf(stderr, - "FAIL: oversized later line did not reach the summary as a truncated stub\n"); - goto cleanup; - } - if (session_load("cli:stalllater", loaded, LOAD_CAP) != 0) - goto cleanup; - if (strstr(loaded, "STALL_B_END") != NULL) { - fprintf(stderr, "FAIL: oversized later line is still raw instead of summarized\n"); - goto cleanup; - } - if (strstr(loaded, "Summary of earlier") == NULL) { - fprintf(stderr, "FAIL: expected a summary of the truncated oversized turn\n"); - goto cleanup; + for (i = 0; i < 8; i++) { + if (agent_run(cfg, "cli:stalllater", "hi", &compaction_provider, NULL, 0, + response_buf, sizeof(response_buf)) != 0) + goto cleanup; + if (session_load("cli:stalllater", loaded, LOAD_CAP) != 0) + goto cleanup; + nmsg = count_session_messages(loaded); + if (i == 0) { + if (strstr(loaded, "STALL_B_END") == NULL) { + fprintf(stderr, + "FAIL: oversized later line was deleted after a %d-byte predecessor filled the summary\n", + PRE_SIZE); + goto cleanup; + } + if (strstr(loaded, "PRE_END") != NULL) { + fprintf(stderr, + "FAIL: predecessor that fit the summary was kept raw\n"); + goto cleanup; + } + if (strstr(loaded, "Summary of earlier") == NULL) { + fprintf(stderr, + "FAIL: expected a summary of the predecessor, not a sliver of the oversized line\n"); + goto cleanup; + } + } + if (strstr(loaded, "STALL_B_END") == NULL && nmsg <= 4 + 2) { + shrunk = 1; + break; + } } - if (count_session_messages(loaded) > 4 + 2) { - fprintf(stderr, "FAIL: session is not back to max_ctx after the oversized turn\n"); + if (!shrunk) { + nmsg = count_session_messages(loaded); + fprintf(stderr, + "FAIL: oversized later line never left the session or count stayed at %d\n", + nmsg); goto cleanup; } /* The pass after a partial pass must keep making progress, and once the @@ -854,6 +883,7 @@ static int test_compaction_progresses_when_later_message_is_oversized(void) cleanup: config_free(cfg); free(session_json); + free(pre); free(big); free(loaded); cleanup_early: From ac283fbd60328b0c4c4a340587453a735d9423fb Mon Sep 17 00:00:00 2001 From: Adrianno Esnarriaga Sereno Date: Sun, 18 Oct 2026 11:27:59 -0300 Subject: [PATCH 2/5] fix(agent): fold the partial-pass summary into an oversized assistant line The only-empty stub rule lets a pass defer a never-fit line to the next one, which works for a user line: the pass folds its summary into the kept line, the line lands at the head of the session and the next pass stubs it against a full 64 KiB source. An assistant line cannot take that path. The kept tail starts on it, a user summary gets prepended in front of it, and the next pass sees the same pair again - the wrapper always counts toward used, so no pass ever reaches the only-empty stub. The count then only grows until SESSION_JSON_MAX stops the saves and the stored session stays above max_context_messages forever. The new regression test reproduces the stall on the exact post-partial-pass state: without the fix the count stayed at 6..22 across eight turns. Fix shape: a partial pass that consumed exactly one line and breaks on a never-fit non-user line now folds its summary into that line instead of prepending a wrapper. The line keeps its content and role, the count shrinks, and the next pass starts on the line with an empty source and stubs it whole. Chosen over forcing a truncated stub at the stuck pass: that restores progress but re-creates the sliver this branch exists to prevent whenever the one consumed line was large; the fold always reaches the full stub. The wrapper variant (not counting the summary toward used) was rejected: the wrapper cannot be told from real content structurally and dropping it from the source would lose the accumulated summary. The full-buffer next-pass test now expects the count to shrink on the first partial pass (5 + 2 instead of 6 + 2): the fold consumes the head line together with the oversized line, where the old half-source threshold left the count unchanged for one pass. The full-buffer stub assertion itself is unchanged and still holds. --- CHANGELOG.md | 1 + src/core/agent.c | 32 ++++++++--- tests/test_agent.c | 130 +++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 151 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e0d55fe..f3b69f1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,7 @@ All notable changes to ShellClaw are documented here. Format follows [Keep a Cha - Session compaction no longer persists the summarized transcript before the user turn finishes: a failed follow-up provider call leaves the raw stored session unchanged, and when the kept tail already starts with a user turn the summary is folded into that turn so consecutive user roles cannot break the next Anthropic call (supersedes #126). - Session compaction summarizes a message larger than the summary source as a truncated `[truncated]` stub, so a single oversized line can no longer abort the pass with an empty summary source or keep the session above `max_context_messages` forever. - Session compaction stubs that oversized line only when the summary source is still empty. A later oversized message is left in the session until the next pass, instead of being cut to whatever sliver the earlier lines left and then deleted. +- Session compaction folds the partial-pass summary into an oversized assistant line instead of leaving a user summary in front of it, so a pass that cannot stub the line still shrinks the session; the line reaches the head of the next pass and gets the full 64 KiB stub instead of stalling the count above `max_context_messages` until the session blob cap freezes it. - WebChat `ws_send_to` drops a send whose session has no live `wsi` (still returning 0) and `ws_unregister_conn` drops frames still queued for the closing session, so ~64 abandoned replies cannot pin the shared outgoing queue at `MAX_QUEUE` (64) and stall every live client until restart. - `read_file` reads that do not fit the result buffer now fail with a JSON error instead of returning a clipped prefix, so a later `write_file` cannot persist a silently truncated file. diff --git a/src/core/agent.c b/src/core/agent.c index 47d34fe..55fc849 100644 --- a/src/core/agent.c +++ b/src/core/agent.c @@ -165,8 +165,10 @@ static int kept_tail_starts_with_user(const cJSON *root, int start) return cJSON_IsString(role) && role->valuestring && strcmp(role->valuestring, "user") == 0; } -/** Fold the summary into an existing user turn. A second user message 400s on Anthropic. */ -static int fold_summary_into_user(cJSON *item, const char *summary) +/** Fold the summary into the first kept message: a user turn (a second user + * message 400s on Anthropic) or the oversized line a partial pass broke on, + * so the next pass starts on it with an empty source for the full stub. */ +static int fold_summary_into_message(cJSON *item, const char *summary) { const cJSON *content; const char *old; @@ -245,6 +247,7 @@ static int compact_session_via_llm(const char *session_id, char *session_buf, si source_buf[0] = '\0'; size_t used = 0; int next_unsummarized = to_summarize_n; + int fold_into_oversized = 0; for (int i = 0; i < to_summarize_n; i++) { const cJSON *item = cJSON_GetArrayItem(root, i); const char *role = "user"; @@ -268,6 +271,19 @@ static int compact_session_via_llm(const char *session_id, char *session_buf, si summary_line_never_fits(role, content) && summary_source_append_truncated(source_buf, SUMMARY_SOURCE_MAX, &used, role, content)) continue; + /* A never-fit line that is not a user turn and sits directly + * behind the only line this pass consumed would leave the + * count unchanged: the tail starts on it, a user summary is + * prepended in front of it, and the next pass sees the same + * pair again - the wrapper always counts toward used, so no + * pass ever reaches the only-empty stub and the count only + * grows until SESSION_JSON_MAX stops the saves. Fold this + * pass's summary into the line instead: the count shrinks + * and the next pass starts on the line with an empty source, + * so it gets the full 64 KiB stub (see the tail build). */ + if (i == 1 && strcmp(role, "user") != 0 && + summary_line_never_fits(role, content)) + fold_into_oversized = 1; next_unsummarized = i; break; } @@ -304,10 +320,12 @@ static int compact_session_via_llm(const char *session_id, char *session_buf, si if (start < 0) start = 0; /* Tail can start on a user turn (odd max_context_messages, or a cut that * lands on user). Inserting another user role makes Anthropic reject the - * session on this turn and every later one. */ + * session on this turn and every later one. fold_into_oversized breaks on + * that rule for the line that stopped the pass: it is not a user turn, so + * the summary folds into it instead of a prepended wrapper (see the loop). */ merge = kept_tail_starts_with_user(root, start); if (!new_arr) { cJSON_Delete(root); return -1; } - if (!merge && append_summary_user(new_arr, summary_buf) != 0) { + if (!merge && !fold_into_oversized && append_summary_user(new_arr, summary_buf) != 0) { cJSON_Delete(new_arr); cJSON_Delete(root); return -1; @@ -318,8 +336,8 @@ static int compact_session_via_llm(const char *session_id, char *session_buf, si if (!item) continue; dup = cJSON_Duplicate(item, 1); if (!dup) continue; - if (merge && i == start) { - if (fold_summary_into_user(dup, summary_buf) != 0) { + if ((merge || fold_into_oversized) && i == start) { + if (fold_summary_into_message(dup, summary_buf) != 0) { cJSON_Delete(dup); cJSON_Delete(new_arr); cJSON_Delete(root); @@ -329,7 +347,7 @@ static int compact_session_via_llm(const char *session_id, char *session_buf, si } cJSON_AddItemToArray(new_arr, dup); } - if (merge && !folded) { + if ((merge || fold_into_oversized) && !folded) { cJSON_Delete(new_arr); cJSON_Delete(root); return -1; diff --git a/tests/test_agent.c b/tests/test_agent.c index 881bc68..6e756cf 100644 --- a/tests/test_agent.c +++ b/tests/test_agent.c @@ -1060,8 +1060,9 @@ static int test_compaction_gives_oversized_line_full_buffer_on_next_pass(void) response_buf, sizeof(response_buf)) != 0) goto cleanup; /* Pass 1: the head line consumed most of the source, so the oversized - * assistant line must stay raw; a stub would only see the left-over - * bytes. With the count untouched this is the merge == 0 partial pass. */ + * assistant line must stay whole; a stub would only see the left-over + * bytes. The merge == 0 partial pass now folds its summary into the + * line (the head is consumed too), so the count shrinks by one. */ if (strstr(compaction_last_source, "[truncated]") != NULL) { fprintf(stderr, "FAIL: pass 1 stubbed the oversized line into the left-over bytes\n"); goto cleanup; @@ -1069,11 +1070,11 @@ static int test_compaction_gives_oversized_line_full_buffer_on_next_pass(void) if (session_load("cli:fullbuff", loaded, LOAD_CAP) != 0) goto cleanup; if (strstr(loaded, "BIGASST_END") == NULL) { - fprintf(stderr, "FAIL: pass 1 destroyed the oversized message instead of keeping it raw\n"); + fprintf(stderr, "FAIL: pass 1 destroyed the oversized message instead of keeping it whole\n"); goto cleanup; } - if (count_session_messages(loaded) != 6 + 2) { - fprintf(stderr, "FAIL: partial pass changed the count, expected unchanged\n"); + if (count_session_messages(loaded) != 5 + 2) { + fprintf(stderr, "FAIL: partial pass did not fold its summary into the oversized line\n"); goto cleanup; } /* Pass 2: the oversized line starts near the top of a fresh source, so @@ -1125,6 +1126,124 @@ static int test_compaction_gives_oversized_line_full_buffer_on_next_pass(void) return failed; } +/* An oversized ASSISTANT line cannot reach the head of the session the way a + * user line can: a partial pass prepends a user summary in front of it (merge + * == 0), that wrapper always counts toward used, so the only-empty stub never + * fires and the count never shrinks again. This seeds the exact state a + * merge==0 partial pass leaves behind and requires convergence anyway. */ +static int test_compaction_progresses_when_assistant_message_is_oversized(void) +{ + enum { OVERSIZE = 65 * 1024, JSON_CAP = 96 * 1024, LOAD_CAP = 128 * 1024 }; + int failed = 1; + int shrunk = 0; + config_t *cfg = NULL; + char *session_json = NULL; + char *loaded = NULL; + char *big = NULL; + const char *db_path = "build/test_agent_compact_stall_assistant.db"; + const char *config_path = "build/test_agent_compact_stall_assistant.toml"; + size_t off; + int i; + int nmsg; + char response_buf[4096]; + char errbuf[256]; + FILE *cf; + + memory_cleanup(); + compaction_call_count = 0; + if (memory_init(db_path) != 0) + goto cleanup_early; + session_json = malloc(JSON_CAP); + big = malloc((size_t)OVERSIZE + 1); + loaded = malloc(LOAD_CAP); + if (!session_json || !big || !loaded) + goto cleanup; + memset(big, 'A', (size_t)OVERSIZE); + memcpy(big + OVERSIZE - 8, "ASST_END", 8); + big[OVERSIZE] = '\0'; + off = 0; + off += (size_t)snprintf(session_json + off, JSON_CAP - off, "["); + /* The state a merge == 0 partial pass leaves behind: a summary user + * line in front of an oversized assistant line. */ + off += (size_t)snprintf(session_json + off, JSON_CAP - off, + "{\"role\":\"user\",\"content\":\"Summary of earlier turns.\"}"); + off += (size_t)snprintf(session_json + off, JSON_CAP - off, + ",{\"role\":\"assistant\",\"content\":\"%s\"}", big); + for (i = 2; i < 6; i++) { + off += (size_t)snprintf(session_json + off, JSON_CAP - off, + ",{\"role\":\"user\",\"content\":\"m%d\"}", i); + } + off += (size_t)snprintf(session_json + off, JSON_CAP - off, "]"); + if (off >= JSON_CAP) + goto cleanup; + if (session_save("cli:stallasst", session_json) != 0) + goto cleanup; + cf = fopen(config_path, "w"); + if (!cf) + goto cleanup; + fprintf(cf, "[agent]\nmodel = \"test\"\nmax_context_messages = 4\n[memory]\npath = \"%s\"\n", + db_path); + fclose(cf); + if (config_load(config_path, &cfg, errbuf, sizeof(errbuf)) != 0) + goto cleanup; + response_buf[0] = '\0'; + for (i = 0; i < 8; i++) { + if (agent_run(cfg, "cli:stallasst", "hi", &compaction_provider, NULL, 0, + response_buf, sizeof(response_buf)) != 0) + goto cleanup; + if (session_load("cli:stallasst", loaded, LOAD_CAP) != 0) + goto cleanup; + nmsg = count_session_messages(loaded); + if (i == 0) { + /* The first partial pass must keep the oversized line + * whole: only a later pass with an empty source may + * stub it, never the leftover sliver. */ + if (strstr(loaded, "ASST_END") == NULL) { + fprintf(stderr, + "FAIL: oversized assistant line was stubbed into a sliver, not deferred\n"); + goto cleanup; + } + } + if (strstr(loaded, "ASST_END") == NULL && nmsg <= 4 + 2) { + shrunk = 1; + break; + } + } + if (!shrunk) { + nmsg = count_session_messages(loaded); + fprintf(stderr, + "FAIL: oversized assistant line never left the session or count stayed at %d\n", + nmsg); + goto cleanup; + } + /* Once the count is back at max_ctx the summary stays inside the sent + * window. */ + if (agent_run(cfg, "cli:stallasst", "hi", &compaction_provider, NULL, 0, + response_buf, sizeof(response_buf)) != 0) + goto cleanup; + if (session_load("cli:stallasst", loaded, LOAD_CAP) != 0) + goto cleanup; + if (count_session_messages(loaded) > 4 + 2) { + fprintf(stderr, "FAIL: session never shrinks back to max_ctx after the assistant turn\n"); + goto cleanup; + } + if (!compaction_spy_sent_summary()) { + fprintf(stderr, "FAIL: summary of the assistant turn is outside the sent window\n"); + goto cleanup; + } + failed = 0; +cleanup: + config_free(cfg); + free(session_json); + free(big); + free(loaded); +cleanup_early: + remove(config_path); + remove(db_path); + memory_cleanup(); + return failed; +} + static int save_alternating_session(const char *session_id, int count) { char *json; @@ -2104,6 +2223,7 @@ int main(void) RUN(test_compaction_progresses_when_later_message_is_oversized()); RUN(test_compaction_stub_ends_on_utf8_boundary_when_oversized()); RUN(test_compaction_gives_oversized_line_full_buffer_on_next_pass()); + RUN(test_compaction_progresses_when_assistant_message_is_oversized()); RUN(test_compaction_failure_keeps_original_session()); RUN(test_compaction_merges_summary_when_tail_starts_with_user()); RUN(test_local_offline_note_when_active_is_local()); From 7cf7bebeea104a428a6400c10dd070430b582bf1 Mon Sep 17 00:00:00 2001 From: Adrianno Esnarriaga Sereno Date: Sun, 18 Oct 2026 11:40:02 -0300 Subject: [PATCH 3/5] fix(agent): fold any non-user line that stops the pass at index 1 A line that fits a fresh source but not the wrapped remainder behind the previous head used to miss the fold (it is not "never-fit"), take the old prepend path and freeze the session at the same count: every pass rebuilt [summary, bigA] and the count only grew until SESSION_JSON_MAX. The fold now fires for any non-user line that stops the pass at index 1, never-fit or not; regression pins the 65500-byte band (stalled red at count 14 after 4 turns, converges by turn 3 green). --- src/core/agent.c | 23 +++++---- tests/test_agent.c | 118 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 129 insertions(+), 12 deletions(-) diff --git a/src/core/agent.c b/src/core/agent.c index 55fc849..a3df259 100644 --- a/src/core/agent.c +++ b/src/core/agent.c @@ -271,18 +271,17 @@ static int compact_session_via_llm(const char *session_id, char *session_buf, si summary_line_never_fits(role, content) && summary_source_append_truncated(source_buf, SUMMARY_SOURCE_MAX, &used, role, content)) continue; - /* A never-fit line that is not a user turn and sits directly - * behind the only line this pass consumed would leave the - * count unchanged: the tail starts on it, a user summary is - * prepended in front of it, and the next pass sees the same - * pair again - the wrapper always counts toward used, so no - * pass ever reaches the only-empty stub and the count only - * grows until SESSION_JSON_MAX stops the saves. Fold this - * pass's summary into the line instead: the count shrinks - * and the next pass starts on the line with an empty source, - * so it gets the full 64 KiB stub (see the tail build). */ - if (i == 1 && strcmp(role, "user") != 0 && - summary_line_never_fits(role, content)) + /* A non-user line directly behind the only line this pass consumed would + * leave the count unchanged: the tail starts on it, a user summary + * is prepended in front of it, and the next pass sees the same + * pair again - the wrapper always counts toward used, so no pass + * ever reaches the only-empty stub and the count only grows until + * SESSION_JSON_MAX stops the saves. That holds for a never-fit + * line and for one that merely fits a fresh source but not the + * wrapped remainder, so any non-user line stopping the pass at + * index 1 folds: the count shrinks now and the next pass handles + * the folded line against a fresh source. */ + if (i == 1 && strcmp(role, "user") != 0) fold_into_oversized = 1; next_unsummarized = i; break; diff --git a/tests/test_agent.c b/tests/test_agent.c index 6e756cf..b3215a8 100644 --- a/tests/test_agent.c +++ b/tests/test_agent.c @@ -1131,6 +1131,123 @@ static int test_compaction_gives_oversized_line_full_buffer_on_next_pass(void) * == 0), that wrapper always counts toward used, so the only-empty stub never * fires and the count never shrinks again. This seeds the exact state a * merge==0 partial pass leaves behind and requires convergence anyway. */ +/* A line that fits a fresh source but not behind the summary wrapper used + * to miss the fold (it is not "never-fit"), take the old prepend path and + * freeze the session: every pass rebuilt [summary, bigA] and the count + * only grew. Any non-user line that stops the pass at index 1 must fold. */ +static int test_compaction_folds_assistant_line_that_fits_a_fresh_source(void) +{ + enum { OVERSIZE = 65500, JSON_CAP = 96 * 1024, LOAD_CAP = 128 * 1024 }; + int failed = 1; + int shrunk = 0; + int shrink_turn = 99; + config_t *cfg = NULL; + char *session_json = NULL; + char *loaded = NULL; + char *big = NULL; + const char *db_path = "build/test_agent_compact_band_assistant.db"; + const char *config_path = "build/test_agent_compact_band_assistant.toml"; + size_t off; + int i; + int nmsg; + char response_buf[4096]; + char errbuf[256]; + FILE *cf; + + memory_cleanup(); + compaction_call_count = 0; + if (memory_init(db_path) != 0) + goto cleanup_early; + session_json = malloc(JSON_CAP); + big = malloc((size_t)OVERSIZE + 1); + loaded = malloc(LOAD_CAP); + if (!session_json || !big || !loaded) + goto cleanup; + memset(big, 'A', (size_t)OVERSIZE); + memcpy(big + OVERSIZE - 8, "ASST_END", 8); + big[OVERSIZE] = '\0'; + off = 0; + off += (size_t)snprintf(session_json + off, JSON_CAP - off, "["); + off += (size_t)snprintf(session_json + off, JSON_CAP - off, + "{\"role\":\"user\",\"content\":\"Summary of earlier turns.\"}"); + off += (size_t)snprintf(session_json + off, JSON_CAP - off, + ",{\"role\":\"assistant\",\"content\":\"%s\"}", big); + for (i = 2; i < 6; i++) { + off += (size_t)snprintf(session_json + off, JSON_CAP - off, + ",{\"role\":\"user\",\"content\":\"m%d\"}", i); + } + off += (size_t)snprintf(session_json + off, JSON_CAP - off, "]"); + if (off >= JSON_CAP) + goto cleanup; + if (session_save("cli:bandasst", session_json) != 0) + goto cleanup; + cf = fopen(config_path, "w"); + if (!cf) + goto cleanup; + fprintf(cf, "[agent]\nmodel = \"test\"\nmax_context_messages = 4\n[memory]\npath = \"%s\"\n", + db_path); + fclose(cf); + if (config_load(config_path, &cfg, errbuf, sizeof(errbuf)) != 0) + goto cleanup; + response_buf[0] = '\0'; + for (i = 0; i < 4 && !shrunk; i++) { + if (agent_run(cfg, "cli:bandasst", "hi", &compaction_provider, NULL, 0, + response_buf, sizeof(response_buf)) != 0) + goto cleanup; + if (session_load("cli:bandasst", loaded, LOAD_CAP) != 0) + goto cleanup; + nmsg = count_session_messages(loaded); + if (i == 0) { + if (strstr(loaded, "ASST_END") == NULL) { + fprintf(stderr, + "FAIL: band line was stubbed into a sliver instead of deferred\n"); + goto cleanup; + } + } + if (strstr(loaded, "ASST_END") == NULL && nmsg <= 4 + 2) { + shrunk = 1; + shrink_turn = i; + } + } + if (!shrunk) { + nmsg = count_session_messages(loaded); + fprintf(stderr, + "FAIL: band assistant line stalled compaction (count %d after 4 turns)\n", + nmsg); + goto cleanup; + } + if (shrink_turn > 2) { + fprintf(stderr, + "FAIL: compaction needed turn %d, expected convergence by turn 3\n", + shrink_turn + 1); + goto cleanup; + } + if (agent_run(cfg, "cli:bandasst", "hi", &compaction_provider, NULL, 0, + response_buf, sizeof(response_buf)) != 0) + goto cleanup; + if (session_load("cli:bandasst", loaded, LOAD_CAP) != 0) + goto cleanup; + if (count_session_messages(loaded) > 4 + 2) { + fprintf(stderr, "FAIL: session never shrinks back to max_ctx after the band turn\n"); + goto cleanup; + } + if (!compaction_spy_sent_summary()) { + fprintf(stderr, "FAIL: summary of the band case is outside the sent window\n"); + goto cleanup; + } + failed = 0; +cleanup: + config_free(cfg); + free(session_json); + free(big); + free(loaded); +cleanup_early: + remove(config_path); + remove(db_path); + memory_cleanup(); + return failed; +} + static int test_compaction_progresses_when_assistant_message_is_oversized(void) { enum { OVERSIZE = 65 * 1024, JSON_CAP = 96 * 1024, LOAD_CAP = 128 * 1024 }; @@ -2223,6 +2340,7 @@ int main(void) RUN(test_compaction_progresses_when_later_message_is_oversized()); RUN(test_compaction_stub_ends_on_utf8_boundary_when_oversized()); RUN(test_compaction_gives_oversized_line_full_buffer_on_next_pass()); + RUN(test_compaction_folds_assistant_line_that_fits_a_fresh_source()); RUN(test_compaction_progresses_when_assistant_message_is_oversized()); RUN(test_compaction_failure_keeps_original_session()); RUN(test_compaction_merges_summary_when_tail_starts_with_user()); From 7dfbfc104a9577938af1c5e6083ba4dd3d0937c5 Mon Sep 17 00:00:00 2001 From: Adrianno Esnarriaga Sereno Date: Sun, 18 Oct 2026 11:40:41 -0300 Subject: [PATCH 4/5] test(agent): pin compaction convergence turns and tool mid-history The assistant regression now requires convergence within three turns instead of accepting any of eight, and a tool-role oversized line behind two consumed lines pins the normal prepend path: the first pass must shrink the count and the session must still converge like the assistant cases. --- tests/test_agent.c | 107 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) diff --git a/tests/test_agent.c b/tests/test_agent.c index b3215a8..8be45d8 100644 --- a/tests/test_agent.c +++ b/tests/test_agent.c @@ -1131,6 +1131,104 @@ static int test_compaction_gives_oversized_line_full_buffer_on_next_pass(void) * == 0), that wrapper always counts toward used, so the only-empty stub never * fires and the count never shrinks again. This seeds the exact state a * merge==0 partial pass leaves behind and requires convergence anyway. */ +/* A tool-role oversized line consumed after two lines takes the normal + * prepend path (i >= 2, no fold): the count must shrink on the very first + * pass, and the line converges later like the assistant cases. */ +static int test_compaction_progresses_when_tool_message_is_oversized_mid_history(void) +{ + enum { OVERSIZE = 65 * 1024, JSON_CAP = 96 * 1024, LOAD_CAP = 128 * 1024 }; + int failed = 1; + int shrunk = 0; + config_t *cfg = NULL; + char *session_json = NULL; + char *loaded = NULL; + char *big = NULL; + const char *db_path = "build/test_agent_compact_tool_mid.db"; + const char *config_path = "build/test_agent_compact_tool_mid.toml"; + size_t off; + int i; + int nmsg; + char response_buf[4096]; + char errbuf[256]; + FILE *cf; + + memory_cleanup(); + compaction_call_count = 0; + if (memory_init(db_path) != 0) + goto cleanup_early; + session_json = malloc(JSON_CAP); + big = malloc((size_t)OVERSIZE + 1); + loaded = malloc(LOAD_CAP); + if (!session_json || !big || !loaded) + goto cleanup; + memset(big, 'B', (size_t)OVERSIZE); + memcpy(big + OVERSIZE - 8, "TOOL_END", 8); + big[OVERSIZE] = '\0'; + off = 0; + off += (size_t)snprintf(session_json + off, JSON_CAP - off, "["); + off += (size_t)snprintf(session_json + off, JSON_CAP - off, + "{\"role\":\"user\",\"content\":\"Summary of earlier turns.\"}"); + off += (size_t)snprintf(session_json + off, JSON_CAP - off, + ",{\"role\":\"user\",\"content\":\"m2\"}"); + off += (size_t)snprintf(session_json + off, JSON_CAP - off, + ",{\"role\":\"tool\",\"content\":\"%s\",\"tool_use_id\":\"t1\"}", big); + for (i = 4; i < 7; i++) { + off += (size_t)snprintf(session_json + off, JSON_CAP - off, + ",{\"role\":\"user\",\"content\":\"m%d\"}", i); + } + off += (size_t)snprintf(session_json + off, JSON_CAP - off, "]"); + if (off >= JSON_CAP) + goto cleanup; + if (session_save("cli:toolmid", session_json) != 0) + goto cleanup; + cf = fopen(config_path, "w"); + if (!cf) + goto cleanup; + fprintf(cf, "[agent]\nmodel = \"test\"\nmax_context_messages = 4\n[memory]\npath = \"%s\"\n", + db_path); + fclose(cf); + if (config_load(config_path, &cfg, errbuf, sizeof(errbuf)) != 0) + goto cleanup; + response_buf[0] = '\0'; + for (i = 0; i < 6 && !shrunk; i++) { + if (agent_run(cfg, "cli:toolmid", "hi", &compaction_provider, NULL, 0, + response_buf, sizeof(response_buf)) != 0) + goto cleanup; + if (session_load("cli:toolmid", loaded, LOAD_CAP) != 0) + goto cleanup; + nmsg = count_session_messages(loaded); + if (i == 0 && nmsg >= 6 + 2) { + fprintf(stderr, + "FAIL: tool line at index 2 did not shrink the count on the first pass\n"); + goto cleanup; + } + if (strstr(loaded, "TOOL_END") == NULL && nmsg <= 4 + 2) + shrunk = 1; + } + if (!shrunk) { + nmsg = count_session_messages(loaded); + fprintf(stderr, + "FAIL: oversized tool line never left the session or count stayed at %d\n", + nmsg); + goto cleanup; + } + if (!compaction_spy_sent_summary()) { + fprintf(stderr, "FAIL: summary of the tool case is outside the sent window\n"); + goto cleanup; + } + failed = 0; +cleanup: + config_free(cfg); + free(session_json); + free(big); + free(loaded); +cleanup_early: + remove(config_path); + remove(db_path); + memory_cleanup(); + return failed; +} + /* A line that fits a fresh source but not behind the summary wrapper used * to miss the fold (it is not "never-fit"), take the old prepend path and * freeze the session: every pass rebuilt [summary, bigA] and the count @@ -1253,6 +1351,7 @@ static int test_compaction_progresses_when_assistant_message_is_oversized(void) enum { OVERSIZE = 65 * 1024, JSON_CAP = 96 * 1024, LOAD_CAP = 128 * 1024 }; int failed = 1; int shrunk = 0; + int shrink_turn = 99; config_t *cfg = NULL; char *session_json = NULL; char *loaded = NULL; @@ -1323,6 +1422,7 @@ static int test_compaction_progresses_when_assistant_message_is_oversized(void) } if (strstr(loaded, "ASST_END") == NULL && nmsg <= 4 + 2) { shrunk = 1; + shrink_turn = i; break; } } @@ -1333,6 +1433,12 @@ static int test_compaction_progresses_when_assistant_message_is_oversized(void) nmsg); goto cleanup; } + if (shrink_turn + 1 > 3) { + fprintf(stderr, + "FAIL: compaction needed turn %d for the oversized assistant line, expected <= 3\n", + shrink_turn + 1); + goto cleanup; + } /* Once the count is back at max_ctx the summary stays inside the sent * window. */ if (agent_run(cfg, "cli:stallasst", "hi", &compaction_provider, NULL, 0, @@ -2340,6 +2446,7 @@ int main(void) RUN(test_compaction_progresses_when_later_message_is_oversized()); RUN(test_compaction_stub_ends_on_utf8_boundary_when_oversized()); RUN(test_compaction_gives_oversized_line_full_buffer_on_next_pass()); + RUN(test_compaction_progresses_when_tool_message_is_oversized_mid_history()); RUN(test_compaction_folds_assistant_line_that_fits_a_fresh_source()); RUN(test_compaction_progresses_when_assistant_message_is_oversized()); RUN(test_compaction_failure_keeps_original_session()); From 712e06882016404892dc668a615fb6a491b58568 Mon Sep 17 00:00:00 2001 From: Adrianno Esnarriaga Sereno Date: Sun, 18 Oct 2026 11:40:51 -0300 Subject: [PATCH 5/5] docs(changelog): merge the compaction stub bullets into the final behavior The stub-only-empty, defer-whole and fold behaviors are one mechanism; the release note now reads as the resulting guarantee instead of three incremental fixes. --- CHANGELOG.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f3b69f1..ba3502d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,9 +22,7 @@ All notable changes to ShellClaw are documented here. Format follows [Keep a Cha - Session compaction includes the boundary turn (index `msg_count - max_context_messages`) in the summary window, so the turn at the keep-tail boundary is not deleted without being summarized. - Session compaction keeps a message that does not fit the 64 KiB summary source, and everything after it, instead of skipping the long line and deleting it anyway. - Session compaction no longer persists the summarized transcript before the user turn finishes: a failed follow-up provider call leaves the raw stored session unchanged, and when the kept tail already starts with a user turn the summary is folded into that turn so consecutive user roles cannot break the next Anthropic call (supersedes #126). -- Session compaction summarizes a message larger than the summary source as a truncated `[truncated]` stub, so a single oversized line can no longer abort the pass with an empty summary source or keep the session above `max_context_messages` forever. -- Session compaction stubs that oversized line only when the summary source is still empty. A later oversized message is left in the session until the next pass, instead of being cut to whatever sliver the earlier lines left and then deleted. -- Session compaction folds the partial-pass summary into an oversized assistant line instead of leaving a user summary in front of it, so a pass that cannot stub the line still shrinks the session; the line reaches the head of the next pass and gets the full 64 KiB stub instead of stalling the count above `max_context_messages` until the session blob cap freezes it. +- Session compaction summarizes a single oversized line as a truncated `[truncated]` stub only against an empty summary source: a later oversized message is deferred whole instead of cut to the sliver the earlier lines left, a pass whose summary would strand it folds the summary into the line so the count keeps shrinking, and the line is finally stubbed against a full 64 KiB source. - WebChat `ws_send_to` drops a send whose session has no live `wsi` (still returning 0) and `ws_unregister_conn` drops frames still queued for the closing session, so ~64 abandoned replies cannot pin the shared outgoing queue at `MAX_QUEUE` (64) and stall every live client until restart. - `read_file` reads that do not fit the result buffer now fail with a JSON error instead of returning a clipped prefix, so a later `write_file` cannot persist a silently truncated file.