Repository navigation
Change the duplicate-docs eval subject to an upload retention period - #12
Merged
Merged
Conversation
The old workspace was identical to example 3 in the skill's maintenance examples, so a pass did not show whether Claude applied the consolidation rule or reproduced the example. The new workspace states a retention rule in a README section and in docs/operations.md, with no link between them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
duplicate-docseval case tests an upload retention period instead of a default port.The old workspace was identical to example 3 in
skills/doc-maintenance/examples/maintenance.md, so a pass did not show whether Claudeapplied the consolidation rule or reproduced the example.
Changes
cleanup.pyreplacesserver.py.RETENTION_DAYS = 30is the default, andUPLOAD_RETENTION_DAYSoverrides it.docs/operations.mdstate the retention rule indifferent words. The README has no link to the operations page, so the consolidation has
to add one. In example 3, the link already exists.
cleanup.pyanddocs/operations.md.backups-untouchedreplaceslog-level-untouched. Thereadme-consolidatedcriteria accept a reference that namesthe subject or the variable.
Validation
claude plugin validate .passes with Claude Code 2.1.308. The one warning is theaccepted
CLAUDE.mdwarning.Eval results for
duplicate-docs, 3 runs per model, Sonnet judge, no baseline arm:readme-consolidateddocs/operations.md#upload-retention.The Opus result matches the 7 of 7 on the old workspace, so the old result was not an
effect of the shared example text.
Related measurement, recorded here because Rename the skill to doc-maintenance and set the version to 0.5.0 #11 listed it as untested: Opus with the
Sonnet judge on
obsolete-comment, 3 runs. Every grader passed, with 3 of 3 judge voteson
comment-judgment. The runs kept the constraint as a comment, moved it into thedocstring, and removed it, and the judge passed each form. The
comment-judgmentfailures in Rename the skill to doc-maintenance and set the version to 0.5.0 #11 were Haiku judge defects.
No shell script changed.
Known limitations
copies of a fact can diverge.