Skip to content

Change the duplicate-docs eval subject to an upload retention period - #12

Merged
MirzaMerdovic merged 1 commit into
mainfrom
duplicate-docs-workspace
Oct 28, 2026
Merged

MirzaMerdovic merged 1 commit into
mainfrom
duplicate-docs-workspace

Conversation

@MirzaMerdovic

Copy link
Copy Markdown
Contributor

Summary

The duplicate-docs eval case tests an upload retention period instead of a default port.
The old workspace was identical to example 3 in
skills/doc-maintenance/examples/maintenance.md, so a pass did not show whether Claude
applied the consolidation rule or reproduced the example.

Changes

  • cleanup.py replaces server.py. RETENTION_DAYS = 30 is the default, and
    UPLOAD_RETENTION_DAYS overrides it.
  • The README "Retention" section and docs/operations.md state the retention rule in
    different words. The README has no link to the operations page, so the consolidation has
    to add one. In example 3, the link already exists.
  • The prompt is "Change the default upload retention from 30 days to 14 days."
  • The graders check cleanup.py and docs/operations.md. backups-untouched replaces
    log-level-untouched. The readme-consolidated criteria accept a reference that names
    the subject or the variable.

Validation

  • claude plugin validate . passes with Claude Code 2.1.308. The one warning is the
    accepted CLAUDE.md warning.

  • Eval results for duplicate-docs, 3 runs per model, Sonnet judge, no baseline arm:

    Model readme-consolidated Other graders
    Opus 3 of 3. Each run replaced the README section with a link to docs/operations.md#upload-retention. All passed.
    Sonnet 0 of 3. Each run set 14 days in both documents and kept both copies. All passed.
  • The Opus result matches the 7 of 7 on the old workspace, so the old result was not an
    effect of the shared example text.

  • Related measurement, recorded here because Rename the skill to doc-maintenance and set the version to 0.5.0 #11 listed it as untested: Opus with the
    Sonnet judge on obsolete-comment, 3 runs. Every grader passed, with 3 of 3 judge votes
    on comment-judgment. The runs kept the constraint as a comment, moved it into the
    docstring, and removed it, and the judge passed each form. The comment-judgment
    failures in Rename the skill to doc-maintenance and set the version to 0.5.0 #11 were Haiku judge defects.

  • No shell script changed.

Known limitations

  • Sonnet does not consolidate the two documents. The grader stays strict, because two
    copies of a fact can diverge.
  • The no-plugin baseline arm did not run.

The old workspace was identical to example 3 in the skill's maintenance
examples, so a pass did not show whether Claude applied the consolidation
rule or reproduced the example. The new workspace states a retention rule
in a README section and in docs/operations.md, with no link between them.
@MirzaMerdovic
MirzaMerdovic merged commit afae0be into main Oct 28, 2026
1 check passed
@MirzaMerdovic
MirzaMerdovic deleted the duplicate-docs-workspace branch October 28, 2026 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant