Skip to content

Near-miss merging (--max-gap-lines): review findings from #1020 #1022

Description

@kucherenko

Follow-ups on the gap-merge pass shipped in #1020 (merge_gapped_clones in rust/crates/cpd-core/src/detect.rs), from the review of that PR. None of them affect default runs (the pass is skipped at --max-gap-lines 0).

  • Chain search for repeated blocks. The pass only tries to extend the last clone of a file pair. When a block is copied twice with an inserted line each, the sorted list interleaves the pairs, the chain breaks, and the option silently does nothing for that pair. Search all open chains of a pair (or retry the previous clone) and add a fixture with a block copied twice.
  • Bound the gap in tokens and add a similarity floor. The gap is measured in lines only, so a single 3000-token line bridges two unrelated clones into a similar ~0.05 result. Add a token bound and/or a minimum similarity for merged clones.
  • --max-gap-lines N tolerates N+2 inserted lines. Exact windows already reach into the inserted region on both sides, so a 3-line insertion measures as a 1-line gap. Measure from the matched lines, or document the real rule.
  • Deep overlaps. continuation() accepts any overlap as long as start and end increase; A B C vs A B junk B C merges into one ~0.70 clone that reuses A's tokens for two B regions. Cap the tolerated overlap (one boundary token) or subtract it correctly on both sides.
  • Cross-format pairs never merge. Primary and secondary passes stamp different formats on the two halves of a js/ts pair; drop format from the pair key or normalize it in both passes.
  • Clone order changes when the flag is on even if nothing merges (the pass sorts the whole list). Emit in the original order of each chain's first member.
  • Gap lines count as duplicated lines in statistics, so --threshold gets stricter when merging is enabled. Count matched lines only, or exclude gaps from the percentage.
  • Merged renamed halves lose kind: renamed. Decide precedence (similar subsumes renamed, or keep renamed when every part was renamed) and document it.
  • Rename the SARIF rules so the vocabulary lines up: renamedjscpd/renamed-code, similarjscpd/similar-code (currently jscpd/similar-code is the renamed rule and jscpd/near-miss-code the similar one). Unreleased, so cheap to change now.
  • Fixture README inaccuracies in fixtures/type3-demo/README.md: the shared boundary is 3 tokens (not one ;), the wide-gap second half starts on line 9's const payment, the JSON similarity for wide-gap is 0.905.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureA new functionality request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions