Problem
comment_style() in rust/crates/cpd-tokenizer/src/generic.rs has no arm for "markdown" / "md", so Markdown falls through to the catch-all _ => CommentStyle::CStyle.
Markdown prose routinely contains the C comment openers:
/* — a glob such as docs/**, a code span, or plain prose.// — any URL (https://...).
Under the C-style fallback, a stray /* opens a block comment that is never closed for the rest of the file, and // opens a line comment. Everything after the opener is silently dropped from tokenization.
Nothing signals the loss: the file is still reported as analyzed, and its token total still looks plausible. Two files that share a paragraph are simply not reported as a clone if a /* or // appears anywhere earlier in either file.
This compounds with the prose-tokenization fix from #883 (already merged), which routes the Markdown prose body through tokenize_generic(.., "markdown") — so prose-only .md files are the most exposed.
Repro
$ jscpd . --min-tokens 20 --min-lines 2
No duplicates found.
markdown | 2 files | 14 lines | 416 tokens | 0 clones
$ # same two files, minus one line holding `docs/**`
Clone found (markdown)
- first.md:markdown [3:1 - 6:79] (4 lines, 54 tokens)
second.md:markdown [4:1 - 7:79]
Removing a single line containing a glob is enough to flip the result from "no duplicates" to a 54-token clone.
Expected
Markdown should use CommentStyle::None.
CommonMark defines no comment syntax of its own. Its only comment is the HTML comment (HTML block type 2), which is not /* */ or //. Markdown should therefore not inherit C's comment style.
Status
A fix is ready — the missing "markdown" | "md" => CommentStyle::None arm plus unit and integration tests. PR to follow.
Problem
comment_style()inrust/crates/cpd-tokenizer/src/generic.rshas no arm for"markdown"/"md", so Markdown falls through to the catch-all_ => CommentStyle::CStyle.Markdown prose routinely contains the C comment openers:
/*— a glob such asdocs/**, a code span, or plain prose.//— any URL (https://...).Under the C-style fallback, a stray
/*opens a block comment that is never closed for the rest of the file, and//opens a line comment. Everything after the opener is silently dropped from tokenization.Nothing signals the loss: the file is still reported as analyzed, and its token total still looks plausible. Two files that share a paragraph are simply not reported as a clone if a
/*or//appears anywhere earlier in either file.This compounds with the prose-tokenization fix from #883 (already merged), which routes the Markdown prose body through
tokenize_generic(.., "markdown")— so prose-only.mdfiles are the most exposed.Repro
Removing a single line containing a glob is enough to flip the result from "no duplicates" to a 54-token clone.
Expected
Markdown should use
CommentStyle::None.CommonMark defines no comment syntax of its own. Its only comment is the HTML comment (HTML block type 2), which is not
/* */or//. Markdown should therefore not inherit C's comment style.Status
A fix is ready — the missing
"markdown" | "md" => CommentStyle::Nonearm plus unit and integration tests. PR to follow.