[tsantalis]

Refs #485

This PR fixes the converter problems from §1 of #485 in AbstractTreeSitterNgGenerator and works around the two tree-sitter-ng binding problems from §2 on the GumTree side. The binding itself can only be fixed in bonede/tree-sitter-ng.

Changes

Performance

• The tree is converted in one TSTreeCursor traversal. The types of the ancestors are kept in a list for the ancestor rules (a b c in rules.yml). The old recursive traversal called getParent() for every node and every rule set. ts_node_parent walks down from the root each time, so the cost grew with depth × tree size. It also made several JNI calls per node (getChild(i), getType, getStartPoint, …).
• The rule sets of each language (ignored, label_ignored, flattened, aliased) are precomputed once and cached: direct types go in a hash set, plus the maximum ancestor depth. Before, every rule string was split again for every node.
• TSParser, TSTree and TSTreeCursor are closed with try-with-resources.

Correctness

• UTF-16 offsets: the UTF-8 byte offsets of tree-sitter are converted to char offsets through a table. The table is skipped for pure ASCII input. Before, the conversion only corrected for characters on previous lines, so any node after a non-ASCII character on the same line got a wrong pos/length (§1).
• Labels are the source text, content.substring(pos, end). Before, multi-line leaves and flattened nodes (Python triple-quoted strings, template literals, …) were labelled with only part of their text.
• BOM / U+FEFF: the content is parsed with TSParser.parse(byte[], …, TSInputEncodingUTF8) instead of parseString. The binding's ConvertUTF.c silently drops U+FEFF, U+FFFE and U+FFFF, which shifted every following node by 3 bytes (§2.1).

API note: the protected static helpers tsNode2GumTree, getLabel and calculateOffset were removed. No subclass in the repository uses them. matchNodeOrAncestorTypes is kept, and now only calls getParent() when no direct rule matched.

Tests

New tests in AbstractTreeSitterNgGeneratorTest. Six of them fail on main and pass with this PR:

• testMultiByteOffsetOnSameLine, testSurrogatePairOffsetOnSameLine: offsets after é / 🐍 on the same line.
• testMultiLineLabel, testMultiLineLabelCrlf: full multi-line labels, with LF and CRLF.
• testByteOrderMark: positions after a leading BOM.
• testFlattenedAndAncestorRules: flattening plus ignored/ancestor rules (default_parameter =) still apply.

Each test also checks that every leaf label equals its source range. ./gradlew :gen.treesitter-ng:check :gen.kotlin:check passes.

Experiments

Corpus: all source files of the test datasets of RefactoringMiner, from src/test/resources/oracle/commits. These are the before/after files of the commits in its Python, Kotlin, TypeScript/JavaScript and C++ refactoring oracles. 5,467 files, 12.7M GumTree nodes.

Setup:

• Apple M4 (10 cores), 32 GB, Java 25.
• tree-sitter-ng 0.26.6, the version on main.
• Each configuration runs in its own JVM, one after another, gated on load average < 3.
• Each configuration does 2 warm-up rounds and then 5 measured rounds over all files of the group.
• The table gives the median round time.
• parse = TSParser.parseString only, the lower bound.
• main = GumTreeDiff/gumtree 6e9a863.
• PR = this branch.

Language (grammar)	Files	Nodes	parse only	main	PR	Speed-up
Python	1,578	1,529,976	2.8 s	121.3 s	14.1 s	8.6×
Kotlin	1,962	1,941,050	3.0 s	171.0 s	16.9 s	10.1×
C++ (.cpp/.cc/.c/.h/.hpp)	751	6,747,935	7.3 s	113.2 s	48.7 s	2.3×
JavaScript (.js/.jsx)	563	1,324,347	1.2 s	19.5 s	9.5 s	2.1×
TypeScript	541	1,046,368	1.0 s	15.5 s	7.4 s	2.1×
TSX	72	119,193	0.1 s	1.8 s	0.9 s	2.1×
Total	5,467	12,708,869	15.4 s	442.4 s	97.5 s	4.5×

The round-to-round spread is below 1.5% in all configurations. Python and Kotlin gain the most because their rules.yml sections use ancestor rules, which triggered the quadratic getParent() walks. In the other grammars the remaining cost is mostly the JNI calls for each node.

Equivalence check: each tree was dumped in preorder as (depth, type, pos, length, label) with both versions and compared node by node:

• Tree structure and types are identical for all 5,467 files: same shape, same types, same node counts as listed above.

• pos/length differ only in files that contain non-ASCII characters or a BOM. In every such case the new range is the correct one:

  	Python	Kotlin	C++	JavaScript	TypeScript	TSX
  files with corrected offsets	70	168	42	39	179	11
  nodes with corrected offsets	1,213	3,294	512	2,266	7,163	132
  multi-line labels now complete	12,450	6,425	1,094	1,620	2,079	51

• With this PR, every leaf label equals content.substring(pos, pos + length), and every range is within the source. On main, multi-line labels were shifted text windows. For example, a Kotlin KDoc was labelled " \n * Get a Task with the given ID. Will return null i…" instead of "/**\n * Get a Task with the given ID. …". In a Python file with a BOM, the label of import was "\ufeffimp".

Not in this PR (tree-sitter-ng)

These belong in bonede/tree-sitter-ng:

• ConvertUTF.c skips 0xFEFF, 0xFFFE and 0xFFFF while converting the string, so parseString parses a different input than it was given. This PR avoids it by passing UTF-8 bytes. The fix in the binding needs its natives rebuilt.
• Every TSNode accessor first calls isNull(), which is a native call (ts_node_is_null), so each accessor costs two JNI calls. isNull() could be idPtr == 0 in Java. This PR keeps the number of accessor calls per node small, but the binding fix would help every user.

🤖 Generated with Claude Code