Repository navigation
Conversation
|
Superseded: the branch was rewritten, see the comment below. |
OptimizerTest.MultiTableExtendOverMultiTableExtend expects E(t)Filter()E(m)Filter()S(p) for MATCH (p:P)-[r:p_x|:p_y]->(m)-[:grp]->(t) WHERE p.id > 0 AND r.w % 2 = 0, the plan the two *AfterMultiTableExtendRefill cases of match/fork_extend.test exercise. A plan that scans m first and extends to p costs exactly as much, so the chain wins only by enumeration order, and any change to either plan's estimates can flip it. Ten P rows now fail p.id > 0, one of them with 200 edges to new Y nodes. The scan-first plan extends from all of them, while the chain filters p first. The results are unchanged: no new row passes the filter, and the Y nodes have no grp edges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An extend from an unflat bound group flattens that group in its schema without a FLATTEN operator, and the group's multiplier, an earlier hop's fan-out, was lost with it. Flat estimates multiply a plan's cardinality by the multipliers of its unflat groups, so after two hops they counted only the last one. For LDBC IC7 the lookup-join gate saw 1 outer row instead of about 220 and chose a per-row primary-key lookup on liker, after which the next MATCH hashed every like in the graph. The extend now multiplies the bound group's multiplier into the neighbour group's. Its cardinality stays the factorized input, so this hop's rate is not counted twice, and a flat bound group's multiplier, already counted by a FLATTEN or carried by an earlier extend from the same node, is not carried again. OptimizerTest.LookupJoinCountsEarlierHopFanOut plans IC7's shape over 100 middle nodes with 20 edges each. Before this change the gate counts 20 probes and chooses the lookup; now it counts 2,000, over the cap, and the plan joins by hash join. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d5a82c0 to
7e9fb28
Compare
|
The branch is now two commits on current The join enumerator reaches a node an extend has bound through a join with that node's bare scan, which
|
zahariash
changed the title
This work was produced with the help of language models.
Part of #1411.
LDBC IC7 on SF1 is the query #1457 finds 11.4x slower than lance-csr. Its first
MATCH,(person:Person {ID: $personId})<-[:postHasCreator|:commentHasCreator]-(message:Post:Comment)<-[like:likePost|:likeComment]-(liker:Person), was planned as if the person had one message, so the lookup-join gate of #1370 saw 1 outer row instead of 220, chose a per-rowQUERY_PRIMARY_KEY_LOOKUPonliker, and the nextMATCHhashed every like in the graph. With the first hop counted, IC7 joinslikerby hash join and the nextMATCHmasks itsPersonscans. In the benchmark's own harness on 4 pinned cores, IC7 now takes 45 ms, against 952 ms onmainand 42 ms for lance-csr 0.2.0.Fix.
LogicalExtendflattens an unflat bound group in its schema without aFLATTENoperator, andappendNonRecursiveExtenddropped that group's multiplier, an earlier hop's fan-out. It now multiplies it into the neighbour group's multiplier, so flat estimates (FLATTEN, hash join build keys, lookup join probes) count it. The extend's cardinality stays the factorized input, so the double count #1065 removed stays removed, and a flat bound group's multiplier, already counted, is not carried again.Left open in #1411. The extend's cost still uses its factorized input, and
appendRecursiveExtendleaves out an earlier hop's fan-out the same way. Counting the flattened rows in the extend's cardinality instead makes chains of extends lose to hash joins they do not need: the join enumerator charges that cardinality to the join with a bound node's bare scan, whichRemoveUnnecessaryJoinOptimizerremoves afterwards.Plans. 8 of 58 LDBC SF1 plans change (IC1–IC14, the complex14 set and the 30-query suite as qN), all with the same rows; none of the 9 LSQB SF0.1 plans do. IC8 and complex14 q8 only swap their two hash joins. Medians of 9 runs on 4 pinned cores in two rounds, and malloc calls per execution at one thread:
main, msmainIC4 and complex14 q4 keep
main's plans, malloc calls and instructions.Cost. q7 and complex14 q7 lose primary-key lookups; q7's estimate rises from 1 to 207 rows, and 207 × 40.6 exceeds the 1,460 places, so the gate rejects its
Placelookup. complex14 q7 runs 8.7% more instructions and takes 4 to 6 ms longer, q7 1.1% more. A runtime fallback such as #1307 proposes could keep these lookups. q25 runs half again as many instructions, about 0.2 ms.Tests. The first commit only changes test data.
OptimizerTest.MultiTableExtendOverMultiTableExtendand the two*AfterMultiTableExtendRefillcases ofmatch/fork_extend.testrely on a chain plan that ties at equal cost with a plan that scansmfirst, and counting the earlier hop's fan-out tips the tie; tenProws that fail the filter, one with 200 edges, give the chain a clear margin with the same results. In the second commit,OptimizerTest.LookupJoinCountsEarlierHopFanOutplans IC7's shape over 100 middle nodes with 20 edges each:maincounts 20 probes and chooses the lookup, this PR counts 2,000, over the cap, and joins by hash join. Full ctest passes, 3,443 tests.EXPLAINplan diffs,mainto this PR, operator IDs removed: IC7, q7, complex14 q7