Skip to content

perf(graph): cache the visit order of graphs - #1053

Open
marco-c wants to merge 1 commit into
taskcluster:mainfrom
marco-c:cache_visit_order
Open

marco-c wants to merge 1 commit into
taskcluster:mainfrom
marco-c:cache_visit_order

Conversation

@marco-c

@marco-c marco-c commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Graphs are immutable, but visit_postorder and visit_preorder sorted them topologically again every time they were called. The full task graph is visited once per registered verification (11 times in taskgraph alone), then again to serialize it. The target task graph is visited several times during optimization. Now the order is computed once per graph and direction, and cached like links_and_reverse_links_dict already is.

Also, during optimization:

  • index paths are gathered by iterating over the tasks directly, as the order doesn't matter;
  • remove_tasks uses the cached reverse links instead of building them again.

On a synthetic graph of 20,210 tasks and 40,200 edges, verifying the full task graph takes 0.34s instead of 0.73s, and optimizing it takes 1.57s instead of 2.19s.

@marco-c
marco-c requested a review from a team as a code owner September 29, 2026 13:43
@marco-c
marco-c requested a review from jcristau September 29, 2026 13:43
@codspeed

codspeed Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 24 untouched benchmarks
🆕 4 new benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
🆕 test_for_each_task_repeated[btree] N/A 668.1 ms N/A
🆕 test_for_each_task_repeated[diamond] N/A 3 s N/A
🆕 test_for_each_task_repeated[fan] N/A 697.4 ms N/A
🆕 test_for_each_task_repeated[linear] N/A 665.2 ms N/A

Comparing marco-c:cache_visit_order (2894a55) with main (c6f48bb)

Open in CodSpeed

Graphs are immutable, but visit_postorder and visit_preorder sorted them
topologically again every time they were called. The full task graph is
visited once per registered verification (11 times in taskgraph alone),
then again to serialize it. The target task graph is visited several
times during optimization. Now the order is computed once per graph and
direction, and cached like links_and_reverse_links_dict already is.

Also, during optimization:
- index paths are gathered by iterating over the tasks directly, as the
  order doesn't matter;
- remove_tasks uses the cached reverse links instead of building them
  again.

On a synthetic graph of 20,210 tasks and 40,200 edges, verifying the
full task graph takes 0.34s instead of 0.73s, and optimizing it takes
1.57s instead of 2.19s.
@marco-c

marco-c commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

The codspeed results were overstating things because the cache was not cleared during those tests.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant