How does Serena compare to token-saving focus tools Headroom and RTK?

[jqnatividad]

I realize that Semantic search saves tokens as well, but these token-compression/saving tools have been getting a lot of attention of late.

Further, can Serena interoperate with them as complementary tools?

[paul-basanets]

I'd say Serena isn't about saving tokens, it's rather about better precision in code navigation and advanced edits.

RTK has no relation to Serena as it doesn't work with MCP.
I didn't use Headroom before but according to the docs it works with MCP so probably it may give some benefit. But i doubt that it can save significant amount of tokens.
I did some experiments with Serena's responses, some of the tools could benefit from more token efficient format (f.x. TOON) but in general estimated savings are almost negligible and depends on actual usage scenarios.

But maybe the authors can share more.

P.S.
Just as illustration that actual count of token spend heavily depends on specific operation, there's count of in/out token for a tool on the right side:

[opcode81]

I'd say Serena isn't about saving tokens, it's rather about better precision in code navigation and advanced edits.

Serena is very much about saving tokens, too, because better precision and code navigation frequently result in spending fewer tokens.

For example, ...

• if you can efficiently retrieve only what is needed (without reading any superfluous content), you save tokens.
• if you can reliably edit & refactor (without having to grep, check occurrences and modify only the actually matching ones), you save tokens.

More generally, if you have better tools that reliably do precisely what needs to be done (without unnecessary detours), you will save tokens.

[MischaPanch]

We are running benchmarks of serena on various real-world tasks and are planning to publish the results soon. Serena does save many tokens in many scenarios because the tools let the agent operate more efficiently.

Compressing tools like rtk are orthogonal and can be used on top. RTK saves tokens in specific cases (long shell commands), but we found that they appear fairly rarely in our benchmark and the savings of RTK were very marginal compared to savings with Serena. There is essentially no downside to just using RTK though.

Regarding headroom, I recently had a look and a lot of what it is doing is actively damaging and counterproductive in my opinion. While RTK just makes shell output shorter, headroom completely rewires what the agent sees for everything. Doing that right always requires full-blown frontier AI by itself, so it is best left to the agent than to some compressing heuristics outside of it (agents can do that very well when given the right tools). Even occasional mistakes in such compression can have absolutely devastating effects on a session, wiping out any savings from other scenarios. While we might include headroom in our benchmark, personally I would steer away from it based on what I understand about it.

[jqnatividad]

Having benchmarks would be great!

Hopefully, they get picked up on Hacker News once they're out - as that's how RTK and Headroom got viral.

As for RTK and Headroom - I tried them out, but ended up uninstalling them and just going back to using Serena.