Skip to content
excsnPublic

About

High Performance, Rust Native, Deduplicating Build Cache

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

51 Commits

Folders and files

Repository files navigation

DamasMus - High Performance, Rust Native, Deduplicating Build Cache

damasmus is a rustc cache and Cargo target dir deduplicator written in Rust. It caches library, proc-macro, build-script and incremental compiles in one store that every project on the machine shares. It also replaces duplicate build artifacts with copy-on-write clones.

A second target dir of snapfire's FSR workspace, about 800 crates, builds in 19.7 s with DamasMus, against 60.1 s with no wrapper and 42.8 s with sccache. Proc-macros and incremental crates come from the store as well, which sccache does not cache at all. A checkout of the same workspace at another path hits too.

Dedup freed 5.8 GiB across 29 target dirs on a working machine without Cargo rebuilding anything afterwards.

Performance Highlights

Apple M4 Pro, high power mode, rustc 1.98.1, sccache 0.18.0. Setup, cache counts, store size and dedup numbers are in BENCHMARKS.md.

Build No wrapper sccache DamasMus
damasmus, empty cache 11.30 s 12.96 s 9.87 s
damasmus, second target dir 10.69 s 10.65 s 5.61 s
fst_incremental tests, empty cache 8.92 s 8.98 s 7.39 s
fst_incremental tests, second target dir 9.11 s 7.38 s 4.02 s
snapfire fsr, empty cache 58.25 s 58.98 s 49.79 s
snapfire fsr, second target dir 60.11 s 42.78 s 19.71 s
snapfire fsr, checkout at another path 60.11 s 42.91 s 29.08 s
Edit snapfire_fsr_ir after a hit 9.22 s 13.39 s 10.81 s
Edit it again 11.09 s 12.44 s 8.59 s
Revert the edit 12.87 s 11.39 s 3.25 s

Install

cargo install damasmus
damasmus install --server

damasmus install makes it Cargo's rustc-wrapper in ~/.cargo/config.toml. --server runs the cache server under launchd or systemd --user. Without it the first build starts a server on its own. --replace swaps out a wrapper that is already set, such as sccache. damasmus uninstall puts it back.

From a checkout, scripts/install.py --server builds it, installs it to ~/.local/bin and runs the same install.

Runs on macOS (APFS) and Linux (Btrfs, XFS).

What it does

  • Measures every Cargo target dir under the roots you give it: where the bytes go and how many are duplicates.
  • Dedups identical artifacts across target dirs by replacing them with copy-on-write clones. Cargo still sees the target dirs as fresh.
  • Prunes the incremental state of crates you haven't built in a while.
  • Caches rustc as Cargo's RUSTC_WRAPPER. Library, proc-macro and build-script compiles, incremental ones included, go into one store that every project on the machine shares, so a dependency built once is reused everywhere.

Commands

Each command has its own chapter in README.USAGE.md.

To Run
See where the disk under your target dirs goes damasmus measure
Free the space duplicate artifacts take damasmus dedup
Free incremental state of crates you no longer build damasmus prune
Stop recompiling the same dependencies in every project damasmus install
Keep the cache server running from login damasmus install --server
Go back to the wrapper you had before damasmus uninstall
Use the cache in CI or a container without a server DAMASMUS_MODE=local
See hits and misses damasmus cache stats
See why a crate missed damasmus cache explain <crate>
Keep the store under a size damasmus cache gc or the server's own GC
Check or stop the cache server damasmus server --status, damasmus server --stop

Compared with sccache

  • sccache does not cache proc-macros or incremental compiles. DamasMus caches both.
  • sccache's server has frozen mid-build here and needed a restart.
  • DamasMus remaps the workspace and the target dir in rustc's output on its own, so a checkout at another path hits. sccache needs SCCACHE_BASEDIRS set to match paths across checkouts.

Limits

  • The first edit after a hit is slower once. The store restores each crate's incremental session, so rustc reuses its type checking, but it redoes code generation for the edited crate and every crate downstream of it.
  • Crates a workspace takes by path from outside itself compile again in a checkout at another path, with every crate that depends on them. Cargo derives their -C metadata from where they sit.
  • Debug info and panic messages write the workspace path as . and the target dir as target/<profile>. A debugger finds the sources when started from the workspace root.
  • Binaries pass straight to rustc. Only build scripts built without debug info are cached.

About

High Performance, Rust Native, Deduplicating Build Cache

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages