Database of all the Hurwitz lab's Nextflow processes — reusable tool wrappers
(process blocks) shared across every lab pipeline, so a bug fix or new
container path only has to happen in one place.
This repo holds processes only. It has no pipeline of its own, no workflow {}
entry point, and no pipeline config of its own (just a template to copy — see
templates/base.config below — plus a generated container config for Hazel,
see Containers below) — it's a library that pipeline repos (like
viral_inference_benchmark)
include from.
Clone this repo as a sibling of the pipeline repo that uses it, under the same parent directory:
some_parent_dir/
your_pipeline_repo/
nextflow_modules/
Then include a process by relative path from wherever your subworkflow or
workflow file lives, e.g. from your_pipeline_repo/nextflow_pipeline/subworkflows/:
include { genomad } from '../../../nextflow_process_library/modules/genomad/genomad.nf'Nextflow resolves include paths as literal strings at parse time — they
can't be driven by a config variable — so the exact number of ../ depends
on how deep your including file is. Count directory levels up to the shared
parent, then down into nextflow_process_library/modules/.
Pinning: cloning this repo gives you whatever commit is on main at
clone time. If you need a pipeline to keep working against a known-good
version regardless of later changes here, checkout a specific tag/commit in
your local clone rather than tracking main:
cd nextflow_process_library
git checkout <tag-or-commit>Every process reads its container, output directory, and tool options from
params.* — it never defines them itself. The consuming pipeline's own
.config (or a config it includes) must define, for each process it uses:
params.container__<tool>— path to the tool's container imageparams.<process>_outdir— where output gets published, for a process carrying apublish_final/publish_intermediatelabel (or a*_pooledvariant, for a process whose output is one file for all samples, e.g.mmseqs_easy_cluster-- its Closure receives thepooledidinstead of thesampleid). Keyed by the process name, not the file's tool name:bbmap.nf'spileuppublishes toparams.pileup_outdir. (A process with its own in-modulepublishDiruses whateverparams.*_outdirthat line names — usually<tool>_outdir.)params.<tool>_<option>— any tool-specific options the process references
The consuming pipeline is also expected to define withLabel:process_single,
withLabel:process_low, withLabel:process_medium, and
withLabel:process_high resource selectors, plus the
withLabel:publish_final/withLabel:publish_intermediate blocks (and their
*_pooled variants) that give labeled processes a publishDir — every process tags itself with one
resource label (and at most one publish label), but resource amounts are the
pipeline's call, not this repo's.
Starting point: templates/base.config defines
all eight labels. Copy it into your pipeline (e.g. as conf/base.config),
includeConfig it from your nextflow.config, and tune the numbers to what
your tools actually use (check the run's trace/report files or seff). Copy
it rather than includeConfig-ing it from this repo, so later edits here
don't change your pipeline's resources underneath it.
If a process you need references a param your pipeline's config doesn't have
yet, add it — see the process's file for the exact params.* names it expects.
Processes never name an image — each reads params.container__<tool>, so the
same .nf file works anywhere you can supply a container.
On Hazel, the BRC container repository
(/usr/local/usrapps/brc/brc_modules/images) already has most of them.
includeConfig the generated config to get every available one, at its
newest version:
// your_pipeline_repo/nextflow.config
includeConfig '../nextflow_process_library/conf/containers_hazel.config'To pin a different version for one tool, set that param after the include
(see conf/container_catalog.tsv for the versions available). Tools with no
image on Hazel yet are listed as comments at the bottom of the config — define
those yourself if you use them.
Elsewhere, define the params.container__<tool> values yourself.
conf/container_catalog.tsv has a public_uri column (e.g.
docker://quay.io/biocontainers/samtools:1.24--h9dcdb79_1) for every image
that came from a public registry. A locally built image has no public_uri;
its custom_build column instead links to the .def recipe directory in
custom_container_module
(same lookup as the BRC infra site), so you can build it yourself.
Regenerating: conf/container_catalog.tsv (one row per tool version)
and conf/containers_hazel.config (the default per tool) are both written by
python3 conf/make_containers.py, which reads the BRC modulefiles and image
directory. Images are matched to a param by exact name only; a param whose
name differs from its image's (bbmap → bbtools, gatk → gatk4), a
pinned default, or a deliberately unset param goes in the script's
OVERRIDES. Rerun it after adding a module or when new images land, and
review the diff before committing — a new image version becomes the default
only through that commit.
Tests are run with nf-test, on Hazel using local BRC images, via the
tests/run_tests.sh utility (submit from tests/):
cd tests
sbatch run_tests.sh # every test, default container versions
sbatch run_tests.sh modules/samtools # extra args pass through to nf-test (e.g: run only the samtools tests)
sbatch run_tests.sh --all-versions # each tested module, once per image version in the catalog
sbatch run_tests.sh --stale # only tests whose inputs changed since they last passedOnly re-run what changed (--stale). Each test file gets a pass record next to it,
<process>.nf.test.pass, written when it passes under --stale. The record holds a fingerprint:
a hash of the test, its snapshot, the text of the process it tests (and of any process its
setup block runs), the module's code outside process blocks, the fixtures it references, its
resolved container image, tests/nextflow.config, nf-test.config and the Nextflow/nf-test
versions run_tests.sh loads. A test is stale when it has no record or the fingerprint changed;
editing one process in samtools.nf makes only that process's test stale, and editing a comment
between processes makes none stale. Hashes are used rather than timestamps because commit dates
survive rebases and merges unchanged, and uncommitted edits have no commit date.
Commit the updated .pass files with your change — that's how others (and review) know the
change was tested. python3 tests/test_ledger.py check exits 1 if any test is stale. The
fingerprint covers the default (Hazel) image only; --all-versions runs don't write records.
Merging to main requires no stale tests. The GitHub Actions workflow
.github/workflows/stale-tests.yml runs
python3 tests/test_ledger.py check (check name No stale tests) on every PR to main and every
push to main. It only compares hashes against the committed .pass records, so it needs no
containers and takes seconds; the tests themselves still run on Hazel. On a PR it checks the PR
merged into main, so a test also goes stale when main changed something it depends on after
you ran it. Each stale test is annotated on its file in the PR, and the run's summary lists what
changed.
Two repository rulesets protect main (Settings → Rules → Rulesets), with no bypass:
| Ruleset | Enforces |
|---|---|
main: no stale tests |
No stale tests must pass, and the branch must be up to date with main before merging; no force pushes or deletion of main |
PR approval requirement |
Changes reach main through a PR with at least one approving review |
If No stale tests fails on your PR:
python3 tests/test_ledger.py status --why # from the repo root: which tests are stale, and what changed
sbatch tests/run_tests.sh --stale # run them on Hazel; passes write new .pass records
git add modules/*/tests/*.nf.test.pass && git commit -m "Update test pass records" && git pushIf GitHub says the branch is out of date, update it from main first (merge or rebase), then
re-check — main's changes can make more of your tests stale.
General testing layout The testing suite for a process has a general layout that all processes should follow:
modules/
└── <tool_name>/ # e.g: samtools
├── <tool_name>.nf # File with nextflow processes
└── tests/ # Directory containing all testing-related files
├── <process_A>.nf.test # A nf-test file, testing a single process from <tool_name>.nf
├── <process_A>.nf.test.snap # The intended output of the *.test file, used to validate correctness
├── <process_A>.nf.test.pass # A record of the last time the test passed and what was used as an input.
├── <process_B>.nf.test # A second process from the <tool_name>.nf file
├── <process_B>.nf.test.snap
├── <process_B>.nf.test.pass
└── testdata/ # Optional directory containing process-specific (not-shared) test data
Test data: shared inputs live in tests/data/ — a small,
generated "mini world" around the phiX174 genome (contigs, paired/amplicon/long reads, proteins,
transcripts, SAM/BAMs, VCFs), referenced from a test as file("${baseDir}/tests/data/<name>").
Inputs only one module needs go in modules/<tool>/tests/testdata/. Tests are one file per
process, modules/<tool>/tests/<process>.nf.test.
--all-versions requires every version to match the same snapshot, so
snapshot only what shouldn't change between tool versions — e.g. core
alignment fields rather than BAM headers (samtools appends version-stamped
@PG lines). Create or update snapshots with a default run
(sbatch run_tests.sh --update-snapshot ...); --all-versions refuses to.
modules/— the reviewed set. If a pipeline canincludeit today, it lives here.modules/dev/— staging for processes converted or written but not yet reconciled against what's already inmodules/. Nothing here is guaranteed to work, and nothing here should beincluded by a pipeline.
Promoting a process out of dev/:
- Check whether
modules/already has a file for the same tool. - No existing file — move it up as a new file.
- Existing file, different process name(s) — merge your process(es)
into the existing file as additional
processblocks (one file per tool, permodules/CLAUDE.md), then remove the file fromdev/. - Existing file, same process name — a real collision. Don't just pick one and delete the other; check whether they're actually redundant or cover different cases, and reconcile by hand (rename, merge logic, or consolidate) before promoting either.
This repo holds 100+ tool processes as of this writing — a hand-maintained
table here goes stale within a day given how often processes get added or
promoted from modules/dev/. To see what's actually available:
ls modules/*/*.nf # every process file
grep -h '^process\|^ process' modules/*/*.nf | sort # every process name, across all filesSome tools have multiple processes in one file (e.g. bbmap.nf has
bbwrap + pileup; samtools.nf has several BAM/SAM conversion steps) —
see File layout in modules/CLAUDE.md for why.
- Read
modules/CLAUDE.md— house rules for file layout, process block structure, shell interpolation style, andparams.*naming. - Copy
modules/TEMPLATE.nfas a starting point. - Work through the checklist at the bottom of
CLAUDE.mdbefore considering it done. - In your PR description, list every
params.*name the new process references, so pipeline maintainers know what to add to their own config. - Run
python3 conf/make_containers.pyso the newcontainer__<tool>param lands in the catalog and Hazel config (add anOVERRIDESentry if the image is named differently — see Containers above).
This repo gets edited concurrently — by different lab members, and often by
more than one Claude Code session at once. That's exactly how a real
process-name collision (two processes both defining a process called
vibrant) happened here. Before adding a process:
- Check
modules/*/*.nffor an existing file with the same tool name — don't assume a name is free just because your own source material doesn't mention an existing module. - If it already exists, add your process(es) to that file instead of
creating a duplicate (see Repo layout above for the full promotion
decision tree if you're coming from
modules/dev/). - Follow
modules/CLAUDE.mdandmodules/TEMPLATE.nfeither way (see Adding a new module above).
modules/bbmap.nf copy— a stray file (content differs frombbmap.nf, looks like an accidental Finder duplicate). Should be deleted or explained; it isn't a real process.modules/dev/phamb.nfandmodules/dev/viralverify.nf— both completely empty (0 bytes), placeholders with noprocessblock at all.modules/dev/virfinder.nf— unbalanced braces (8{vs. 7}), so its finalprocessblock never closes.
None of the three dev/ files above should be promoted or included as-is.
Whatever you change here — a process, CLAUDE.md, this README, anything —
record it in the notebook/ directory before you're done, not just in the commit
message. The commit says what changed; the notebook says why and what
happened (results, errors, decisions, flags for whoever picks this up next),
which is exactly the context a diff can't carry on its own.
notebook/ at the repo root is a chronological log of work sessions on this
repo — one file per session per person, named YYYY-MM-DD_<short-slug>.md,
covering what was done, what happened (concrete results/errors, not a diff
restatement), and the decision or next step. Each entry is its own file so
entries written on different branches never cause merge conflicts. Never edit
anyone else's or any past entry; write a new one instead.
If you use Claude Code, this convention is checked in as a project skill at
.claude/skills/notebook/ — anyone who clones this repo gets the same
/notebook behavior automatically, no setup needed. If you don't use Claude
Code, follow the rules and format in notebook/README.md.
If you're using Claude Code (or a similar assistant) here, /notebook alone
does this. If you're prompting a general assistant, or want to be explicit
about what you expect, something like this works:
Before we wrap up, review what we changed in this session and write one
dated entry as a new file in notebook/ at the repo root, following the rules
and format in notebook/README.md. Cover:
- what was done/tried
- what happened — concrete results, errors, numbers, not a restatement of
the diff
- any decision made or flags left for whoever works on this next
If I already have an entry file for today, add to it rather than starting a
new file. Never edit any other entry.