Skip to content

About

Database of all the Hurwitz lab's Nextflow processes

Resources

Stars

0 stars

Watchers

1 watching

Forks

Repository files navigation

Nextflow Process Library

Database of all the Hurwitz lab's Nextflow processes — reusable tool wrappers (process blocks) shared across every lab pipeline, so a bug fix or new container path only has to happen in one place.

This repo holds processes only. It has no pipeline of its own, no workflow {} entry point, and no pipeline config of its own (just a template to copy — see templates/base.config below — plus a generated container config for Hazel, see Containers below) — it's a library that pipeline repos (like viral_inference_benchmark) include from.

Using these processes in a pipeline

Clone this repo as a sibling of the pipeline repo that uses it, under the same parent directory:

some_parent_dir/
  your_pipeline_repo/
  nextflow_modules/

Then include a process by relative path from wherever your subworkflow or workflow file lives, e.g. from your_pipeline_repo/nextflow_pipeline/subworkflows/:

include { genomad } from '../../../nextflow_process_library/modules/genomad/genomad.nf'

Nextflow resolves include paths as literal strings at parse time — they can't be driven by a config variable — so the exact number of ../ depends on how deep your including file is. Count directory levels up to the shared parent, then down into nextflow_process_library/modules/.

Pinning: cloning this repo gives you whatever commit is on main at clone time. If you need a pipeline to keep working against a known-good version regardless of later changes here, checkout a specific tag/commit in your local clone rather than tracking main:

cd nextflow_process_library
git checkout <tag-or-commit>

What a process expects from the pipeline that includes it

Every process reads its container, output directory, and tool options from params.* — it never defines them itself. The consuming pipeline's own .config (or a config it includes) must define, for each process it uses:

  • params.container__<tool> — path to the tool's container image
  • params.<process>_outdir — where output gets published, for a process carrying a publish_final/publish_intermediate label (or a *_pooled variant, for a process whose output is one file for all samples, e.g. mmseqs_easy_cluster -- its Closure receives the pooledid instead of the sampleid). Keyed by the process name, not the file's tool name: bbmap.nf's pileup publishes to params.pileup_outdir. (A process with its own in-module publishDir uses whatever params.*_outdir that line names — usually <tool>_outdir.)
  • params.<tool>_<option> — any tool-specific options the process references

The consuming pipeline is also expected to define withLabel:process_single, withLabel:process_low, withLabel:process_medium, and withLabel:process_high resource selectors, plus the withLabel:publish_final/withLabel:publish_intermediate blocks (and their *_pooled variants) that give labeled processes a publishDir — every process tags itself with one resource label (and at most one publish label), but resource amounts are the pipeline's call, not this repo's.

Starting point: templates/base.config defines all eight labels. Copy it into your pipeline (e.g. as conf/base.config), includeConfig it from your nextflow.config, and tune the numbers to what your tools actually use (check the run's trace/report files or seff). Copy it rather than includeConfig-ing it from this repo, so later edits here don't change your pipeline's resources underneath it.

If a process you need references a param your pipeline's config doesn't have yet, add it — see the process's file for the exact params.* names it expects.

Containers

Processes never name an image — each reads params.container__<tool>, so the same .nf file works anywhere you can supply a container.

On Hazel, the BRC container repository (/usr/local/usrapps/brc/brc_modules/images) already has most of them. includeConfig the generated config to get every available one, at its newest version:

// your_pipeline_repo/nextflow.config
includeConfig '../nextflow_process_library/conf/containers_hazel.config'

To pin a different version for one tool, set that param after the include (see conf/container_catalog.tsv for the versions available). Tools with no image on Hazel yet are listed as comments at the bottom of the config — define those yourself if you use them.

Elsewhere, define the params.container__<tool> values yourself. conf/container_catalog.tsv has a public_uri column (e.g. docker://quay.io/biocontainers/samtools:1.24--h9dcdb79_1) for every image that came from a public registry. A locally built image has no public_uri; its custom_build column instead links to the .def recipe directory in custom_container_module (same lookup as the BRC infra site), so you can build it yourself.

Regenerating: conf/container_catalog.tsv (one row per tool version) and conf/containers_hazel.config (the default per tool) are both written by python3 conf/make_containers.py, which reads the BRC modulefiles and image directory. Images are matched to a param by exact name only; a param whose name differs from its image's (bbmap → bbtools, gatk → gatk4), a pinned default, or a deliberately unset param goes in the script's OVERRIDES. Rerun it after adding a module or when new images land, and review the diff before committing — a new image version becomes the default only through that commit.

Testing

Tests are run with nf-test, on Hazel using local BRC images, via the tests/run_tests.sh utility (submit from tests/):

cd tests
sbatch run_tests.sh                    # every test, default container versions
sbatch run_tests.sh modules/samtools   # extra args pass through to nf-test (e.g: run only the samtools tests)
sbatch run_tests.sh --all-versions     # each tested module, once per image version in the catalog
sbatch run_tests.sh --stale            # only tests whose inputs changed since they last passed

Only re-run what changed (--stale). Each test file gets a pass record next to it, <process>.nf.test.pass, written when it passes under --stale. The record holds a fingerprint: a hash of the test, its snapshot, the text of the process it tests (and of any process its setup block runs), the module's code outside process blocks, the fixtures it references, its resolved container image, tests/nextflow.config, nf-test.config and the Nextflow/nf-test versions run_tests.sh loads. A test is stale when it has no record or the fingerprint changed; editing one process in samtools.nf makes only that process's test stale, and editing a comment between processes makes none stale. Hashes are used rather than timestamps because commit dates survive rebases and merges unchanged, and uncommitted edits have no commit date.

Commit the updated .pass files with your change — that's how others (and review) know the change was tested. python3 tests/test_ledger.py check exits 1 if any test is stale. The fingerprint covers the default (Hazel) image only; --all-versions runs don't write records.

Merging to main requires no stale tests. The GitHub Actions workflow .github/workflows/stale-tests.yml runs python3 tests/test_ledger.py check (check name No stale tests) on every PR to main and every push to main. It only compares hashes against the committed .pass records, so it needs no containers and takes seconds; the tests themselves still run on Hazel. On a PR it checks the PR merged into main, so a test also goes stale when main changed something it depends on after you ran it. Each stale test is annotated on its file in the PR, and the run's summary lists what changed.

Two repository rulesets protect main (Settings → Rules → Rulesets), with no bypass:

Ruleset Enforces
main: no stale tests No stale tests must pass, and the branch must be up to date with main before merging; no force pushes or deletion of main
PR approval requirement Changes reach main through a PR with at least one approving review

If No stale tests fails on your PR:

python3 tests/test_ledger.py status --why    # from the repo root: which tests are stale, and what changed
sbatch tests/run_tests.sh --stale            # run them on Hazel; passes write new .pass records
git add modules/*/tests/*.nf.test.pass && git commit -m "Update test pass records" && git push

If GitHub says the branch is out of date, update it from main first (merge or rebase), then re-check — main's changes can make more of your tests stale.

General testing layout The testing suite for a process has a general layout that all processes should follow:

modules/
   └── <tool_name>/                               # e.g: samtools
            ├── <tool_name>.nf                    # File with nextflow processes
            └── tests/                            # Directory containing all testing-related files
                  ├── <process_A>.nf.test         # A nf-test file, testing a single process from <tool_name>.nf 
                  ├── <process_A>.nf.test.snap    # The intended output of the *.test file, used to validate correctness
                  ├── <process_A>.nf.test.pass    # A record of the last time the test passed and what was used as an input.
                  ├── <process_B>.nf.test         # A second process from the <tool_name>.nf file
                  ├── <process_B>.nf.test.snap    
                  ├── <process_B>.nf.test.pass
                  └── testdata/                   # Optional directory containing process-specific (not-shared) test data 

Test data: shared inputs live in tests/data/ — a small, generated "mini world" around the phiX174 genome (contigs, paired/amplicon/long reads, proteins, transcripts, SAM/BAMs, VCFs), referenced from a test as file("${baseDir}/tests/data/<name>"). Inputs only one module needs go in modules/<tool>/tests/testdata/. Tests are one file per process, modules/<tool>/tests/<process>.nf.test.

--all-versions requires every version to match the same snapshot, so snapshot only what shouldn't change between tool versions — e.g. core alignment fields rather than BAM headers (samtools appends version-stamped @PG lines). Create or update snapshots with a default run (sbatch run_tests.sh --update-snapshot ...); --all-versions refuses to.

Repo layout: modules/ vs modules/dev/

  • modules/ — the reviewed set. If a pipeline can include it today, it lives here.
  • modules/dev/ — staging for processes converted or written but not yet reconciled against what's already in modules/. Nothing here is guaranteed to work, and nothing here should be included by a pipeline.

Promoting a process out of dev/:

  1. Check whether modules/ already has a file for the same tool.
  2. No existing file — move it up as a new file.
  3. Existing file, different process name(s) — merge your process(es) into the existing file as additional process blocks (one file per tool, per modules/CLAUDE.md), then remove the file from dev/.
  4. Existing file, same process name — a real collision. Don't just pick one and delete the other; check whether they're actually redundant or cover different cases, and reconcile by hand (rename, merge logic, or consolidate) before promoting either.

Available Processes

This repo holds 100+ tool processes as of this writing — a hand-maintained table here goes stale within a day given how often processes get added or promoted from modules/dev/. To see what's actually available:

ls modules/*/*.nf                                   # every process file
grep -h '^process\|^ process' modules/*/*.nf | sort  # every process name, across all files

Some tools have multiple processes in one file (e.g. bbmap.nf has bbwrap + pileup; samtools.nf has several BAM/SAM conversion steps) — see File layout in modules/CLAUDE.md for why.

Adding a new process

  1. Read modules/CLAUDE.md — house rules for file layout, process block structure, shell interpolation style, and params.* naming.
  2. Copy modules/TEMPLATE.nf as a starting point.
  3. Work through the checklist at the bottom of CLAUDE.md before considering it done.
  4. In your PR description, list every params.* name the new process references, so pipeline maintainers know what to add to their own config.
  5. Run python3 conf/make_containers.py so the new container__<tool> param lands in the catalog and Hazel config (add an OVERRIDES entry if the image is named differently — see Containers above).

Contributing

This repo gets edited concurrently — by different lab members, and often by more than one Claude Code session at once. That's exactly how a real process-name collision (two processes both defining a process called vibrant) happened here. Before adding a process:

  1. Check modules/*/*.nf for an existing file with the same tool name — don't assume a name is free just because your own source material doesn't mention an existing module.
  2. If it already exists, add your process(es) to that file instead of creating a duplicate (see Repo layout above for the full promotion decision tree if you're coming from modules/dev/).
  3. Follow modules/CLAUDE.md and modules/TEMPLATE.nf either way (see Adding a new module above).

Known issues

  • modules/bbmap.nf copy — a stray file (content differs from bbmap.nf, looks like an accidental Finder duplicate). Should be deleted or explained; it isn't a real process.
  • modules/dev/phamb.nf and modules/dev/viralverify.nf — both completely empty (0 bytes), placeholders with no process block at all.
  • modules/dev/virfinder.nf — unbalanced braces (8 { vs. 7 }), so its final process block never closes.

None of the three dev/ files above should be promoted or included as-is.

Working on this repo

Whatever you change here — a process, CLAUDE.md, this README, anything — record it in the notebook/ directory before you're done, not just in the commit message. The commit says what changed; the notebook says why and what happened (results, errors, decisions, flags for whoever picks this up next), which is exactly the context a diff can't carry on its own.

Lab notebook conventions

notebook/ at the repo root is a chronological log of work sessions on this repo — one file per session per person, named YYYY-MM-DD_<short-slug>.md, covering what was done, what happened (concrete results/errors, not a diff restatement), and the decision or next step. Each entry is its own file so entries written on different branches never cause merge conflicts. Never edit anyone else's or any past entry; write a new one instead.

If you use Claude Code, this convention is checked in as a project skill at .claude/skills/notebook/ — anyone who clones this repo gets the same /notebook behavior automatically, no setup needed. If you don't use Claude Code, follow the rules and format in notebook/README.md.

A prompt to record your session

If you're using Claude Code (or a similar assistant) here, /notebook alone does this. If you're prompting a general assistant, or want to be explicit about what you expect, something like this works:

Before we wrap up, review what we changed in this session and write one
dated entry as a new file in notebook/ at the repo root, following the rules
and format in notebook/README.md. Cover:
- what was done/tried
- what happened — concrete results, errors, numbers, not a restatement of
  the diff
- any decision made or flags left for whoever works on this next

If I already have an entry file for today, add to it rather than starting a
new file. Never edit any other entry.

About

Database of all the Hurwitz lab's Nextflow processes

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages