PlexPipe-flow: Nextflow orchestration for PlexPipe

PlexPipe-flow runs the PlexPipe multiplex imaging pipeline with Nextflow, so that tissue microarrays and whole-slide images are processed in parallel, locally or on a Slurm cluster.

This repository contains only orchestration: Nextflow processes, resources and retries. Every process calls the plexpipe command that comes with the PlexPipe package (see Command Line (plexpipe)), so the analysis code exists in one place only.

It manages:

• Resource allocation: the right CPU, memory and GPU for each step.
• Data flow: the outputs of one step become the inputs of the next.
• Fault tolerance: segmentation and quantification are retried (up to 3 times) with more resources.

The pipeline runs in two phases separated by a barrier:

• Phase 1 – cutting: one job per channel image cuts all ROIs from it.
• Barrier: .collect() waits until every channel has been cut.
• Phase 2 – per ROI: each ROI is assembled, segmented and quantified independently.

Requirements

• Nextflow (Java 17 or newer).

• PlexPipe 0.1.0 installed in an environment the jobs can use, so that plexpipe --version works there:

  pip install "plex_pipe[all]==0.1.0"

  The first job checks the version and stops with a message if it differs (see --plexpipe_version).

• A PlexPipe config YAML with absolute paths (general.image_dir, general.analysis_dir, and any other path you set), and the ROIs already defined (rois.pkl).

Installing on a cluster

Three pieces, all on the cluster's shared filesystem so that every compute node sees them:

1. Nextflow on the login node: the cluster's module (module load nextflow, if there is one) or the self-installing script (curl -s https://get.nextflow.io | bash, then move nextflow to ~/bin). Needs Java 17 or newer.

2. PlexPipe in its own environment, in your home or project space:

   uv venv ~/envs/plexpipe-0.1.0 --python 3.12
   uv pip install --python ~/envs/plexpipe-0.1.0/bin/python "plex_pipe[all]==0.1.0"

   On Linux, PyTorch from PyPI is the CUDA build, so segmentation can use the GPU nodes. Pass the full path of the command to the pipeline: --plexpipe ~/envs/plexpipe-0.1.0/bin/plexpipe.

3. The segmentation model weights, downloaded once on the login node: compute nodes often have no internet access, and InstanSeg / Cellpose download their weights on first use. Run the model once there, e.g.

   ~/envs/plexpipe-0.1.0/bin/python -c "from instanseg import InstanSeg; InstanSeg('fluorescence_nuclei_and_cells')"

   (use the model named in your config).

The pipeline code itself doesn't need installing: Nextflow fetches a tagged version from GitHub.

nextflow run plexpipe/plex-pipe-nextflow -r v0.1.0 -profile hpc,bridges2 \
    --exp_config /abs/path/to/config.yaml \
    --plexpipe ~/envs/plexpipe-0.1.0/bin/plexpipe

hpc sets the resources and retries of each step and works on any Slurm cluster. The second profile adds one cluster's settings (partitions, account, GPU request): bridges2 (PSC Bridges-2; options --account, --gpu_type) and pitt_crc are in conf/. For another cluster, write the same few lines in your own file and pass it with -c my_cluster.config instead, so you don't edit this repository. Start Nextflow inside tmux or screen, or as a small Slurm job of its own, so it keeps running after you log out.

Versions: plex-pipe-nextflow v0.1.0 works with PlexPipe 0.1.0.

Quick start

From a clone of this repository. Local run:

nextflow run main.nf -profile local \
    --exp_config /abs/path/to/config.yaml

Slurm cluster, here PSC Bridges-2 (see Installing on a cluster for other clusters):

nextflow run main.nf -profile hpc,bridges2 \
    --exp_config /abs/path/to/config.yaml \
    --plexpipe /abs/path/to/env/bin/plexpipe

With images on a remote Globus collection (see Globus Setup):

nextflow run main.nf -profile hpc,bridges2 \
    --exp_config /abs/path/to/config.yaml \
    --globus_config /abs/path/to/globus_config.yaml \
    --cleanup

Parameters

Parameter	Default	Meaning
--exp_config	(required)	PlexPipe config YAML (absolute paths).
--plexpipe	plexpipe	The plexpipe command; give the full path if it is not on PATH in the jobs.
--account / --gpu_type	your default / v100-32	Bridges-2 only: allocation to charge, GPU for segmentation.
--plexpipe_version	0.1.0	PlexPipe version required; '' skips the check (e.g. for a development install).
--globus_config	none	Globus config YAML; fetch the images from a remote collection.
--from_collection / --to_collection	remote_source / local_workstation	Collection keys in the Globus config.
--file_checksum	false	Transfer images that are already present too.
--wait_hours	12	How long a cutting job waits for its Globus transfer.
--cleanup	false	Delete each transferred image once it is cut.
--roi_cleanup	false	Delete each ROI's TIFFs once it is assembled.
--overwrite	false	Overwrite existing masks and tables. Retries always overwrite what the failed attempt wrote.
--logdir	<launch dir>/pipeline_logs	Nextflow timeline, report and trace.

PlexPipe's own logs (one per job) are written to the analysis logs folder.

Notes

• Cutting jobs are not retried. With --cleanup, the transferred image is deleted after cutting, so a retry would have no input.
• -resume and the analysis folder. PlexPipe writes the ROI stores in place in the analysis folder, and the processes pass their paths as values. -resume therefore re-runs a step only when its inputs (config, ROI list) change, not when the files in the analysis folder changed. To redo steps, use --overwrite, or run single ROIs with plexpipe directly.