samirelanduk/nf-diffuser

Introduction

samirelanduk/nf-diffuser is a generative AI pipeline that creates images from text prompts using latent diffusion. It takes a samplesheet of text prompts, with optional negative prompts, plus a Stable Diffusion 1.5 .safetensors checkpoint (either your own, or one of several popular models downloaded from Hugging Face by name), and produces one JPEG per prompt along with the intermediate latents and conditioning tensors.

In case the image above is not loading, please have a look at the static version.

1. Create a random starting latent at the requested size (pydiffuse noise create)
2. Build the noise schedule (pydiffuse noise schedule)
3. Encode the prompt and negative prompt with CLIP: tokenize, embed and encode (pydiffuse clip)
4. Denoise the latent with the UNet using classifier-free guidance (pydiffuse sample denoise)
5. Decode the denoised latent into an image with the VAE (pydiffuse vae decode)
6. Assess each image's quality and prompt adherence: basic image statistics, CLIPScore, and optionally PickScore
7. Present QC metrics, a thumbnail gallery and software versions (MultiQC)

Usage

Note

If you are new to Nextflow and nf-core, please refer to this page on how to set-up Nextflow. Make sure to test your setup with -profile test before running the workflow on actual data.

First, prepare a samplesheet with your prompts that looks as follows:

samplesheet.csv:

sample,prompt,negative_prompt,width,height,steps,cfg,sampler,schedule
TREE,"A photo of a tree","animals, people, text",768,512,30,5,heun,exponential
SKY,"A beautiful panorama of the sky",,,,,,,

Each row represents one image to generate. Only sample and prompt are required; the remaining columns can be left empty or omitted to use their defaults.

Now, you can run the pipeline using:

nextflow run samirelanduk/nf-diffuser \
   -profile <docker/singularity/conda> \
   --input samplesheet.csv \
   --outdir <OUTDIR>

Alternatively, to generate a single image with default settings, replace --input samplesheet.csv with --prompt "A photo of a tree".

Warning

Please provide pipeline parameters via the CLI or Nextflow -params-file option. Custom config files including those provided by the -c Nextflow option can be used to provide any configuration except for parameters; see docs.

For more details and further functionality, please refer to the usage documentation.

Pipeline output

For more details about the output files and reports, please refer to the output documentation.

Credits

samirelanduk/nf-diffuser was originally written by Sam M. Ireland.

Contributions and Support

If you would like to contribute to this pipeline, please see the contributing guidelines.

Citations

An extensive list of references for the tools used by the pipeline can be found in the CITATIONS.md file.

This pipeline uses code and infrastructure developed and maintained by the nf-core community, reused here under the MIT license.

The nf-core framework for community-curated bioinformatics pipelines.

Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.

Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.