Skip to content

Protein trace: record what happens to every input protein - #106

Merged
alpae merged 5 commits into
devfrom
feat/protein-trace
Oct 22, 2026
Merged

alpae merged 5 commits into
devfrom
feat/protein-trace

Conversation

@ali-yz

@ali-yz ali-yz commented Oct 19, 2026 •

Copy link
Copy Markdown

Adds an optional protein trace (--protein_trace true) that records, for every input protein, what happened to it during the run, and writes:

  • protein_fates.tsv: one row per input protein with exactly one final fate (e.g. in_orthoxml, too_short, lost_in_split_hog, singleton_roothog_dropped, ...)
  • protein_events.tsv.gz: all recorded events, sorted by species and protein
  • protein_trace_summary.txt: counts per fate, the rootHOGs that lose the most proteins, and consistency checks

How it works

  • infer_roothogs and infer_subhogs write per-task protein_events_*.events files, but only when the new --protein-trace option is given. Nextflow passes it when params.protein_trace is set; it is off by default, and then nothing is written and no extra work is done.
  • A new protein_trace process merges the event files at the end. It sorts them on disk (GNU sort) and handles one protein at a time, so memory stays at a few hundred bytes per protein. Its memory request scales with the size of the event files (conf/base.config).

How to use it

To turn it on, add --protein_trace true to your Nextflow command. e.g.

nextflow run FastOMA.nf -profile docker \
    --container_version "sha-$(git rev-list --max-count=1 --abbrev-commit HEAD)" \
    --input testdata/in_folder \
    --output_folder myresult/
    --protein_trace true

@ali-yz
ali-yz force-pushed the feat/protein-trace branch from adcb620 to c4bda92 Compare October 19, 2026 14:02
@ali-yz
ali-yz requested a review from alpae October 19, 2026 15:58

@alpae alpae left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @ali-yz ,

PR looks already quite good. I found 3 points that should be fixed, (two easy, one might need a bit more thinking).

I hope it's clear what I mean, otherwise ping me, np!

Comment thread conf/base.config Outdated
Comment thread FastOMA/_utils_roothog.py
Comment thread FastOMA/protein_trace.py Outdated
@alpae

alpae commented Oct 20, 2026

Copy link
Copy Markdown
Member

the broken tests are unrelated. I fixed them in #107. If you rebase, things should become green again ;-)

- Event logging in infer_roothogs and infer_subhogs is now off unless
  --protein-trace is given; Nextflow passes it when params.protein_trace
  is set. When off, no .events files are written and the extra
  bookkeeping (rootHOG assignment dicts, per-level member sets) is skipped.
- fastoma-protein-trace sorts the event files on disk (GNU sort, stable)
  and processes one protein at a time instead of holding all events in
  memory; repeated strings (species, step, event, rootHOG) are interned.
  Peak memory on 500k synthetic proteins: 949 MB -> 224 MB, same output.
  protein_fates.tsv is now sorted by species and protein.
- New withName "protein_trace" resources scaled by the event file size.
@ali-yz
ali-yz force-pushed the feat/protein-trace branch from c4bda92 to a9dd74d Compare October 22, 2026 09:06
@ali-yz

ali-yz commented Oct 22, 2026

Copy link
Copy Markdown
Author

@alpae Thank you so much for your comments, I believe they are addressed now. You can go ahead with the merge whenever ready.

@alpae
alpae merged commit f0d065c into dev Oct 22, 2026
24 checks passed
@alpae
alpae deleted the feat/protein-trace branch October 22, 2026 11:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants