Multi-animal offline analysis: tracking, ethograms, labelling and social features - #180
Merged
Merged
Conversation
PoseData stays single-animal. PoseTracks holds N of them and refuses any set that cannot describe one video -- mismatched lengths, mismatched keypoints, or slot ids with a gap in them. Each of those would otherwise surface as a silent misalignment much further downstream, where the individuals named in a DLC CSV no longer mean what they say.
A tracker asked to follow two mice for twenty minutes produces dozens of ids, and 'animal 0' has to mean one mouse for the whole video rather than for the length of one fragment. Seeding is the first half: drop what is too short to be an animal, take the N longest, and number them by first appearance so two runs of the same video agree about who animal 0 was.
Replace bare constraint restatement with actionable error that shows which names differ and explains why the mismatch fails: animals whose keypoints do not line up cannot be compared, written to one DLC CSV, or fed to one model.
The limit is a speed rather than a distance, because an animal out of view for a second is legitimately further away than one out of view for a frame. Overlap is refused outright: one animal cannot be in two places, and that constraint is the whole value of knowing the count up front.
Greedy and longest-first, with the ceiling named in the docstring rather than discovered later. A fragment no slot will take is recorded, not discarded: a video that drops many of them is one whose tuning is wrong, and that should be visible without re-running an hour of inference. Asking for two animals and finding one returns an empty second slot. That is a real answer -- one mouse was never seen -- and a far better one than raising after the inference is already paid for.
Consolidation joins fragments across gaps on the strength of a plausible speed. That beats dropping them, and it is also exactly where a swap gets baked in, so the frames it happened on are written down: close when two animals were near enough to confuse, stitched when the link was inferred, gap when nothing was seen. Sparse on purpose -- a row means doubt, and absent means measured. The column takes its name from the live tracking CSV's, because the same word should not mean two things across two files an analyst joins.
The _centroids helper hand-rolls NaN-aware mean over keypoints to avoid RuntimeWarnings on all-NaN rows. Prior tests used a helper that broadcasts a single (x,y) pair to all keypoints, so no test could catch errors like dividing by fixed keypoint count instead of per-frame finite count. Add two direct PoseData construction tests: - One finite, one NaN keypoint: centroid must equal the finite one exactly - Multiple finite keypoints at different positions, some NaN: centroid must be the mean of the finite ones Both import _centroids directly and assert on computed values, not isfinite. Also fix module docstring: the claim that identity_flag matches the live tracking CSV column is false on this branch (that column exists only on the shelved claude/multi-animal-topdown-spec branch). Replace with forward-looking justification: the name pre-aligns with that branch if revived, so the two files join on one vocabulary.
Four header rows rather than three, with individuals between scorer and bodyparts. This is the interop artifact -- SimBA, DLC and Keypoint-MoSeq read it directly -- so its shape is not ours to improvise. The single-animal writer is untouched, and a PoseTracks of one still writes four rows: which path produced a file should stay legible from the file itself.
…elpers Review found write_tracks_meta carried arena_gate through from tracks.metadata (the container), but every producer sets it on the first animal's PoseData.metadata instead -- the same place resolution is already read from. No PoseTracks any code produces could ever carry the block, silently disabling the gate-mismatch check that cohort_speed and classify rely on. Also extracts the x/y/likelihood interleave shared by _build_dataframe and _build_dataframe_multi into _interleave_xyc, and the resolution-parsing and best-effort sidecar write shared by write_pose_meta and write_tracks_meta into _validated_resolution and _write_meta_payload. write_pose_meta's output is unchanged -- verified by the roundtrip/meta tests, run unmodified.
Header depth is decided from the second line rather than by counting, and a three-row file reads exactly as it always has -- every pose CSV in every existing project is three rows. A multi-animal file with no individual= raises. Handing back the first animal would be indistinguishable from a single-animal read, and silently analysing one arbitrary mouse out of a social recording is the exact failure this feature exists to end.
_pick_candidate becomes the first element of a ranking, so the single-animal path is unchanged and stays obviously unchanged, and the multi-animal path keeps the best N of the same ordering -- in-arena first, then by confidence. A bench-floor blob still loses to real animals, and a frame that had a usable detection still cannot become a dropout. Refused by name for DeepLabCut and SLEAP: those are single-instance here and have nowhere to put a second animal, so answering with one arbitrary mouse in slot 0 would be worse than stopping.
…oint validation The arena-validation try/except appeared in both infer_video and infer_video_tracks with slightly different wording, inviting drift. Extracted into _resolve_arena_for_video to validate once and use the same warning text everywhere. Also: move the empty keypoint_names check to after model identification (so DeepLabCut and SLEAP models aren't told "is a YOLO checkpoint") and remove the stale single-animal docstring line that misrepresented what the module does.
zone_events.csv has carried an object_id since it was written; occupancy did not, so two animals in one arena produced one set of dwell times with no way to tell whose. Both writers now lead with the animal, and the single-animal path writes the same 'subject' it always wrote to events. The live/offline-video runner counted a frame as "in zone" once per zone regardless of how many tracked animals were in it, which undercounts as soon as two share a zone; occupancy there is now tallied per (zone, track) to match the enter/exit events it already logs per track. docs-site and the two tests that read the occupancy CSV by column position are updated for the new leading column.
A zone nobody ever entered had silently stopped appearing in zone_occupancy.csv: _write_occupancy nested its loop under frames_in_zone[zone.id], so an empty per-track dict wrote no row at all, making "never entered" indistinguishable from "never configured" - and that's often the headline result of an assay. Restored the one-row-per-configured-zone guarantee: an empty zone now writes frames_in_zone=0, seconds=0.000, object_id="". Also, three more review findings on the same change: - Added a two-simultaneous-track test double so the per-track occupancy tally (the reason this task exists) actually gets exercised by two TrackedObjects with distinct track_ids sharing a zone. - ZoneScoring now carries its own object_id, set once in score_pose, instead of zone_scoring.write_zone_csvs re-deriving it from the first zone event - which defaulted a named animal back to "subject" whenever it never entered a zone (empty events list). - The existing "no zone" test only checked the CSV existed, which wouldn't have caught the mislabeling above; it now asserts the row content, and uses the file's OUTSIDE constant instead of a raw literal.
run_batch now accepts n_animals (default 1). When n_animals=1, the original single-animal path runs unchanged, keeping existing projects' output byte-identical. When n_animals > 1, each video is tracked into N animals and written as one four-row DLC CSV plus an identity sidecar. Each animal is gated, filtered and zone-scored independently; the new _process_multi and _score_zones_multi functions mirror the single-animal paths (gate_to_arena, smooth, zone scoring) one slot at a time.
…h(n_animals=2) The original 5 multi-animal tests all called run_batch without gate=, filtering=, or zones=, so _process_multi's riskiest paths -- the per-slot gate_to_arena loop and PoseTracks rebuild, the per-slot smooth rebuild, _raw companion ordering, and identity_min_separation_px -- shipped with no regression protection. Production code was reviewed and is unchanged; this adds the missing coverage, mirroring test_batch.py's single-animal gating idiom.
Not detected, and deliberately so: the count is the constraint that lets tracking stitch a fragmented track back into one animal, and a guessed one would be worse than no constraint at all. The tooltip says what a wrong value looks like rather than restating the field name -- too high splits one mouse into two that each vanish for half the video.
The tuning table described min_fragment_frames as a seeding knob -- short fragments "can still be joined onto a slot, but never seed one". It is the reverse: seed_slots filters by length before it splits seeds from the rest, so a fragment under the floor is offered to nothing and joins nothing. A reader raising it to make seeding stricter was in fact discarding tracker output, and the surrounding prose agreed with the wrong version. Worse, nothing counts what it discards. Only fragments that assignment_cost rejected reach the dropped list the sidecar reports, so a video quietly losing short fragments to the floor looks clean in its own metadata. That limitation is now stated where the count is described, rather than left for someone to find in a cohort that did not reproduce. Also: the multi-animal CSV example was missing its likelihood column, and the changelog implied DeepLabCut and SLEAP had always been refused by name when the refusal itself is new here -- the architecture is what is old.
- CHANGELOG: document zone_occupancy.csv's schema break under ### Changed (object_id column order, per-(zone,track) rows, NaN-empty object_id on a zero row), not buried under ### Added. - Multi-animal inference now reports a real progress total instead of a permanent indeterminate spinner: infer_video's frame-count probe is lifted into _probe_frame_count and shared with infer_video_tracks, which also gains a tqdm bar when progress=True. - _score_zones_multi reads resolution from tracks[0].metadata, matching its single-animal twin and write_tracks_meta instead of the container level; added run_batch coverage asserting zone_occupancy.csv carries rows for every animal, which nothing previously exercised. - gate_pose_csv now refuses a multi-animal CSV with a GUI-readable message (names the animals, drops the individual= kwarg instruction that a lab user has no way to act on); docs name what else can't yet read these files and say it's planned, not a bug. - consolidate() now counts fragments dropped for falling under min_fragment_frames (ConsolidationResult.below_floor), surfaced in infer_video_tracks's consolidation metadata so a re-run that only changes the floor is distinguishable after the fact; docs updated to match. - write_zone_csvs delegates to write_zone_csvs_multi instead of duplicating ~35 lines of row formatting; single-animal gate tests pass unmodified.
A multi-animal batch wrote a single four-row DeepLabCut CSV, and nothing in GLIDER could open it: seventeen from_dlc_csv call sites refuse a four-row file rather than guess which animal you meant, which is right and also left the rest of the application unable to read what the new button produced. Each animal now gets its own three-row CSV under <stem>_animals/, in the format every tool already reads. One animal, one file, one source of truth -- the earlier design put per-animal files beside the four-row one and had to warn that the two copies could drift apart. The subdirectory is invisible to both discovery paths on purpose: find_pose_csv globs flat, and the cohort collector needs DLC_ in the stem, which animal0.csv does not have. Those are two independent conditions in two files, and the tests pin both. n_animals=1 is untouched: one three-row CSV at the same path, no subdirectory. The _raw companion stays four-row, being a diagnostic rather than something a tool opens.
…shaped run left A video tracked single-animal and later multi-animal (or the reverse) left both outputs on disk, so find_pose_csv could hand back a superseded track as current. The skip/resume check also tested primary.exists(), which _process_multi never writes, so a multi-animal rerun never skipped and switching a video to multi-animal with the default overwrite=False silently wrote nothing while reporting success. Shrinking n_animals also orphaned the dropped slot's CSV, since the per-slot write loop only overwrites slots the new run produces. Reconcile at write time, mirroring _drop_stale_ungated: drop the stale counterpart (primary+companions, or animals_dir) once the new shape's output is safely on disk, fix the skip check to test the artifact this run would actually produce, and clear orphaned per-animal slots before writing. Also stop the multi-animal WROTE event from naming primary, which it never writes.
Every other reconciliation in this file writes the replacement first and deletes what it supersedes second, so a run that dies partway leaves the video no worse than it found it. The orphan-slot cleanup was doing the reverse: had a per-animal write thrown midway, the removed slot would already be gone while the kept slots still held the previous run's coordinates -- a mix of old and deleted rather than simply unchanged. The slots it removes are disjoint from the slots just written, so the order was never load-bearing in the first place.
…ession find_pose_csv asks which pose CSV belongs to a video, and a session tracked with two animals has no single answer. It now returns None for those, and logs the video, the animal count and the directory -- the log being the only breadcrumb a researcher gets when a single-animal tool shows them an empty state. Raising was tried first and rejected. Roughly fourteen callers across seven files include folder scans, and a raise turns one multi-animal session into an aborted scan for every other video beside it, as well as bypassing project.py's `or` fallback. Right semantically, wrong mechanically. find_pose_csvs returns the whole set instead, in numeric slot order -- sorted() on the names would put animal10 before animal2, because the slot is an integer and only looks like a string.
find_pose_csv has resolved "several pose CSVs for one video" by most recent mtime for as long as it has had a tie-break, and its docstring says why: alphabetical order would silently prefer exp-5 over exp-7, quietly scoring a cohort with a superseded pose model. The _animals lookup added beside it did exactly that. Two models can each leave their own directory -- dlc_output_path is keyed on (video, model), so the reconciliation that clears a stale one does not apply across models -- and find_pose_csvs took the first alphabetically. Given expA (one animal, older) and expB (two animals, newer), it returned expA's single file. Both lookups now share the same _mtime rule and log the choice, so the function that names a directory and the function that returns its files can no longer describe different sessions. The log test went with it: asserting "2" and "s1" appeared somewhere passed on the video name alone, and that log line is the only thing a researcher gets when a multi-animal session goes invisible to a single-animal tool.
Adds "Export multi-animal DLC CSV" to the Batch Pose Tracking window: reads a session's per-animal CSVs, rebuilds a PoseTracks, and writes the four-row file SimBA/DLC/Keypoint-MoSeq read with the existing to_dlc_csv_multi. It is derived fresh every time, never a parallel primary, so it cannot drift from the per-animal files it comes from. export_actions.py is Qt-free (mirrors arena_actions.py's regate_videos); export_worker.py carries it onto a QThread (mirrors RegateWorker). A single-animal session reports "nothing to export" and writes nothing; per-animal files that disagree are reported with the session name rather than raising past the log.
PoseTracks already refused slots that disagree on frame count or keypoint names, because either means the animals cannot be compared. It said nothing about fps, and the multi-animal export took whichever rate the lowest-numbered slot happened to carry. Nothing GLIDER writes can produce the disagreement -- consolidate hands one rate to every slot, and gating and smoothing preserve it through a copy. But per-animal CSVs are plain three-row files in the format everything reads, so substituting one by hand is a thing a lab will do, and a substituted file recorded at another rate would have been silently re-stamped rather than refused. That one is worse than the two already caught: a wrong frame count shows up as a shape error somewhere, while a wrong rate just quietly computes every feature windowed in seconds over the wrong span.
classify_pose_tracks loops classify_pose_data over each slot, keyed by slot id. Tuning kwargs forward through **kw rather than being re-declared, so classify_pose_data's defaults can't drift out of sync with a second copy here.
classify_pose_tracks scores every slot; write_animal_ethograms writes each
slot's rows to animals_dir(pose_csv)/animal{slot}_ethogram.csv with the
existing write_ethogram_csv, reusing animals_dir rather than re-deriving the
naming. No individual column anywhere, per D2 spec §5 -- SessionView needs
no change.
A slot never filled by consolidation is entirely NaN; classify_pose_data
already scores that the same as a dropout frame (one row per scored frame,
blank behavior), so its ethogram is written like every other slot's rather
than skipped or special-cased.
classify_pose_tracks and write_animal_ethograms existed, were tested and
exported, but classify() only reached batch_apply when pose_csv_in was set
-- and find_pose_csv returns None for a multi-animal session by design, so
the flow always fell through to the single-animal LiveInferencePipeline.
find_pose_csvs returning more than one path is now checked alongside
find_pose_csv (same reuse_existing_poses gate), and a multi-animal session
rebuilds a PoseTracks from its per-animal CSVs and writes one ethogram per
slot via write_animal_ethograms, returning {slot: ethogram_path}. Refuses
an annotated video or a speed-only run for a multi-animal session rather
than silently mis-scoring. The single-animal branch is untouched -- the new
code only runs when a second discovery call finds more than zero files.
classify()'s multi-animal branch wrote run.json only to output_dir, but SessionView reads the manifest from ethogram_csv.parent (_load_applied_thresholds, _load_scale) -- which for a multi-animal ethogram is animal_dir, not output_dir. Opening animal0_ethogram.csv in Session Review therefore silently lost the applied freeze/dart thresholds (nothing else records them), the recorded px_per_mm, and the video association, while still rendering labels and poses -- quiet degradation. Writes the same manifest payload to animal_dir right after the existing output_dir write, skipped when they're already the same directory.
…e pool compute_cohort_thresholds's cohort collector matches anything with DLC_ in its stem, which is exactly the shape of the four-row multi-animal export -- now the in-GLIDER way to produce one. from_dlc_csv refuses to guess which animal to read from a four-row file without individual=, and that ValueError was unguarded, so one export sitting in a cohort folder killed pooling for every other session in it. Wraps the read in try/except ValueError: log and continue, matching the skip-and-continue idiom already used four lines below for a session with no usable speed samples. Covers any unreadable CSV, not just the export case.
write_animal_ethograms writes a blank ethogram for a slot consolidation never filled -- the right call, every slot gets a file -- but a blank ethogram is indistinguishable from a scoring failure at a glance. Adds one logger.info per all-NaN slot, naming it and saying the animal was never found rather than that scoring failed.
PoseTracks' fps-mismatch error blamed only a CSV "replaced by hand". A lost or corrupt .meta.json sidecar produces the same symptom -- from_dlc_csv falls back to DEFAULT_FPS for that slot alone -- and is the cause a user can actually act on (restore or regenerate the sidecar). Message text only.
find_pose_csv compared an _animals directory's own mtime against the newest flat CSV to decide which is current. On APFS a directory's mtime only moves when an entry is added or removed, not when a file already inside it is overwritten in place -- exactly what a same-model re-track does to animal0.csv, animal1.csv, etc. -- so a stale flat CSV from a different model could outrank a just-retracked multi-animal session. Rank by the newest per-animal CSV inside the directory instead. Also close the resume-check's raw `.exists()` on the same directory: a leftover empty _animals/ made a complete single-animal session look unfinished, forcing needless re-inference under overwrite=False. Reuse _animal_csvs, matching how _pick_animals_dir already decides whether a directory counts.
…es across animals
_assemble_sessions gains individuals: list[int | None] | None, positionally aligned with sessions, forwarded to AnnotationStore.load_csv so each session trains only on its own animal's zones. A two-animal video becomes two sessions sharing one annotations CSV, differing only in individuals. A length mismatch between individuals and sessions raises loudly instead of zipping short. Also refuses spec.include_social with mirror_augment, matching the existing motion_features + mirror_augment guard: mirroring flips only the subject, not the other animals in `others`, so social columns would measure the subject against a partner on the wrong side of the arena. Threaded individuals through train_model, train_hybrid_model, and the shared _assemble_and_filter helper so both call sites of _assemble_sessions (the held-out-test assembly and the main training assembly) forward it.
…g one train_model was forwarding `individuals` (aligned with `sessions`) into the held-out-test `_assemble_sessions` call, which assembles `holdout_sessions` -- a different list that can legally have a different length (e.g. train on both animals of video A, hold out one single-animal session of video B). That either raised on the length check for valid input, or silently labelled the test set from the wrong animals when the lengths happened to match. Adds `holdout_individuals`, positionally aligned with `holdout_sessions` and independent of `individuals`, and forwards it at the holdout call site instead. Default None keeps today's behaviour (every holdout zone). Also documents `individuals` in train_hybrid_model and _assemble_and_filter, which had the parameter but no docstring mention.
Add a test that produces one flat single-animal session alongside a multi-animal one in the same folder and asserts the single-animal video routes through propose_clips_multi tagged individual=0, the multi-animal video routes through propose_clips_for_animal tagged with the chosen subject, and the two quotas sum back to the requested total. Also drop a dead `or []` in the exclude-zones list comprehension for the single-animal batch path: that branch only runs when exclude_labeled is already true, so _exclude_zones_for never returns None there.
… can be learned _assemble_sessions computed features with no `others`, so train_model(spec= FeatureSpec(include_social=True), ...) raised unconditionally on every session -- compute_features' own guard fired every time, before a single row was ever assembled. Social features were reachable only from the annotator's clip-diversity sampler. Gather `others` from the session's `_animals/` siblings (via vision.pose.batch._animal_csvs, mirroring the layout convention gui.behavior.window._individual_for_pose_csv already reads) and forward them into compute_features. A session whose pose CSV isn't laid out that way now raises a ValueError naming the offending path, instead of falling through to compute_features' generic "no others were given". _assemble_for_cv (cross_validate_sessions / cross_validate_and_train) gets an explicit refusal instead of the same treatment -- it has no session- list plumbing for this yet, and refusing beats silently training columns that were never actually social. Corrected docs-site/camera-behavior/behavior.md and CHANGELOG.md, which (accurately, before this) said training never forwards another animal's pose.
_seed_clip_zones picked the maximally-overlapping zone in the whole store, ignoring `individual`. Two animals' zones legitimately overlap -- the sampler is designed to propose animal 1's clips over frames animal 0 is already labelled on -- so an animal-1 clip got bound to animal 0's zone, displayed as already done with the other animal's behaviour, and the first keypress ran store.remove on it. One keypress, no warning, annotation gone. Skip zones whose individual differs from the clip's. Legacy zones and single-animal clips are both 0, so nothing changes for them. The existing re-trim test forced _clip_zone by hand, skipping the very seeding step that was broken; it now lets _seed_clip_zones do the binding, and a new test produces the data-loss case: animal 0's zone on disk, an animal-1 clip over the same frames, label it, assert animal 0's zone survives. That test fails against the unfixed code.
Review mode and resume both build their clip list here, and every clip came back individual=0. So the overlay highlighted animal 0 whoever the zone belonged to, and a resumed session then seeded animal 0's clips over other animals' zones. The test that pinned this flattening as the requirement now pins only the dataclass default (positional construction relies on it), and a new test asserts zones_to_clips carries each zone's own animal through.
The button routed through propose_clips_multi, which stamps every clip individual=0. A labeller who launched a pass for animal 1 pressed it and was labelling animal 0 from then on, unannounced -- and on Resume it also sampled animal 0's pose CSV, because the resume path resolves slot 0 for every multi-animal video. make_more_sampler now takes the per-video animal CSVs and the subject slot and goes through propose_clips_for_animal, which picks the right animal's pose and stamps the clip. Same divmod quota and per-video seed offset as before, so a folder with no multi-animal video returns exactly what it did. The subject is derived from the queue itself in _open_annotator -- a pass labels one animal per video, so the first clip for a video says which. That keeps Launch and Resume on one rule; Resume never asks for a subject.
Training with spec.include_social worked and nothing could consume the result: every offline apply path called compute_features(pose, spec) with no `others`, so it raised -- and its message told the operator to "classify the recording offline where every animal's track is available", which is exactly what they were doing. classify_pose_tracks is the one place every animal's PoseData is in hand, so it now passes each slot the other slots as `others`, through a new optional keyword on classify_pose_data. Defaulted and ignored for a non-social model, so every single-animal caller is untouched. The other three compute_features callers: * evaluation._windowed_for is plumbed the same way -- it holds the pose CSV path, so pipeline._other_animal_poses finds the session's other animals exactly as training does, and raises by name when the layout cannot support it. * scale_guard.scale_warning reads body_length only, which is computed from the subject alone; it now asks for the spec with include_social cleared rather than needing tracks nothing hands it. Before, a social model lost its scale warning silently inside the diagnostic's own except-and-return-None. * sequence.assemble_sequences refuses a social spec up front, by name. It could load the other animals, but a CNN sequence model cannot be applied to a multi-animal session at all (apply_behavior_model refuses one), so the bundle could never be scored. docs-site says what is now true: a social model is scored by classifying the recording offline, never live, with the CV and sequence-model corners named.
…imal An untracked animal is made ineligible with +inf, so `nearest` silently switches to a farther one the frame a closer animal drops out -- and np.gradient over social_distance then subtracts one animal's distance from another's. On a real 20-minute two-mouse recording with dropouts that manufactures a violent approach and an equally violent retreat on frames where nobody moved: a wrong answer that looks plausible, which is the dangerous kind. NaN social_approach wherever the derivative's stencil spans a change of which animal is nearest -- "nobody tracked" counting as its own identity, since argmin over an all-inf column returns slot 0 rather than a sighting. The same rule social_distance already follows when nobody is there. The test produces the condition rather than forcing it: two others, the nearer one dropping out mid-sequence, everything stationary, so every defensible approach value is exactly 0 and any non-zero number is fiction. The docstring and the docs-site table say "nearest *tracked* other animal".
_assemble_for_cv loads each session's annotations with no `individual` filter, while the training path filters correctly. Same session list, two GUI buttons, one of them wrong: every animal was trained and scored against BOTH animals' labels, and with "...and fit a model on all sessions" ticked CrossValidateWorker saved that mislabelled bundle to disk. Refuse beside the include_social refusal already there, naming the offending pose CSV and pointing at train_model(individuals=...), which does filter. A loud refusal beats a silent wrong number. The test builds three real multi-animal sessions -- so the run is otherwise perfectly foldable and the guard is the only thing that can stop it; against the unfixed code it does not raise at all.
_tracks_for dropped every track on any single per-animal CSV failure and said nothing; _draw_pose then returned the frame unchanged. The animals in this assay are visually identical, so the overlay is the only thing telling the labeller which one a clip is about -- they were left with two indistinguishable mice, no markers and no reason why, and every zone the pass wrote carried a guess. Record the reason in track_errors and put it on the status bar with no timeout, naming the file that would not parse. Recorded and said out loud, never raised -- the same policy _load_speed_now applies to the speed trace. Not through warn_about_load_errors: that fires once at startup, and these CSVs are decoded lazily on the first frame drawn, long after the dialog has gone. Its load_errors also disables labelling for the video, which would be wrong here -- the annotations file is fine. Dropping all tracks on one failure is kept and now explained: skipping the unreadable file would shift every later animal's index, and the subject is picked BY index, so the overlay would highlight the wrong animal -- worse than none. Adds the failure-branch test that was previously deferred.
…bels _center_in_labelled_zone asked zones_at_frame for EVERY animal's zones, so on the second animal's labelling pass -- the workflow this branch ships -- a well-labelled video rejected nearly every fresh clip, because animal 1's unlabelled frames are mostly frames where animal 0 is already labelled. The status bar then blamed "already-labelled regions filtered", which was true of the wrong animal. Match window.py's launch-time exclusion list, which already filters to the subject's own zones for exactly this reason.
evaluate_model loaded the annotations CSV with no individual filter. That was harmless while it refused every social bundle outright; now that `others` is plumbed through and social evaluation succeeds, every social score silently mixed the partner's zones into ground truth -- and a social model comes from a multi-animal session by definition. Where the two animals' zones contradicted, frames went AMBIGUOUS and quietly vanished from the denominator; where only the partner was labelled, its behavior became the subject's ground truth. Both produce a plausible wrong number. So: take `individuals`, positionally aligned with `sessions`, exactly as train_model does; and when none was given but the file holds more than one animal, refuse the way _assemble_for_cv refuses, naming the argument that fixes it. Single-animal sessions hold exactly one individual and are untouched.
A duplicate box on one animal fills both slots, and the identity sidecar flags every frame close rather than presenting two animals. A persistent false positive on cage hardware outsits an intermittently-occluded animal and takes its slot; the refused fragments land in dropped, so the loss is recorded rather than silent. Both observed running a YOLO pose model over CalMS21 mouse001. Neither is fixed here: the only signal separating stationary hardware from a frozen mouse is motion, and freezing is a behaviour these recordings measure. Arena gating upstream is the defence.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GLIDER has only ever followed one animal. Every recording of a social assay went through a pipeline that picked the most confident mouse in each frame and discarded the rest, so a pair assay could be recorded but not analysed.
This makes the whole offline path multi-animal for YOLO — tracking, ethograms, labelling, and features that measure one animal against another. SLEAP and DeepLabCut stay single-animal and are refused by name rather than silently returning one arbitrary mouse in slot 0.
The decision everything else follows from: a multi-animal session is N single-animal sessions, not one session with an instance axis.
PoseTracksis a dumb{slot: PoseData}container with no analysis methods of its own. Gating, filtering, zone scoring, feature extraction and classification are reused verbatim, because each one still receives exactly what it always received. Pose stays one CSV per animal, so the same file opens in the multi-animal tools and the single-animal ones and the two can never diverge.What changed
D1 — tracking. ByteTrack yields more fragments than there are animals;
consolidate.pystitches them greedily, longest-first, into exactly N lifelong slots, using the known animal count as a hard constraint. Identity honesty is a sparse_identity.csvsidecar: a row exists only where there is doubt, flaggedclose,stitchedorgap. Zone scoring and occupancy became per-track, and a zone nobody entered still emits a row — a zero is a result, not the absence of one.D2 — behaviour.
classify_pose_tracksruns an existing single-animal model over each slot and writes one ethogram per animal. Discovery answers only what it can:find_pose_csvreturnsNonefor a multi-animal session rather than guessing, andfind_pose_csvsreturns every animal's CSV in slot order. The four-row DLC CSV is an opt-in export, generated fresh from the per-animal files so it cannot drift.D3 — labelling and social features.
BehaviorZonegained anindividual; the overlap rule now keys on(behavior, individual), because two animals grooming at once is the assay rather than an edge case. The annotator labels one animal per pass, draws every animal with the subject highlighted, and stamps each zone with its subject.FeatureSpec.include_socialadds five columns measured against whichever tracked other animal is nearest that frame — so it works for N > 2 and survives a dropout.Social features are refused where they cannot be honest: on the live path (one animal, nothing to measure against), with mirror augmentation (the partner is not mirrored), and in cross-validation (which cannot filter by individual).
evaluate_modelrefuses ambiguous ground truth rather than scoring against both animals' labels.What a reader should know
An annotations CSV with no
individualcolumn, or a blank cell, reads as animal 0 — every existing file keeps working.include_socialdefaults off, so existing models' feature frames stay bit-identical. Both GUI doors are wired: the annotator launch asks which animal, and the training picker resolves a per-animal CSV to the session's shared annotations.This also fixes a latent bug found on the way:
FeatureSpec.to_dict()silently droppedinclude_trajectoryandtrajectory_min_step, so a spec saved with either changed reloaded with the defaults.Verified, and not
5339 passed, 3 skipped, 5 deselected. ruff and black clean; mypy unchanged from baseline. Every task was reviewed, and the whole-branch review found one Critical — labelling one animal over another's frames deleted that animal's annotation — plus six Important, all fixed.
Then it was run over a real two-mouse recording (CalMS21
mouse001, 21,364 frames). The pipeline runs end to end, consolidates into two slots, and writes the sidecar. Two failure shapes from that run are now pinned as tests: a duplicate box on one animal fills both slots but flags every frameclose, and a persistent false positive on cage hardware outsits an intermittently-occluded animal and takes its slot, with the refused fragments recorded indropped.What has not been validated. That recording has one black and one white mouse, so identity is recoverable from pixels alone — it cannot produce the silent slot swap that fixed-N consolidation exists to prevent. Nothing here has met two animals you genuinely cannot tell apart. The five social columns are reasoned, not validated, and cannot be until this ships and produces the annotations that would validate them; treat the first trained social model as a probe. Arena gating is the defence against the false-positive shape above, and a session configured without an arena has none.