feat(camera): several cameras in the picture, with layout sections - #1026
Draft
christian-wr wants to merge 22 commits into
Draft
christian-wr wants to merge 22 commits into
christian-wr wants to merge 22 commits into
Conversation
Each camera of the webcams list (or the legacy single-camera fields) gets its own capture and encoder on the recording's T0. A camera that cannot be opened is dropped with an indexed webcam-unavailable warning; one whose samples fail mid-take is disabled on its own. recording-stopped keeps webcamPath and adds webcamPaths. MFEncoder now only balances an MFStartup it made: a dropped camera's never-initialized encoder ran an unmatched MFShutdown from its destructor and stopped every other encoder in the process.
A camera whose encoder initialize() or capture start() fails is now warned about with the indexed webcam-unavailable event and dropped, instead of ending the take; its encoder is finalized and its empty file removed. The screen encoder's failure stays fatal. mf_encoder_color_test pins the MFEncoder fix: finalizing a never-initialized encoder must leave a live one able to write and finalize.
Two webcams of the same model report the same name, and the browser id never matches a device path, so every such camera selected the first device and the second open failed as busy. The take now owns a claim set: each camera that opens adds its device (MF symbolic link or DirectShow DevicePath, normalized so both paths agree), and later cameras pick the best unclaimed match. The selection rule lives in device_selection.{h,cpp} with its own unit test.
A label already used by camera 1 or an earlier extra gets " (2)", " (3)" by occurrence, so unavailable and dropped cameras of the same model can be told apart.
The recorder caps at three extra cameras, but a validator that ships with a cap cannot be loosened later for older builds. The HUD and Electron still cap.
Two webcams of the same model share a name. When an extra carries a deviceId and camera 1 does not, the name says nothing about whether they are the same device, so the extra is no longer dropped as a duplicate of camera 1.
The helper deletes the file of a camera it drops at start, but ignored a failed DeleteFileW. It now logs a WARNING with the path and GetLastError, and Electron keeps the dropped cameras' paths so stop and discard remove a 0-byte stub left behind.
A camera the helper disables mid-take keeps its partial file in the take, but nobody was told. A camera whose file was kept (size > 0) yet is missing from recording-stopped.webcamPaths is now named in a "Stopped early" notice after the take, camera 1 included. Paths compare case-insensitively with either separator. Only an event that carries webcamPaths can say so, so the helper now prints the list, possibly empty, whenever a camera wrote a file of its own; an older helper or a missing event never produces the notice.
The checklist now notes that with several identical cameras plugged in the recorded ones follow Windows' enumeration order, adds a 1-vs-2-camera screen pacing comparison at 4K (getopenscreen#945) and a stopped-early check. The helper README says the camera index counts after entries without camPath are skipped, and the extras-need-camera-1 assumption (R6) is noted where links drop them.
A failed ReadSample was counted and retried forever, so an unplugged camera kept its file running to the end on its last picture and stayed in webcamPaths. The Media Foundation capture now latches lost on a device invalidated or hardware start failure, on end of stream during the take, or after a second of consecutive read failures; the DirectShow fallback latches it on EC_DEVICE_LOST (removal), EC_ERRORABORT or EC_STREAM_ERROR_STOPPED. The writer loop disables a lost camera like any mid-take failure, so its file ends at the loss and the app names it as stopped early.
A Full Camera section can now show a webcam tilted down onto the desk: "Desk view" in the inspector turns the camera 180°, switches the mirror off for that section (otherwise text on the page reads back to front) and shows the whole camera frame instead of the face crop. The timeline marks such sections with a rotate icon. The moment the camera is tilted is covered at both ends of the section: the whole camera picture is blurred and dimmed for the Full Camera grow (and the shrink), fading in and out around it, with a translated "Desk mode" label over it that can be switched off per section. Preview and export render it identically through the compositor's frame plan. - Rust: per-frame orientation (u/v bound swaps, crop bypass) and cover strength, a `LayerCB.cover` lane (HLSL, WGSL, MSL), the blur clamped to the camera's valid area so aligned decoder padding never bleeds in; the label is drawn at the cover strength of the same frame, so it fades exactly with the cover. - App: `rotation` / `mirror` / `deskLabel` on Full Camera regions (defaults are not stored), store and persistence, inspector controls, one generated label annotation per projected piece, translations in every locale. - Tests for each layer; WGSL is validated with naga on every host.
Up to four recorded cameras can now be on screen together. Layout sections on the timeline choose a template — screen + small cameras, one camera full + small ones, two cameras side by side, or another camera full — and which camera goes into each place; the picture glides between sections. Each camera can be turned, mirrored, cropped and perspective-corrected (a homography calibrated once with four corner handles, optionally placed by detected ArUco markers from a printable sheet). - Compositor: a list of camera layers per frame instead of one webcam rect, planned in `camera_layers.rs` (glide and fade between neighbouring sections, incl. a direct glide to and from an adjacent Full Camera section), extra camera decoders opened only for cameras a layout shows (preview and export, a camera that ends early never shortens the clip), a homography and a transparency lane in `LayerCB` (HLSL, WGSL, MSL), the corrected camera cover-fitted into its box. - Editor: Full Camera regions stay as they are (camera 1, desk view); every other layout lives in `cameraLayoutRegions`, clip-anchored like Full Camera regions, in the same timeline lane with no overlap across the two lists. "Add layout" menu, a layout inspector (template, camera per place, reset windows), PiP places movable and resizable in the preview, a "Cameras" section with per-camera settings, and a calibration dialog (perspective and crop) with a loupe, target format, margin and a live rectified preview. Marker detection uses js-aruco2 (MIT; the OpenCV 4x4 dictionary codes are BSD-3, both listed in THIRD-PARTY-NOTICES.md). - Every change is one undo step; projects without layout sections render exactly as before.
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Put several recorded cameras in the picture at once — the desk camera full with your face small on top, two cameras side by side, or the screen with two small cameras — and switch between such arrangements along the timeline with a glide.
camera_layers.rs: glide and fade between neighbouring sections, also directly to and from an adjacent Full Camera section), extra camera decoders opened only for cameras a layout shows (preview and export; a camera that ends early never shortens the clip), and a homography + transparency lane appended toLayerCB(HLSL, WGSL and MSL agree; pinned by the size/offset test). Projects without layout sections render exactly as before.cameraLayoutRegions, clip-anchored like Full Camera regions, with no overlap across the two lists (also enforced for the AI agent's Full Camera tools). Per-camera settings incameraSettings. All additive — no schema bump.js-aruco22.0.0 (MIT, pure JS) for marker detection; it embeds OpenCV's 4×4 dictionary codes (BSD-3). Both are listed inTHIRD-PARTY-NOTICES.md, since the code ships inside the renderer bundle.Related issue
None — new feature.
Type of change
Release impact
Desktop impact
Screenshots / video
Can follow on request (desk camera full + face small, side by side, the calibration dialog).
Testing
npm run test(≈4430 passed;LeftPanel.copyMessage.test.tsxoccasionally times out under full-suite load and passes alone — untouched here), bothtscconfigs,npm run lint(0 errors),npm run i18n:check.cargo test -p openscreen-compositor --libon Windows ARM64 (the 8pipeline_windows::teststhat need hardware video decode fail on this host with or without this change); Linux and macOS compositor tests incl. new wgpu pixel tests passed in CI on a fork._electron, on a copy of a real Brio + built-in camera recording): add a "camera full + small" section, choose and swap cameras, convert a Full Camera section and back, undo/redo, drag a camera window, calibrate a perspective, save and reopen — 27/27 checks, live preview inside the sections, camera 1 large fills the frame edge to edge (measured). Script:scripts/editor-multicam-smoke.mjs.Not covered yet (in the checklist): the printed marker sheet and a real desk camera under real light, and an export of a real-camera project with layout sections. Translations other than English and German would benefit from a native speaker's look.