Conversation
RtspCamera (dimos/hardware/sensors/camera/rtsp): RTSP URL, file or a generated clip in; video (encoded, untouched), color_image and color_jpeg out. PyAV only, url is required, set_video_enabled() and set_jpeg_rate() are RPCs. Blueprint rtsp-camera-vis. SiyiA8Gimbal (dimos/hardware/gimbal/siyi): gimbal tf chain, camera_info (a8_camera_info() is the one source of the A8 intrinsics), aim requests on gimbal_target. ip is a config field with no default; with it set the SIYI SDK client polls the zoom and camera_info is withheld off 1x. New px4 extra: av. Tests: pytest dimos/hardware/gimbal/siyi dimos/hardware/sensors/camera/rtsp -> 37 passed (gimbal 22, camera 15).
github-actions
Bot
added
the
first-time-contributor
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| gimbal_rot = Quaternion.from_euler( | ||
| Vector3( | ||
| math.radians(att.roll_deg), -math.radians(att.pitch_deg), -math.radians(att.yaw_deg) | ||
| ) | ||
| ) |
There was a problem hiding this comment.
When the airframe pitches, this body-relative transform applies the A8’s earth-stabilized pitch without compensating for airframe attitude. The published optical frame is misoriented, giving camera rays and target locations incorrect directions. With 20° of airframe pitch, the optical orientation was 20° from the expected orientation.
Artifacts
Authored camera-orientation check and invocation
- The executed Python source feeds a synthetic gimbal attitude through the real publisher and compares its optical TF orientation with an earth-stabilized model.
- The command, working directory, exit code, and output show that the published optical orientation matches the expected orientation when the airframe is level.
Nonlevel-airframe defect output
- The command, working directory, exit code, and output show a 20° optical-orientation error after modeling 20° of airframe pitch, confirming the defect.
| zoom = self._sdk.query_zoom() | ||
| if zoom is not None: # a lost reply keeps the last known zoom | ||
| with self._lock: | ||
| self._zoom = zoom |
There was a problem hiding this comment.
After a successful 1× zoom reading, failed polls leave _zoom at 1×. publish_camera_info() keeps issuing newly timestamped 1× intrinsics although the current zoom can no longer be verified. If the camera changes zoom during that loss of communication, image consumers receive incorrect calibration.
Artifacts
Camera-info service before poll failures
- Ran the service methods with unknown zoom and then a valid 1x reply; publication began only after the valid reply.
Camera-info service after poll failures
- Ran the same service methods through three failed polls and a later 2.5x reply; fixed 1x intrinsics continued publishing through the failures.
Authored zoom-poll service repro source
- This script supplies controlled SDK replies and asserts the observed service returns and publication counts.
Authored artifact upload script
- This script PUT the two execution logs and both scripts to preallocated artifact slots.
Artifact upload command and responses
- Captured the upload command, working directory, exit code, and four HTTP 200 OK responses.
| if self._is_file() and cfg.replay_realtime: | ||
| due = t_start + packets / fps | ||
| delay = due - time.monotonic() | ||
| if delay > 0: | ||
| time.sleep(delay) |
There was a problem hiding this comment.
For a file with variable frame durations, replay schedules packets by count and average frame rate rather than their timestamps. A captured 320 ms frame gap replayed as 87 ms, shifting frame timestamps relative to other streams and preventing faithful synchronized replay.
Artifacts
Variable-timestamp H.265 replay check source
- The authored script generates a variable-PTS file and runs both the timestamp-paced reference and the PR relay, showing exactly how timing was measured.
Timestamp-paced reference output
- The executed reference paced packets from file timestamps and observed the intended 320 ms gap, establishing the comparison.
- The executed PR relay emitted all six frames but reduced the 320 ms file gap to 87 ms, confirming the timing defect.
| self.relay_once() | ||
| except (av.FFmpegError, OSError, IndexError, ValueError) as exc: | ||
| self._count("video", errors=1) | ||
| logger.warning("camera stream unavailable", url=self._source, error=str(exc)) |
There was a problem hiding this comment.
If an RTSP URL contains credentials, a connection failure writes the unredacted URL to console output and the persistent JSON log. Anyone with access to those logs can recover the camera credentials. Log a credential-free endpoint instead; the stream-ended warning has the same exposure.
How this was verified: A failed connection with a dummy password placed that password in both the console warning and the JSON log.
Artifacts
Failed RTSP connection before mitigation
- The executed PR code logged a dummy password to the console and persistent JSON record, confirming credential exposure.
Failed RTSP connection with in-memory mitigation
- The same simulated failure ran with raw-URL logging removed in memory and left the dummy password out of both outputs.
Local RTSP failure reproduction script
- The script invoked the changed relay path with a fake credential and inspected the real logger handlers' JSON output.
Artifact upload command script
- The script uploaded the captured proof through allocated artifact slots without using camera credentials.
- The captured upload command reported HTTP 200 OK for the after log and reproduction scripts; the before log was already uploaded.
| "max_hz": {"world/color_jpeg": 2.0}, | ||
| # Raw decoded frames stay on this machine; the viewer decodes the H.265 itself. | ||
| "visual_override": {"world/color_image": None}, |
There was a problem hiding this comment.
This viewer suppresses color_image, the only image output to which the bridge attaches camera calibration. Its displayed video and JPEG outputs receive no pinhole calibration, limiting their usefulness for interpreting camera geometry. This is a non-blocking viewer limitation; JPEG resizing also requires appropriately scaled intrinsics.
Artifacts
Bridge output without RTSP image suppression
- Chromium rendered the captured before-run output, which shows a Pinhole on the raw image entity; upload receipt: https://app.greptile.com/trex/artifacts/c31947fb-9687-41ed-b7a8-6320df3ac48c.
Bridge output with RTSP image suppression
- Chromium rendered the captured after-run output, which shows JPEG and video entities but no Pinhole; upload receipt: https://app.greptile.com/trex/artifacts/9d185c3d-762b-4806-8476-12b0be9b0b91.
Authored and executed bridge reproduction script
- Chromium rendered the authored script used for both runs; upload receipt: https://app.greptile.com/trex/artifacts/abcfd98a-e435-4bc0-886b-d267f68400bc.
Executed artifact upload script
- Chromium rendered the upload script used to submit the proof; upload receipt: https://app.greptile.com/trex/artifacts/fe71f284-dd7f-4016-b8e8-27c9cc146dae.
Successful artifact upload output
- Chromium rendered the upload command's captured HTTP 200 responses; upload receipt: https://app.greptile.com/trex/artifacts/0826853b-a928-43f5-8122-e3b22282433e.
- The final upload command recorded the prior uploads and an HTTP 200 for the upload-output image, confirming submission.
| from dimos.hardware.sensors.camera.rtsp.camera import RtspCamera | ||
| from dimos.hardware.sensors.camera.rtsp.synthetic import SQUARE, square_origin, write_synthetic_h265 | ||
|
|
||
| pytestmark = pytest.mark.skipif_no_turbojpeg |
There was a problem hiding this comment.
The module-wide turbojpeg marker skips all 15 camera tests locally when the native library is unavailable, including passthrough, decoded-image, lifecycle, and missing-library fallback tests. A successful local run can therefore exercise none of those paths. This loss of useful local coverage does not independently block merging.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Artifacts
Camera pytest execution commands
- The uploaded script records and executes collection, local, and CI-mode camera pytest commands from the repository root, making the verification reproducible.
Camera pytest collection and run output without libturbojpeg
- The captured execution shows 15 collected, 15 locally skipped, and 12 passing plus 3 failing in CI mode, confirming that the module-wide skip conceals runnable tests.
Comments Outside DiffThese findings sit on lines the diff does not cover, so they could not be posted inline. Each one leaves this list once its file changes.
|
This branch has not been deployed
1 participant
What is this feature?
Camera and gimbal drivers for the PX4 stack. Nothing here depends on PX4.
RtspCamera(dimos/hardware/sensors/camera/rtsp): RTSP H.265 in;video(encoded, untouched),color_image(decoded, rate-capped),color_jpeg(small, for a slow link) out. PyAV only.urlis an RTSP URL, a file, orsynthetic. RPCs:set_video_enabled(enabled),set_jpeg_rate(hz).SiyiA8Gimbal(dimos/hardware/gimbal/siyi): readsgimbal_attitude, publishes the gimbal tf chain andcamera_info, turnstarget_losintogimbal_targetaim requests. Opens no MAVLink socket. The SIYI SDK (zoom poll) is off unlessipis set.px4extra:av.Why do we need this?
The A8 is the drone's camera. dimOS has no RTSP/H.265 source and no gimbal model.
How to Test
The last command streams 25 fps video with no camera attached. An earlier revision decoded the real A8 stream on a Jetson Orin Nano.
Stack
Five PRs, all against
main; each contains the ones above it. Merge in order. Review only this PR's own commit: 5bbf22cWhich issue(s) does this PR close?
None. New platform: PX4 multicopters with a SIYI A8 gimbal camera.
Checklist
🤖 Generated with Claude Code