Conversation
RtspCamera (dimos/hardware/sensors/camera/rtsp): RTSP URL, file or a generated clip in; video (encoded, untouched), color_image and color_jpeg out. PyAV only, url is required, set_video_enabled() and set_jpeg_rate() are RPCs. Blueprint rtsp-camera-vis. SiyiA8Gimbal (dimos/hardware/gimbal/siyi): gimbal tf chain, camera_info (a8_camera_info() is the one source of the A8 intrinsics), aim requests on gimbal_target. ip is a config field with no default; with it set the SIYI SDK client polls the zoom and camera_info is withheld off 1x. New px4 extra: av. Tests: pytest dimos/hardware/gimbal/siyi dimos/hardware/sensors/camera/rtsp -> 37 passed (gimbal 22, camera 15).
sensor_msgs.NavSatFix and sensor_msgs.BatteryState wrap the dimos_lcm types and take their enum values from them. px4_msgs.VehicleStatus is a hand-written LCM type with the wire layout of sensor_msgs/ImuInfo (fingerprint, Header, big-endian fields); its base hash is an arbitrary constant, there is no .lcm schema. The wire helpers are in dimos/msgs/lcm_wire.py. No registry, pyproject or lock change. The producer is Px4DroneConnection in the next commit: its gps, battery and vehicle_status outputs. Tests: pytest dimos/msgs/px4_msgs dimos/msgs/sensor_msgs/test_NavSatFix.py dimos/msgs/sensor_msgs/test_BatteryState.py -> 6 passed (43 with the payload tests below).
dimos/robot/px4: - connection.py: Px4DroneConnection, the one MAVLink link. The vehicle as streams (odometry, imu, gps, battery, gimbal_attitude, vehicle_status) and the operator RPCs takeoff, go_to, land, hold, set_guidance_mode, estop*. No arm, mode or raw-setpoint RPC. - supervisor_core.py: 20 Hz state machine, no I/O. Preflight, RC enable switch, fence and ceiling, pilot override, E-STOP latch, go-to, TELEOP. When the tick stops the setpoints stop and PX4's Offboard-loss failsafe takes over. - mavlink.py: pymavlink socket, vehicle state, PX4 modes, NED/FLU frames. - blueprints.py: px4-basic, px4-drone, px4-sitl, px4-teleop, px4-sitl-teleop. tool_sitl_gate.py runs px4-sitl against PX4 SITL. px4 extra += pymavlink; stubs/pymavlink/mavutil.pyi is extended for mypy. Tests: pytest dimos/robot/px4 -> 103 passed (connection 10, mavlink 21, supervisor_core 4, supervisor_operator 68); 146 with the payload and message tests below.
github-actions
Bot
added
the
first-time-contributor
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| # HEARTBEAT is 1 Hz and px4_stale_s is 1.0 s, so heartbeat age alone sits on the | ||
| # threshold and a few ms of jitter would abort a flight (seen in SITL). Any message | ||
| # from 1/1 proves the link. | ||
| return min(st.heartbeat_age, st.px4_msg_age) |
There was a problem hiding this comment.
If HEARTBEAT stops while other PX4 telemetry continues, this check treats the cached flight mode as current. The supervisor can continue Offboard setpoints after a pilot mode change, or send Hold during an unrelated abort instead of yielding to the pilot. Check heartbeat freshness separately before relying on its mode.
Artifacts
Mocked heartbeat freshness check source
- The authored Python script feeds mocked messages through VehicleState and executes SupervisorCore.step and stream in both conditions, showing exactly what was tested.
Heartbeat-only comparator output
- The executed comparator used heartbeat age for freshness and recorded an abort with no setpoint, establishing the same-scope comparison.
Current generic-freshness output
- The executed unchanged code continued a setpoint with a stale heartbeat and recorded the distinct RC-loss and fresh-pilot-mode outcomes, confirming the defect.
| f.append("local position stale") | ||
| if st.gps is None or st.gps.fix < c.min_fix_type: | ||
| f.append("GPS fix") | ||
| elif not math.isnan(st.gps.eph) and st.gps.eph > c.max_eph_m: |
There was a problem hiding this comment.
A zero or absent GPS horizontal-accuracy reading becomes NaN, which this condition exempts from the 1.5 m limit. With an otherwise valid 3D fix, takeoff preflight advances without confirming the required accuracy. Treat unknown accuracy as a preflight failure.
Artifacts
Executed GPS accuracy preflight check source
- The authored Python command feeds GPS messages through the real state and supervisor code and asserts both control and unknown-accuracy outcomes.
Preflight with measured GPS accuracy
- The executed control run shows that 0.8 m passes while 3.0 m is rejected, establishing that the configured limit works for measured values.
Preflight with zero or absent GPS accuracy
- The executed candidate run shows both inputs becoming NaN and advancing to STREAMING with no preflight failures.
| batt = st.batt_pct | ||
| if batt >= 0 and batt < c.min_batt_pct: | ||
| f.append(f"battery {batt}%") |
There was a problem hiding this comment.
Without SYS_STATUS, battery percentage is -1 and bypasses the minimum; a previously received high reading is also accepted without a freshness check. Takeoff preflight can advance without confirming the required 40% battery level. Require a present, recent reading.
Artifacts
Battery preflight reproduction script
- The executed script feeds MAVLink-like messages into the real vehicle-state and supervisor code for fresh, missing, and retained battery readings; it provides the reproducible check.
Preflight run with low battery readings
- The command ran the reproduction with fresh and retained 20% readings and exited successfully; both kept takeoff in PREFLIGHT.
Preflight run with missing and stale battery telemetry
- The command ran the same reproduction path without SYS_STATUS and with a 100-second-old 80% reading and exited successfully; both advanced takeoff to STREAMING.
| def hold_cmd( | ||
| self, st: VehicleSnapshot, m: Px4Actuator, now: float | None = None | ||
| ) -> Rejection | None: | ||
| if self._holds_px4(st): | ||
| m.set_mode(MAIN_AUTO, SUB_AUTO_LOITER) | ||
| self.sp = None | ||
| self.teleop = None | ||
| self.entered_offboard = False | ||
| self.goto("IDLE", "operator hold", now) |
There was a problem hiding this comment.
If PX4 rejects or misses a Hold or Land command, Hold, E-STOP, Land, and abort still clear setpoints and progress as though the handoff succeeded. The operator can be told the aircraft is holding or landing while PX4 remains armed in Offboard. Confirm the resulting mode before reporting completion.
Artifacts
Mock PX4 mode-change reproduction script
- The authored harness invokes the real connection and supervisor paths against synthetic PX4 feedback for accepted, rejected, and lost commands, making the comparison reproducible.
- Ran all four actions with PX4 accepting their mode commands; reported vehicle mode changed to Hold or Land.
- Ran the same actions with PX4 rejecting commands; ACK result 2 and OFFBOARD feedback coexist with accepted RPCs and completed supervisor transitions.
- Ran the same actions without delivering commands or ACKs to PX4; OFFBOARD feedback persists while the supervisor reports Hold, E-STOP, Landing, or IDLE after abort.
| # Out of the connection: the supervisor. | ||
| ("supervisor_state", String): _zenoh_transport("supervisor_state", String), | ||
| # Camera and gimbal. | ||
| ("video", CompressedVideo): _zenoh_transport("video", CompressedVideo, latest_wins=True), |
There was a problem hiding this comment.
This transport permits encoded video packets to be dropped under congestion, and the default viewer bridge can discard intermediate packets when it falls behind. With a slowed bridge, only 21 of 50 H.265 packets arrived and those packets decoded to one frame. Provide delivery or recovery that preserves usable video without blocking flight control.
Artifacts
Source for the camera-to-bridge H.265 reproduction
- This authored script was run in both modes against synthetic camera replay and the PX4 video transport, with a slow callback in the bridge run; it defines the comparison.
- The executed before command decoded all 50 camera-produced packets into 50 frames and exited 0, establishing that the generated stream is decodable.
Congested default-bridge packet delivery
- The executed after command observed 21 of 50 packets at the default bridge and decoded one frame before exiting 0, demonstrating loss and broken downstream decode.
| RtspCamera.blueprint( | ||
| url=SYNTHETIC_URL, | ||
| frame_id="a8_optical", | ||
| capture_latency_s=0.08, | ||
| ), | ||
| SiyiA8Gimbal.blueprint(mount_xyz=GIMBAL_MOUNT_XYZ_UNMEASURED), |
There was a problem hiding this comment.
The synthetic camera produces 320×180 images, while its paired gimbal publishes intrinsics for 1280×720. Geometry consumers project simulated pixels in the wrong directions, reducing the usefulness of SITL vision checks. This is non-blocking for aircraft operation; match the calibration to the synthetic images.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Artifacts
SITL camera and gimbal reproduction script
- Runs the modules selected by the SITL blueprint and compares actual calibration with an isolated corrected calibration, without changing tracked source.
SITL camera output with published intrinsics
- The executed camera produced a 320×180 image while the gimbal published 1280×720 intrinsics, reversing the sample pixel’s projected angle.
SITL camera output with matching intrinsics
- The same flow with an isolated 320×180 intrinsics override produced matching dimensions and the expected projected angle.
| if now - self._last_aim_mono < 1.0 / self.config.aim_hz: | ||
| return False | ||
| self._last_aim_mono = now | ||
| self.gimbal_target.publish( |
There was a problem hiding this comment.
The public aim() request reports success after publishing gimbal_target, but the PX4 aircraft blueprint has no consumer for that output and the connection has no gimbal-target input. An accepted request therefore issues no A8 actuator command. Connect the target to a command sender or reject requests that cannot be acted on.
Artifacts
Executable gimbal aim contract scenario
- The authored Python scenario invokes the public RPC, observes its output, and inspects the PX4 blueprint contract.
Gimbal aim contract before the PX4 change
- The parent-revision run returned true and published a target; that revision had no PX4 blueprint.
Gimbal aim contract with the PX4 blueprint
- The current-revision run returned true and published a target while finding no blueprint consumer or PX4 connection input.
| px4_visualization(), | ||
| px4_control(), | ||
| RtspCamera.blueprint(url=A8_RTSP_URL, frame_id="a8_optical", capture_latency_s=0.08), | ||
| SiyiA8Gimbal.blueprint(mount_xyz=GIMBAL_MOUNT_XYZ_UNMEASURED), |
There was a problem hiding this comment.
If the aircraft’s A8 is zoomed away from 1×, this blueprint supplies no SDK address, so the gimbal never polls zoom and still publishes fixed 1× intrinsics. Vision consumers then receive calibration for the wrong focal length. Configure zoom polling and withhold calibration when zoom is unknown or unsupported.
Artifacts
Aircraft blueprint and mocked zoom-camera reproduction script
- The authored script starts the gimbal using real aircraft blueprint arguments and compares them with an SDK-IP-configured control.
Control run with SDK address configured
- The executed control queried a mocked 2.5x camera twice and published no CameraInfo, showing the zoom gate working.
Run with the unmodified aircraft blueprint
- The executed blueprint setup made zero zoom queries and published 1x CameraInfo, confirming the missing-address path.
| zoom = self._sdk.query_zoom() | ||
| if zoom is not None: # a lost reply keeps the last known zoom | ||
| with self._lock: | ||
| self._zoom = zoom |
There was a problem hiding this comment.
A missing zoom reply retains the last 1× value indefinitely. If the camera reconnects at another zoom before replies resume, incorrect 1× intrinsics continue to be published. This is a non-blocking calibration concern during outages; expire the cached reading so publication pauses until zoom is known.
Artifacts
Clock and mock zoom reproduction script
- Run this script in before or after mode to exercise the source zoom loop and camera-info gate without hardware.
Original zoom loop after lost replies and reconnect
- The executed source loop retained cached 1x and published four camera-info messages after the simulated 2.5x reconnect, confirming the defect.
In-memory cache invalidation after lost replies
- The same executed scenario with only the loop patched in memory withheld camera info after replies stopped, preventing publication at 2.5x.
| # 2. Camera. | ||
| print(f"video: {frames} frames in {args.seconds:.0f}s") | ||
| ok &= frames > 0 |
There was a problem hiding this comment.
One video packet at the start of the listening window passes this check even if the stream then stalls. The gate can report a healthy camera without detecting a stopped stream, weakening its video check. This is non-blocking; check recent or sustained packet arrivals.
Comments Outside DiffThese findings sit on lines the diff does not cover, so they could not be posted inline. Each one leaves this list once its file changes.
|
This branch has not been deployed
1 participant
What is this feature?
Fly and command a PX4 multicopter from dimOS.
Px4DroneConnection(dimos/robot/px4/connection.py): the only module that talks to PX4. One MAVLink socket (pymavlink, one setpoint writer), vehicle streams (odometry, tf, imu, gps, battery, gimbal attitude, vehicle status), and the flight supervisor. MirrorsR1ProConnection.supervisor_core.py: 20 Hz flight state machine, pure logic, no I/O.IDLE > PREFLIGHT > STREAMING > OFFBOARD_REQ > ARMING > TAKEOFF > HOVER > LANDING, withTELEOPandGOTOfromHOVER. Preflight checks, RC enable switch, fence and ceiling, abort reasons, E-STOP.mavlink.py: socket, vehicle state, PX4 modes, NED/FLU frames.takeoff(altitude_m),go_to(...),land,hold,set_guidance_mode,estop. No arm, mode or raw-setpoint RPC. Goals outside the fence or ceiling are refused.TELEOPholds altitude, and holds position when keys stop.px4-basic,px4-drone,px4-sitl,px4-teleop,px4-sitl-teleop. Gate:tool_sitl_gate.py.mavsdk_serverheartbeats as a ground station with no off switch (PX4 arms with no GCS and cannot detect GCS loss), opens an unauthenticated gRPC port, and in SITL had setpoint timing p99 ~10 ms vs ~1 ms.Why do we need this?
dimOS has no aerial platform. This is the base the skills (#4292) and target following (#4293) build on.
How to Test
uv sync --extra px4 uv run pytest dimos/robot/px4 # PX4 SITL running (HEADLESS=1 make px4_sitl gz_x500), QGC closed: uv run python dimos/robot/px4/tool_sitl_gate.py --flyThe gate (PX4 v1.16.2): takeoff cancelled by the enable switch never arms; takeoff to 2 m; go-to 2 m south at 3 m; go-to past the fence refused; a held key moves 2 m at a locked 3 m and holds on release; land to
IDLE.dimos/robot/px4/README.mdhas the vehicle setup.Hardware, props off (Pixhawk 6C, PX4 1.17, Jetson Orin Nano): takeoff refused with the enable switch low; with it high the chain runs to
TAKEOFF; switch low aborts to Hold; a stick move hands control to the pilot (PILOT_OVERRIDE). Not flown yet.Stack
Five PRs, all against
main; each contains the ones above it. Merge in order. Review only this PR's own commit: 3635440Which issue(s) does this PR close?
None. New platform: PX4 multicopters with a SIYI A8 gimbal camera.
Checklist
🤖 Generated with Claude Code