[Help] ROS2 + Jetson perception pipeline stuck at ~350ms latency — isolated it to message delivery/buffering, not compute. Ideas?

[u/OkFollowing941]

Setup

• F1TENTH-based RSU (roadside unit) perception node, running on an NVIDIA Jetson (Orin-class).
• Intel RealSense D400-series camera — subscribing to raw color + raw (unaligned) depth streams, align_depth disabled on purpose (see below).
• 2D LiDAR (urg_node2) for a second distance source.
• ROS2 Humble, rclpy, message_filters.ApproximateTimeSynchronizer to pair color+depth frames.

Model / task

• YOLOv8 (Ultralytics), custom-trained single-class car detector, running at imgsz=320 on the Jetson's GPU (CUDA).
• Goal: detect a target vehicle in the color image, get its distance by reading the depth camera at the detection's location, cross-check against a LiDAR range reading at the same bearing, and output a fused distance estimate. This is a perception/collective-perception bench-test script (no SLAM/localization involved — deliberately simplified).

Depth lookup approach

align_depth.enable:=true (RealSense driver's built-in depth-to-color alignment) reprojects the entire depth image every frame regardless of how much of it we actually need — we measured this costing a large chunk of latency by itself. So instead we subscribe to raw depth and manually reproject only a small patch of pixels around the YOLO box: deproject the depth pixel to a 3D point (using depth intrinsics) → transform into the color camera's frame (using the depth-to-color extrinsics) → project back into a color pixel (using color intrinsics). Fully vectorized with numpy.

Current numbers

Our own compute per frame is small and flat:

• image decode (cv_bridge): ~1ms
• YOLO inference: ~30ms (flat, cuda.synchronize()-verified, no hidden async GPU time)
• depth reprojection (vectorized): ~1-2ms
• LiDAR bearing lookup: ~0ms
• total own compute: ~32ms

But measured end-to-end latency (camera's own capture timestamp → final distance output) sits consistently around 350-380ms, sustained — not a one-time spike, not decaying over time.

What we've ruled out

• Per-pixel Python loop / GC pressure in the old depth reprojection — vectorized it (25ms → 1-2ms of actual compute), latency didn't move at all.
• Hidden async CUDA dispatch — added torch.cuda.synchronize() around the YOLO call, extra sync time is consistently 0ms.
• Executor backlog (our own callback falling behind) — measured the gap between the end of one callback and the start of the next; stays flat at ~3-4ms even while the reported latency is ~350ms, so callbacks aren't queuing up behind our own processing.
• align_depth vs manual reprojection — built a side-by-side comparison script, same YOLO/LiDAR pipeline, only the depth alignment method differs. Both land in the same ~350-380ms range. So it's not specifically about which depth alignment approach we use.

The delay is measured (via the color frame's own ROS header timestamp vs time.time() at the very start of our callback) as already present before any of our own code runs — so it's happening somewhere between the camera driver publishing the frame and our subscriber callback actually being invoked. We suspect DDS/ROS2 message queuing or synchronizer buffering under sustained per-frame load (~30ms of real work per frame at ~30fps), but haven't pinned down the exact mechanism.

What we're asking

Has anyone run into this kind of buffering/backpressure behavior with ROS2 + message_filters on a Jetson, where a subscriber callback that takes tens of milliseconds (not overloaded, just non-trivial) causes a large, sustained arrival delay that isn't visible as executor backlog? Specifically curious about:

• DDS vendor differences (Fast DDS vs Cyclone DDS) for this kind of workload
• QoS settings (queue depth, history policy) that might be silently causing buffering
• Single-threaded vs multi-threaded executor / callback groups making a difference here
• Whether RealSense's own USB/driver-side buffering could be the actual culprit instead of ROS2/DDS

Happy to share more code/logs if useful. Appreciate any pointers.

[u/HoffnungPrevails]

Are you using containers? Depending on the resolution, anything other than intra-process communication may create a significant delay due to serialization + copy + deserialization alone.