5.2.8 ROS2 MIPI Camera Usage

This section provides a camera driver and a set of ROS 2 nodes based on the JDK library for acquiring and processing MIPI CSI camera data. The JDK library encapsulates hardware features such as camera access (JdkCamera), video encoding (JdkEncoder), decoding (JdkDecoder), and video output (JdkVo). ROS 2 nodes invoke these features and publish data in ROS 2 using custom messages. The system architecture is as follows: camera data is collected and published to /camera_frames by the camera_node; raw frames can be H.264 encoded by the venc_node and published to /encoded_frames; the vdec_node decodes encoded frames and publishes them to /decoded_frames; the vo_node displays the frames on screen (via hardware-accelerated output); and the infer_node performs object detection inference using camera frames (by default using the YOLOv8 model). Users can subscribe to the relevant topics to obtain and process frame data. This documentation covers system background, dependencies, build instructions, node startup and parameters, interface explanations, and examples to help users integrate and use this package quickly.

1. Background Introduction

● Hardware Environment

This driver targets embedded systems that support MIPI CSI camera interfaces. A typical use case is the Spacemit KX series development boards. The JDK library (JDK SDK) directly interfaces with V4L2 to efficiently access camera and video codec hardware.

● System Architecture

Based on the JDK SDK’s functionalities, this toolkit defines a structure of ROS 2 packages and nodes. The main components include:

• JDK Library Components: Provides C++ interfaces for:
  • Camera capture (JdkCamera)
  • Frame encapsulation (JdkFrame)
  • Hardware encoding (JdkEncoder)
  • Decoding (JdkDecoder)
  • Video output (JdkVo)
  • DMA frame transfer (JdkDma)
• ROS 2 Interface Package: The jdk_interfaces package defines a custom message type JdkFrameMsg used to transmit frame data.
• FD Transfer Package: The jdk_fd_transmitter package implements a zero-copy frame transfer mechanism using UNIX sockets, allowing multiple nodes to share DMA buffers and avoid memory copy overhead.
• Functional Nodes:
  • jdk_camera_node: Captures camera frames and publishes them to /camera_frames
  • jdk_venc_node: Encodes the frames and publishes them to /encoded_frames
  • jdk_vdec_node: Decodes frames and publishes them to /decoded_frames
  • jdk_vo_node: Displays frames using hardware acceleration
  • jdk_infer_node: Performs object detection on /camera_frames using the YOLOv8 model

The following sections will introduce the installation environment, compilation process, node startup parameters, interface definition and usage examples in detail.

2.Installation and Dependencies

● Operating System

It is recommended to use Bianbu 2.2 and install ROS 2 Humble (Humble Hawksbill). Refer to the official ROS 2 documentation at [docs.ros.org](https://docs.ros.org/en/humble/Installation.html#:~:text=* Ubuntu Linux ,04) for installation steps. Ensure your system is updated and install the full desktop version of ROS 2:sudo apt install ros-humble-desktop

● Camera Driver

A driver that supports MIPI CSI cameras is required (e.g., the Bianbu kernel supports multiple CSI interfaces by default). You can check available camera device nodes using:v4l2-ctl --list-devices

Common paths include /dev/video0, /dev/video50, etc.

---

● Required Dependencies

Install the following tools and libraries needed for building and running:

• C/C++ build tools: build-essential, cmake, etc.
• UUID library: uuid-dev – used to generate frame IDs
• V4L2 headers (optional): libv4l-dev – for V4L2 header files
• ROS 2 dependencies: Already included in the desktop version: ament_cmake, rclcpp, std_msgs, rosidl_default_generators, etc.

2.1 Example Installation Command (on Bianbu 2.2)

sudo apt update
sudo apt install -y ros-humble-desktop build-essential cmake libuuid1 uuid-dev libv4l-dev

Once installed, follow the ROS 2 official instructions for setting up sources and keys: docs.ros.org - ROS 2 Humble Installation

2.2 Workspace Setup

After installing ROS 2, create a workspace (e.g., ~/ros2_ws) and place this project's source code (the jdk folder) inside the src directory:

mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
cp -r path_to_jdk_folder ./jdk
cd ~/ros2_ws

2.3JDK SDK Build (Optional)

If you want to install the JDK library into your system, follow the steps in the jdk_sdk directory inside the project and install it to /opt/jdk:

cd jdk/jdk_ros2_ws/jdk_sdk
cmake -DCMAKE_INSTALL_PREFIX=/opt/jdk .
make install

Then set the environment variables:

export CMAKE_PREFIX_PATH=/opt/jdk:$CMAKE_PREFIX_PATH
export LD_LIBRARY_PATH=/opt/jdk/lib:$LD_LIBRARY_PATH

This allows other programs to link to the JDK library using find_package(jdk REQUIRED).

2.4 Dependency Check

• Ensure that the required dependencies are installed before compiling. Common packages include:
  • ros-humble-rclcpp, ros-humble-std-msgs – ROS 2 C++ client library
  • ros-humble-rosidl-default-generators – for generating custom message types
  • ros-humble-rcl-interfaces – usually installed with ROS 2
  • libssl-dev – may be required for inference plugins, encryption, or specific backends

3.Build and Compilation

Once the source code is placed in your ROS 2 workspace, use colcon to build all ROS 2 packages. The general procedure is as follows:

3.1 Configure the ROS 2 Environment

In a new terminal, first source the ROS 2 environment:

source /opt/ros/humble/setup.bash

This loads the necessary environment variables for ROS 2.

3.2 Navigate to the Workspace

Change to the root directory of your workspace that contains the src folder:

cd ~/ros2_ws

3.3 Compile Interface and Transmission Packages

To generate message types first, it’s recommended to build the jdk_interfaces and jdk_fd_transmitter packages first:

colcon build --packages-select jdk_interfaces
source install/setup.bash
colcon build --packages-select jdk_fd_transmitter
source install/setup.bash

💡 Tip: In most cases, you can compile all packages together; the steps above are just for clarity.

3.4 Build All Packages

To build all packages in the workspace at once:

colcon build --symlink-install

This will compile all ROS 2 packages in the workspace. After a successful build, you should see directories such as build/, install/, and log/ inside your workspace.

3.5 Source the Workspace

After building, you must source the workspace before using it:

source install/setup.bash

This ensures the environment can locate the compiled packages and executables.

⚠️ Note: If you build outside of a ROS workspace (e.g., in an isolated directory), you may need to adjust LD_LIBRARY_PATH to include the JDK library path (/opt/jdk/lib, if installed there). The simplest approach is to keep everything in one workspace and build with colcon build.

4. Node Launch Methods and Parameters

This project contains several ROS 2 executable nodes. Each node can be launched using ros2 run or ros2 launch, and their behavior can be adjusted through parameters.

4.1 Camera Node (Capture Node)

• Executable name: camera_node
• Package name: jdk_camera_node
• Launch command:

ros2 run jdk_camera_node camera_node [--ros-args -p <param_name>:=<value> ...]

• Supported Parameters:
  • device (string): Path to the video device. Default: "/dev/video50"
  • width (int): Image width. Default: 1280
  • height (int): Image height. Default: 720
  • socket_path (string): FD transfer UNIX socket path. Default: "/tmp/jdk_fd_socket"
• Description: The camera node opens the specified V4L2 device and captures NV12-format frames, publishing them periodically to the /camera_frames topic. It also launches a UNIX socket server (at socket_path) for DMA-based frame transfer. Example parameter usage:

-p device:=/dev/video0 -p width:=1280 -p height:=720

4.2 Hardware Encoding Node (VENC)

• Executable name: venc_node
• Package name: jdk_venc_node
• Launch command:

ros2 run jdk_venc_node venc_node [--ros-args -p <param_name>:=<value> ...]

• Supported Parameters:
  • width (int): Input frame width. Default: 1920
  • height (int): Input frame height. Default: 1080
  • payload (int): Encoding format. Default: CODING_H264 (H.264)
  • format (int): Pixel format. Default: PIXEL_FORMAT_NV12
  • use_dma (bool): Use DMA transfer. Default: true
  • venc_fd_socket (string): Output FD socket path. Default: "/tmp/jdk_fd_enc2out.sock"

📝 Internal input socket defaults to "/tmp/jdk_fd_socket" and typically doesn't need to be changed.

• Description: Subscribes to /camera_frames (with SensorData QoS), encodes the raw frames using H.264, and publishes encoded frames to /encoded_frames using the custom message JdkFrameMsg. You can change payload to other encoding types (e.g., CODING_H265), and toggle DMA usage with use_dma.

4.3 Hardware Decoding Node (VDEC)

• Executable name: vdec_node
• Package name: jdk_vdec_node
• Launch command:

ros2 run jdk_vdec_node vdec_node [--ros-args -p <param_name>:=<value> ...]

• Supported Parameters:
  • width, height: Decoded frame dimensions. Default: 1920x1080
  • payload: Encoding format. Default: CODING_H264
  • format: Pixel format. Default: PIXEL_FORMAT_NV12
  • use_dma: Use DMA. Default: true
  • dec_fd_socket: FD socket for decoded frames. Default: "/tmp/jdk_fd_dec2out.sock"
  • venc_fd_socket: Input socket from encoder. Default: "/tmp/jdk_fd_enc2out.sock"
• Description: Subscribes to /encoded_frames, decodes incoming frames, and publishes the raw decoded data to /decoded_frames. Make sure venc_fd_socket matches the encoder's socket path.

4.4 Video Output Node (VO)

• Executable name: vo_node
• Package name: jdk_vo_node
• Launch command:

ros2 run jdk_vo_node vo_node [--ros-args -p <param_name>:=<value> ...]

• Supported Parameters:
  • width, height: Frame dimensions. Default: 1920x1080
  • format: Pixel format. Default: PIXEL_FORMAT_NV12
  • use_dma: Use DMA. Default: true
  • cam_fd_socket: Socket path for camera frames. Default: "/tmp/jdk_fd_socket"
• Description: Subscribes to /camera_frames and displays them on screen using hardware-accelerated output via JdkVo. Useful for validating the camera capture. This node is optional if display is not needed.

4.5 Inference Node

• Executable name: infer_node
• Package name: jdk_infer_node
• Launch command:

ros2 run jdk_infer_node infer_node [--ros-args -p <param_name>:=<value> ...]

• Supported Parameters:
  • socket_path (string): UNIX socket path. Default: "/tmp/jdk_fd_socket"
• Description: Subscribes to /camera_frames and performs object detection using the built-in YOLOv8 ONNX model (yolov8n.q.onnx). Results are printed in the terminal. It accesses frame data via file descriptors (FD), independent of the display.

---

Parameter Summary Table

Node	Parameter	Type	Default	Description
camera_node	device	string	"/dev/video50"	Camera device path (V4L2)
	width	int	1280	Frame width in pixels
	height	int	720	Frame height in pixels
	socket_path	string	"/tmp/jdk_fd_socket"	FD transfer socket path
venc_node	width	int	1920	Input frame width
	height	int	1080	Input frame height
	payload	int	CODING_H264 (H.264)	Encoding format
	format	int	PIXEL_FORMAT_NV12	Pixel format (FourCC)
	use_dma	bool	true	Whether to use DMA
	venc_fd_socket	string	"/tmp/jdk_fd_enc2out.sock"	Output socket for encoded frames
vdec_node	width, height	int	1920, 1080	Output frame size after decoding
	payload	int	CODING_H264	Encoding format
	format	int	PIXEL_FORMAT_NV12	Pixel format
	use_dma	bool	true	Use DMA transfer
	dec_fd_socket	string	"/tmp/jdk_fd_dec2out.sock"	Socket for decoded output
	venc_fd_socket	string	"/tmp/jdk_fd_enc2out.sock"	Socket from encoder node
vo_node	width, height	int	1920, 1080	Frame display dimensions
	format	int	PIXEL_FORMAT_NV12	Pixel format
	use_dma	bool	true	Use DMA
	cam_fd_socket	string	"/tmp/jdk_fd_socket"	Input socket from camera
infer_node	socket_path	string	"/tmp/jdk_fd_socket"	FD socket path for inference

Use --ros-args -p param:=value to override parameters, for example: -p device:=/dev/video0

5.Node Interface Descriptions

The system mainly uses a custom message type jdk_interfaces/msg/JdkFrameMsg to transmit image frames between nodes. Typical topic interfaces are as follows:

---

Topic: /camera_frames

• Message Type: jdk_interfaces/JdkFrameMsg
• Publisher: camera_node

Each message contains raw camera data (default in NV12 format). The message fields include:

• stamp: Timestamp
• frame_id: UUID-based Frame ID
• width, height: Frame dimensions
• stride: Image row size in bytes
• size: Frame data size
• pix_format: V4L2 FourCC pixel format
• is_dma: Whether DMA is used
• dma_fd: File descriptor for DMA buffer (if used)
• data: Raw image byte array (if not using DMA)

🔍 When is_dma=true, actual image data is stored in the buffer pointed to by dma_fd, and the data field is empty. When is_dma=false, image bytes are stored in data with NV12 layout.

---

Topic: /encoded_frames

• Message Type: jdk_interfaces/JdkFrameMsg
• Publisher: venc_node

This topic contains encoded frame data, typically H.264 stream data.

• pix_format: Corresponds to the encoding type (e.g., H264 or H265)
• data: Stores encoded bytes or references DMA buffer via dma_fd

---

Topic: /decoded_frames

• Message Type: jdk_interfaces/JdkFrameMsg
• Publisher: vdec_node

This topic contains decoded raw image frames in NV12 format. These can be used for further processing or displayed using the VO node.

---

📌 Note: Currently, no ROS services or actions are defined in this system. All inter-node communication is handled through the above topics.

Users can subscribe to /camera_frames or other topics and retrieve the JdkFrameMsg. You can then convert NV12 image data into displayable formats (e.g., RGB) using libraries like OpenCV.

6.Usage Example

The following examples demonstrate a typical workflow from compilation to running the system and obtaining camera frame data.

---

6.1 Compile the Workspace

Assuming your workspace is located at ~/ros2_ws, build all packages using the steps below:

cd ~/ros2_ws
source /opt/ros/humble/setup.bash
colcon build --symlink-install
source install/setup.bash

At this point, all nodes are compiled and ready to run.

---

6.2 Run the Camera Node

In a terminal, launch the camera capture node:

ros2 run jdk_camera_node camera_node --ros-args \
  -p device:=/dev/video50 -p width:=1280 -p height:=720

This command opens /dev/video50 (or other specified device) and starts publishing frames to /camera_frames.

---

6.3 Run Other Nodes

In additional terminals, launch other nodes as needed:

ros2 run jdk_venc_node venc_node
ros2 run jdk_vdec_node vdec_node
ros2 run jdk_vo_node vo_node
ros2 run jdk_infer_node infer_node

• The encoder node encodes /camera_frames to H.264 and publishes to /encoded_frames
• The decoder node decodes and publishes to /decoded_frames
• The display node renders frames directly on the screen
• The inference node prints object detection results in the terminal

---

6.4 View Topic Data

After launching the nodes, use ROS 2 CLI tools to inspect the topics:

ros2 topic list | grep jdk
ros2 topic echo /camera_frames

Note: ros2 topic echo shows only metadata. Image data is binary and must be processed by custom code.

---

6.5 Python Subscriber Example

Below is a simple Python example that subscribes to /camera_frames and displays the image using OpenCV (for non-DMA data):

import rclpy
from rclpy.node import Node
from jdk_interfaces.msg import JdkFrameMsg
import numpy as np
import cv2

class CameraListener(Node):
    def __init__(self):
        super().__init__('cam_listener')
        self.sub = self.create_subscription(
            JdkFrameMsg, 'camera_frames', self.callback, 10)

    def callback(self, msg):
        if not msg.is_dma:
            width, height = msg.width, msg.height
            nv12 = np.frombuffer(msg.data, dtype=np.uint8)
            yuv = nv12.reshape((height * 3 // 2, width))  # NV12 layout
            bgr = cv2.cvtColor(yuv, cv2.COLOR_YUV2BGR_NV12)
            cv2.imshow('Camera', bgr)
            cv2.waitKey(1)

def main():
    rclpy.init()
    node = CameraListener()
    rclpy.spin(node)
    rclpy.shutdown()

if __name__ == '__main__':
    main()

This script subscribes to /camera_frames, reads the msg.data (assuming is_dma=False), decodes NV12 to a BGR image using OpenCV, and displays it.

⚠️ If DMA mode is used (is_dma=True), you must handle the dma_fd manually. This example does not cover DMA access.

---

✅ Final Note

These examples show a complete workflow from building to real-time camera frame processing. Users can customize startup parameters, subscribe to topics, or write their own nodes for advanced image processing and analysis.

With this toolkit, you can efficiently access MIPI CSI camera data in a ROS 2 environment, and leverage encoding, decoding, and inference functions for downstream development.