Skip to main content

6.2.1 ROS2 MIPI Camera Usage Guide

Last Version: 11/09/2025

Overview​

This section introduces a camera driver and a set of ROS 2 nodes based on the JDK library for capturing and processing MIPI CSI camera data.

The JDK library provides hardware interfaces for camera access (JdkCamera), video encoding (JdkEncoder), decoding (JdkDecoder), and video output (JdkVo). ROS 2 nodes use these interfaces to publish data via custom messages.

The system architecture is as follows:

  • camera_node captures camera frames and publishes them to /camera_frames.
  • venc_node encodes raw frames to H.264 and publishes to /encoded_frames.
  • vdec_node decodes encoded frames and publishes them to /decoded_frames.
  • vo_node subscribes to camera frames and displays them on screen (with hardware acceleration).
  • infer_node performs object detection on camera frames (default model: YOLOv8).

Users can subscribe to the relevant topics to access frame data for further processing.

Background​

Hardware Environment​

This driver targets embedded systems with MIPI CSI camera support, such as Spacemit KX series development boards. The JDK SDK interfaces directly with V4L2 for efficient access to camera and video codec hardware.

System Architecture​

Based on the JDK SDK’s functionalities, this toolkit defines a structure of ROS 2 packages and nodes. The main components include:

  • JDK Library Components: Provides C++ interfaces for:
    • Camera capture (JdkCamera)
    • Frame encapsulation (JdkFrame)
    • Hardware encoding (JdkEncoder)
    • Decoding (JdkDecoder)
    • Video output (JdkVo)
    • DMA frame transfer (JdkDma)
  • ROS 2 Interface Package: The jdk_interfaces package defines a custom message type JdkFrameMsg used to transmit frame data.
  • FD Transfer Package: The jdk_fd_transmitter package implements a zero-copy frame transfer mechanism using UNIX sockets, allowing multiple nodes to share DMA buffers and avoid memory copy overhead.
  • Functional Nodes:
    • jdk_camera_node: Captures camera frames and publishes them to /camera_frames
    • jdk_venc_node: Encodes the frames and publishes them to /encoded_frames
    • jdk_vdec_node: Decodes frames and publishes them to /decoded_frames
    • jdk_vo_node: Displays frames using hardware acceleration
    • jdk_infer_node: Performs object detection on /camera_frames using the YOLOv8 model

The following sections will introduce the installation environment, compilation process, node startup parameters, interface definition and usage examples in detail.

Installation and Dependencies​

System Requirements and Environment Setup​

  • Operating System Use Bianbu 2.2 and install ROS 2 Humble (Humble Hawksbill). Follow the official guide: ROS 2 Humble Installation.

    Make sure your system is up to date and install the ROS 2 desktop version:

    sudo apt install ros-humble-desktop

  • Camera Driver Ensure drivers are available for MIPI CSI cameras (the Bianbu kernel supports multiple CSI interfaces by default). Check available camera devices using:

    v4l2-ctl --list-devices

    Example device nodes: /dev/video0, /dev/video50, etc.

  • Required Dependencies Install the required tools and libraries for building and running ROS 2 applications:

    • C/C++ build tools: build-essential, cmake, etc.

    • UUID library: uuid-dev (used for generating frame IDs). Reference: ROS 2 Unique Identifier Messages

    • V4L2 headers (optional): libv4l-dev

    • ROS 2 dependencies: ament_cmake (build tool), rclcpp, std_msgs, rosidl_default_generators, and other common ROS 2 packages (these are included when installing the ROS 2 desktop version)

Install Required Packages (Bianbu 2.2)​

Update your system and install ROS 2 Humble and build dependencies:

sudo apt update
sudo apt install -y ros-humble-desktop build-essential cmake libuuid1 uuid-dev libv4l-dev

After installation, follow the official ROS 2 instructions to set up sources and keys: ROS 2 Humble Installation Guide

Setup Workspace​

Create a ROS 2 workspace (e.g., ~/ros2_ws) and add this project’s source code (jdk folder) into the src directory:

mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
cp -r path_to_jdk_folder ./jdk
cd ~/ros2_ws

JDK SDK Build (Optional)​

If you want to install the JDK library system-wide, build it from the jdk_sdk directory and install to /opt/jdk:

cd jdk/jdk_ros2_ws/jdk_sdk
cmake -DCMAKE_INSTALL_PREFIX=/opt/jdk .
make install

Then set the environment variables:

export CMAKE_PREFIX_PATH=/opt/jdk:$CMAKE_PREFIX_PATH
export LD_LIBRARY_PATH=/opt/jdk/lib:$LD_LIBRARY_PATH

Allows other programs to link to the JDK library with:

find_package(jdk REQUIRED)

Verify Dependencies​

Ensure that the required dependencies are installed before compiling. Common packages include:

  • ros-humble-rclcpp, ros-humble-std-msgs – ROS 2 C++ client library
  • ros-humble-rosidl-default-generators – for generating custom message types
  • ros-humble-rcl-interfaces – usually installed with ROS 2
  • libssl-dev – may be required for inference plugins, encryption, or specific backends

Build and Compilation​

This section explains how to build ROS 2 packages using colcon. Ensure your source code is placed inside your ROS 2 workspace

Configure the ROS 2 Environment​

Open a new terminal, load the ROS 2 environment variables:

source /opt/ros/humble/setup.bash

Navigate to the root of your ROS 2 workspace (the folder containing src):

cd ~/ros2_ws

Compile Interface and Transmission Packages​

To generate message types before building other packages, compile these two packages (jdk_interfaces and jdk_fd_transmitter) first:

colcon build --packages-select jdk_interfaces
source install/setup.bash
colcon build --packages-select jdk_fd_transmitter
source install/setup.bash

Tip: In most cases, you can simply build all packages at once. The steps above are for clarity.

Build All Packages​

To build every package in the workspace at once, run:

colcon build --symlink-install

According to the Official ROS documentation, this will build all packages together without compiling them individually.

After a successful build, you should see directories such as build/, install/, and log/ inside your workspace.

Source the Workspace​

Before running your packages, source the newly built workspace:

source install/setup.bash

According to the Official ROS documentation, this ensures the environment can locate the compiled packages and executables.

Note:

  • If you build outside of a ROS workspace (e.g., in an isolated directory), you may need to adjust LD_LIBRARY_PATH to include the JDK library path (/opt/jdk/lib, if installed).
  • The simplest approach is to keep everything in one workspace and build with colcon build.

Node Launch and Parameters​

The project contains several ROS 2 executable nodes. Each node can be started using ros2 run or ros2 launch and can be configured with parameters. The main nodes and their usage are as follows:

Camera Node (Capture)​

  • Executable name: camera_node (package: jdk_camera_node)

  • Launch command:

    ros2 run jdk_camera_node camera_node [--ros-args -p <param_name>:=<value> ...]

  • Supported parameters:

    • device (string): Path to the camera device. Default: "/dev/video50". Adjust if needed (e.g., "/dev/video0").
    • width (int): Image width in pixels. Default: 1280.
    • height (int): Image height in pixels. Default: 720.
    • socket_path (string): UNIX socket path for FD (file descriptor) transfer. Default: "/tmp/jdk_fd_socket". Used for passing frame buffers.

  • Description:

    • The camera node opens the V4L2 device specified by device, captures NV12 frames, and periodically publishes them to the /camera_frames topic.

    • At the same time, starts a UNIX socket service (at socket_path) for DMA-based frame buffer transfer.

    • Custom settings can be applied using

      -p device:=/dev/video0 -p width:=1280 -p height:=720

Hardware Encoder Node (VENC)​

  • Executable name: venc_node (package: jdk_venc_node)

  • Launch command:

    ros2 run jdk_venc_node venc_node [--ros-args -p <param_name>:=<value> ...]

  • Supported Parameters:

    • width (int): Input frame width. Default: 1920.
    • height (int): Input frame height. Default: 1080.
    • payload (int): Encoding format. Default: CODING_H264 (H.264).
    • format (int): Pixel format. Default: PIXEL_FORMAT_NV12 (NV12).
    • use_dma (bool): Use DMA for transfer. Default: true.
    • venc_fd_socket (string): Socket path for encoded frame output. Default: "/tmp/jdk_fd_enc2out.sock". (Note: There is also an internal subscription socket, defaulting to "/tmp/jdk_fd_socket", which usually does not need to be modified.)

  • Description:

    • The encoder node subscribes to the /camera_frames topic (using SensorData QoS).
    • It receives raw frames and performs H.264 encoding in an internal thread.
    • Encoded frames are then published to the /encoded_frames topic using the custom JdkFrameMsg message type.
    • To use a different encoding format, adjust the payload parameter (for example, set it to CODING_H265).
    • The use_dma parameter controls whether DMA is used for frame transfer.

Hardware Decoder Node (VDEC)​

  • Executable name: venc_node (package: jdk_venc_node)

  • Launch command:

    ros2 run jdk_venc_node venc_node [--ros-args -p <param_name>:=<value> ...]

  • Supported Parameters:

    • width (int): Input frame width. Default: 1920.
    • height (int): Input frame height. Default: 1080.
    • payload (int): Encoding format. Default: CODING_H264 (H.264).
    • format (int): Pixel format. Default: PIXEL_FORMAT_NV12 (NV12).
    • use_dma (bool): Use DMA for transfer. Default: true.
    • dec_fd_socket (string): UNIX socket path for decoded frame output. Default: "/tmp/jdk_fd_dec2out.sock".
    • venc_fd_socket (string): UNIX socket path for encoded output. Default: "/tmp/jdk_fd_enc2out.sock"(matches the encoder node’s venc_fd_socket).

  • Description:

    • The VDEC node subscribes to the /encoded_frames topic and decodes incoming encoded frames.
    • It then publishes the decoded raw frames to the /decoded_frames topic.
    • In most cases, venc_fd_socket should match the encoder node’s output path for getting the encoded data (The default parameters already match the encoder node’s output socket path, so manual changes are usually unnecessary).

Video Output Node (VO)​

  • Executable name: vo_node (package: jdk_vo_node)

  • Launch command:

    ros2 run jdk_vo_node vo_node [--ros-args -p <param_name>:=<value> ...]

  • Supported parameters:

    • width (int): Frame width. Default: 1920.
    • height (int): Frame height. Default: 1080.
    • format (int): Pixel format. Default: PIXEL_FORMAT_NV12.
    • use_dma (bool): Use DMA for transfer. Default: true.
    • cam_fd_socket (string): Camera frame FD socket path. Default: "/tmp/jdk_fd_socket".

  • Description:

    • The VO node subscribes to the /camera_frames topic.
    • It uses the hardware video output (JdkVo) to display frames directly on the screen (overlay output).
    • This node is typically used to verify camera capture.
    • If display functionality is not needed, this node can be skipped.

Inference Node (Infer)​

  • Executable name: infer_node (package: jdk_infer_node)

  • Launch command:

    ros2 run jdk_infer_node infer_node [--ros-args -p <param_name>:=<value> ...]

  • Supported parameters:

    • socket_path (string): FD socket path. Default: "/tmp/jdk_fd_socket" (same as camera_node).

  • Description:

    • The Infer node subscribes to the /camera_frames topic.
    • It uses the built-in YOLOv8 ONNX model (yolov8n.q.onnx) to perform object detection on camera frames.
    • Detection results are printed in the terminal.
    • Frame data is obtained via FD transfer, so no display functionality is required.
    • Users can refere to the sample in yolov8n.q.onnx to replace the default model with their own ONNX model if desired.

Parameter Summary Table​

NodeParameterTypeDefaultDescription
camera_nodedevicestring"/dev/video50"Camera device path (V4L2)
widthint1280Frame width in pixels
heightint720Frame height in pixels
socket_pathstring"/tmp/jdk_fd_socket"FD transfer socket path
venc_nodewidthint1920Input frame width
heightint1080Input frame height
payloadintCODING_H264 (H.264)Encoding format
formatintPIXEL_FORMAT_NV12Pixel format (FourCC)
use_dmabooltrueUse DMA transfer
venc_fd_socketstring"/tmp/jdk_fd_enc2out.sock"Output socket for encoded frames
vdec_nodewidth, heightint1920, 1080Output frame size after decoding
payloadintCODING_H264Encoding format
formatintPIXEL_FORMAT_NV12Pixel format
use_dmabooltrueUse DMA transfer
dec_fd_socketstring"/tmp/jdk_fd_dec2out.sock"Socket for decoded output
venc_fd_socketstring"/tmp/jdk_fd_enc2out.sock"Socket from encoder node
vo_nodewidth, heightint1920, 1080Frame display dimensions
formatintPIXEL_FORMAT_NV12Pixel format
use_dmabooltrueUse DMA
cam_fd_socketstring"/tmp/jdk_fd_socket"Input socket from camera
infer_nodesocket_pathstring"/tmp/jdk_fd_socket"FD socket path for inference

Use --ros-args -p param:=value to override parameters, for example: -p device:=/dev/video0

5. Node Interface Descriptions​

The system mainly uses a custom message type jdk_interfaces/msg/JdkFrameMsg to transmit image frames between nodes. Typical topic interfaces are as follows:

/camera_frames​

  • Message Type: jdk_interfaces/JdkFrameMsg

  • Publisher: camera_node

  • Each message contains raw camera data (default in NV12 format). The message fields include:

    • stamp: Timestamp
    • frame_id: UUID-based Frame ID
    • width and height: Frame dimensions
    • stride: Image row size in bytes
    • size: Frame data size
    • pix_format: V4L2 FourCC pixel format
    • is_dma: Whether the frame uses DMA transfer
    • dma_fd: DMA file descriptor (used if is_dma = true)
    • data: Byte array of image data (used if is_dma = false; NV12 layout)

    Note:

    • When is_dma=true, actual image data is stored in the buffer pointed to by dma_fd, and the data field is empty.
    • When is_dma=false, image bytes are stored in data with NV12 layout.

/encoded_frames​

  • Message Type: jdk_interfaces/JdkFrameMsg

  • Publisher: venc_node

  • This topic contains encoded frame data, typically H.264 stream data.

    • pix_format: Corresponds to the encoding type (e.g., H264 or H265)
    • data: Data is stored either in data or transferred via dma_fd.

/decoded_frames​

  • Message Type: jdk_interfaces/JdkFrameMsg

  • Publisher: vdec_node

  • This topic contains decoded raw image frames in NV12 format. These can be used for further processing or displayed using the VO node.

Notes​

  • Currently, no ROS services or actions are defined in this system. All inter-node communication is handled through the above topics.
  • Users can subscribe to /camera_frames or other topics and retrieve the JdkFrameMsgin callbacks, and convert NV12 data to a viewable format (e.g., RGB using OpenCV) for processing or visualization.

Usage Example​

The following examples demonstrate a typical workflow from compilation to running the system and obtaining camera frame data.

Compile the Workspace​

Assuming your workspace is located at ~/ros2_ws, build all packages using the steps below:

cd ~/ros2_ws
source /opt/ros/humble/setup.bash
colcon build --symlink-install
source install/setup.bash

At this point, all nodes are compiled and ready to run. Please refer to the ROS2 Official Document for more details.

Run the Camera Node​

In a terminal, launch the camera capture node:

ros2 run jdk_camera_node camera_node --ros-args \
  -p device:=/dev/video50 -p width:=1280 -p height:=720
  • This command opens /dev/video50 (or other specified device) and starts publishing frames to /camera_frames.
  • For Customization
    • Change device to match your camera (e.g., /dev/video0)
    • Adjust width and height to set the frame size

Run Other Nodes​

Launch other nodes in separate terminals as needed:

ros2 run jdk_venc_node venc_node
ros2 run jdk_vdec_node vdec_node
ros2 run jdk_vo_node vo_node
ros2 run jdk_infer_node infer_node

What each node does:

  • The encoder node encodes /camera_frames to H.264 and publishes to /encoded_frames
  • The decoder node decodes and publishes to /decoded_frames
  • The display node renders frames directly on the screen
  • The inference node prints object detection results in the terminal

View Topic Data​

After launching the nodes, use ROS 2 CLI tools to inspect the topics:


ros2 topic list | grep jdk
ros2 topic echo /camera_frames

Using ros2 topic echo, you can view the metadata of JdkFrameMsg.

Note: The image data is transmitted in binary format, so you cannot view it directly in the terminal.

You can also write a subscriber node to:

  1. Receive JdkFrameMsg messages.
  2. Extract the NV12 image data.
  3. Convert it to a displayable format (e.g., RGB with OpenCV).
  4. Save or show the images as needed.

Subscriber Example (Python)​

The following example shows how to subscribe to camera frames in a custom ROS 2 node and convert NV12 data to an OpenCV image:

import rclpy
from rclpy.node import Node
from jdk_interfaces.msg import JdkFrameMsg
import numpy as np
import cv2

class CameraListener(Node):
    def **init**(self):
        super().**init**('cam_listener')
        self.sub = self.create_subscription(
            JdkFrameMsg, 'camera_frames', self.callback, 10)

    def callback(self, msg):
        # If data is not using DMA (is_dma=False), read the data array
        if not msg.is_dma:
            width, height = msg.width, msg.height
            # NV12 data is stored in msg.data
            nv12 = np.frombuffer(msg.data, dtype=np.uint8)
            yuv = nv12.reshape((height * 3 // 2, width))  # NV12 total length = height*1.5*width
            # Convert NV12 to BGR image for OpenCV
            bgr = cv2.cvtColor(yuv, cv2.COLOR_YUV2BGR_NV12)
            cv2.imshow('Camera', bgr)
            cv2.waitKey(1)

def main():
    rclpy.init()
    node = CameraListener()
    rclpy.spin(node)
    rclpy.shutdown()

if **name** == '**main**':
    main()
  • This example subscribes to /camera_frames.
  • In the callback, it reads msg.data (assuming is_dma=False) and converts the NV12 frame into a BGR image that can be displayed using OpenCV.
  • If DMA mode is used (is_dma=True), you would access the frame buffer via dma_fd (not shown in this example).

Summary​

  • These examples demonstrate a full workflow, from building the workspace to processing real-time camera frames.

  • Users can customize launch parameters, subscribe to topics, or create their own nodes for more advanced image processing and analysis.

  • This toolkit allows efficient access to MIPI CSI camera data in a ROS 2 environment, while providing encoding, decoding, and inference capabilities for downstream development.