🛠 Boîte de vision RTSP : caméra MIPI, accélération NPU RKNN et VPU

Objectives

Build a vision box using the RK3568's 1 TOPS INT8 NPU (RKNN framework, supporting TensorFlow/TF-Lite/PyTorch/Caffe/ONNX) and VPU (4Kp60 H.265/H.264/VP9 decoding, 1080p60 encoding): camera on the 2-lane MIPI CSI connector, frame inference, and RTSP streaming. Radxa documents each component: official RTSP, OpenCV, and AI/RKNN sections in docs.radxa.com (app-development).

Hardware

• Radxa ROCK 3B running Radxa OS
• Compatible 2-lane MIPI CSI camera (ribbon cable to the board's CSI connector)
• Cooling recommended (sustained VPU+NPU load), SSD/eMMC for videos

Full Code

# 1. Activer la caméra (Radxa OS) : rsetup -> Camera overlay, reboot, puis :
ls /dev/video* ; v4l2-ctl --list-devices        # chercher la pipeline mipi-csi

# 2. Chaîne d'acquisition (gstreamer, décodage/encodage matériel) :
sudo apt install gstreamer1.0-tools gstreamer1.0-plugins-good
gst-launch-1.0 v4l2src device=/dev/video11 ! video/x-raw,width=1280,height=720 ! \
  videoconvert ! autovideosink                  # aperçu live depuis la CSI

# 3. Inférence RKNN : installer la toolchain rknn-toolkit2 (hôte x86) pour
#    convertir le modèle ONNX -> .rknn, puis runtime rknnlite sur la carte :
sudo pip3 install rknn-toolkit-lite2            # wheel selon la doc AI Radxa
python3 detect.py --model model.rknn --cam 0    # script du tuto AI officiel

# 4. Diffusion RTSP (encodage 1080p60 matériel) :
sudo apt install v4l2loopback-dkms
# gst-launch-1.0 ... ! mpph264enc ! rtspclientsink location=rtsp://... (voir tuto RTSP)

Expected Verification

v4l2-ctl --list-devices lists the CSI pipeline; gst-launch displays the camera stream; the RKNN script displays boxes/labels and latency per frame (the 1 TOPS NPU handles real-time with lightweight INT8-quantized models like YOLO); a VLC client reads the RTSP stream. cat /sys/class/devfreq/*npu*/load (or rknn top) shows the NPU active.

Common Errors

• Testing RKNN under a mainline/custom Armbian image: the NPU depends on the vendor kernel — libmali/RKNN blobs and panfrost do not mix (documented issue on self-built images).
• CSI ribbon cable poorly inserted or contacts reversed: no /dev/video device for the pipeline.
• Confusing decoding and encoding: the VPU decodes at 4Kp60 and encodes at 1080p60 (the radxa.com marketing sheet that swaps the two is a typo).
• Converting a model not supported by RKNN: use the ONNX converter and quantize to INT8 on an x86 PC, not on the board.
• Overheating under sustained VPU+NPU load: add a heatsink/fan before concluding it is a software throttle.