Rockchip RMSL Linux Developer Guide
:::info 文档信息
文档标识：RK-KF-YF-355 · 发布版本：V1.0.1 · 发布日期：2020-12-14 · 页数：12 · 语言：英文
原始文档：Rockchip_Developer_Guide_Linux_RMSL_EN.pdf
:::
章节大纲
- Rockchip RMSL Linux Developer Guide（P1）
- Introduction to RMSL Module Interface（P4）
- Overview（P4）
- RMSL Module（P4）
- Configurations in Buildroot（P4）
- Build and Run（P5）
- Demo Code Introduction（P8）
- RMSL Control Interface（P8）
- Get Data Stream（P10）
- Decoding（P10）
- Display（P10）
- Integration with QT application（P11）
- FAQs（P12）
- Device Disconnected after Opening Camera（P12）
- Introduction to RMSL Module Interface（P4）
在线预览
若预览无法显示，请直接下载 PDF 后用本地阅读器打开。
1. Introduction to RMSL Module Interface
1.1 Overview
Rockchip RMSL module is an USB plug and play device, there are three outputs at the same time: Depth, RGB and IR. Depth outputs YUYV data, RGB and IR output MJPG data.
| Video | Max. Resolution | Format | Frame Rate | Description |
|---|---|---|---|---|
| Depth | 640x480 | YUYV | 15fps | Output disparity image, which can be converted into depth image or point cloud image according to the algorithm |
| RGB | 1920x1080 | MJPG | 30fps | Output RGB image, which can be decoded to NV12 format |
| IR | 640x480 | MJPG | 15fps | Output Infrared image, which can be decoded to NV12 format |
1.2 RMSL Module
The model number of RK RMSL can be obtained from SN number. The currently supported RK structured light models are as follows:
| Model | Matching SN No. | Interface |
|---|---|---|
| RMSL201-1301 | R2011301xxxxxxxxx | USB |
1.3 Configurations in Buildroot
In the Buildroot Linux SDK, the development interfaces and reference demos are located in the app/demo/rmsl directory. The default SDK release version will not build the program. You have to enable the Buildroot configuration BR2_PACKAGE_APP_DEMO_RMSL to enable building. The following application packages that this program depends on should be enabled too:
BR2_PACKAGE_CAMERA_ENGINE_RKISP, used to get V4L2 data stream BR2_PACKAGE_MPP, used to decode MJPG BR2_PACKAGE_LINUX_RGA, used to process YUV data for display BR2_PACKAGE_LIBDRM, used to display on the screen (optional)
Note：
If there is no macro switch or rmsl demo code, please update to the latest SDK or obtain it from the github repository. external/camera_engine_rkisp/ should be updated to 86dc5bf1 apps: rkisp_api: add usb camera supports . If the SDK is not updated to the latest version, it can be obtained through the github repository.
1.4 Build and Run
Enter the SDK directory, build the modules with the following command:
# make app_demo
Please refer to the following command when need to rebuild:
# make app_demo-dirclean
# make app_demo
After finishing the building, two executable binaries rmsl_linux_demo and rmsl_tool will be generated.
# ls -1 /usr/bin/rmsl*
/usr/bin/rmsl_linux_demo
/usr/bin/rmsl_tool
Because of the large number of dependencies, it is recommended to make a clean build after section 1.3.
Connecting the RMSL module, the rmsl_tool can display the device information.
Get node information of device
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | # rmsl tool --list devices _ _ Device /dev/video6 info: usb interface: UVC DEPTH driver name: uvcvideo card type: RV1108 bus info: usb-ff340000.usb-1 _ Device /dev/video8 info: usb interface: UVC RGB driver name: uvcvideo card type: RV1108 bus info: usb-ff340000.usb-1 _ Device /dev/video10 info: usb interface: UVC IR driver name: uvcvideo card type: RV1108 bus info: usb-ff340000.usb-1 _ |
|---|---|
| Note: T |
It also displays other information such as USB device node, driver information
Search the SN Number and software version of the RMSL:
| 1 # rmsl tool --get sn --get version --device /dev/video6 _ _ _ 2 SN: R2011301200801448 3 Version: 2.2.0 | |
|---|---|
| 1 2 3 | # rmsl tool --get sn --get version --device /dev/video6 _ _ _ SN: R2011301200801448 Version: 2.2.0 |
The device should be initialized before getting data stream:
| 1 # rmsl tool --init --device /dev/video6 _ | |
|---|---|
| 1 | # rmsl tool --init --device /dev/video6 _ |
The device should be de-initialized after ending the data flow:
| 1 # rmsl tool --deinit --device /dev/video6 _ | |
|---|---|
| 1 | # rmsl tool --deinit --device /dev/video6 _ |
After ending the data flow, you can also reset the device
| 1 # rmsl tool --reset --device /dev/video6 _ | |
|---|---|
| 1 | # rmsl tool --reset --device /dev/video6 _ |
Once the RMSL was initialized, some open source tools such as v4l2-ctl and vlc are able to get the data stream, but you have to pay attention to the format and size of the data stream.
Another tool rmsl_linux_demo provided in rmsl demo is used to demonstrate complete usage including init: get data stream, decode, deinit, display and other functions. Because the rmsl_linux_demo has initialized the RMSL properly, it is not necessary to use the rmsl_tool to do the initialization first.
The usage help of rmsl_linux_demo:
| 1 # rmsl linux demo --help _ _ 2 Usage of rmsl linux demo: _ _ 3 To display and/or save decode frames to files: 4 --screen-width, screen width, required if need display 5 --screen-height, screen height, required if need display 6 --no-display, disable display, by default it's enabled 7 --save-ir-to, path to save IR decoded frames in NV12 8 --save-depth-to, path to save DEPTH decoded frames with bpp = 16 9 --save-rgb-to, path to save RGB decoded frames in NV12 | |
|---|---|
| 1 2 3 4 5 6 7 8 9 | # rmsl linux demo --help _ _ Usage of rmsl linux demo: _ _ To display and/or save decode frames to files: --screen-width, screen width, required if need display --screen-height, screen height, required if need display --no-display, disable display, by default it's enabled --save-ir-to, path to save IR decoded frames in NV12 --save-depth-to, path to save DEPTH decoded frames with bpp = 16 --save-rgb-to, path to save RGB decoded frames in NV12 |
| As sh |
it on the screen. The display function is turned on by default, if you do not need display, use the --no-display parameter to disable.
Display the image on the screen:
| 1 # rmsl linux demo --screen-width 1536 --screen-height 2048 _ _ 2 3 UVC DEPTH: /dev/video5: draw in (0, 480)[640 x 480] 4 UVC RGB: /dev/video7: draw in (320, 0)[640 x 480] 5 UVC IR: /dev/video9: draw in (640, 480)[640 x 480] 6 | |
|---|---|
| 1 2 3 4 5 6 | # rmsl linux demo --screen-width 1536 --screen-height 2048 _ _ UVC DEPTH: /dev/video5: draw in (0, 480)[640 x 480] UVC RGB: /dev/video7: draw in (320, 0)[640 x 480] UVC IR: /dev/video9: draw in (640, 480)[640 x 480] |
The pictures above show three images on RK3288 screen (1536x2048), from top to bottom and left to right, they are RGB, DEPTH, IR respectively. The size are all 640x480, in which the depth image displays the disparity image in RGB565 format directly.
Don't display the image, but save the frame after decoding
| 1 # rmsl linux demo --no-display \ _ _ 2 --save-ir-to /tmp/ir.bin \ 3 --save-rgb-to /tmp/rgb.bin \ 4 --save-depth-to /tmp/depth.bin | |
|---|---|
| 1 2 3 4 | # rmsl linux demo --no-display \ _ _ --save-ir-to /tmp/ir.bin \ --save-rgb-to /tmp/rgb.bin \ --save-depth-to /tmp/depth.bin |
In the above example, the display function is disabled, and the three frame data are saved to files.
ir.bin, save 640x480 NV12 frame data rgb.bin, save 640x480 NV12 frame data depth.bin, save 640x480 depth data, 2 bytes per pixel
In the above three examples, the data stream size is 640 * 480 by default, but the RGB camera can support 1080p output. YUVPlayer (windows environment) or mplayer (Linux environment) can be used to view the captured files when they are on a PC. and please refer to the following commands for mplayer usage.
| 1 # mplayer /tmp/ir.bin -loop 0 -demuxer rawvideo -fps 15 \ 2 -rawvideo w=640:h=480:size=$((640*480*3/2)):format=nv12 3 4 # mplayer /tmp/rgb.bin -loop 0 -demuxer rawvideo -fps 15 \ 5 -rawvideo w=640:h=480:size=$((640*480*3/2)):format=nv12 6 7 # mplayer /tmp/depth.bin -loop 0 -demuxer rawvideo -fps 15 \ 8 -rawvideo w=640:h=480:size=$((640*480*2)):format=rgb16 | |
|---|---|
| 1 2 3 4 5 6 7 8 | # mplayer /tmp/ir.bin -loop 0 -demuxer rawvideo -fps 15 \ -rawvideo w=640:h=480:size=$((640*480*3/2)):format=nv12 # mplayer /tmp/rgb.bin -loop 0 -demuxer rawvideo -fps 15 \ -rawvideo w=640:h=480:size=$((640*480*3/2)):format=nv12 # mplayer /tmp/depth.bin -loop 0 -demuxer rawvideo -fps 15 \ -rawvideo w=640:h=480:size=$((640*480*2)):format=rgb16 |
2. Demo Code Introduction
The source code is located in the app/demo/rmsl/ directory, which contains RMSL settings, querying interface, getting data stream and decoding and display functions.
| 1 # tree app/demo/rmsl/ 2 app/demo/rmsl/ 3 ├── CMakeLists.txt 4 ├── main.c # main of rmsl linux demo _ _ 5 ├── rkdrm display.c # use DRM to display directly on Overlay Plane of _ Rockchip devices 6 ├── rkdrm display.h _ 7 ├── rmsl api.h # RMSL API Definition _ 8 ├── rmsl ctrl.c # Implementation of rmsl control, query and other _ routines 9 ├── rmsl depth.c # Implementation of depth image conversion _ 10 ├── rmsl tool.c # The rmsl tool source code _ _ 11 ├── vpu decode.c # vpu jpeg hardware decoding interface _ 12 └── vpu decode.h _ | |
|---|---|
| 1 2 3 4 5 6 7 8 9 10 11 12 | # tree app/demo/rmsl/ app/demo/rmsl/ ├── CMakeLists.txt ├── main.c # main of rmsl linux demo _ _ ├── rkdrm display.c # use DRM to display directly on Overlay Plane of _ Rockchip devices ├── rkdrm display.h _ ├── rmsl api.h # RMSL API Definition _ ├── rmsl ctrl.c # Implementation of rmsl control, query and other _ routines ├── rmsl depth.c # Implementation of depth image conversion _ ├── rmsl tool.c # The rmsl tool source code _ _ ├── vpu decode.c # vpu jpeg hardware decoding interface _ └── vpu decode.h _ |
| 2.1 R |
Get the version:
| 1 int rmsl get version(int fd, char *ver, int size); _ _ | |
|---|---|
| 1 | int rmsl get version(int fd, char *ver, int size); _ _ |
Explanation of each parameter:
fd, the file descriptor of the /dev/video node ver, the returned version number size, the size of the ver array, should not be less than RMSL_DATA_SIZE_QUERY
Return 0 on success, or error value if an error occurred.
Get SN number:
| 1 int rmsl get sn(int fd, char *sn, int size); _ _ | |
|---|---|
| 1 | int rmsl get sn(int fd, char *sn, int size); _ _ |
Explanation of each parameter:
fd, the file descriptor of the /dev/video node sn, the returned SN number size, the size of the sn array, should not be less than RMSL_DATA_SIZE_QUERY
Return 0 on success, or error value if an error occurred.
Initialize/de-initialize the device:
int rmsl_init_device(int fd);
int rmsl_deinit_device(int fd);
Initialize or de-initialize the device so that the video node will output data. Explanation of each parameter:
fd, the file descriptor of the /dev/video node
Return 0 on success, or error value if an error occurred.
Reset the device
| 1 int rmsl reset device(int fd); _ _ | |
|---|---|
| 1 | int rmsl reset device(int fd); _ _ |
Explanation of each parameter:
fd, the file descriptor of the /dev/video node
Return 0 on success, or error value if an error occurred.
Enter suspend mode:
| 1 int rmsl suspend device(int fd); _ _ | |
|---|---|
| 1 | int rmsl suspend device(int fd); _ _ |
Explanation of each parameter:
fd, the file descriptor of the /dev/video node
Return 0 on success, or error value if an error occurred.
Get the file path of video devices:
int rmsl_get_devices(char *dev_depth, char *dev_ir, char *dev_rgb, int
silent);
Get the file path of three video devices, such as /dev/video6, /dev/video8, etc.
Explanation of each parameter:
dev_depth, return the path of the Depth device dev_ir, return the path of the IR device dev_rgb, return the path of the RGB device silent, 1 means no log output; 0 will print some logs
Return 0 on success, or error value if an error occurred.
Convert Depth frame data to point cloud image and depth image:
int rmsl_get_point_cloud_depth(const uint16_t *pIn, struct rmsl_pc *pc_out,
uint16_t *depth_out, int width, int height);
Convert to point cloud image and depth image according to disparity image.
Explanation of each parameter:
pIn, disparity image frame data from Depth device, and the BPP is 16 pc_out, output point cloud image. If NULL, the point cloud image is not calculated depth_out, output depth image. If NULL, the depth image is not calculated width, the width of the image height, the height of the image
Return values: 0 means successful calculation A negative means error A positive 0 means that the original frame data of the module is already a depth image and no calculation is required. (No such case currently)
2.2 Get Data Stream
The rkisp_api.so is used to get data stream in the rmsl_linux_demo. Compared with the common USB Camera module, one more initialization operation is required.
For more details about rkisp_api.so interface, please refer to "Rockchip_Developer_Guide_Linux_Camera_EN.pdf".
2.3 Decoding
The Rockchip MPP library is used to decode MJPG in this demo. Which will send the data stream got from rkisp_api to MPP for decoding. Because the decoded buffer may be resized/copied by RGA and then display on screen, this demo allocate buffer from RGA as decoded buffer.
Source code for decoding:
| 1 ret = vpu decode jpeg init(&ctx->decoder, ctx->width, ctx->height); _ _ _ 2 while (has new frame) 3 vpu decode jpeg doing(&ctx->decoder, ctx->cur frame->buf, _ _ _ _ 4 ctx->cur frame->size, ctx->rga bo fd, ctx->rga bo.ptr); _ _ _ _ 5 vpu decode jpeg done(&ctx->decoder); _ _ _ | |
|---|---|
| 1 2 3 4 5 | ret = vpu decode jpeg init(&ctx->decoder, ctx->width, ctx->height); _ _ _ while (has new frame) vpu decode jpeg doing(&ctx->decoder, ctx->cur frame->buf, _ _ _ _ ctx->cur frame->size, ctx->rga bo fd, ctx->rga bo.ptr); _ _ _ _ vpu decode jpeg done(&ctx->decoder); _ _ _ |
Note:
decoder, is vpu decoding handle rga_bo_fd, is RGA buffer handle rga_bo.ptr, is virtual address of RGA buffer
2.4 Display
In order to process the data stream as efficiently as possible and take the characteristics of different Rockchip chips into account, the display of rmsl_linux_demo directly uses the libdrm interface, and the three images are copied to the specific offset of the target buffer through RGA, and finally sent to the overlay Plane of Rockchip VOP.
The main process is shown below. RGB camera MJPG frame IR camera MJPG frame
MPP decode MPP decode
Depth camera frame RGB NV12 frame Infrared NV12 frame
Algorithm calculation Algorithm calculation RGA copy RGA transcode to rgb565 and copy RGA transcode to rgb565 and copy
Depth image Point cloud image Display buffer after three images merge
VOP overlay level display
There are two assumptions here:
Suppose only one overlay layer is available. So use RGA to copy images to different offsets of the target buffer. For example, there is only one overlay layer in px30/rk3326 Display the disparity image as RGB565 which is just for demonstration, the BPP of Depth happens to be 16 as RGB565 is. The outline of the Depth frame is quite clear if display in RGB565 format.
Although there is an additional step of RGA transcoding and copying, the RGA hardware acceleration make it efficiently. The demo can be displayed at 15fps on px30, rk3288, and rk3399 platforms. Because IR and Depth Camera are only 15fps, the maximum frame rate is only 15fps. The following methods can be considered to optimize performance:
If multiple overlay layers are available, NV12 format can be displayed in a separate plane directly. A total of three overlay layers are required, two for RGB and infrared NV12 frame display, and the other for depth image display. In this way, RGA transcoding and/or copying steps can be skipped. RGB Camera can also reach a frame rate of 30fps. Multithreading can improve concurrency, especially when needing to process point cloud images, which involving floating-point operations.
2.4.1 Integration with QT application
If you need QT applications to draw UI components such as menus and buttons, a video area can be reserved on main UI for video overlay plane.
It should be noted that the overlay layer (video) is always above UI, so if there is a pop-up dialog, menu, etc. that overlaps with the video area, we can hide the video overlay plane temporarily.
The above is just to provide a kind of idea, please make your own design as need.
3. FAQs
3.1 Device Disconnected after Opening Camera
The reason is that the power supply is insufficient. Please connect to the USB hub of an external power supply.