Rockchip RKNPU User Guide RKNN API

:::info 文档信息

页数：45　·　语言：英文

原始文档：Rockchip_RKNPU_User_Guide_RKNN_API_V1.2.0_EN.pdf

:::

在线预览

若预览无法显示，请直接下载 PDF 后用本地阅读器打开。

---

Classification Level: Top Secret( ) Secret( ) Internal( ) Public(√)

RKNN API For RKNPU User Guide

(Technology Department, Graphic Compute Platform Center)

Mark: [ ] Changing [√] Released	Version:	1.2.0
	Author:	HPC/NPU Team
	Completed Date:	14/Jan/2022
	Reviewer:	Vincent
	Reviewed Date:	14/Jan/2022

瑞芯微电子股份有限公司

Rockchip Electronics Co., Ltd.

(Copyright Reserved)

Revision History

Version	Modifier	Date	Modify Description	Reviewer
v0.6.0	HPC Team	2/Mar/2021	Initial version	Vincent
v0.7.0	HPC Team	22/Apr/2021	Remove description of swapping input data channels	Vincent
v1.0.0	HPC Team	30/Apr/2021	Release version	Vincent
v1.1.0	HPC Team	13/Aug/2021	1. Add rknn tensor mem flags _ _ _ 2. Add query commands for input/output Tensor native attributes 3. Add the memory layout of NC1HWC2	Vincent
v1.2.0b1	NPU Team	07/Jan/2022	1. Add RK3588/RK3588s pla 2. Add rknn set core mask interface _ _ _ 3. Add rknn dump context interface _ _ 4. Add details for input and output API	Vincent
V1.2.0	HPC Team	14/Jan/2022	1. Add the Glossary 2. Add the NPU SDK guide and building and compiling instruction 3. Add the section of how to do debugging 4. Add the description of C2 in the section NATIVE LAYOUT _	Vincent

Overview

RKNN SDK provides programming interfaces for RK3566/RK3568 chip platforms with NPU,

which can help users deploy RKNN models exported from RKNN-Toolkit2 and accelerate the

implementation of AI applications.

Supported Hardware Platforms

This document applies to the following hardware platforms:

RK3566, RK3568, RK3588, RK3588S

Note: RK356X is used to indicate RK3566/RK3568 in this document. RK3588 is used to

indicate RK3588/RK3588S in this document.

Glossary

RKNN Model: It is the binary file running on the RKNPU, with the suffix .rknn.

Inference with board: It refer to use RKNN-Toolkit2 API interface to run the model. Actually,

the model is running on the NPU on the development board.

HIDL: It denotes interface description language (IDL) to specify the interface between an

android HAL and its users.

CTS: It is the Compatibility Test Suite from android automatic testing kit.

VTS: It is the Vendor Test Suite from android automatic testing kit.

DRM: It is the Direct Rendering Manager, a main stream of graph displaying framework on

linux.

tensor: A multi-dimensional array of data.

fd: It is the file descriptor for representing the buffer.

NATIVE_LAYOUT: It refers to the native layout of NPU. In other words, it has the best

performance when NPU is processing data with this memory layout. i8 model: A quantized RKNN model which running by 8-bit signed integer data.

fp16 model: A non-quantized RKNN model which running by 16-bit half-float data.

Instructions

4.1 RKNN SDK Development Process

Before using the RKNN SDK, users first need to utilize the RKNN-Toolkit2 to convert the user's

model to the RKNN model.

After getting the RKNN model file, users can choose using C interface to develop the

application. The following chapters will explain how to develop applications based on the RKNN

SDK on RK356X/RK3588 platform.

4.2 RKNN Linux Platform Development Instructions

4.2.1

RKNN API Library For Linux

For the RK3566/ RK3568, the SDK library file is librknnrt.so under the directory of

<sdk>/rknpu2/ runtime.

4.2.2

Example Usage

The SDK provides MobileNet image classification, SSD object detection, YOLOv5 object

detection demos. These demos are the reference for developers to develop applications based on the

RKNN SDK. The demo is located in the path of <sdk>/rknpu2/examples. Let's take RK356X

rknn_mobilenet_demo as an example to explain how to get started quickly.

1. Compile Demo Source Code

cd examples/rknn_mobilenet_demo

# set GCC_COMPILER in build-linux.sh to the correct compiler path ./build-linux_RK356X.sh

2. Deploy to the RK356X device

adb push install/rknn_mobilenet_demo_Linux /userdata/

3. Run Demo

adb shell
cd /userdata/rknn_mobilenet_demo_Linux/
export LD_LIBRARY_PATH=./lib

./rknn_mobilenet_demo model/RK356X/mobilenet_v1.rknn model/dog_224x224.jpg

4.3 RKNN Android Platform Development Instructions

4.3.1

RKNN API Library For Android

There are two ways to call the RKNN API on the Android platform

1. The application can link librknnrt.so directly

2. Application link to librknn_api_android.so implemented by HIDL on Android platform

For Android devices that need to pass the CTS/VTS test, you can use the RKNN API based on

the Android platform HIDL implementation. If the device does not need to pass the CTS/VTS test, it

is recommended to directly link and use librknnrt.so as this way can shorten the callingand linking

processing of each interface to offer better performance.

The code for the RKNN API implemented using Android HIDL is located in the

vendor/rockchip/hardware/interfaces/neuralnetworks directory of the RK356X Android system SDK.

When the Android system is compiled, some NPU-related libraries will be generated (for

applications only need to link librknn_api_android.so), as shown below:

/system/lib/librknn_api_android.so /system/lib/librknnhal_bridge.rockchip.so /system/lib64/librknn_api_android.so /system/lib64/librknnhal_bridge.rockchip.so /vendor/lib64/rockchip.hardware.neuralnetworks@1.0.so /vendor/lib64/rockchip.hardware.neuralnetworks@1.0-adapter-helper.so /vendor/lib64/librknnrt.so /vendor/lib64/hw/rockchip.hardware.neuralnetworks@1.0-impl.so

Alternatively, user can use the following command to recompile and generate the above library separately.

mmm vendor/rockchip/hardware/interfaces/neuralnetworks/ -j8

4.3.2

Example Usage

The SDK currently provides examples of MobileNet image classification, SSD object detection

and YOLOv5 object detection. The demo code is located in the <sdk>/rknpu2/examples directory.

Users can use NDK to compile the demo executed in the Android command line. Let's take RK356X

rknn_mobilenet_demo as an example to explain how to use this demo on the Android platform:

1. Compile Demo Source Code

cd examples/rknn_mobilenet_demo

#set ANDROID_NDK_PATH under build-android.sh to the correct NDK path ./build-android_RK356X.sh

2. Deploy to the RK3566 device

adb push install/rknn_mobilenet_demo_Android /data/

3. Run Demo

adb shell
cd /data/rknn_mobilenet_demo_Andorid/
export LD_LIBRARY_PATH=./lib

./rknn_mobilenet_demo model/RK356X/mobilenet_v1.rknn model/dog_224x224.jpg

Note: The above demo uses the librknnrt.so by default. For developers who want to use

librknn_api_android.so, replacing the librknnrt.so with the librknn_api_android.so on the corresponding

CMakeList.txt of that demo and compiling it again using above steps.

4.4 RKNN C API

4.4.1

API Process Description

At present, there are two groups of APIs that can be used on RK356X, namely the general API interface and the API interface for the zero-copy process. The main difference between the two sets

of APIs is that each time the general interface updates the frame data, the data allocated externally

needs to be copied to the input memory of the NPU during runtime, while the interface of the zero-

copy process uses pre-allocated memory directly (including memory created by NPU or other frame,

such as DRM), reducing the cost of copying memory. When the user input data has only a virtual

address, only the common API interface can be used; when the user input data has a physical address

or fd, both sets of interfaces can be used.

For the general API interface, the rknn_input structure is initialized at first. The frame data is

contained in the structure. The input of models is set by using rknn_inputs_set function. After the

completion of inference, the inference output can be acquired by the rknn_outputs_get function and

performed for post-processing. The frame data must be updated before inference. The process of

general API call is shown in Figure 3-1, and user behavior is indicated by the process in yellow font. Figure 3-1: General process of calling general interface of API

For the zero-copy interface of API , after allocating memory, the rknn_tensor_mem structure is

initialized by the memory information from allocated memory. Creating this structure before the

inference is necessary. And the memory information can be retrieved by reading this structure after

inference. According to whether the user needs to allocate the modular memory inside model

(input/output/weights/intermediate result) and the differences of memory representation (file

descriptor/physical address, etc.), there are the following three typical zero-copy calling processes, as

shown in Figure 3-2, Figure 3-3 and Figure 3-4 respectively. The red font indicates the interface and data structure specially added for zero copy and the italic indicates the data structure passed between

interface calls.

1. Input/Output memory is allocated by API during runtime

Figure 3-2: Process of zero-copy API interface call (input/output allocated internally)

As shown in Figure 3-2, the input/output memory information structure created by the

rknn_create_mem interface includes the file descriptor and physical addresses. The RGA interface

utilizes the memory information allocated by the NPU during runtime. In the above figure,

preprocess_rga represents the RGA interface, and stream_fd represents the input source buffer represented by fd in RGA, postprocess_cpu represents the CPU implementation of post processing.

2. Input/output memory is allocated externally

Figure 3-3: The process zero-copy API interface call (input/output allocated externally)

As shown Figure 3-3, flush_cache indicates that the user needs to call the interface associated

with the memory type to flush the output buffer.

3. Input/output/weights/internal tensor memory is allocated externally Figure 3-4: The process of zero-copy API interface call (input/output/weights/internal tensor

allocated externally) 4.4.1.1 API internal processing flow

When inferring the RKNN model, the original data need to go through three major processes:

input processing, running model on NPU, and output processing. Currently, according to different

model input formats and quantization methods, there are two different processes inside the general

API interface.

1. int8 quantized model and the number of input channels is 1 or 3 or 4

The processing flow of the original data is shown in Figure 3-5. Assuming that the input is a 3-

channel model, the user must ensure that the color sequence of the R, G, and B channels is consistent

with the training model. Normalization, quantization, and model inference are all running on the

NPU. The output data layout format and dequantization process is running on the CPU.

Figure 3-5: Optimized data processing flow

2. int8 quantized model and non-quantized model when number of input channels is 2

or >= 4

The flow of data processing is shown in Figure 3-6. The normalization, quantification, data

layout format conversion, and dequantization of data are all running on the CPU except for the

inference of the model which runs on the NPU. In this scenario, the processing efficiency of the input

data flow will be lower than the optimized one in Figure 3-5.

Figure 3-6: Common data processing flow

For the zero-copy API, there is only one processing flow for the internal process of the API, as

shown in Figure 3-5. Conditions for using zero copy are listed below:

1. The number of input channels is 1, 3 or 4. 2. The input width is 8 pixel aligned.

2. Int8 asymmetric quantized model.

4.4.1.2 Quantification and Dequantization

The quantization method, quantization data type and quantization parameters used in

quantization and dequantization can be queried through the rknn_query interface.

At the moment, the NPU of RK3566/RK3568 only supports asymmetric quantization, and does

not support dynamic fixed-point quantization. The combination of data type and quantization

method includes:

int8 (asymmetric quantization) 

int16 (asymmetric quantization, not yet implemented) 

float16 

Generally speaking, the normalized data is stored in 32-bit floating point data. For conversion of

32-bit floating point data to 16-bit floating point data, it is better for referring to the IEEE-754

standard. The following describes the quantization process, assuming that the normalized 32-bit

floating point data is D.

1. float32 to int8(asymmetric quantization)

S , ZP , the data Assuming that the asymmetric quantization parameter of the input tensor is q

quantization process is expressed as the following formula:

)) , , / ( ( -128 ZP S D clamp round D   q q

In the above formula, clamp means to limit the value to a certain range. round means rounding

processing.

2. float32 to int16(asymmetric quantization)

S , ZP , the data Assuming that the asymmetric quantization parameter of the input tensor is q

quantization process is expressed as the following formula: )) , , / ( ( 32767 -32768 ZP S D clamp round D   q q

The dequantization process is the inverse process of quantization, and the dequantization

formula can be deduced according to the above quantization formula, which will not be repeated

here.

4.4.2

API Reference

4.4.2.1 rknn_init

The rknn_init will do the following things, such as, creating the rknn_context object, loading the

RKNN model, and performing specific initialization behaviors according to the flag and

rknn_init_extend structure.

API	rknn init _
Description	Initialize rknn
Parameters	rknn context *context: The pointer of rknn context object. After the function is called, the _ _ obejct of the context will be returned with the information.
	void *model: Binary data for the RKNN model or the path of RKNN model.
	uint32 t size: When model is stored in binary data, it indicates the size of the model. The _ size is 0 when model is given as the path.
	uint32 t flag: A specific initialization flag. Now, it only has the following flags. _ RKNN FLAG COLLECT PERF MASK: For querying the time it took to run each _ _ _ _ particular layer during runtime. RKNN FLAG MEM ALLOC OUTSIDE: Used to indicate that model inputs, outputs, _ _ _ _ weights, and intermediate tensor memory are all allocated by the user.
	rknn init extend: The extended information during specific initialization. It is disabled at _ _ the moment, which indicates this must be passed by the NULL.
Return	int: Error code (See RKNN Error Code).

Sample Code: rknn_context ctx;

int ret = rknn_init(&ctx, model_data, model_data_size, 0, NULL);

4.4.2.2 rknn_set_core_mask

This function sets the specific cores running inside the NPU. For now, it only works at the

RK3588.

API	rknn set core mask _ _ _
Description	Set the cores for the NPU.
Parameters	rknn context context: The object of rknn context. _
	rknn core mask core mask: The specification of NPU core setting. It has the following _ _ _ choices: RKNN NPU CORE AUTO : Referring to automatic mode, meaning that it will select _ _ _ the idle core inside the NPU. RKNN NPU CORE 0 : Running on the NPU0 core _ _ _ RKNN NPU CORE 1: Runing on the NPU1 core _ _ _ RKNN NPU CORE 2: Runing on the NPU2 core _ _ _ RKNN NPU CORE 0 1: Running on both NPU0 and NPU1 core simultaneously. _ _ _ _ (Unavailable currently) RKNN NPU CORE 0 1 2: Running on both NPU0, NPU1 and NPU2 simultaneously. _ _ _ _ _ (Unavailable currently)
Return	int: Error code (See RKNN Error Code).

Sample Code:

rknn_context ctx;

rknn_core_mask core_mask = RKNN_NPU_CORE_0;
int ret = rknn_set_core_mask(ctx, core_mask);

4.4.2.3 rknn_dup_context

The rknn_dup_context creates a new context object referring to the same model. The new

context is useful in the condition where the weight of the model need to be reused during executing

the same model on the multi-threading. (Unavailable for RK3588)

API	rknn dup context _ _
Description	Creates a new context for the same model, to reuse the weight of the model.
Parameters	rknn context *context in: The pointer of rknn context object. After the function is called, _ _ _ the obejct of the input context will be initialized.
	rknn context *context out: The pointer of a rknn context output object where information _ _ _ about this new created object is returned.
Return	int: Error code (See RKNN Error Code).

Sample Code:

rknn_context ctx_in; rknn_context ctx_out;

int ret = rknn_dup_context(&ctx_in, &ctx_out);

4.4.2.4 rknn_destroy

This function is used to release the rknn_context and its related resources.

API	rknn destroy _
Description	Destroy the rknn context object and its related resources. _
Parameters	rknn context context: The rknn context object that is going to be destroyed. _ _
Return	int: Error code (See RKNN Error Code).

Sample Code:

rknn_context ctx;

int ret = rknn_destroy(ctx);

4.4.2.5 rknn_query

The rknn_query function can query the information of models and SDK, including model input

and output information, layer-by-layer running time, total model inference time, SDK version,

memory usage information, user-defined strings and other information.

API	rknn query _
Description	Query the information about the model and the SDK.
Parameters	rknn context context: The object of rknn contex. _ _
	rknn query cmd cmd: The query command. _ _
	void* info: The structure object that stores the result of the query.
	uint32 t size: The size of the info structure object. _
Return	int: Error code (See RKNN Error Code).

Currently, the SDK supports following query commands:

Query command	Return result structure	Function
RKNN QUERY IN OUT NUM _ _ _ _	rknn input output num _ _ _	Query the number of input and output Tensors.
RKNN QUERY INPUT ATTR _ _ _	rknn tensor attr _ _	When using the general API interface, query the input Tensor attribute.
RKNN QUERY OUTPUT ATTR _ _ _	rknn tensor attr _ _	When using the general API interface, query the output Tensor attribute.
RKNN QUERY PERF DETAIL _ _ _	rknn perf detail _ _	Query the running time of each layer of the network. It only works when the flag of RKNN FLAG COLLECT PERF _ _ _ _ MASK is set via using the rknn init. _
RKNN QUERY PERF RUN _ _ _	rknn perf run _ _	Query the total time of inference (excluding setting input/output) in microseconds .
RKNN QUERY SDK VERSION _ _ _	rknn sdk version _ _	Query the SDK version.
RKNN QUERY MEM SIZE _ _ _	rknn mem size _ _	Query the memory size allocated to the weights and internal tensors in the network.
RKNN QUERY CUSTOM STRING _ _ _	rknn custom string _ _	Query the user-defined strings in the RKNN model.
RKNN QUERY NATIVE INPUT A _ _ _ _ TTR	rknn tensor attr _ _	When using the zero-copy API interface, it queries the native input Tensor attribute, which is the model input attribute directly read by the NPU.
RKNN QUERY NATIVE OUTPUT _ _ _ ATTR _	rknn tensor attr _ _	When using the zero-copy API interface, query the native output Tensor attribute, which is the model output attribute directly from the NPU.

Next we will explain each query command in detail.

1）Query the SDK version

The RKNN_QUERY_SDK_VERSION command can be used to query the version information of the RKNN SDK. You need to create the rknn_sdk_version structure object first.

Sample Code:

rknn_sdk_version version; ret = rknn_query(ctx, RKNN_QUERY_SDK_VERSION, &version,

sizeof(rknn_sdk_version));
printf("sdk api version: %s\n", version.api_version);
printf("driver version: %s\n", version.drv_version);

2）Query the number of input and output Tensor

The RKNN_QUERY_IN_OUT_NUM command can be used to query the number of model

input and output Tensor. You need to create the rknn_input_output_num structure object first.

Sample Code:

rknn_input_output_num io_num; ret = rknn_query(ctx, RKNN_QUERY_IN_OUT_NUM, &io_num,

sizeof(io_num));

printf("model input num: %d, output num: %d\n", io_num.n_input, io_num.n_output);

3）Query input Tensor attribute (for general API interface)

The RKNN_QUERY_INPUT_ATTR command can be used to query the attribute of the model

input Tensor. You need to create the rknn_tensor_attr structure object first.

Sample Code:

rknn_tensor_attr input_attrs[io_num.n_input];

memset(input_attrs, 0, sizeof(input_attrs));
for (int i = 0; i < io_num.n_input; i++) {
input_attrs[i].index = i;

ret = rknn_query(ctx, RKNN_QUERY_INPUT_ATTR, &(input_attrs[i]),

sizeof(rknn_tensor_attr));

}

4）Query output Tensor attribute (for general API interface)

The RKNN_QUERY_OUTPUT_ATTR command can be used to query the attribute of the model

output Tensor. You need to create the rknn_tensor_attr structure object first.

Sample Code: rknn_tensor_attr output_attrs[io_num.n_output];

memset(output_attrs, 0, sizeof(output_attrs));
for (int i = 0; i < io_num.n_output; i++) {
output_attrs[i].index = i;

ret = rknn_query(ctx, RKNN_QUERY_OUTPUT_ATTR, &(output_attrs[i]),

sizeof(rknn_tensor_attr));

}

5）Query layer-by-layer inference time of model

After the rknn_run interface is called, the RKNN_QUERY_PERF_DETAIL command can be

used to query the layer-by-layer inference time in microseconds. The premise of using this command

is that the flag parameter of the rknn_init interface needs to include the

RKNN_FLAG_COLLECT_PERF_MASK flag.

Sample Code:

rknn_context ctx; int ret = rknn_init(&ctx, model_data, model_data_size, RKNN_FLAG_COLLECT_PERF_MASK, NULL); ...

ret = rknn_run(ctx,NULL);

... rknn_perf_detail perf_detail; ret = rknn_query(ctx, RKNN_QUERY_PERF_DETAIL, &perf_detail,

sizeof(perf_detail));

6）Query total inference time of model

After the rknn_run interface is called, the RKNN_QUERY_PERF_RUN command can be used

to query the inference time of the model (not including setting input/output) in microseconds.

Sample Code: rknn_context ctx;

int ret = rknn_init(&ctx, model_data, model_data_size, 0, NULL);

ret = rknn_run(ctx,NULL);

... rknn_perf_run perf_run; ret = rknn_query(ctx, RKNN_QUERY_PERF_RUN, &perf_run,

sizeof(perf_run));

7）Query the memory allocation of the model

After the rknn_init interface is called, when the user needs to allocate memory for network, the

RKNN_QUERY_MEM_SIZE command can be used to query the weights of the model and the

internal memory (excluding input and output) in the network. The requirement of using this

command is that the flag parameter of the rknn_init interface needs to enable the

RKNN_FLAG_MEM_ALLOC_OUTSIDE flag.

Sample Code:

rknn_context ctx; int ret = rknn_init(&ctx, model_data, model_data_size, RKNN_FLAG_MEM_ALLOC_OUTSIDE , NULL); rknn_mem_size mem_size; ret = rknn_query(ctx, RKNN_QUERY_MEM_SIZE, &mem_size,

sizeof(mem_size));

8）Query User-defined string in the model

After the rknn_init interface is called, if the user has added custom strings when generating the

RKNN model, the RKNN_QUERY_CUSTOM_STRING command can be used to retrieve user-

defined strings.

Sample Code:

rknn_context ctx;

int ret = rknn_init(&ctx, model_data, model_data_size, 0, NULL);

rknn_custom_string custom_string; ret = rknn_query(ctx, RKNN_QUERY_CUSTOM_STRING, &custom_string,

sizeof(custom_string));

9）Query native input tensor attribute (for zero-copy API interface) The RKNN_QUERY_NATIVE_INPUT_ATTR command can be used to query the native

attribute of the model input Tensor. You need to create the rknn_tensor_attr structure object first.

Sample Code:

rknn_tensor_attr input_attrs[io_num.n_input];

memset(input_attrs, 0, sizeof(input_attrs));
for (int i = 0; i < io_num.n_input; i++) {
input_attrs[i].index = i;

ret = rknn_query(ctx, RKNN_QUERY_NATIVE_INPUT_ATTR,

&(input_attrs[i]), sizeof(rknn_tensor_attr));

}

10） Query native output tensor attribute (for zero-copy API interface)

The RKNN_QUERY_NATIVE_OUTPUT_ATTR command can be used to query the native

attribute of the model output Tensor. You need to create the rknn_tensor_attr structure object first.

Sample Code:

rknn_tensor_attr output_attrs[io_num.n_output];

memset(output_attrs, 0, sizeof(output_attrs));
for (int i = 0; i < io_num.n_output; i++) {
output_attrs[i].index = i;

ret = rknn_query(ctx, RKNN_QUERY_NATIVE_OUTPUT_ATTR,

&(output_attrs[i]), sizeof(rknn_tensor_attr));

}

4.4.2.6 rknn_inputs_set

The input data of the model can be set by the rknn_inputs_set function. This function can

support multiple inputs, each of which is a rknn_input structure object. The user needs to set these

object field before passing in rknn_inputs_set function.

API	rknn inputs set _ _
Description	Set the model input data.
Parameter	rknn context context: The object of rknn contex. _ _
	uint32 t n inputs: Number of inputs. _ _
	rknn input inputs[]: Array of rknn input. _ _
Return	int: Error code (See RKNN Error Code).

Sample Code:

rknn_input inputs[1];

memset(inputs, 0, sizeof(inputs));
inputs[0].index = 0;
inputs[0].type = RKNN_TENSOR_UINT8;
inputs[0].size = img_width*img_height*img_channels;
inputs[0].fmt = RKNN_TENSOR_NHWC;
inputs[0].buf = in_data;
inputs[0].pass_through = 0;
ret = rknn_inputs_set(ctx, 1, inputs);

4.4.2.7 rknn_run

The rknn_run function will perform a model inference. The input data need to be configured by

the rknn_inputs_set function or zero-copy interface before rknn_run is called.

API	rknn run _
Description	Perform a model inference.
Parameter	rknn context context: The object of rknn contex. _ _
	rknn run extend* extend: Reserved for extension. It is not used currently and accepted _ _ NULL only.
Return	int: Error code (See RKNN Error Code).

Sample Code:

ret = rknn_run(ctx, NULL);

4.4.2.8 rknn_wait

This interface is used for non-blocking mode inference and has not been implemented yet.

4.4.2.9 rknn_outputs_get

The rknn_outputs_get function can get the output data from the model. This function can get

multiple output data, each of which is a rknn_output structure object that needs to be created and

initialized in turn before the function is called. There are two ways to store buffers for output data:

1. The user allocate and release buffers themselves. In this case, the rknn_output.is_prealloc

needs to be set to 1, and the rknn_output.buf points to users’ allocated buffer;

2. The other is allocated by rknn. At this time, the rknn_output .is_prealloc needs to be set to

0. After the function is executed, rknn_output.buf will be created and store the output data.

API	rknn outputs get _ _
Description	Get model inference output data.
Parameter	rknn context context: The object of rknn context. _ _
	uint32 t n outputs: Number of output. _ _
	rknn output outputs[]: Array of rknn output. _ _
	rknn run extend* extend: Reserved for extension, currently not used yet. Accepting _ _ NULL only.
Return	int: Error code (See RKNN Error Code).

Sample Code:

rknn_output outputs[io_num.n_output];

memset(outputs, 0, sizeof(outputs));
for (int i = 0; i < io_num.n_output; i++) {
outputs[i].index = i;
outputs[i].is_prealloc = 0;
outputs[i].want_float = 1;

}

ret = rknn_outputs_get(ctx, io_num.n_output, outputs, NULL);

4.4.2.10 rknn_outputs_release

The rknn_outputs_release function will release the relevant resources of the rknn_output object.

API	rknn outputs release _ _
Description	Release the rknn output object _
Parameter	rknn context context: rknn context object _ _
	uint32 t n outputs: Number of output. _ _
	rknn output outputs[]: The array of rknn output to be released. _ _
Return	int: Error code (See RKNN Error Code).

Sample Code:

ret = rknn_outputs_release(ctx, io_num.n_output, outputs);

4.4.2.11 rknn_create_mem_from_mb_blk

It has not been implemented Currently.

4.4.2.12 rknn_create_mem_from_phys

When the user wants to allocate memory for NPU, the rknn_create_mem_from_phys function

can create a rknn_tensor_mem structure and return its pointer. This function will pass the physical

address, virtual address and size, and the information related to the external memory to the

rknn_tensor_mem structure.

API	rknn create mem from phys _ _ _ _
Description	Create rknn tensor mem structure and allocate memory through physical address _ _
Parameter	rknn context context: rknn context object. _ _
	uint64 t phys addr: The physical address of buffer. _ _
	void *virt addr: The virtual address of buffer. _
	uint32 t size: The size of buffer. _
Return	rknn tensor mem*: The tensor memory information structure pointer. _ _

Sample Code:

//suppose we have got buffer information as input_phys, input_virt and size

rknn_tensor_mem* input_mems [1];

input_mems[0] = rknn_create_mem_from_phys(ctx, input_phys, input_virt, size);

4.4.2.13 rknn_create_mem_from_fd

When the user wants to allocate for NPU, the rknn_create_mem_from_fd function a

rknn_tensor_mem structure and return its pointer. This function filled with the file descriptor, logical

address and size, and the information related to the external memory will be assigned to the

rknn_tensor_mem structure.

API	rknn create mem from fd _ _ _ _
Description Parameter	Create rknn tensor mem structure and allocate memory through file descriptor _ _
	rknn context context: rknn context object. _ _
	int32 t fd: The file descriptor of buffer. _
	void *virt addr: The virtual address of buffer, which indicates the beginning of fd. _
	uint32 t size: The size of buffer. _
	int32 t offset: The offset corresponding for file descriptor and virtual address _
Return	rknn tensor mem*: The tensor memory information structure pointer. _ _

Sample Code:

//suppose we have got buffer information as input_fd, input_virt and size

rknn_tensor_mem* input_mems [1];

input_mems[0] = rknn_create_mem_from_fd(ctx, input_fd, input_virt, size, 0);

4.4.2.14 rknn_create_mem

When the user wants to allocate memory which can be used directly in the NPU, the

rknn_create_mem function can create a rknn_tensor_mem structure and get its pointer. This function

passes in the memory size and initializes the rknn_tensor_mem structure at runtime.

API	rknn create mem _ _
Description	Create rknn tensor mem structure internally and allocate memory during runtime _ _
Parameter	rknn context context: rknn context object. _ _
	uint32 t size: The size of buffer. _
Return	rknn tensor mem*: The tensor memory information structure pointer. _ _

Sample Code:

//suppose we have got buffer size

rknn_tensor_mem* input_mems [1];

input_mems[0] = rknn_create_mem(ctx, size);

4.4.2.15 rknn_destory_mem

The rknn_destory_mem function destroys the rknn_tensor_mem structure. However, the memory

allocated by the user needs to be released manually.

API	rknn destory mem _ _
Description	Destroy rknn tensor mem structure _ _
Parameter	rknn context context: rknn context object. _ _
	rknn tensor mem*: The tensor memory information structure pointer. _ _
Return	int: Error code (See RKNN Error Code).

Sample Code:

rknn_tensor_mem* input_mems [1];

int ret = rknn_destroy_mem(ctx, input_mems[0]);

4.4.2.16 rknn_set_weight_mem

If the user has allocated memory for the network weights, after initializing the corresponding

rknn_tensor_mem structure, the NPU can use the memory through the rknn_set_weight_mem

function. This function must be called before calling rknn_run.

API	rknn set weight mem _ _ _
Description	Set up the rknn tensor mem structure containing weights memory information _ _
Parameter	rknn context context: rknn context object. _ _
	rknn tensor mem*: The tensor memory information structure pointer. _ _
Return	int: Error code (See RKNN Error Code).

Sample Code:

rknn_tensor_mem* weight_mems [1];

int ret = rknn_set_weight_mem(ctx, weight_mems[0]);

4.4.2.17 rknn_set_internal_mem

If the user has allocated memory for the internal tensor in network, after initializing the

corresponding rknn_tensor_mem structure, the NPU can use the memory through the

rknn_set_internal_mem function. This function must be called before calling rknn_run.

API	rknn set internal mem _ _ _
Description	Set up the rknn tensor mem structure containing internal tensor memory information in _ _ network
Parameter	rknn context context: rknn context object. _ _
	rknn tensor mem*: The pointer to the tensor memory information structure. _ _
Return	int: Error code (See RKNN Error Code).

Sample Code:

rknn_tensor_mem* internal_tensor_mems [1];

int ret = rknn_set_internal_mem(ctx, internal_tensor_mems[0]);

4.4.2.18 rknn_set_io_mem

If the user has allocated memory for the input/output tensor in network, after initializing the

corresponding structure, the NPU can use the memory through the rknn_tensor_mem

rknn_set_io_mem function. This function must be called before calling rknn_run.

API	rknn set io mem _ _ _
Description	Set up the rknn tensor mem structure containing input/output tensor memory information _ _ in network
Parameter	rknn context context: rknn context object. _ _
	rknn tensor mem*: The pointer to the tensor memory information structure . _ _
Return	int: Error code (See RKNN Error Code).

Sample Code:

rknn_tensor_mem* io_tensor_mems [1];

int ret = rknn_set_io_mem(ctx, io_tensor_mems[0]);

4.4.3

RKNN Definition of Data Structcture

4.4.3.1 rknn_sdk_version

The structure rknn_sdk_version is used to indicate the version information of the RKNN SDK.

The following table shows the definition:

Field	Type	Meaning
api version _	char[]	SDK API version information.
drv version _	char[]	SDK driver version information.

4.4.3.2 rknn_input_output_num

The structure rknn_input_output_num represents the number of input and output Tensor，The

following table shows the definition:

Field	Type	Meaning
n input _	uint32 t _	The number of input tensor
n output _	uint32 t _	The number of output tensor

4.4.3.3 rknn_tensor_attr

The structure rknn_tensor_attr represents the attribute of the model's Tensor. The following table

shows the definition:

Field	Type	Meaning
index	uint32 t _	Indicates the index position of the input and output Tensor.
n dims _	uint32 t _	The number of Tensor dimensions.
dims	uint32 t[] _	Values for each dimension.
name	char[]	Tensor name.
n elems _	uint32 t _	The number of Tensor data elements.
size	uint32 t _	The memory size of Tensor data.
fmt	rknn tensor format _ _	The format of Tensor dimension, has the following format: RKNN TENSOR NCHW _ _ RKNN TENSOR NHWC _ _ RKNN TENSOR NC1HWC2 _ _
type	rknn tensor type _ _	Tensor data type, has the following data types: RKNN TENSOR FLOAT32 _ _ RKNN TENSOR FLOAT16 _ _ RKNN TENSOR INT8 _ _ RKNN TENSOR UINT8 _ _ RKNN TENSOR INT16 _ _ RKNN TENSOR UINT16 _ _

		RKNN TENSOR INT32 _ _ RKNN TENSOR INT64 _ _ RKNN TENSOR BOOL _ _
qnt type _	rknn tensor qnt type _ _ _	Tensor Quantization Type, has the following types of quantization: RKNN TENSOR QNT NONE: Not quantified; _ _ _ RKNN TENSOR QNT DFP: Dynamic fixed _ _ _ point quantization; RKNN TENSOR QNT AFFINE ASYMMETR _ _ _ _ IC: Asymmetric quantification.
fl	int8 t _	RKNN TENSOR QNT DFP: quantization _ _ _ parameter
zp	int32 t _	RKNN TENSOR QNT AFFINE ASYMMETRIC _ _ _ _ quantization parameter.
scale	float	RKNN TENSOR QNT AFFINE ASYMMETRIC _ _ _ _ quantization parameter.
stride	uint32 t _	The number of pixels that actually store one line of image data, which is equal to the number of valid data pixels in one row + the number of invalid pixels that are filled in for the hardware to quickly jump to the next line.
size with stride _ _	uint32 t _	The actual byte size of image data (including the byte size of filled invalid pixels).
pass through _	uint8 t _	0: means unconverted data. 1: means converted data, Note: Conversion includes normalization and quantization.

4.4.3.4 rknn_perf_detail

The structure rknn_perf_detail represents the performance details of the model. The definition of

the structure is shown in the following table:

Field	Type	Meaning
perf data _	char*	The performance details contain the running time of each layer of the network stored in string type.
data len _	uint64 t _	The data length of string type of perf data _

4.4.3.5 rknn_perf_run

The structure rknn_perf_run represents the total inference time of the model. The definition of

the structure is shown in the following table:

Field	Type	Meaning
run duration _	int64 t _	The total inference time of the network (not including setting input/output) in microseconds.

4.4.3.6 rknn_mem_size

The structure rknn_mem_size represents the memory allocation when the model is initialized.

The definition of the structure is shown in the following table:

Field	Type	Meaning
total weight size _ _	uint32 t _	The memory size allocated for weights of the network.
total internal size _ _	uint32 t _	The memory size allocated for internal tensor of the network.

4.4.3.7 rknn_tensor_mem

The structure rknn_tensor_mem represents the tensor memory information. The definition of

the structure is shown in the following table:

Field	Type	Meaning
virt addr _	void*	The virtual address of tensor.
phys addr _	uint64 t _	The physical address of tensor.
fd	int32 t _	The file descriptor of tensor.
offset	int32 t _	The offset of fd and virtual address.
size	uint32 t _	The actual size of tensor.
flags	uint32 t _	rknn tensor mem has the following type of flags: _ _ RKNN TENSOR MEMORY FALGS ALLO _ _ _ _ C INSIDE: indicates that the rknn tensor mem _ _ _ structure is created at runtime. The user does not need to pay attention to the flags.
priv data _	void*	private data.

4.4.3.8 rknn_input

The structure rknn_input represents a data input to the model, used as a parameter to the

rknn_inputs_set function. The following table shows the definition:

Field	Type	Meaning
index	uint32 t _	The index position of this input.
buf	void*	The pointer of the input data buffer.
size	uint32 t _	The memory size of the input data buffer.
pass through _	uint8 t _	When set to 1, buf will be directly set to the input node of the model without any pre-processing.
type	rknn tensor type _ _	The type of input data.
fmt	rknn tensor format _ _	The format of input data.

4.4.3.9 rknn_output

The structure rknn_output represents a data output of the model, used as a parameter to the

rknn_outputs_get function. This structure will be assigned with data after calling rknn_outputs_get.

The following table shows the definition of each member of rknn_output:

Field	Type	Meaning
want float _	uint8 t _	Indicates whether the output data needs to be converted to float type.
is prealloc _	uint8 t _	Indicates whether the buffer that stores the output data is pre-allocated.
index	uint32 t _	The index position of this output.
buf	void*	The pointer pointing to the output data buffer.
size	uint32 t _	Output data buffer size in byte.

4.4.3.10 rknn_init_extend

The structure rknn_init_extend represents the extended information when the runtime is

initialized. It is not available currently. The definition of the structure is shown in the following

table:

Field	Type	Meaning
ctx	rknn context _	The initialized rknn context object. _
reserved	uint8 t[128] _	The reserved data.

4.4.3.11 rknn_run_extend

The structure rknn_run_extend represents the extended information during model inference. It is

not available currently. The definition of the structure is shown in the following table:

Field	Type	Meaning
frame id _	uint64 t _	The index of frame of current inference.
non block _	int32 t _	0 means blocking mode, 1 means non-blocking mode, non-blocking mode means that the rknn run _ call returns immediately.
timeout ms _	int32 t _	The timeout of inference in milliseconds.
fence fd _	int32 t _	For the non-blocking inference. (Not Available)

4.4.3.12 rknn_output_extend

The structure rknn_output_extend means to obtain the extended information of the output. It is

not available currently. The definition of the structure is shown in the following table:

Field	Type	Meaning
frame id _	int32 t _	The frame index of the output result.

4.4.3.13 rknn_custom_string

The structure rknn_custom_string represents the custom string set by the user when converting

the RKNN model. The definition of the structure is shown in the following table:

Field	Type	Meaning
string	char[]	User-defined string.

4.4.4

Instruction of API Input and Output

4.4.4.1 General Input and Output of API (Without Zero Copy)

1. rknn_inputs_set

When calling this API, it can set up the parameter from rknn_inputs_set, that is, like the

example shown on the section of 4.4.2.6, the input data type and size can be configured by the rknn_input

structure.

The detail explanation of input: The pass_through is one of the member of rknn_input

structure. If pass_through is set as 1, then the input data will be imported and computed inside the model

in which none of the conversion will happen. If it is 0, the input tensor will be transformed first the rknn

api based on the fmt and type from the tensor (inside the RKNN API) and imported into model. As shown

on the table below, as an example, when the model has the type of int8, with the pass_through set as 1,

the only available input data type of the tensor is int8. Besides, the tensor data layout should be NHWC

when the number of input channels is 1, 3, 4. For other channel numbers, the input tensor data layout is

NC1HWC2. When the pass_through is set as 0, the only supported data type for input tensor is NHWC.

Table-1: The NPU supports input configuration without zero copy

Data Type of Input	pass through _	Number of Channels	Supported input layout	Remark
uint8 t _	0	-	NHWC
float32	0	-	NHWC
int8	1	1,3,4	NHWC	Only support i8 model
int8	1	None 1,3,4 channel	NC1HWC2	Only support i8 model
float16	0	-	NHWC
float16	1	-	NC1HWC2	Only support fp16 model

2. rknn_outputs_get

When calling this function, it can set up want_float, which is one of the member of rknn_output

structure, to select the input data type, layout, etc. Since the default setting will set the data type of output layer as float32 when converting the model into rknn model via rknn-toolkit2, the querying result

for non-quantified models will still be float32 type even if the want_float is set as 0. The zero copy

should be utilized if the non float32 data result is desired.

Table-2: The NPU supports input configuration without zero copy

want float _	Output data Type	Supported output layout
1	float32	NCHW
0	float32	NCHW
	int8	NCHW
	float16	NCHW

4.4.4.2 Input and Output with Zero Copy of API

Two functions need to use for this purpose:

rknn_create_mem,

rknn_set_io_mem

The specification of input data (involving the size, data type, layout, etc.) should be configured

according to the actual data that need to be allocated by the zero copy memory.

Explanation of input: pass_through is the member of rknn_tensor_attr structure. It is the same as the

one mentioned in the rknn_input.

Table-3: The NPU supports input configuration with zero copy

Data Type of Input	pass through _	Number of Channels	Supported input layout	Remark
uint8 t _	0	-	NHWC
float32	0	-	NHWC
int8	1	1,3,4	NHWC	Only support i8 model
int8	1	None 1,3,4 channel	NC1HWC2	Only support i8 model
float16	0	-	NHWC
float16	1	-	NC1HWC2	Only support fp16 model

For the output data, when not in the mode of NATIVE_LAYOUT, it is suggested considering the

actual data inside the zero copy memory to configure the corresponding size, data type and data layout

for output. When in the NATIVE_LAYOUT mode, it is recommended utilizing the default configuration

that is from the one queried from the RKNN_QUERY_NATIVE_INPUT_ATTR. The specification of

output is shown below.

Table-4: NPU supports output setting. (With zero copy interface)

Output Data Type	Available Output Layout
float32	NCHW
int8	NCHW
int8	NC1HWC2
float16	NCHW
float16	NC1HWC2

4.4.4.3 Instruction of looking up NATIVE_LAYOUT parameter

There are two parameters listed below.

RKNN_QUERY_NATIVE_INPUT_ATTR

RKNN_QUERY_NATIVE_OUTPUT_ATTR

Those two can query the attribute of input and output tensor. In most of the time, this native attribute

has the best performance. When querying this attribute, the rknn_tensor_format might be the NC1HWC2,

which is a special memory arrangement. The memory address from low to high is C2->W->H->C1->N,

among which the C2 has the fastest changing rate and N is the slowest one. The value of C2 may vary

depending on the platform and data type. The specification of C2 is shown in the table below.

Table-5: The value of C2 under different platform

	int8(quantization)	float16(non-quantization)
RK356X	8	4
RK3588	16	8

For example, assuming on the RK356X, there is a output tensor whose memory layout is 1x1x1x32

(NHWC) from the orignal model. Then, the original model is converted to RKNN model. When

querying the output tensor from this new model with the native attribute, the memory layout of this output tensor will be converted to 1x4x1x1x8 (NC1HWC2), where C1 = [32/8] =4, C2 = 8, the [] denotes

the rounding up to cloest integer.

When invoking the rknn_query, this API will return the configuration with the best performance to

the rknn_tensor_attr. If the user has got different input or output data configuration differed from

what they desire, they can customize the configuration of the query part. However, it must be

consistent with the table 3 and table 4 shown above. For example, if the queried data type is uint8

and users want to configure this output data type as float32, users should multiply the size in

rknn_tensor_attr structure by 4 since the float32 is 4 bytes long. Also, the data type in

rknn_tensor_attr should be changed to RKNN_TENSOR_FLOAT32. With regrading to querying in

the native mode using the above two parameters, users should use the zero copy interface to fetch the

result of input and output.

4.4.5

RKNN Error Code

The return code of the RKNN API function is defined as shown in the following table.

Error Code	Message
RKNN SUCC（0） _	Execution is successful
RKNN ERR FAIL (-1) _ _	Execution error
RKNN ERR TIMEOUT (-2) _ _	Execution timeout
RKNN ERR DEVICE UNAVAILABLE (-3) _ _ _	NPU device is unavailable
RKNN ERR MALLOC FAIL (-4) _ _ _	Memory allocation is failed
RKNN ERR PARAM INVALID (-5) _ _ _	Parameter error
RKNN ERR MODEL INVALID (-6) _ _ _	RKNN model is invalid
RKNN ERR CTX INVALID (-7) _ _ _	rknn context is invalid _
RKNN ERR INPUT INVALID (-8) _ _ _	rknn input object is invalid _
RKNN ERR OUTPUT INVALID (-9) _ _ _	rknn output object is invalid _
RKNN ERR DEVICE UNMATCH (-10) _ _ _	Version does not match
RKNN ERR INCOMPATILE OPTIMIZATION LEVEL VERSION (-12) _ _ _ _ _	This RKNN model use optimization level mode, but not compatible with current driver.
RKNN ERR TARGET PLATFORM UNMATCH (-13) _ _ _ _	This RKNN model doesn’t compatible with current platform.

4.5 NPU SDK Instruction

4.5.1

RKNN Error Code

The NPU SDK for both RK356X and RK3588 contains examples for the API usage, including the

NPU library, server programs, doc. The server program is called rknn_server, which resides on the

development board. With the communication set up by the server program, The developer should connect

the development board with the PC via the USB and use the rknn-toolkit2 API with python interface to

run the model. The detail instruction of how to install is written in rknn_server_proxy.md.

The overall structure of the content is shown below, ├──doc ├──examples # Example Program │├──3rdparty │├──rknn_api_demo # Zero Copy API Demo │├──rknn_mobilenet_demo │├──rknn_multiple_input_demo │├──rknn_ssd_demo │└──rknn_yolov5_demo ├──README.md ├──rknn_server_proxy.md # The instruction of server program (used for │ inference with board) └──runtime ├──RK356X # RK356X library and content of server program │ on the board side │├──Android ││├──librknn_api ││└──rknn_server │└──Linux │ ├──librknn_api │ └──rknn_server └──RK3588 # RK3588 library and content of server program │ on the board side ├──Android │├──librknn_api │└──rknn_server └──Linux ├──librknn_api └──rknn_server

4.5.2

Compiling and Building

Despite offering the same API interface, there are some differences on these platforms. The table

below shows the difference. (Only for the API varied on those two platform)

Table-6: API supporting condition on RK356X and RK3588

API/ Platform	RK356X	RK3588
rknn dup context _ _	√	×
rknn set core mask _ _ _	×	√

4.6 Debugging

4.6.1 Log level

The library for NPU will rely on the environment variable on the board to generate a set of

debugging logs or files that are suitable for developers to debug the program. The command for setting

up debugging log is shown below,

export RKNN_LOG_LEVEL=<level_number>

, where <level_number> represents the category of debugging log, ranging from 0 to 5. The detail of

each type of log is listed below.

Table-7: The output log of each category

Level	Output Log
0	Printing error message only
1	Printing both error and warning log
2	Printing both hints and log in level 1
3	Printing both debugging log and log in level 2
4	Printing both log in level 3 and information in each layer, influencing the performance of rknn run when enabling this log _
5	Printing both message in level 4 and data in intermediate layer, influencing the performance of rknn run when enabling this log _

As an example, for querying the performance of each layer, the developer can set the following

environment variable.

export RKNN_LOG_LEVEL=4

When completing the debugging, to reset the debugging log to level 0, using the following

commands,

unset RKNN_LOG_LEVEL

4.6.2 Profiling

4.5.2.1 Checking the environment on the development board

Generally, the frequency of each hardware unit on the development board is not fixed, resulting in

the fluctuation and uncertainty on the performance of module running on the board. To avoid this, the

developer should fix the frequency of each hardware unit before testing modules for assessment.

1. Fix Frequency for CPU

(1) Query the frequency of CPU

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq

or

cat /sys/kernel/debug/clk/clk_summary | grep arm

(2) Fix the frequency of CPU

# Query avilable frequency of CPU

cat /sys/devices/system/cpu/cpufreq/policy0/scaling_available_frequencies

408000 600000 816000 1008000 1200000 1416000 1608000 1704000 # Fix the CPU frequency, e.g., fix at 1.7GHz

echo userspace > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor
echo 1704000 > /sys/devices/system/cpu/cpufreq/policy0/scaling_setspeed

2. DDR related commands

(1) Query the frequency of DDR

cat /sys/class/devfreq/dmc/cur_freq

or

cat /sys/kernel/debug/clk/clk_summary | grep ddr

(2) Fix the frequency of DDR (Need extra firmware)

# Query the avilable frequency of DDR

cat /sys/class/devfreq/dmc/available_frequencies

# Fix the frequency of DDR, e.g., at 1560MHz

echo userspace > /sys/class/devfreq/dmc/governor
echo 1560000000 > /sys/class/devfreq/dmc/userspace/set_freq

3. NPU related commands (1) Query the frequency of NPU:

For RK356X:

cat /sys/kernel/debug/clk/clk_summary | grep npu

or

cat /sys/class/devfreq/fde40000.npu/cur_freq

For RK3588 (Need extra firmware):

cat /sys/kernel/debug/clk/clk_summary | grep clk_npu_dsu0

(2) Fix the frequency of NPU：

For RK356X:

# Query the available frequency of NPU

cat /sys/class/devfreq/fde40000.npu/available_frequencies

# Fix NPU frequency，e.g., fix at 1 GHz

echo userspace > /sys/class/devfreq/fde40000.npu/governor
echo 1000000000 > /sys/kernel/debug/clk/clk_scmi_npu/clk_rate

For RK3588 (Need extra firmware) :

# Fix the NPU frequency, e.g., at 1GHz

echo 1000000000 > /sys/kernel/debug/clk/clk_npu_dsu0/clk_rate