Rspress

# #Basic Sample User Guide

The Basic Sample Package is designed to help you familiarize yourself with and learn the interfaces and various advanced functions related to model inference. The Basic Sample Package can also be used as a starting point for new projects to help you quickly build a basic framework and conduct further development and customization on this basis.

The Basic Sample Package provides samples in three aspects:

- Provides a quick start sample based on the resnet50 model. You can experience and develop applications based on this sample to lower the development threshold.

- Provides a sample of using the model inference interface. You can use this sample to better familiarize yourself with the use of various basic functions.

- Provides advanced sample of various special functions. You can choose the appropriate function for model inference according to the actual usage scenario.

## #Release Description

The Basic Sample package is located in the `samples/ucp_tutorial/dnn/basic_samples/` path of the release package and consists the following main contents:

RELEASE PACKAGE | DESCRIPTIONS
--- | ---
basic_samples | including sample codes and scripts.

Directory of the Basic Sample Package is shown as below:

`├── code # source code of samples │ ├── 00_quick_start # a quick-start single image inference sample using the resnet50 model │ │ ├── CMakeLists.txt │ │ ├── resnet_nv12 │ │ └── resnet_rgb │ ├── 01_api_tutorial # BPU SDK API sample code │ │ ├── CMakeLists.txt │ │ ├── mem │ │ ├── model │ │ ├── padding │ │ └── quanti │ ├── 02_advanced_samples # Advanced samples │ │ ├── CMakeLists.txt │ │ ├── crop │ │ ├── multi_model_batch │ │ └── roi_infer │ ├── build.sh # compilation script │ ├── build_aarch64.sh # compilation script, running in aarch64 linux environment │ ├── build_x86.sh # compilation script, running in x86 environment │ ├── CMakeLists.txt │ └── resolve.sh # run dependency acquisition script ├── runtime │ ├── data # preset data │ │ └── cls_data │ ├── model │ │ ├── README.md │ │ └── runtime -> ../../../../../model_zoo/runtime/nash-e/basic_samples # soft link, pointed to the model folder in OE package. In the board-side runtime environment, you need to specify the model path │ ├── script # aarch64 scripts to run samples │ │ ├── 00_quick_start │ │ ├── 01_api_tutorial │ │ ├── 02_advanced_samples │ │ ├── aarch64 # aarch64 executable programs generated by the compilation and dependencies │ │ └── README.md │ └── script_x86 # x86 scripts to run samples │ ├── 00_quick_start │ ├── 01_api_tutorial │ ├── 02_advanced_samples │ ├── x86 # x86 executable programs generated by the compilation and dependencies │ └── README.md └── README.md`

- The code directory contains source code.

- code/00_quick_start: A quick start sample, based on the model inference interface, using resnet50 to perform single image model inference and result analysis.

- code/01_api_tutorial: Samples of using the model inference interface, including adding/removing padding sample `padding`, quantization and dequantization samples `quanti`, memory usage samples `mem`, model loading and information acquisition samples `model`.

- code/02_advanced_samples: Advanced samples of model inference, including cropping an image as model input sample `crop`, the inference sample `roi_infer` of the model with roi input and the multi-model batch inference sample `multi_model_batch`.

- The code/build.sh is a application compilation script, and you need to specify the compilation parameters, called by build_aarch64.sh and build_x86.sh.

- The code/build_aarch64.sh is a quick application compilation script, and the compiled application runs in the aarch64 linux environment.

- The code/build_x86.sh is a quick application compilation script, and the compiled application runs in the x86 environment.

- The code/resolve.sh is the program running dependency acquisition script, which needs to be executed before the user compiles the model.

- The runtime folder contains scripts to run samples, preset data and related models.

## #Environment Configuration

### #Prepare the Dev Board

1. After getting the development board, upgrade the system image file to the version recommended by the sample package.

2. Make sure the local development machine and development board can be connected remotely.

### #Compilation

Take the following steps to perform the compilation:

1. Follow the instructions in the [Environment Deployment] (../../env_install) section to install the cross-compilation tool.

2. In the `ucp_tutorial/dnn/basic_samples/code` directory, there is a pre-configured build script `build.sh` .

3. After executing the compilation script, the executable programs and dependency files required to run the samples will be generated and copied to the runtime/script or runtime/script_x86 directory.

Note

By default, the cross-compilation tool specified in the `build.sh` script is located in the /usr/bin directory. You can manually modify the build.sh script if you want to change the location.

## #How to Use (Basic Samples)

Note

If you want to run the sample in an x86 environment, please use the script under runtime/script_x86.

Sample scripts are in the runtime/script folder, and the directory structure after compilation is shown as follows:

`script: ├── 00_quick_start │ ├── README.md │ ├── run_resnet_nv12.sh │ └── run_resnet_rgb.sh ├── 01_api_tutorial │ ├── README.md │ ├── model.sh │ ├── padding.sh │ ├── quanti.sh │ └── sys_mem.sh ├── 02_advanced_samples │ ├── README.md │ ├── run_crop.sh │ ├── run_multi_model_batch.sh │ └── roi_infer.sh ├── aarch64 # executable programs generated by the compilation and dependencies │ ├── bin │ │ ├── run_resnet_nv12 │ │ ├── model_example │ │ ├── padding_example │ │ ├── quanti_example │ │ ├── roi_infer │ │ ├── run_crop │ │ ├── run_multi_model_batch │ │ ├── run_resnet_rgb │ │ └── sys_mem_example │ └── lib │ ├── libdnn.so │ ├── libhb_arm_rpc.so │ ├── libhbrt4.so │ ├── libhbtl.so │ ├── libhbucp.so │ ├── libhlog.so │ ├── libhlog.so.1 │ ├── libhlog.so.1.15.1 │ ├── libhlog_wrapper.so │ ├── libopencv_world.so.3.4 │ └── libperfetto_sdk.so └── README.md`

Before running, deploy the runtime directory to the board end and execute the corresponding script.

Note

- To obtain the model publisher for the `basic_samples` sample package, you can execute the `resolve_runtime_sample.sh` in the `samples/ai_toolchain/model_zoo/runtime/${march}/basic_samples` dictionary of the OE package. The `${march}` in the path can be set to `nash-e` or `nash-p` , corresponding respectively to the `S100/S100P` and `S600` platforms. You can choose the appropriate one based on your needs.

- To obtain other dependencies of the `basic_samples` sample package, you can execute the `resolve.sh` script in the `samples/ucp_tutorial/dnn/basic_samples/code` directory of the OE package.

- The `model` folder contains the path of the model, the `runtime` folder is a soft link and the default link path is `../../../model_zoo/runtime/nash-e/basic_samples`. If you need to run a model that is not based on the `nash-e` architecture, you can manually change `nash-e` in the URL path to `nash-b` or `nash-p` .

### #quick_start

Under 00_quick_start directory, we provide a quick-start sample to introduce the use process of model inference. Taking the resnet50 as an example, it shows the process of model inference for nv12 input and rgb input respectively, including the complete process code from data preparation to model inference, to execution of post-processing, and finally to generating classification results.

The code is mainly divided into six parts:

- Load the model and get the model handle: the main interfaces involved are hbDNNInitializeFromFiles, hbDNNGetModelNameList and hbDNNGetModelHandle.

- Prepare input and output tensors, and complete the dynamic information of the input: the main interfaces involved are hbDNNGetInputCount, hbDNNGetOutputCount, hbDNNGetInputTensorProperties, hbDNNGetOutputTensorProperties, hbDNNGetInputName, hbDNNGetOutputName and hbUCPMallocCached.

- Prepare input data: For nv12 input, the image needs to be converted into color space, and for rgb input, pre-processing from U8 to S8 is required. The interface involved is hbUCPMemFlush.

- Model inference: The main interfaces involved are hbDNNInferV2, hbUCPSubmitTask and hbUCPWaitTaskDone.

- Output data post-processing: The resnet50 model is a classification model, and you can obtain classification results with high confidence through post-processing. Before processing the output, you need to call the hbUCPMemFlush interface for data synchronization.

- Release tasks, memory and models: The main interfaces involved are hbUCPReleaseTask, hbUCPFree and hbDNNRelease.

The directory structure of the script is as follows:

`00_quick_start/ ├── README.md ├── run_resnet_nv12.sh └── run_resnet_rgb.sh`

When using it, enter the 00_quick_start directory, and then directly execute `sh run_resnet_rgb.sh` and `sh run_resnet_nv12.sh`. After the script is successfully executed, it will print the names of the model input and output and the 5 classification results with the highest confidence. The classification result with the highest confidence number `340` indicates that it is a zebra.

### #api_tutorial

The API tutorial refers to the samples in the 01_api_tutorial folder. These samples help developers understand how to use the model inference APIs. This folder contains the following scripts:

`├── padding.sh ├── quanti.sh ├── model.sh └── sys_mem.sh`

#### #padding

This sample helps you become familiar with how to add padding to model input and how to remove padding from model output. The main data types involved are `validShape` and `stride` in hbDNNTensorProperties.

In order to speed up calculations, the model has alignment requirements for input data, see alignment rules for details. This sample is divided into two parts:

- For model input, when additional alignment operations are required for data, use the `add_padding` method provided in the sample to add padding.

- For model output, when the output contains padding, use the `remove_padding` method provided in the sample to remove padding to obtain valid data.

When using it, directly enter the 01_api_tutorial directory, and then directly execute `sh padding.sh`. After the script is successfully executed, it will print the original input and the input data after adding padding, the original output and the output data after removing padding.

#### #quanti

This sample mainly helps you get familiar with the quantization and dequantization of the model input and output. The data types involved are `quantiType`, `scale`, and `quantizeAxis` in hbDNNTensorProperties.

When the model input contains quantization information, the input floating point number needs to be quantized according to the specified rules, when the model output contains quantization information, the output needs to be dequantized before post-processing the data. For the processing method, please refer to hbDNNQuantiScale.

This sample is mainly divided into four parts:

- Quantization of input per tensor

- Quantization of input per axis

- Dequantization of output per tensor

- Dequantization of output per axis

When using it, just go to the 01_api_tutorial directory and execute `sh quanti.sh` directly. After the script is successfully executed, each part will print the original input/output and the quantized/dequantized results.

#### #model

This sample mainly helps you get familiar with the use of interfaces related to model loading and model information acquisition.

The interfaces involved mainly include hbDNNInitializeFromFiles, hbDNNGetModelNameList, hbDNNGetModelHandle, hbDNNGetInputCount, hbDNNGetOutputCount, hbDNNGetInputTensorProperties, hbDNNGetOutputTensorProperties and hbDNNRelease.

When using it, directly enter the 01_api_tutorial directory, and then execute `sh model.sh` directly. After the script is executed successfully, the basic information of the model will be printed out.

#### #sys_mem

This sample mainly helps you get familiar with the use of memory interfaces. The main interfaces involved are hbUCPMalloc, hbUCPMallocCached, hbUCPMemFlush and hbUCPFree.

For cacheable memory, the usage steps are as follows:

- Apply for cacheable memory.

- Write data to memory.

- Call the hbUCPMemFlush interface and use the `HB_SYS_MEM_CACHE_CLEAN` parameter to synchronize data to DDR.

- After model inference or operator calculation, call the hbUCPMemFlush interface and use the `HB_SYS_MEM_CACHE_INVALIDATE` parameter to synchronize data to the cache.

- Read data for processing.

- Release memory.

When using it, directly enter the 01_api_tutorial directory and execute `sh sys_mem.sh`.

### #advanced_samples

This sample refers to the samples in the 02_advanced_samples directory, which is intended to introduce the use of advanced samples. Its directory contains the following scripts:

`├── run_crop.sh ├── run_multi_model_batch.sh └── roi_infer.sh`

#### #crop

This sample mainly helps you get familiar with how to crop images and use them as model input for inference. The overall process of the sample code is the same as 00_quick_start, and the difference lies mainly in the preparation of input data.

The principle of cropping in this sample is to offset the memory address of the existing image to the upper left corner of the ROI, and mask the redundant part of the image by controlling the size of the stride to prepare the input of the model.

Limitations of sample use:

- Image: The image resolution is required to be large, at least larger than the input of the model, and the image is read into the BPU memory in advance.

- Model: The input validShape of the model is required to be fixed and the stride is dynamic, so that the image can be cropped by controlling the size of the stride.

- Cropping position: Since cropping is to offset the image memory, and the first address of the input memory requires `32` alignment, there is a limit on the size of the offset.

Preparation of input data:

- Image: The size of the zebra image is 376x376. We read the image and convert it into Y and UV input and store them in two BPU memories after `32` alignment. The width of the zebra image after `32` alignment is `image_width_stride = 384`.

- Model: The model has two inputs

  - y: validShape = (1,224,224,1), stride = (-1,-1, 1,1)

  - uv: validShape = (1,112,112,2), stride = (-1,-1,2,1)

- Cropping: We need to crop a 224x224 area in the image as the input of the model. Selecting (64,50) as the upper left corner point can just get most of the zebra image, and the offset memory address is exactly `32` aligned, which meets the requirements. As shown in the figure below.

- Input tensor preparation:

  - The stride of the Y input is [224 * image_width_stride, image_width_stride, 1, 1], and the coordinates of the upper left corner of the cropped image are [0, 50, 64, 0], then the address offset is `50 * image_width_stride + 64 * 1`.

  - The stride of the UV input is [112 * image_width_stride, image_width_stride, 2, 1], and the coordinates of the upper left corner of the cropped image are [0, 25, 32, 0], then the address offset is `25 * image_width_stride + 32 * 2`.

When using, enter directly 02_advanced_samples directory, and then directly execute `sh run_crop.sh`. After the script is successfully executed, the five classification results with the highest confidence will be printed. The classification result with the highest confidence number `340` indicates that it is a zebra.

#### #multi_model_batch

This sample mainly helps you get familiar with the function of batch processing of small models.

When there are multiple small models running, if each model task runs separately, the scheduling time of the entire framework will be relatively large. In order to avoid performance loss, you can combine multiple small model tasks into one task for inference to reduce the proportion of framework scheduling time.

This sample takes two small models `googlenet` and `resnet50` as an example, and creates/adds tasks by calling the hbDNNInferV2 interface multiple times.

When using it, directly enter the 02_advanced_samples directory and execute `sh run_multi_model_batch.sh`. After the script is successfully executed, the classification results with the highest confidence of the two models will be printed. The classification number `340` indicates that it is a zebra.

#### #roi_infer

This sample mainly helps you understand how to prepare data for model inference when there is a roi input.

The overall process of the sample code is the same as 00_quick_start, and the difference lies mainly in the preparation of input data. The sample model is `mobilenetv1`, and the input is 3 in total, as shown below:

- name: data_y; validShape: (1,-1,-1,1); stride: (-1,-1,1,1);

- name: data_uv; validShape: (1,-1,-1,2); stride: (-1,-1,2,1);

- name: data_roi; validShape: (1,4); stride: (16,4);

`data_y` and `data_uv` are dynamic inputs. They are read from the image and converted into Y and UV data for alignment as input, and `validShape` and `stride` are completed according to the size of the image. For dynamic input, please refer to Introduction to Dynamic Input.

`data_roi` is the coordinates of the upper left and lower right corners of the ROI area in the image, in the order of left, upper, right, and lower.

When using it, directly enter the 01_api_tutorial directory, and then execute `sh roi_infer.sh` directly. After the script is successfully executed, it will print the five classification results with the highest confidence for the two sets of inputs of the model. The classification result with the highest confidence number `340` indicates that it is a zebra.

## #Helper Tools (Log)

There are 2 types of logs: sample log and dnn log. Wherein, sample log refers to the log in the Basic Sample Package release package, while dnn log refers to the log in the embedded dnn library. Developers can specify logs according to their own needs.

### #Sample Log

The sample log mainly uses hlog, which is mainly divided into 7 levels:

The log level can be set to 0, 1, 2, 3, 4, 5 and 6, corresponding to trace, debug, info, warn, error, critical and never, with the default being info.

### #dnn Log

For the configuration of `dnn` logs, please read the Configuration Info section in the Model Inference API Introduction.

On This Page