Rspress

# #Model Inference

The `hrt_model_exec infer` tool can be used for model inference, where the input data are defined by user and one frame is inferred.

The program runs a single frame of data in a single thread and outputs the time of the model execution.

## #Support Range

The support range of the model: QAT(Quantized Awareness Training) model, PTQ (Post-Training Quantization) model.

## #Usage

`Usage: hrt_model_exec [Option...] [Parameter] [Option] [instruction] --------------------------------------------------------------------------------------------------------------- -h --help Display this information -v --version Display this version [Option] [Parameter] --------------------------------------------------------------------------------------------------------------- --model_file [string]: Model file paths, separate by comma, each represents one model file path. --model_name [string]: Model name. When model_file has one more model and Subcommand is infer or perf, "model_name" must be specified! --input_file [string]: Input file paths, separate by comma, each represents one input. The extension of files should be one of [jpg, JPG, jpeg, JPEG, png, PNG, bin, txt, npy] bin for binary such as image data, nv12 or yuv444 etc. txt for plain data such as image info. --core_id [string]: core id, 0 for any core, 1 for core 0, 2 for core 1 and etc, default is 0. Please confirm the number of bpu cores on the board before setting up. When you need to specify multiple cores, separate them with commas, such as "1,2". --frame_count [int] : frame count for run loop, default 200, valid when perf_time is 0 in perf mode; default 1 for infer mode. --dump_intermediate [string]: dump intermediate layer input and output. The default is 0. Subcommand must be infer. --enable_dump [bool] : flag for dump infer input and output. The default is false. Subcommand must be infer. --dump_precision [int] : Output dump precision for float32/float64 in txt file. Default is 9 decimal places. --dequantize_process [bool] : dequantize the model infer output. The default is false. Subcommand must be infer, enable_dump set as true --remove_padding_process [bool] : remove padding of the model infer output. The default is false. Subcommand must be infer, enable_dump set as true --dump_format [string]: output dump format, only support [bin, txt, npy]. The default is bin. When dumping data in .npy format, the tool will always remove padding; the remove_padding_process parameter has no effect. Subcommand must be infer. --dump_txt_axis [int] : The txt file of dump is expanded according to the specified axis; the default is -1, which means there is only one data per line; Subcommand must be infer, dump_format must be txt. range:[0, tensor_dimension]. --enable_cls_post_process [bool] : flag for classfication post process, only for ptq model now. Subcommand must be infer. --dump_path [string]: dump file path, --enable_dump or --dump_intermediate will dump model nodes inputs and outputs files. --input_img_properties [string]: Specify the color space of the image type input. Each image needs to specify the color space, separated by commas. The supported color spaces are [Y, UV]. --input_valid_shape [string]: Complete the validshape of the model input, allowing only the dynamic part to change. Provide two ways to set: 1. This only needs to be set when the validShape of the model input is dynamic. 2. Set for all inputs. Different inputs are separated by semicolons, and different dimensions are separated by commas. For example: --input_valid_shape="1,376,376,1;1,188,188,2". --input_stride [string]: Complete the stride of the model input, allowing only the dynamic part to change. Provide two ways to set: 1. This only needs to be set when the stride of model input is dynamic. 2. Set for all inputs. Different inputs are separated by semicolons, and different dimensions are separated by commas. For example: --input_stride="144384,384,1,1;72192,384,2,1". [Examples] --------------------------------------------------------------------------------------------------------------- hrt_model_exec infer --model_file --model_name --core_id --input_file --input_img_properties --input_valid_shape --input_stride --frame_count --dump_intermediate --enable_dump --dump_precision --dequantize_process --dump_path --remove_padding_process --dump_format --dump_txt_axis --enable_cls_post_process`

## #Parameters Introduction

Parameter | Data type | Parameter Description | Correlated Parameters
--- | --- | --- | ---
`-h, --help` | None. | Display the help information. | None.
`-v, --version` | None. | View the version number of the DNN prediction library of the tool. | None.
`infer` | None. | Perform model inference and get model inference results. | This parameter should be used together with `input_file` to specify the input image path, and the tool resizes the image according to the model information and organizes the model input information.
`model_file` | string | Model file path, multiple paths can be separated by commas. | None.
`model_name` | string | Specify the name of a model. | None.
`input_file` | string | Model input information. The input of the image type, it must have one of the following file name suffixes: `PNG`/`JPG`/`JPEG`/`png`/`jpg`/`jpeg`/`bin`/`txt`/`npy`. The inputs should be separated by commas `,`, such as `xxx.jpg,input.txt`. | This parameter needs to be used in conjunction with the subcommand `infer`to specify the input image path, and the tool resizes the image according to the model information and organizes the model input information.
`core_id` | string | Specify the running core, `0` means arbitrary core, `1` means core0, `2` means core1 and etc. Default value is `0`. When you need to specify multiple cores, separate them with commas, such as `"1,2"`. | None.
`input_img_properties` | string | The color space information of the model image input, range [`Y`, `UV`]. | This parameter should be used together with `input_file`, each image type input in `input_file` needs to specify a `Y`/`UV` type, and each input color space needs to be separated by an English character comma `,`, such as: `Y,UV`.
`input_valid_shape` | string | Model dynamic `validShape` input information. If the model input attribute `validShape` contains `-1`, the `-1` part needs to be completed, and multiple `validShape` are separated by English semicolons. For example: `--input_valid_shape="1,376,376,1;1,188,188,2"`. | None.
`input_stride` | string | Model dynamic `stride` input information. If the model input attribute `stride` contains `-1`, the `-1` part needs to be completed, and multiple `stride`s are separated by English semicolons. For example:`--input_stride="50176,224,1,1;25088,224,2,1"`. | None.
`frame_count` | int | The number of running frames of the execution model. When the subcommand is `infer`, defaults to `1`. When the subcommand is `perf`, defaults to `200`. | When the subcommand is `perf`, it takes effect when `perf_time` is not set.
`dump_intermediate` | string | dump model each layer of input and output, range [0, 3]. Default value is `0`. When `dump_intermediate=0`, the dump function is turned off by default. When `dump_intermediate=1`, the input and output data of each node layer in the model are saved as `bin`, where inputs and outputs of node are `stride` data. When `dump_intermediate=2`, the input and output data of each node layer in the model are saved as `bin` and `txt`, where the inputs and outputs of node are `stride` data. When `dump_intermediate=3`, the input and output data of each node layer in the model are saved as `bin` and `txt`, where the inputs and outputs of node are `valid` data. | None.
`enable_dump` | bool | Enables dump model input and output. Default value is `false`. | None.
`dump_precision` | int | Controls the number of decimal places of the float type data output in txt format. Default value is `9`. | None.
`dequantize_process` | bool | Inverse quantization of model output. Default value is `false`. | Effective when the `enable_dump` parameter is `true`.
`remove_padding_process` | bool | Remove padding of model output. Default value is `false`. | Effective when the `enable_dump` parameter is `true`.
`dump_format` | string | The format of the dump model input and output, range [`bin`, `txt`, `npy`]. Default value is `bin`. | When dumping data in `.npy` format, the tool will always remove padding; the `remove_padding_process` parameter has no effect.
`dump_txt_axis` | int | Control line feed rules for txt format input and output. If output dimension = n, then parameter range: [0, n]. Defaults to -1, which means one data per row. | None.
`enable_cls_post_process` | bool | Enables classification post-processing. Default value is `false`. | This parameter needs to be used in conjunction with the subcommand `infer`. Currently, it only supports post-processing of the ptq classification model and printing of classification results.
`dump_path` | string | The path of dump model input and output. | Effective when the `enable_dump` or `dump_intermediate` parameter is set.

## #Usage Example

### #Model Inference

#### #Single Model Inference

`hrt_model_exec infer --model_file=xxx.hbm --input_file=xxx.bin ../aarch64/bin/hrt_model_exec infer --model_file=resnet50_224x224_nv12.hbm --input_file=zebra_cls.jpeg,zebra_cls.jpeg --input_img_properties=Y,UV --input_stride=50176,224,1,1;25088,224,2,1 Load model to DDR cost 1965.03ms. [I][35143][06-28][10:39:51:373][file_util.cpp:527][hrt_model_exec][HRT_MODEL_EXEC] The input valid shape is (1,224,224,1), and the image [zebra_cls.jpeg] will be scaled to 224x224 [I][35143][06-28][10:39:51:377][file_util.cpp:527][hrt_model_exec][HRT_MODEL_EXEC] The input valid shape is (1,112,112,2), and the image [zebra_cls.jpeg] will be scaled to 224x224 ---------------------Frame 0 begin--------------------- Infer time: 1.464 ms ---------------------Frame 0 end---------------------`

#### #Multiple Models Inference

`hrt_model_exec infer --model_file=xxx.hbm,xxx.hbm --model_name=xx --input_file=xxx.jpg`

### #Model Input Instructions

#### #Multi-input Model Description

The tool `infer` supports inference for multiple input models, supporting image input, binary file input, text file input, and NumPy array file input, with input data separated by commas. The model input information can be viewed via `model_info`.

`hrt_model_exec infer --model_file=xxx.hbm --input_file=xxx.jpg,input.txt`

#### #Dynamic input instructions

If the model input is dynamic, you need to use the `input_valid_shape` and `input_stride` parameters to complete the dynamic information according to the actual input situation. You can choose to specify the parameters in the following two ways:

- Only give the `validShape` or `stride` information of the dynamic input.

- Give the `validShape` or `stride` information of all inputs. The information of non-dynamic inputs must be consistent with the model information.

Note

The tool will automatically complete the dynamic input information to the maximum extent, which is convenient for you to perform performance evaluation more easily. You can choose whether to let the tool automatically complete it according to the actual situation.

- If the input `stride` is dynamic and `validShape` is fixed, you can not specify the `input_stride` parameter. The tool will automatically complete it according to the minimum alignment rule and print the alignment information.

- If both `validShape` and `stride` are dynamic:

  - If the input is a picture type, you need to specify the color space information of the picture `input_img_properties`, and the other parameters can be left unspecified. The tool will fill in the `input_valid_shape` and `input_stride` information according to the size of the picture.

  - If the input is a type other than a picture or is not specified, you need to set the `input_valid_shape` information, and the `input_stride` information will be automatically filled in and printed.

Taking the model in the Dynamic Input Introduction section as an example, you can run the model with the following command:

`# Only given dynamically input information hrt_model_exec infer --model_file=xxx.hbm --input_file="input_y.bin,input_uv.bin,input_roi.bin" --input_valid_shape="1,220,220,1;1,110,110,2" --input_stride="49280,224,1,1;24640,224,2,1" # Given all the input information hrt_model_exec infer --model_file=xxx.hbm --input_file="input_y.bin,input_uv.bin,input_roi.bin" --input_valid_shape="1,220,220,1;1,110,110,2;1,4" --input_stride="49280,224,1,1;24640,224,2,1;16,4"`

#### #Image type input instructions

When `input_file` is given an image input, you need to use the `input_img_properties` parameter to specify which color space of the image you want to use as input to the model. Currently, only `Y` and `UV` color spaces are supported.

`hrt_model_exec infer --model_file=xxx.hbm --input_file="img.jpg,img.jpg,input_roi.bin" --input_img_properties="Y,UV"`
On This Page