Rspress

# #PTQ Conversion Principle and Process

## #Overview

The process of the model from training to conversion to running on the development board is shown below:

Model Training: It is the process of getting a usable model by using public deep learning frameworks such as TensorFlow, PyTorch, Caffe, etc. The trained model will serve as the input for the model conversion stage. The toolchain does not provide training-related libraries or tools. For the detailed supported public learning frameworks, please refer to the instructions in the Floating-point Model Preparation.

Model Conversion: Taking the floating-point model obtained from model training as input, this stage transforms floating-point models into models that can run efficiently on the Horizon computing platform through important steps such as model structure optimization, model calibration quantization and model compilation. For more information, please refer to Model Quantization and Compilation section.

Embedded Application development: The toolchain supports application development in both X86 simulation environment and real embedded environment. In case you are not convenient to use the development board, you can debug the program and verify the calculation results in the simulation environment. In order to reduce the cost of simulation verification, the toolchain provides the exact same simulation library interface as the embedded interface, only with different compilation configurations. For more information, please refer to Embedded Application Development section.

## #PTQ Conversion Process

The complete the model development process with the Horizon toolchain involves five important stages: Floating-point Model Preparation, Model Checking, Model Conversion, Performance Evaluation, and Accuracy Evaluation, as shown in the figure below.

The Floating-point model, as the output of the Floating-point Model Preparation stage, will serve as the input of the model conversion tool. The floating-point model is usually trained on basis of some open source deep learning frameworks. Note that the model must be exported to a format supported by Horizon Robotics. For more information, please refer to the Floating-point Model Preparation.

The Model Checking stage is used to ensure that the model is computing platform compliant. We provides specified tools to complete model validation, and for non-compliance, such tools will explicitly give you the specific operator information for the non-compliance, so that you can easily adjust the model with the description of the operator constraints. For more information, please refer to the Model Checking.

The Model Conversion stage converts the floating-point model to the board-side deployable model supported by Horizon Robotics. To run models efficiently on the Horizon computing platform, critical steps such as model optimization, quantization, and compilation are completed by Horizon's model conversion tools. Horizon's model quantization method has undergone long-term technological and production validation, and can guarantee an accuracy loss of about 1% on most typical deep learning models. For more details about model conversion please refer to the Data Preparation and Model Quantization and Compilation.

The Performance Evaluation stage contains a series of tools to evaluate the model performance. Before deploying your application, you can use these tools to verify that the model performance meets application requirements. For some cases where the performance is not as good as expected, you can optimize the models based on Horizon's model optimization advices. For more information, please refer to the Model Performance Analysis.

The Accuracy Evaluation stage contains a series of tools to evaluate the accuracy of the model. In most cases, Horizon's converted-models can maintain almost the same accuracy as the original floating-point model. Before application deployment, you can use these tools to verify that the accuracy of the model meets the expectations. For some cases where the accuracy is not as good as expected, you can optimize the models based on Horizon's model optimization advices. For more information about evaluation, please refer to the Model Accuracy Analysis.

Attention

- In general, you can get a qualified runtime model after model conversion. However, make sure that the performance and accuracy of the model are in line with the application requirements. Horizon Robotics strongly suggest you that each conversion shall be followed by the evaluation steps of performance and accuracy.
- The model conversion process will generate onnx models, which are intermediate products and only facilitate you to verify the accuracy of the model. Therefore, the compatibility between versions is not guaranteed. When using the evaluation script in the sample to evaluate the onnx model in a single image or on a test set, please use the onnx model generated by the current version of the tool.

## #Model Conversion Process Detailed Explanation

So how to convert the floating-point models trained by using the opensource ML frameworks (such as Caffe, TensorFlow, PyTorch, etc.) to the Horizon hardware supported fixed-point models?

In most cases, the threshold values and weights of the floating-point models obtained from either the opensource ML frameworks or trained by yourself are floating-point numbers (float32) and each number occupies 4 bytes.

However, by converting the floating-point numbers to lower-bit computational numbers, thus the computation operations in the embedded runtime can be dramatically reduced.

Therefore, it brings significant performance boost by converting the floating-point models to lower-bit computational models with no loss or very small loss.

Typically, model conversion can be divided into the following steps:

1. Check if there are unsupported OPs in the models to be converted.

2. Prepare 20~100 images for calibration use at the conversion stage.

3. Convert the floating-point models to fixed-point models using the floating-point conversion tools.

4. Evaluate the performance and accuracy of the converted models to ensure that there isn't huge difference in model accuracy before and after the conversion.

5. Run models in simulator/dev board to validate model performance and accuracy.

### #Model Checking

Note

If you need to do this process in the sample folder, you need to execute the `00_init.sh` script in the folder first to get the corresponding original model and dataset.

Before converting the floating-point models into the fixed-point models, we should check if there are Horizon hardwares unsupported OPs in the floating-point models using the `hb_compile` tool. If yes, the tool will report the unsupported OP(s). The usage of the `hb_compile` tool for checking the model can be found in Model Checking section.

Hint

More information about Horizon hardware supported OPs, refer to Toolchain Operator Support Constraint List.

### #Calibration Image Preparation

Note

If you need to do this process in the sample folder, you need to execute the `00_init.sh` script in the folder first to get the corresponding original model and dataset.

When converting the floating-point models, you need to prepare 20~100 images for calibration use at the calibration stage.

Input image formats may vary by input type and layout. In this stage, because both original (JPG, etc.) and the processed images are valid, you can either feed the calibration images used in the model training or feed your own processed images.

Note

The format of the calibration dataset should be `npy` .

We recommend you preprocessing the calibration images on your own, you need to complete the operations such as image channel (BGR/RGB), data layout (NHWC/NCHW), and image resizing/padding (Resize&Padding). The tool will then feed the images to the calibration stage after loading them as npy format.

Taking Resnet50 as an example, the required transformer operations are as follows:

`transformers = [ PILResizeTransformer(size=256), # Adjust the short side to 256 with bilinear interpolation method PILCenterCropTransformer(size=224), # Perform center cropping with 224*224 with PIL method HWC2CHWTransformer(), # Convert NHWC to NCHW ScaleTransformer(scale_value=1 / 255), # Scale pixel value ranges to [0,1] MeanTransformer(means=np.array([0.485, 0.456, 0.406])), # Normalize the data by the average of the data set ScaleTransformer( scale_value=np.array([1 / 0.229, 1 / 0.224, 1 / 0.225])) # Normalize the data by the standard deviation of the data set ]`

### #Model Conversion

When you confirm that the floating-point model can be successfully converted, you can then convert the floating-point model to a Horizon hardware supported fixed-point model by using the `hb_compile` tool.

This process requires you to pass a configuration file(*.yaml) containing conversion requirements. For specific configuration file settings and the insturctions of each parameter, refer to the descriptions in sections Specific Parameter Information and Configuration File Template .

When the model conversion process ends, it also prints the level of similarity between the floating-point model and fixed-point model to the log, you can therefore judge the similarity before and after conversion according to the `Quantized Cosine` field.

If the value of `Quantized Cosine` is very close to 1, so the performance of the fixed-point model should be very close to that of the floating-point model before the conversion.

Note

The `CosineSimilarity` in the log refers to the very first image in the calibration images, it cannot fully represent the model accuracy before and after the conversion.

After a successful model conversion, the model outputs, information files and the model static performance evaluation files for each phase will be generated in the generated folder (`model_output` by default), where the model output files will be used in subsequent phases.

Note

- You can use the `03_classification/03_resnet50/03_build.sh` script to experience the `hb_compile` tool doing the model quantized compilation.
- If you want to learn more about the model conversion workflow, please read Model Quantification and Compilation .

### #Single Image Inference

The accuracy of the fixed-point model generated by the floating-point conversion must be evaluated.

You should have good understanding of the input/output structures of the model. You should also be able to accurately preprocess the input images of the model, postprocess the model outputs, and write the model execution scripts on your own.

You can refer to the sample code in `04_inference.sh` of the sample in Horizon model convert sample package.

The code logic is as follows:

`import numpy as np # Load Horizon dependency library from horizon_tc_ui.hb_runtime import HBRuntime from postprocess import postprocess # Prepare the input for model running, here the `input.py` is the processed data data = np.load("input.npy") # Load model file sess = HBRuntime("model.bc") # Obtain the model input & output node information input_names = sess.input_names output_names = sess.output_names # Model inference, here we assume the model has only one input # The return value is a list that corresponds in order to the names specified by output_names output = sess.run(output_names, {input_names[0]: data}) # Postprocessing top_five_label_probs = postprocess(output)`

Once the script used, you can verify its own accuracy by inputting a single image.

For example, the input to this script is a picture of a zebra, preprocessing the image data from rgb to the data type configured by input_type_rt(for informaiton about intermediate types, refer to the Model Conversion Interpretation ). Then, infer the model by passing the above image data by using the `HB_HBIRRuntime` command, post-processing after inference, and finally print out 5 of its most likely types.

The output of the script is shown as follows with the most possible class being `label: 340`:

`I0108 18:11:47.398328 140427646048000 cls_inference.py:89] The input picture is classified to be: label 340: prob 0.97 label 292: prob 0.02 label 282: prob 0.00 label 83: prob 0.00`

`label` uses the ImageNet label classes, where the corresponding class of `340` is zebra, so the inference result is correct.

### #Model Accuracy Verification

It's insufficient to determine the model accuracy by single image inference, so you still need to use scripts to evalute the model accuracy after the conversion.

To do so, you need some coding work to enable the model to loop the image inference and compare the inference results with standard results to get model accuracy results.

In model accuracy evaluations, images must be pre-processed and the model data must be post-processed, so here we provide a Python script as a sample.

The logic of this script is the same as that of single image inference, yet it must run on the entire dataset.

The script can evaluate the model output results and generate evaluation results.

Because it takes a long time to run the script, you can set the number of threads to run the evaluation by specifying the `PARALLEL_PROCESS_NUM` environment variable.

After the execution of this script, you can get the accuracy of the converted fixed-point model from the output of the script.

Note

- Model accuracy may vary slightly due to the differences of operating systems and dependencies.
- Model accuracy may vary slightly by iteration.

### #[Reference] Supported Calibration Methods

We currently support the following calibration methods:

1. Default

`Default` is a strategy that automatically searches the calibrated quantization parameters to obtain a relatively good combination.

2. Mix

`Mix` is a search strategy that integrates multiple calibration methods, which automatically identifies quantization-sensitive nodes, selects the best from a group of calibration methods at node granularity, and finally build a hybrid calibration method absorbing the advantages of multiple calibration methods.

3. KL

`KL` learns from the Solution proposed by TensorRT, uses the KL entropy value to traverse the data distribution of each quantized layer, and determines threshold value by searching for the lowest KL entropy value.

As this method can cause more data saturation and smaller data quantization granularity, it more suitable than `max` for those neural network models with more concentrated data distribution.

4. Max

`Max` refers to a calibration method that automatically selects the max value in quantized layer as the threshold.

This method can cause oversized quantization granularity; however, it also causes less saturated points than the KL method, which makes it suitable for those neural network models with more discrete data distribution.

### #[Reference] OP List

For more information about the operators and corresponding constraints currently supported by Horizon Algorithm Toolchain, please refer to Toolchain Operator Support Constraint List.

On This Page