Rspress

# #Environment Deployment

## #Foreword

Horizon OpenExplorer currently provides 2 sets of model quantization schemes at the same time.

- PTQ: Post-training Quantization.
- QAT: Quantization Awareness Training, which only supports the Pytorch framework for now.

Among them:

- Both solutions do not interfere with the training phase of the floating-point model, which is your own responsibility. Horizon has also provided some open-source implementations of the public version of Pytorch for efficient models in classification, detection, segmentation, and other scenarios in the `samples/ai_toolchain/horizon_model_train_sample` for reference, with the support for training and replication on the host.

- For PTQ scheme, you need to quantize the model in the host development environment, and then copy the compiled `.hbm` model to the dev board environment for subsequent deployment.

- For the QAT scheme, you need to complete the QAT training of the model in the host development environment, perform the quantization conversion, and then copy the compiled `.hbm` model to the dev board environment for subsequent deployment.

For both of the above quantization schemes and the development environment of the efficient model, Horizon provides both local manual installation and Docker containers. We strongly recommend using Docker containers as they do not pollute the local environment and is easy to use, descriptions of these two are included in the following sections.

## #Development Environment Deployment

### #Development Machine Preparation

In order to use the toolchain smoothly, we recommends that the development machine you choose should meet the following requirements:

HW/OS | REQUIREMENTS
CPU | CPU above I3 or same level processor as E3/E5
Memory Size | 16G or above
GPU | CUDA12.6, Drive Version: Linux: >= 550.163.01 Adapted graphics cards include but are not limited to: 1)GeForce RTX 3090 2)GeForce RTX 2080 Ti 3)NVIDIA TITAN V 4)Tesla V100S-PCIE-32GB 5)A100
OS | Native Ubuntu 22.04

For more information about CUDA compatibility with graphics cards, refer to NVIDIA website information.

### #Docker Container Deployment

#### #Docker Base Environment

Horizon requires the following Docker base environment, please complete the installation on your host computer in advance.

- Docker (20.10.10 and above, 20.10.10 is recommended), refer to the Docker Installation Manual.
- NVIDIA Container Toolkit (1.16.2 and above, 1.17.8 is recommended), refer to installation manual NVIDIA Container Toolkit installation manual.

After completing the installation of the Docker environment, remember to add non-root users into Docker users group by running below command:

`sudo groupadd docker sudo gpasswd -a ${USER} docker sudo service docker restart`

#### #Docker Image Usage

To help you quickly use the toolchain, we provide a Docker image containing the complete development environment, which greatly simplifies the deployment process of the development environment.

Hint

If you have downloaded the offline image, you need to use the following command to load the image locally first.

`docker load -i ai_toolchain_ubuntu_22_s100_s600_xxx.tar.gz`

You can start the Docker container corresponding to the current OE version by running the following script directly from the first level of the OE package:

`bash run_docker.sh data/`

Where `data` is the path to the evaluation dataset folder. Please create the path before running the command, otherwise loading problems may occur.

If you want to use the CPU version of the Docker image, you need to add the `cpu` parameter:

`bash run_docker.sh data/ cpu`

The download links for the relevant publicly available evaluation datasets that the OE package samples rely on can be accessed by referring to the introduction in section Dataset Download.

If you want to start the Docker container manually, you can refer to the following command, where `{version}` is the OE version number you are currently using.

Note

For your convenience, we provide CPU Docker and GPU Docker for you to choose on demand.

Attention

1. Since the environment variables `PATH` and `LD_LIBRARY_PATH` are configured during the build process of the OE Docker image, not using the recommended way (e.g., docker attach) to enter the container may result in the environment variables not being loaded correctly, which may lead to the use of abnormalities in tools such as Cmake, GCC, CUDA, and so on.

2. If you want the Docker container to exit without removing it, use the command line `docker run -it` to start it manually, without the `--rm` option.

3. If you want the Docker container to run background after startup, add the `-d` option after the command line `docker run -it`, the container ID will be returned after the container is started, and then you can enter the container again with the command `docker exec -it {container ID} /bin/bash`.

### #Local Manual Installation

This section describes the local manual installation environment method, and introduces the environment-related dependencies and descriptions for each of the two quantization schemes and Horizon open-source efficient model training. We recommend prioritizing the easy-to-use PTQ quantization scheme after the floating-point model training, and switch to the QAT quantization scheme only when accuracy issues cannot be solved.

#### #Local Manual Installation Environment Method

For local manual environment setup, you can enter the package/host directory and install the required files.

#### #PTQ Quantization Environment Dependence

The PTQ scheme has the following software dependencies on the base software of the development machine operating environment:

- Operating system: Ubuntu22.04
- Python3.10
- libpython3.10
- python3-devel
- python3-pip
- gcc&g++: 12.2.1
- graphviz

#### #QAT Quantization Environment Dependence

The QAT quantization environment is installed in the local environment and you need to ensure that the following basic environmental conditions are met.

The environmental dependencies required for the quantitative training tool to be trained are listed below:

HW/OS | GPU | CPU
--- | --- | ---
os | Ubuntu22.04 | Ubuntu22.04
cuda | 12.6 | N/A
python | 3.10 | 3.10
torch | 2.6.0+cu126 | 2.6.0+cpu
torchvision | 0.21.0+cu126 | 0.21.0+cpu
Recommended Graphics Cards | titan v/2080ti/v100/3090 | N/A

After completing the training of the QAT model, you can install the relevant toolkits in the current training environment and complete the subsequent model conversion directly through the interface call.

## #Runtime Environment Deployment

Once the model has been quantized, the compiled model can be deployed on the dev board environment for inference and execution.

To deploy the runtime environment, you need to prepare a dev board with the system image programmed, and then copy the relevant supplementary files to the dev board.

### #Dev Board Preparation

At this stage, you need to verify the usability of the dev board and program the available system images to the board.

### #Board Side Tool Installation

Some of the supplementary tools of the toolchain are not included in the system image, but can be copied to the dev board by running the installation script in the OE package in the host environment, as follows:

`cd package/board/hrt_model_exec bash build_aarch64.sh`
Note

After successfully compiling hrt_model_exec, please copy the generated output_shared_runtime_aarch64 to a writable path on the development board, then run the scripts in the script directory to use this tool.

On This Page