D-Robotics Official Website
D-Robotics Official Website
English

Simple Development

In this chapter, we will introduce the basic usage workflow of D-Robotics-LLM to help you get started quickly.

Qwen2.5 Model Version Selection

We provide both Base and Instruct versions of the Qwen2.5 model to meet your diverse development and application needs. The differences between them are as follows:

  • The Base version is a foundational text generation model suitable for subsequent model training tasks. Its model name does not contain the word Instruct.

  • The Instruct version is derived from the Base version through instruction fine-tuning and is better suited for conversational scenarios. Its model name does contain the word Instruct.

Here, we use the Qwen2.5-1.5B-Instruct model as an example to demonstrate its usage.

Preparing the Model and Deployment Package

  • Download and extract the provided deployment package: D-Robotics_LLM_{version}.tar.gz.

  • Download the provided model file: Qwen2.5_1.5B_Instruct_1024.hbm.

Note

For the download link of the .hbm model, please refer to the resolve_model.txt file located in the model folder of oellm_runtime.

After preparing both the deployment package and the model, integrate the model (*.hbm) with the oellm_runtime SDK from the deployment package. The reference directory structure is as follows:

llm └── D-Robotics_LLM_{version} └── oellm_runtime ├── model │ ├── resolve_model.txt │ ├── Qwen2.5_1.5B_1024.hbm │ ├── Qwen2.5_1.5B_Instruct_1024.hbm │ ├── Qwen2.5_7B_1024.hbm │ └── Qwen2.5_7B_Instruct_1024.hbm ├── config │ ├── Qwen2.5_1.5B_config │ ├── Qwen2.5_1.5B_Instruct_config │ ├── Qwen2.5_7B_config │ └── Qwen2.5_7B_Instruct_config ├── example │ ├── oellm_run │ ├── oellm_multichat │ └── oellm_ppl ├── include ├── lib └── set_performance_mode.sh

On-Device Setup

Create a working directory on the device using the following commands as a reference:

# Create working directory mkdir -p /home/root/llm cd /home/root/llm

Copy the integrated folder from your development machine to this on-device directory. Reference command:

scp -r llm/* root@{board_ip}:/home/root/llm

Finally, under the path /home/root/llm/D-Robotics_LLM_{version}/oellm_runtime, configure the LD_LIBRARY_PATH. Reference commands:

# Modify hardware register values to switch the device to performance mode sh set_performance_mode.sh # Set environment variable lib=/home/root/llm/D-Robotics_LLM_{version}/oellm_runtime/lib export LD_LIBRARY_PATH=${lib}:${LD_LIBRARY_PATH}

Running on Device

Reference command to run the model on the device:

cd ./example/oellm_run ./oellm_run --hbm_path ../../model/Qwen2.5_1.5B_Instruct_1024.hbm \ --tokenizer_dir ../../config/Qwen2.5_1.5B_Instruct_config/ \ --template_path ../../config/Qwen2.5_1.5B_Instruct_config/Qwen2.5_1.5B_Instruct.jinja \ --model_type 7

Program input parameters are as follows:

ParameterDescriptionOptional/Required
-h, --helpDisplay help information./
--hbm_pathSpecifies the path to the quantized model file (*.hbm).Required
--tokenizer_dirSpecifies the tokenizer configuration directory.Required
--template_pathSpecifies the conversation template path for Instruct models; optional when loading Base models.Optional
--model_typeSpecifies the model type to run; currently, the Qwen2.5 model type is 7.Required

Running Results

After successful execution, you can perform simple conversation tests, as shown below:

[User] <<< Briefly describe the development of artificial intelligence. [Assistant] >>> The development of Artificial Intelligence (AI) can be briefly described as follows: 1. From Theory to Implementation - AI research began as early as the 1920s, but significant progress started with Alan Turing's concept of machine intelligence in 1936. Turing proposed a computational model—the Turing Machine—and defined effective computation as equivalent to human understanding of results. - Subsequent researchers explored simulating brain-like neural networks, machine learning, and endowing machines with intelligence. - From the 1960s onward, AI entered a period of rapid growth, with mature concepts and technologies emerging, such as genetic algorithms, neural networks, and fuzzy logic. - In the 1990s, deep learning (deep neural networks) gained prominence, significantly advancing machine learning—especially in image and pattern recognition. - In recent years, AI has achieved rapid progress in speech recognition, natural language processing, computer vision, machine translation, and intelligent robotics. 2. From Theory to Practice - Between 1950 and 2010 (~60 years), AI theory and practice developed rapidly. - From 2010 to 2014, deep learning surged, marking a new era of accelerated AI development. By 2014, over 800,000 AI-related papers had been published, with citations reaching 500 million. - By 2017, deep learning achieved remarkable accuracy in image and speech recognition, though challenges like massive datasets and model complexity emerged. Research shifted toward addressing data overload and model complexity. 3. From Development to Maturity - In 2015, AlphaGo defeated world Go champion Lee Sedol in a landmark event that shattered previous assumptions about AI capabilities, signaling technological maturity. - By 2020, AI was widely applied across healthcare, smart manufacturing, smart cities, and other domains. Performance prefill: 1939.39 tokens/s decode: 24.43 tokens/s