D-Robotics Official Website
D-Robotics Official Website
English

Simple Development

In this chapter, we will introduce the basic usage workflow of D-Robotics-LLM to help you get started quickly. Here, we take the DeepSeek-R1-Distill-Qwen-1.5B model as an example to demonstrate its usage.

Model and Deployment Package Preparation

  • Download the provided D-Robotics_LLM_{version}.tar.gz deployment package and extract it.

  • Download the provided DeepSeek_R1_Distill_Qwen_1.5B_1024.hbm model.

Note

For the download link of the hbm model, please refer to the resolve_model.txt file in the model folder of oellm_runtime.

After preparing both the deployment package and the model, integrate the model (*.hbm) with the oellm_runtime SDK from the deployment package. The reference directory structure is as follows:

llm └── D-Robotics_LLM_{version} └── oellm_runtime ├── model │ ├── resolve_model.txt │ ├── DeepSeek_R1_Distill_Qwen_1.5B_1024.hbm │ ├── DeepSeek_R1_Distill_Qwen_1.5B_1024_q4.hbm │ ├── DeepSeek_R1_Distill_Qwen_1.5B_4096.hbm │ ├── DeepSeek_R1_Distill_Qwen_1.5B_4096_q4.hbm │ └── DeepSeek_R1_Distill_Qwen_7B_1024.hbm ├── config │ ├── DeepSeek_R1_Distill_Qwen_1.5B_config │ └── DeepSeek_R1_Distill_Qwen_7B_config ├── example │ ├── oellm_run │ ├── oellm_multichat │ └── oellm_ppl ├── include ├── lib └── set_performance_mode.sh

On-Device Runtime Preparation

Create a working directory on the device using the following commands as a reference:

# Create working directory mkdir -p /home/root/llm cd /home/root/llm

Copy the integrated folder from your development machine to this device directory. Reference command:

scp -r llm/* root@{board_ip}:/home/root/llm

Finally, configure LD_LIBRARY_PATH under the path /home/root/llm/D-Robotics_LLM_{version}/oellm_runtime. Reference commands:

# Modify hardware register values to switch the device to performance mode sh set_performance_mode.sh # Set environment variables lib=/home/root/llm/D-Robotics_LLM_{version}/oellm_runtime/lib export LD_LIBRARY_PATH=${lib}:${LD_LIBRARY_PATH}

On-Device Execution

Reference command for running the model on the device:

cd ./example/oellm_run ./oellm_run --hbm_path ../../model/DeepSeek_R1_Distill_Qwen_1.5B_4096.hbm \ --tokenizer_dir ../../config/DeepSeek_R1_Distill_Qwen_1.5B_config/ \ --template_path ../../config/DeepSeek_R1_Distill_Qwen_1.5B_config/DeepSeek_R1_Distill_Qwen_1.5B.jinja \ --model_type 1

The runtime parameters are as follows:

ParameterDescriptionOptional/Required
-h, --helpDisplay help information./
--hbm_pathSpecifies the path to the quantized model file (*.hbm).Required
--tokenizer_dirSpecifies the tokenizer configuration path.Required
--template_pathSpecifies the conversation template path.Required
--model_typeSpecifies the model type to run; currently, the DeepSeek model type is 1.Required

Execution Results

A simple dialogue test example is shown below:

[User] <<< Briefly describe the development of artificial intelligence. [Assistant] >>> <｜begin▁of▁sentence｜> The development of Artificial Intelligence (AI) can be divided into several major stages: 1. **Early AI**: - **Artificial Intelligence**: Initially used for specific tasks such as gaming and customer service. - **Machine Learning**: In the 1950s, computers began learning tasks like automatic recognition and speech recognition. - **Expert Systems**: In the 1970s, systems like "MYCIN" simulated human experts. 2. **Computer Vision**: - **Image Recognition**: In the 1980s, computers recognized simple images, such as handwritten digits. - **Natural Language Processing**: In the 1990s, technologies enabled automatic search and editing, e.g., on Wikipedia. 3. **Deep Learning**: - **Neural Networks**: In the 1980s, neural networks started processing complex data. - **Convolutional Neural Networks (CNNs)**: In the 1990s, CNNs were applied to image recognition, such as in autonomous vehicles. - **Deep Learning**: In the 2010s, models like GPT and BERT revolutionized natural language processing. 4. **Reinforcement Learning**: - **Robotics Control**: Starting in the 1980s, robots learned actions through trial and error. - **Autonomous Driving**: In the 2010s, reinforcement learning contributed significantly to self-driving cars. 5. **Deep Learning and Neural Networks**: - **Image Recognition**: Tasks like classification, segmentation, and generation. - **Natural Language Processing**: Including text generation, translation, and dialogue systems. - **Speech Recognition**: Enabling transcription and speech synthesis. 6. **AI Applications**: - **Healthcare**: For diagnosis and drug discovery. - **Transportation**: Including autonomous vehicles and traffic management systems. - **Education**: Intelligent tutoring systems. - **Finance**: Algorithmic trading and risk management. 7. **Ethics and Challenges**: - **Privacy Issues**: Data breaches and privacy violations. - **Ethical Concerns**: Algorithmic bias and fairness issues. 8. **Future Outlook**: - **AI Chips**: Dedicated hardware for training and inference. - **Edge AI**: Running AI directly on devices to reduce data transmission. - **Multimodal AI**: Integrating vision, audio, and other modalities. - **Human Assistants**: Chatbots and life-support systems. AI will continue advancing across multiple domains, driving technological progress and societal transformation. Performance prefill: 2348.62 tokens/s decode: 27.08 tokens/s