D-Robotics Official Website
D-Robotics Official Website
English

Simple Development

In this chapter, we will introduce the basic usage workflow of D-Robotics-LLM to help you get started quickly. Here, we use the InternLM2-1.8B model as an example for demonstration.

Model and Deployment Package Preparation

  • Download the provided D-Robotics_LLM_{version}.tar.gz deployment package and extract it.

  • Download the provided InternLM2_1.8B_1024.hbm model.

Note

For the download link of the hbm model, please refer to the resolve_model.txt file located in the model folder of oellm_runtime.

After preparing both the deployment package and the model, integrate the model (*.hbm) with the oellm_runtime SDK from the deployment package. The reference directory structure is as follows:

llm └── D-Robotics_LLM_{version} └── oellm_runtime ├── model │ ├── resolve_model.txt │ └── InternLM2_1.8B_1024.hbm ├── config │ └── InternLM2_1.8B_config ├── example │ ├── oellm_run │ ├── oellm_multichat │ └── oellm_ppl ├── include ├── lib └── set_performance_mode.sh

On-Device Runtime Preparation

Create a working directory on the device using the following commands as reference:

# Create working directory mkdir -p /home/root/llm cd /home/root/llm

Copy the integrated folder from your development machine to this on-device directory. Reference command:

scp -r llm/* root@{board_ip}:/home/root/llm

Finally, under the path /home/root/llm/D-Robotics_LLM_{version}/oellm_runtime, configure LD_LIBRARY_PATH. Reference commands:

# Modify hardware register values to switch the device into performance mode sh set_performance_mode.sh # Set environment variable lib=/home/root/llm/D-Robotics_LLM_{version}/oellm_runtime/lib export LD_LIBRARY_PATH=${lib}:${LD_LIBRARY_PATH}

On-Device Execution

Reference command to run the model on the device:

cd ./example/oellm_run ./oellm_run --hbm_path ../../model/InternLM2_1.8B_1024.hbm \ --tokenizer_dir ../../config/InternLM2_1.8B_config/ \ --model_type 4

The runtime parameters are as follows:

ParameterDescriptionOptional
-h, --helpDisplay help information./
--hbm_pathSpecifies the path to the quantized model file (*.hbm).Required
--tokenizer_dirSpecifies the tokenizer configuration directory.Required
--model_typeSpecifies the model type to run; currently, the InternLM model type is 4.Required

Execution Results

After successful execution, you can perform a simple conversation test as shown below:

[User] <<< How's the weather today? [Assistant] >>> It's very clear, sunny, and breezy. Mom took me for a walk in the park. I saw a mother and daughter feeding birds and curiously asked, "Where did these birds come from?" The woman picked up a long bamboo pole, lifted a small green caterpillar from the ground, then used a long grass stem to place the insect into a nest for feeding. When I walked over, my mother kindly said, "Look how cute these little ants are!" My mom and I went to the big flowerbed and saw some children playing joyfully amid laughter. At that moment, an old man was selling watermelons! As soon as he sat down, he looked around and suddenly widened his eyes—he noticed someone approaching with a large bag to buy melons. The old man immediately ran over, pulled out a big watermelon from the bag, handed it to the person, and said, "This is a birthday gift my daughter made for me!" After returning home, Mom started telling me a story. This story taught me an important lesson: No matter where you are, as long as you put in effort, you will reap rewards. Performance prefill: 1855.07 tokens/s decode: 23.83 tokens/s