#The oellm_build Tool

The oellm_build tool is provided by D-Robotics to convert floating-point model into quantized model. It completes model quantization and compilation based on the original floating-point model, a json configuration file(optional), and calibration data(optional), and finally generates a deployable *.hbm model.

#Usage

#Parameters Introduction

#JSON Configuration File Description

1. The JSON configuration file for text calibration data, with a sample shown below:

   [ {"text": "<|User|>Please elaborate on the future development of artificial intelligence.<|Assistant|><think>Analyze in depth from the perspectives of technology, ethics, and industrial development, and predict possible future trends.</think><|Assistant|>"}, {"text": "<|User|>Write a quatrain about spring.<|Assistant|><think>Construct verses around spring imagery such as wind, flowers, birds, and willows.</think><|Assistant|>"} ]

2. Calibrate data required for the Qwen2.5-Omni model JSON Configuration File Description, with a sample shown below:

   [ { "conversation": [ { "role": "system", "content": [ {"type": "text", "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."} ] }, { "role": "user", "content": [ {"type": "text", "text": "Please answer me."}, {"type": "video", "video": "./draw.mp4"}, {"type": "image", "image": "./images/image4.jpg"}, {"type": "audio", "audio": "./audios/audio2.mp3"} ] } ] } ]

Description of configuration file parameters:

(1) When "role" is "system", the first element in the content list must be a text element containing the text field; otherwise, an error will occur when accessing text during template formatting.

(2) When "role" is "user", the content list supports types including text,audio,image,video.The specific rules are as follows:

• When the type in the content list is "text":

  • Format restrictions:No special format requirements; plain text, sentences with punctuation, short instructions, and long paragraphs are all supported.

  • Supported sources:No fixed source restrictions.

  • Example reference:

  [ { "conversation": [ { "role": "user", "content": [ {"type": "text", "text": "Please answer me."} ] } ] } ]

• When the type in the content list is "video":

  • Format restrictions:MP4、MKV.

  • Supported sources:Local video files, local file URLs, and web URLs.

  • Example reference:

  [ { "conversation": [ { "role": "user", "content": [ {"type": "video", "video": "./draw.mp4"}, {"type": "video", "video": "file://server/audios/draw.mp4"}, {"type": "video", "video": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2.5-Omni/draw.mp4"} ] } ] } ]

• When the type in the content list is "image"`:

  • Format restrictions:PNG、JPG、JPEG、BMP.

  • Supported sources:Local image files, local file URLs, web URLs, and Data URI.

  • Example reference:

  [ { "conversation": [ { "role": "user", "content": [ {"type": "image", "image": "./images/image4.jpg"}, {"type": "image", "image": "file://server/audios/image1.jpg"}, {"type": "image", "image": "https://huggingface.co/OpenGVLab/InternVL2-1B/blob/main/examples/image1.jpg"}, {"type": "image", "image": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVQIHWP4//8/AwAI/AL+4J4fNQAAAABJRU5ErkJggg=="} ] } ] } ]

• When the type in the content list is "audio":

  • Format restrictions:WAV、MP3、FLAC.

  • Supported sources:Local audio files, local file URLs, web URLs, Data URI.

  • Example reference:

  [ { "conversation": [ { "role": "user", "content": [ {"type": "audio", "audio": "./audios/audio2.mp3"}, {"type": "audio", "audio": "file://server/audios/image1.flac"}, {"type": "audio", "audio": "https://huggingface.co/datasets/hf-internal-testing/dummy-flac-single-example/blob/main/example.flac"}, {"type": "audio", "audio": "data:audio/wav;base64,xxxxx...AA"} ] } ] } ]

#Usage Example

• The DeepSeek-R1-Distill-Qwen model uses the oellm_build tool for model quantization. Refer to the following command:

  oellm_build \ --model_name ${model_name} \ --input_model_path ${input_model_path} \ --output_model_path ${output_model_path} \ --march ${march} \ --calib_text_path ${calib_text_path} \ --chunk_size ${chunk_size} \ --cache_len ${cache_len}

• The DeepSeek-R1-Distill-Qwen model uses the oellm_build tool for model quantization and perform consistency verification on the quantized HBM model. Refer to the following command:

  oellm_build \ --model_name ${model_name} \ --input_model_path ${input_model_path} \ --output_model_path ${output_model_path} \ --march ${march} \ --calib_text_path ${calib_text_path} \ --chunk_size ${chunk_size} \ --cache_len ${cache_len} \ --verifier \ --remote_ip ${march} \

• The InternLM2 model uses the oellm_build tool to perform model quantization. Refer to the following command:

  oellm_build \ --model_name ${model_name} \ --input_model_path ${input_model_path} \ --output_model_path ${output_model_path} \ --march ${march} \ --calib_text_path ${calib_text_path} \ --chunk_size ${chunk_size} \ --cache_len ${cache_len}

• The Qwen2.5 model uses the oellm_build tool to perform model quantization. Refer to the following command:

  oellm_build \ --model_name ${model_name} \ --input_model_path ${input_model_path} \ --output_model_path ${output_model_path} \ --march ${march} \ --calib_text_path ${calib_text_path} \ --chunk_size ${chunk_size} \ --cache_len ${cache_len}

• The Qwen2.5-Omni model uses the oellm_build tool for model quantization. Refer to the following command:

  oellm_build \ --model_name ${model_name} \ --input_model_path ${input_model_path} \ --output_model_path ${output_model_path} \ --calib_conversation_path ${calib_conversation_path} \ --march ${march} \ --chunk_size ${chunk_size} \ --cache_len ${cache_len}