Rspress

# #S600 Benchmark of Model Performance

- Test Board: S600

- Range: Models under the `samples/ucp_tutorial/dnn/ai_benchmark` path in the OE package

- Runtime Environment: Linux

## #MODEL ACCURACY

MODEL NAME | INPUT SIZE | ACCURACY | Dataset
--- | --- | --- | ---
ResNet50 | 1x3x224x224 | Top1: 0.7704(FLOAT)/0.7661(INT8) | ImageNet
GoogleNet | 1x3x224x224 | Top1: 0.7018(FLOAT)/0.6995(INT8) | ImageNet
EfficientNet_Lite1 | 1x240x240x3 | Top1: 0.7652(FLOAT)/0.7602(INT8) | ImageNet
EfficientNet_Lite2 | 1x260x260x3 | Top1: 0.7734(FLOAT)/0.7696(INT8) | ImageNet
EfficientNet_Lite3 | 1x280x280x3 | Top1: 0.7917(FLOAT)/0.7885(INT8) | ImageNet
EfficientNet_Lite4 | 1x300x300x3 | Top1: 0.8063(FLOAT)/0.8041(INT8) | ImageNet
Vargconvnet | 1x3x224x224 | Top1: 0.7793(FLOAT)/0.7765(INT8) | ImageNet
Efficientnasnet_m | 1x3x300x300 | Top1: 0.7935(FLOAT)/0.7927(INT8) | ImageNet
Efficientnasnet_s | 1x3x280x280 | Top1: 0.7441(FLOAT)/0.7512(INT8) | ImageNet
ResNet18 | 1x3x224x224 | Top1: 0.7170(FLOAT)/0.7159(INT8) | ImageNet
YOLOv2_Darknet19 | 1x3x608x608 | [IoU=0.50:0.95]= 0.2760(FLOAT)/0.2707(INT8) | COCO
YOLOv3_Darknet53 | 1x3x416x416 | [IoU=0.50:0.95]= 0.3370(FLOAT)/0.3370(INT8) | COCO
YOLOv5x_v2.0 | 1x3x672x672 | [IoU=0.50:0.95]= 0.4810(FLOAT)/0.4670(INT8) | COCO
Centernet_resnet101 | 1x3x512x512 | [IoU=0.50:0.95]= 0.3420(FLOAT)/0.3270(INT8) | COCO
YOLOv3_VargDarknet | 1x3x416x416 | [IoU=0.50:0.95]= 0.3280(FLOAT)/0.3260(INT8) | COCO
Deeplabv3plus_efficientnetb0 | 1x3x1024x2048 | mIoU: 0.7630(FLOAT)/0.7569(INT8) | Cityscapes
Fastscnn_efficientnetb0 | 1x3x1024x2048 | mIoU: 0.6997(FLOAT)/0.6909(INT8) | Cityscapes
Deeplabv3plus_efficientnetm1 | 1x3x1024x2048 | mIoU: 0.7794(FLOAT)/0.7756(INT8) | Cityscapes
Deeplabv3plus_efficientnetm2 | 1x3x1024x2048 | mIoU: 0.7882(FLOAT)/0.7854(INT8) | Cityscapes
Bev_gkt_mixvargenet_multitask | image: 6x3x512x960 points(0-8): 6x64x64x2 | NDS: 0.2810(FLOAT)/0.2788(INT8) MeanIOU: 0.4852(FLOAT)/0.4839(INT8) mAP: 0.1990(FLOAT)/0.1992(INT8) | Nuscenes
Bev_ipm_4d_efficientnetb0_multitask | image: 6x3x512x960 points: 6x128x128x2 prev_feat: 1x164x28x128 prev_point: 1x128x128x2 | NDS: 0.3722(FLOAT)/0.3723(INT8) MeanIOU: 0.5287(FLOAT)/0.5389(INT8) mAP: 0.2201(FLOAT)/0.2217(INT8) | Nuscenes
Bev_ipm_efficientnetb0_multitask | image: 6x3x512x960 points: 6x128x128x2 | NDS: 0.3055(FLOAT)/0.3041(INT8) MeanIOU: 0.5145(FLOAT)/0.5103(INT8) mAP: 0.2170(FLOAT)/0.2166(INT8) | Nuscenes
Bev_lss_efficientnetb0_multitask | image: 6x3x256x704 points(0&1): 10x128x128x2 | NDS: 0.3006(FLOAT)/0.3008(INT8) MeanIOU: 0.5180(FLOAT)/0.5172(INT8) mAP: 0.2061(FLOAT)/0.2043(INT8) | Nuscenes
Detr3d_efficientnetb3 | coords(0-3): 6x4x256x2 image: 6x3x512x1408 masks: 1x4x256x24 | NDS: 0.3304(FLOAT)/0.3306(INT8) mAP: 0.2753(FLOAT)/0.2742(INT8) | Nuscenes
Petr_efficientnetb3 | image: 6x3x512x1408 pos_embed: 1x96x44x256 | NDS: 0.3765(FLOAT)/0.3748(INT8) mAP: 0.3038(FLOAT)/0.2942(INT8) | Nuscenes
Bevformer_tiny_resnet50_detection | img: 6x3x480x800 prev_bev: 1x2500x256 prev_bev_ref: 1x50x50x2 queries_rebatch_grid: 6x20x32x2 restore_bev_grid: 1x100x50x2 reference_points_rebatch: 6x640x4x2 bev_pillar_counts: 1x2500x1 | NDS: 0.3713(FLOAT)/0.3700(INT8) mAP: 0.2673(FLOAT)/0.2644(INT8) | Nuscenes
Flashocc_henet_lss_occ3d_nuscenes | img: 6x3x512x960 points: 10x128x128x2 points_depth: 10x128x128x2 | mIoU: 0.3675(FLOAT)/0.3685(INT8) | Nuscenes
Horizon_swin_transformer | 1x3x224x224 | Top1: 0.8024(FLOAT)/0.7982(INT8) | ImageNet
Vargnetv2 | 1x3x224x224 | Top1: 0.7342(FLOAT)/0.7332(INT8) | ImageNet
Vit_small | 1x3x224x224 | Top1: 0.7950(FLOAT)/0.7924(INT8) | ImageNet
Centerpoint_pointpillar | points: 300000x5 voxel_feature: 1x5x20x40000 coors: 40000x4 | NDS: 0.5832(FLOAT)/0.5820(INT8) mAP: 0.4804(FLOAT)/0.4783(INT8) | Nuscenes
Detr_efficientnetb3 | 1x3x800x1333 | [IoU=0.50:0.95]= 0.3720(FLOAT)/0.3584(INT8) | MS COCO
Detr_resnet50 | 1x3x800x1333 | [IoU=0.50:0.95]= 0.3569(FLOAT)/0.3168(INT8) | MS COCO
FCOS3D_efficientnetb0 | 1x3x512x896 | NDS: 0.3061(FLOAT)/0.3022(INT8) mAP: 0.2133(FLOAT)/0.2067(INT8) | nuscenes
Fcos_efficientnetb0 | 1x3x512x512 | [IoU=0.50:0.95]= 0.3626(FLOAT)/0.3562(INT8) | MS COCO
Ganet_mixvargenet | 1x3x320x800 | F1Score: 0.7949(FLOAT)/0.7884(INT8) | CuLane
Deformable_detr_resnet50 | 1x3x800x1333 | [IoU=0.50:0.95]= 0.4413(FLOAT)/0.4499(INT8) | MS COCO
Stereonetplus_mixvargenet | 2x3x544x960 | EPE: 1.1270(FLOAT)/1.1336(INT8) | SceneFlow
Centerpoint_mixvargnet_multitask | points: 300000x5 voxel_feature: 1x5x20x40000 coors: 40000x4 | NDS: 0.5809(FLOAT)/0.5766(INT8) MeanIOU: 0.9128(FLOAT)/0.9126(INT8) mAP: 0.4727(FLOAT)/0.4649(INT8) | Nuscenes
Unet_mobilenetv1 | 1x3x1024x2048 | mIoU: 0.6801(FLOAT)/0.6764(INT8) | Cityscapes
Densetnt_vectornet | goals_2d: 30x1x2048x2 goals_2d_mask: 30x1x2048x1 instance_mask: 30x1x96x1 lane_feat: 30x9x64x11 traj_feat: 30x19x32x9 | minFDA: 1.2975(FLOAT)/1.2989(INT8) | Argoverse 1
Maptroe_henet_tinym_bevformer | img: 6x3x480x800 osm_mask: 1x1x50x100 queries_rebatch_grid: 6x20x100x2 restore_bev_grid: 1x100x100x2 reference_points_rebatch: 6x2000x4x2 bev_pillar_counts: 1x5000x1 | mAP: 0.6633(FLOAT)/0.6569(INT8) | Nuscenes
Qcnet_oe | valid_mask: 1x30x10 valid_mask_a2a: 1x10x30x30 agent_type: 1x30x1 x_a_cur: 1x1x30x1,1x1x30x1,1x1x30x1,1x1x30x1 r_pl2a_cur: 1x1x30x80,1x1x30x80,1x1x30x80 r_t_cur: 1x1x30x6,1x1x30x6,1x1x30x6,1x1x30x6 r_a2a_cur: 1x1x30x30,1x1x30x30,1x1x30x30 x_a_mid_emb: 1x30x2x128 x_a: 1x30x6x128 pl_type,is_intersection: 1x80 r_pl2pl: 1x1x80x80,1x1x80x80,1x1x80x80 r_pt2pl: 1x1x80x50,1x1x80x50,1x1x80x50 mask_pl2pl: 1x80x80 magnitude,pt_type,side,mask: 1x80x50 mask_a2m: 1x30x30 mask_dst: 1x30x1 type_pl2pl: 1x80x80 | hitrate: 0.8026(FLOAT)/0.7979(INT8) | Argoverse 2

## #MODEL PERFORMANCE

- FPS = Frames per second. This data is obtained by running the `hrt_model_exec` tool in multi-threaded mode. For detailed usage instructions, please refer to the the hrt_model_exec tool introduction - model performance evaluation section.

- Latency = Inference latency, with the unit of ms(milliseconds). This data is obtained by running the `hrt_model_exec` tool in single-threaded mode. For detailed usage instructions, please refer to the the hrt_model_exec tool introduction - model performance evaluation section.

Note

Before performance evaluation, please refer to the following method to set the number of threads in the thread pool for CPU operator inference in model inference to 12 via environment variables:

`export HB_UCP_ENABLE_CPU_BACKEND_CORE_NUM=12`

MODEL NAME | INPUT SIZE | Latency(ms) | FPS | FPS Configuration
--- | --- | --- | --- | ---
ResNet50 | 1x3x224x224 | 0.587 | 6519.27 | thread_num:10
GoogleNet | 1x3x224x224 | 0.379 | 18480.92 | thread_num:12
EfficientNet_Lite1 | 1x240x240x3 | 0.382 | 17998.63 | thread_num:12
EfficientNet_Lite2 | 1x260x260x3 | 0.447 | 14056.08 | thread_num:10
EfficientNet_Lite3 | 1x280x280x3 | 0.512 | 11572.69 | thread_num:10
EfficientNet_Lite4 | 1x300x300x3 | 0.656 | 8327.86 | thread_num:10
Vargconvnet | 1x3x224x224 | 0.523 | 12423.31 | thread_num:12
Efficientnasnet_m | 1x3x300x300 | 0.576 | 9260.61 | thread_num:10
Efficientnasnet_s | 1x3x280x280 | 0.376 | 19348.62 | thread_num:12
ResNet18 | 1x3x224x224 | 0.384 | 14316.09 | thread_num:12
YOLOv2_Darknet19 | 1x3x608x608 | 2.229 | 1987.00 | thread_num:8
YOLOv3_Darknet53 | 1x3x416x416 | 2.453 | 1752.26 | thread_num:8
YOLOv5x_v2.0 | 1x3x672x672 | 8.202 | 494.50 | thread_num:8
Centernet_resnet101 | 1x3x512x512 | 2.446 | 1786.98 | thread_num:8
YOLOv3_VargDarknet | 1x3x416x416 | 1.628 | 2673.33 | thread_num:8
Deeplabv3plus_efficientnetb0 | 1x3x1024x2048 | 2.952 | 1468.62 | thread_num:8
Fastscnn_efficientnetb0 | 1x3x1024x2048 | 1.741 | 2647.19 | thread_num:8
Deeplabv3plus_efficientnetm1 | 1x3x1024x2048 | 5.126 | 818.14 | thread_num:8
Deeplabv3plus_efficientnetm2 | 1x3x1024x2048 | 7.330 | 561.92 | thread_num:8
Bev_gkt_mixvargenet_multitask | image: 6x3x512x960 points(0-8): 6x64x64x2 | 6.911 | 605.09 | thread_num:8
Bev_ipm_4d_efficientnetb0_multitask | image: 6x3x512x960 points: 6x128x128x2 prev_feat: 1x164x28x128 prev_point: 1x128x128x2 | 4.615 | 940.42 | thread_num:8
Bev_ipm_efficientnetb0_multitask | image: 6x3x512x960 points: 6x128x128x2 | 4.499 | 967.65 | thread_num:8
Bev_lss_efficientnetb0_multitask | image: 6x3x256x704 points(0&1): 10x128x128x2 | 3.075 | 1454.60 | thread_num:8
Detr3d_efficientnetb3 | coords(0-3): 6x4x256x2 image: 6x3x512x1408 masks: 1x4x256x24 | 15.168 | 258.88 | thread_num:8
Petr_efficientnetb3 | image: 6x3x512x1408 pos_embed: 1x96x44x256 | 21.444 | 187.70 | thread_num:8
Bevformer_tiny_resnet50_detection | img: 6x3x480x800 prev_bev: 1x2500x256 prev_bev_ref: 1x50x50x2 queries_rebatch_grid: 6x20x32x2 restore_bev_grid: 1x100x50x2 reference_points_rebatch: 6x640x4x2 bev_pillar_counts: 1x2500x1 | 14.164 | 276.44 | thread_num:8
Flashocc_henet_lss_occ3d_nuscenes | img: 6x3x512x960 points: 10x128x128x2 points_depth: 10x128x128x2 | 5.519 | 742.29 | thread_num:8
Horizon_swin_transformer | 1x3x224x224 | 1.694 | 2556.99 | thread_num:8
Vargnetv2 | 1x3x224x224 | 0.332 | 19838.28 | thread_num:12
Vit_small | 1x3x224x224 | 1.092 | 4245.19 | thread_num:8
Centerpoint_pointpillar | points: 300000x5 voxel_feature: 1x5x20x40000 coors: 40000x4 | 7.530 | 1067.16 | thread_num:16
Detr_efficientnetb3 | 1x3x800x1333 | 8.499 | 390.55 | thread_num:8
Detr_resnet50 | 1x3x800x1333 | 10.795 | 317.51 | thread_num:8
FCOS3D_efficientnetb0 | 1x3x512x896 | 1.475 | 3567.10 | thread_num:10
Fcos_efficientnetb0 | 1x3x512x512 | 0.887 | 7227.14 | thread_num:10
Ganet_mixvargenet | 1x3x320x800 | 0.550 | 12439.15 | thread_num:12
Deformable_detr_resnet50 | 1x3x800x1333 | 77.201 | 29.28 | thread_num:8
Stereonetplus_mixvargenet | 2x3x544x960 | 2.022 | 2147.39 | thread_num:8
Centerpoint_mixvargnet_multitask | points: 300000x5 voxel_feature: 1x5x20x40000 coors: 40000x4 | 6.263 | 1638.08 | thread_num:18
Unet_mobilenetv1 | 1x3x1024x2048 | 0.881 | 6401.99 | thread_num:10
Densetnt_vectornet | goals_2d: 30x1x2048x2 goals_2d_mask: 30x1x2048x1 instance_mask: 30x1x96x1 lane_feat: 30x9x64x11 traj_feat: 30x19x32x9 | 3.560 | 1108.38 | thread_num:12
Maptroe_henet_tinym_bevformer | img: 6x3x480x800 osm_mask: 1x1x50x100 queries_rebatch_grid: 6x20x100x2 restore_bev_grid: 1x100x100x2 reference_points_rebatch: 6x2000x4x2 bev_pillar_counts: 1x5000x1 | 6.439 | 625.40 | thread_num:8
Qcnet_oe | valid_mask: 1x30x10 valid_mask_a2a: 1x10x30x30 agent_type: 1x30x1 x_a_cur: 1x1x30x1,1x1x30x1,1x1x30x1,1x1x30x1 r_pl2a_cur: 1x1x30x80,1x1x30x80,1x1x30x80 r_t_cur: 1x1x30x6,1x1x30x6,1x1x30x6,1x1x30x6 r_a2a_cur: 1x1x30x30,1x1x30x30,1x1x30x30 x_a_mid_emb: 1x30x2x128 x_a: 1x30x6x128 pl_type,is_intersection: 1x80 r_pl2pl: 1x1x80x80,1x1x80x80,1x1x80x80 r_pt2pl: 1x1x80x50,1x1x80x50,1x1x80x50 mask_pl2pl: 1x80x80 magnitude,pt_type,side,mask: 1x80x50 mask_a2m: 1x30x30 mask_dst: 1x30x1 type_pl2pl: 1x80x80 | 2.611 | 1723.45 | thread_num:8

On This Page