Inferencing Time in Tinker Edge R

[marco@bigsis.tech 0]

I have developed a CNN model for image classification with TensorFlow and then converted it to TensorFlow-lite. I have done a comparison between the inferencing time in the tinker edge and RPi 4B 4GB. It turned out that on the RPi 4 the inferencing time is faster ~ 1sec compared to the tinker edge r which is ~ 2sec. How is that possible. shouldn't the tinker edge have an AI accelerator unit?  Am I missing something? 

I would please appreciate your help.

 

thanks

[tooz 52]

hello @marco@bigsis.tech, 

On 12/2/2022 at 11:08 PM, marco@bigsis.tech said:

I have developed a CNN model for image classification with TensorFlow and then converted it to TensorFlow-lite.

this is processed by cpu only. 

this is the reference for rknn-toolkit for model conversion, inferene ... etc

https://github.com/rockchip-linux/rknn-toolkit

here's an example on converting tflite model to rknn model: 

https://github.com/rockchip-linux/rknn-toolkit/tree/master/examples/tflite/mobilenet_v1 

[marco@bigsis.tech 0]

On 12/5/2022 at 10:42 AM, tooz said:

hello @marco@bigsis.tech, 

this is processed by cpu only. 

this is the reference for rknn-toolkit for model conversion, inferene ... etc

https://github.com/rockchip-linux/rknn-toolkit

here's an example on converting tflite model to rknn model: 

https://github.com/rockchip-linux/rknn-toolkit/tree/master/examples/tflite/mobilenet_v1 

Hi Tooz,

Thank you very much for your response it was very helpful.

I have converted the tf lite model to a rknn model and i had an improvement from 2 sec to 0.72 sec  for inference. however is that the best that the tinker edge r can achieve? I didn't do the quantization in the rknn.build function, does it affect the speed in the inference?

below is the code I used:

 

rknn = RKNN()

print('--> Loading model')
ret = rknn.load_tflite(model = '/home/linaro/AI-big-data-1b.tflite')
if ret != 0:
    print('Load failed!')
    exit(ret)
print('done')

# Build model
print('--> Building model')
ret = rknn.build(do_quantization=True)
if ret != 0:
    print('Build *.rknn failed!')
    exit(ret)
print('done')

# Export rknn model
print('--> Export RKNN model')
ret = rknn.export_rknn("ai-module2.rknn")
if ret != 0:
    print('Export *.rknn failed!')
    exit(ret)
print('done')
 

[marco@bigsis.tech 0]

if i do      ret = rknn.build(do_quantization=True, dataset='./dataset.txt', pre_compile=False)

 

i get the below error but the model get built at the end.

to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block2_add/add_113:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block2_add/add_114:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_1_conv/Conv2D_115:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_1_conv/Conv2D_116:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_2_conv/Conv2D_117:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_2_conv/Conv2D_118:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_3_conv/BiasAdd1_119:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_add/add_120:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_add/add_121:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/avg_pool/Mean_122:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/dense/BiasAdd_123:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @StatefulPartitionedCall/0_124:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv2_block1_add/add_12_concat_126:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv2_block1_add/add_12_conv_127:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
done
--> Export RKNN model
done

[tooz 52]

hello @marco@bigsis.tech, 

On 12/9/2022 at 11:17 PM, marco@bigsis.tech said:

however is that the best that the tinker edge r can achieve? I didn't do the quantization in the rknn.build function, does it affect the speed in the inference?

it depends on how the original model is trained -- in the document of rknn toolkit, ch. 3.2.2: rknn-toolkit/Rockchip_User_Guide_RKNN_Toolkit_V1.7.3_EN.pdf at master · rockchip-linux/rknn-toolkit (github.com) listed the quantize sizes supported; using low precision integer will indeed make the inference faster

 

On 12/10/2022 at 12:15 AM, marco@bigsis.tech said:

to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block2_add/add_113:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block2_add/add_114:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_1_conv/Conv2D_115:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_1_conv/Conv2D_116:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_2_conv/Conv2D_117:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_2_conv/Conv2D_118:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_3_conv/BiasAdd1_119:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_add/add_120:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_add/add_121:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/avg_pool/Mean_122:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/dense/BiasAdd_123:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @StatefulPartitionedCall/0_124:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv2_block1_add/add_12_concat_126:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv2_block1_add/add_12_conv_127:out0 seems to be always 0, user might try to remove the correlative layer manually[0m

these are warning messages and should not be affecting anything as long as the model gets built out.