Jump to content
Sign in to follow this  
marco@bigsis.tech

Inferencing Time in Tinker Edge R

Recommended Posts

I have developed a CNN model for image classification with TensorFlow and then converted it to TensorFlow-lite. I have done a comparison between the inferencing time in the tinker edge and RPi 4B 4GB. It turned out that on the RPi 4 the inferencing time is faster ~ 1sec compared to the tinker edge r which is ~ 2sec. How is that possible. shouldn't the tinker edge have an AI accelerator unit?  Am I missing something? 

I would please appreciate your help.

 

thanks

Share this post


Link to post
Share on other sites

hello @marco@bigsis.tech

On 12/2/2022 at 11:08 PM, marco@bigsis.tech said:

I have developed a CNN model for image classification with TensorFlow and then converted it to TensorFlow-lite.

this is processed by cpu only. 

this is the reference for rknn-toolkit for model conversion, inferene ... etc

https://github.com/rockchip-linux/rknn-toolkit

here's an example on converting tflite model to rknn model: 

https://github.com/rockchip-linux/rknn-toolkit/tree/master/examples/tflite/mobilenet_v1 

Share this post


Link to post
Share on other sites
On 12/5/2022 at 10:42 AM, tooz said:

hello @marco@bigsis.tech

this is processed by cpu only. 

this is the reference for rknn-toolkit for model conversion, inferene ... etc

https://github.com/rockchip-linux/rknn-toolkit

here's an example on converting tflite model to rknn model: 

https://github.com/rockchip-linux/rknn-toolkit/tree/master/examples/tflite/mobilenet_v1 

Hi Tooz,

Thank you very much for your response it was very helpful.

I have converted the tf lite model to a rknn model and i had an improvement from 2 sec to 0.72 sec  for inference. however is that the best that the tinker edge r can achieve? I didn't do the quantization in the rknn.build function, does it affect the speed in the inference?

below is the code I used:

 

rknn = RKNN()


print('--> Loading model')
ret = rknn.load_tflite(model = '/home/linaro/AI-big-data-1b.tflite')
if ret != 0:
    print('Load failed!')
    exit(ret)
print('done')

# Build model
print('--> Building model')
ret = rknn.build(do_quantization=True)
if ret != 0:
    print('Build *.rknn failed!')
    exit(ret)
print('done')

# Export rknn model
print('--> Export RKNN model')
ret = rknn.export_rknn("ai-module2.rknn")
if ret != 0:
    print('Export *.rknn failed!')
    exit(ret)
print('done')
 

Share this post


Link to post
Share on other sites

if i do      ret = rknn.build(do_quantization=True, dataset='./dataset.txt', pre_compile=False)

 

i get the below error but the model get built at the end.

to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block2_add/add_113:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block2_add/add_114:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_1_conv/Conv2D_115:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_1_conv/Conv2D_116:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_2_conv/Conv2D_117:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_2_conv/Conv2D_118:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_3_conv/BiasAdd1_119:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_add/add_120:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_add/add_121:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/avg_pool/Mean_122:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/dense/BiasAdd_123:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @StatefulPartitionedCall/0_124:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv2_block1_add/add_12_concat_126:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv2_block1_add/add_12_conv_127:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
done
--> Export RKNN model
done

Share this post


Link to post
Share on other sites

hello @marco@bigsis.tech

On 12/9/2022 at 11:17 PM, marco@bigsis.tech said:

however is that the best that the tinker edge r can achieve? I didn't do the quantization in the rknn.build function, does it affect the speed in the inference?

it depends on how the original model is trained -- in the document of rknn toolkit, ch. 3.2.2: rknn-toolkit/Rockchip_User_Guide_RKNN_Toolkit_V1.7.3_EN.pdf at master · rockchip-linux/rknn-toolkit (github.com) listed the quantize sizes supported; using low precision integer will indeed make the inference faster

 

On 12/10/2022 at 12:15 AM, marco@bigsis.tech said:

to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block2_add/add_113:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block2_add/add_114:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_1_conv/Conv2D_115:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_1_conv/Conv2D_116:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_2_conv/Conv2D_117:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_2_conv/Conv2D_118:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_3_conv/BiasAdd1_119:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_add/add_120:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv5_block3_add/add_121:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/avg_pool/Mean_122:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/dense/BiasAdd_123:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @StatefulPartitionedCall/0_124:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv2_block1_add/add_12_concat_126:out0 seems to be always 0, user might try to remove the correlative layer manually[0m
[33mW tensor @sequential/resnet50/conv2_block1_add/add_12_conv_127:out0 seems to be always 0, user might try to remove the correlative layer manually[0m

these are warning messages and should not be affecting anything as long as the model gets built out. 

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×
×
  • Create New...