Evaluation of models on OAK devices

[tersekmatija]

Start with the why:

We've accumulated a lot of different models that are available in luxonis/depthai-experiments and this Model ZOO. With the development of web based search engine for models available on OAK devices, it would be beneficial to evaluate the models on the data sets they were trained on, with execution on OAKs. During the conversion some layers might be replaced, and models based on FP32 are converted to FP16. Such evaluation would show whether the model was converted successfully, emphasize possible issues, as well as enable us to compare which model performs better on OAK devices.

Move to the what:

• Evaluation of models in model ZOO on popular task related data sets:

  • depth_estimation_mbnv2_240x320,
  • depth_estimation_mbnv2_480x640,
  • dmcount_540x960,
  • east_text_detection_256x256,
  • fast_depth_256x320,
  • fast_depth_480x640,
  • head-pose-estimation_160x160,
  • hrdepth_192x640,
  • inceptionv4_299x299,
  • megadepth,
  • mobile_object_localizer_192x192,
  • scdepth_256x832,
  • yolop_320x320.

• Provide information on which data sets the models in the zoo were trained on.

Move to the how:

This can be executed in two steps:

1. provide an API that exposes the OAK as a backbone for computation, so evaluation on device is as easy as few lines of code,
2. download required data sets and adapt models' evaluation scripts so that OAK is used as a backbone.

As this is a tedious process, help from the community is much appreciated!