[Model] Add tinypose single && pipeline model (PaddlePaddle#177)

* Add tinypose model

* Add PPTinypose python API

* Fix picodet preprocess bug && Add Tinypose examples

* Update tinypose example code

* Update ppseg preprocess if condition

* Update ppseg backend support type

* Update permute.h

* Update README.md

* Update code with comments

* Move files dir

* Delete premute.cc

* Add single model pptinypose

* Delete pptinypose old code in ppdet

* Code format

* Add ppdet + pptinypose pipeline model

* Fix bug for posedetpipeline

* Change Frontend to ModelFormat

* Change Frontend to ModelFormat in __init__.py

* Add python posedetpipeline/

* Update pptinypose example dir name

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Create keypointdetection_result.md

* Create README.md

* Create README.md

* Create README.md

* Update README.md

* Update README.md

* Create README.md

* Fix det_keypoint_unite_infer.py bug

* Create README.md

* Update PP-Tinypose by comment

* Update by comment

* Add pipeline directory

* Add pptinypose dir

* Update pptinypose to align accuracy

* Addd warpAffine processor

* Update GetCpuMat to  GetOpenCVMat

* Add comment for pptinypose && pipline

* Update docs/main_page.md

* Add README.md for pptinypose

* Add README for det_keypoint_unite

* Remove ENABLE_PIPELINE option

* Remove ENABLE_PIPELINE option

* Change pptinypose default backend

* PP-TinyPose Pipeline support multi PP-Detection models

* Update pp-tinypose comment

* Update by comments

* Add single test example

Co-authored-by: Jason <[email protected]>

Author: felixhjh

CMakeLists.txt

@@ -184,10 +184,11 @@ file(GLOB_RECURSE DEPLOY_TRT_SRCS ${PROJECT_SOURCE_DIR}/${CSRCS_DIR_NAME}/fastde
 file(GLOB_RECURSE DEPLOY_OPENVINO_SRCS ${PROJECT_SOURCE_DIR}/${CSRCS_DIR_NAME}/fastdeploy/backends/openvino/*.cc)
 file(GLOB_RECURSE DEPLOY_LITE_SRCS ${PROJECT_SOURCE_DIR}/${CSRCS_DIR_NAME}/fastdeploy/backends/lite/*.cc)
 file(GLOB_RECURSE DEPLOY_VISION_SRCS ${PROJECT_SOURCE_DIR}/${CSRCS_DIR_NAME}/fastdeploy/vision/*.cc)
+file(GLOB_RECURSE DEPLOY_PIPELINE_SRCS ${PROJECT_SOURCE_DIR}/${CSRCS_DIR_NAME}/fastdeploy/pipeline/*.cc)
 file(GLOB_RECURSE DEPLOY_VISION_CUDA_SRCS ${PROJECT_SOURCE_DIR}/${CSRCS_DIR_NAME}/fastdeploy/vision/*.cu)
 file(GLOB_RECURSE DEPLOY_TEXT_SRCS ${PROJECT_SOURCE_DIR}/${CSRCS_DIR_NAME}/fastdeploy/text/*.cc)
 file(GLOB_RECURSE DEPLOY_PYBIND_SRCS ${PROJECT_SOURCE_DIR}/${CSRCS_DIR_NAME}/fastdeploy/pybind/*.cc ${PROJECT_SOURCE_DIR}/${CSRCS_DIR_NAME}/fastdeploy/*_pybind.cc)
-list(REMOVE_ITEM ALL_DEPLOY_SRCS ${DEPLOY_ORT_SRCS} ${DEPLOY_PADDLE_SRCS} ${DEPLOY_POROS_SRCS} ${DEPLOY_TRT_SRCS} ${DEPLOY_OPENVINO_SRCS} ${DEPLOY_LITE_SRCS} ${DEPLOY_VISION_SRCS} ${DEPLOY_TEXT_SRCS})
+list(REMOVE_ITEM ALL_DEPLOY_SRCS ${DEPLOY_ORT_SRCS} ${DEPLOY_PADDLE_SRCS} ${DEPLOY_POROS_SRCS} ${DEPLOY_TRT_SRCS} ${DEPLOY_OPENVINO_SRCS} ${DEPLOY_LITE_SRCS} ${DEPLOY_VISION_SRCS} ${DEPLOY_TEXT_SRCS} ${DEPLOY_PIPELINE_SRCS})
 
 set(DEPEND_LIBS "")
 
@@ -389,6 +390,7 @@ if(ENABLE_VISION)
     list(APPEND DEPLOY_VISION_SRCS ${DEPLOY_VISION_CUDA_SRCS})
   endif()
   list(APPEND ALL_DEPLOY_SRCS ${DEPLOY_VISION_SRCS})
+  list(APPEND ALL_DEPLOY_SRCS ${DEPLOY_PIPELINE_SRCS})
   include_directories(${PROJECT_SOURCE_DIR}/third_party/yaml-cpp/include)
   include(${PROJECT_SOURCE_DIR}/cmake/opencv.cmake)
 
@@ -586,7 +588,8 @@ if(BUILD_FASTDEPLOY_PYTHON)
 
   if(NOT ENABLE_VISION)
     file(GLOB_RECURSE VISION_PYBIND_SRCS ${PROJECT_SOURCE_DIR}/${CSRCS_DIR_NAME}/fastdeploy/vision/*_pybind.cc)
-    list(REMOVE_ITEM DEPLOY_PYBIND_SRCS ${VISION_PYBIND_SRCS})
+    file(GLOB_RECURSE PIPELINE_PYBIND_SRCS ${PROJECT_SOURCE_DIR}/${CSRCS_DIR_NAME}/fastdeploy/pipeline/*_pybind.cc)
+    list(REMOVE_ITEM DEPLOY_PYBIND_SRCS ${VISION_PYBIND_SRCS} ${PIPELINE_PYBIND_SRCS})
   endif()
 
   if (NOT ENABLE_TEXT)

docs/api/vision_results/README.md

@@ -6,8 +6,9 @@ FastDeploy根据视觉模型的任务类型，定义了不同的结构体(`fastd
 | :----- | :--- | :---- | :------- |
 | ClassifyResult | [C++/Python文档](./classification_result.md) | 图像分类返回结果 | ResNet50、MobileNetV3等 |
 | SegmentationResult | [C++/Python文档](./segmentation_result.md) | 图像分割返回结果 | PP-HumanSeg、PP-LiteSeg等 |
-| DetectionResult | [C++/Python文档](./detection_result.md) | 目标检测返回结果 | PPYOLOE、YOLOv7系列模型等 |
+| DetectionResult | [C++/Python文档](./detection_result.md) | 目标检测返回结果 | PP-YOLOE、YOLOv7系列模型等 |
 | FaceDetectionResult | [C++/Python文档](./face_detection_result.md) | 目标检测返回结果 | SCRFD、RetinaFace系列模型等 |
+| KeyPointDetectionResult | [C++/Python文档](./keypointdetection_result.md) | 关键点检测返回结果 | PP-Tinypose系列模型等 |
 | FaceRecognitionResult | [C++/Python文档](./face_recognition_result.md) | 目标检测返回结果 | ArcFace、CosFace系列模型等 |
 | MattingResult | [C++/Python文档](./matting_result.md) | 目标检测返回结果 | MODNet系列模型等 |
 | OCRResult | [C++/Python文档](./ocr_result.md) | 文本框检测，分类和文本识别返回结果 | OCR系列模型等 |

docs/api/vision_results/keypointdetection_result.md

@@ -0,0 +1,45 @@
+# KeyPointDetectionResult 目标检测结果
+
+KeyPointDetectionResult 代码定义在`fastdeploy/vision/common/result.h`中，用于表明图像中目标行为的各个关键点坐标和置信度。
+
+## C++ 定义
+
+`fastdeploy::vision::KeyPointDetectionResult`
+
+```c++
+struct KeyPointDetectionResult {
+  std::vector<std::array<float, 2>> keypoints;
+  std::vector<float> scores;
+  int num_joints = -1;
+  void Clear();
+  std::string Str();
+};
+```
+
+- **keypoints**: 成员变量，表示识别到的目标行为的关键点坐标。`keypoints.size()= N * J * 2`，
+    - `N`：图片中的目标数量
+    - `J`：num_joints（一个目标的关键点数量）
+    - `3`:坐标信息[x, y]
+- **scores**: 成员变量，表示识别到的目标行为的关键点坐标的置信度。`scores.size()= N * J`
+    - `N`：图片中的目标数量
+    - `J`:num_joints（一个目标的关键点数量）
+- **num_joints**: 成员变量，一个目标的关键点数量
+
+- **num_joints**: 成员变量，一个目标的关键点数量
+- **Clear()**: 成员函数，用于清除结构体中存储的结果
+- **Str()**: 成员函数，将结构体中的信息以字符串形式输出（用于Debug）
+
+## Python 定义
+
+`fastdeploy.vision.KeyPointDetectionResult`
+
+- **keypoints**(list of list(float)): 成员变量，表示识别到的目标行为的关键点坐标。
+                                      `keypoints.size()= N * J * 2`
+                                      `N`:图片中的目标数量
+                                      `J`:num_joints（关键点数量）
+                                     `3`:坐标信息[x, y, conf]
+- **scores**(list of float): 成员变量，表示识别到的目标行为的关键点坐标的置信度。
+                             `scores.size()= N * J`
+                             `N`:图片中的目标数量
+                             `J`:num_joints（一个目标的关键点数量）
+- **num_joints**(int): 成员变量，一个目标的关键点数量

docs/api_docs/cpp/main_page.md

@@ -26,5 +26,6 @@ Currently, FastDeploy supported backends listed as below,
 | Task | Model | API | Example |
 | :---- | :---- | :---- | :----- |
 | object detection | PaddleDetection/PPYOLOE | [fastdeploy::vision::detection::PPYOLOE](./classfastdeploy_1_1vision_1_1detection_1_1PPYOLOE.html) | [C++](./)/[Python](./) |
+| keypoint detection | PaddleDetection/PPTinyPose | [fastdeploy::vision::keypointdetection::PPTinyPose](./classfastdeploy_1_1vision_1_1keypointdetection_1_1PPTinyPose.html) | [C++](./)/[Python](./) |
 | image classification | PaddleClassification serials | [fastdeploy::vision::classification::PaddleClasModel](./classfastdeploy_1_1vision_1_1classification_1_1PaddleClasModel.html) | [C++](./)/[Python](./) |
 | semantic segmentation | PaddleSegmentation serials | [fastdeploy::vision::classification::PaddleSegModel](./classfastdeploy_1_1vision_1_1segmentation_1_1PaddleSegModel.html) | [C++](./)/[Python](./) |

docs/api_docs/python/keypoint_detection.md

@@ -1,3 +1,9 @@
 # Keypoint Detection API
 
-comming soon...
+## fastdeploy.vision.keypointdetection.PPTinyPose
+
+```{eval-rst}
+.. autoclass:: fastdeploy.vision.keypointdetection.PPTinyPose
+    :members:
+    :inherited-members:
+```

docs/api_docs/python/vision_results_cn.md

@@ -40,6 +40,19 @@ API:`fastdeploy.vision.FaceDetectionResult` , 该结果返回:
 - **landmarks**(list of list(float)): 成员变量，表示单张图片检测出来的所有人脸的关键点.
 - **landmarks_per_face**(int): 成员变量，表示每个人脸框中的关键点的数量.
 
+## KeyPointDetectionResult
+KeyPointDetectionResult 代码定义在`fastdeploy/vision/common/result.h`中，用于表明图像中目标行为的各个关键点坐标和置信度。
+
+API:`fastdeploy.vision.KeyPointDetectionResult` , 该结果返回:
+- **keypoints**(list of list(float)): 成员变量，表示识别到的目标行为的关键点坐标。`keypoints.size()= N * J * 2`，
+    - `N`：图片中的目标数量
+    - `J`：num_joints（一个目标的关键点数量）
+    - `3`:坐标信息[x, y]
+- **scores**(list of float): 成员变量，表示识别到的目标行为的关键点坐标的置信度。`scores.size()= N * J`
+    - `N`：图片中的目标数量
+    - `J`:num_joints（一个目标的关键点数量）
+- **num_joints**(int): 成员变量，表示一个目标的关键点数量
+
 
 ## FaceRecognitionResult
 FaceRecognitionResult 代码定义在`fastdeploy/vision/common/result.h`中，用于表明人脸识别模型对图像特征的embedding.

docs/api_docs/python/vision_results_en.md

@@ -40,6 +40,19 @@ API: `fastdeploy.vision.FaceDetectionResult`, The FaceDetectionResult will retur
 - **landmarks**(list of list(float)): Member variables that represent the key points of all faces detected by a single image.
 - **landmarks_per_face**(int):Member variable indicating the number of key points in each face frame.
 
+## KeyPointDetectionResult
+The KeyPointDetectionResult code is defined in `fastdeploy/vision/common/result.h` and is used to indicate the coordinates and confidence of each keypoint of the target behavior in the image.
+
+API:`fastdeploy.vision.KeyPointDetectionResult`, The KeyPointDetectionResult will return:
+- **keypoints**(list of list(float)): Member variable, representing the key point coordinates of the identified target behavior. `keypoints.size()= N * J * 2`，
+    - `N`: number of objects in the picture
+    - `J`: num_joints（number of keypoints for a target）
+    - `3`: 坐标信息[x, y]
+- **scores**(list of float): Member variable, representing the confidence of the keypoint coordinates of the recognized target behavior. `scores.size()= N * J`
+    - `N`: number of objects in the picture
+    - `J`: num_joints（number of keypoints for a target）
+- **num_joints**(int): Member variable, representing the number of keypoints for a target
+
 ## FaceRecognitionResult
 The FaceRecognitionResult code is defined in `fastdeploy/vision/common/result.h` and is used to indicate the embedding of the image features by the face recognition model.
 

examples/vision/README.md

@@ -8,6 +8,7 @@
 | Segmentation   | 语义分割，输入图像，给出图像中每个像素的分类及置信度          | [SegmentationResult](../../docs/api/vision_results/segmentation_result.md) |
 | Classification | 图像分类，输入图像，给出图像的分类结果和置信度             | [ClassifyResult](../../docs/api/vision_results/classification_result.md)   |
 | FaceDetection | 人脸检测，输入图像，检测图像中人脸位置，并返回检测框坐标及人脸关键点             | [FaceDetectionResult](../../docs/api/vision_results/face_detection_result.md)   |
+| KeypointDetection   | 关键点检测，输入图像，返回图像中人物行为的各个关键点坐标和置信度         | [KeyPointDetectionResult](../../docs/api/vision_results/keypointdetection_result.md) |
 | FaceRecognition | 人脸识别，输入图像，返回可用于相似度计算的人脸特征的embedding            | [FaceRecognitionResult](../../docs/api/vision_results/face_recognition_result.md)   |
 | Matting | 抠图，输入图像，返回图片的前景每个像素点的Alpha值            | [MattingResult](../../docs/api/vision_results/matting_result.md)   |
 | OCR | 文本框检测，分类，文本框内容识别，输入图像，返回文本框坐标，文本框的方向类别以及框内的文本内容            | [OCRResult](../../docs/api/vision_results/ocr_result.md)   |

examples/vision/keypointdetection/README.md

@@ -0,0 +1,17 @@
+# 关键点检测模型
+
+FastDeploy目前支持两种关键点检测任务方式的部署
+
+| 任务 | 说明 | 模型格式 | 示例 | 版本 |
+| :---| :--- | :--- | :------- | :--- |
+| 单人关键点检测 | 部署PP-TinyPose系列模型，输入图像仅包含单人 | Paddle | 参考[tinypose目录](./tiny_pose/) |  [Release/2.5](https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.5/configs/keypoint/tiny_pose) |
+| 单人/多人关键点检测 | 部署PicoDet + PP-TinyPose的模型串联任务，输入图像先通过检测模型，得到独立的人像子图后，再经过PP-TinyPose模型检测关键点 | Paddle | 参考[det_keypoint_unite目录](./det_keypoint_unite/) |[Release/2.5](https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.5/configs/keypoint/tiny_pose) |
+
+# 预训练模型准备
+本文档提供了如下预训练模型，开发者可直接下载使用
+| 模型 | 说明 | 模型格式 | 版本 |
+| :--- | :--- | :------- | :--- |
+| [PP-TinyPose-128x96](https://bj.bcebos.com/paddlehub/fastdeploy/PP_TinyPose_128x96_infer.tgz) | 单人关键点检测模型 | Paddle | [Release/2.5](https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.5/configs/keypoint/tiny_pose) |
+| [PP-TinyPose-256x192](https://bj.bcebos.com/paddlehub/fastdeploy/PP_TinyPose_256x192_infer.tgz) | 单人关键点检测模型 | Paddle | [Release/2.5](https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.5/configs/keypoint/tiny_pose) |
+| [PicoDet-S-Lcnet-Pedestrian-192x192](https://bj.bcebos.com/paddlehub/fastdeploy/PP_PicoDet_V2_S_Pedestrian_192x192_infer.tgz) + [PP-TinyPose-128x96](https://bj.bcebos.com/paddlehub/fastdeploy/PP_TinyPose_128x96_infer.tgz) | 单人关键点检测串联配置 | Paddle |[Release/2.5](https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.5/configs/keypoint/tiny_pose) |
+| [PicoDet-S-Lcnet-Pedestrian-320x320](https://bj.bcebos.com/paddlehub/fastdeploy/PP_PicoDet_V2_S_Pedestrian_320x320_infer.tgz) + [PP-TinyPose-256x192](https://bj.bcebos.com/paddlehub/fastdeploy/PP_TinyPose_256x192_infer.tgz)  | 多人关键点检测串联配置 | Paddle |[Release/2.5](https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.5/configs/keypoint/tiny_pose) |

examples/vision/keypointdetection/det_keypoint_unite/README.md

@@ -0,0 +1,38 @@
+# PP-PicoDet + PP-TinyPose 联合部署(Pipeline)
+
+## 模型版本说明
+
+- [PaddleDetection release/2.5](https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.5)
+
+目前FastDeploy支持如下模型的部署
+
+- [PP-PicoDet + PP-TinyPose系列模型](https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.5/configs/keypoint/tiny_pose/README.md)
+
+## 准备PP-TinyPose部署模型
+
+PP-TinyPose以及PP-PicoDet模型导出，请参考其文档说明[模型导出](https://github.com/PaddlePaddle/PaddleDetection/blob/release/2.5/deploy/EXPORT_MODEL.md)  
+
+**注意**:导出的推理模型包含`model.pdmodel`、`model.pdiparams`和`infer_cfg.yml`三个文件，FastDeploy会从yaml文件中获取模型在推理时需要的预处理信息。
+
+## 下载预训练模型
+
+为了方便开发者的测试，下面提供了PP-PicoDet + PP-TinyPose（Pipeline）导出的部分模型，开发者可直接下载使用。
+
+| 应用场景                         |  模型                                 | 参数文件大小 |  AP(业务数据集) | AP(COCO Val 单人/多人) | 单人/多人推理耗时 (FP32) | 单人/多人推理耗时（FP16) |
+|:-------------------------------|:--------------------------------- |:----- |:----- | :----- | :----- | :----- |
+| 单人模型配置 |[PicoDet-S-Lcnet-Pedestrian-192x192](https://bj.bcebos.com/paddlehub/fastdeploy/PP_PicoDet_V2_S_Pedestrian_192x192_infer.tgz) + [PP-TinyPose-128x96](https://bj.bcebos.com/paddlehub/fastdeploy/PP_TinyPose_128x96_infer.tgz) | 4.6MB + 5.3MB | 86.2% | 52.8% | 12.90ms | 9.61ms |
+| 多人模型配置 |[PicoDet-S-Lcnet-Pedestrian-320x320](https://bj.bcebos.com/paddlehub/fastdeploy/PP_PicoDet_V2_S_Pedestrian_320x320_infer.tgz) + [PP-TinyPose-256x192](https://bj.bcebos.com/paddlehub/fastdeploy/PP_TinyPose_256x192_infer.tgz)  | 4.6M + 5.3MB | 85.7% | 49.9% | 47.63ms | 34.62ms |
+
+**说明**
+- 关键点检测模型的精度指标是基于对应行人检测模型检测得到的检测框。
+- 精度测试中去除了flip操作，且检测置信度阈值要求0.5。
+- 速度测试环境为qualcomm snapdragon 865，采用arm8下4线程推理。
+- Pipeline速度包含模型的预处理、推理及后处理部分。
+- 精度测试中，为了公平比较，多人数据去除了6人以上（不含6人）的图像。
+
+更多信息请参考：[PP-TinyPose 官方文档](https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.5/configs/keypoint/tiny_pose/README.md)
+
+## 详细部署文档
+
+- [Python部署](python)
+- [C++部署](cpp)