diff --git a/README.md b/README.md index 0d5ee5e..90b8629 100644 --- a/README.md +++ b/README.md @@ -1,58 +1,43 @@ -# MAMMAL core - -This repository contains the core code of MAMMAL system. The system is written with C++17. - -# Environment -The code has been tested on Windows 10 system with NVIDIA RTX 2080Ti GPU and CUDA 10.2. - -After unzip this project, you need to download some necessary data from [Google Drive](https://drive.google.com/file/d/1m9hKCMhI_VJb1muM-sS_01TgYdDqaz3D/view?usp=sharing). The file you download is `data.zip`, just unzip it and put it under the MAMMAL_core main folder. Besides, download my pre-compiled third-party libraries from the link (https://drive.google.com/file/d/1Df-p3nQLE5lPo999eOvaKY59sMQurmW9/view?usp=sharing). Then unzip the third-party libraries and put it under the MAMMAL_core main folder. You will get a folder structure like -``` -\-- MAMMAL_core\ - |-- 3rdparty\ - |-- data\ - |-- annotator\ - |-- articulation\ - |-- configs\ - |-- posesolver\ - |-- props\ - |-- render\ - |-- utils\ - |-- .gitignore - |-- README.md - |-- MAMMAL_core.sln -``` -Then, download data files for render from [Google Drive](https://drive.google.com/file/d/1xZqepoemvG6aPBnMBn69cvaUKz5JMlZd/view?usp=sharing) and unzip it under `render/` folder like -``` -\-- MAMMAL_core\ - \-- 3rdparty\ - |-- data\ - |-- annotator\ - |-- articulation\ - |-- configs\ - |-- posesolver\ - |-- props\ - |-- render\ - \-- data\ - |-- shader\ - |-- utils\ - |-- .gitignore - |-- README.md - |-- MAMMAL_core.sln -``` -To open and run the project, we recommend to download and install Visual Studio 2017 Community from their official website (https://visualstudio.microsoft.com/). After installing Visual Studio 2017, you should install CUDA Runtime API 10.2 (or higher version. However, if you use higher version, you need to change some property configurations after opening the project. ). To install it, you should download from NVIDIA developer website (https://developer.nvidia.com/cuda-10.2-download-archive) and run the .exe file directly. Usually, both Visual Studio 2017 and CUDA are installed under C:/ drive by default. After installing above softwares, you can open the project now! - -# Open the project -By simply clicking `MAMMAL_core.sln` file, the Visual Studio 2017 would open it. -Then, you will see five `projects` under the `MAMMAL_core` solution: -``` -annotator -articulation -posesolver -render -utils -``` -You can choose to set one of `annotator`, `articulation`, `posesolver` or `render` as the "start project". We recommend to try `render` first because its dependencies are the least. The, set the project compile properties as `Release` + `x64`. Finally, click the green triangle button to compile and run the project. If everything goes right, you will see a rendering of pig liveing environment model. - - -# Run demo on BamaPig3D dataset -The tutorial on how to run demo on BamaPig3D will be provided soon. \ No newline at end of file +# MAMMAL +![](pics/teaser.png) +This repository contains the core code of **MAMMAL** system proposed in paper _Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL_. +This is the **first** system demonstrating pose and shape reconstruction of socially interacting animals. Moreover, it is the **first** system to analyse pig behavior in three-dimensional space. + +## Installation +The code is written using C++ Std 17 on Visual Studio Community 2017. Please refer to `docs/INSTALL.md` for installation guide. + +## Quick demo +Please refer to `docs/Demo.md`. + +## The PIG model +To enable the MAMMAL system, we proposed a new pig articulation model named `PIG` model, as in the link https://github.com/anl13/PIG_model. This link contains necessary details and code for you to understand the model. +![](pics/model.jpg) + +## BamaPig2D and BamaPig3D dataset +Check out link https://github.com/anl13/MAMMAL_datasets for descriptions of these datasets. Briefly, BamaPig2D dataset contains 3340 images with more than 10k pig instances fully labeled with bounding boxes, silhouettes and keypoints, and is formated in COCO style. We also provide a modified COCO annotation file suitable for training [SLEAP](https://sleap.ai). BamaPig3D dataset provides synchronized 70-second videos of 10 views (1750 frames, 25FPS). We manually labeled 2D keypoints, 2D silhouettes and 3D mesh/keypoints every 25 frames, resulting in 70 fully labeled frames. BamaPig3D dataset is used for evaluating the performance of MAMMAL system. However, it is also suitable for training 3D algorithms (e.g. [DANNCE](https://github.com/spoonsso/dannce) or [LiftPose3D](https://github.com/NeLy-EPFL/LiftPose3D)). The figure shows examples of BamaPig2D dataset. +![](pics/BamaPig2D.jpg) + +## MAMMAL detection +MAMMAL detection contains two seperate parts: pig bounding box and silhouette detection modified from Detectron2, and pig pose detection modified from HRNet. Please refer to https://github.com/anl13/pig_silhouette_det and https://github.com/anl13/pig_pose_det for more details. + +## Evaluation +Key evaluation code is released at https://github.com/anl13/MAMMAL_evaluation. It helps reproduce some figures of the paper. The results used for evaluation are also provided together with the code. Our provided results perfactly match the paper figures. However, if run the algorithm again, it may change a little (because we have updated the code recently), but not much. + +## Pig behavior analysis +Please refer to code https://github.com/anl13/MAMMAL_behavior for behavior analysis performed in the paper. + +## Rendering +To effectively render the model in python environment, we developed a `pig_renderer` as in https://github.com/anl13/pig_renderer. + +## Citation +If you use these datasets in your research, please cite the paper + +```BibTex +@article{MAMMAL, + author = {An, Liang and Ren, Jilong and Yu, Tao and Hai, Tang and Jia, Yichang and Liu, Yebin}, + title = {Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL}, + booktitle = {}, + month = {July}, + year = {2022} +} +``` \ No newline at end of file diff --git a/docs/INSTALL.md b/docs/INSTALL.md index 7375e84..caab29f 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -75,7 +75,7 @@ Note that, on the left of the green triangle buttom "Local Windows Debugger", ch The whole MAMMAL\_core solution contains five projects named `annotator`, `articulation`, `posesolver`, `render`, `utils`. Their functions are * annotator -It is a GUI (graphical user interface) for pig pose tunning. See `Annotator.md` for more details. +It is a GUI (graphical user interface) for pig pose tunning. See `Demo.md` for more details. * articulation It contain the core code for how to pose the PIG model and how to optimize the poses according to observations. diff --git a/pics/BamaPig2D.jpg b/pics/BamaPig2D.jpg new file mode 100644 index 0000000..70b6992 Binary files /dev/null and b/pics/BamaPig2D.jpg differ diff --git a/pics/model.jpg b/pics/model.jpg new file mode 100644 index 0000000..fc44cc6 Binary files /dev/null and b/pics/model.jpg differ diff --git a/pics/teaser.png b/pics/teaser.png new file mode 100644 index 0000000..296e770 Binary files /dev/null and b/pics/teaser.png differ