AICommons - AI4Good Competition

Edge-Based Early Detection and Alerting Systems for Forest Guards - Team Lisa

Content

1. Repository Tree
2. Datasets Used
3. Datasets Generated
4. Notebooks
5. Bottleneck Features
6. Models
7. Mobile App Client
8. Video Demo
9. Team Members
10. References

---

Repository Tree

├───apk
	└───forest_guard_client.apk

├───labels_and_classescsv
	├───ESC-50_Mel-Spectogram_dataset_Meta_data.csv
	├───ESC-50_WavePlot_dataset_Meta_data.csv
	└───labels_and_target_classes(ESC50).csv

├───models
	├───ForestAI.tflite
	└───illegal_logging_classifier_model.h5

├───notebooks
	├───Dataset_exploration_and__Generation_by_Author_Appau_Ernest.ipynb
	├───iteration1_Building_a_CNN_model_to_classify_audio_events_from_the_ESC_50_WavePlot_dataset.ipynb
	└───Iteration2_Building_a_CNN_model_to_classify_audio_events_from_the_ESC_50_MelSpectogram_dataset.ipynb

├───pickle_files
	├───test_data.pkl
	├───test_labels.pkl
	├───train_labels.pkl
	├───train_data.pkl
	├───validation_labels.pkl
	└───validation_data.pkl
	
├───forest-guard-ai4good-firebase-adminsdk-1fw9p-232da17716.json

└───Testing_final_model_Team_Lisa.ipynb

Open Testing_final_notebook to make a copy of this notebook to run sample test in Colab. It contains an end to end pipeline with instructions to run inference on sound inputs.

Datasets Used

1. ESC-50 Dataset (Primary): https://github.com/karolpiczak/ESC-50
2. Sampled Audio For Testing (Secondary): https://research.google.com/audioset

Datasets Generated

1. ESC-50 waveplot/amplitude portfolio dataset
2. ESC-50 MelSpectogram portfolio dataset

Notebooks

1. Notebook 1: Open In Colab
   This notebook involves a feature engineering approach to generate two distinct datasets to help us in the modelling process in an attempt to solve the challenge of illegal logging activities in the forests. Generated Datasets include:

   • ESC-50 Waveplot dataset
   • ESC-50 MelSpectogram dataset

2. Notebook 2: Open In Colab
   The first iteration by building a base Convolutional Neural Network to classify audio waveplot/amplitude profile. Dataset used in this notebook includes the Generated ESC-50 waveplot profiles of the original audio files.
   Results from this iteration was a poor performing model with a high bias and variance between training and validation splits. In conclusion, we moved on to the melspectogram dataset to achieve a better performance after dozen hyperparameter tuning exercises.

3. Notebook 3: Open In Colab
   In this notebook, our model performed a bit better than the previous model in notebook 2 but also displayed a high variance during training process.
   After a dozen hyperparameter tuning and regularization, we concluded on the lack of our generated dataset not well feature-engineered enough to accommodate a generalized model.

4. Notebook 4: Open In Colab
   We adopted an advanced feature engineering approach by converting our audio files to a concatenated stack of 3 channels based on three main operations to generate feature maps. This includes splitting the 5 second audio clips into 5 folds increasing the dataset from 2000 to 10000 audio files. We combined three feature maps which include the Mel spectogram, log-scaled spectogram and the delta mel-scaled spectogram to form one feature map profile for an audio clip.

5. Notebook 5: Open In Colab
   Contains an end to end pipeline to run simulated test on the model. Currently it is performing averagely.

We then iterated over a number of pre-trained CNN to serve as feature extracts but amongst the lot, vgg16 performed very well on the ImageNet weights. Finally, we clipped off the Multi-Linear Perceptron classifier layer and built a sequential model with regularized dense and units to enable us train a multi-class classifier.

Bottleneck Features

To make this iteration reproducible, we have provided access to the bottle-necked features for the train, test and validation splits. We finally trained and tested our model which has a far better variance compared to the other approaches used in the first two iterations and notebooks. Per class accuracy metrics were evaluated on the model with the essential classes performing in a standard capacity. The bottle neck features extracted from the vgg16 model include :

• Train
  train_data.pkl
  train_labels.pkl

• Test
  test_data.pkl
  test_labels.pkl

• Validation
  validation_data.pkl
  validation_labels.pkl

Models

1. illegal_logging_classifier_model.h5 (Keras variant)
2. ForestAI.tflite (Tensorflow-Lite variant)

We provide a converted/optimized TFLite format as well as an h5 Keras format of our model. There is an inference pipeline script at the end of the notebook to enable one test the model on their audio files.

Mobile App Client

The android apk can be found in the apk/forest_guard_client.apk folder. You can click here to download from Play Store.
After installing the app, log in with AgentID: 1234 and Forest ID: 1.
When the prediction is sent to the database, deployed cloud functions trigger notifications for Human/Illegal (Chainsaw, etc) activity.
You can check the source code of the mobile app.

Video Demo

Click on the image below to play the video demo.

Team Members

Name	Role	Profile
Appau Ernest (Lead)	AI Model Development/Deployment/I-IOT	Github / LinkedIn
Debrah Kwesi Buabeng	Mobile Developer	Github / LinkedIn
Akpalu Larry	Data Collection and Annotation	Github / LinkedIn
Baidoo Mabel	Data Collection and Annotation	Github / LinkedIn
Kpene Godsway Edem	Documentation	Github / LinkedIn
Appau Roberta	UI/UX Designer	N/A

We would like to express our sincere gratitude to all members of Team Lisa as well as the mentors, host of this competition for being resourceful in our journey to seeing this through.

References

• Hands-On Mathematics for Deep Learning - Packt publishing
• Environmental Sound Classification With Convolutional Neural Networks - Karol Piczak, 2015
• Deep Learning - Ian Goodfellow and Yoshua Bengio
• Deep learning with Keras workshop - Packt Publishing
• TinyML - O'Reilly
• Keras
• Tensorflow
• Handouts by Superfluid Labs and AI4Good
• Mentors from Superfluid labs and AI4GOOD
• Google Search
• Wikipedia