Face recognition PR

community_projects/face_recognition/README.md

@@ -0,0 +1,167 @@
+# Face Recognition System
+
+This project is a face recognition system built using Python, Flask, and LanceDB. It provides a web interface for managing face recognition data, including adding, updating, and deleting persons and their associated images. The system also supports real-time face recognition using GStreamer pipelines.
+
+---
+
+## Features
+
+- **Web Interface**: Manage persons, images, and thresholds via a user-friendly web interface.
+- **Real-Time Face Recognition**: Detect and recognize faces in real-time using GStreamer pipelines.
+- **Database Management**: Store and manage face embeddings and metadata using LanceDB.
+- **Telegram Notifications**: Send alerts for detected faces via Telegram.
+- **Visualization**: Visualize embeddings and confidence circles in a 2D plot.
+
+---
+
+## Prerequisites
+
+- Python 3.8+
+- Pipenv or virtualenv for dependency management
+- Required Python libraries (see `requirements.txt`)
+- GStreamer installed on the system
+- LanceDB installed for database management
+
+---
+
+## Installation
+
+1. Clone the repository:
+   ```bash
+   git clone https://github.com/your-repo/face-recognition.git
+   cd face-recognition
+   ```
+
+2. Install dependencies:
+   ```bash
+   pip install -r requirements.txt
+   ```
+
+3. Install GStreamer:
+   ```bash
+   sudo apt-get install gstreamer1.0-tools gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly
+   ```
+
+4. Set up the database:
+   ```bash
+   python -c "from db_handler import init_database; init_database()"
+   ```
+
+5. Download resources:
+   ```bash
+   chmod +x download_resources.sh
+   ./download_resources.sh --all
+   ```
+
+---
+
+## Usage
+
+### Run the Web Application
+
+1. Start the Flask web server:
+   ```bash
+   python web/web_app.py
+   ```
+
+2. Open your browser and navigate to:
+   ```
+   http://localhost:5002
+   ```
+
+### Run the Face Recognition Pipeline
+
+1. Start the GStreamer pipeline:
+   ```bash
+   python app_db.py
+   ```
+
+2. Choose the mode:
+   - `run`: Real-time face recognition
+   - `run-save`: Save new faces detected
+   - `train`: Train the system with new images
+   - `delete`: Clear the database
+
+---
+
+## Web Interface
+
+### Features
+
+- **View Persons**: Displays all persons in the database with their images and metadata.
+- **Update Name**: Modify the name of a person.
+- **Delete Image**: Remove a specific image associated with a person.
+- **Update Threshold**: Adjust the classification confidence threshold for a person.
+- **Delete Person**: Remove a person and all their associated data.
+
+---
+
+## Telegram Notifications
+
+- Configure the `TELEGRAM_TOKEN` and `TELEGRAM_CHAT_ID` in `app_db.py` to enable Telegram notifications.
+- Notifications are sent when a face is detected, with an image and confidence score.
+
+---
+
+## File Structure
+
+```
+face-recognition/
+├── web/
+│   ├── templates/
+│   │   └── index.html       # Web interface template
+│   ├── web_app.py           # Flask web application
+├── resources/
+│   ├── train/               # Training images
+│   ├── faces/               # Detected face images
+│   ├── database/            # LanceDB database
+├── db_handler.py            # Database management
+├── app_db.py                # GStreamer pipeline and Telegram integration
+├── face_recognition_pipeline_db.py # GStreamer face recognition pipeline
+├── download_resources.sh    # Script to download resources
+└── README.md                # Documentation
+```
+
+---
+
+## API Endpoints
+
+### `GET /`
+- **Description**: Displays the web interface.
+- **Response**: HTML page with persons and their data.
+
+### `POST /delete_image`
+- **Description**: Deletes a specific image of a person.
+- **Parameters**:
+  - `global_id`: The person's global ID.
+  - `face_id`: The ID of the face to delete.
+
+### `POST /update_person_name`
+- **Description**: Updates the name of a person.
+- **Parameters**:
+  - `global_id`: The person's global ID.
+  - `new_name`: The new name.
+
+### `POST /update_person_threshold`
+- **Description**: Updates the classification confidence threshold for a person.
+- **Parameters**:
+  - `global_id`: The person's global ID.
+  - `new_threshold`: The new threshold value.
+
+---
+
+## Troubleshooting
+
+- **Database Issues**: Ensure the database is initialized using `init_database()`.
+- **GStreamer Errors**: Verify GStreamer is installed and configured correctly.
+- **Telegram Notifications**: Check the `TELEGRAM_TOKEN` and `TELEGRAM_CHAT_ID` values.
+
+---
+
+## Acknowledgments
+
+- [Flask](https://flask.palletsprojects.com/)
+- [GStreamer](https://gstreamer.freedesktop.org/)
+- [LanceDB](https://lancedb.github.io/)
+- [Telegram Bot API](https://core.telegram.org/bots/api)
+```

community_projects/face_recognition/app_db.py

@@ -0,0 +1,86 @@
+# region imports
+import datetime
+from datetime import datetime, timedelta
+from io import BytesIO
+
+import gi
+gi.require_version('Gst', '1.0')
+from gi.repository import Gst
+from PIL import Image
+import telebot
+
+import hailo
+from hailo_apps_infra.hailo_rpi_common import app_callback_class
+from face_recognition_pipeline_db import GStreamerFaceRecognitionApp
+from db_handler import clear_table, init_database as db_init
+# endregion
+
+TELEGRAM_TOKEN = '7544346062:AAFSvYjJlvlby-rmJoUF3sWoXQh-7dxj2RY'
+TELEGRAM_CHAT_ID = '7520285462'
+
+class user_callbacks_class(app_callback_class):
+    def __init__(self):
+        super().__init__()
+        self.ids_msg_sent = {}  # Dictionary to store the last sent time for each person
+        if TELEGRAM_TOKEN and TELEGRAM_CHAT_ID:
+            self.bot = telebot.TeleBot(TELEGRAM_TOKEN)
+            self.chat_id = TELEGRAM_CHAT_ID
+
+    def send_notification(self, name, global_id, distance, frame):
+        if not TELEGRAM_TOKEN or not TELEGRAM_CHAT_ID:
+            return
+        caption = None  # Initialize caption with a default value
+        current_time = datetime.now()
+        last_sent_time = self.ids_msg_sent.get(global_id)
+        if last_sent_time is None or current_time - last_sent_time > timedelta(hours=1):  # Check if the notification was sent more than an hour ago or never sent
+            if not name:
+                caption = "🚨 Unknown person detected!"  # For Unknown person there is no classification hence no classification confidence
+            else:
+                if name == 'Unknown':
+                    caption = f"Detected {global_id} (confidence: {(1 - distance):.2f})"
+                else:
+                    caption = f"Detected {name} (confidence: {(1 - distance):.2f})"
+        if caption:
+            self.ids_msg_sent[global_id] = current_time  # Update the last sent time
+            image = Image.fromarray(frame)  # Open the image from the file path
+            image_byte_array = BytesIO()  # Save the image to a byte stream
+            image.save(image_byte_array, format='PNG')
+            image_byte_array.seek(0)
+            try:
+                self.bot.send_photo(self.chat_id, image_byte_array, caption)
+            except Exception as e:
+                print(f'Error sending Telegram notification: {str(e)}')
+
+def app_callback(pad, info, user_data):
+    buffer = info.get_buffer()
+    if buffer is None:
+        return Gst.PadProbeReturn.OK
+    user_data.increment()
+    string_to_print = f'Frame count: {user_data.get_count()}\n'
+    roi = hailo.get_roi_from_buffer(buffer)
+    detections = roi.get_objects_typed(hailo.HAILO_DETECTION)
+    for detection in detections:
+        label = detection.get_label()
+        detection_confidence = detection.get_confidence()
+        if label == "face":
+            track_id = 0
+            track = detection.get_objects_typed(hailo.HAILO_UNIQUE_ID)
+            if len(track) > 0:
+                track_id = track[0].get_id()
+            string_to_print += f'Detection track ID: {track_id} (Confidence: {detection_confidence:.1f})\n'
+            classifications = detection.get_objects_typed(hailo.HAILO_CLASSIFICATION)
+            if len(classifications) > 0:
+                for classification in classifications:
+                    string_to_print += f'Classification: {classification.get_label()} (Confidence: {classification.get_confidence():.1f})'
+    # print(string_to_print)
+    return Gst.PadProbeReturn.OK
+
+if __name__ == "__main__":    
+    user_data = user_callbacks_class()
+    app = GStreamerFaceRecognitionApp(app_callback, user_data)
+    if app.options_menu.mode == 'delete':
+        db, tbl_persons = db_init()
+        clear_table()
+        print("All records deleted from the database")
+    else:  # run, run-save, train
+        app.run()

community_projects/face_recognition/compile_postprocess.sh

@@ -0,0 +1,60 @@
+#!/bin/bash
+
+# Set the project directory name
+PROJECT_DIR="."
+
+# Enable strict error handling
+set -e
+
+# Check if Meson and Ninja are installed
+if ! command -v meson &> /dev/null; then
+    echo "Error: Meson is not installed. Please install it and try again."
+    exit 1
+fi
+
+if ! command -v ninja &> /dev/null; then
+    echo "Error: Ninja is not installed. Please install it and try again."
+    exit 1
+fi
+
+# Get the build mode from the command line (default to release)
+if [ "$1" = "debug" ]; then
+    BUILD_MODE="debug"
+elif [ "$1" = "clean" ]; then
+    BUILD_MODE="release"  # Default to release for cleanup
+    CLEAN=true
+else
+    BUILD_MODE="release"
+fi
+
+# Set up the build directory
+BUILD_DIR="$PROJECT_DIR/build.$BUILD_MODE"
+
+# Handle cleanup
+if [ "$CLEAN" = true ]; then
+    echo "Cleaning build directory..."
+    rm -rf "$BUILD_DIR"
+    exit 0
+fi
+
+# Create the build directory
+mkdir -p "$BUILD_DIR"
+cd "$BUILD_DIR"
+
+# Configure the project with Meson if not already configured
+if [ ! -f "build.ninja" ]; then
+    echo "Configuring project with Meson..."
+    meson setup .. --buildtype="$BUILD_MODE"
+else
+    echo "Build directory already configured. Skipping setup."
+fi
+
+# Compile the project using Ninja with parallel jobs
+echo "Building project with Ninja..."
+ninja -j$(nproc)
+
+# Install the project (optional)
+echo "Installing project..."
+ninja install
+
+echo "Build completed successfully!"

community_projects/face_recognition/cpp/arcface.cpp

@@ -0,0 +1,73 @@
+/**
+ * Copyright (c) 2021-2022 Hailo Technologies Ltd. All rights reserved.
+ * Distributed under the LGPL license (https://www.gnu.org/licenses/old-licenses/lgpl-2.1.txt)
+ **/
+#include <vector>
+#include "common/tensors.hpp"
+#include "common/math.hpp"
+#include "arcface.hpp"
+#include "hailo_tracker.hpp"
+#include "hailo_xtensor.hpp"
+#include "xtensor/xadapt.hpp"
+#include "xtensor/xarray.hpp"
+
+#define OUTPUT_LAYER_NAME_RGB "arcface_mobilefacenet/fc1"
+#define OUTPUT_LAYER_NAME_RGBA "arcface_mobilefacenet_rgbx/fc1"
+#define OUTPUT_LAYER_NAME_NV12 "arcface_mobilefacenet/fc1"
+
+std::string tracker_name = "hailo_face_tracker";
+
+void arcface(HailoROIPtr roi, std::string layer_name)
+{
+    if (!roi->has_tensors())
+    {
+        return;
+    }
+
+    std::string jde_tracker_name = tracker_name + "_" + roi->get_stream_id();
+    auto unique_ids = hailo_common::get_hailo_track_id(roi);
+    // Remove previous matrices
+    if(unique_ids.empty())
+        roi->remove_objects_typed(HAILO_MATRIX);
+    else
+        HailoTracker::GetInstance().remove_matrices_from_track(jde_tracker_name, unique_ids[0]->get_id());
+    // Convert the tensor to xarray.
+    auto tensor = roi->get_tensor(layer_name);
+    xt::xarray<float> embeddings = common::get_xtensor_float(tensor);
+
+    // vector normalization
+    auto normalized_embedding = common::vector_normalization(embeddings);
+
+    HailoMatrixPtr hailo_matrix = hailo_common::create_matrix_ptr(normalized_embedding);
+    if(unique_ids.empty())
+    {
+        roi->add_object(hailo_matrix);
+    }
+    else
+    {
+        // Update the tracker with the results
+        HailoTracker::GetInstance().add_object_to_track(jde_tracker_name,
+                                                        unique_ids[0]->get_id(),
+                                                        hailo_matrix);
+    }
+}
+
+void arcface_rgb(HailoROIPtr roi)
+{
+    arcface(roi, OUTPUT_LAYER_NAME_RGB);
+}
+
+void arcface_rgba(HailoROIPtr roi)
+{
+    arcface(roi, OUTPUT_LAYER_NAME_RGBA);
+}
+
+void arcface_nv12(HailoROIPtr roi)
+{
+    arcface(roi, OUTPUT_LAYER_NAME_NV12);
+}
+
+void filter(HailoROIPtr roi)
+{
+    arcface(roi, OUTPUT_LAYER_NAME_RGB);
+}