notebooks/04_video_understanding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Video Undestanding\n",
    "\n",
    "This Jupiter notebook contains the code to process a video using [Amazon Nova models](https://docs.aws.amazon.com/nova/) to [video understanding](https://docs.aws.amazon.com/nova/latest/userguide/modalities-video.html). If the video is less than 25MB, it is converted to base64, and if it's larger, it is uploaded to an Amazon S3 bucket, which must be added as a variable in **you_bucket**. \n",
    "\n",
    "![Diagram](data/video_understanding.png)\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Nova family introduces five specialized models tailored to different use cases:\n",
    "- **Nova Pro:** A high-capability multimodal model balancing accuracy, speed, and cost\n",
    "- **Nova Lite:** A faster, cost-effective multimodal model optimized for processing various media\n",
    "types\n",
    "- **Nova Micro:** A text-only model delivering low-latency responses at minimal cost\n",
    "- **Nova Canvas:** An image generation model with professional-grade outputs and customization options\n",
    "- **Nova Reel:** A video generation model with quality outputs and motion control features\n",
    "\n",
    "> Using Nova pro and Nova Lite models you don't need to create a video embedding or store the video in a vector database, Amazon Nova does everything for you. Learn more about in [The Amazon Nova Family of Models:\n",
    "Technical Report and Model Card](https://assets.amazon.science/10/0a/0b61d39a4e9aaec16f71ad3d9168/the-amazon-nova-family-of-models-technical-report-and-model-card2-26.pdf) and find more code samples in [aws-samples repo](https://github.com/aws-samples/amazon-nova-samples/)\n",
    "\n",
    "![Diagram](./data/nova-models.png)\n",
    "\n",
    "The credentials used to run this notebook requires permission for the *bedrock:InvokeModel* action to invoke Amazon Nova models and to [*PutObjet* to S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html). \n",
    "\n",
    "Additionally, remember to activate the model in the console; follow the [Getting started with Amazon Bedrock steps.](https://docs.aws.amazon.com/bedrock/latest/userguide/getting-started.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requirements: \n",
    "- Install boto3 - This is the [AWS SDK for Python ](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingTheBotoAPI.html)that allows interacting with AWS services. Install with `pip install boto3`.\n",
    "- [Configure AWS credentials](https://docs.aws.amazon.com/braket/latest/developerguide/braket-using-boto3.html) - Boto3 needs credentials to make API calls to AWS.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install boto3\n",
    "#!pip install langchain\n",
    "#!pip install psycopg2-binary\n",
    "#!pip install langchain_experimental\n",
    "# or install requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3 # to interact with AWS services.\n",
    "import json\n",
    "import base64\n",
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "PRO_MODEL_ID = \"us.amazon.nova-pro-v1:0\"\n",
    "LITE_MODEL_ID = \"us.amazon.nova-lite-v1:0\"\n",
    "MICRO_MODEL_ID = \"us.amazon.nova-micro-v1:0\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bedrock_client              = boto3.client(\"bedrock-runtime\", region_name='us-east-1') \n",
    "boto3_bedrock               = boto3.client('bedrock', region_name='us-east-1')\n",
    "s3_client                   = boto3.client('s3')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validate the connection\n",
    "We can check the client works by trying out the list_foundation_models() method, which will tell us all the models available for us to use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[models['modelId'] for models in boto3_bedrock.list_foundation_models()['modelSummaries']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def handle_video(video_path,prompt):\n",
    "    # Convert 64MB to bytes\n",
    "    SIZE_THRESHOLD = 64 * 1024 * 1024  # 64MB in bytes\n",
    "    \n",
    "    file_size = Path(video_path).stat().st_size\n",
    "    file_name = os.path.basename(video_path)\n",
    "    content_type = file_name.lower().split('.')[-1]\n",
    "    \n",
    "    try:\n",
    "        if file_size <= SIZE_THRESHOLD:\n",
    "            # For files under 64MB, create base64\n",
    "            print(f\"Video size: {file_size / (1024 * 1024):.2f} MB - Creating base64 - content type: {content_type}\")\n",
    "            with open(video_path, \"rb\") as file:\n",
    "                media_bytes = file.read()\n",
    "                \n",
    "\n",
    "            messages = [\n",
    "                        {\n",
    "                            \"role\": \"user\",\n",
    "                            \"content\": [\n",
    "                                {\"video\": {\"format\": \"mp4\", \"source\": {\"bytes\": media_bytes}}},\n",
    "                                {\"text\": prompt},\n",
    "                            ],\n",
    "                        }\n",
    "                            ]\n",
    "                \n",
    "        else:\n",
    "            # For files over 64MB, upload to S3\n",
    "            print(f\"Video size: {file_size / (1024 * 1024):.2f} MB - Uploading to S3\")\n",
    "            \n",
    "            bucket_name = s3_uri.split('/')[2]\n",
    "            media_uri = f'{s3_uri}/{file_name}'\n",
    "\n",
    "            with open(video_path, \"rb\") as data:\n",
    "                s3_client.upload_fileobj(data,bucket_name, media_uri)\n",
    "            print(\"Put file in s3://{}{}{}\".format(bucket_name,s3_uri,file_name))\n",
    "\n",
    "            #put objet\n",
    "            s3_client.put_object(\n",
    "                Bucket=bucket_name,\n",
    "                Key=file_name,\n",
    "                Body=open(video_path, 'rb'),\n",
    "                ContentType=f'video/{content_type}'\n",
    "            )\n",
    "\n",
    "            messages = [\n",
    "                            {\n",
    "                                \"role\": \"user\",\n",
    "                                \"content\": [\n",
    "                                    {\n",
    "                                        \"video\": {\n",
    "                                            \"format\": content_type,\n",
    "                                            \"source\": {\n",
    "                                                \"s3Location\": {\n",
    "                                                    #Replace the s3 bucket URI \n",
    "                                                    \"uri\": media_uri\n",
    "                                                }\n",
    "                                            },\n",
    "                                        }\n",
    "                                    },\n",
    "                                    {\"text\": prompt},\n",
    "                                ],\n",
    "                            }\n",
    "                        ]\n",
    "\n",
    "        inf_params = {\"maxTokens\": 300, \"topP\": 0.1, \"temperature\": 0.3}\n",
    "\n",
    "        model_response = bedrock_client.converse(\n",
    "                modelId=LITE_MODEL_ID, messages=messages, inferenceConfig=inf_params\n",
    "            )\n",
    "\n",
    "        #print(\"\\n[Full Response]\")\n",
    "        #print(json.dumps(model_response, indent=2))\n",
    "\n",
    "        print(\"\\n[Response Content Text]\")\n",
    "        response = model_response[\"output\"][\"message\"][\"content\"][0][\"text\"]\n",
    "        print(response)\n",
    "    \n",
    "            \n",
    "    except Exception as e:\n",
    "        print(f\"Error processing video: {str(e)}\")\n",
    "        raise\n",
    "\n",
    "      \n",
    "    return response\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "video_path = \"demo-files/moderation-video.mp4\"\n",
    "prompt = \"Describe the following video\"\n",
    "response = handle_video(video_path,prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How Amazon Nova Pro and Lite Understands Video \n",
    "When a video is submitted to Nova Pro or Lite, the system processes it as a sequence of frames sampled from the video. These frames are processed through the model’s vision encoders, which transform the visual information into embeddings that can be processed by the transformer architecture. \n",
    "\n",
    "For videos less than or equal to 16 minutes in duration, a 1 frame per second (FPS) sampling rate is employed.\n",
    "\n",
    "However, for videos exceeding 16 minutes in length, the sampling rate decreases in order to maintain a consistent 960 frames sampled, with the frame sampling rate varying accordingly. \n",
    "\n",
    "This approach is designed to provide more accurate scene-level video understanding for shorter videos compared to longer video content. \n",
    "\n",
    "Learn more in [Video size information page](https://docs.aws.amazon.com/nova/latest/userguide/modalities-video.html#modalities-video-size)\n",
    "\n",
    "> We recommend that you keep the video length less than 1 hour for low motion, and less than 16 minutes for anything with higher motion.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}