demo/STS_STT_Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1da4e8f1-3054-423d-88cf-c2742cd419e6",
   "metadata": {},
   "source": [
    "# Sign-to-Sign/Sign-to-Text Service Demo\n",
    "\n",
    "## This demo will showcase how to use the One Touch Sign-to-Sign & Sign-to-Text to enable sign language translation into both text and another sign language from a different country."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9f1b331-4031-430a-a75d-71609743830b",
   "metadata": {},
   "source": [
    "<img src=\"../images/One_Touch_MVP_Architecture_Diagram.jpg\" alt=\"architecture\" width=\"80%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ac571ff-d567-4534-9b99-ede1c48b41ce",
   "metadata": {},
   "source": [
    "## For this demo, we will focus on interpreting American Sign Language (ASL) into Chinese Sign Language (CSL) and Spanish Sign Language (SSL)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fbbae51-b4be-489a-896d-256ba98612a7",
   "metadata": {},
   "source": [
    "### Import required libaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f62c10b3-31bf-4a40-9f28-bbc275fb6a6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import os\n",
    "from IPython.display import Video"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b953c6f-f795-425b-b29b-ac760c0aef8c",
   "metadata": {},
   "source": [
    "## STS/STT Parameters\n",
    "### There are four input parameters for the service:\n",
    "- **source** - the source language for translation (asl)\n",
    "- **target** - the target language for translation (ssl | csl) \n",
    "- **output_format** - the desired output format for the translation (sign_lanague | text | speech |)\n",
    "- **video_uri** - the uri of a jsonl file in cloud object storage containing path and info of a video containing sign lanuguage\n",
    "\n",
    "For this demo, we provide a uri of a person signing \"hello\" and \"how are you\" in ASL \n",
    "- \"gs://asl_signed_videos/hello_video_prediction.jsonl\"\n",
    "- \"gs://asl_signed_videos/how_are_you_video_prediction.jsonl\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdc355bc-3845-4a40-8e1c-116295307b07",
   "metadata": {},
   "source": [
    "# Enable Sign Language to Sign Language translation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4acbbbd-274a-40c5-b1e0-2b6f3923e9d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "payload = {'source': 'asl', 'target': 'ssl', 'output_format': 'sign_language', 'video_uri': 'gs://asl_signed_videos/hello_video_prediction.jsonl'}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8409521-2afd-40b1-b600-fb047b9c415e",
   "metadata": {},
   "source": [
    "## Call the `sign_translation` endpoint to begin translation from ASL to SSL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48e4652d-710e-4f76-8986-c3a1484454a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'http://localhost:5000/sign_translation'\n",
    "\n",
    "r = requests.post(url, json=payload)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71b5b40a-1d9e-4303-a6ad-fb2e155ad494",
   "metadata": {},
   "source": [
    "## Get the translated sign language output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0e2642f-232c-47e0-ad6a-45daf19cd0f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('output.mp4', 'wb') as f:\n",
    "    for chunk in r.iter_content(chunk_size=128):\n",
    "        f.write(chunk)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6588f1da-a9af-4c80-8c16-3c1d8f11affa",
   "metadata": {},
   "outputs": [],
   "source": [
    "Video(\"output.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cb9b04d-a951-45e4-9934-de5511d716fb",
   "metadata": {},
   "source": [
    "## Change the above payload `target` parameter to `csl` and rerun to see the CSL intrepretation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c70cca2f-bf2c-4db1-a9e4-d10290249aee",
   "metadata": {},
   "source": [
    "# Now let's enable Sign language to Speech\n",
    "### modify the output format to `speech`. You should hear the translation in the target language, but the bytes to save as an audio file are returned from the service as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02503013-d049-4c11-b3bc-10b3100a4d2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "payload = {'source': 'asl', 'target': 'csl', 'output_format': 'speech', 'video_uri': 'gs://asl_signed_videos/how_are_you_video_prediction.jsonl'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fd9e73f-af84-429f-8c25-75759b6a2b9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "r = requests.post(url, json=payload)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2eaf54ec-c791-4646-a6b2-2585a9deea1e",
   "metadata": {},
   "source": [
    "# Finally, enable Sign Language to Text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "648e172d-22f4-4b81-b4a8-54dce00c2b9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "payload = {'source': 'asl', 'target': 'ssl', 'output_format': 'text','video_uri': 'gs://asl_signed_videos/how_are_you_video_prediction.jsonl'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71ae82be-e703-4ec5-b8e4-5335a96a4492",
   "metadata": {},
   "outputs": [],
   "source": [
    "r = requests.post(url, json=payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1543a585-b218-4122-a731-90a5f275bedd",
   "metadata": {},
   "outputs": [],
   "source": [
    "r.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd16cbb9-eefd-40e4-b7d5-b3cd3218e182",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "hackathon",
   "language": "python",
   "name": "hackathon"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}