{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":10019986,"sourceType":"datasetVersion","datasetId":6141303}],"dockerImageVersionId":30786,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2024-11-22T17:12:15.493189Z","iopub.execute_input":"2024-11-22T17:12:15.493613Z","iopub.status.idle":"2024-11-22T17:12:15.517076Z","shell.execute_reply.started":"2024-11-22T17:12:15.493574Z","shell.execute_reply":"2024-11-22T17:12:15.515865Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# AI DocParser\n\nAn application framework developed using the latest AI technologies to extract the values of specific pre-defined keys from a given PDF document. Also generating a document summary using the key & values extracted in the while doing so. \n\n**Application Input:**\n\nA set of PDF documents, the domain of the document, and a set of field names (also called keys) for which the values need to be extracted.\n\n**Application output:**\n\nThe values against the keys in a document extracted and stored in a CSV file with key & value columns.\nSummary of the document","metadata":{}},{"cell_type":"markdown","source":"## Our Example PDF\n\nWe will use the recent contracts happened between Disney and Reliance.\n\n### Keys to extract:\n\nName of the 1st Party\n\nName of the 2nd Party\n\nData of announcement\n\nInvestment amount\n\nTransaction value\n\nExclusive rights","metadata":{}},{"cell_type":"markdown","source":"# Setup","metadata":{}},{"cell_type":"markdown","source":"### PDF file path","metadata":{}},{"cell_type":"code","source":"pdf_path = '/kaggle/input/reliance-hotstar-ocr/reliance_disney_agreement.pdf'","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T13:55:55.539104Z","iopub.execute_input":"2024-11-26T13:55:55.539491Z","iopub.status.idle":"2024-11-26T13:55:55.545015Z","shell.execute_reply.started":"2024-11-26T13:55:55.539457Z","shell.execute_reply":"2024-11-26T13:55:55.543509Z"}},"outputs":[],"execution_count":3},{"cell_type":"markdown","source":"### Pretty table for printing key-value like dictionary pairs","metadata":{}},{"cell_type":"code","source":"from prettytable import PrettyTable \ndef print_pretty_table(extracted_values):\n \"\"\"\n Prints the extracted key-value pairs in a formatted table.\n\n Args:\n extracted_values (dict): Dictionary of extracted key-value pairs.\n \"\"\"\n # Create a PrettyTable instance\n table = PrettyTable()\n\n # Set column names\n table.field_names = [\"Key\", \"Value\"]\n\n # Add rows\n for key, value in extracted_values.items():\n table.add_row([key, value])\n\n # Print the table\n print(table)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T13:55:59.865961Z","iopub.execute_input":"2024-11-26T13:55:59.866361Z","iopub.status.idle":"2024-11-26T13:55:59.911655Z","shell.execute_reply.started":"2024-11-26T13:55:59.866327Z","shell.execute_reply":"2024-11-26T13:55:59.910489Z"}},"outputs":[],"execution_count":4},{"cell_type":"markdown","source":"# Step 1: Data Processing and Preprocessing\n\n**What we are doing**\n\n* Extract text content from PDFs.\n* Clean and preprocess text (remove headers, footers, and noise).\n* Normalize text to ensure uniformity (e.g., removing special characters, handling encoding issues).","metadata":{}},{"cell_type":"markdown","source":"## 1.1 Extract text content from PDFs\n\n### Option 1 - PyPDF2, for simple PDF extraction\n- **Lightweight** and easy to use for basic text extraction.\n- Works well for simple PDFs with **linear text** (no complex layouts, images, or tables).\n- Stable and actively maintained.\n\n### Option 2 - pdfplumber, for handling layouts better\n- Excellent for extracting text from PDFs with **complex layouts**.\n- Can extract tables, images, and even coordinates of text within a page.\n- Provides a **high degree of customization**, such as selecting specific page areas or identifying text within bounding boxes.\n- Supports well-structured text extraction for scanned documents (if OCR is pre-applied).\n\n### Option 3 - PyMuPDF library (via fitz), \n- Extremely fast and efficient.\n- Extracts text, images, and metadata with high accuracy.\n- **Handles multi-column PDFs**, tables, and even rotated text better than PyPDF2.\n- Can extract text from specific regions of a page (bounding box selection).\n- **Supports embedded fonts and handles PDFs with complex structures more robustly**.\n\n\n\n### Let's opt for option 3 as default, but we can use 1 and 2 by changing argument","metadata":{}},{"cell_type":"code","source":"!pip install PyPDF2 PyMuPDF pdfplumber","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T13:56:15.558851Z","iopub.execute_input":"2024-11-26T13:56:15.559249Z","iopub.status.idle":"2024-11-26T13:56:32.949171Z","shell.execute_reply.started":"2024-11-26T13:56:15.559216Z","shell.execute_reply":"2024-11-26T13:56:32.947182Z"}},"outputs":[{"name":"stdout","text":"Collecting PyPDF2\n Downloading pypdf2-3.0.1-py3-none-any.whl.metadata (6.8 kB)\nCollecting PyMuPDF\n Downloading PyMuPDF-1.24.14-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (3.4 kB)\nCollecting pdfplumber\n Downloading pdfplumber-0.11.4-py3-none-any.whl.metadata (41 kB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.0/42.0 kB\u001b[0m \u001b[31m1.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hCollecting pdfminer.six==20231228 (from pdfplumber)\n Downloading pdfminer.six-20231228-py3-none-any.whl.metadata (4.2 kB)\nRequirement already satisfied: Pillow>=9.1 in /opt/conda/lib/python3.10/site-packages (from pdfplumber) (10.3.0)\nCollecting pypdfium2>=4.18.0 (from pdfplumber)\n Downloading pypdfium2-4.30.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (48 kB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m48.5/48.5 kB\u001b[0m \u001b[31m1.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hRequirement already satisfied: charset-normalizer>=2.0.0 in /opt/conda/lib/python3.10/site-packages (from pdfminer.six==20231228->pdfplumber) (3.3.2)\nRequirement already satisfied: cryptography>=36.0.0 in /opt/conda/lib/python3.10/site-packages (from pdfminer.six==20231228->pdfplumber) (42.0.8)\nRequirement already satisfied: cffi>=1.12 in /opt/conda/lib/python3.10/site-packages (from cryptography>=36.0.0->pdfminer.six==20231228->pdfplumber) (1.16.0)\nRequirement already satisfied: pycparser in /opt/conda/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=36.0.0->pdfminer.six==20231228->pdfplumber) (2.22)\nDownloading pypdf2-3.0.1-py3-none-any.whl (232 kB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m232.6/232.6 kB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mta \u001b[36m0:00:01\u001b[0m\n\u001b[?25hDownloading PyMuPDF-1.24.14-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (19.8 MB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m19.8/19.8 MB\u001b[0m \u001b[31m55.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n\u001b[?25hDownloading pdfplumber-0.11.4-py3-none-any.whl (59 kB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m59.2/59.2 kB\u001b[0m \u001b[31m1.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n\u001b[?25hDownloading pdfminer.six-20231228-py3-none-any.whl (5.6 MB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.6/5.6 MB\u001b[0m \u001b[31m67.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n\u001b[?25hDownloading pypdfium2-4.30.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.8 MB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.8/2.8 MB\u001b[0m \u001b[31m50.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n\u001b[?25hInstalling collected packages: pypdfium2, PyPDF2, PyMuPDF, pdfminer.six, pdfplumber\nSuccessfully installed PyMuPDF-1.24.14 PyPDF2-3.0.1 pdfminer.six-20231228 pdfplumber-0.11.4 pypdfium2-4.30.0\n","output_type":"stream"}],"execution_count":5},{"cell_type":"code","source":"import fitz # PyMuPDF\nimport pdfplumber\nfrom PyPDF2 import PdfReader\nimport re\n\ndef extract_text_from_pdf(pdf_path, method=\"fitz\"):\n \"\"\"\n Extracts text from a PDF using the specified library (fitz, pdfplumber, or PyPDF2).\n\n Args:\n pdf_path (str): Path to the PDF file.\n method (str): The library to use for extraction ('fitz', 'pdfplumber', 'pypdf2').\n Defaults to 'fitz' (PyMuPDF).\n\n Returns:\n str: Extracted and cleaned text from the PDF.\n \"\"\"\n try:\n raw_text = \"\"\n \n # pdfplumber method\n if method == \"pdfplumber\":\n with pdfplumber.open(pdf_path) as pdf:\n for page in pdf.pages:\n raw_text += page.extract_text() or \"\"\n\n # PyPDF2 method\n elif method == \"pypdf2\":\n reader = PdfReader(pdf_path)\n for page in reader.pages:\n raw_text += page.extract_text() or \"\"\n\n # PyMuPDF (fitz) method\n else:\n document = fitz.open(pdf_path)\n for page_num in range(len(document)):\n page = document[page_num]\n raw_text += page.get_text()\n document.close()\n\n if not raw_text.strip():\n raise ValueError(\"No extractable text found in the PDF.\")\n else:\n return raw_text\n\n except Exception as e:\n return f\"Error processing the PDF with {method}: {str(e)}\"","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T13:56:49.195477Z","iopub.execute_input":"2024-11-26T13:56:49.195968Z","iopub.status.idle":"2024-11-26T13:56:49.804367Z","shell.execute_reply.started":"2024-11-26T13:56:49.195913Z","shell.execute_reply":"2024-11-26T13:56:49.802898Z"}},"outputs":[],"execution_count":6},{"cell_type":"markdown","source":"## 1.2 Accuracy check","metadata":{}},{"cell_type":"code","source":"file_path = '/kaggle/input/reliance-hotstar-ocr/reliance_disney_ocr.txt'","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T13:57:26.550392Z","iopub.execute_input":"2024-11-26T13:57:26.550806Z","iopub.status.idle":"2024-11-26T13:57:26.555722Z","shell.execute_reply.started":"2024-11-26T13:57:26.550770Z","shell.execute_reply":"2024-11-26T13:57:26.554456Z"}},"outputs":[],"execution_count":8},{"cell_type":"markdown","source":"**Loading the OCR file text**","metadata":{}},{"cell_type":"code","source":"# Function to read ground truth from a file\ndef read_ground_truth(file_path):\n try:\n with open(file_path, 'r') as file:\n ground_truth = file.read().strip() \n return ground_truth\n except FileNotFoundError:\n print(f\"Error: The file at '{file_path}' was not found.\")\n return None\n except Exception as e:\n print(f\"An error occurred while reading the file: {e}\")\n return None","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T13:57:39.434364Z","iopub.execute_input":"2024-11-26T13:57:39.434907Z","iopub.status.idle":"2024-11-26T13:57:39.441244Z","shell.execute_reply.started":"2024-11-26T13:57:39.434871Z","shell.execute_reply":"2024-11-26T13:57:39.440035Z"}},"outputs":[],"execution_count":9},{"cell_type":"markdown","source":"**Preprocessing text before checking accuracy**","metadata":{}},{"cell_type":"code","source":"import re\n\ndef preprocess_text(text):\n \"\"\"\n Preprocesses the input text by:\n - Removing extra whitespaces\n - Normalizing newlines\n - Converting to lowercase\n - Optional: Removing punctuation or stopwords\n \"\"\"\n # Remove extra spaces and newlines\n text = re.sub(r'\\s+', ' ', text).strip() # Replace multiple spaces and trim\n text = text.replace('\\n', ' ') # Replace newlines with a space\n\n # Convert to lowercase\n text = text.lower()\n\n # Optional: Remove punctuation (if needed)\n # text = re.sub(r'[^\\w\\s]', '', text) # Uncomment if punctuation removal is desired\n\n return text","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T13:58:11.380790Z","iopub.execute_input":"2024-11-26T13:58:11.381177Z","iopub.status.idle":"2024-11-26T13:58:11.387157Z","shell.execute_reply.started":"2024-11-26T13:58:11.381143Z","shell.execute_reply":"2024-11-26T13:58:11.385934Z"}},"outputs":[],"execution_count":10},{"cell_type":"markdown","source":"**Word accuracy code**","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import precision_score, recall_score, f1_score\n\ndef word_match_accuracy(extracted_text, ground_truth):\n # Preprocess both texts\n extracted_text = preprocess_text(extracted_text)\n ground_truth = preprocess_text(ground_truth)\n\n # Convert the texts into sets of words\n extracted_words = set(extracted_text.split())\n ground_truth_words = set(ground_truth.split())\n\n # Calculate precision, recall, and F1-score\n intersection = len(extracted_words & ground_truth_words)\n precision = intersection / len(extracted_words) if len(extracted_words) > 0 else 0\n recall = intersection / len(ground_truth_words) if len(ground_truth_words) > 0 else 0\n f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0\n \n return precision, recall, f1","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T13:58:15.988438Z","iopub.execute_input":"2024-11-26T13:58:15.988870Z","iopub.status.idle":"2024-11-26T13:58:16.744432Z","shell.execute_reply.started":"2024-11-26T13:58:15.988831Z","shell.execute_reply":"2024-11-26T13:58:16.743103Z"}},"outputs":[],"execution_count":11},{"cell_type":"markdown","source":"**Similiarity check code**","metadata":{}},{"cell_type":"code","source":"pip install python-Levenshtein","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T13:58:20.371878Z","iopub.execute_input":"2024-11-26T13:58:20.372445Z","iopub.status.idle":"2024-11-26T13:58:33.043167Z","shell.execute_reply.started":"2024-11-26T13:58:20.372409Z","shell.execute_reply":"2024-11-26T13:58:33.041614Z"}},"outputs":[{"name":"stdout","text":"Collecting python-Levenshtein\n Downloading python_Levenshtein-0.26.1-py3-none-any.whl.metadata (3.7 kB)\nCollecting Levenshtein==0.26.1 (from python-Levenshtein)\n Downloading levenshtein-0.26.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.2 kB)\nCollecting rapidfuzz<4.0.0,>=3.9.0 (from Levenshtein==0.26.1->python-Levenshtein)\n Downloading rapidfuzz-3.10.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)\nDownloading python_Levenshtein-0.26.1-py3-none-any.whl (9.4 kB)\nDownloading levenshtein-0.26.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (162 kB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m162.6/162.6 kB\u001b[0m \u001b[31m3.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n\u001b[?25hDownloading rapidfuzz-3.10.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.1/3.1 MB\u001b[0m \u001b[31m37.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n\u001b[?25hInstalling collected packages: rapidfuzz, Levenshtein, python-Levenshtein\nSuccessfully installed Levenshtein-0.26.1 python-Levenshtein-0.26.1 rapidfuzz-3.10.1\nNote: you may need to restart the kernel to use updated packages.\n","output_type":"stream"}],"execution_count":12},{"cell_type":"code","source":"import Levenshtein\n\ndef levenshtein_accuracy(extracted_text, ground_truth):\n # Preprocess both texts\n extracted_text = preprocess_text(extracted_text)\n ground_truth = preprocess_text(ground_truth)\n \n # Compute Levenshtein distance and return the similarity ratio\n distance = Levenshtein.distance(extracted_text, ground_truth)\n max_len = max(len(extracted_text), len(ground_truth))\n \n # Similarity ratio\n similarity_ratio = 1 - (distance / max_len)\n return similarity_ratio","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T13:58:50.989545Z","iopub.execute_input":"2024-11-26T13:58:50.990015Z","iopub.status.idle":"2024-11-26T13:58:51.020914Z","shell.execute_reply.started":"2024-11-26T13:58:50.989977Z","shell.execute_reply":"2024-11-26T13:58:51.019291Z"}},"outputs":[],"execution_count":13},{"cell_type":"markdown","source":"### Accuracy check - 1. Using fitz model","metadata":{}},{"cell_type":"code","source":"ground_truth = read_ground_truth(file_path)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T13:58:56.279383Z","iopub.execute_input":"2024-11-26T13:58:56.279793Z","iopub.status.idle":"2024-11-26T13:58:56.290169Z","shell.execute_reply.started":"2024-11-26T13:58:56.279758Z","shell.execute_reply":"2024-11-26T13:58:56.288996Z"}},"outputs":[],"execution_count":14},{"cell_type":"code","source":"extracted_text = extract_text_from_pdf(pdf_path, method=\"fitz\")\n\n# Check Levenshtein similarity\nsimilarity_ratio = levenshtein_accuracy(extracted_text, ground_truth)\nprint(f\"Levenshtein Similarity: {similarity_ratio * 100:.2f}%\")\n\n# Check word match accuracy\nprecision, recall, f1 = word_match_accuracy(extracted_text, ground_truth)\nprint(f\"Precision: {precision:.3f}, Recall: {recall:.3f}, F1-Score: {f1:.3f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T13:58:59.734391Z","iopub.execute_input":"2024-11-26T13:58:59.735325Z","iopub.status.idle":"2024-11-26T13:58:59.827782Z","shell.execute_reply.started":"2024-11-26T13:58:59.735252Z","shell.execute_reply":"2024-11-26T13:58:59.826550Z"}},"outputs":[{"name":"stdout","text":"Levenshtein Similarity: 99.77%\nPrecision: 0.998, Recall: 0.998, F1-Score: 0.998\n","output_type":"stream"}],"execution_count":15},{"cell_type":"markdown","source":"### Accuracy check - 2. Using pdfplumber","metadata":{}},{"cell_type":"code","source":"extracted_text = extract_text_from_pdf(pdf_path, method=\"pdfplumber\")\n\n# Check Levenshtein similarity\nsimilarity_ratio = levenshtein_accuracy(extracted_text, ground_truth)\nprint(f\"Levenshtein Similarity: {similarity_ratio * 100:.2f}%\")\n\n# Check word match accuracy\nprecision, recall, f1 = word_match_accuracy(extracted_text, ground_truth)\nprint(f\"Precision: {precision:.3f}, Recall: {recall:.3f}, F1-Score: {f1:.3f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T13:59:03.072259Z","iopub.execute_input":"2024-11-26T13:59:03.072693Z","iopub.status.idle":"2024-11-26T13:59:04.092355Z","shell.execute_reply.started":"2024-11-26T13:59:03.072654Z","shell.execute_reply":"2024-11-26T13:59:04.091322Z"}},"outputs":[{"name":"stdout","text":"Levenshtein Similarity: 99.95%\nPrecision: 0.993, Recall: 0.996, F1-Score: 0.995\n","output_type":"stream"}],"execution_count":16},{"cell_type":"markdown","source":"### Accuracy check - 3. Using pypdf2","metadata":{}},{"cell_type":"code","source":"extracted_text = extract_text_from_pdf(pdf_path, method=\"pypdf2\")\n\n# Check Levenshtein similarity\nsimilarity_ratio = levenshtein_accuracy(extracted_text, ground_truth)\nprint(f\"Levenshtein Similarity: {similarity_ratio * 100:.2f}%\")\n\n# Check word match accuracy\nprecision, recall, f1 = word_match_accuracy(extracted_text, ground_truth)\nprint(f\"Precision: {precision:.3f}, Recall: {recall:.3f}, F1-Score: {f1:.3f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T13:59:05.894219Z","iopub.execute_input":"2024-11-26T13:59:05.894681Z","iopub.status.idle":"2024-11-26T13:59:06.178648Z","shell.execute_reply.started":"2024-11-26T13:59:05.894627Z","shell.execute_reply":"2024-11-26T13:59:06.176603Z"}},"outputs":[{"name":"stdout","text":"Levenshtein Similarity: 99.22%\nPrecision: 0.914, Recall: 0.937, F1-Score: 0.925\n","output_type":"stream"}],"execution_count":17},{"cell_type":"markdown","source":"## 1.3 Clean and normalise text\n\nextra preprocesing of extracted text for our main objectives","metadata":{}},{"cell_type":"code","source":"# Clean the text\ndef clean_txt(raw_text):\n cleaned_text = re.sub(r\"(page \\d+ of \\d+)\", \"\", raw_text, flags=re.IGNORECASE) # Remove page numbers\n cleaned_text = re.sub(r\"(\\n\\s*\\n)|(\\r\\n|\\r|\\n)\", \"\\n\", cleaned_text) # Remove extra newlines\n cleaned_text = re.sub(r\"\\s+\", \" \", cleaned_text) # Normalize whitespace\n cleaned_text = re.sub(r\"[^\\x00-\\x7F₹$\\s]+\", \" \", cleaned_text) # Remove non-ASCII characters\n normalized_text = cleaned_text.strip()\n\n return normalized_text","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T13:59:09.688829Z","iopub.execute_input":"2024-11-26T13:59:09.689242Z","iopub.status.idle":"2024-11-26T13:59:09.696820Z","shell.execute_reply.started":"2024-11-26T13:59:09.689210Z","shell.execute_reply":"2024-11-26T13:59:09.694844Z"}},"outputs":[],"execution_count":18},{"cell_type":"markdown","source":"# Step 2: Key-Value Pair Extraction\n\n**What we are doing**\n\n\n* Annotate domain-specific training data (e.g., contracts, finance documents).\n* Identify and extract values corresponding to predefined keys using AI models.\n\n**Libraries:**\n\nNER models: SpaCy\n\nre: Regular Expression","metadata":{}},{"cell_type":"markdown","source":"### Optional : Fine-Tune a Pre-Trained NER Model (Custom NER Model)\n\n* **Domain-Specific Adaptation**: Fine-tuning allows the model to recognize entities specific to the domain, improving accuracy.\n* **Reusability**: Pre-trained models like BERT can be adapted for multiple domains with minimal effort.\n* **Scalability**: The approach scales well for different domains as long as labeled data is available\n\n⚠️ More data of PDFs will be needed for this!\n\n### Or we can use Hugging Face's pretrained models","metadata":{}},{"cell_type":"markdown","source":"## Using the NER Model + Regular Expressions (re) for Key Extraction\n\n* Load the model and tokenizer to predict keys from the text\n* For better accuracy, will use Regular Expressions for dates and amount\n* Saving extracted keys in CSV file","metadata":{}},{"cell_type":"code","source":"!pip install spacy\n!python -m spacy download en_core_web_sm","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T14:00:02.305594Z","iopub.execute_input":"2024-11-26T14:00:02.306030Z","iopub.status.idle":"2024-11-26T14:00:34.209178Z","shell.execute_reply.started":"2024-11-26T14:00:02.305989Z","shell.execute_reply":"2024-11-26T14:00:34.207761Z"}},"outputs":[{"name":"stdout","text":"Requirement already satisfied: spacy in /opt/conda/lib/python3.10/site-packages (3.8.2)\nRequirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /opt/conda/lib/python3.10/site-packages (from spacy) (3.0.12)\nRequirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /opt/conda/lib/python3.10/site-packages (from spacy) (1.0.5)\nRequirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /opt/conda/lib/python3.10/site-packages (from spacy) (1.0.10)\nRequirement already satisfied: cymem<2.1.0,>=2.0.2 in /opt/conda/lib/python3.10/site-packages (from spacy) (2.0.8)\nRequirement already satisfied: preshed<3.1.0,>=3.0.2 in /opt/conda/lib/python3.10/site-packages (from spacy) (3.0.9)\nRequirement already satisfied: thinc<8.4.0,>=8.3.0 in /opt/conda/lib/python3.10/site-packages (from spacy) (8.3.2)\nRequirement already satisfied: wasabi<1.2.0,>=0.9.1 in /opt/conda/lib/python3.10/site-packages (from spacy) (1.1.3)\nRequirement already satisfied: srsly<3.0.0,>=2.4.3 in /opt/conda/lib/python3.10/site-packages (from spacy) (2.4.8)\nRequirement already satisfied: catalogue<2.1.0,>=2.0.6 in /opt/conda/lib/python3.10/site-packages (from spacy) (2.0.10)\nRequirement already satisfied: weasel<0.5.0,>=0.1.0 in /opt/conda/lib/python3.10/site-packages (from spacy) (0.4.1)\nRequirement already satisfied: typer<1.0.0,>=0.3.0 in /opt/conda/lib/python3.10/site-packages (from spacy) (0.12.3)\nRequirement already satisfied: tqdm<5.0.0,>=4.38.0 in /opt/conda/lib/python3.10/site-packages (from spacy) (4.66.4)\nRequirement already satisfied: requests<3.0.0,>=2.13.0 in /opt/conda/lib/python3.10/site-packages (from spacy) (2.32.3)\nRequirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /opt/conda/lib/python3.10/site-packages (from spacy) (2.9.2)\nRequirement already satisfied: jinja2 in /opt/conda/lib/python3.10/site-packages (from spacy) (3.1.4)\nRequirement already satisfied: setuptools in /opt/conda/lib/python3.10/site-packages (from spacy) (70.0.0)\nRequirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.10/site-packages (from spacy) (21.3)\nRequirement already satisfied: langcodes<4.0.0,>=3.2.0 in /opt/conda/lib/python3.10/site-packages (from spacy) (3.4.1)\nRequirement already satisfied: numpy>=1.19.0 in /opt/conda/lib/python3.10/site-packages (from spacy) (1.26.4)\nRequirement already satisfied: language-data>=1.2 in /opt/conda/lib/python3.10/site-packages (from langcodes<4.0.0,>=3.2.0->spacy) (1.2.0)\nRequirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.10/site-packages (from packaging>=20.0->spacy) (3.1.2)\nRequirement already satisfied: annotated-types>=0.6.0 in /opt/conda/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy) (0.7.0)\nRequirement already satisfied: pydantic-core==2.23.4 in /opt/conda/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy) (2.23.4)\nRequirement already satisfied: typing-extensions>=4.6.1 in /opt/conda/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy) (4.12.2)\nRequirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy) (3.3.2)\nRequirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy) (3.7)\nRequirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy) (1.26.18)\nRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests<3.0.0,>=2.13.0->spacy) (2024.8.30)\nRequirement already satisfied: blis<1.1.0,>=1.0.0 in /opt/conda/lib/python3.10/site-packages (from thinc<8.4.0,>=8.3.0->spacy) (1.0.1)\nRequirement already satisfied: confection<1.0.0,>=0.0.1 in /opt/conda/lib/python3.10/site-packages (from thinc<8.4.0,>=8.3.0->spacy) (0.1.5)\nCollecting numpy>=1.19.0 (from spacy)\n Downloading numpy-2.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m60.9/60.9 kB\u001b[0m \u001b[31m1.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n\u001b[?25hRequirement already satisfied: click>=8.0.0 in /opt/conda/lib/python3.10/site-packages (from typer<1.0.0,>=0.3.0->spacy) (8.1.7)\nRequirement already satisfied: shellingham>=1.3.0 in /opt/conda/lib/python3.10/site-packages (from typer<1.0.0,>=0.3.0->spacy) (1.5.4)\nRequirement already satisfied: rich>=10.11.0 in /opt/conda/lib/python3.10/site-packages (from typer<1.0.0,>=0.3.0->spacy) (13.7.1)\nRequirement already satisfied: cloudpathlib<1.0.0,>=0.7.0 in /opt/conda/lib/python3.10/site-packages (from weasel<0.5.0,>=0.1.0->spacy) (0.19.0)\nRequirement already satisfied: smart-open<8.0.0,>=5.2.1 in /opt/conda/lib/python3.10/site-packages (from weasel<0.5.0,>=0.1.0->spacy) (7.0.4)\nRequirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2->spacy) (2.1.5)\nRequirement already satisfied: marisa-trie>=0.7.7 in /opt/conda/lib/python3.10/site-packages (from language-data>=1.2->langcodes<4.0.0,>=3.2.0->spacy) (1.2.0)\nRequirement already satisfied: markdown-it-py>=2.2.0 in /opt/conda/lib/python3.10/site-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy) (3.0.0)\nRequirement already satisfied: pygments<3.0.0,>=2.13.0 in /opt/conda/lib/python3.10/site-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy) (2.18.0)\nRequirement already satisfied: wrapt in /opt/conda/lib/python3.10/site-packages (from smart-open<8.0.0,>=5.2.1->weasel<0.5.0,>=0.1.0->spacy) (1.16.0)\nRequirement already satisfied: mdurl~=0.1 in /opt/conda/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy) (0.1.2)\nDownloading numpy-2.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.5 MB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m19.5/19.5 MB\u001b[0m \u001b[31m51.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n\u001b[?25hInstalling collected packages: numpy\n Attempting uninstall: numpy\n Found existing installation: numpy 1.26.4\n Uninstalling numpy-1.26.4:\n Successfully uninstalled numpy-1.26.4\n\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\napache-beam 2.46.0 requires cloudpickle~=2.2.1, but you have cloudpickle 3.0.0 which is incompatible.\napache-beam 2.46.0 requires dill<0.3.2,>=0.3.1.1, but you have dill 0.3.8 which is incompatible.\napache-beam 2.46.0 requires numpy<1.25.0,>=1.14.3, but you have numpy 2.0.2 which is incompatible.\napache-beam 2.46.0 requires pyarrow<10.0.0,>=3.0.0, but you have pyarrow 17.0.0 which is incompatible.\ncatboost 1.2.7 requires numpy<2.0,>=1.16.0, but you have numpy 2.0.2 which is incompatible.\ngensim 4.3.3 requires numpy<2.0,>=1.18.5, but you have numpy 2.0.2 which is incompatible.\ngensim 4.3.3 requires scipy<1.14.0,>=1.7.0, but you have scipy 1.14.1 which is incompatible.\nibis-framework 7.1.0 requires numpy<2,>=1, but you have numpy 2.0.2 which is incompatible.\nibis-framework 7.1.0 requires pyarrow<15,>=2, but you have pyarrow 17.0.0 which is incompatible.\nlibpysal 4.9.2 requires packaging>=22, but you have packaging 21.3 which is incompatible.\nlibpysal 4.9.2 requires shapely>=2.0.1, but you have shapely 1.8.5.post1 which is incompatible.\nmatplotlib 3.7.5 requires numpy<2,>=1.20, but you have numpy 2.0.2 which is incompatible.\ntensorflow 2.16.1 requires numpy<2.0.0,>=1.23.5; python_version <= \"3.11\", but you have numpy 2.0.2 which is incompatible.\ntensorflow-transform 0.14.0 requires numpy<2,>=1.16, but you have numpy 2.0.2 which is incompatible.\nxarray 2024.9.0 requires packaging>=23.1, but you have packaging 21.3 which is incompatible.\nydata-profiling 4.10.0 requires scipy<1.14,>=1.4.1, but you have scipy 1.14.1 which is incompatible.\u001b[0m\u001b[31m\n\u001b[0mSuccessfully installed numpy-2.0.2\nCollecting en-core-web-sm==3.8.0\n Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl (12.8 MB)\n\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.8/12.8 MB\u001b[0m \u001b[31m64.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m0:01\u001b[0m\n\u001b[?25h\u001b[38;5;2m✔ Download and installation successful\u001b[0m\nYou can now load the package via spacy.load('en_core_web_sm')\n","output_type":"stream"}],"execution_count":19},{"cell_type":"markdown","source":"**Extraction of key-value pairs**","metadata":{}},{"cell_type":"code","source":"import spacy\nimport re\n\n# Loading the spaCy model for Named Entity Recognition (NER)\nnlp = spacy.load(\"en_core_web_sm\")\n\ndef key_value_extraction(text, keys):\n \"\"\"\n Extract key values from a cleaned contract text based on the provided keys.\n\n Parameters:\n - text (str): Cleaned and preprocessed contract text.\n - keys (list, optional): A list of keys (strings) to extract. If None, all keys will be extracted.\n\n Returns:\n - dict: A dictionary containing the extracted key-value pairs.\n \"\"\"\n # Process the text with spaCy for Named Entity Recognition (NER)\n doc = nlp(text)\n\n # Extract company names using NER (ORG - Organizations)\n companies = [ent.text for ent in doc.ents if ent.label_ == \"ORG\"]\n\n # Extract dates using regex pattern for dates\n date_pattern = r\"\\d{1,2}[a-z]{2}\\s[A-Za-z]+\\s\\d{4}\"\n dates = re.findall(date_pattern, text)\n\n # Extract financial values (e.g., investment amounts, transaction values) using regex\n amount_pattern = r\"₹[\\d,]+(?:\\s*crore|\\s*\$.*\$)\"\n amounts = re.findall(amount_pattern, text)\n\n # Define helper function to check for exclusive rights\n def check_exclusive_rights(text):\n \"\"\"\n Function to check if the contract mentions exclusive rights.\n Returns True if exclusive rights are granted, otherwise False.\n \"\"\"\n # Keywords or phrases that indicate exclusive rights\n exclusive_rights_keywords = [\"exclusive rights\", \"granted exclusive rights\", \"exclusive distribution rights\"]\n \n # Search for any of the keywords in the text\n for keyword in exclusive_rights_keywords:\n if re.search(rf\"{keyword}\", text, re.IGNORECASE):\n return True\n \n # If no keywords are found, return False\n return False\n\n # Check if the contract mentions exclusive rights\n exclusive_rights = check_exclusive_rights(text)\n\n # Prepare the default extracted information, note that the below config depends upon predefined\n # keywords and it can be done better for more domains\n extracted_info = {\n keys[0]: companies[0] if len(companies) > 0 else None, # First company (Party 1)\n keys[1]: companies[1] if len(companies) > 1 else None, # Second company (Party 2)\n keys[2]: dates[0] if dates else None, # First date found\n keys[3]: amounts[0] if amounts else None, # First investment amount found\n keys[4]: amounts[2] if len(amounts) > 1 else None, # Second transaction value found\n keys[5]: \"Yes\" if exclusive_rights else \"No\" # Boolean value indicating exclusive rights\n }\n\n # Return the filtered dictionary of extracted information\n return extracted_info\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T14:02:04.452797Z","iopub.execute_input":"2024-11-26T14:02:04.453488Z","iopub.status.idle":"2024-11-26T14:02:05.279215Z","shell.execute_reply.started":"2024-11-26T14:02:04.453446Z","shell.execute_reply":"2024-11-26T14:02:05.277966Z"}},"outputs":[],"execution_count":21},{"cell_type":"markdown","source":"**Save to CSV**","metadata":{}},{"cell_type":"code","source":"import csv","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T14:02:11.850631Z","iopub.execute_input":"2024-11-26T14:02:11.851127Z","iopub.status.idle":"2024-11-26T14:02:11.856447Z","shell.execute_reply.started":"2024-11-26T14:02:11.851090Z","shell.execute_reply":"2024-11-26T14:02:11.855076Z"}},"outputs":[],"execution_count":22},{"cell_type":"code","source":"def save_to_csv(key_values, output_file):\n \"\"\"\n Saves the extracted key-value pairs to a CSV file.\n\n Args:\n extracted_values (dict): Dictionary of extracted key-value pairs.\n output_file (str): Path to the output CSV file.\n \"\"\"\n with open(output_file, mode='w', newline='') as file:\n writer = csv.writer(file)\n writer.writerow([\"Key\", \"Value\"])\n for key, value in key_values.items():\n writer.writerow([key, value])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T14:02:15.186026Z","iopub.execute_input":"2024-11-26T14:02:15.186460Z","iopub.status.idle":"2024-11-26T14:02:15.193470Z","shell.execute_reply.started":"2024-11-26T14:02:15.186424Z","shell.execute_reply":"2024-11-26T14:02:15.191932Z"}},"outputs":[],"execution_count":23},{"cell_type":"markdown","source":"## Problem - Keys are to be exact?\n\nHandling Flexible Keys for Extraction:\nDomain-Specific Synonyms:\n\nFor each domain, I used some standard variations of keys (e.g., for contracts: \"First Party,\" \"Party One,\" \"Party 1\" all refer to the same entity).\nThis is done by specifying multiple synonyms in the domain-specific patterns.\nCase Insensitivity:\n\nThe regex patterns in the code are case-insensitive (e.g., (?i)), so \"Start Date\" will match \"start date\" or \"START DATE\" in the text.\n\n\n\n\n### Solution - Custom Key Mapping:\n\nYou can have a custom key mapping system where a user can provide a list of keys (e.g., \"Start Date\", \"Start Date of Contract\"), and these would map to predefined extraction patterns in your system.\nEnhancement for Flexible Matching:\nHere’s how you can modify the code to allow for more flexible or fuzzy key matching:\n\nAllow for Multiple Variants of Keys (Domain-Specific Variations):\n\nAdd different forms of keys within the same domain rules so that the same key can be captured even if it appears in different formats.\nAllow Users to Define Their Own Keys:\n\nUsers can input the keys they need, and you can check for matches in the text regardless of slight variations.\n","metadata":{}},{"cell_type":"markdown","source":"**Example of Custom Key Mapping:**","metadata":{}},{"cell_type":"code","source":"# Domain-specific patterns\ndomain_rules = {\n \"Contracts\": {\n \"Name of the 1st Party\": r\"(?:First Party|Party 1|Party One):?\\s*([^\\n,]+)\",\n \"Name of the 2nd Party\": r\"(?:Second Party|Party 2|Party Two):?\\s*([^\\n,]+)\",\n \"Contract Start Date\": r\"(?:Effective Date|Start Date|Commencement):?\\s*(\\d{1,2}[-/]\\d{1,2}[-/]\\d{2,4})\",\n \"Contract End Date\": r\"(?:End Date|Termination Date):?\\s*(\\d{1,2}[-/]\\d{1,2}[-/]\\d{2,4})\",\n \"Scope of Work\": r\"(?:Scope of Work|Services):?\\s*([^\\n]+)\",\n \"Penalty Amount\": r\"(?:Penalty|Fine):?\\s*\\$?([\\d,]+)\"\n },\n \"Finance\": {\n \"Transaction Amount\": r\"(?:Amount|Transaction):?\\s*\\$?([\\d,]+)\",\n \"Date of Transaction\": r\"(?:Transaction Date|Date):?\\s*(\\d{1,2}[-/]\\d{1,2}[-/]\\d{2,4})\",\n \"Account Number\": r\"(?:Account Number|Acc No):?\\s*([^\\n,]+)\",\n \"Bank Name\": r\"(?:Bank|Financial Institution):?\\s*([^\\n,]+)\"\n },\n \"Legal\": {\n \"Case Number\": r\"(?:Case Number|Case ID):?\\s*([^\\n,]+)\",\n \"Filing Date\": r\"(?:Filing Date|Date Filed):?\\s*(\\d{1,2}[-/]\\d{1,2}[-/]\\d{2,4})\",\n \"Petitioner\": r\"(?:Petitioner|Claimant):?\\s*([^\\n,]+)\",\n \"Respondent\": r\"(?:Respondent|Defendant):?\\s*([^\\n,]+)\",\n \"Court Name\": r\"(?:Court|Jurisdiction):?\\s*([^\\n,]+)\"\n },\n \"HR\": {\n \"Employee Name\": r\"(?:Employee Name|Name):?\\s*([^\\n,]+)\",\n \"Employee ID\": r\"(?:Employee ID|ID):?\\s*([^\\n,]+)\",\n \"Joining Date\": r\"(?:Joining Date|Start Date):?\\s*(\\d{1,2}[-/]\\d{1,2}[-/]\\d{2,4})\",\n \"Department\": r\"(?:Department|Team):?\\s*([^\\n,]+)\",\n \"Salary\": r\"(?:Salary|Compensation):?\\s*\\$?([\\d,]+)\"\n },\n \"Invoices\": {\n \"Invoice Number\": r\"(?:Invoice Number|Invoice ID):?\\s*([^\\n,]+)\",\n \"Invoice Date\": r\"(?:Invoice Date|Date):?\\s*(\\d{1,2}[-/]\\d{1,2}[-/]\\d{2,4})\",\n \"Client Name\": r\"(?:Client Name|Customer Name):?\\s*([^\\n,]+)\",\n \"Total Amount\": r\"(?:Total Amount|Total):?\\s*\\$?([\\d,]+)\",\n \"Due Date\": r\"(?:Due Date|Payment Due):?\\s*(\\d{1,2}[-/]\\d{1,2}[-/]\\d{2,4})\"\n }\n}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-22T17:13:07.445959Z","iopub.execute_input":"2024-11-22T17:13:07.446511Z","iopub.status.idle":"2024-11-22T17:13:07.466388Z","shell.execute_reply.started":"2024-11-22T17:13:07.446457Z","shell.execute_reply":"2024-11-22T17:13:07.464654Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Step 3: Document Summarization\n\n**What are we doing**\n\n* Models: facebook/bart-large-cnn\n* Utilize extracted keys and values to contextualize summaries using the model (hugging-face)\n* Key-Value Pairs: The model will prioritize and emphasize these key points (e.g., party names, contract start and end dates, penalty amount) in the summary.\n\nInput: The model will take in:\n\n- Text: The raw document text.\n- Keys: A list of predefined keys that are relevant to the domain or document type. \n- Values: A list of corresponding values that describe the content or information related to the keys.\n- Word Limit: A maximum word count for the summary (default to 1000 words).","metadata":{}},{"cell_type":"code","source":"from transformers import pipeline\nimport re\n\n# Step 1: Load the summarization pipeline\nsummarizer = pipeline(\"summarization\", model=\"facebook/bart-large-cnn\")\n\n# Step 2: Custom Summarization Function based on Keys and Values\ndef custom_summarization(text, keys, values):\n \"\"\"\n Summarize text by focusing on specific keys and values.\n \n Args:\n - text (str): The full text to summarize.\n - keys (list): List of keys (such as 'Investment Amount', 'Party 1', etc.)\n - values (list): List of corresponding values (such as 'Reliance', '₹11,500 crore', etc.)\n \n Returns:\n - summary (str): A summary that focuses on the provided keys and values.\n \"\"\"\n \n relevant_text = \"\"\n \n # Try to match key-value pairs and extract related text\n for key, value in zip(keys, values):\n # Find sentences containing the key and its value (this is a simple heuristic)\n pattern = rf\"([^.]*{re.escape(value)}[^.]*\\.)\"\n matches = re.findall(pattern, text)\n \n # Combine the relevant sentences into the relevant_text\n relevant_text += \" \".join(matches)\n \n # If no relevant text is found, use the whole text for summarization\n if not relevant_text:\n relevant_text = text\n \n # Summarize the extracted relevant text\n summary = summarizer(relevant_text, max_length=150, min_length=50, do_sample=False)\n return summary[0]['summary_text']","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-26T14:02:32.653126Z","iopub.execute_input":"2024-11-26T14:02:32.653562Z","iopub.status.idle":"2024-11-26T14:03:07.341104Z","shell.execute_reply.started":"2024-11-26T14:02:32.653526Z","shell.execute_reply":"2024-11-26T14:03:07.338320Z"}},"outputs":[{"output_type":"display_data","data":{"text/plain":"config.json: 0%| | 0.00/1.58k [00:00

FilesExpand file tree

AI DocParser.ipynb

Latest commit

History

AI DocParser.ipynb

File metadata and controls