Merge pull request #14 from DataFog/v2.3.2

updated README.MD

Author: sidmohan0

Diff for: .gitignore

@@ -13,7 +13,7 @@ build/
 /src/datafog/pii_tools/__pycache__/
 /tests/__pycache__/
 /tests/scratch.py
-node_modules
+node_modules/
 datafog_debug.log
 sotu_2023.txt
 .DS_Store

Diff for: README.md

@@ -3,88 +3,105 @@
 </p>
 
 <p align="center">
-    <b>Open-source PII Detection for Retrieval Systems</b>. <br />
-    Scan, redact, and manage PII in your documents before they get uploaded to a Generative AI system. 
+    <b>Open-source DevSecOps for Generative AI Systems</b>. <br />
 </p>
 
 <p align="center">
   <a href="https://pypi.org/project/datafog/"><img src="https://img.shields.io/pypi/v/datafog.svg?style=flat-square" alt="PyPi Version"></a>
   <a href="https://pypi.org/project/datafog/"><img src="https://img.shields.io/pypi/pyversions/datafog.svg?style=flat-square" alt="PyPI pyversions"></a>
   <a href="https://github.com/datafog/datafog-python"><img src="https://img.shields.io/github/stars/datafog/datafog-python.svg?style=flat-square&logo=github&label=Stars&logoColor=white" alt="GitHub stars"></a>
   <a href="https://pypistats.org/packages/datafog"><img src="https://img.shields.io/pypi/dm/datafog.svg?style=flat-square" alt="PyPi downloads"></a>
-</p>
-
-<p align="center">
+  <a href="https://discord.gg/bzDth394R4"><img src="https://img.shields.io/discord/1173803135341449227?style=flat" alt="Discord"></a>
   <a href="https://github.com/psf/black"><img src="https://img.shields.io/badge/code%20style-black-000000.svg?style=flat-square" alt="Code style: black"></a>
-</p>
-
-<p align="center">
   <a href="https://codecov.io/gh/datafog/datafog-python"><img src="https://img.shields.io/codecov/c/github/datafog/datafog-python.svg?style=flat-square" alt="codecov"></a>
+  <a href="https://github.com/datafog/datafog-python/issues"><img src="https://img.shields.io/github/issues/datafog/datafog-python.svg?style=flat-square" alt="GitHub Issues"></a>
 </p>
 
 ## Overview
 
-DataFog works by scanning and redacting-out PII in files **before** they get uploaded to a Generative AI system.
+### What is DataFog?
+
+DataFog is an open-source DevSecOps platform that lets you scan and redact Personally Identifiable Information (PII) out of your Generative AI applications.
+
+### What problem are we solving?
+
+**Context**
+
+The primary use case today is Retrieval Augmented Generation (RAG) systems. As a refresher, RAG systems operate by retrieving information from a custom knowledge base—constructed by you or your team—and leverage this information, either by directly citing the files in a response or inferred through the model's responses. This knowledge base is assembled through a deliberate process, which involves uploading files into a workflow. These files are then segmented into logical information blocks and tagged according to their contextual significance. There are a thousand ways to add nuance to this characterization, but this suffices for the vast majority of cases!
+
+**Problem**
+
+How do you keep:
+
+- Customer PII
+- Employee PII
+- Sensitive company information pertaining to org changes or restructurings
+- Pending M&A activity
+- Conversations with external counsel on material corporate matters (i.e. product recall, etc)
+- and more
+
+from entering a Generative AI environment in the first place? What you need is a tool to scan and redact your RAG-bound documents based on your organization or team needs.
 
-## How it works
+That's where DataFog comes in. Our solution to this problem is through two major approaches:
 
-<img src="https://www.datafog.ai/hero.png" alt="DataFog Overview">
+**PII Observability** Take in your batch/streaming data and return a scan indicating character-level detection of entities
+**Privacy Filter** DataFog can slot in as a pre-processor that redacts PII from your files before they get uploaded to a RAG database
+
+With this SDK, you can import it into a Python environment (like a Google Colab notebook, check out our [Getting Started](examples/getting-started.ipynb)) and within a few lines of code you're up and running.
+
+### How it works
+
+<img src="https://www.datafog.ai/hero.png" alt="DataFog Overview" style="width:50%;">
+
+### There's lots of PII tools out there; why DataFog?
+
+If you look at the landscape of PII detection tools, their very existence was in many cases driven by regulatory requirements (i.e. 'comply with CCPA/GDPR/HIPAA').
+In this scenario, there's a very defined problem, a specific set of immutable entities to look for, and a relatively static universe of document schema to work with. What that means as an end-result is that the products
+are purpose-built for the problem that they are solving.
+
+However, Generative AI changes how we think about privacy. There's now a changing set of privacy requirements (new M&A deals, internal discussions means new terms to scan/redact) as well as different and varying document sources to contend with. PII detection is no longer just about compliance, it's an active - and for some, new - internal security threat for CISOs and Eng Leaders to contend with. We want DataFog to be built and driven to meet the needs of the open-source community as they tackle this challenge.
 
 ## Installation
 
 DataFog can be installed via pip:
 
 ```bash
-pip install datafog # python client
+pip install datafog
 ```
 
-## Usage
-
-We're going to build up functionality starting with support for the Microsoft Presidio library. If you have any custom requests that would be of benefit to the community, please let us know!
+and in your python environment:
 
 ```
-  import requests
-  from datafog import PresidioEngine as presidio
-
-  # Example: Detecting PII in a String
-  pii_detected = presidio.scan("My name is John Doe and my email is [email protected]")
-  print("PII Detected:", pii_detected)
-
-  # Example: Detecting PII in a File
-  sample_filepath = "/Users/sidmohan/Desktop/v2.0.0/datafog-python/tests/files/input_files/sample.csv"
-  with open(sample_filepath, "r") as f:
-      original_value = f.read()
-  pii_detected = presidio.scan(original_value)
-  print("PII Detected in File:", pii_detected)
-
-  # Example: Detecting PII in a URL
-  sample_url = "https://gist.githubusercontent.com/sidmohan0/1aa3ec38b4e6594d3c34b113f2e0962d/raw/42e57146197be0f85a5901cd1dcdd9ad15b31bab/sotu_2023.txt"
-  response = requests.get(sample_url)
-  original_value = response.text
-  pii_detected = presidio.scan(original_value)
-  print("PII Detected in URL Content:", pii_detected)
-
+from datafog import PresidioEngine as presidio
 ```
 
-Depending on your input, the output will be a list of detected PII entities:
+## Examples
+
+Here are some examples of datafog being used to redact information in business contexts. Please see '/examples' for our [Getting Started](examples/getting-started.ipynb) notebook. We'll be regularly updating content and providing comprehensive guides to using DataFog in production contexts. If you have any ideas for a tutorial or guide that you would like to see, please let us know!
 
 ```
-PII Detected: [type: EMAIL_ADDRESS, start: 36, end: 53, score: 1.0, type: PERSON, start: 11, end: 19, score: 0.85, type: URL, start: 44, end: 53, score: 0.5]
+  ceo_email_chunk = "I'm announcing on Friday that Jeff is going to be CTO."
+
+  scan_results1 = presidio.scan(ceo_email_chunk)
+  print("PII Detected - base case:", scan_results1)
+  # PII Detected - base case: [type: PERSON, start: 30, end: 34, score: 0.85]
+
+
+  scan_results2 = presidio.scan(ceo_email_chunk, deny_list=['CTO'])
+  print("PII Detected with deny list:", scan_results2)
+  # PII Detected with deny list: [type: CUSTOM_PII, start: 50, end: 53, score: 1.0, type: PERSON, start: 30, end: 34, score: 0.85]
+
 ```
 
 ## Contributing
 
-This is an open-source project and we welcome contributions. If you have any questions, please feel free to reach out to us, join our Discord or email me directly at [email protected].
+DataFog is a community-driven **open-source** platform and we've been fortunate to have a small and growing contributor base. We'd love to hear ideas, feedback, suggestions for improvement - anything on your mind about what you think can be done to make DataFog better! Join our [Discord](https://discord.gg/bzDth394R4) and join our growing community.
 
 ### Dev Notes
 
-- Clone repo
-- Run 'poetry install' to install dependencies (recommend entering poetry shell for preserving dependencies)
 - Justfile commands:
   - `just format` to apply formatting.
   - `just lint` to check formatting and style.
-  - `just tag` to tag your project on git
-  - `just upload` to publish to PyPi.
 
 ### Testing
 
@@ -96,7 +113,7 @@ tox
 
 ```
 
-### License
+## License
 
 This software is published under the [MIT
 license](https://en.wikipedia.org/wiki/MIT_License).

Diff for: examples/getting-started.ipynb

@@ -4,7 +4,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# <img src=\"../public/colorlogo.png\" width=\"20%\"/>\n"
+    "<img src=\"../public/colorlogo.png\" width=\"50%\"/>\n",
+    "\n",
+    "\n",
+    "[Homepage](https://www.datafog.ai) | \n",
+    "[Discord](https://discord.gg/bzDth394R4) | \n",
+    "[Github](https://github.com/datafog/datafog-python) | \n",
+    "[Contact](mailto:[email protected])\n",
+    "\n"
    ]
   },
   {
@@ -20,10 +27,15 @@
    "source": [
     "Welcome to the DataFog Python SDK! This notebook will walk you through the basics of using the SDK to scan text for PII.\n",
     "\n",
-    "Please consider this a living document - we will be focused on adding new content and guides to make using DataFog as easy as possible! If you have any questions or need help, please reach out to us on our Discord or email us at [email protected].\n",
+    "Please consider this a living document - we will be focused on adding new content and guides to make using DataFog as easy as possible! If you have any questions or need help, please reach out to us on our [Discord](https://discord.gg/bzDth394R4) or email us at [email protected].\n",
     "\n"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -42,7 +54,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 80,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -67,18 +79,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 81,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "                                                text\n",
-      "0  Mr. Speaker, Madam Vice President, our First L...\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "# Download the transcript of the US State of the Union 2023\n",
     "url = \"https://bit.ly/datafog-sample-text-sotu-2023\"\n",
@@ -113,18 +116,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 82,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "PII Detected - base case: [type: PERSON, start: 30, end: 34, score: 0.85]\n",
-      "PII Detected with deny list: [type: CUSTOM_PII, start: 50, end: 53, score: 1.0, type: PERSON, start: 30, end: 34, score: 0.85]\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "ceo_email_chunk = \"I'm announcing on Friday that Jeff is going to be CTO.\"\n",
     "\n",