Readme Fixes (#340)

* Update names of Models and Datasets used

* Update README

* Update Table of contents for Notebooks

Signed-off-by: Mitali Potnis <[email protected]>

* Update Table of contents for Notebooks

Signed-off-by: Mitali Potnis <[email protected]>

---------

Signed-off-by: Mitali Potnis <[email protected]>

Author: mitalipo

DATASETS.md

@@ -12,4 +12,6 @@ This is a comprehensive list of public datasets used by this repository.
 | [ImageNet (TorchVision)](https://pytorch.org/vision/main/generated/torchvision.datasets.ImageNet.html) | PyTorch | Image Classification |
 | [IMDB Reviews](https://ai.stanford.edu/~amaas/data/sentiment/) | PyTorch | Text Classification |
 | [MNIST (TorchVision)](https://pytorch.org/vision/main/generated/torchvision.datasets.MNIST.html) | PyTorch | Image Classification |
-| [SMS Spam Collection](https://archive.ics.uci.edu/dataset/228/sms+spam+collection) | PyTorch | Text Classification |
+| [SMS Spam Collection](https://archive.ics.uci.edu/dataset/228/sms+spam+collection) | PyTorch | Text Classification |
+| [ToxicChat](https://huggingface.co/datasets/lmsys/toxic-chat) | PyTorch | Toxicity Model Benchmarking |
+| [Jigsaw Unintended Bias](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification) | PyTorch | Toxicity Model Benchmarking |

MODELS.md

@@ -0,0 +1,7 @@
+# Models
+
+This is a comprehensive list of public models used by this repository.
+
+| Model Name (Link/Source) | Framework | Model Hub |
+|--------------------| --------- | -------- |
+| [ toxic-prompt-roberta ](https://huggingface.co/Intel/toxic-prompt-roberta) | PyTorch | Hugging Face |

README.md

@@ -139,6 +139,6 @@ Intel is committed to the respect of human rights and avoiding complicity in hum
 Intel® Explainable AI Tools is licensed under Apache License Version 2.0.
 
 #### Datasets and Models
-To the extent that any data, datasets, or models are referenced by Intel or accessed using tools or code on this site such data, datasets and models are provided by the third party indicated as the source of such content. Intel does not create the data, datasets, or models, provide a license to any third-party data, datasets, or models referenced, and does not warrant their accuracy or quality. By accessing such data, dataset(s) or model(s) you agree to the terms associated with that content and that your use complies with the applicable license. [DATASETS](DATASETS.md)
+To the extent that any data, datasets, or models are referenced by Intel or accessed using tools or code on this site such data, datasets and models are provided by the third party indicated as the source of such content. Intel does not create the data, datasets, or models, provide a license to any third-party data, datasets, or models referenced, and does not warrant their accuracy or quality. By accessing such data, dataset(s) or model(s) you agree to the terms associated with that content and that your use complies with the applicable license. [DATASETS](DATASETS.md), [MODELS](MODELS.md)
 
 Intel expressly disclaims the accuracy, adequacy, or completeness of any data, datasets or models, and is not liable for any errors, omissions, or defects in such content, or for any reliance thereon. Intel also expressly disclaims any warranty of non-infringement with respect to such data, dataset(s), or model(s). Intel is not liable for any liability or damages relating to your use of such data, datasets, or models.

notebooks/README.md

@@ -20,5 +20,7 @@ This directory has Jupyter notebooks that demonstrate explainability and model c
 | [Generating a Model Card with PyTorch](model_card_gen/model_card_generation_with_pytorch/adult-pytorch-model-card.ipynb) | Numerical/Categorical: Tabular Classification | PyTorch | Demonstrates training a multilayer network using the "Adult" dataset from the UCI repository to predict whether a person has a salary greater or less than $50,000, then uses the Model Card Generator to create a model card with interactive graphics to analyze the model. |
 | [Detecting Issues in Fairness by Generate Model Card from TensorFlow Estimators](model_card_gen/compas_with_model_card_gen/compas-model-card-tfx.ipynb) | Numerical/Categorical: Tabular Classification  | TensorFlow | Uses a TFX pipeline to train and evaluate a model using the COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) dataset to generate a risk score indended to determine a defendant's likelihood of reoffending. The Model Card Generator is then used to create interative graphics visualizing racial bias in the model's predictions. |
 | [Creating Model Card for Toxic Comments Classification in TensorFlow](model_card_gen/toxic_comments_classification/toxicity-tfma-model-card.ipynb) | Numerical/Categorical: Tabular Classification | TensorFlow | Adapts a [TensorFlow Fairness Exercise notebook](https://colab.research.google.com/github/google/eng-edu/blob/main/ml/pc/exercises/fairness_text_toxicity_part1.ipynb?utm_source=practicum-fairness&utm_campaign=colab-external&utm_medium=referral&utm_content=fairnessexercise1-colab#scrollTo=2z_xzJ40j9Q-) to use the Model Card Generator. The notebook trains a model to detect toxicity in online coversations and graphically analyzes accuracy metrics by gender. |
+| [Creating Model Card for Hate Speech Detection using Hugging Face model](model_card_gen/hugging_face_model_card) | Numerical/Categorical: Tabular Classification | PyTorch | Utilizes a model hosted on Hugging Face Hub for detecting hatespeech in English language using the HateXplain dataset. The Model Card Generator is then used to create a model card with interactive graphics to analyze the model performance metrics at threshold and Bias AUC metric for target groups. |
+| [Multiclass classification of Hate Speech using Hugging Face model](model_card_gen/multiclass_classification) | Numerical/Categorical: Tabular Classification | PyTorch | Uses a model hosted on Hugging Face Hub for classifying hate speech into Hate, Offensive, or Normal categories using the HateXplain dataset. The Model Card Generator is then used to create a model card with individual interactive graphics for each class to analyze the model performance metrics at threshold and the Bias AUC metric for target groups. |
 
 *Other names and brands may be claimed as the property of others. [Trademarks](http://www.intel.com/content/www/us/en/legal/trademarks.html)

notebooks/model_card_gen/README.ipynb

@@ -12,7 +12,8 @@
     "| [Generating a Model Card with PyTorch](./model_card_generation_with_pytorch)| Numerical/Categorical: Tabular Classification | PyTorch | Demonstrates training a multilayer network using the \"Adult\" dataset from the UCI repository to predict whether a person has a salary greater or less than $50,000. The Model Card Generator is then used to to create a model card with interactive graphics to analyze the model. |\n",
     "| [Detecting Issues in Fairness by generating a Model Card from TensorFlow Estimators](./compas_with_model_card_gen) | Numerical/Categorical: Tabular Classification  | TensorFlow | Utilizes a TFX pipeline to train and evaluate a model using the COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) dataset to generate a risk score indended to determine a defendant's likelihood of reoffending. The Model Card Generator is then used to create interative graphics visualizing racial bias in the model's predictions. |\n",
     "| [Creating Model Card for Toxic Comments Classification in TensorFlow](./toxic_comments_classification) | Numerical/Categorical: Tabular Classification | TensorFlow | Adapts a [TensorFlow Fairness Exercise notebook](https://colab.research.google.com/github/google/eng-edu/blob/main/ml/pc/exercises/fairness_text_toxicity_part1.ipynb?utm_source=practicum-fairness&utm_campaign=colab-external&utm_medium=referral&utm_content=fairnessexercise1-colab#scrollTo=2z_xzJ40j9Q-) to use the Model Card Generator. The notebook trains a model to detect toxicity in online coversations and graphically analyzes accuracy metrics by gender. |\n",
-    "\n",
+    "| [Creating Model Card for Hate Speech Detection using Hugging Face model](hugging_face_model_card) | Numerical/Categorical: Tabular Classification | PyTorch | Utilizes a model hosted on Hugging Face Hub for detecting hatespeech in English language using the HateXplain dataset. The Model Card Generator is then used to create a model card with interactive graphics to analyze the model performance metrics at threshold and Bias AUC metric for target groups. |\n",
+    "| [Multiclass classification of Hate Speech using Hugging Face model](multiclass_classification) | Numerical/Categorical: Tabular Classification | PyTorch | Uses a model hosted on Hugging Face Hub for classifying hate speech into Hate, Offensive, or Normal categories using the HateXplain dataset. The Model Card Generator is then used to create a model card with individual interactive graphics for each class to analyze the model performance metrics at threshold and the Bias AUC metric for target groups. |\n",
     "\n"
    ]
   }

notebooks/model_card_gen/README.md

@@ -6,5 +6,7 @@ This directory has Jupyter notebooks that demonstrate model card generation usin
 | [Generating a Model Card with PyTorch](model_card_generation_with_pytorch) | Numerical/Categorical: Tabular Classification | PyTorch | Demonstrates training a multilayer network using the "Adult" dataset from the UCI repository to predict whether a person has a salary greater or less than $50,000. The Model Card Generator is then used to create a model card with interactive graphics to analyze the model. |
 | [Detecting Issues in Fairness by generating a Model Card from TensorFlow Estimators](compas_with_model_card_gen) | Numerical/Categorical: Tabular Classification  | TensorFlow | Utilizes a TFX pipeline to train and evaluate a model using the COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) dataset to generate a risk score indended to determine a defendant's likelihood of reoffending. The Model Card Generator is then used to create interative graphics visualizing racial bias in the model's predictions. |
 | [Creating Model Card for Toxic Comments Classification in TensorFlow](toxic_comments_classification) | Numerical/Categorical: Tabular Classification | TensorFlow | Adapts a [TensorFlow Fairness Exercise notebook](https://colab.research.google.com/github/google/eng-edu/blob/main/ml/pc/exercises/fairness_text_toxicity_part1.ipynb?utm_source=practicum-fairness&utm_campaign=colab-external&utm_medium=referral&utm_content=fairnessexercise1-colab#scrollTo=2z_xzJ40j9Q-) to use the Model Card Generator. The notebook trains a model to detect toxicity in online coversations and graphically analyzes accuracy metrics by gender. |
+| [Creating Model Card for Hate Speech Detection using Hugging Face model](hugging_face_model_card) | Numerical/Categorical: Tabular Classification | PyTorch | Utilizes a model hosted on Hugging Face Hub for detecting hatespeech in English language using the HateXplain dataset. The Model Card Generator is then used to create a model card with interactive graphics to analyze the model performance metrics at threshold and Bias AUC metric for target groups. |
+| [Multiclass classification of Hate Speech using Hugging Face model](multiclass_classification) | Numerical/Categorical: Tabular Classification | PyTorch | Uses a model hosted on Hugging Face Hub for classifying hate speech into Hate, Offensive, or Normal categories using the HateXplain dataset. The Model Card Generator is then used to create a model card with individual interactive graphics for each class to analyze the model performance metrics at threshold and the Bias AUC metric for target groups. |
 
 *Other names and brands may be claimed as the property of others. [Trademarks](http://www.intel.com/content/www/us/en/legal/trademarks.html)

plugins/benchmark/classification_metrics/README.md

@@ -13,12 +13,10 @@ For evaluating a target toxicity detection LLM, we use the ToxicChat and Jigsaw
     - accuracy
     - auprc (area under precision recall curve)
     - auroc
-    - auprc (area under precision recall curve)
     - f1
     - fpr (false positive rate)
     - precision
     - recall
-    - fpr (false positive rate)
 
 ## Get Started