BERT_for_text_classification_Finnish.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7edc3587-2e2b-418a-ab50-628cef59c320",
   "metadata": {
    "tags": []
   },
   "source": [
    "# BERT text classification in Finnish. \n",
    "<br/>\n",
    "Ekta Vats, Centre for Digital Humanities (CDHU), Department of ALM, Uppsala University <br>\n",
    "ekta.vats@abm.uu.se <br> <br>\n",
    "\n",
    "Matti La Mela, Department of ALM, Uppsala University <br> <br>\n",
    "\n",
    "Using Simple Transformers - an NLP library based on the Transformers library by HuggingFace. <br>\n",
    "Link: https://huggingface.co/docs/transformers/index\n",
    "\n",
    "Dataset used: Berry corpus. <br>\n",
    "Classification of OCR-ed texts into 2 categories (binary classification): <br>\n",
    "Category 0: DESCRIPTIVE (i.e. descriptive articles) <br>\n",
    "Category 1: ECONOMIC (i.e. economic-industrial articles) <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e184316-0e38-4f93-b51c-9f2891fcabda",
   "metadata": {},
   "source": [
    "The binary division is between articles where berries / berry picking is mentioned for some contextual or descriptive reason. <br>\n",
    "For example: <br> \n",
    "Snake bite a berry picking child => 0 <br>\n",
    "Articles regarding selling berries, exports, industrial production, etc. => 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dca77c0e-e4f7-408a-b5bb-f52ce7b9ea74",
   "metadata": {},
   "source": [
    "Prerequisite: Install transformers: <br> \n",
    "Link https://github.com/ThilinaRajapakse/simpletransformers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6098558-eb0a-451d-a943-c8ec3ec949a4",
   "metadata": {},
   "source": [
    "BERT models: we are using Finnish BERT models, and more models can be explored here: <br>\n",
    "Link: https://huggingface.co/models?sort=downloads <br>\n",
    "Use the search function to explore!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "281e1040-bbf5-4993-bace-39477adb765a",
   "metadata": {},
   "source": [
    "Note: This program runs on a CPU, and one can add cuda support for processing on a GPU. <br>\n",
    "Remove \"use_cuda=False\" from the ClassificationModel instance <br>\n",
    "Install: <br>\n",
    "conda install pytorch>=1.6 cudatoolkit=11.0 -c pytorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e4be247f-0a40-479a-961f-1781d75951e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b60587a0-d119-444f-a427-4b606cbc1447",
   "metadata": {},
   "outputs": [],
   "source": [
    "class_list = ['0','1'] # 0 is descriptive; 1 is economic-industrial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e26ab612-e176-4bc1-beab-aa1bfd7282ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# read the data\n",
    "df = pd.read_csv('berries_class_binary.csv', sep=';', lineterminator='\\n',encoding='utf8',names=[\"pred_class\", \"ocr_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a353c66-b455-45e4-a041-25f1aece9793",
   "metadata": {},
   "source": [
    "Note: It is always nice to use verified OCR-ed texts, free from errors, for training. For testing, they can have errors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b0804d1c-6721-4a9b-acd2-b52f8334a009",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['ocr_text'] = df['ocr_text'].str.replace('\\r', \"\") # clean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "35fa114d-8d40-43b1-810e-6c4cc4b571d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['ocr_text'] = df['ocr_text'].str.replace('\\n', \"\") # clean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2717c649-d81b-43fc-9fe5-156d3e7f2504",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pred_class</th>\n",
       "      <th>ocr_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Säten tallan laillifen ebesroasta= uksen uhall...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>Käärmeen pistolta pelastunut. Eräs kaupnntimme...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>Kadonnut. Talollisen lesken Oma Tirkkosen, Lcp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>Lapsi kadonnut- Sunnuntaina t. k. 16 p:nä ilta...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>Kertomus Lllppccnilln- Nllu MMachtoiscstll Pal...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   pred_class                                           ocr_text\n",
       "0           0  Säten tallan laillifen ebesroasta= uksen uhall...\n",
       "1           0  Käärmeen pistolta pelastunut. Eräs kaupnntimme...\n",
       "2           0  Kadonnut. Talollisen lesken Oma Tirkkosen, Lcp...\n",
       "3           0  Lapsi kadonnut- Sunnuntaina t. k. 16 p:nä ilta...\n",
       "4           0  Kertomus Lllppccnilln- Nllu MMachtoiscstll Pal..."
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "87262a00-c4fd-4384-bf5e-57c01411d48e",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[['ocr_text','pred_class']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8be76c85-543f-498e-be90-de4bc9b57c02",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(415, 2)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ocr_text</th>\n",
       "      <th>pred_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Säten tallan laillifen ebesroasta= uksen uhall...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Käärmeen pistolta pelastunut. Eräs kaupnntimme...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Kadonnut. Talollisen lesken Oma Tirkkosen, Lcp...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Lapsi kadonnut- Sunnuntaina t. k. 16 p:nä ilta...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Kertomus Lllppccnilln- Nllu MMachtoiscstll Pal...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            ocr_text  pred_class\n",
       "0  Säten tallan laillifen ebesroasta= uksen uhall...           0\n",
       "1  Käärmeen pistolta pelastunut. Eräs kaupnntimme...           0\n",
       "2  Kadonnut. Talollisen lesken Oma Tirkkosen, Lcp...           0\n",
       "3  Lapsi kadonnut- Sunnuntaina t. k. 16 p:nä ilta...           0\n",
       "4  Kertomus Lllppccnilln- Nllu MMachtoiscstll Pal...           0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(df.shape)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d0c77d55-2995-43a7-9120-d40bb549ab7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7caaf718-237e-4b3d-afe4-4c55513f5192",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train shape:  (373, 2)\n",
      "test shape:  (42, 2)\n"
     ]
    }
   ],
   "source": [
    "train_df, test_df = train_test_split(df, test_size=0.10) # 90% training and 10% testing\n",
    "print('train shape: ',train_df.shape)\n",
    "print('test shape: ',test_df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b674b25-9c4f-4d4e-8272-c30ec19e3488",
   "metadata": {},
   "source": [
    "# Load pre-trained model\n",
    "\n",
    "Create a ClassificationModel instance with parameters:</br>\n",
    "\n",
    "Architecture (e.g. \"bert\") </br>\n",
    "Pre-trained model (\"TurkuNLP/bert-base-finnish-cased-v1\")</br>\n",
    "No. of class labels (2)</br>\n",
    "Hyperparameter for training (train_args)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "422d2960-ea7c-4b0a-8333-975c22025787",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at TurkuNLP/bert-base-finnish-cased-v1 were not used when initializing BertForSequenceClassification: ['cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.weight']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at TurkuNLP/bert-base-finnish-cased-v1 and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "from simpletransformers.classification import ClassificationModel\n",
    "\n",
    "# define hyperparameter\n",
    "train_args ={\"reprocess_input_data\": True,\n",
    "             \"fp16\":False,\n",
    "             \"num_train_epochs\": 4}\n",
    "\n",
    "# Create a ClassificationModel\n",
    "model = ClassificationModel(\n",
    "    \"bert\", \"TurkuNLP/bert-base-finnish-cased-v1\",\n",
    "    num_labels=2,\n",
    "    use_cuda=False, #cpu only\n",
    "    args=train_args\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44a30868-fa88-48c7-8a45-f612bee29892",
   "metadata": {},
   "source": [
    "# Model training/fine-tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "937aafac-567c-4ffc-ad02-e6923cec64d5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ektva999/miniforge3/envs/st/lib/python3.9/site-packages/simpletransformers/classification/classification_model.py:585: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3fa9c4793aeb4651bc3a493a50aafac7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/373 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ektva999/miniforge3/envs/st/lib/python3.9/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use thePyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2746a84e0f8b44c2876fac16b0984e55",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e26cc2d5df51465ea0b17cff54720f09",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Running Epoch 0 of 4:   0%|          | 0/47 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "65d24873a41341f8a9ee0aa422b8da4a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Running Epoch 1 of 4:   0%|          | 0/47 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7480786fee514c6bbb26fbf64b58594d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Running Epoch 2 of 4:   0%|          | 0/47 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a4719c342aee4098b5e0ae53a51b2a39",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Running Epoch 3 of 4:   0%|          | 0/47 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(188, 0.153899671698108)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.train_model(train_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b215b257-846e-4391-8253-1a9e358983f3",
   "metadata": {},
   "source": [
    "# Evaluate the results of training\n",
    "\n",
    "Using a simple helper function f1_multiclass(), which is used to calculate the f1_score. <br/>\n",
    "Derived from https://www.philschmid.de/bert-text-classification-in-a-different-language"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "be3dd9bd-2d00-4ec9-89c3-707deb5338ce",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ektva999/miniforge3/envs/st/lib/python3.9/site-packages/simpletransformers/classification/classification_model.py:1426: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "04f1afa391a04809830fec9535358b7e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/42 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7cfe8f8694a64c95b9b3e2cd4ecf5467",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Running Evaluation:   0%|          | 0/6 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import f1_score, accuracy_score\n",
    "\n",
    "def f1_multiclass(labels, preds):\n",
    "    return f1_score(labels, preds, average='micro')\n",
    "\n",
    "result, model_outputs, wrong_predictions = model.eval_model(test_df, f1=f1_multiclass, acc=accuracy_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7bc4028-4c43-411c-b495-55fe082a59f1",
   "metadata": {},
   "source": [
    "# Results!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "6d1e0d9e-2663-4d9a-afd0-8570e43aaeb6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mcc': 0.9468641529479986,\n",
       " 'tp': 13,\n",
       " 'tn': 28,\n",
       " 'fp': 1,\n",
       " 'fn': 0,\n",
       " 'auroc': 0.9973474801061009,\n",
       " 'auprc': 0.9945054945054943,\n",
       " 'f1': 0.9761904761904762,\n",
       " 'acc': 0.9761904761904762,\n",
       " 'eval_loss': 0.07071196141381127}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "8d1b1a85-bbb3-4f14-a63f-3ec3f9419f00",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The accuracy is 0.9761904761904762\n"
     ]
    }
   ],
   "source": [
    "print(\"The accuracy is\",result[\"acc\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "cc382cd8-3a02-4592-bdd4-cdbafb12960c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#print(wrong_predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bf73765-ec39-4dda-9323-fc46b27f9b04",
   "metadata": {},
   "source": [
    "# Save the trained model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb9ff538-58b9-4ba7-8f4c-3c7184f8db27",
   "metadata": {},
   "source": [
    "The model is automatically saved in the default directory \"outputs\" after every 2000 steps and in the end of the training."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40e92dc4-4b65-4188-be69-f164055c46f8",
   "metadata": {},
   "source": [
    "# Predict!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff494895-7bda-4399-9dab-05f1ccbc921d",
   "metadata": {},
   "source": [
    "Use an unseen berry corpus' OCR-ed text to predict the category (economic or industrial) that it belongs to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "095e8447-0192-4731-a90b-b477658937f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from simpletransformers.classification import ClassificationModel\n",
    "\n",
    "# define hyperparameter\n",
    "train_args ={\"reprocess_input_data\": True,\n",
    "             \"fp16\":False,\n",
    "             \"num_train_epochs\": 4}\n",
    "\n",
    "# Create a ClassificationModel with our trained model\n",
    "model = ClassificationModel(\n",
    "    \"bert\", 'outputs/',\n",
    "    num_labels=2,\n",
    "    use_cuda=False, # cpu only\n",
    "    args=train_args\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "618aea70-c9ad-4de0-95f1-202dbd0528b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "class_list = ['0','1']\n",
    "#class_list = ['ECONOMIC','INDUSTRIAL']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "bd39cbcf-5933-49e0-ae13-f80e92d6354e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d86882d7b8db4b178e77acd79e2aa2bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c3df4caefc944f99913232eb4278e791",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "# test_ocr1 is taken from excel file, last row\n",
    "test_ocr1 = \"Ilmoitus. Kaikenlainen metsästäminen, kalastaminen, marjain keruu ja metsien raihnaaminen falon< uhalla kielletään omistamiltani metsiltä ja rantamilta, Ilmarila, Rauhala ja loukola (Pört« sykänlahti, Pörtsykän-niemi ja Kolusalmi.) Samallainen kielto koskee entistä Kapakka» saarta nykyistä Walamoa. Käkisalmella 21 päiwänä heiniik. 1905. A. W. Mausnerus. Antti Ahtiainen.\"\n",
    "\n",
    "predictions, raw_outputs = model.predict([test_ocr1])\n",
    "\n",
    "print(class_list[predictions[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f635d75d-c6cf-40cf-9ebe-fee26199b899",
   "metadata": {},
   "source": [
    "# Expert's reflection on test_ocr1:\n",
    "Berry picking is forbidding on this person’s land. Should be in category 0 (thus descriptive, non-economic)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "45c99705-41ae-4ee1-b5bf-6982e0030ec4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "99720495b3894cefb3a18a604e2ae168",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a559870a27a743e3943030ce95d5c470",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "# test_ocr2 is a \"seen\" category 0 used only for verification\n",
    "test_ocr2 = \"Säten tallan laillifen ebesroasta= uksen uhalla kalastamisen, metsästämisen, tuin myös kaikenlaisen metsän haaskuun. etenkin marjan poimimisen Kukkarin saarista Sumasmeden rannalta. @nonlat)besfa Huh- tikuun 29 p. 1885. Aleks. Koponen.\"\n",
    "predictions, raw_outputs = model.predict([test_ocr2])\n",
    "\n",
    "print(class_list[predictions[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "fdad9670-da5f-4f36-83b4-8d990eb6e952",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "791709b14cdb489f91ffd6107be02018",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d36f8959a52641958a9a7462c52ef50f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    }
   ],
   "source": [
    "# test_ocr3 is a \"seen\" category 1 used only for verification\n",
    "test_ocr3 = \"Puolukkain wienti. Waasasta on taas hiljan lähetetty ..Iris laimassa Satsaan 1.100 laatikkoa puolukoita, kussakin laatikossa 22 a 23 kappaa. Walmiina lähctettämätsi ..Björn laimalla on miclä 1,900 laatikkoa, sisältäen yhteenjä noin 40,000 tappaa. Kun puolukkain hinta on 20 p. kapalta, omat ne siis samassa hinnassa kuin kaurat. — W. T.\"\n",
    "predictions, raw_outputs = model.predict([test_ocr3])\n",
    "\n",
    "print(class_list[predictions[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "3148ac44-dd38-4a9a-b9b8-24d8a0512e93",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ocr_text</th>\n",
       "      <th>pred_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>350</th>\n",
       "      <td>Marjojen säilljttäminen ja (täyttäminen ta-lou...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>320</th>\n",
       "      <td>Puolukkain wienti. Waasasta on taas hiljan läh...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>Susi on taas tappanut lapsen. Torppari Isat Ha...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>153</th>\n",
       "      <td>Keisarillisten olosta Barösundissa kirjoitetaa...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>Käärmeenpistoon kuollut. Itäwä tapaus satwi Ka...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              ocr_text  pred_class\n",
       "350  Marjojen säilljttäminen ja (täyttäminen ta-lou...           1\n",
       "320  Puolukkain wienti. Waasasta on taas hiljan läh...           1\n",
       "39   Susi on taas tappanut lapsen. Torppari Isat Ha...           0\n",
       "153  Keisarillisten olosta Barösundissa kirjoitetaa...           0\n",
       "85   Käärmeenpistoon kuollut. Itäwä tapaus satwi Ka...           0"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.head() #to visualise test samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "8b56a007-2956-4b58-b7d4-a143bae03b58",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0aa3a2edc2504ce7b579924bbd152d7a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9aee4d61f26d4a21947312a0469d2303",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "# test_ocr4 is an unseen test sample no. 39\n",
    "test_ocr4 = \"Susi on taas tappanut lapsen. Torppari Isak Hartman'in Wehmalaisten kylässä Karjalassa !> wuotias poika on joutunut suden saaliiksi wiime kuun 22 p:nä. Onneton oli marjoja poimimassa nuoremman »veljensä kanssa, kun peto hänen kohtasi ja tappoi — niin on taaskin luettaivana Turun sanomissa. Tämä on jo sitte wiime huhtikuun kuudes lapsi, joka meidän pienessä maassamme «n joutunut suden saaliiksi. Woi kauheata!\"\n",
    "predictions, raw_outputs = model.predict([test_ocr4])\n",
    "\n",
    "print(class_list[predictions[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d98703de-976b-4c1a-ac0a-fab5a74e945d",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}