determine_training_species.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os.path\n",
    "import pandas as pd\n",
    "\n",
    "busco_stats_path = \"data/WBPS19_busco_stats.tsv\"\n",
    "outdir = \"data/helixer_training_species/\"\n",
    "\n",
    "full_df = pd.read_csv(busco_stats_path, delimiter=\"\\t\")\n",
    "\n",
    "# Take genus to be first part of species name\n",
    "full_df[\"genus\"] = full_df[\"genome\"].str.split(\"_\").apply(lambda x: x[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A number of species failed pre-processing by GeenuFF, so sould be removed from contention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(outdir, \"failed_set.txt\")) as f:\n",
    "    fail_set = f.read().splitlines()\n",
    "full_df = full_df[~full_df[\"genome\"].isin(fail_set)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Determinine candidates for rerunning Helixer/BRAKER on once a good model has been trained. These have the biggest difference in quality of assembly (generally good) vs quality of existing annotation (generally poor). If we were to take a species with a poor annotation but an equally poor assembly, we would understandably not gain much improved BUSCO scores regardless of the annotation tool we choose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>genome</th>\n",
       "      <th>C_as</th>\n",
       "      <th>C_an</th>\n",
       "      <th>C_diff</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>106</th>\n",
       "      <td>heterorhabditis_bacteriophora_prjna13977</td>\n",
       "      <td>87.1</td>\n",
       "      <td>31.4</td>\n",
       "      <td>55.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>231</th>\n",
       "      <td>teladorsagia_circumcincta_prjna72569</td>\n",
       "      <td>71.8</td>\n",
       "      <td>38.4</td>\n",
       "      <td>33.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ancylostoma_ceylanicum_prjna72583</td>\n",
       "      <td>87.2</td>\n",
       "      <td>58.9</td>\n",
       "      <td>28.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>globodera_pallida_prjna702104</td>\n",
       "      <td>59.5</td>\n",
       "      <td>38.6</td>\n",
       "      <td>20.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>angiostrongylus_cantonensis_prjna350391</td>\n",
       "      <td>83.6</td>\n",
       "      <td>67.4</td>\n",
       "      <td>16.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       genome  C_as  C_an  C_diff\n",
       "106  heterorhabditis_bacteriophora_prjna13977  87.1  31.4    55.7\n",
       "231      teladorsagia_circumcincta_prjna72569  71.8  38.4    33.4\n",
       "5           ancylostoma_ceylanicum_prjna72583  87.2  58.9    28.3\n",
       "86              globodera_pallida_prjna702104  59.5  38.6    20.9\n",
       "8     angiostrongylus_cantonensis_prjna350391  83.6  67.4    16.2"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_df[\"C_diff\"] = full_df[\"C_as\"] - full_df[\"C_an\"]\n",
    "reann_set = full_df.sort_values(\"C_diff\", ascending=False).head(5)\n",
    "reann_set[\"genome\"].to_csv(os.path.join(outdir, \"reannotation_set.txt\"), sep=\"\\n\", index=False, header=False)\n",
    "reann_set[[\"genome\", \"C_as\", \"C_an\", \"C_diff\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the training set and validation sets, we should ensure a minimum BUSCO Completeness of **60** (as a broad threshold), and that the species isn't already earmarked for reannotation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = full_df[(full_df[\"C_an\"] >= 60) & (~full_df[\"genome\"].isin(reann_set[\"genome\"]))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For training set, we'll pick one from each genus, prioritising the one with the highest annotation BUSCO Completeness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>genome</th>\n",
       "      <th>C_an</th>\n",
       "      <th>D_an</th>\n",
       "      <th>F_an</th>\n",
       "      <th>M_an</th>\n",
       "      <th>N_an</th>\n",
       "      <th>C_as</th>\n",
       "      <th>D_as</th>\n",
       "      <th>F_as</th>\n",
       "      <th>M_as</th>\n",
       "      <th>N_as</th>\n",
       "      <th>genus</th>\n",
       "      <th>C_diff</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>caenorhabditis_briggsae_qx1410prjna784955</td>\n",
       "      <td>98.5</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.6</td>\n",
       "      <td>982</td>\n",
       "      <td>98.5</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.7</td>\n",
       "      <td>0.8</td>\n",
       "      <td>982</td>\n",
       "      <td>caenorhabditis</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264</th>\n",
       "      <td>wuchereria_bancrofti_prjna275548</td>\n",
       "      <td>98.2</td>\n",
       "      <td>2.3</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.9</td>\n",
       "      <td>982</td>\n",
       "      <td>97.5</td>\n",
       "      <td>1.9</td>\n",
       "      <td>1.7</td>\n",
       "      <td>0.8</td>\n",
       "      <td>982</td>\n",
       "      <td>wuchereria</td>\n",
       "      <td>-0.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <td>loa_loa_prjna37757</td>\n",
       "      <td>96.3</td>\n",
       "      <td>0.7</td>\n",
       "      <td>3.7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>982</td>\n",
       "      <td>96.0</td>\n",
       "      <td>0.7</td>\n",
       "      <td>3.7</td>\n",
       "      <td>0.3</td>\n",
       "      <td>982</td>\n",
       "      <td>loa</td>\n",
       "      <td>-0.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>cercopithifilaria_johnstoni_prjeb47283</td>\n",
       "      <td>95.9</td>\n",
       "      <td>0.7</td>\n",
       "      <td>2.3</td>\n",
       "      <td>1.8</td>\n",
       "      <td>982</td>\n",
       "      <td>94.4</td>\n",
       "      <td>0.8</td>\n",
       "      <td>4.2</td>\n",
       "      <td>1.4</td>\n",
       "      <td>982</td>\n",
       "      <td>cercopithifilaria</td>\n",
       "      <td>-1.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>diploscapter_coronatus_prjdb3143</td>\n",
       "      <td>94.4</td>\n",
       "      <td>88.9</td>\n",
       "      <td>3.6</td>\n",
       "      <td>2.0</td>\n",
       "      <td>982</td>\n",
       "      <td>91.5</td>\n",
       "      <td>67.9</td>\n",
       "      <td>4.6</td>\n",
       "      <td>3.9</td>\n",
       "      <td>982</td>\n",
       "      <td>diploscapter</td>\n",
       "      <td>-2.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        genome  C_an  D_an  F_an  M_an  N_an  \\\n",
       "34   caenorhabditis_briggsae_qx1410prjna784955  98.5   0.6   0.9   0.6   982   \n",
       "264           wuchereria_bancrofti_prjna275548  98.2   2.3   0.9   0.9   982   \n",
       "117                         loa_loa_prjna37757  96.3   0.7   3.7   0.0   982   \n",
       "52      cercopithifilaria_johnstoni_prjeb47283  95.9   0.7   2.3   1.8   982   \n",
       "62            diploscapter_coronatus_prjdb3143  94.4  88.9   3.6   2.0   982   \n",
       "\n",
       "     C_as  D_as  F_as  M_as  N_as              genus  C_diff  \n",
       "34   98.5   0.6   0.7   0.8   982     caenorhabditis     0.0  \n",
       "264  97.5   1.9   1.7   0.8   982         wuchereria    -0.7  \n",
       "117  96.0   0.7   3.7   0.3   982                loa    -0.3  \n",
       "52   94.4   0.8   4.2   1.4   982  cercopithifilaria    -1.5  \n",
       "62   91.5  67.9   4.6   3.9   982       diploscapter    -2.9  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set = df.sort_values(\"C_an\", ascending=False).drop_duplicates(\"genus\")\n",
    "train_set[\"genome\"].sort_values().to_csv(os.path.join(outdir, \"training_set.txt\"), sep=\"\\n\", index=False, header=False)\n",
    "train_set.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For validation set, we'll pick the second best species from each criteria, based on previous criteria."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_9827/1398248434.py:1: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
      "  valid_set = df.sort_values(\"C_an\", ascending=False)[~df[\"genome\"].isin(train_set[\"genome\"])].drop_duplicates(\"genus\")\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>genome</th>\n",
       "      <th>C_an</th>\n",
       "      <th>D_an</th>\n",
       "      <th>F_an</th>\n",
       "      <th>M_an</th>\n",
       "      <th>N_an</th>\n",
       "      <th>C_as</th>\n",
       "      <th>D_as</th>\n",
       "      <th>F_as</th>\n",
       "      <th>M_as</th>\n",
       "      <th>N_as</th>\n",
       "      <th>genus</th>\n",
       "      <th>C_diff</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>caenorhabditis_zanzibari_prjeb12596</td>\n",
       "      <td>98.5</td>\n",
       "      <td>1.9</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>982</td>\n",
       "      <td>98.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>1.2</td>\n",
       "      <td>0.6</td>\n",
       "      <td>982</td>\n",
       "      <td>caenorhabditis</td>\n",
       "      <td>-0.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>loa_loa_prjna246086</td>\n",
       "      <td>94.7</td>\n",
       "      <td>1.3</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.8</td>\n",
       "      <td>982</td>\n",
       "      <td>97.5</td>\n",
       "      <td>1.3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>982</td>\n",
       "      <td>loa</td>\n",
       "      <td>2.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>176</th>\n",
       "      <td>pristionchus_maxplancki_prjeb27334</td>\n",
       "      <td>93.5</td>\n",
       "      <td>13.4</td>\n",
       "      <td>4.2</td>\n",
       "      <td>2.3</td>\n",
       "      <td>982</td>\n",
       "      <td>90.6</td>\n",
       "      <td>9.4</td>\n",
       "      <td>5.5</td>\n",
       "      <td>3.9</td>\n",
       "      <td>982</td>\n",
       "      <td>pristionchus</td>\n",
       "      <td>-2.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>151</th>\n",
       "      <td>oscheius_tipulae_prjeb15512</td>\n",
       "      <td>89.8</td>\n",
       "      <td>3.4</td>\n",
       "      <td>5.7</td>\n",
       "      <td>4.5</td>\n",
       "      <td>982</td>\n",
       "      <td>89.4</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.2</td>\n",
       "      <td>7.4</td>\n",
       "      <td>982</td>\n",
       "      <td>oscheius</td>\n",
       "      <td>-0.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>249</th>\n",
       "      <td>trichinella_spiralis_prjna257433</td>\n",
       "      <td>88.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.7</td>\n",
       "      <td>5.4</td>\n",
       "      <td>982</td>\n",
       "      <td>89.7</td>\n",
       "      <td>2.3</td>\n",
       "      <td>7.1</td>\n",
       "      <td>3.2</td>\n",
       "      <td>982</td>\n",
       "      <td>trichinella</td>\n",
       "      <td>0.8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  genome  C_an  D_an  F_an  M_an  N_an  C_as  \\\n",
       "51   caenorhabditis_zanzibari_prjeb12596  98.5   1.9   1.0   0.5   982  98.2   \n",
       "116                  loa_loa_prjna246086  94.7   1.3   3.5   1.8   982  97.5   \n",
       "176   pristionchus_maxplancki_prjeb27334  93.5  13.4   4.2   2.3   982  90.6   \n",
       "151          oscheius_tipulae_prjeb15512  89.8   3.4   5.7   4.5   982  89.4   \n",
       "249     trichinella_spiralis_prjna257433  88.9   3.0   5.7   5.4   982  89.7   \n",
       "\n",
       "     D_as  F_as  M_as  N_as           genus  C_diff  \n",
       "51    1.3   1.2   0.6   982  caenorhabditis    -0.3  \n",
       "116   1.3   2.0   0.5   982             loa     2.8  \n",
       "176   9.4   5.5   3.9   982    pristionchus    -2.9  \n",
       "151   3.0   3.2   7.4   982        oscheius    -0.4  \n",
       "249   2.3   7.1   3.2   982     trichinella     0.8  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_set = df.sort_values(\"C_an\", ascending=False)[~df[\"genome\"].isin(train_set[\"genome\"])].drop_duplicates(\"genus\")\n",
    "valid_set[\"genome\"].sort_values().to_csv(os.path.join(outdir, \"validation_set.txt\"), sep=\"\\n\", index=False, header=False)\n",
    "valid_set.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total species:\t\t\t263\n",
      "Species in...\n",
      "\treannotation set:\t5\n",
      "\ttraining set:\t\t84\n",
      "\tvalidation set:\t\t40\n"
     ]
    }
   ],
   "source": [
    "print(\"Total species:\\t\\t\\t{}\".format(len(full_df)))\n",
    "print(\"Species in...\")\n",
    "print(\"\\treannotation set:\\t{}\".format(len(reann_set)))\n",
    "print(\"\\ttraining set:\\t\\t{}\".format(len(train_set)))\n",
    "print(\"\\tvalidation set:\\t\\t{}\".format(len(valid_set)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top 40 Training Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_set_40 = train_set.head(40)\n",
    "train_set_40[\"genome\"].sort_values().to_csv(os.path.join(outdir, \"training_set_top40.txt\"), sep=\"\\n\", index=False, header=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll rebuild the validation set now that more species will be available after being filtered from the training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_9827/3457853504.py:1: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
      "  valid_set_40 = df.sort_values(\"C_an\", ascending=False)[~df[\"genome\"].isin(train_set_40[\"genome\"])].drop_duplicates(\"genus\")\n"
     ]
    }
   ],
   "source": [
    "valid_set_40 = df.sort_values(\"C_an\", ascending=False)[~df[\"genome\"].isin(train_set_40[\"genome\"])].drop_duplicates(\"genus\")\n",
    "valid_set_40[\"genome\"].sort_values().to_csv(os.path.join(outdir, \"validation_set_top40.txt\"), sep=\"\\n\", index=False, header=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Species in...\n",
      "\ttraining set:\t\t40\n",
      "\tvalidation set:\t\t67\n"
     ]
    }
   ],
   "source": [
    "print(\"Species in...\")\n",
    "print(\"\\ttraining set:\\t\\t{}\".format(len(train_set_40)))\n",
    "print(\"\\tvalidation set:\\t\\t{}\".format(len(valid_set_40)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mixed quality training and validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: ylabel='Frequency'>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAGdCAYAAAAIbpn/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAgOElEQVR4nO3de3BU5f3H8c+SkCVgAgiGBAlJCChCooJYykWrhSoXFaFjBaGGS61oWqKIQmSQUoQErRlsbUEdBazcZIqXagEhgMiI3IUybcNVEiAI5ZILSIDs8/vDYX9dEkiy2eTwbN6vmcw0Zw8n34fTJW/Pnk1cxhgjAAAAC9VzegAAAAB/ETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArBXq9ADV4fF4dOTIEUVERMjlcjk9DgAAqARjjIqKitSyZUvVq1e9aypWh8yRI0cUGxvr9BgAAMAPeXl5atWqVbWOYXXIRERESPrhLyIyMtLhaQAAQGUUFhYqNjbW+328OqwOmUsvJ0VGRhIyAABYJhC3hXCzLwAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArBXq9AAAgOAVP+GzGjnut5n9a+S4sA9XZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtR0OmtLRUkyZNUkJCgsLDw5WYmKipU6fKGOPkWAAAwBKhTn7xGTNmaNasWZo3b546duyoLVu2aMSIEWrcuLHGjBnj5GgAAMACjobMV199pQEDBqh///6SpPj4eC1cuFCbNm1yciwAAGAJR19a6t69u7Kzs7V7925J0o4dO7R+/Xr17du33P1LSkpUWFjo8wEAAOouR6/ITJgwQYWFhWrfvr1CQkJUWlqqadOmaejQoeXun5GRoSlTptTylACcEj/hsxo79reZ/Wvs2DWlpv4+bPy7AC5x9IrMBx98oPnz52vBggXatm2b5s2bpz/84Q+aN29eufunp6eroKDA+5GXl1fLEwMAgGuJo1dknn/+eU2YMEGDBw+WJCUnJ+vgwYPKyMhQSkpKmf3dbrfcbndtjwkAAK5Rjl6ROXv2rOrV8x0hJCREHo/HoYkAAIBNHL0i8+CDD2ratGlq3bq1OnbsqO3btysrK0sjR450ciwAAGAJR0PmT3/6kyZNmqSnn35ax44dU8uWLfXkk0/qpZdecnIsAABgCUdDJiIiQjNnztTMmTOdHAMAAFiK37UEAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqOh8zhw4c1bNgwNWvWTOHh4UpOTtaWLVucHgsAAFgg1MkvfurUKfXo0UP33nuvli1bphtuuEF79uxR06ZNnRwLAABYwtGQmTFjhmJjYzVnzhzvtoSEBAcnAgAANnH0paVPPvlEXbp00SOPPKKoqCh16tRJb7/9tpMjAQAAizgaMvv379esWbPUrl07rVixQk899ZTGjBmjefPmlbt/SUmJCgsLfT4AAEDd5ehLSx6PR126dNH06dMlSZ06ddKuXbs0e/ZspaSklNk/IyNDU6ZMqe0xgaAQP+Ezp0cAgIBz9IpMTEyMOnTo4LPtlltuUW5ubrn7p6enq6CgwPuRl5dXG2MCAIBrlKNXZHr06KGcnByfbbt371ZcXFy5+7vdbrnd7toYDQAAWMDRKzLPPvusvv76a02fPl179+7VggUL9NZbbyk1NdXJsQAAgCUcDZk777xTH374oRYuXKikpCRNnTpVM2fO1NChQ50cCwAAWMLRl5Yk6YEHHtADDzzg9BgAAMBCjv+KAgAAAH8RMgAAwFqEDAAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALCWXyGzf//+QM8BAABQZX6FTNu2bXXvvffq/fff17lz5wI9EwAAQKX4FTLbtm3TrbfeqrFjxyo6OlpPPvmkNm3aFOjZAAAArsqvkLn99tv1+uuv68iRI3r33XeVn5+vnj17KikpSVlZWTp+/Hig5wQAACijWjf7hoaGatCgQVqyZIlmzJihvXv3aty4cYqNjdXjjz+u/Pz8QM0JAABQRrVCZsuWLXr66acVExOjrKwsjRs3Tvv27dPKlSt15MgRDRgwIFBzAgAAlBHqzx/KysrSnDlzlJOTo379+um9995Tv379VK/eD12UkJCguXPnKj4+PpCzAgAA+PArZGbNmqWRI0dq+PDhiomJKXefqKgovfPOO9UaDgAA4Gr8Cpk9e/ZUuE9YWJhSUlL8OTwAAECl+HWPzJw5c7RkyZIy25csWaJ58+ZVeygAAIDK8CtkMjIy1Lx58zLbo6KiNH369GoPBQAAUBl+hUxubq4SEhLKbI+Li1Nubm61hwIAAKgMv0ImKipKO3fuLLN9x44datasWbWHAgAAqAy/QmbIkCEaM2aM1qxZo9LSUpWWlmr16tVKS0vT4MGDAz0jAABAufx619LUqVP17bffqlevXgoN/eEQHo9Hjz/+OPfIAACAWuNXyISFhWnx4sWaOnWqduzYofDwcCUnJysuLi7Q8wEAAFyRXyFzyU033aSbbropULMAAABUiV8hU1paqrlz5yo7O1vHjh2Tx+PxeXz16tUBGQ4AAOBq/AqZtLQ0zZ07V/3791dSUpJcLleg5wIAAKiQXyGzaNEiffDBB+rXr1+g5wEAAKg0v95+HRYWprZt2wZ6FgAAgCrxK2See+45vf766zLGBHoeAACASvPrpaX169drzZo1WrZsmTp27Kj69ev7PL506dKADAdcq+InfFZjx/42s3+NHRv/rybPoW34u4DN/AqZJk2aaODAgYGeBQAAoEr8Cpk5c+YEeg4AAIAq8+seGUm6ePGiVq1apTfffFNFRUWSpCNHjqi4uDhgwwEAAFyNX1dkDh48qD59+ig3N1clJSX62c9+poiICM2YMUMlJSWaPXt2oOcEAAAow68rMmlpaerSpYtOnTql8PBw7/aBAwcqOzs7YMMBAABcjV9XZL788kt99dVXCgsL89keHx+vw4cPB2QwAACAivh1Rcbj8ai0tLTM9kOHDikiIqLaQwEAAFSGXyFz3333aebMmd7PXS6XiouLNXnyZH5tAQAAqDV+vbT02muv6f7771eHDh107tw5PfbYY9qzZ4+aN2+uhQsXBnpGAACAcvkVMq1atdKOHTu0aNEi7dy5U8XFxRo1apSGDh3qc/MvAABATfIrZCQpNDRUw4YNC+QsAAAAVeJXyLz33ntXffzxxx/3axgAAICq8Ctk0tLSfD6/cOGCzp49q7CwMDVs2JCQAQAAtcKvdy2dOnXK56O4uFg5OTnq2bMnN/sCAIBa4/fvWrpcu3btlJmZWeZqDQAAQE0JWMhIP9wAfOTIkUAeEgAA4Ir8ukfmk08+8fncGKP8/Hy98cYb6tGjR0AGAwAAqIhfIfPwww/7fO5yuXTDDTfopz/9qV577bVAzAUAAFAhv0LG4/EEeg4AAIAqC+g9MgAAALXJrysyY8eOrfS+WVlZ/nwJAACACvkVMtu3b9f27dt14cIF3XzzzZKk3bt3KyQkRJ07d/bu53K5AjMlAABAOfwKmQcffFARERGaN2+emjZtKumHH5I3YsQI3XXXXXruuecCOiQAAEB5/LpH5rXXXlNGRoY3YiSpadOmevnll3nXEgAAqDV+hUxhYaGOHz9eZvvx48dVVFRU7aEAAAAqw6+QGThwoEaMGKGlS5fq0KFDOnTokP72t79p1KhRGjRoUKBnBAAAKJdf98jMnj1b48aN02OPPaYLFy78cKDQUI0aNUqvvvpqQAcEAAC4Er9CpmHDhvrLX/6iV199Vfv27ZMkJSYmqlGjRgEdDgAA4Gqq9QPx8vPzlZ+fr3bt2qlRo0YyxgRqLgAAgAr5FTInTpxQr169dNNNN6lfv37Kz8+XJI0aNYq3XgMAgFrjV8g8++yzql+/vnJzc9WwYUPv9kcffVTLly8P2HAAAABX49c9Mp9//rlWrFihVq1a+Wxv166dDh48GJDBAAAAKuLXFZkzZ874XIm55OTJk3K73dUeCgAAoDL8Cpm77rpL7733nvdzl8slj8ejV155Rffee2/AhgMAALgav0LmlVde0VtvvaW+ffvq/PnzeuGFF5SUlKR169ZpxowZfg2SmZkpl8ulZ555xq8/DwAA6h6/QiYpKUm7d+9Wz549NWDAAJ05c0aDBg3S9u3blZiYWOXjbd68WW+++aZuvfVWf8YBAAB1VJVv9r1w4YL69Omj2bNna+LEidUeoLi4WEOHDtXbb7+tl19+udrHAwAAdUeVr8jUr19fO3fuDNgAqamp6t+/v3r37l3hviUlJSosLPT5AAAAdZdfb78eNmyY3nnnHWVmZlbriy9atEjbtm3T5s2bK7V/RkaGpkyZUq2viWtP/ITPauS432b2r5HjAnBeTf27UdP4dynw/AqZixcv6t1339WqVat0xx13lPkdS1lZWRUeIy8vT2lpaVq5cqUaNGhQqa+bnp6usWPHej8vLCxUbGxs1YYHAABBo0ohs3//fsXHx2vXrl3q3LmzJGn37t0++7hcrkoda+vWrTp27Jj3OJJUWlqqdevW6Y033lBJSYlCQkJ8/ozb7ebn1AAAAK8qhUy7du2Un5+vNWvWSPrhVxL88Y9/VIsWLar8hXv16qV//vOfPttGjBih9u3ba/z48WUiBgAA4HJVCpnLf7v1smXLdObMGb++cEREhJKSkny2NWrUSM2aNSuzHQAAoDx+/RyZSy4PGwAAgNpUpSsyLperzD0wlb0npjLWrl0bsGMBAIDgV+WXloYPH+694fbcuXMaPXp0mXctLV26NHATAgAAXEGVQiYlJcXn82HDhgV0GAAAgKqoUsjMmTOnpuYAAACosmrd7AsAAOAkQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYK1Qpwe4lsVP+KxGjvttZv8aOS581dT5AwBcO7giAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGs5GjIZGRm68847FRERoaioKD388MPKyclxciQAAGARR0Pmiy++UGpqqr7++mutXLlSFy5c0H333aczZ844ORYAALBEqJNffPny5T6fz507V1FRUdq6davuvvtuh6YCAAC2cDRkLldQUCBJuv7668t9vKSkRCUlJd7PCwsLa2UuAABwbXIZY4zTQ0iSx+PRQw89pNOnT2v9+vXl7vO73/1OU6ZMKbO9oKBAkZGRAZ8pfsJnAT9mTfs2s7/TI1SZjX/PAFBX1MT3lcLCQjVu3Dgg37+vmXctpaamateuXVq0aNEV90lPT1dBQYH3Iy8vrxYnBAAA15pr4qWl3/zmN/r000+1bt06tWrV6or7ud1uud3uWpwMAABcyxwNGWOMfvvb3+rDDz/U2rVrlZCQ4OQ4AADAMo6GTGpqqhYsWKCPP/5YEREROnr0qCSpcePGCg8Pd3I0AABgAUfvkZk1a5YKCgp0zz33KCYmxvuxePFiJ8cCAACWcPylJQAAAH9dM+9aAgAAqCpCBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1Qp0eAHaIn/CZ0yMAAFAGV2QAAIC1CBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgrWsiZP785z8rPj5eDRo0UNeuXbVp0yanRwIAABZwPGQWL16ssWPHavLkydq2bZtuu+023X///Tp27JjTowEAgGuc4yGTlZWlJ554QiNGjFCHDh00e/ZsNWzYUO+++67TowEAgGtcqJNf/Pz589q6davS09O92+rVq6fevXtrw4YNZfYvKSlRSUmJ9/OCggJJUmFhYY3M5yk5WyPHrUn8XQAAAqkmvq9cOqYxptrHcjRk/vvf/6q0tFQtWrTw2d6iRQv95z//KbN/RkaGpkyZUmZ7bGxsjc1om8YznZ4AABBMavL7SlFRkRo3blytYzgaMlWVnp6usWPHej/3eDw6efKkmjVrJpfL5dcxCwsLFRsbq7y8PEVGRgZq1GsSaw1edWm9rDV41aX11vW1GmNUVFSkli1bVvv4joZM8+bNFRISou+++85n+3fffafo6Ogy+7vdbrndbp9tTZo0CcgskZGRQf9/pktYa/CqS+tlrcGrLq23Lq+1uldiLnH0Zt+wsDDdcccdys7O9m7zeDzKzs5Wt27dHJwMAADYwPGXlsaOHauUlBR16dJFP/rRjzRz5kydOXNGI0aMcHo0AABwjXM8ZB599FEdP35cL730ko4eParbb79dy5cvL3MDcE1xu92aPHlymZesghFrDV51ab2sNXjVpfWy1sBxmUC89wkAAMABjv9APAAAAH8RMgAAwFqEDAAAsBYhAwAArFVnQubw4cMaNmyYmjVrpvDwcCUnJ2vLli3ex40xeumllxQTE6Pw8HD17t1be/bscXBi/1W01uHDh8vlcvl89OnTx8GJ/RcfH19mLS6XS6mpqZKkc+fOKTU1Vc2aNdN1112nn//852V+AKMtKlrrPffcU+ax0aNHOzy1f0pLSzVp0iQlJCQoPDxciYmJmjp1qs/vZQmm52xl1htMz9uioiI988wziouLU3h4uLp3767Nmzd7Hw+mc1vRWm0+r+vWrdODDz6oli1byuVy6aOPPvJ5vDLn8eTJkxo6dKgiIyPVpEkTjRo1SsXFxVUbxNQBJ0+eNHFxcWb48OFm48aNZv/+/WbFihVm79693n0yMzNN48aNzUcffWR27NhhHnroIZOQkGC+//57ByevusqsNSUlxfTp08fk5+d7P06ePOng1P47duyYzzpWrlxpJJk1a9YYY4wZPXq0iY2NNdnZ2WbLli3mxz/+senevbuzQ/uporX+5Cc/MU888YTPPgUFBc4O7adp06aZZs2amU8//dQcOHDALFmyxFx33XXm9ddf9+4TLM9ZYyq33mB63v7iF78wHTp0MF988YXZs2ePmTx5somMjDSHDh0yxgTXua1orTaf13/84x9m4sSJZunSpUaS+fDDD30er8x57NOnj7ntttvM119/bb788kvTtm1bM2TIkCrNUSdCZvz48aZnz55XfNzj8Zjo6Gjz6quveredPn3auN1us3DhwtoYMWAqWqsxPzxxBgwYUDsD1bK0tDSTmJhoPB6POX36tKlfv75ZsmSJ9/F///vfRpLZsGGDg1MGxv+u1ZgfQiYtLc3ZoQKkf//+ZuTIkT7bBg0aZIYOHWqMCa7nrDEVr9eY4Hnenj171oSEhJhPP/3UZ3vnzp3NxIkTg+rcVrRWY4LnvF4eMpU5j//617+MJLN582bvPsuWLTMul8scPny40l+7Try09Mknn6hLly565JFHFBUVpU6dOuntt9/2Pn7gwAEdPXpUvXv39m5r3Lixunbtqg0bNjgxst8qWusla9euVVRUlG6++WY99dRTOnHihAPTBtb58+f1/vvva+TIkXK5XNq6dasuXLjgc17bt2+v1q1bW3deL3f5Wi+ZP3++mjdvrqSkJKWnp+vs2bMOTum/7t27Kzs7W7t375Yk7dixQ+vXr1ffvn0lBddzVqp4vZcEw/P24sWLKi0tVYMGDXy2h4eHa/369UF1bita6yXBcF4vV5nzuGHDBjVp0kRdunTx7tO7d2/Vq1dPGzdurPTXcvwn+9aG/fv3a9asWRo7dqxefPFFbd68WWPGjFFYWJhSUlJ09OhRSSrz04RbtGjhfcwWFa1Vkvr06aNBgwYpISFB+/bt04svvqi+fftqw4YNCgkJcXgF/vvoo490+vRpDR8+XJJ09OhRhYWFlfnFojae18tdvlZJeuyxxxQXF6eWLVtq586dGj9+vHJycrR06VLnBvXThAkTVFhYqPbt2yskJESlpaWaNm2ahg4dKklB9ZyVKl6vFDzP24iICHXr1k1Tp07VLbfcohYtWmjhwoXasGGD2rZtG1TntqK1SsFzXi9XmfN49OhRRUVF+TweGhqq66+/vkrnuk6EjMfjUZcuXTR9+nRJUqdOnbRr1y7Nnj3b+809WFRmrYMHD/bun5ycrFtvvVWJiYlau3atevXq5cjcgfDOO++ob9++Afm18Ne68tb661//2vu/k5OTFRMTo169emnfvn1KTEx0Yky/ffDBB5o/f74WLFigjh076ptvvtEzzzyjli1bBt1zVqrceoPpefvXv/5VI0eO1I033qiQkBB17txZQ4YM0datW50eLeAqWmswnVen1ImXlmJiYtShQwefbbfccotyc3MlSdHR0ZJU5t0s3333nfcxW1S01vK0adNGzZs31969e2t6vBpz8OBBrVq1Sr/61a+826Kjo3X+/HmdPn3aZ18bz+v/Km+t5enataskWXlen3/+eU2YMEGDBw9WcnKyfvnLX+rZZ59VRkaGpOB6zkoVr7c8Nj9vExMT9cUXX6i4uFh5eXnatGmTLly4oDZt2gTdub3aWstj83n9X5U5j9HR0Tp27JjP4xcvXtTJkyerdK7rRMj06NFDOTk5Ptt2796tuLg4SVJCQoKio6OVnZ3tfbywsFAbN25Ut27danXW6qporeU5dOiQTpw4oZiYmJoer8bMmTNHUVFR6t+/v3fbHXfcofr16/uc15ycHOXm5lp3Xv9XeWstzzfffCNJVp7Xs2fPql4933+eQkJC5PF4JAXXc1aqeL3lCYbnbaNGjRQTE6NTp05pxYoVGjBgQNCd20vKW2t5guG8SpV7jnbr1k2nT5/2uRK3evVqeTwe73+IVUr171W+9m3atMmEhoaaadOmmT179pj58+ebhg0bmvfff9+7T2ZmpmnSpIn5+OOPzc6dO82AAQOsfLtfRWstKioy48aNMxs2bDAHDhwwq1atMp07dzbt2rUz586dc3h6/5SWlprWrVub8ePHl3ls9OjRpnXr1mb16tVmy5Ytplu3bqZbt24OTBkYV1rr3r17ze9//3uzZcsWc+DAAfPxxx+bNm3amLvvvtuhSasnJSXF3Hjjjd63Iy9dutQ0b97cvPDCC959guU5a0zF6w225+3y5cvNsmXLzP79+83nn39ubrvtNtO1a1dz/vx5Y0xwndurrdX281pUVGS2b99utm/fbiSZrKwss337dnPw4EFjTOXOY58+fUynTp3Mxo0bzfr16027du14+/WV/P3vfzdJSUnG7Xab9u3bm7feesvncY/HYyZNmmRatGhh3G636dWrl8nJyXFo2uq52lrPnj1r7rvvPnPDDTeY+vXrm7i4OPPEE0+Yo0ePOjhx9axYscJIKvd8ff/99+bpp582TZs2NQ0bNjQDBw40+fn5DkwZGFdaa25urrn77rvN9ddfb9xut2nbtq15/vnnrf05MoWFhSYtLc20bt3aNGjQwLRp08ZMnDjRlJSUePcJpudsResNtuft4sWLTZs2bUxYWJiJjo42qamp5vTp097Hg+ncXm2ttp/XNWvWGEllPlJSUowxlTuPJ06cMEOGDDHXXXediYyMNCNGjDBFRUVVmsNlzP/86EgAAACL1Il7ZAAAQHAiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFjr/wCGnT3pwlogEwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.sort_values(\"C_an\", ascending=False).drop_duplicates(\"genus\")[\"C_an\"].plot(kind=\"hist\", bins=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_pool = df.sort_values(\"C_an\", ascending=False).drop_duplicates(\"genus\")\n",
    "sample_size = round(len(df_pool) / 2)\n",
    "train_set_mix = df_pool.sample(sample_size)\n",
    "valid_set_mix = df_pool[~df_pool[\"genome\"].isin(train_set_mix[\"genome\"])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: ylabel='Frequency'>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAGdCAYAAAAIbpn/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAg0UlEQVR4nO3de3BU5f3H8c+SkCVAEq4hQSDhjshFgZZG0aqgcilF6LTKpQZEW5TWIKIYGQVKMahjRjpasI6CVhBkimhtAeUulVuQi2jLHRIgCAXJJiALZJ/fH/7YuiSQZLPJybN5v2Z2pnv2cPJ9ctzk3c3ZxGWMMQIAALBQDacHAAAACBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBakU4PUB4+n0/Hjh1TTEyMXC6X0+MAAIBSMMYoPz9fTZs2VY0a5XtNxeqQOXbsmJo3b+70GAAAIAg5OTlq1qxZuY5hdcjExMRI+v4TERsb6/A0AACgNDwej5o3b+7/Pl4eVofM5R8nxcbGEjIAAFgmFJeFcLEvAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArOVoyEyZMkUulyvg1qFDBydHAgAAFnH8by3dcMMNWrFihf9+ZKTjIwEAAEs4Xg2RkZFKSEhwegwAAGAhx6+R2bt3r5o2bapWrVpp+PDhys7Ovuq+Xq9XHo8n4AYAAKovlzHGOPXBly5dqoKCArVv3165ubmaOnWqjh49ql27dikmJqbI/lOmTNHUqVOLbM/Ly1NsbGxljAwAKMmUuAo8dl7FHRuVxuPxKC4uLiTfvx0NmSudOXNGSUlJyszM1OjRo4s87vV65fV6/fc9Ho+aN29OyABAVULIoAShDBnHr5H5oXr16qldu3bat29fsY+73W653e5KngoAAFRVjl8j80MFBQXav3+/EhMTnR4FAABYwNGQmTBhgtauXatDhw7p888/1+DBgxUREaGhQ4c6ORYAALCEoz9aOnLkiIYOHapTp06pcePG6tWrlzZu3KjGjRs7ORYAALCEoyGzYMECJz88AACwXJW6RgYAAKAsCBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWqjIhM2PGDLlcLo0bN87pUQAAgCWqRMhs2bJFr7/+urp06eL0KAAAwCKOh0xBQYGGDx+uN954Q/Xr13d6HAAAYBHHQ2bs2LEaMGCA+vTpU+K+Xq9XHo8n4AYAAKqvSCc/+IIFC/TFF19oy5Ytpdo/IyNDU6dOreCpADhuSlyFHTr5/PxS7XdoxoAKm6FMKvBzoSl513w4+el/BHXYQ7WC+melcrWZqsz5QqVz7BWZnJwcpaWlad68eapVq3T/1aenpysvL89/y8nJqeApAQBAVebYKzJbt27ViRMn1K1bN/+2wsJCrVu3Tq+++qq8Xq8iIiIC/o3b7Zbb7a7sUQEAQBXlWMj07t1bX375ZcC2UaNGqUOHDpo4cWKRiAEAALiSYyETExOjTp06BWyrU6eOGjZsWGQ7AABAcRx/1xIAAECwHH3X0pXWrFnj9AgAAMAivCIDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArBVUyBw4cCDUcwAAAJRZUCHTpk0b3XHHHXr33Xd1/vz5UM8EAABQKkGFzBdffKEuXbpo/PjxSkhI0G9/+1tt3ry5zMeZNWuWunTpotjYWMXGxiolJUVLly4NZiQAAFANBRUyN954o2bOnKljx47prbfeUm5urnr16qVOnTopMzNTJ0+eLNVxmjVrphkzZmjr1q3KysrSnXfeqUGDBumrr74KZiwAAFDNlOti38jISA0ZMkSLFi3SCy+8oH379mnChAlq3ry5HnjgAeXm5l7z3w8cOFD9+/dX27Zt1a5dO02fPl1169bVxo0byzMWAACoJsoVMllZWXr00UeVmJiozMxMTZgwQfv379enn36qY8eOadCgQaU+VmFhoRYsWKCzZ88qJSWl2H28Xq88Hk/ADQAAVF8uY4wp6z/KzMzUnDlztHv3bvXv318PPfSQ+vfvrxo1/tdFR44cUXJysi5dunTNY3355ZdKSUnR+fPnVbduXc2fP1/9+/cvdt8pU6Zo6tSpRbbn5eUpNja2rMtAVTAlrgKPnRfUP0t++h8hHkQ6NGNAyI8plX7WQ7WGVcjHx/8kn59fIcfl3AWy7vMc5NehcOfxeBQXFxeS799BvSIza9YsDRs2TIcPH9aSJUv0s5/9LCBiJCk+Pl5vvvlmicdq3769tm/frk2bNumRRx5Ramqqvv7662L3TU9PV15env+Wk5MTzPgAACBMRAbzj/bu3VviPlFRUUpNTS3Vfm3atJEkde/eXVu2bNHMmTP1+uuvF9nX7XbL7XaXfWAAABCWgnpFZs6cOVq0aFGR7YsWLdLbb79droF8Pp+8Xm+5jgEAAKqHoEImIyNDjRo1KrI9Pj5ezz//fKmPk56ernXr1unQoUP68ssvlZ6erjVr1mj48OHBjAUAAKqZoH60lJ2drZYtWxbZnpSUpOzs7FIf58SJE/63acfFxalLly5avny57rrrrmDGAgAA1UxQIRMfH6+dO3cqOTk5YPuOHTvUsGHDUh+nNBcDAwAAXE1QP1oaOnSoHnvsMa1evVqFhYUqLCzUqlWrlJaWpvvvvz/UMwIAABQrqFdkpk2bpkOHDql3796KjPz+ED6fTw888ECZrpEBAAAoj6BCJioqSgsXLtS0adO0Y8cORUdHq3PnzkpKSgr1fAAAAFcVVMhc1q5dO7Vr1y5UswAAAJRJUCFTWFiouXPnauXKlTpx4oR8Pl/A46tWrQrJcAAAANcSVMikpaVp7ty5GjBggDp16iSXyxXquQAAAEoUVMgsWLBA77///lX/uCMAAEBlCOrt1z/8+0gAAABOCSpknnjiCc2cOVPGmFDPAwAAUGpB/Whp/fr1Wr16tZYuXaobbrhBNWvWDHh88eLFIRkOAADgWoIKmXr16mnw4MGhngUAAKBMggqZOXPmhHoOAACAMgvqGhlJunTpklasWKHXX39d+fn5kqRjx46poKAgZMMBAABcS1CvyBw+fFh9+/ZVdna2vF6v7rrrLsXExOiFF16Q1+vV7NmzQz0nAABAEUG9IpOWlqYePXro22+/VXR0tH/74MGDtXLlypANBwAAcC1BvSLz2Wef6fPPP1dUVFTA9uTkZB09ejQkgwEAAJQkqFdkfD6fCgsLi2w/cuSIYmJiyj0UAABAaQQVMnfffbdeeeUV/32Xy6WCggJNnjyZP1sAAAAqTVA/Wnr55Zd1zz33qGPHjjp//ryGDRumvXv3qlGjRnrvvfdCPSMAAECxggqZZs2aaceOHVqwYIF27typgoICjR49WsOHDw+4+BcAAKAiBRUykhQZGakRI0aEchYAAIAyCSpk3nnnnWs+/sADDwQ1DAAAQFkEFTJpaWkB9y9evKhz584pKipKtWvXJmQAAEClCOpdS99++23AraCgQLt371avXr242BcAAFSaoP/W0pXatm2rGTNmFHm1BgAAoKKELGSk7y8APnbsWCgPCQAAcFVBXSPz0UcfBdw3xig3N1evvvqqbrnllpAMBgAAUJKgQubee+8NuO9yudS4cWPdeeedevnll0MxFwAAQImCChmfzxfqOQAAAMospNfIAAAAVKagXpEZP358qffNzMwM5kMAAACUKKiQ2bZtm7Zt26aLFy+qffv2kqQ9e/YoIiJC3bp18+/ncrlCMyUAAEAxggqZgQMHKiYmRm+//bbq168v6ftfkjdq1CjdeuuteuKJJ0I6JAAAQHGCukbm5ZdfVkZGhj9iJKl+/fr64x//yLuWAABApQkqZDwej06ePFlk+8mTJ5Wfn1/uoQAAAEojqJAZPHiwRo0apcWLF+vIkSM6cuSI/va3v2n06NEaMmRIqGcEAAAoVlDXyMyePVsTJkzQsGHDdPHixe8PFBmp0aNH66WXXgrpgAAAAFcTVMjUrl1bf/7zn/XSSy9p//79kqTWrVurTp06IR0OAADgWsr1C/Fyc3OVm5urtm3bqk6dOjLGhGouAACAEgUVMqdOnVLv3r3Vrl079e/fX7m5uZKk0aNH89ZrAABQaYIKmccff1w1a9ZUdna2ateu7d9+3333admyZSEbDgAA4FqCukbmk08+0fLly9WsWbOA7W3bttXhw4dDMhgAAEBJgnpF5uzZswGvxFx2+vRpud3ucg8FAABQGkGFzK233qp33nnHf9/lcsnn8+nFF1/UHXfcEbLhAAAAriWoHy29+OKL6t27t7KysnThwgU99dRT+uqrr3T69Gn961//CvWMAAAAxQrqFZlOnTppz5496tWrlwYNGqSzZ89qyJAh2rZtm1q3bh3qGQEAAIpV5ldkLl68qL59+2r27NmaNGlSRcwEAABQKmV+RaZmzZrauXNnRcwCAABQJkH9aGnEiBF68803Qz0LAABAmQR1se+lS5f01ltvacWKFerevXuRv7GUmZkZkuEAAACupUwhc+DAASUnJ2vXrl3q1q2bJGnPnj0B+7hcrtBNBwAAcA1lCpm2bdsqNzdXq1evlvT9nyT405/+pCZNmlTIcAAAANdSpmtkrvzr1kuXLtXZs2dDOhAAAEBpBXWx72VXhg0AAEBlKlPIuFyuItfAcE0MAABwSpmukTHGaOTIkf4/DHn+/HmNGTOmyLuWFi9eHLoJAQAArqJMIZOamhpwf8SIESEdBgAAoCzKFDJz5sypqDkAAADKrFwX+wIAADjJ0ZDJyMjQj370I8XExCg+Pl733nuvdu/e7eRIAADAIo6GzNq1azV27Fht3LhRn376qS5evKi7776b300DAABKJai/tRQqy5YtC7g/d+5cxcfHa+vWrbrtttscmgoAANjC0ZC5Ul5eniSpQYMGxT7u9Xrl9Xr99z0eT6XMBQAAqqYqEzI+n0/jxo3TLbfcok6dOhW7T0ZGhqZOnVp5Q02Jq6Dj5lXMcREg+el/OD0CqpFDtYY5PQKqoor6PiLxveT/VZl3LY0dO1a7du3SggULrrpPenq68vLy/LecnJxKnBAAAFQ1VeIVmd/97nf6+OOPtW7dOjVr1uyq+7ndbv9vFQYAAHA0ZIwx+v3vf68PPvhAa9asUcuWLZ0cBwAAWMbRkBk7dqzmz5+vDz/8UDExMTp+/LgkKS4uTtHR0U6OBgAALODoNTKzZs1SXl6ebr/9diUmJvpvCxcudHIsAABgCcd/tAQAABCsKvOuJQAAgLIiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFiLkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGAtQgYAAFjL0ZBZt26dBg4cqKZNm8rlcmnJkiVOjgMAACzjaMicPXtWXbt21WuvvebkGAAAwFKRTn7wfv36qV+/fk6OAAAALOZoyJSV1+uV1+v13/d4PA5OAwAAnGZVyGRkZGjq1KlOj1F+U+Iq5LDJ5+dXyHEr0qFaFXnsYRV38LKaUjGHrcjPH4AqrpK+lxyaMaBCPk6oWPWupfT0dOXl5flvOTk5To8EAAAcZNUrMm63W2632+kxAABAFWHVKzIAAAA/5OgrMgUFBdq3b5///sGDB7V9+3Y1aNBALVq0cHAyAABgA0dDJisrS3fccYf//vjx4yVJqampmjt3rkNTAQAAWzgaMrfffruMMU6OAAAALMY1MgAAwFqEDAAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxFyAAAAGsRMgAAwFqEDAAAsBYhAwAArEXIAAAAaxEyAADAWoQMAACwFiEDAACsRcgAAABrETIAAMBahAwAALAWIQMAAKxVJULmtddeU3JysmrVqqWePXtq8+bNTo8EAAAs4HjILFy4UOPHj9fkyZP1xRdfqGvXrrrnnnt04sQJp0cDAABVnOMhk5mZqYcfflijRo1Sx44dNXv2bNWuXVtvvfWW06MBAIAqLtLJD37hwgVt3bpV6enp/m01atRQnz59tGHDhiL7e71eeb1e//28vDxJksfjqZgBvaZijltBfN5zTo9QZh6XXZ9jAM6qqK9zfC36nys/xxXxPfbyMY0p/+fd0ZD573//q8LCQjVp0iRge5MmTfSf//ynyP4ZGRmaOnVqke3NmzevsBnt8iunByizOKcHAGCZivk6x9eiHwr8HMe9UnEfKT8/X3Fx5fvsOxoyZZWenq7x48f77/t8Pp0+fVoNGzaUy+UK6pgej0fNmzdXTk6OYmNjQzVqlcRaw1d1Wi9rDV/Vab3Vfa3GGOXn56tp06blPr6jIdOoUSNFRETom2++Cdj+zTffKCEhocj+brdbbrc7YFu9evVCMktsbGzY/8d0GWsNX9Vpvaw1fFWn9VbntZb3lZjLHL3YNyoqSt27d9fKlSv923w+n1auXKmUlBQHJwMAADZw/EdL48ePV2pqqnr06KEf//jHeuWVV3T27FmNGjXK6dEAAEAV53jI3HfffTp58qSee+45HT9+XDfeeKOWLVtW5ALgiuJ2uzV58uQiP7IKR6w1fFWn9bLW8FWd1staQ8dlQvHeJwAAAAc4/gvxAAAAgkXIAAAAaxEyAADAWoQMAACwVrUJmaNHj2rEiBFq2LChoqOj1blzZ2VlZfkfN8boueeeU2JioqKjo9WnTx/t3bvXwYmDV9JaR44cKZfLFXDr27evgxMHLzk5uchaXC6Xxo4dK0k6f/68xo4dq4YNG6pu3br6xS9+UeQXMNqipLXefvvtRR4bM2aMw1MHp7CwUM8++6xatmyp6OhotW7dWtOmTQv4uyzh9JwtzXrD6Xmbn5+vcePGKSkpSdHR0br55pu1ZcsW/+PhdG5LWqvN53XdunUaOHCgmjZtKpfLpSVLlgQ8XprzePr0aQ0fPlyxsbGqV6+eRo8erYKCgrINYqqB06dPm6SkJDNy5EizadMmc+DAAbN8+XKzb98+/z4zZswwcXFxZsmSJWbHjh3m5z//uWnZsqX57rvvHJy87Eqz1tTUVNO3b1+Tm5vrv50+fdrBqYN34sSJgHV8+umnRpJZvXq1McaYMWPGmObNm5uVK1earKws85Of/MTcfPPNzg4dpJLW+tOf/tQ8/PDDAfvk5eU5O3SQpk+fbho2bGg+/vhjc/DgQbNo0SJTt25dM3PmTP8+4fKcNaZ06w2n5+2vfvUr07FjR7N27Vqzd+9eM3nyZBMbG2uOHDlijAmvc1vSWm0+r//85z/NpEmTzOLFi40k88EHHwQ8Xprz2LdvX9O1a1ezceNG89lnn5k2bdqYoUOHlmmOahEyEydONL169brq4z6fzyQkJJiXXnrJv+3MmTPG7Xab9957rzJGDJmS1mrM90+cQYMGVc5AlSwtLc20bt3a+Hw+c+bMGVOzZk2zaNEi/+P//ve/jSSzYcMGB6cMjR+u1ZjvQyYtLc3ZoUJkwIAB5sEHHwzYNmTIEDN8+HBjTHg9Z40peb3GhM/z9ty5cyYiIsJ8/PHHAdu7detmJk2aFFbntqS1GhM+5/XKkCnNefz666+NJLNlyxb/PkuXLjUul8scPXq01B+7Wvxo6aOPPlKPHj30y1/+UvHx8brpppv0xhtv+B8/ePCgjh8/rj59+vi3xcXFqWfPntqwYYMTIwetpLVetmbNGsXHx6t9+/Z65JFHdOrUKQemDa0LFy7o3Xff1YMPPiiXy6WtW7fq4sWLAee1Q4cOatGihXXn9UpXrvWyefPmqVGjRurUqZPS09N17tw5B6cM3s0336yVK1dqz549kqQdO3Zo/fr16tevn6Twes5KJa/3snB43l66dEmFhYWqVatWwPbo6GitX78+rM5tSWu9LBzO65VKcx43bNigevXqqUePHv59+vTpoxo1amjTpk2l/liO/2bfynDgwAHNmjVL48eP1zPPPKMtW7boscceU1RUlFJTU3X8+HFJKvLbhJs0aeJ/zBYlrVWS+vbtqyFDhqhly5bav3+/nnnmGfXr108bNmxQRESEwysI3pIlS3TmzBmNHDlSknT8+HFFRUUV+cOiNp7XK125VkkaNmyYkpKS1LRpU+3cuVMTJ07U7t27tXjxYucGDdLTTz8tj8ejDh06KCIiQoWFhZo+fbqGDx8uSWH1nJVKXq8UPs/bmJgYpaSkaNq0abr++uvVpEkTvffee9qwYYPatGkTVue2pLVK4XNer1Sa83j8+HHFx8cHPB4ZGakGDRqU6VxXi5Dx+Xzq0aOHnn/+eUnSTTfdpF27dmn27Nn+b+7hojRrvf/++/37d+7cWV26dFHr1q21Zs0a9e7d25G5Q+HNN99Uv379QvJn4au64tb6m9/8xv+/O3furMTERPXu3Vv79+9X69atnRgzaO+//77mzZun+fPn64YbbtD27ds1btw4NW3aNOyes1Lp1htOz9u//vWvevDBB3XdddcpIiJC3bp109ChQ7V161anRwu5ktYaTufVKdXiR0uJiYnq2LFjwLbrr79e2dnZkqSEhARJKvJulm+++cb/mC1KWmtxWrVqpUaNGmnfvn0VPV6FOXz4sFasWKGHHnrIvy0hIUEXLlzQmTNnAva18bz+UHFrLU7Pnj0lycrz+uSTT+rpp5/W/fffr86dO+vXv/61Hn/8cWVkZEgKr+esVPJ6i2Pz87Z169Zau3atCgoKlJOTo82bN+vixYtq1apV2J3ba621ODaf1x8qzXlMSEjQiRMnAh6/dOmSTp8+XaZzXS1C5pZbbtHu3bsDtu3Zs0dJSUmSpJYtWyohIUErV670P+7xeLRp0yalpKRU6qzlVdJai3PkyBGdOnVKiYmJFT1ehZkzZ47i4+M1YMAA/7bu3burZs2aAed19+7dys7Otu68/lBxay3O9u3bJcnK83ru3DnVqBH45SkiIkI+n09SeD1npZLXW5xweN7WqVNHiYmJ+vbbb7V8+XINGjQo7M7tZcWttTjhcF6l0j1HU1JSdObMmYBX4latWiWfz+f/P2KlUv5rlau+zZs3m8jISDN9+nSzd+9eM2/ePFO7dm3z7rvv+veZMWOGqVevnvnwww/Nzp07zaBBg6x8u19Ja83PzzcTJkwwGzZsMAcPHjQrVqww3bp1M23btjXnz593ePrgFBYWmhYtWpiJEycWeWzMmDGmRYsWZtWqVSYrK8ukpKSYlJQUB6YMjautdd++feYPf/iDycrKMgcPHjQffvihadWqlbntttscmrR8UlNTzXXXXed/O/LixYtNo0aNzFNPPeXfJ1yes8aUvN5we94uW7bMLF261Bw4cMB88sknpmvXrqZnz57mwoULxpjwOrfXWqvt5zU/P99s27bNbNu2zUgymZmZZtu2bebw4cPGmNKdx759+5qbbrrJbNq0yaxfv960bduWt19fzd///nfTqVMn43a7TYcOHcxf/vKXgMd9Pp959tlnTZMmTYzb7Ta9e/c2u3fvdmja8rnWWs+dO2fuvvtu07hxY1OzZk2TlJRkHn74YXP8+HEHJy6f5cuXG0nFnq/vvvvOPProo6Z+/fqmdu3aZvDgwSY3N9eBKUPjamvNzs42t912m2nQoIFxu92mTZs25sknn7T298h4PB6TlpZmWrRoYWrVqmVatWplJk2aZLxer3+fcHrOlrTecHveLly40LRq1cpERUWZhIQEM3bsWHPmzBn/4+F0bq+1VtvP6+rVq42kIrfU1FRjTOnO46lTp8zQoUNN3bp1TWxsrBk1apTJz88v0xwuY37wqyMBAAAsUi2ukQEAAOGJkAEAANYiZAAAgLUIGQAAYC1CBgAAWIuQAQAA1iJkAACAtQgZAABgLUIGAABYi5ABAADWImQAAIC1CBkAAGCt/wM4ro2RIEAC+QAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "valid_set_mix[\"C_an\"].plot(kind=\"hist\", bins=20)\n",
    "train_set_mix[\"C_an\"].plot(kind=\"hist\", bins=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_set_mix[\"genome\"].sort_values().to_csv(os.path.join(outdir, \"training_set_mixed.txt\"), sep=\"\\n\", index=False, header=False)\n",
    "valid_set_mix[\"genome\"].sort_values().to_csv(os.path.join(outdir, \"validation_set_mixed.txt\"), sep=\"\\n\", index=False, header=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Species in...\n",
      "\ttraining set:\t\t42\n",
      "\tvalidation set:\t\t42\n"
     ]
    }
   ],
   "source": [
    "print(\"Species in...\")\n",
    "print(\"\\ttraining set:\\t\\t{}\".format(len(train_set_mix)))\n",
    "print(\"\\tvalidation set:\\t\\t{}\".format(len(valid_set_mix)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### High Quality Training Set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us filter  the training set so that BUSCO Annotation Completeness is >= 75"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "66"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set_hq = train_set[train_set[\"C_an\"] >= 75]\n",
    "train_set_hq[\"genome\"].sort_values().to_csv(os.path.join(outdir, \"training_set_hq.txt\"), sep=\"\\n\", index=False, header=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll rebuild the validation set now that more species will be available after being filtered from the training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_95164/452821630.py:1: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
      "  valid_set_hq = df.sort_values(\"C_an\", ascending=False)[~df[\"genome\"].isin(train_set_hq[\"genome\"])].drop_duplicates(\"genus\")\n"
     ]
    }
   ],
   "source": [
    "valid_set_hq = df.sort_values(\"C_an\", ascending=False)[~df[\"genome\"].isin(train_set_hq[\"genome\"])].drop_duplicates(\"genus\")\n",
    "valid_set_hq[\"genome\"].sort_values().to_csv(os.path.join(outdir, \"validation_set_hq.txt\"), sep=\"\\n\", index=False, header=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Species in...\n",
      "\ttraining set:\t\t66\n",
      "\tvalidation set:\t\t55\n"
     ]
    }
   ],
   "source": [
    "print(\"Species in...\")\n",
    "print(\"\\ttraining set:\\t\\t{}\".format(len(train_set_hq)))\n",
    "print(\"\\tvalidation set:\\t\\t{}\".format(len(valid_set_hq)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}