doc/orca_tutorial_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. [How to use ORCA](#how-to-use-orca)\n",
    "\t1. [Launch experiments through `ini` files](#launch-experiments-through-ini-files)\n",
    "\t\t1. [Syntax of `ini` files](#syntax-of-ini-files)\n",
    "\t\t2. [Hyper-parameter optimization](#hyper-parameter-optimization)\n",
    "\t\t3. [Experimental results and reports](#experimental-results-and-reports)\n",
    "\t2. [Running algorithms with ORCA API](#running-algorithms-with-orca-api)\n",
    "\t\t1. [Run a pair of train-test files with fitpredict](#run-a-pair-of-train-test-files-with-fitpredict)\n",
    "\t\t2. [Using performance metrics](#using-performance-metrics)\n",
    "\t\t3. [Visualizing projections](#visualizing-projections)\n",
    "\t\t4. [Visualizing projections and decision thresholds](#visualizing-projections-and-decision-thresholds)\n",
    "\t3. [Using ORCA with your own datasets](#using-orca-with-your-own-datasets)\n",
    "\t\t1. [Data format](#data-format)\n",
    "\t\t2. [Data partitions for the experiments](#data-partitions-for-the-experiments)\n",
    "\t\t3. [Generating your own partitions](#generating-your-own-partitions)\n",
    "\t\t4. [Warning about highly imbalanced datasets](#warning-about-highly-imbalanced-datasets)\n",
    "2. [References](#references)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to use ORCA\n",
    "\n",
    "ORCA is an experimental framework focused on productivity and experiments reproducibility for machine learning researchers. Although initially created to collect ordinal classification methods, it is also suitable for other classifiers.\n",
    "\n",
    "First, you will need to install the framework. To do so, please visit [ORCA Quick Install Guide](orca_quick_install.md). Note that you should be able to perform the test when the framework is successfully installed.\n",
    "\n",
    "This tutorial uses three small datasets (`pasture`, `tae`, `toy`) contained in folder [example data](../exampledata/30-holdout). The datasets are already partitioned with a 30-holdout experimental design.\n",
    "\n",
    "This tutorial has been tested in Octave 4.2 and 4.4, but it should work with minor changes in Matlab. \n",
    "\n",
    "*NOTE:*\n",
    "\n",
    "Small datasets like the ones used in this tutorial usually produce warning messages such as:\n",
    "```MATLAB\n",
    "Warning: Matrix is close to singular or badly scaled. Results may be inaccurate. RCOND =\n",
    "1.747151e-17.\n",
    "Warning: Maximum likelihood estimation did not converge.  Iteration limit\n",
    "exceeded.  You may need to merge categories to increase observed counts.\n",
    "\n",
    "```\n",
    "\n",
    "You can disable these messages by using the following code in Matlab:\n",
    "```MATLAB\n",
    "warning('off','MATLAB:nearlySingularMatrix')\n",
    "warning('off','stats:mnrfit:IterOrEvalLimit')\n",
    "```\n",
    "\n",
    "In Octave, to disable `warning: matrix singular to machine precision` we need to disable all the warnings: \n",
    "```MATLAB\n",
    "warning('off','all');\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Launch experiments through `ini` files\n",
    "\n",
    "In this section, we run several experiments to compare the performance of three methods in a set of datasets: POM (Proportional Odds Model) [1], SVORIM (Support Vector Machines with IMplicit constrains) [2] and SVC1V1 (SVM classifier with 1-vs-1 binary decomposition) [3]. POM is a linear ordinal model, with limited performance but easy interpretation. SVORIM is an ordinal nonlinear model, with one of the most competitive performances according to several studies. SVC1V1 is the nominal counterpart of SVORIM, so that we can check the benefits of considering the order of the classes. To learn more about ordinal performance metrics see [4].\n",
    "\n",
    "From the Octave console load packages and add orca files to path:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "% Install dataframe, add src to path and disable some warnings\n",
    "pkg install -forge dataframe\n",
    "warning('off','all');\n",
    "\n",
    "addpath('../src/Algorithms/')\n",
    "addpath('../src/Measures/')\n",
    "addpath('../src/Utils/')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The set of experiments described in INI file `tutorial/config-files/pom.ini` can be run by (the syntax of these files will be explained in the [next subsection](#Syntax-of-ini-files)):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Utilities.runExperiments('tutorial/config-files/pom.ini')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can be observed, ORCA analyses all the files included in the folder of the dataset, where training and test partitions are included (a pair of files `train_dataset.X` and `test_dataset.X` for each dataset, where `X` is the number of partition). For each partition, a model is trained on training data and tested on test data.\n",
    "\n",
    "After this, you can also run the experiments for SVORIM and SVC1V1:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Utilities.runExperiments('tutorial/config-files/svorim-3holdout.ini')\n",
    "Utilities.runExperiments('tutorial/config-files/svc1v1-3holdout.ini')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the experiments are finished, the corresponding results can be found in the `Experiments` subfolder, as described in the [corresponding section](#Experimental-results-and-reports) of this tutorial.\n",
    "\n",
    "Each experiment has a different folder, and each folder should include two CSV files with results similar to the following (some columns are omitted):\n",
    "\n",
    "POM results ([download CSV](tutorial/reference-results/pom-mean-results_test.csv)):\n",
    "\n",
    "| Dataset-Experiment | MeanMAE | StdMAE | MeanAMAE | StdAMAE | MeanTrainTime | StdTrainTime |\n",
    "| --- | --- | --- | --- | --- | --- | --- |\n",
    "| pasture-pom-tutorial | 0.6 | 0.230866 | 0.6 | 0.230866 | 0.070958 | 0.004822 |\n",
    "| tae-pom-tutorial | 0.615789 | 0.100766 | 0.616952 | 0.101876 | 0.324884 | 0.087447 |\n",
    "| toy-pom-tutorial | 0.980889 | 0.038941 | 1.213242 | 0.059357 | 0.038949 | 0.002738 |\n",
    "\n",
    "SVORIM results ([download CSV](tutorial/reference-results/svorim-mean-results_test.csv)):\n",
    "\n",
    "| Dataset-Experiment | MeanMAE | StdMAE | MeanAMAE | StdAMAE | MeanTrainTime | StdTrainTime |\n",
    "| --- | --- | --- | --- | --- | --- | --- |\n",
    "| pasture-svorim-mae-real | 0.322222 | 0.106614 | 0.322222 | 0.106614 | 0.013843 | 0.002601 |\n",
    "| tae-svorim-mae-real | 0.475439 | 0.069086 | 0.473291 | 0.068956 | 0.042999 | 0.023227 |\n",
    "| toy-svorim-mae-real | 0.017778 | 0.012786 | 0.019631 | 0.015726 | 0.071385 | 0.025767 |\n",
    "\n",
    "SVC1V1 results ([download CSV](tutorial/reference-results/svc1v1-mean-results_test.csv)):\n",
    "\n",
    "| Dataset-Experiment | MeanMAE | StdMAE | MeanAMAE | StdAMAE | MeanTrainTime | StdTrainTime |\n",
    "| --- | --- | --- | --- | --- | --- | --- |\n",
    "| pasture-svc1v1-mae-tutorial | 0.314815 | 0.127468 | 0.314815 | 0.127468 | 0.014363 | 0.003297 |\n",
    "| tae-svc1v1-mae-tutorial | 0.534211 | 0.108865 | 0.533832 | 0.110083 | 0.017699 | 0.004122 |\n",
    "| toy-svc1v1-mae-tutorial | 0.051556 | 0.023419 | 0.044367 | 0.022971 | 0.015869 | 0.003786 |\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "***Exercise 1***: apparently, POM is the slowest method, but here we are not considering the crossvalidation time. Check the detailed CSV results to conclude which is the method with the lowest computational cost (taking crossvalidation, training and test phases into account).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, you can plot a bar plot to graphically compare the performance of the methods. Let's analyse for that the `toy` dataset. This is a synthetic dataset proposed by Herbrich et al. in their paper \"Support vector learning for ordinal regression\" (1997):\n",
    "\n",
    "![Synthetic toy dataset](tutorial/images/toy.png)\n",
    "\n",
    "The following code plots the figure below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pkg load dataframe\n",
    "pomT = dataframe('tutorial/reference-results/pom-mean-results_test.csv');\n",
    "svorimT = dataframe('tutorial/reference-results/svorim-mean-results_test.csv');\n",
    "svc1v1T = dataframe('tutorial/reference-results/svc1v1-mean-results_test.csv');\n",
    "datasets = {'pasture','tae','toy'};\n",
    "\n",
    "bar([pomT.MeanAMAE svorimT.MeanAMAE svc1v1T.MeanAMAE])\n",
    "set (gca, 'xticklabel',datasets)\n",
    "legend('POM  ', 'SVORIM', 'SVC1V1')\n",
    "title('AMAE performance (smaller is better)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "---\n",
    "\n",
    "***Exercise 2***: you can repeat this barplot but now considering:\n",
    "- A `global` (i.e. a metric where the class a priori probability is not considered) **nominal** metric.\n",
    "- A `global` **ordinal** metric.\n",
    "- A **nominal** metric specifically designed for imbalanced datasets.\n",
    "- An **ordinal** metric specifically designed for imbalanced datasets.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Syntax of `ini` files\n",
    "\n",
    "ORCA experiments are specified in configuration `ini` files, which execute an algorithm for a collection of datasets (each dataset with a given number of partitions). The folder [src/config-files](../src/config-files) contains example configuration files for running all the algorithms included in ORCA for all the algorithms and datasets of the [review paper](http://www.uco.es/grupos/ayrna/orreview). The following code is an example for running the Proportion Odds Model (POM), a.k.a. Ordinal Logistic Regression. Note that the execution of this `ini` file can take several hours:\n",
    "```INI\n",
    "; Experiment ID\n",
    "[pom-real\n",
    "{general-conf}\n",
    "seed = 1\n",
    "; Datasets path\n",
    "basedir = ../../../datasets/ordinal/real/30-holdout\n",
    "; Datasets to process (comma separated list or 'all' to process all)\n",
    "datasets = automobile,balance-scale,bondrate,car,contact-lenses,ERA,ESL,eucalyptus,LEV,marketing,newthyroid,pasture,squash-stored,squash-unstored,SWD,tae,thyroid,toy,winequality-red,winequality-white\n",
    "; Activate data standardization\n",
    "standarize = true\n",
    "\n",
    "; Method: algorithm and parameter\n",
    "{algorithm-parameters}\n",
    "algorithm = POM\n",
    "```\n",
    "\n",
    "`ini` files include **Subsections** to help organize the configuration. These sections are mandatory:\n",
    " - `{general-conf}`: generic parameters of the experiment, including the seed considered for random number generation, the directory containing the datasets, the datasets to be processed... All the parameters included here are the same for all the algorithms.\n",
    " - `{algorithm-parameters}`: here you can specify the algorithm to run and the parameters which are going to be fixed (not optimized through cross validation).\n",
    " - `{algorithm-hyper-parameters-to-cv}`: algorithms' hyper-parameters to optimise. For more details, see [Hyper-parameter optimization](#Hyper-parameter-optimization).\n",
    "\n",
    "The above file tells ORCA to run the algorithm `POM` for all the datasets specified in the list `datasets`. You can also use `datasets = all` to process all the datasets in `basedir`). Each of these datasets should be found at folder `basedir`, in such a way that ORCA expects one subfolder for each dataset, where the name of the subfolder must match the name of the dataset. Other directives are:\n",
    " - INI section `[pom-real]` sets the experiment identifier.\n",
    " - The `standarize` flag activates the standardization of the data (by using the mean and standard deviation of the train set).\n",
    " - The rest of the parameters of the model depend on the specific algorithm (and they should be checked in the documentation of the algorithm). For instance, the kernel type is set up with `kernel` parameter.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyper-parameter optimization\n",
    "\n",
    "Many machine learning methods are very sensitive to the value considered for the hyper-parameters (consider, for example, support vector machines and the two associated parameters, cost and kernel width). ORCA automates hyper-parameter optimization by using a grid search with an internal nested *k*-fold cross-validation considering only the training partition. Let see an example for the optimisation of the two hyper-parameters of SVORIM: cost (`C`) and kernel width parameter (`k`, a.k.a. *gamma*):\n",
    "```ini\n",
    "; Experiment ID\n",
    "[svorim-mae-real]\n",
    "{general-conf}\n",
    "seed = 1\n",
    "; Datasets path\n",
    "basedir = datasets/ordinal/real/30-holdout\n",
    "; Datasets to process (comma separated list)\n",
    "datasets = all\n",
    "; Activate data standardization\n",
    "standarize = true\n",
    "; Number of folds for the parameters optimization\n",
    "num_folds = 5\n",
    "; Crossvalidation metric\n",
    "cvmetric = mae\n",
    "\n",
    "; Method: algorithm and parameter\n",
    "{algorithm-parameters}\n",
    "algorithm = SVORIM\n",
    "kernel = rbf\n",
    "\n",
    "; Method's hyper-parameter values to optimize\n",
    "{algorithm-hyper-parameters-to-cv}\n",
    "C = 10.^(-3:1:3)\n",
    "k = 10.^(-3:1:3)\n",
    "```\n",
    "\n",
    "The directive for configuring the search process is included in the general section. The directives associated to hyper-parameter optimisation are:\n",
    "- `seed`: is the value to initialize MATLAB random number generator. This can be helpful to debug algorithms.\n",
    "- `num_folds`: *k* value for the nested *k*-fold cross validation over the training data.\n",
    "- `cvmetric`: metric used to select the best hyper-parameters in the grid search. The metrics available are: `AMAE`,`CCR`,`GM`,`MAE`,`MMAE`,`MS`,`MZE`,`Spearman`,`Tkendall` and `Wkappa`.\n",
    "- The list of hyper-parameters to be optimised and values considered for each parameter during the grid search are specified in subsection `{algorithm-hyper-parameters-to-cv}`;\n",
    "    - `C`: add a new parameter with name `C` and a set of values of `10.^(-3:1:3)` (10<sup>-3</sup>,10<sup>-2</sup>,...,10<sup>3</sup>). The same apples for `k`.\n",
    "\n",
    "The parameter optimization can also be done by using the API (full example is in [exampleParamOptimization.m](../src/code-examples/exampleParamOptimization.m) script):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "% Load the different partitions of the dataset\n",
    "load ../exampledata/1-holdout/toy/matlab/train_toy.0\n",
    "load ../exampledata/1-holdout/toy/matlab/test_toy.0\n",
    "\n",
    "% \"patterns\" refers to the input variables and targets to the output one\n",
    "train.patterns = train_toy(:,1:end-1);\n",
    "train.targets = train_toy(:,end);\n",
    "test.patterns = test_toy(:,1:end-1);\n",
    "test.targets = test_toy(:,end);\n",
    "\n",
    "% Assumes training set in structure 'train'\n",
    "% Create the algorithm object\n",
    "algorithmObj = KDLOR();\n",
    "% Create vectors of values to test\n",
    "param.C = 10.^(-3:1:3);\n",
    "param.k = 10.^(-3:1:3);\n",
    "param.u = [0.01,0.001,0.0001,0.00001];\n",
    "\n",
    "% Optimizing parameters for KDLOR with metric MAE (default metric)\n",
    "optimalp = paramopt(algorithmObj,param,train)\n",
    "\n",
    "% Optimizing parameters for KDLOR with metric GM\n",
    "optimalp = paramopt(algorithmObj,param,train, 'metric', GM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experimental results and reports\n",
    "\n",
    "ORCA uses the `Experiments` folder to store all the results of the different experiments. Each report is placed in a subfolder of `Experiments` named with the current date, time and the name of the configuration file (for example 'exp-2015-7-14-10-5-57-pom'). After a successful experiment, this folder should contain the following information:\n",
    " - Individual experiment configuration files for each dataset and partition.\n",
    " - A `Results` folder with the following information:\n",
    "    - `mean-results_train.csv` and `mean-results_test.csv` which are the reports in CSV format (easily read by Excel or LibreOffice Calc). They contain the mean and standard deviation for each performance measure (`AMAE`,`CCR`,`GM`,`MAE`,`MMAE`,`MS`,`MZE`,`Spearman`,`Tkendall` and `Wkappa`) and the computational time. These averages and standard deviations are obtained for all the partitions of each algorithm and dataset.\n",
    "    - The `Results` folder contains one subfolder for each dataset with the following data:\n",
    "        - Train and test confusion matrices (`matrices.txt`).\n",
    "        - Name of the folder used for the experiments (`dataset`).\n",
    "        - Individual results for each partition in CSV format (`results.csv`).\n",
    "        - Models of each partition in `.mat` format (`Models` folder). These models are structures and their fields depend on the specific algorithm.\n",
    "        - Decision values used to obtain the predicted labels for training and test partitions ('Guess' folder). For threshold models, this is the one dimensional mapping before applying the discretisation based on the thresholds. The rest of models may have multidimensional mappings.\n",
    "        - Labels predicted by the models for each partition ('Predictions' folder).\n",
    "        - Optimal hyper-parameters values obtained after nested cross-validation ('OptHyperparams').\n",
    "        - Computational time results ('Times').\n",
    "\n",
    "If you provide the option `report_sum = true` in `{general-conf}`, additionally the same metrics will be calculated with a matrix that is the sum of the generalization matrices (as Weka does). **Note that this only makes sense in the case of a k-fold experimental design**. With this option active, two additional reports will be generated (`mean-results_matrices_sum_train.csv` and `mean-results_matrices_sum_test.csv`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running algorithms with ORCA API\n",
    "\n",
    "### Run a pair of train-test files with fitpredict\n",
    "\n",
    "ORCA algorithms can be used from your own Matlab/Octave code. All algorithms included in the [Algorithms](../src/Algorithms) have a `fitpredict` method, which can be used for running the algorithms with your data. The method receives a structure with the matrix of training data and labels, the equivalent for test data and a structure with the values of the parameters associated to the method. With respect to other tools, parameters are a mandatory argument for the method to avoid the use of default values.\n",
    "\n",
    "For example, the [KDLOR (Kernel Discriminant Learning for Ordinal Regression)](../src/Algorithms/KDLOR.m) [5]  method has a total of five parameters. Two of them (the type of kernel, `kernelType`, and the optimisation routine considered, `optimizationMethod`) are received in the constructor of the corresponding class, and the other three parameters (cost, `C`, kernel parameter, `k`, and value to avoid singularities, `u`) are supposed to have to be fine-tuned for each dataset and partition, so they are received in a structure passed to the `fitpredict` method.\n",
    "\n",
    "This an example of execution of KDLOR from the Matlab console:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "addpath('../src/Algorithms/')\n",
    "kdlorAlgorithm = KDLOR('kernelType','rbf','optimizationMethod','quadprog');\n",
    "kdlorAlgorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load train and test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "load ../exampledata/1-holdout/toy/matlab/train_toy.0\n",
    "load ../exampledata/1-holdout/toy/matlab/test_toy.0\n",
    "train.patterns = train_toy(:,1:(size(train_toy,2)-1));\n",
    "train.targets = train_toy(:,size(train_toy,2));\n",
    "test.patterns = test_toy(:,1:(size(test_toy,2)-1));\n",
    "test.targets = test_toy(:,size(test_toy,2));\n",
    "param.C = 10;\n",
    "param.k = 0.1;\n",
    "param.u = 0.001;\n",
    "param"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit the model and test prediction with generalization data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "info = kdlorAlgorithm.fitpredict(train,test,param);\n",
    "fieldnames(info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fprintf('Accuracy Train %f, Accuracy Test %f\\n',\n",
    "    sum(train.targets==info.predictedTrain)/size(train.targets,1),\n",
    "    sum(test.targets==info.predictedTest)/size(test.targets,1));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the methods return a structure with the main information about the execution of the algorithm. The fields of this structure are:\n",
    "- `projectedTrain`: decision values for the training set.\n",
    "- `predictedTrain`: labels predicted for the training set.\n",
    "- `trainTime`: time in seconds needed for training the model.\n",
    "- `projectedTest`: decision values for the test set.\n",
    "- `predictedTest`: labels predicted for the test set.\n",
    "- `testTime`: time in seconds needed for the test phase.\n",
    "- `model`: structure containing the model (its coefficients, parameters, etc.). Note that, although most of the fields of this structure depend on the specific algorithm considered, we will always find the `algorithm` and `parameters` fields. These are the fields for KDLOR:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fieldnames(info.model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "i.e., the algorithm used for training (`algorithm`), the weight given to each pattern in the kernel model (`projection`), the set of threshold values (`thresholds`), the parameters used for training (`parameters`), the type of kernel considered (`kernelType`) and the training data (`train`). As can be checked, at least, this structure should contain the information for performing the test phase. In this way, for KDLOR, the prediction phase needs to apply the kernel to each training point and the test point being evaluated (using `kernelType`, `train` and `parameters.K`) and perform the weighted sum of these values (using `projection`). After that, the thresholds are used to obtain the labels.\n",
    "\n",
    "The corresponding script ([exampleKDLOR.m](../src/code-examples/exampleKDLOR.m)) can be found and run in the [code example](../src/code-examples) folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using performance metrics\n",
    "\n",
    "Ordinal classification problems should be evaluated with specific metrics that consider the magnitude of the prediction errors in different ways. ORCA includes a set of these metrics in [Measures](../src/Measures) folder. Given the previous example, we can calculate different performance metrics with the actual and predicted labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "addpath ../Measures/\n",
    "CCR.calculateMetric(test.targets,info.predictedTest)\n",
    "MAE.calculateMetric(test.targets,info.predictedTest)\n",
    "AMAE.calculateMetric(test.targets,info.predictedTest)\n",
    "Wkappa.calculateMetric(test.targets,info.predictedTest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same results can be obtained from the confusion matrix:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = confusionmat(test.targets,info.predictedTest)\n",
    "CCR.calculateMetric(cm)\n",
    "MAE.calculateMetric(cm)\n",
    "AMAE.calculateMetric(cm)\n",
    "Wkappa.calculateMetric(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing projections\n",
    "\n",
    "Many ordinal regression methods belong to the category of threshold methods, which briefly means that models project the patterns into a one-dimensional latent space. We can visualize this projection and thresholds in the following way to observe the effect of the kernel width parameter starting from the previous example:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "figure; hold on;\n",
    "param.k = 1;\n",
    "info1 = kdlorAlgorithm.fitpredict(train,test,param);\n",
    "hist(info1.projectedTest', 30);\n",
    "legend('KDLOR k=1','Location','NorthWest')\n",
    "hold off;\n",
    "\n",
    "figure; hold on;\n",
    "param.k = 100;\n",
    "info2 = kdlorAlgorithm.fitpredict(train,test,param);\n",
    "hist(info2.projectedTest', 30);\n",
    "legend('KDLOR k=100','Location','NorthWest')\n",
    "hold off;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you can compare performance using AMAE:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "amae1 = AMAE.calculateMetric(test.targets,info1.predictedTest)\n",
    "amae2 = AMAE.calculateMetric(test.targets,info2.predictedTest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The whole example is available at [exampleProjections.m](../src/code-examples/exampleProjections.m)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing projections and decision thresholds\n",
    "\n",
    "The `model` structure stores decision thresholds in the field thresholds. Starting from the previous example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "% Run algorithm\n",
    "info1 = kdlorAlgorithm.fitpredict(train,test,param);\n",
    "amaeTest1 = AMAE.calculateMetric(test.targets,info1.predictedTest);\n",
    "% Build legend text\n",
    "msg{1} = sprintf('KDLOR k=%f. AMAE=%f', param.k, amaeTest1);\n",
    "msg{2} = 'Thresholds';\n",
    "\n",
    "figure; hold on;\n",
    "info1 = kdlorAlgorithm.fitpredict(train,test,param);\n",
    "hist(info1.projectedTest', 30);\n",
    "y1=get(gca,'ylim');\n",
    "plot(info1.model.thresholds, ...\n",
    "    zeros(length(info1.model.thresholds),1),...\n",
    "    'r+', 'MarkerSize', 10)\n",
    "legend(msg)\n",
    "legend('Location','NorthWest')\n",
    "hold off;\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The whole example is available at [exampleProjectionsThresholds.m](../src/code-examples/exampleProjectionsThresholds.m)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using ORCA with your own datasets\n",
    "\n",
    "This section shows how to use ORCA with custom datasets. First of all, you should take into account the structure of the files and then the way you should include them in the corresponding folder.\n",
    "\n",
    "### Data format\n",
    "\n",
    "ORCA uses the default text file format for MATLAB. This is, one pattern per row with the following structure:\n",
    "```\n",
    "attr1 attr2 ... attrN label\n",
    "attr1 attr2 ... attrN label\n",
    "attr1 attr2 ... attrN label\n",
    "```\n",
    "ORCA is intended to be used for ordinal regression problems, so the labels should be integer numbers: `1` for the first class in the ordinal scale, `2` for the second one, ..., `Q` for the last one, where `Q` is the number of classes of the problem. Please, take into account that all the attributes should be numeric, i.e. categorical variables needs to be transformed into several binary dummy variables before using ORCA.\n",
    "\n",
    "### Data partitions for the experiments\n",
    "\n",
    "The datasets should be partitioned before applying the ORCA algorithms, i.e. ORCA needs all the pairs of training and test files for each dataset. This is because, in this way, we are sure all the methods will consider the same partitions, which is very important to obtain reliable unbiased estimation of the performance and be able to perform fair comparisons. The partitions would be used to train and measure generalization performance of each algorithm.\n",
    "\n",
    "For each dataset, ORCA will look for a subfolder called `matlab`, which will contain the training and test partitions. If the name of the dataset is `dataset`, the name of the files will be `train_dataset.X` for training partitions, and `test_dataset.X` for the test ones, where `X` is the number of partitions. This format has to be respected.\n",
    "\n",
    "For instance, for the `toy` dataset, we have the following folder and file arrangement to perform `30` times a stratified holdout validation:\n",
    "```\n",
    "toy\n",
    "toy/matlab/\n",
    "toy/matlab/test_toy.0\n",
    "toy/matlab/train_toy.0\n",
    "toy/matlab/test_toy.1\n",
    "toy/matlab/train_toy.1\n",
    "\n",
    "toy/matlab/test_toy.29\n",
    "toy/matlab/train_toy.29\n",
    "```\n",
    "ORCA will train a model for all the training/test pairs, and the performance results will be used for the reports. The website of the review paper associated to ORCA includes the [partitions](http://www.uco.es/grupos/ayrna/ucobigfiles/datasets-orreview.zip) for all the datasets considered in the experimental part.\n",
    "\n",
    "### Generating your own partitions\n",
    "\n",
    "If you are using your own dataset, you will probably have to generate your own partitions. There are many options for performing the training/test partitions of a dataset, but the two most common ones are:\n",
    "- `k`-fold cross-validation: in this case, the dataset is randomly divided in `k` subsets or folds. `k` training/test partitions will be considered, where, for each partition, one fold will be used for test and the remaining ones will be used for training.\n",
    "- Holdout validation: in this case, the dataset is simply divided in two random subsets, one for training and the other one for test. It is quite common to use the following percentages: 75% for training and 25% for test.\n",
    "- Repeated holdout validation (`h`-holdout): in order to avoid the dependence of the random partition, the holdout process is repeated a total of `h` times. A common value for `h` is 30, given that it is high enough for obtaining reliable estimations of the performance from a statistical point of view.\n",
    "\n",
    "For classification (ordinal or nominal), both methods (`k`-fold and `h`-holdout) should be applied in a stratified way, i.e. the partitions are generated respecting the initial proportions of patterns in the original dataset. This is especially important for imbalanced datasets.\n",
    "\n",
    "Now, we are going to generate the partitions for a given dataset. We will use the dataset ERA. This is its description:\n",
    "> The ERA data set was originally gathered during an academic decision-making\n",
    "> experiment aiming at determining which are the most important qualities of\n",
    "> candidates for a certain type of jobs. Unlike the ESL data set (enclosed)\n",
    "> which was collected from expert recruiters, this data set was collected\n",
    "> during a MBA academic course.\n",
    "> The input in the data set are features of a candidates such as past\n",
    "> experience, verbal skills, etc., and the output is the subjective judgment of\n",
    "> a decision-maker to which degree he or she tends to accept the applicant to\n",
    "> the job or to reject him altogether (the lowest score means total tendency to\n",
    "> reject an applicant and vice versa).\n",
    "\n",
    "Specifically, 4 input attributes are used for evaluating each individual, and the variable to be predicted is the final judgment of the decision maker. We can load the dataset in MATLAB by using the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "load ../exampledata/ERA.csv\n",
    "ERA(1:20,:)\n",
    "pkg load statistics;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "targets = ERA(:,end);\n",
    "k=10;\n",
    "CVO = cvpartition(targets,'KFold',k);\n",
    "nameDataset = 'era';\n",
    "rootDir = fullfile('..', 'exampledata', '10-fold', nameDataset);\n",
    "mkdir(rootDir);\n",
    "rootDir = fullfile(rootDir,'matlab');\n",
    "mkdir(rootDir);\n",
    "numTests = get(CVO,'NumTestSets');\n",
    "for ff = 1:numTests\n",
    "    trIdx = training(CVO,ff);\n",
    "    teIdx = ~trIdx;\n",
    "    dlmwrite(fullfile(rootDir,sprintf('train-%s.%d',nameDataset,ff-1)),ERA(trIdx,:),' ');\n",
    "    dlmwrite(fullfile(rootDir,sprintf('test-%s.%d',nameDataset,ff-1)),ERA(teIdx,:),' ');\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will generate all the partitions for a `10`fold crossvalidation experimental design. The source code of this example is in [exampleERAKFold.m](../src/code-examples/exampleERAKFold.m).\n",
    "\n",
    "In order to obtain a `30`holdout design, the code will be a bit different. As MATLA/Octave does not included a native way of repeating holdout, we will do it manually:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "load ../exampledata/ERA.csv\n",
    "ERA(1:20,:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "% Extract targets\n",
    "targets = ERA(:,end);\n",
    "% Generate h holdout partitions\n",
    "h=30;\n",
    "% Prepare filesystem\n",
    "nameDataset = 'era';\n",
    "rootDir = fullfile('..', 'exampledata', '30-holdout', nameDataset);\n",
    "mkdir(rootDir);\n",
    "rootDir = fullfile(rootDir,'matlab');\n",
    "mkdir(rootDir);\n",
    "% For each partitions\n",
    "for ff = 1:h\n",
    "    CVO = cvpartition(targets,'HoldOut',0.25); % 25% of patterns for the test set\n",
    "    trIdx = training(CVO,1);\n",
    "    teIdx = ~trIdx;\n",
    "    dlmwrite(fullfile(rootDir,sprintf('train-%s.%d',nameDataset,ff-1)),ERA(trIdx,:),' ');\n",
    "    dlmwrite(fullfile(rootDir,sprintf('test-%s.%d',nameDataset,ff-1)),ERA(teIdx,:),' ');\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The source code of this example is in [exampleERAHHoldout.m](../src/code-examples/exampleERAHHoldout.m). As can be checked, the `cvpartition` function performs the partitions, receiving the target vector. The targets are used in order to obtain a stratified partition.\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "***Exercise 3***: you should prepare a `30holdout` set of partitions for the dataset `ESL`, which is included in the folder [exampledata](/exampledata). Try to find the description of this dataset in the Internet and spot the main differences with respect to ERA.\n",
    "\n",
    "---\n",
    "\n",
    "***Exercise 4***: train classifiers for both `ERA` and `ESL` datasets, using the same experimental design you used in the [experiment section](orca_tutorial_1.md#launch-experiments-through-ini-files). Compare the results obtained for both datasets. Generate bar plots for comparing accuracy and AMAE. Which one is better classified? Which one is better ordered?\n",
    "\n",
    "---\n",
    "\n",
    "---\n",
    "\n",
    "### Warning about highly imbalanced datasets\n",
    "\n",
    "ORCA is an tool to automate experiments for algorithm comparison. The default experimental setup is a n-hold-out (n=10). However, if your dataset has only less than 10-15 patterns in one or more classes, it is very likely that there will not be enough data to do the corresponding partitions, so there will be folds with varying number of classes. This can cause some errors since the confusion matrices dimensions do not agree.\n",
    "\n",
    "# References\n",
    "\n",
    "1. P. McCullagh, \"Regression models for ordinal data\",  Journal of the Royal Statistical Society. Series B (Methodological), vol. 42, no. 2, pp. 109–142, 1980.\n",
    "1. W. Chu and S. S. Keerthi, \"Support Vector Ordinal Regression\", Neural Computation, vol. 19, no. 3, pp. 792–815, 2007. http://10.1162/neco.2007.19.3.792\n",
    "1. C.-W. Hsu and C.-J. Lin. \"A comparison of methods for multi-class support vector machines\", IEEE Transaction on Neural Networks,vol. 13, no. 2, pp. 415–425, 2002. https://doi.org/10.1109/72.991427\n",
    "1. M. Cruz-Ramírez, C. Hervás-Martínez, J. Sánchez-Monedero and P. A. Gutiérrez, \"Metrics to guide a multi-objective evolutionary algorithm for ordinal classification\", Neurocomputing, Vol. 135, July, 2014, pp. 21-31. https://doi.org/10.1016/j.neucom.2013.05.058\n",
    "1. B.-Y. Sun, J. Li, D. D. Wu, X.-M. Zhang, and W.-B. Li, \"Kernel discriminant learning for ordinal regression\", IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 6, pp. 906-910, 2010. https://doi.org/10.1109/TKDE.2009.170\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Octave",
   "language": "octave",
   "name": "octave"
  },
  "language_info": {
   "file_extension": ".m",
   "help_links": [
    {
     "text": "GNU Octave",
     "url": "https://www.gnu.org/software/octave/support.html"
    },
    {
     "text": "Octave Kernel",
     "url": "https://github.com/Calysto/octave_kernel"
    },
    {
     "text": "MetaKernel Magics",
     "url": "https://github.com/calysto/metakernel/blob/master/metakernel/magics/README.md"
    }
   ],
   "mimetype": "text/x-octave",
   "name": "octave",
   "version": "4.2.1"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}