Problem with exclude datasets in benchmark

[toncho11]

moabb/moabb/benchmark.py

Line 333 in ca79b90

datasets_codes = [d.code for d in datasets]

It seems if you have more than 1 excludes than the code indexing is no longer correct resulting in removing another dataset - not the desired one. This is because two lists datasets_codes and are no longer the same length and so the indexing becomes incorrect.

I will look into a solution now.

[toncho11]

This is the documentation:

include_datasets: list of str or Dataset object
        Datasets (dataset.code or object) to include in the benchmark run.
        By default, all suitable datasets are included. If both include_datasets
        and exclude_datasets are specified, raise an error.
    exclude_datasets: list of str or Dataset object
        Datasets to exclude from the benchmark run

It is actually unclear if it is class codes, class names or both. Until now I assumed that it is class names only for the exclude parameter of benchmark.

[toncho11]

The fact that it allows both codes and class names leads to confusion.

[toncho11]

There is (as before) a tricky problem with the exclude datasets code. I tried:

exclude_datasets=["EPFLP300","FakeVirtualRealityDataset","Huebner2017","Huebner2018","Lee2019_ERP","Sosulski2019"]

exclude_datasets=[EPFLP300(),FakeVirtualRealityDataset(),Huebner2017(),Huebner2018(),Lee2019_ERP(),Sosulski2019()],

but I get errors each time.

I am putting this here for other people who are suffering like me. Try my PR and explanation: #656 (comment) if you can not use the exclude_datasets.