-
Notifications
You must be signed in to change notification settings - Fork 4
Home
Chalab is a tool, which helps you design data science or machine learning challenges. A step-by-step Wizard guides you through the process. When you are done, you can compile your challenge as a self-contained zip file (competition bundle) and upload it to a challenge platform. Currently, only Codalab accepts competition bundles created by Chalab. You can view a sample competition of the style you can create with Chalab.
Although Codalab allows you to design very elaborate challenges (with many datasets and phases, and elaborate means of scoring results), this version of Chalab follows a rather rigid pattern to generate "classic" data science challenges. However, such challenges are both with result and CODE submission. This permits comparing solutions proposed by participants in a fair way, using the same computational resources. Instructors using challenges in their classes can easily evaluate and check solutions submitted.
Codalab allows you to submit any kind of Linux executable. You can run your code in a docker of your choice. For simplificy however, all the examples we provide are in Python. We use Jupyter notebook for the starting kit and the scikit-learn machine learning library, which includes an excellent tutorial.
Your point of entry into Chalab is the wizard home page, which allows you to select a challenge to edit or create a new challenge. You are then led to the Wizard page allowing you to design a challenge, one step at a time! Conveniently, you may use as template challenges previously created by others (i.e. there is a lot of information already filled in that provides you with further guidance). To understand how to select or create a template, see the Profile and the Group pages.
The Chalab challenge design includes 6 steps:
Data science challenges designed with Chalab propose supervised learning tasks of CLASSIFICATION or REGRESSION. You must prepare your dataset in the AutoML challenge format, which supports data represented as feature vectors. Full and sparse (LIBSVM-style) formats are supported. See the data page for details. We supply several example datasets, which you can choose from in a menu, if you are not ready yet to upload your own data.
Chalab wants to split your data 3-way into a:
- training set (with labels supplied to the participants to train their learning machine)
- validation set (with labels concealed to the participants who must predict them)
- (final) test set (also with labels concealed to the participants)
The performances of the participants are evaluated by comparing the "predictions" they provide in their submissions to the "ground truth" (or "solution"), known only to the organizers. This is achieved by using a metric:
metric(solution, prediction)
See how to supply your own [metric](https://github.com/codalab/chalab/wiki/Help:-Wizard-‐-Challenge-‐-Metric) or select from one of the metrics provided. The performances of various methods can be strongly affected by the choice of metrics, which also depend on the data (balanced or not) and task chosen (binary/multi-class/multilabel classification or regression). For example, in a binary classification problem, if one class has 99% of the data, it is preferable NOT to use error rate as a metric (because predicting always the most abundant class is a trivial way to get good performance). One would rather use the "balanced error rate", which averages the error rates of both classes.
We do not give much choice of protocol design. All challenges generated by this version of Chalab have 2 phases (development phase and final phase) and most phase parameters are fixed (maximum 5 submissions per day, 500 seconds of execution time per submission). You only get to select the start and end date of the development phase. See instructions. The data you supplied will automatically appear in the challenge platform in the "right" place:
- Public data: the training set and its labels, and the two unlabeled test sets will be downloadable by the participants.
- Input data: the public data will also be made available to the code submitted by the participants on the challenge platform, so training and prediction can happen on the platform.
- Solutions: the validation and test labels will be made available to the scoring program, which will compute performances using the metric that you supplied.
During the **final test phase** there is no submission. The last submission of the development phase is forwarded automatically to the final test phase. For competitions with **result submission only**, the participants must therefore include predictions on both the validation set AND the test set in their last submission.
Last but not least, you must provide information to your participants on what your challenge is about (overview), how you are going to rank them (evaluation), give them instructions on how to prepare a submission (get data, etc.), and inform them about the rules (terms and conditions). This is supplied in the form of HTML pages, which will appear automatically in your challenge website. We recommend that you include a pictures, a slide show, or even a video describing the objectives of your challenge to motivate participants. Check the videos that students created to present challenges they designed a project.