Advanced Tutorial
Here is an advanced tutorial. If you are new to CodaBench, please refer to get started tutorial first. In this article, you'll learn how to use more advanced features and how to create benchmarks using either the editor or bundles. Before proceeding to our tutorial, make sure you have registered for an account on the Codabench website.
- Creating a Benchmark by Editor
- Creating a Benchmark by Bundle
- Benchmark Examples
- How do I set up submission comments for multiple submissions?
The image below is an overview of the benchmark creation process
In this chapter, I'll take you step by step through the Editor's approach to creating benchmark, including algorithm type and dataset type.
When you click on it, you will see the screen as shown in the screenshot below.
When you click on the Manage Tasks/Datasets button, you will see the screenshot shown below
Click the Add Dataset button in the diagram to upload the resource files needed to create the competition.
Creating a phase will require a bundle of the following types (.zip format), I'll give you more details on how to write these bundle later.
Now you just need to have this concept in your mind
Here are the screenshots of the 5 types of bundles after they were uploaded
Creating a benchmark through bundles is a much more efficient way than using editors.
Click on the link above to download the bundle in the screenshot.
After the bundle has been uploaded, you will see the screenshot shown below.
Example bundles for code & dataset competition can be found here:
https://github.com/codalab/competition-examples/tree/master/codabench
Iris Codabench Bundle is a simple benchmark involving two phases, code submission and results submission.
Two versions of the Automated Weakly Supervised Learning Benchmark:
Mini-AutoML Bundle is a benchmark template for Codabench, featuring code submission to multiple datasets (tasks).
