Home

Documentation

Welcome to the Codabench wiki!

Codabench is a platform allowing you to flexibly specify a benchmark. First you define tasks, e.g. datasets and metrics of success, then you specify the API for submissions of code (algorithms), add some documentation pages, and [CLICK] your benchmark is created, ready to accept submissions of new algorithms. Participant results get appended to an ever-growing leaderboard.

You may also create inverted benchmarks in which the role of datasets and algorithms are swapped. You specify reference algorithms and your participants submit datasets.

Table of contents

1. Participants
2. Organizers
3. Developers and administrators
4. How you can contribute
5. How to contact us

1. Participants

• List of Current Benchmarks and Competitions
• Participating in a Competition

2. Organizers

General Information (Page Diagrams and FAQ)

• Competition Detail Page
• Competition Management & List Page
• Queue Management
• Task & Dataset Management
• Competition Creation: Form
• Competition Creation: Bundle
• Dataset competition creation and participate instruction
• Leaderboard Features
• How to Deploy Server

Organizer Information

• Getting started tutorial
• Competition YAML Structure
• Competition Bundle Structure
• Compute Worker Management & Setup
• Admin Features
• Competition examples
• Advanced Tutorial
• Example: Cancer Benchmarks

3. Developers and administrators

Codabench Platform Technical Information and Management

• Overall Architecture
• Administrator procedures
• Submission Docker Container Layout
• Backups: Creating, Automating, and Restoring
• Codabench Docker Architecture
• Submission Process Overview
• Codabench Basic Installation Guide
• Special prod settings
• Robot submissions
• Running tests

4. How you can contribute

• Use Codabench by either participating in a competition or hosting a new competition.
• Find a bug? Got a feature request? Submit a GitHub issue.
• Submit pull requests on GitHub to implement new features or fix bugs.
• Improve this documentation.
• Let others know about Codabench!

5. How to contact us

• The preferred way is via posting a GitHub issue.
• In case of emergency, contact us directly.
• If you wish to get in touch with the community, you can use the Google Groups.