Home

Documentation

Welcome to the Codabench wiki!

Codabench is a platform allowing you to flexibly specify a benchmark. First you define tasks, e.g. datasets and metrics of success, then you specify the API for submissions of code (algorithms), add some documentation pages, and [CLICK] your benchmark is created, ready to accept submissions of new algorithms. Participant results get appended to an ever-growing leaderboard.

You may also create inverted benchmarks in which the role of datasets and algorithms are swapped. You specify reference algorithms and your participants submit datasets.

Table of contents

1. Participants
2. Organizers
3. Developers and administrators
4. How you can contribute
5. How to contact us

1. Participants

• List of Current Benchmarks and Competitions
• Participating in a Competition

2. Organizers

Benchmark creation

• Getting Started Tutorial
• Advanced Tutorial
• Competition Creation: Form
• Competition Creation: Bundle
• Competition YAML Structure
• Competition Bundle Structure
• Dataset competition creation and participate instruction
• Leaderboard Features
• Competition Examples
• Example: Cancer Benchmarks

Running a benchmark

• Benchmark Management & List Page
• Benchmark Detail Page
• Queue Management
• Task & Dataset Management
• Compute Worker Management & Setup
• Admin Features

3. Developers and administrators

Codabench Platform Technical Information and Management

• Overall Architecture
• Codabench Basic Installation Guide
• How to Deploy Server
• Administrator procedures
• Codabench Docker Architecture
• Submission Docker Container Layout
• Backups: Creating, Automating, and Restoring
• Submission Process Overview
• Special prod settings
• Robot submissions
• Running tests

4. How you can contribute

• Use Codabench by either participating in a competition or hosting a new competition.
• Find a bug? Got a feature request? Submit a GitHub issue.
• Submit pull requests on GitHub to implement new features or fix bugs.
• Improve this documentation.
• Let others know about Codabench!

5. How to contact us

• The preferred way is via posting a GitHub issue.
• In case of emergency, contact us directly.
• If you wish to get in touch with the community, you can use the Google Groups.