Submit independently to tasks

[Didayolo]

General idea

Right now, submitting to only one task results in having n/a scores to the other tasks. It would be interesting to be able to submit seperately to different tasks and have the results merged on the same leaderboard row.

Potential problems

• In some cases, we want the exact same submission to be evaluated on the different tasks. Therefore, allowing participants to submit different models / results to the different tasks, and then merging the scores, may be problematic.

• The leaderboard definition (score names, etc.) is the same for each task.

Potential implementations

Different possible approaches to discuss:

• Using the current system of selecting tasks to which submit. Then, the submissions would be added to the leaderboard depending on the submission rule (manually by the user, or the best first, etc.). It is not really clear how to implement this without deep changes.

• Using multiple leaderboards (multiple leaderboards discussion #1553)

• Allowing phases to be simultaneous, instead of sequential. This way, the different tasks could be separated in different phases. That may require deep changes.

Current work-around

The current work-around, used by competitions organizers, is to open separate competitions (with separate URLs) for each task.

Related issue

• Feature Request: Support for Separate Leaderboards or Parallel Phases #1679
• Score n/a for one of the subtasks when submitting for both subtasks at once. #1570
• Features suggestions #1334
• The meta scoring problem could be done on tasks, not on columns: [Feature] Meta scoring program: organizer provides the code for computation between leaderboard columns #406

[Didayolo]

The current system is not that bad:

• If the tasks are similar and are intended to all receive the same submission, then the current system is fine (submit to all tasks at once and get your scores)
• If the tasks are differents (different data, scores, solutions...), then having separated competitions is a fine work-around, that is actually used a lot. Maybe we could allow organizers to create pages to group competition togethers.

[shmuhammadd]

We are experiencing the same problem now. Since our issue (#1686) is yet to be resolved, we followed this alternative and are now stuck with this issue. We have three different tasks (data, scores, and solutions), and our only option is to follow your suggestion to have separate competitions as a workaround.