-
Notifications
You must be signed in to change notification settings - Fork 31
[Newsletter] CodaLab in 2024
Welcome to the first edition of CodaLab’s newsletter! This year has been full of novelty, success, and scientific progress. The platform is breaking records of participation and number of organized competitions, and Codabench, the new version of CodaLab, had a very promising launch. Let’s dive into more details.
In October, Codabench has registered its 10,000th user! From about 100 daily submissions in January, it is now more than 500 daily submissions that are handled on the servers. In 2024, more than 20,000 new users have registered on CodaLab, whereas more than 12,000 on Codabench.
249 public competitions have been created on CodaLab, whereas 193 on Codabench: +15% of total competitions of both platforms compared to 2023. CodaLab continues to handle a large number of submissions, as many past competitions remain active even after they have officially ended: 240,000 submissions on CodaLab whereas 83,000 on Codabench.
Contributors community is very active with 143 pull requests this year. Since the platform is still relatively new, the primary focus has been on bug fixes, security and performance enhancements, and administrative features, accounting for approximately two-thirds of the pull requests. Nevertheless, we are keen on improving the experience for both participants and organizers. We have set a versioning and a release-notes follow-up to give more visibility to the platform evolution and maturity.
Codabench, the modernized version of CodaLab, was released in summer 2023, and presented at JCAD days in November 2024! Codabench platform software is now concentrating all development effort of the community. In addition to CodaLab features, it offers improved performance, live logs, more transparency, data-centric benchmarks and more!
We warmly encourage you to use codabench.org for all your new competitions and benchmarks. Note that CodaLab bundles are compatible with Codabench, easing the transition, as explained in the following Wiki page: How-to-transition-from-CodaLab-to-Codabench
CodaLab and Codabench are hosted on servers located at Paris-Saclay university, maintained by LISN lab.
The most popular competition this year, featuring 520 participants, was the SemEval task Bridging the Gap in Text-Based Emotion Detection. The task of this competition focused on identifying the emotion that most people would associate with a speaker based on a given sentence or short text snippet.
The competition with the highest reward, a prize pool of $100,000, was the Global Artificial Intelligence Championships, track Maths 2024, a pioneering contest that aimed to advance the development of artificial intelligence tools for solving advanced mathematical problems across multiple levels of difficulty (full report here).
Codabench featured interesting NeurIPS 2024 challenges:
- The LLM Privacy Challenge, where participants were divided into two teams: Red Team and Blue Team, each one aiming at developing attack and defense approaches for data privacy in LLMs.
- Fair Universe — Higgs Uncertainy Challenge, exploring uncertainty-aware AI techniques for High Energy Physics (HEP).
- The Concordia Challenge, focusing on improving the cooperative intelligence of AI systems: promise-keeping, negotiation, reciprocity, reputation, partner choice, compromise, and sanctioning.
- The Erasing the Invisible Challenge, where the task is to remove invisible watermarks from images, while preserving the overall image quality.
- Codabench also hosted the NeurIPS 2024 Checklist Assistant, a self-service tool for the NeurIPS 2024 checklist. This verification assistant helps authors confirm their papers’ compliance and submit improved versions based on the Assistant’s feedback. For more details check out the blog post and the experiment results.
Last but not least, Codabench featured several competitions in the fields of medicine and biology, such as the Dental CBCT Scans, Panoramic X-ray Images, Butterfly Hybrid Detection, and Monitoring Age-related Macular Degeneration Progression In Optical Coherence Tomography.
We’d like to thank the whole community for these exceptional scientific contributions. You can explore more challenges in the public listing.
We always develop new features to serve the state-of-the art of Machine Learning science. We plan to invest effort in handling medical data, improving federated learning use-cases, and allowing human-in-the loop feedback for better AI experience.
The platform interface will also be improved for better visibility and facilitated re-use of datasets, tasks or solutions generated through it.
Keep in touch and give us feedback on your experience on Codabench !
Reminder on our communication tools:
- Join our google forum to emphasize your competitions and events
- Contact us for any question: [email protected]
- Write an issue on github about interesting suggestions
Please cite one of these papers when working with our platforms:
@article{codabench,
title = {Codabench: Flexible, easy-to-use, and reproducible meta-benchmark platform},
author = {Zhen Xu and Sergio Escalera and Adrien Pavão and Magali Richard and
Wei-Wei Tu and Quanming Yao and Huan Zhao and Isabelle Guyon},
journal = {Patterns},
volume = {3},
number = {7},
pages = {100543},
year = {2022},
issn = {2666-3899},
doi = {https://doi.org/10.1016/j.patter.2022.100543},
url = {https://www.sciencedirect.com/science/article/pii/S2666389922001465}
}
@article{codalab_competitions_JMLR,
author = {Adrien Pavao and Isabelle Guyon and Anne-Catherine Letournel and Dinh-Tuan Tran and Xavier Baro and Hugo Jair Escalante and Sergio Escalera and Tyler Thomas and Zhen Xu},
title = {CodaLab Competitions: An Open Source Platform to Organize Scientific Challenges},
journal = {Journal of Machine Learning Research},
year = {2023},
volume = {24},
number = {198},
pages = {1--6},
url = {http://jmlr.org/papers/v24/21-1436.html}
}
Thank you for reading the first edition of our newsletter. We look forward to sharing more exciting updates, competitions, and breakthroughs with you soon. Until then, keep exploring, keep competing, and stay inspired!
