Skip to content

Commit 45a9b9a

Browse files
committed
Update for rolling leaderboard
1 parent 1ee83e2 commit 45a9b9a

File tree

1 file changed

+5
-48
lines changed

1 file changed

+5
-48
lines changed

docs/GETTING_STARTED.md

Lines changed: 5 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,7 @@
1818
- [Docker Tips](#docker-tips)
1919
- [Score your Submission](#score-your-submission)
2020
- [Running workloads](#running-workloads)
21-
- [Package your Submission code](#package-your-submission-code)
22-
- [Package Logs for Self-Reporting Submissions](#package-logs-for-self-reporting-submissions)
21+
- [Submit your Submission](#submit-your-submission)
2322

2423
## Set Up and Installation
2524

@@ -31,7 +30,7 @@ The specs on the benchmarking machines are:
3130
- 8xV100 16GB GPUs
3231
- 240 GB in RAM
3332
- 2 TB in storage (for datasets).
34-
3. Install the algorithmic package and dependencies either in a [Python virtual environment](#python-virtual-environment) or use a [Docker](#docker) (recommended) or [Singularity/Apptainer container](#using-singularityapptainer-instead-of-docker).
33+
3. Install the `algoperf` package and dependencies either in a [Python virtual environment](#python-virtual-environment) or use a [Docker](#docker) (recommended) or [Singularity/Apptainer container](#using-singularityapptainer-instead-of-docker).
3534

3635
### Python Virtual Environment
3736

@@ -252,7 +251,7 @@ python3 submission_runner.py \
252251
--tuning_search_space=<path_to_tuning_search_space>
253252
```
254253
255-
**Pytorch**: to score your submission on a workload, from the algorithmic-efficency directory run:
254+
**PyTorch**: to score your submission on a workload, from the algorithmic-efficency directory run:
256255
257256
```bash
258257
python3 submission_runner.py \
@@ -389,50 +388,8 @@ python score_submissions.py --submission_directory <directory_with_submissions>
389388

390389
We provide the scores and performance profiles for the [paper baseline algorithms](/reference_algorithms/paper_baselines/) in the "Baseline Results" section in [Benchmarking Neural Network Training Algorithms](https://arxiv.org/abs/2306.07179).
391390

392-
## Package your Submission code
391+
## Submit your Submission
393392

394-
If you have registered for the AlgoPerf competition you will receive
395-
an email on 3/27/2024 with a link to a UI to upload a compressed submission folder.
396-
397-
To package your submission modules please make sure your submission folder is structured as follows:
398-
399-
```bash
400-
submission_folder/
401-
├── external_tuning
402-
│ ├── algorithm_name
403-
│ │ ├── helper_module.py
404-
│ │ ├── requirements.txt
405-
│ │ ├── submission.py
406-
│ │ └── tuning_search_space.json
407-
│ └── other_algorithm_name
408-
│ ├── requirements.txt
409-
│ ├── submission.py
410-
│ └── tuning_search_space.json
411-
└── self_tuning
412-
└── algorithm_name
413-
├── requirements.txt
414-
└── submission.py
415-
```
416-
417-
Specifically we require that:
418-
419-
1. There exist subdirectories in the the submission folder named after the ruleset: `external_tuning` or `self_tuning`.
420-
2. The ruleset subdirectories contain directories named according to
421-
some identifier of the algorithm.
422-
3. Each algorithm subdirectory contains a `submission.py` module. Additional helper modules are allowed if prefer to you organize your code into multiple files. If there are additional python packages that have to be installed for the algorithm also include a `requirements.txt` with package names and versions in the algorithm subdirectory.
423-
4. For `external_tuning` algorithms the algorithm subdirectory
424-
should contain a `tuning_search_space.json`.
425-
426-
To check that your submission folder meets the above requirements you can run the `submissions/repo_checker.py` script.
427-
428-
## Package Logs for Self-Reporting Submissions
429-
430-
To prepare your submission for self reporting run:
431-
432-
```bash
433-
python3 package_logs.py --experiment_dir <experiment_dir> --destination_dir <destination_dir>
434-
```
435-
436-
The destination directiory will contain the logs packed in studies and trials required for self-reporting.
393+
To submit your submission, please create a PR on the [submission repository](https://github.com/mlcommons/submissions_algorithms). You can find more details in the submission repositories [How to Submit](https://github.com/mlcommons/submissions_algorithms?tab=readme-ov-file#how-to-submit) section. The working group will review your PR and select the most promising submissions for scoring.
437394

438395
**Good Luck!**

0 commit comments

Comments
 (0)