A command-line interface tool for submitting kernel implementations to BackendBench. This tool streamlines the submission process for both single kernel files and batch submissions from directories.
Client-Server Model:
- CLI (Client): Submit kernels, view submissions
- Server: FastAPI server with SQLite database
- Submit single kernel implementations with detailed metadata
- Batch submit multiple kernels from a directory structure
- View and filter submission history from server
- Simple FastAPI server for receiving and storing submissions
cd leaderboard_cli
pip install -e .
This installs both the CLI and server dependencies.
cd server
python main.py
Server will run at http://localhost:8000
leaderboard login
You'll need a GitHub Personal Access Token:
- Generate at: https://github.com/settings/tokens
- Required scopes:
read:user
,user:email
Open a new terminal:
Submit a single kernel file with operation metadata:
leaderboard submit \
--op add \
--overload Tensor \
--dsl cutedsl \
--device A100 \
--file add_implementation_v1.py
Submit all kernels in a directory. The tool will automatically detect the operation and overload types from the directory structure:
leaderboard submit \
--dsl triton \
--device A100 \
--directory generated_kernels/
Expected directory structure:
generated_kernels/
├── add/
│ ├── Tensor/
│ │ ├── add_v1.py
│ │ └── add_v2.py
│ └── Float/
│ └── add_v1.py
├── mul/
│ └── mul_v1.py
└── matmul/
└── Tensor/
└── matmul_optimized.py
List all submissions:
leaderboard list
Filter by operation, DSL, or device:
leaderboard list --op add --dsl triton --limit 10
Show file content:
leaderboard list --show-content --limit 5
Show details of a specific submission by ID:
leaderboard show 1
Authenticate with GitHub Personal Access Token
Clear authentication and log out
Show current authenticated user
Submit kernel implementation(s) to the server. Requires authentication.
Options:
--op, --operation
: Operation type (e.g., add, mul, matmul) - Required for single file--overload
: Overload type (e.g., Tensor, Float, Int) - Optional--dsl
: DSL type (e.g., cutedsl, triton, cuda) - Required--device
: Device type (e.g., A100, H100, V100) - Required--file
: Path to a single kernel file--directory
: Path to directory containing multiple kernels--endpoint
: API endpoint URL (default: http://localhost:8000/api/submit)
List submitted kernels from the server.
Options:
--op, --operation
: Filter by operation type--dsl
: Filter by DSL type--device
: Filter by device type--limit
: Maximum number of results (default: 20)--endpoint
: API base URL (default: http://localhost:8000)
Show details of a specific submission by ID.
Arguments:
submission_id
: The ID of the submission to display
Options:
--endpoint
: API base URL (default: http://localhost:8000)
- User authenticates → GitHub OAuth via Personal Access Token
- Server issues JWT → Token stored locally in
~/.leaderboard/config.json
- CLI submits kernels → Sends HTTP POST with JWT Bearer token
- Server verifies & stores → Links submission to user in database
- CLI queries server → View all submissions via API
Default endpoint: http://localhost:8000
Authentication:
- Required for: Submissions
- Not required for: Viewing submissions
Data stored on server:
- User information (username, name, email from GitHub)
- Operation type
- Overload type (if specified)
- DSL type
- Device type
- File name and content
- Timestamp
- User who submitted
leaderboard submit \
--op matmul \
--overload Tensor \
--dsl cuda \
--device A100 \
--file kernels/matmul_optimized.cu
leaderboard submit \
--dsl triton \
--device H100 \
--directory ./my_triton_kernels/
leaderboard list --op add --limit 5
leaderboard_cli/
├── leaderboard/ # CLI package
│ ├── __init__.py
│ ├── cli.py # CLI commands
│ └── submit.py # Submission logic
├── server/ # FastAPI server
│ ├── main.py # Server implementation
│ ├── requirements.txt
│ └── README.md
├── examples/ # Example kernel files
├── pyproject.toml
└── README.md
# Install dev dependencies
pip install -e ".[dev]"
# Run tests
pytest
MIT License