Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

benchmark: add sampling.renorm benchmarks #970

Merged
merged 3 commits into from
Mar 25, 2025

Conversation

xslingcn
Copy link
Contributor

@xslingcn xslingcn commented Mar 24, 2025

Partial results:

---
top-p renorm
vocab_size: 128512, batch_size: 512, distrib: normal_distribution(std=1), p: 0.1, duration: 2014.94 us, effective bandwidth: 261.24 GB/s
vocab_size: 128512, batch_size: 512, distrib: normal_distribution(std=1), p: 0.9, duration: 2568.39 us, effective bandwidth: 204.95 GB/s
vocab_size: 128512, batch_size: 512, distrib: gumbel_distribution(beta=1), p: 0.1, duration: 2021.48 us, effective bandwidth: 260.40 GB/s
vocab_size: 128512, batch_size: 512, distrib: gumbel_distribution(beta=1), p: 0.9, duration: 2377.93 us, effective bandwidth: 221.36 GB/s
---
top-k renorm
vocab_size: 128512, batch_size: 512, distrib: normal_distribution(std=1), k: 10, duration: 1761.15 us, effective bandwidth: 298.89 GB/s
vocab_size: 128512, batch_size: 512, distrib: normal_distribution(std=1), k: 100, duration: 2299.25 us, effective bandwidth: 228.94 GB/s
vocab_size: 128512, batch_size: 512, distrib: gumbel_distribution(beta=1), k: 10, duration: 1830.94 us, effective bandwidth: 287.49 GB/s
vocab_size: 128512, batch_size: 512, distrib: gumbel_distribution(beta=1), k: 100, duration: 2286.19 us, effective bandwidth: 230.25 GB/s
---
top-k mask logits
vocab_size: 128512, batch_size: 512, distrib: normal_distribution(std=1), k: 10, duration: 1401.01 us, effective bandwidth: 375.72 GB/s
vocab_size: 128512, batch_size: 512, distrib: normal_distribution(std=1), k: 100, duration: 1661.22 us, effective bandwidth: 316.87 GB/s
vocab_size: 128512, batch_size: 512, distrib: gumbel_distribution(beta=1), k: 10, duration: 1551.09 us, effective bandwidth: 339.36 GB/s
vocab_size: 128512, batch_size: 512, distrib: gumbel_distribution(beta=1), k: 100, duration: 1798.39 us, effective bandwidth: 292.70 GB/s

Copy link
Collaborator

@yzh119 yzh119 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's merge this first, please create another PR for the 2/3 pivot renorm.

@yzh119 yzh119 merged commit 588c2fb into flashinfer-ai:main Mar 25, 2025
2 checks passed
@xslingcn xslingcn deleted the bench-renorm branch March 26, 2025 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants