Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,15 @@ uv pip install -e ".[gpu-cu13]"

The app auto-detects GPU availability at runtime and falls back to CPU if anything goes wrong — no configuration needed. The CPU sklearn path is auto-accelerated by [scikit-learn-intelex](https://github.com/uxlfoundation/scikit-learn-intelex)[^1]. You can also manually select backends (`cuML`, `sklearn`) in the sidebar.

### Reproducibility

To get reproducible projections and clusters, enable **Use fixed seed** in the sidebar and pin the backend instead of `auto`: the GPU backend is `cuML`, the CPU backend is `sklearn` (auto-accelerated by `scikit-learn-intelex` on x86 CPUs). With a seed and a pinned backend, results are identical across app restarts on both backends, with one exception: cuML t-SNE, which never reproduces exactly.

- **PCA** is a deterministic decomposition, with no stochastic optimization involved. cuML PCA uses a full eigendecomposition and always returns the same result, seed or no seed. sklearn can auto-select a randomized SVD solver, so the app passes the seed to make it reproducible.
- **UMAP** and **KMeans** reproduce exactly on both backends when a seed is set. (Seeded UMAP trades some speed for determinism.)
- **t-SNE** reproduces on `sklearn` when a seed is set. cuML's implementation is highly parallelized and documented as [not completely deterministic between runs, even with the same `random_state`](https://docs.rapids.ai/api/cuml/stable/api/generated/cuml.manifold.tsne/) (see [rapidsai/cuml#2980](https://github.com/rapidsai/cuml/issues/2980)). Select `sklearn` when t-SNE results need to be reproducible.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@egrace479 very interesting comment it made here.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to have corrected the 26.4 statement to the actual release of 26.04 but then fallen back to semver to say well this clearly should be fine since 26.4 would come after 26.08 😵‍💫

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Experiment

max pairwise diff across 3 seeded runs, seed 614, 2000×768 danaus embeddings, A100 on ascend

┌────────────┬────────────┬────────────┐
│   method   │ cuML 26.06 │ cuML 26.08 │
├────────────┼────────────┼────────────┤
│ fft        │   122.06   │      0.0   │
├────────────┼────────────┼────────────┤
│ barnes_hut │   501.62   │   506.45   │
├────────────┼────────────┼────────────┤
│ exact      │   109.34   │    97.22   │
└────────────┴────────────┴────────────┘

I think this concludes that, when using cuml for t-SNE, to have reproducible result

  • cuml >= 26.8 AND
  • solver = fft

I was surprised that the exact solver wasn't reproducing on these two versions...

However, the fft solver has the same collapsing issue that we encountered earlier #40

Conclusion

I don't think currently the backend cuml has a t-SNE solver that's both reproducible & collapse-free.

Bump the cuml requirement to 26.08

I'd leave the solver for cuml unchanged, fixed at exact to prevent collapsing on homogenous embeddings. Document the irreproducibility, direct the to sklearn for reproducible results.

- `auto` chooses a backend from data size and hardware, so the same seed can run different algorithms on different machines. Exact coordinates may also differ across library versions and hardware; a seed guarantees repeatability within one environment, not across environments.

## Usage

### Standalone Apps
Expand Down
6 changes: 5 additions & 1 deletion shared/utils/clustering.py
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,9 @@ def _reduce_dim_sklearn(embeddings: np.ndarray, method: str, seed: Optional[int]
effective_workers = -1 if n_workers > 1 else n_workers

if method.upper() == "PCA":
reducer = PCA(n_components=2)
# Pass random_state so the randomized SVD solver (auto-selected for
# large inputs) is reproducible when a seed is set; None keeps it random.
reducer = PCA(n_components=2, random_state=seed)
elif method.upper() == "TSNE":
# Adjust perplexity to be valid for the sample size
n_samples = embeddings.shape[0]
Expand Down Expand Up @@ -244,6 +246,8 @@ def _reduce_dim_cuml(embeddings: np.ndarray, method: str, seed: Optional[int], n

if method.upper() == "PCA":
from cuml.decomposition import PCA as cuPCA
# cuML PCA takes no random_state and needs none: its full-SVD solver
# is deterministic, so results are already reproducible run-to-run.
reducer = cuPCA(n_components=2)
elif method.upper() == "TSNE":
from cuml.manifold import TSNE as cuTSNE
Expand Down
Loading