fix: preserve c-TF-IDF idf diagonal in delete_topics (#2530)

[kiruthick01]

What is wrong

After a model has had topics removed with delete_topics, topics_over_time (and any other path that relies on ctfidf_model.transform) raises:

ValueError: matmul: dimension mismatch with signature (n,k=139),(k=138,m)->(n,m)

This is #2530.

Root cause

ClassTfidfTransformer._idf_diag is a (n_features, n_features) diagonal matrix over the vocabulary (built in ClassTfidfTransformer.fit as sp.diags(idf, shape=(n_features, n_features))), and transform applies it as X = X * self._idf_diag. It is independent of the number of topics.

delete_topics, however, mutated it as if it were indexed by topic, in two places:

1. When adding the -1 topic for the first time, it prepended a row:

   outlier_diag = sp.csr_matrix(([1.0], ([0], [0])), shape=(1, n_features))
   self.ctfidf_model._idf_diag = sp.vstack([outlier_diag, self.ctfidf_model._idf_diag])

2. It then masked rows along the topic axis for the deleted topics:

   mask = np.array([topic not in topics_to_delete for topic in range(self.ctfidf_model._idf_diag.shape[0])])
   self.ctfidf_model._idf_diag = self.ctfidf_model._idf_diag[mask]

Both change the row count of a matrix whose rows are vocabulary features, not topics, leaving _idf_diag non-square so X * _idf_diag fails. (When exactly one topic is deleted from a model that had no -1 yet, the +1/-1 happen to cancel in shape, but the diagonal is then silently misaligned with the vocabulary.)

Reproduction

import numpy as np
from sklearn.cluster import KMeans
from bertopic import BERTopic
from bertopic.dimensionality import BaseDimensionalityReduction

docs = [f"sample document subject {i%6} word{i%9} token{i%4} n{i}" for i in range(120)]
emb = np.random.RandomState(1).rand(len(docs), 12).astype("float32")
ts = [i % 20 for i in range(len(docs))]

m = BERTopic(embedding_model=None, hdbscan_model=KMeans(6, random_state=42),
             umap_model=BaseDimensionalityReduction())
m.fit_transform(docs, embeddings=emb)
m.topics_over_time(docs, ts, nr_bins=4)   # works

m.delete_topics([1, 2])
m.topics_over_time(docs, ts, nr_bins=4)   # ValueError: matmul: dimension mismatch

What changed

delete_topics no longer modifies ctfidf_model._idf_diag. Deleting topics (or adding the -1 outlier) does not change the vocabulary, so the idf diagonal is left as-is. The genuinely topic-indexed attributes (topic_embeddings_, c_tf_idf_, representations, representative docs/images, etc.) are still updated exactly as before.

Tests

Added test_delete_topics_preserves_ctfidf_idf_diag, parametrized over kmeans_pca_topic_model (no -1) and base_topic_model (with -1). It asserts _idf_diag stays square and unchanged in shape after delete_topics, and that topics_over_time succeeds afterwards. The test fails on master (_idf_diag becomes non-square, e.g. (18365, 18367) != (18367, 18367)) and passes with this change.

Validation

• pytest tests/test_reduction/test_delete.py tests/test_variations/test_dynamic.py → 13 passed
• ruff check and ruff format --check on the changed files → clean

Fixes #2530.

I kept the change minimal and focused on the idf-diagonal handling. Happy to adjust the approach or the test based on your preference.