~ 35.63% faster filtering #3330

kaushalprasadhial · 2024-10-29T08:10:50Z

We are submitting PR for speed up of the filtering

	Time
Original	290.59
Updated	187.03
Speedup	35.63%

Experiment was done on AWS r7i.24xlarge

import time
import numpy as np

import pandas as pd

import scanpy as sc
from sklearn.cluster import KMeans

import os
import wget

import warnings


warnings.filterwarnings('ignore', 'Expected ')
warnings.simplefilter('ignore')
input_file = "./1M_brain_cells_10X.sparse.h5ad"

if not os.path.exists(input_file):
    print('Downloading import file...')
    wget.download('https://rapids-single-cell-examples.s3.us-east-2.amazonaws.com/1M_brain_cells_10X.sparse.h5ad',input_file)


# marker genes
MITO_GENE_PREFIX = "mt-" # Prefix for mitochondrial genes to regress out
markers = ["Stmn2", "Hes1", "Olig1"] # Marker genes for visualization

# filtering cells
min_genes_per_cell = 200 # Filter out cells with fewer genes than this expressed
max_genes_per_cell = 6000 # Filter out cells with more genes than this expressed

# filtering genes
min_cells_per_gene = 1 # Filter out genes expressed in fewer cells than this
n_top_genes = 4000 # Number of highly variable genes to retain

# PCA
n_components = 50 # Number of principal components to compute

# t-SNE
tsne_n_pcs = 20 # Number of principal components to use for t-SNE

# k-means
k = 35 # Number of clusters for k-means

# Gene ranking

ranking_n_top_genes = 50 # Number of differential genes to compute for each cluster

# Number of parallel jobs
sc._settings.ScanpyConfig.n_jobs = os.cpu_count()

start=time.time()
adata = sc.read(input_file)
adata.var_names_make_unique()
adata.shape

tr=time.time()
# To reduce the number of cells:
USE_FIRST_N_CELLS = 1300000
adata = adata[0:USE_FIRST_N_CELLS]
adata.shape

sc.pp.filter_cells(adata, min_genes=min_genes_per_cell)
print("Total filter cell : %s" % (time.time()-tr))
sc.pp.filter_cells(adata, max_genes=max_genes_per_cell)
tfg=time.time()
sc.pp.filter_genes(adata, min_cells=min_cells_per_gene)
print("Total filter genes : %s" % (time.time()-tfg))
tnt=time.time()
sc.pp.normalize_total(adata, target_sum=1e4)
print("Total filter normalize_total : %s" % (time.time()-tnt))
print("Total filter and normalize time : %s" % (time.time()-tr))

Closes #
Tests included or not required because:

Release notes not necessary because:

codecov · 2024-10-29T08:29:43Z

Codecov Report

Attention: Patch coverage is 22.44898% with 38 lines in your changes missing coverage. Please review.

Project coverage is 72.25%. Comparing base (a70582e) to head (25e7cd4).
Report is 103 commits behind head on main.

Files with missing lines	Patch %	Lines
src/scanpy/preprocessing/_simple.py	22.44%	38 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3330      +/-   ##
==========================================
- Coverage   76.27%   72.25%   -4.03%     
==========================================
  Files         117      111       -6     
  Lines       12795    12639     -156     
==========================================
- Hits         9760     9132     -628     
- Misses       3035     3507     +472

Files with missing lines	Coverage Δ
src/scanpy/preprocessing/_simple.py	`61.67% <22.44%> (ø)`

... and 65 files with indirect coverage changes