`sepal` numba compilation options changes the results significantly

[selmanozleyen]

I am trying to integrate sq.gr.sepal into rapidsinglecell for a gpu implementation. However, when comparing results I noticed this cpu implementation is very sensitive to numba compilation options. The results are not only different by magnitude it also changes the rankings of the outputs.

Here is my script

import warnings
warnings.filterwarnings("ignore")
import squidpy as sq
import time
import anndata as ad
from pathlib import Path
import os
from argparse import ArgumentParser

HOME = Path(os.path.expanduser("~"))

if __name__ == "__main__":
    parser = ArgumentParser()
    parser.add_argument("--debug", action="store_true")
    args = parser.parse_args()
    # made by
    # adata = sq.datasets.visium_hne_adata()
    #  sq.gr.spatial_neighbors(adata)
    #  adata.write_h5ad(HOME / "data/visium_hne_adata.h5ad")
    adata = ad.read_h5ad(HOME / "data/visium_hne_adata.h5ad")
    start_time = time.time()
    genes = ["Gm29570"]
    result = sq.gr.sepal(adata, max_neighs=6, genes=genes, n_iter=30000, copy=True)
    end_time = time.time()
    print(f"Time taken: {end_time - start_time} seconds")
    result.sort_values(by="sepal_score", ascending=False, inplace=True)
    print(result.head(10))

Here are my configurations: changes to _sepal.pyfiles

• all njit from _sepal.py are commented out

Time taken: 237.62845134735107 seconds
         sepal_score
Gm29570       19.006

• njit stays but all fastmath=False

Time taken: 9.271298170089722 seconds
         sepal_score
Gm29570       16.612

• original

genes ['Gm29570']
Time taken: 5.706198215484619 seconds
         sepal_score
Gm29570        0.014

I noticed similar things on pertpy EDistance adoption to rapidsinglecell. Also probably related: #1030

Here is my env.

List of packages in environment: "/home/ubuntu/miniforge3/envs/sepal"

  Name                              Version                 Build                                Channel    
──────────────────────────────────────────────────────────────────────────────────────────────────────────────
  _libgcc_mutex                     0.1                     conda_forge                          conda-forge
  _openmp_mutex                     4.5                     2_gnu                                conda-forge
  _python_abi3_support              1.0                     hd8ed1ab_2                           conda-forge
  aiohappyeyeballs                  2.6.1                   pyhd8ed1ab_0                         conda-forge
  aiohttp                           3.12.15                 py313h3dea7bd_0                      conda-forge
  aiosignal                         1.4.0                   pyhd8ed1ab_0                         conda-forge
  anndata                           0.12.2                  pypi_0                               pypi       
  anyio                             4.10.0                  pyhe01879c_0                         conda-forge
  aom                               3.9.1                   hac33072_0                           conda-forge
  argon2-cffi                       25.1.0                  pyhd8ed1ab_0                         conda-forge
  argon2-cffi-bindings              25.1.0                  py313h07c4f96_0                      conda-forge
  arrow                             1.3.0                   pyhd8ed1ab_1                         conda-forge
  asttokens                         3.0.0                   pyhd8ed1ab_1                         conda-forge
  async-lru                         2.0.5                   pyh29332c3_0                         conda-forge
  attr                              2.5.2                   h39aace5_0                           conda-forge
  attrs                             25.3.0                  pyh71513ae_0                         conda-forge
  aws-c-auth                        0.9.0                   h0fbd49f_19                          conda-forge
  aws-c-cal                         0.9.2                   he7b75e1_1                           conda-forge
  aws-c-common                      0.12.4                  hb03c661_0                           conda-forge
  aws-c-compression                 0.3.1                   h92c474e_6                           conda-forge
  aws-c-event-stream                0.5.5                   h149bd38_3                           conda-forge
  aws-c-http                        0.10.4                  h37a7233_0                           conda-forge
  aws-c-io                          0.21.2                  h6252d9a_1                           conda-forge
  aws-c-mqtt                        0.13.3                  h19deb91_3                           conda-forge
  aws-c-s3                          0.8.6                   h800fcd2_2                           conda-forge
  aws-c-sdkutils                    0.2.4                   h92c474e_1                           conda-forge
  aws-checksums                     0.2.7                   h92c474e_2                           conda-forge
  aws-crt-cpp                       0.33.1                  hb4fd278_2                           conda-forge
  aws-sdk-cpp                       1.11.606                h31ade35_1                           conda-forge
  azure-core-cpp                    1.16.0                  h3a458e0_0                           conda-forge
  azure-identity-cpp                1.12.0                  ha729027_0                           conda-forge
  azure-storage-blobs-cpp           12.14.0                 hb1c9500_1                           conda-forge
  azure-storage-common-cpp          12.10.0                 hebae86a_2                           conda-forge
  azure-storage-files-datalake-cpp  12.12.0                 h8b27e44_3                           conda-forge
  babel                             2.17.0                  pyhd8ed1ab_0                         conda-forge
  beautifulsoup4                    4.13.5                  pyha770c72_0                         conda-forge
  bleach                            6.2.0                   pyh29332c3_4                         conda-forge
  bleach-with-css                   6.2.0                   h82add2a_4                           conda-forge
  blosc                             1.21.6                  he440d0b_1                           conda-forge
  bokeh                             3.8.0                   pyhd8ed1ab_0                         conda-forge
  branca                            0.8.1                   pyhd8ed1ab_0                         conda-forge
  brotli                            1.1.0                   hb03c661_4                           conda-forge
  brotli-bin                        1.1.0                   hb03c661_4                           conda-forge
  brotli-python                     1.1.0                   py313h7033f15_4                      conda-forge
  brunsli                           0.1                     he3183e4_1                           conda-forge
  bzip2                             1.0.8                   hda65f42_8                           conda-forge
  c-ares                            1.34.5                  hb9d3cd8_0                           conda-forge
  c-blosc2                          2.21.2                  h4cfbee9_0                           conda-forge
  ca-certificates                   2025.8.3                hbd8a1cb_0                           conda-forge
  cached-property                   1.5.2                   hd8ed1ab_1                           conda-forge
  cached_property                   1.5.2                   pyha770c72_1                         conda-forge
  cachetools                        6.2.0                   pyhd8ed1ab_0                         conda-forge
  certifi                           2025.8.3                pyhd8ed1ab_0                         conda-forge
  cffi                              1.17.1                  py313hf01b4d8_1                      conda-forge
  charls                            2.4.2                   h59595ed_0                           conda-forge
  charset-normalizer                3.4.3                   pyhd8ed1ab_0                         conda-forge
  click                             8.2.1                   pyh707e725_0                         conda-forge
  cloudpickle                       3.1.1                   pyhd8ed1ab_0                         conda-forge
  colorama                          0.4.6                   pyhd8ed1ab_1                         conda-forge
  colorcet                          3.1.0                   pyhd8ed1ab_1                         conda-forge
  comm                              0.2.3                   pyhe01879c_0                         conda-forge
  contourpy                         1.3.3                   py313h7037e92_2                      conda-forge
  cpython                           3.13.7                  py313hd8ed1ab_100                    conda-forge
  cucim                             25.08.00                cuda12_py313_250806_g86b4ac7_0       rapidsai   
  cuda-bindings                     12.9.2                  py313h8055221_1                      conda-forge
  cuda-cccl_linux-64                12.8.90                 ha770c72_1                           conda-forge
  cuda-crt-dev_linux-64             12.8.93                 ha770c72_3                           conda-forge
  cuda-crt-tools                    12.8.93                 ha770c72_3                           conda-forge
  cuda-cudart                       12.8.90                 h5888daf_1                           conda-forge
  cuda-cudart-dev                   12.8.90                 h5888daf_1                           conda-forge
  cuda-cudart-dev_linux-64          12.8.90                 h3f2d84a_1                           conda-forge
  cuda-cudart-static                12.8.90                 h5888daf_1                           conda-forge
  cuda-cudart-static_linux-64       12.8.90                 h3f2d84a_1                           conda-forge
  cuda-cudart_linux-64              12.8.90                 h3f2d84a_1                           conda-forge
  cuda-nvcc-dev_linux-64            12.8.93                 he91c749_3                           conda-forge
  cuda-nvcc-impl                    12.8.93                 h85509e4_3                           conda-forge
  cuda-nvcc-tools                   12.8.93                 he02047a_3                           conda-forge
  cuda-nvrtc                        12.8.93                 h5888daf_1                           conda-forge
  cuda-nvvm-dev_linux-64            12.8.93                 ha770c72_3                           conda-forge
  cuda-nvvm-impl                    12.8.93                 he02047a_3                           conda-forge
  cuda-nvvm-tools                   12.8.93                 he02047a_3                           conda-forge
  cuda-pathfinder                   1.2.2                   pyhcf101f3_0                         conda-forge
  cuda-profiler-api                 12.8.90                 h7938cbb_1                           conda-forge
  cuda-python                       12.9.2                  pyhb6702c7_2                         conda-forge
  cuda-version                      12.8                    h5d125a7_3                           conda-forge
  cudf                              25.08.00                cuda12_py313_250806_6cea3743         rapidsai   
  cudf-polars                       25.08.00                cuda12_py313_250806_6cea3743         rapidsai   
  cudf_kafka                        25.08.00                cuda12_py313_250806_6cea3743         rapidsai   
  cugraph                           25.08.00                cuda12_py313_250806_e0f09e50         rapidsai   
  cuml                              25.08.00                cuda12_py313_250806_b081fcd0         rapidsai   
  cupy                              13.6.0                  py313h586c94b_0                      conda-forge
  cupy-core                         13.6.0                  py313h28b6081_0                      conda-forge
  custreamz                         25.08.00                cuda12_py313_250806_6cea3743         rapidsai   
  cuvs                              25.08.00                cuda12_py313_250806_9ce11a0f         rapidsai   
  cuxfilter                         25.08.00                cuda12_py313_250806_3858d64d         rapidsai   
  cycler                            0.12.1                  pyhd8ed1ab_1                         conda-forge
  cyrus-sasl                        2.1.28                  hd9c7081_0                           conda-forge
  cytoolz                           1.0.1                   py313h536fd9c_0                      conda-forge
  dask                              2025.7.0                pyhe01879c_0                         conda-forge
  dask-core                         2025.7.0                pyhe01879c_1                         conda-forge
  dask-cuda                         25.08.00                py313_250806_bde9a4d3                rapidsai   
  dask-cudf                         25.08.00                cuda12_py313_250806_6cea3743         rapidsai   
  datashader                        0.18.2                  pyhd8ed1ab_0                         conda-forge
  dav1d                             1.2.1                   hd590300_0                           conda-forge
  debugpy                           1.8.16                  py313h5d5ffb9_1                      conda-forge
  decorator                         5.2.1                   pyhd8ed1ab_0                         conda-forge
  defusedxml                        0.7.1                   pyhd8ed1ab_0                         conda-forge
  distributed                       2025.7.0                pyhe01879c_0                         conda-forge
  distributed-ucxx                  0.45.01                 py_250806_c25d2cdb_h7e233d3          rapidsai   
  dlpack                            0.8                     h59595ed_3                           conda-forge
  exceptiongroup                    1.3.0                   pyhd8ed1ab_0                         conda-forge
  executing                         2.2.1                   pyhd8ed1ab_0                         conda-forge
  fastrlock                         0.8.3                   py313h5d5ffb9_2                      conda-forge
  folium                            0.20.0                  pyhd8ed1ab_0                         conda-forge
  fonttools                         4.59.2                  py313h3dea7bd_0                      conda-forge
  fqdn                              1.5.1                   pyhd8ed1ab_1                         conda-forge
  freetype                          2.14.0                  ha770c72_1                           conda-forge
  freexl                            2.0.0                   h9dce30a_2                           conda-forge
  frozenlist                        1.7.0                   py313h6b9daa2_0                      conda-forge
  fsspec                            2025.9.0                pyhd8ed1ab_0                         conda-forge
  geopandas                         1.1.1                   pyhd8ed1ab_0                         conda-forge
  geopandas-base                    1.1.1                   pyha770c72_0                         conda-forge
  geos                              3.13.1                  h97f6797_0                           conda-forge
  gflags                            2.2.2                   h5888daf_1005                        conda-forge
  giflib                            5.2.2                   hd590300_0                           conda-forge
  glog                              0.7.1                   hbabe93e_0                           conda-forge
  h11                               0.16.0                  pyhd8ed1ab_0                         conda-forge
  h2                                4.3.0                   pyhcf101f3_0                         conda-forge
  holoviews                         1.21.0                  pyhd8ed1ab_0                         conda-forge
  hpack                             4.1.0                   pyhd8ed1ab_0                         conda-forge
  httpcore                          1.0.9                   pyh29332c3_0                         conda-forge
  httpx                             0.28.1                  pyhd8ed1ab_0                         conda-forge
  hyperframe                        6.1.0                   pyhd8ed1ab_0                         conda-forge
  icu                               75.1                    he02047a_0                           conda-forge
  idna                              3.10                    pyhd8ed1ab_1                         conda-forge
  imagecodecs                       2025.8.2                py313h718aa22_4                      conda-forge
  imageio                           2.37.0                  pyhfb79c49_0                         conda-forge
  importlib-metadata                8.7.0                   pyhe01879c_1                         conda-forge
  ipykernel                         6.30.1                  pyh82676e8_0                         conda-forge
  ipython                           9.5.0                   pyhfa0c392_0                         conda-forge
  ipython_pygments_lexers           1.1.1                   pyhd8ed1ab_0                         conda-forge
  isoduration                       20.11.0                 pyhd8ed1ab_1                         conda-forge
  jedi                              0.19.2                  pyhd8ed1ab_1                         conda-forge
  jinja2                            3.1.6                   pyhd8ed1ab_0                         conda-forge
  joblib                            1.5.2                   pyhd8ed1ab_0                         conda-forge
  json-c                            0.18                    h6688a6e_0                           conda-forge
  json5                             0.12.1                  pyhd8ed1ab_0                         conda-forge
  jsonpointer                       3.0.0                   py313h78bf25f_2                      conda-forge
  jsonschema                        4.25.1                  pyhe01879c_0                         conda-forge
  jsonschema-specifications         2025.9.1                pyhcf101f3_0                         conda-forge
  jsonschema-with-format-nongpl     4.25.1                  he01879c_0                           conda-forge
  jupyter-lsp                       2.3.0                   pyhcf101f3_0                         conda-forge
  jupyter-server-proxy              4.4.0                   pyhd8ed1ab_1                         conda-forge
  jupyter_client                    8.6.3                   pyhd8ed1ab_1                         conda-forge
  jupyter_core                      5.8.1                   pyh31011fe_0                         conda-forge
  jupyter_events                    0.12.0                  pyh29332c3_0                         conda-forge
  jupyter_server                    2.17.0                  pyhcf101f3_0                         conda-forge
  jupyter_server_terminals          0.5.3                   pyhd8ed1ab_1                         conda-forge
  jupyterlab                        4.4.7                   pyhd8ed1ab_0                         conda-forge
  jupyterlab_pygments               0.3.0                   pyhd8ed1ab_2                         conda-forge
  jupyterlab_server                 2.27.3                  pyhd8ed1ab_1                         conda-forge
  jxrlib                            1.1                     hd590300_3                           conda-forge
  keyutils                          1.6.3                   hb9d3cd8_0                           conda-forge
  kiwisolver                        1.4.9                   py313hc8edb43_1                      conda-forge
  krb5                              1.21.3                  h659f571_0                           conda-forge
  lark                              1.2.2                   pyhd8ed1ab_1                         conda-forge
  lazy-loader                       0.4                     pyhd8ed1ab_2                         conda-forge
  lazy_loader                       0.4                     pyhd8ed1ab_2                         conda-forge
  lcms2                             2.17                    h717163a_0                           conda-forge
  ld_impl_linux-64                  2.44                    h1423503_1                           conda-forge
  lerc                              4.0.0                   h0aef613_1                           conda-forge
  libabseil                         20250512.1              cxx17_hba17884_0                     conda-forge
  libaec                            1.1.4                   h3f801dc_0                           conda-forge
  libarchive                        3.8.1                   gpl_h98cc613_100                     conda-forge
  libarrow                          19.0.1                  h487d092_28_cpu                      conda-forge
  libarrow-acero                    19.0.1                  h635bf11_28_cpu                      conda-forge
  libarrow-dataset                  19.0.1                  h635bf11_28_cpu                      conda-forge
  libarrow-substrait                19.0.1                  h3f74fd7_28_cpu                      conda-forge
  libavif16                         1.3.0                   h6395336_2                           conda-forge
  libblas                           3.9.0                   35_h4a7cf45_openblas                 conda-forge
  libbrotlicommon                   1.1.0                   hb03c661_4                           conda-forge
  libbrotlidec                      1.1.0                   hb03c661_4                           conda-forge
  libbrotlienc                      1.1.0                   hb03c661_4                           conda-forge
  libcap                            2.76                    h0b2e76d_0                           conda-forge
  libcblas                          3.9.0                   35_h0358290_openblas                 conda-forge
  libcrc32c                         1.1.2                   h9c3ff4c_0                           conda-forge
  libcublas                         12.8.4.1                h9ab20c4_1                           conda-forge
  libcublas-dev                     12.8.4.1                h9ab20c4_1                           conda-forge
  libcucim                          25.08.00                cuda12_250806_g86b4ac7_0             rapidsai   
  libcudf                           25.08.00                cuda12_250806_6cea3743               rapidsai   
  libcudf_kafka                     25.08.00                cuda12_250806_6cea3743               rapidsai   
  libcufft                          11.3.3.83               h5888daf_1                           conda-forge
  libcufile                         1.13.1.3                h628e99a_1                           conda-forge
  libcufile-dev                     1.13.1.3                h5888daf_1                           conda-forge
  libcugraph                        25.08.00                cuda12_250806_e0f09e50               rapidsai   
  libcugraph_etl                    25.08.00                cuda12_250806_e0f09e50               rapidsai   
  libcuml                           25.08.00                cuda12_250806_b081fcd0               rapidsai   
  libcumlprims                      25.08.00                cuda12_py310_250806_59064b25         rapidsai   
  libcurand                         10.3.9.90               h9ab20c4_1                           conda-forge
  libcurand-dev                     10.3.9.90               h9ab20c4_1                           conda-forge
  libcurl                           8.14.1                  h332b0f4_0                           conda-forge
  libcusolver                       11.7.3.90               h9ab20c4_1                           conda-forge
  libcusolver-dev                   11.7.3.90               h9ab20c4_1                           conda-forge
  libcusparse                       12.5.8.93               h5888daf_1                           conda-forge
  libcusparse-dev                   12.5.8.93               h5888daf_1                           conda-forge
  libcuvs                           25.08.00                cuda12_250806_9ce11a0f               rapidsai   
  libdeflate                        1.24                    h86f0d12_0                           conda-forge
  libedit                           3.1.20250104            pl5321h7949ede_0                     conda-forge
  libev                             4.33                    hd590300_2                           conda-forge
  libevent                          2.1.12                  hf998b51_1                           conda-forge
  libexpat                          2.7.1                   hecca717_0                           conda-forge
  libffi                            3.4.6                   h2dba641_1                           conda-forge
  libfreetype                       2.14.0                  ha770c72_1                           conda-forge
  libfreetype6                      2.14.0                  h73754d4_1                           conda-forge
  libgcc                            15.1.0                  h767d61c_5                           conda-forge
  libgcc-ng                         15.1.0                  h69a702a_5                           conda-forge
  libgcrypt-lib                     1.11.1                  hb9d3cd8_0                           conda-forge
  libgdal-core                      3.11.3                  ha8f0914_2                           conda-forge
  libgfortran                       15.1.0                  h69a702a_5                           conda-forge
  libgfortran5                      15.1.0                  hcea5267_5                           conda-forge
  libgomp                           15.1.0                  h767d61c_5                           conda-forge
  libgoogle-cloud                   2.39.0                  hdb79228_0                           conda-forge
  libgoogle-cloud-storage           2.39.0                  hdbdcf42_0                           conda-forge
  libgpg-error                      1.55                    h3f2d84a_0                           conda-forge
  libgrpc                           1.73.1                  h1e535eb_0                           conda-forge
  libhwy                            1.3.0                   h4c17acf_0                           conda-forge
  libiconv                          1.18                    h3b78370_2                           conda-forge
  libjpeg-turbo                     3.1.0                   hb9d3cd8_0                           conda-forge
  libjxl                            0.11.1                  h6cb5226_4                           conda-forge
  libkml                            1.3.0                   hf539b9f_1021                        conda-forge
  libkvikio                         25.08.00                cuda12_9_250806_2f72c39a             rapidsai   
  liblapack                         3.9.0                   35_h47877c9_openblas                 conda-forge
  liblzma                           5.8.1                   hb9d3cd8_2                           conda-forge
  libmpdec                          4.0.0                   hb9d3cd8_0                           conda-forge
  libnghttp2                        1.67.0                  had1ee68_0                           conda-forge
  libnl                             3.11.0                  hb9d3cd8_0                           conda-forge
  libnsl                            2.0.1                   hb9d3cd8_1                           conda-forge
  libntlm                           1.8                     hb9d3cd8_0                           conda-forge
  libnuma                           2.0.18                  hb9d3cd8_3                           conda-forge
  libnvcomp                         4.2.0.11                hb7e823c_1                           conda-forge
  libnvcomp-dev                     4.2.0.11                hb7e823c_1                           conda-forge
  libnvjitlink                      12.9.86                 h5888daf_1                           conda-forge
  libnvjpeg                         12.3.5.92               h5888daf_1                           conda-forge
  libopenblas                       0.3.30                  pthreads_h94d23a6_2                  conda-forge
  libopentelemetry-cpp              1.21.0                  hb9b0907_1                           conda-forge
  libopentelemetry-cpp-headers      1.21.0                  ha770c72_1                           conda-forge
  libparquet                        19.0.1                  h790f06f_28_cpu                      conda-forge
  libpng                            1.6.50                  h421ea60_1                           conda-forge
  libprotobuf                       6.31.1                  h9ef548d_1                           conda-forge
  libraft                           25.08.00                cuda12_250806_2fb92393               rapidsai   
  libraft-headers                   25.08.00                cuda12_250806_2fb92393               rapidsai   
  libraft-headers-only              25.08.00                cuda12_250806_2fb92393               rapidsai   
  librdkafka                        2.8.0                   h2e2c4f7_0                           conda-forge
  libre2-11                         2025.08.12              h7b12aa8_1                           conda-forge
  librmm                            25.08.00                cuda12_250806_63a4d366               rapidsai   
  librttopo                         1.1.0                   hd718a1a_18                          conda-forge
  libsodium                         1.0.20                  h4ab18f5_0                           conda-forge
  libspatialite                     5.1.0                   he17ca71_14                          conda-forge
  libsqlite                         3.50.4                  h0c1763c_0                           conda-forge
  libssh2                           1.11.1                  hcf80075_0                           conda-forge
  libstdcxx                         15.1.0                  h8f9b012_5                           conda-forge
  libstdcxx-ng                      15.1.0                  h4852527_5                           conda-forge
  libsystemd0                       257.9                   h996ca69_0                           conda-forge
  libthrift                         0.22.0                  h454ac66_1                           conda-forge
  libtiff                           4.7.0                   h8261f1e_6                           conda-forge
  libucxx                           0.45.01                 cuda12_250806_c25d2cdb               rapidsai   
  libudev1                          257.9                   h085a93f_0                           conda-forge
  libutf8proc                       2.10.0                  h202a827_0                           conda-forge
  libuuid                           2.41.1                  he9a06e4_0                           conda-forge
  libuv                             1.51.0                  hb03c661_1                           conda-forge
  libwebp-base                      1.6.0                   hd42ef1d_0                           conda-forge
  libxcb                            1.17.0                  h8a09558_0                           conda-forge
  libxcrypt                         4.4.36                  hd590300_1                           conda-forge
  libxgboost                        3.0.3                   rapidsai_h1399887_0                  rapidsai   
  libxml2                           2.13.8                  h04c0eec_1                           conda-forge
  libzlib                           1.3.1                   hb9d3cd8_2                           conda-forge
  libzopfli                         1.0.3                   h9c3ff4c_0                           conda-forge
  linkify-it-py                     2.0.3                   pyhd8ed1ab_1                         conda-forge
  llvmlite                          0.44.0                  py313hfdae721_2                      conda-forge
  locket                            1.0.0                   pyhd8ed1ab_0                         conda-forge
  lz4                               4.4.4                   py313hdd09ace_1                      conda-forge
  lz4-c                             1.10.0                  h5888daf_1                           conda-forge
  lzo                               2.10                    h280c20c_1002                        conda-forge
  mapclassify                       2.10.0                  pyhd8ed1ab_1                         conda-forge
  markdown                          3.9                     pyhd8ed1ab_0                         conda-forge
  markdown-it-py                    4.0.0                   pyhd8ed1ab_0                         conda-forge
  markupsafe                        3.0.2                   py313h8060acc_1                      conda-forge
  matplotlib-base                   3.10.6                  py313h683a580_1                      conda-forge
  matplotlib-inline                 0.1.7                   pyhd8ed1ab_1                         conda-forge
  mdit-py-plugins                   0.5.0                   pyhd8ed1ab_0                         conda-forge
  mdurl                             0.1.2                   pyhd8ed1ab_1                         conda-forge
  minizip                           4.0.10                  h05a5f5f_0                           conda-forge
  mistune                           3.1.4                   pyhcf101f3_0                         conda-forge
  msgpack-python                    1.1.1                   py313h7037e92_1                      conda-forge
  multidict                         6.6.3                   py313h8060acc_0                      conda-forge
  multipledispatch                  0.6.0                   pyhd8ed1ab_1                         conda-forge
  munkres                           1.1.4                   pyhd8ed1ab_1                         conda-forge
  muparser                          2.3.5                   h5888daf_0                           conda-forge
  mypy                              1.17.1                  pypi_0                               pypi       
  narwhals                          2.4.0                   pyhcf101f3_0                         conda-forge
  nbclient                          0.10.2                  pyhd8ed1ab_0                         conda-forge
  nbconvert-core                    7.16.6                  pyh29332c3_0                         conda-forge
  nbformat                          5.10.4                  pyhd8ed1ab_1                         conda-forge
  nccl                              2.27.7.1                h49b9d9a_2                           conda-forge
  ncurses                           6.5                     h2d0b736_3                           conda-forge
  nest-asyncio                      1.6.0                   pyhd8ed1ab_1                         conda-forge
  networkx                          3.5                     pyhe01879c_0                         conda-forge
  nlohmann_json                     3.12.0                  h3f2d84a_0                           conda-forge
  nodejs                            24.4.1                  heeeca48_0                           conda-forge
  notebook-shim                     0.2.4                   pyhd8ed1ab_1                         conda-forge
  numba                             0.61.2                  py313h50b8c88_1                      conda-forge
  numba-cuda                        0.14.1                  pyhcf101f3_0                         conda-forge
  numpy                             2.2.6                   py313h17eae1a_0                      conda-forge
  nvidia-ml-py                      12.575.51               pyhd8ed1ab_0                         conda-forge
  nvtx                              0.2.13                  py313h07c4f96_0                      conda-forge
  nx-cugraph                        25.08.00                py313_250806_7984845a                rapidsai   
  openjpeg                          2.5.3                   h55fea9a_1                           conda-forge
  openssl                           3.5.2                   h26f9b46_0                           conda-forge
  orc                               2.2.0                   h1bc01a4_0                           conda-forge
  overrides                         7.7.0                   pyhd8ed1ab_1                         conda-forge
  packaging                         25.0                    pyh29332c3_1                         conda-forge
  pandas                            2.3.2                   py313h08cd8bf_0                      conda-forge
  pandocfilters                     1.5.0                   pyhd8ed1ab_0                         conda-forge
  panel                             1.8.0                   pyhd8ed1ab_0                         conda-forge
  param                             2.2.1                   pyhd8ed1ab_0                         conda-forge
  parso                             0.8.5                   pyhcf101f3_0                         conda-forge
  partd                             1.4.2                   pyhd8ed1ab_0                         conda-forge
  pcre2                             10.45                   hc749103_0                           conda-forge
  pexpect                           4.9.0                   pyhd8ed1ab_1                         conda-forge
  pickleshare                       0.7.5                   pyhd8ed1ab_1004                      conda-forge
  pillow                            11.3.0                  py313hf46931b_1                      conda-forge
  pip                               25.2                    pyh145f28c_0                         conda-forge
  platformdirs                      4.4.0                   pyhcf101f3_0                         conda-forge
  polars                            1.31.0                  default_h70f2ef1_1                   conda-forge
  polars-default                    1.31.0                  py39hf521cc8_1                       conda-forge
  proj                              9.6.2                   h18fbb6c_2                           conda-forge
  prometheus-cpp                    1.3.0                   ha5d0236_0                           conda-forge
  prometheus_client                 0.22.1                  pyhd8ed1ab_0                         conda-forge
  prompt-toolkit                    3.0.52                  pyha770c72_0                         conda-forge
  propcache                         0.3.1                   py313h8060acc_0                      conda-forge
  psutil                            7.0.0                   py313h07c4f96_1                      conda-forge
  pthread-stubs                     0.4                     hb9d3cd8_1002                        conda-forge
  ptyprocess                        0.7.0                   pyhd8ed1ab_1                         conda-forge
  pure_eval                         0.2.3                   pyhd8ed1ab_1                         conda-forge
  py-xgboost                        3.0.3                   rapidsai_pyh7b92683_0                rapidsai   
  pyarrow                           19.0.1                  py313h78bf25f_0                      conda-forge
  pyarrow-core                      19.0.1                  py313he5f92c8_0_cpu                  conda-forge
  pycparser                         2.22                    pyh29332c3_1                         conda-forge
  pyct                              0.5.0                   pyhd8ed1ab_1                         conda-forge
  pygments                          2.19.2                  pyhd8ed1ab_0                         conda-forge
  pylibcudf                         25.08.00                cuda12_py313_250806_6cea3743         rapidsai   
  pylibcugraph                      25.08.00                cuda12_py313_250806_e0f09e50         rapidsai   
  pylibraft                         25.08.00                cuda12_py313_250806_2fb92393         rapidsai   
  pynvjitlink                       0.7.0                   12.9_py313_250625_3693c9cd_hd6618c4  rapidsai   
  pynvml                            12.0.0                  pyhd8ed1ab_0                         conda-forge
  pyogrio                           0.11.1                  py313h8b61037_1                      conda-forge
  pyparsing                         3.2.3                   pyhe01879c_2                         conda-forge
  pyproj                            3.7.2                   py313hcfca4fd_1                      conda-forge
  pysocks                           1.7.1                   pyha55dd90_7                         conda-forge
  python                            3.13.7                  h2b335a9_100_cp313                   conda-forge
  python-confluent-kafka            2.8.0                   py313h536fd9c_0                      conda-forge
  python-dateutil                   2.9.0.post0             pyhe01879c_2                         conda-forge
  python-fastjsonschema             2.21.2                  pyhe01879c_0                         conda-forge
  python-gil                        3.13.7                  h4df99d1_100                         conda-forge
  python-json-logger                2.0.7                   pyhd8ed1ab_0                         conda-forge
  python-tzdata                     2025.2                  pyhd8ed1ab_0                         conda-forge
  python_abi                        3.13                    8_cp313                              conda-forge
  pytz                              2025.2                  pyhd8ed1ab_0                         conda-forge
  pyviz_comms                       3.0.6                   pyhd8ed1ab_0                         conda-forge
  pywavelets                        1.9.0                   py313h29aa505_1                      conda-forge
  pyyaml                            6.0.2                   py313h8060acc_2                      conda-forge
  pyzmq                             27.1.0                  py312hfb55c3c_0                      conda-forge
  qhull                             2020.2                  h434a139_5                           conda-forge
  raft-dask                         25.08.00                cuda12_py313_250806_2fb92393         rapidsai   
  rapids                            25.08.00                cuda12_py313_250806_6825ee7c         rapidsai   
  rapids-dask-dependency            25.08.00                250806_fb4cb9f6                      rapidsai   
  rapids-logger                     0.1.1                   h98325ef_0                           rapidsai   
  rapids-xgboost                    25.08.00                cuda12_py313_250806_6825ee7c         rapidsai   
  rapids_singlecell                 0.13.3.dev6+gde2d747b5  pypi_0                               pypi       
  rav1e                             0.7.1                   h8fae777_3                           conda-forge
  rdma-core                         59.0                    hecca717_0                           conda-forge
  re2                               2025.08.12              h5301d42_1                           conda-forge
  readline                          8.2                     h8c095d6_2                           conda-forge
  referencing                       0.36.2                  pyh29332c3_0                         conda-forge
  requests                          2.32.5                  pyhd8ed1ab_0                         conda-forge
  rfc3339-validator                 0.1.4                   pyhd8ed1ab_1                         conda-forge
  rfc3986-validator                 0.1.1                   pyh9f0ad1d_0                         conda-forge
  rfc3987-syntax                    1.1.0                   pyhe01879c_1                         conda-forge
  rich                              14.1.0                  pyhe01879c_0                         conda-forge
  rmm                               25.08.00                cuda12_py313_250806_63a4d366         rapidsai   
  rpds-py                           0.27.1                  py313h843e2db_1                      conda-forge
  s2n                               1.5.23                  h8e187f5_0                           conda-forge
  scikit-image                      0.24.0                  py313ha87cce1_3                      conda-forge
  scikit-learn                      1.7.2                   py313h06d4379_0                      conda-forge
  scipy                             1.16.1                  py313h11c21cd_1                      conda-forge
  send2trash                        1.8.3                   pyh0d859eb_1                         conda-forge
  setuptools                        80.9.0                  pyhff2d567_0                         conda-forge
  shapely                           2.1.1                   py313h393e2d1_1                      conda-forge
  simpervisor                       1.0.0                   pyhd8ed1ab_1                         conda-forge
  six                               1.17.0                  pyhe01879c_1                         conda-forge
  snappy                            1.2.2                   h03e3b7b_0                           conda-forge
  sniffio                           1.3.1                   pyhd8ed1ab_1                         conda-forge
  sortedcontainers                  2.4.0                   pyhd8ed1ab_1                         conda-forge
  soupsieve                         2.8                     pyhd8ed1ab_0                         conda-forge
  spatialdata                       0.5.0                   pypi_0                               pypi       
  sqlite                            3.50.4                  hbc0de68_0                           conda-forge
  squidpy                           1.6.5                   pypi_0                               pypi       
  stack_data                        0.6.3                   pyhd8ed1ab_1                         conda-forge
  streamz                           0.6.4                   pyhd8ed1ab_1                         conda-forge
  svt-av1                           3.1.2                   hecca717_0                           conda-forge
  tblib                             3.1.0                   pyhd8ed1ab_0                         conda-forge
  terminado                         0.18.1                  pyh0d859eb_0                         conda-forge
  threadpoolctl                     3.6.0                   pyhecae5ae_0                         conda-forge
  tifffile                          2025.9.9                pyhd8ed1ab_0                         conda-forge
  tinycss2                          1.4.0                   pyhd8ed1ab_0                         conda-forge
  tk                                8.6.13                  noxft_hd72426e_102                   conda-forge
  tomli                             2.2.1                   pyhe01879c_2                         conda-forge
  toolz                             1.0.0                   pyhd8ed1ab_1                         conda-forge
  tornado                           6.5.2                   py313h07c4f96_1                      conda-forge
  tqdm                              4.67.1                  pyhd8ed1ab_1                         conda-forge
  traitlets                         5.14.3                  pyhd8ed1ab_1                         conda-forge
  treelite                          4.4.1                   py313ha8a5e75_1                      conda-forge
  types-python-dateutil             2.9.0.20250822          pyhd8ed1ab_0                         conda-forge
  typing-extensions                 4.15.0                  h396c80c_0                           conda-forge
  typing_extensions                 4.15.0                  pyhcf101f3_0                         conda-forge
  typing_utils                      0.1.0                   pyhd8ed1ab_1                         conda-forge
  tzdata                            2025b                   h78e105d_0                           conda-forge
  uc-micro-py                       1.0.3                   pyhd8ed1ab_1                         conda-forge
  ucx                               1.18.1                  h990bcc0_2                           conda-forge
  ucx-py                            0.45.00                 py313_250806_d53bdda0                rapidsai   
  ucxx                              0.45.01                 cuda12_py313_250806_c25d2cdb         rapidsai   
  uri-template                      1.3.0                   pyhd8ed1ab_1                         conda-forge
  uriparser                         0.9.8                   hac33072_0                           conda-forge
  urllib3                           2.5.0                   pyhd8ed1ab_0                         conda-forge
  wcwidth                           0.2.13                  pyhd8ed1ab_1                         conda-forge
  webcolors                         24.11.1                 pyhd8ed1ab_0                         conda-forge
  webencodings                      0.5.1                   pyhd8ed1ab_3                         conda-forge
  websocket-client                  1.8.0                   pyhd8ed1ab_1                         conda-forge
  wheel                             0.45.1                  pyhd8ed1ab_1                         conda-forge
  xarray                            2025.9.0                pyhd8ed1ab_0                         conda-forge
  xerces-c                          3.2.5                   h988505b_2                           conda-forge
  xgboost                           3.0.3                   rapidsai_pyh452a892_0                rapidsai   
  xorg-libxau                       1.0.12                  hb9d3cd8_0                           conda-forge
  xorg-libxdmcp                     1.1.5                   hb9d3cd8_0                           conda-forge
  xyzservices                       2025.4.0                pyhd8ed1ab_0                         conda-forge
  yaml                              0.2.5                   h280c20c_3                           conda-forge
  yarl                              1.20.1                  py313h8060acc_0                      conda-forge
  zeromq                            4.3.5                   h387f397_9                           conda-forge
  zfp                               1.0.1                   h909a3a2_3                           conda-forge
  zict                              3.0.0                   pyhd8ed1ab_1                         conda-forge
  zipp                              3.23.0                  pyhd8ed1ab_0                         conda-forge
  zlib                              1.3.1                   hb9d3cd8_2                           conda-forge
  zlib-ng                           2.2.5                   hde8ca8f_0                           conda-forge
  zstandard                         0.24.0                  py313h736c1ce_1                      conda-forge
  zstd                              1.5.7                   hb8e6e7a_2                           conda-forge

[flying-sheep]

Yeah, I think you need to ask the numba people or research the docs if you want to know about how parallelization influences results.

But one thing about “commenting out njit”: If you use Python types, you’re using

1. arbitrarily-sized integers, and
2. 64 bit floats.

while numba will only use fixed-size integers and can use differently-sized floats.

[selmanozleyen]

That's good to know.

I will try out with different flags of fastmath to see if I can pinpoint this issue ( https://llvm.org/docs/LangRef.html#fast-math-flags https://numba.pydata.org/numba-doc/dev/user/performance-tips.html ).

Also will try to find a code that I can refer to as the "correct" results. These differences would be fine if the scores ranks were preserved across genes because the results are basically the number of iteration it takes for the simulation to converge * 0.001

[flying-sheep]

Be careful: The umap numba bug exists because apparently things can be “not fastmath-safe”. So make sure you read about what fastmath is and when you can use it, because apparently that’s not everywhere!

[selmanozleyen]

Ok I am doing some testings and with an epsilon value the results seem more stable.

Here is the branch: https://github.com/selmanozleyen/squidpy/tree/sepal_stability

first commit fastmath or not

selmanozleyen@43f3ade

Ranking data:
         default_rank  no_fastmath_rank
gene                                   
Atp6v1h           9.0              10.0
Lypla1            4.0               5.0
Mrpl15            6.0               7.0
Npbwr1           10.0               1.0
Oprk1             1.0               2.0
Rb1cc1            8.0               9.0
Rgs20             5.0               6.0
Sox17             3.0               4.0
Tcea1             7.0               8.0
Xkr4              2.0               3.0

Rank correlation (Spearman):
                  default_rank  no_fastmath_rank
default_rank          1.000000          0.454545
no_fastmath_rank      0.454545          1.000000
Saved rank correlation heatmap to sepal_rank_correlation.png
Saved score distribution plot to sepal_score_distribution.png
Saved detailed results to sepal_comparison_results.csv

with epsilon fastmath or not

commit
selmanozleyen@bc0e76e

Results are same one to one when epsilon is incorperated.

Ranking data:
         default_rank  no_fastmath_rank
gene                                   
Atp6v1h          10.0              10.0
Lypla1            5.0               5.0
Mrpl15            7.0               7.0
Npbwr1            1.0               1.0
Oprk1             2.0               2.0
Rb1cc1            8.0               8.0
Rgs20             6.0               6.0
Sox17             4.0               4.0
Tcea1             9.0               9.0
Xkr4              3.0               3.0

Rank correlation (Spearman):
                  default_rank  no_fastmath_rank
default_rank               1.0               1.0
no_fastmath_rank           1.0               1.0


gene,default,no_fastmath,default_rank,no_fastmath_rank
Atp6v1h,0.485,0.485,10.0,10.0
Lypla1,0.845,0.845,5.0,5.0
Mrpl15,0.708,0.708,7.0,7.0
Npbwr1,2.606,2.606,1.0,1.0
Oprk1,1.55,1.55,2.0,2.0
Rb1cc1,0.607,0.607,8.0,8.0
Rgs20,0.803,0.803,6.0,6.0
Sox17,1.379,1.379,4.0,4.0
Tcea1,0.606,0.606,9.0,9.0
Xkr4,1.438,1.438,3.0,3.0

[selmanozleyen]

@flying-sheep
Based on these results I think instead of looking for the problem in numba we should do a change like this as I did in the commit I wrote

def _entropy_impl(
    xx: NDArrayA,
) -> float:
    xx_64 = xx.astype(np.float64)
    xnz = xx_64[xx_64 > 0]

    # Handle empty array
    if len(xnz) == 0:
        return 0.0

    # Use float64 for all calculations
    eps = np.finfo(np.float64).eps  # ~2.22e-16
    xs = np.sum(xnz, dtype=np.float64)

    # Avoid division by zero
    if xs < eps:
        return 0.0
    xn = xnz / xs
    # Safe logarithm (avoid log(0))
    xl = np.log(np.maximum(xn, eps))
    entropy = -np.sum(xn * xl, dtype=np.float64)

    return float(entropy)

instead of this

# taken from https://github.com/almaan/sepal/blob/master/sepal/models.py
@njit(fastmath=True)
def _entropy(
    xx: NDArrayA,
) -> float:
    """Get entropy of an array."""
    xnz = xx[xx > 0]
    xs: np.float64 = np.sum(xnz)
    xn = xnz / xs
    xl = np.log(xn)
    return float((-xl * xn).sum())

There are also some things to clean up with sepal since some parameters are never used. It could use some simplification also.

[selmanozleyen]

@flying-sheep @timtreis I created a PR for using a fix like this #1045. It also simplifies some redundant variables