Case study: using array-api-tests in CI and locally

[lucascolley]

I just set up array-api-tests for https://github.com/mdhaber/marray such that the same command can be used in CI and locally to run the tests (and without making array-api-tests a git submodule). It comes down to the following pixi feature:

[tool.pixi.feature.xp-tests.dependencies]
pytest = "*"
pytest-json-report = "*"
hypothesis = "*"
ndindex = "*"
array-api-strict = "*"

[tool.pixi.feature.xp-tests.tasks]
# clean array-api-tests dir
clean-xp-tests = { cwd = ".", cmd = "rm -rf array-api-tests" }
# clone array-api-tests
clone-xp-tests.cmd = "git clone https://github.com/data-apis/array-api-tests.git"
clone-xp-tests.cwd = "."
clone-xp-tests.depends-on = ["clean-xp-tests"]
# checkout array-api-tests commit
checkout-xp-tests.cmd = [
  "git",
  "reset",
  "--hard",
  "606cc4d11fbcb9abdf425ea6bfe21405228d1d89",
  "&&",
  "git",
  "submodule",
  "update",
  "--init",
]
checkout-xp-tests.cwd = "array-api-tests"
checkout-xp-tests.depends-on = ["clone-xp-tests"]
# apply patch to test marray
patch-xp-tests.cmd = "git apply ../tools/xp-tests.patch"
patch-xp-tests.cwd = "array-api-tests"
patch-xp-tests.depends-on = ["checkout-xp-tests"]
# run tests
xp-tests.cmd = [
  "pytest",
  "-v",
  "-rxXfE",
  "-W",
  # https://github.com/data-apis/array-api-tests/issues/284
  "ignore::UserWarning",
  # https://github.com/data-apis/array-api-tests/issues/329
  "--disable-extension",
  "fft",
  "--disable-extension",
  "linalg",
  "--xfails-file",
  "../tools/xp-tests-xfails.txt",
  "--max-examples=100",
  "--derandomize",
  "--disable-deadline",
  "array_api_tests/",
]
xp-tests.cwd = "array-api-tests"
xp-tests.depends-on = ["patch-xp-tests"]

patch file

diff --git a/array_api_tests/__init__.py b/array_api_tests/__init__.py
index 4e0c340..525782e 100644
--- a/array_api_tests/__init__.py
+++ b/array_api_tests/__init__.py
@@ -12,25 +12,30 @@ __all__ = ["xp", "api_version", "xps"]
 
 # You can comment the following out and instead import the specific array module
 # you want to test, e.g. `import array_api_strict as xp`.
-if "ARRAY_API_TESTS_MODULE" in os.environ:
-    xp_name = os.environ["ARRAY_API_TESTS_MODULE"]
-    _module, _sub = xp_name, None
-    if "." in xp_name:
-        _module, _sub = xp_name.split(".", 1)
-    xp = import_module(_module)
-    if _sub:
-        try:
-            xp = getattr(xp, _sub)
-        except AttributeError:
-            # _sub may be a submodule that needs to be imported. WE can't
-            # do this in every case because some array modules are not
-            # submodules that can be imported (like mxnet.nd).
-            xp = import_module(xp_name)
-else:
-    raise RuntimeError(
-        "No array module specified - either edit __init__.py or set the "
-        "ARRAY_API_TESTS_MODULE environment variable."
-    )
+# if "ARRAY_API_TESTS_MODULE" in os.environ:
+#     xp_name = os.environ["ARRAY_API_TESTS_MODULE"]
+#     _module, _sub = xp_name, None
+#     if "." in xp_name:
+#         _module, _sub = xp_name.split(".", 1)
+#     xp = import_module(_module)
+#     if _sub:
+#         try:
+#             xp = getattr(xp, _sub)
+#         except AttributeError:
+#             # _sub may be a submodule that needs to be imported. WE can't
+#             # do this in every case because some array modules are not
+#             # submodules that can be imported (like mxnet.nd).
+#             xp = import_module(xp_name)
+# else:
+#     raise RuntimeError(
+#         "No array module specified - either edit __init__.py or set the "
+#         "ARRAY_API_TESTS_MODULE environment variable."
+#     )
+
+import marray
+import array_api_strict
+xp = marray.get_namespace(array_api_strict)
+xp_name = "marray(array_api_strict)"
 
 
 # If xp.bool is not available, like in some versions of NumPy and CuPy, try

I think this is pretty neat all things considered, but it is pretty complicated. It would be nice if we could provide ways to simplify this process for users, and/or to make the ideas here reusable for other projects. A few observations:

• Instructions to import array module to test incomplete #328
• Having to apply a git patch to test with a module defined at runtime is not nice maintenance-wise. Could we create something which takes a Python script defining xp as input, and returns an appropriately patched array-api-tests for a given version?
• Could we create something which takes the Python module to test, and a list of conda/pypi dependencies for that module, and runs the tests itself? What would the user interface be for this - CLI? Perhaps it could be GHA only (NumPy just runs in CI as far as I can tell), but it is nice to be able to reproduce the CI runs locally.

[rgommers]

It's not immediately clear to me why adding env = { ARRAY_API_TESTS_MODULE = 'marray' } in pixi.toml doesn't work and the mechanism for finding the module under test needs to be patched. Can you elaborate @lucascolley?

[lucascolley]

marray isn't a standard-compatible namespace, but marray(numpy) and marray(array_api_strict) are. The same sort of the consideration should apply for a hypothetical agnostic library adding units like Pint.

[rgommers]

Ah okay, that makes sense. So rather than ARRAY_API_TESTS_MODULE being a name to import, it needs semantics like "use the xp from executing this code". So as a very naive implementation, if the environment variable starts with exec then run the code. Then it can be done like:

ARRAY_API_TESTS_MODULE=exec('import numpy, marray; xp = marray.get_namespace(numpy)')

[lucascolley]

thanks Ralf, that seems like a much nicer user experience. I'll have a go at implementing this.

[adityagoel4512]

Just chiming in, but we run this test suite using pixi weekly and on each PR to find discrepancies. In case it's valuable, our pixi set up is here and the GHA workflow here.

[lucascolley]

The main actionable point is addressed here, so let's close this now