Skip to content

Conversation

Baharis
Copy link
Member

@Baharis Baharis commented Aug 20, 2025

A short PR, to be merged after #137. I had some time today so I added tests for all the simple instamatic._collections and instamatic.tools. These are mostly straightforward but it's a good practice to have them all tested. I removed one duplicate and some unused imports while I was at it. Apparently this pushes instamatic line coverage from 48% to 49%.

When testing with numpy 2.3.2 I have also found and fixed faulty use of np.frombuffer in instamatic/camera/gatansocket3.py that caused numpy 2 to fail tests that numpy 1 passed. In principle, gatansocket3.py implementation correctly tried to buff its messages to a multiple of np.int_ byte size (as np.frombuffer requires) but on modern 64-bit systems the size of np.int_ is 8, not 4, as it was hard-coded before. Numpy 1 buffed the incorrectly-sized bytes as needed, but numpy 2 does not.

def string_to_longarray(string: str, *, dtype: np.dtype = np.int_) -> np.ndarray:
    """Convert the string to a 1D np array of dtype (default np.int_ - C long)
    with numpy2-save padding to ensure length is a multiple of dtype.itemsize.
    """
    s_bytes = string.encode('utf-8')
    dtype_size = np.dtype(dtype).itemsize
    if extra := len(s_bytes) % dtype_size:
        s_bytes += b'\0' * (dtype_size - extra)
    return np.frombuffer(s_bytes, dtype=dtype)

This PR does not introduce any new features. It may fixes some of the issues mentioned by @stefsmeets in #130.

@Baharis Baharis self-assigned this Aug 20, 2025
@Baharis Baharis added the bug label Aug 20, 2025
# Conflicts:
#	src/instamatic/_collections.py
#	src/instamatic/experiments/fast_adt/experiment.py
@Baharis Baharis changed the title Add tests for instamatic collections and tools; improve numpy2 support Add tests for collections and tools; improve numpy2 support Sep 18, 2025
@Baharis Baharis marked this pull request as ready for review September 18, 2025 11:46
@Baharis Baharis requested a review from stefsmeets September 18, 2025 11:46
@Baharis
Copy link
Member Author

Baharis commented Sep 18, 2025

@stefsmeets I guess these were said "non-trivial issues with data type conversions", in which case this fixes #130. Tests work with numpy 2.3.2 for me, I will temporarily force numpy version above 2 for testing on github. In case it passes, I will revert the lower limit to numpy >= 1.17.3, without the , <2 bit.

@Baharis
Copy link
Member Author

Baharis commented Sep 18, 2025

Successfully tested using five six builds (added Python 3.13 to GitHub CI), with all tests passing in each case:

  • CPython 3.9.13 with numpy-2.0.2-cp39-cp39-win_amd64: 87 passed, 15 xfailed, 1 warning in 40.21s (GitHub CI)
  • CPython 3.10.11 with numpy-2.2.6-cp310-cp310-win_amd64: 87 passed, 15 xfailed, 1 warning in 29.59s (GitHub CI)
  • CPython 3.11.9 with numpy-2.3.3-cp311-cp311-win_amd64: 87 passed, 15 xfailed, 1 warning in 28.72s (GitHub CI)
  • CPython 3.12.10 with numpy-2.3.3-cp312-cp312-win_amd64: 87 passed, 15 xfailed, 1 warning in 36.24s (GitHub CI)
  • CPython 3.13.2 with numpy-2.3.2-cp313-cp313-win_amd64: 87 passed, 15 xfailed, 1 warning in 38.04s (Work PC)
  • CPython 3.13.7 with numpy-2.3.3-cp313-cp313-win_amd64: 87 passed, 15 xfailed, 1 warning in 30.08s (GitHub CI)

The only warning comes from "tests/test_simulation/test_grid.py::test_get_array: NotImplementedWarning: Center mark is not implemented yet", something that I guess is / we can address in #140.

Reverting the previous commit to allow python 1.17+ builds, though in the long term I would prefer to switch to numpy > 2 seeing how ubiquitously successful and available it is. Merging tomorrow unless anyone objects.

Copy link
Member

@stefsmeets stefsmeets left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

https://docs.astral.sh/ruff/rules/numpy2-deprecation/ might also catch something not caught by the tests.

@stefsmeets
Copy link
Member

Numpy 2.0 does not drop any Python versions, 3.9 being the lowest supporded version which matches Instamatic. I think we can safely use numpy>=2.

@Baharis
Copy link
Member Author

Baharis commented Sep 18, 2025

@stefsmeets In that case I would absolutely prefer to rip the band-aid. Newer numpy is more consistent and adds cool typing features. Also I have some cleaning to do:

$ python -m ruff check ./src ./tests --select NPY201
warning: Invalid `# noqa` directive on src\instamatic\gui\fast_adt_frame.py:73: expected a comma-separated list of codes (e.g., `# noqa: F401, F841`).
warning: Invalid `# noqa` directive on src\instamatic\calibrate\calibrate_stage_rotation.py:166: expected a comma-separated list of codes (e.g., `# noqa: F401, F841`).
All checks passed!

@Baharis
Copy link
Member Author

Baharis commented Sep 18, 2025

I'd maybe suggest leaving switch to numpy 2 for a few more months. In case someone must use Python 3.9, as it is still supported for a month or so, the only minor numpy 2 version available is 2.0. Planning a switch for 2026 though.

@Baharis Baharis linked an issue Sep 19, 2025 that may be closed by this pull request
@Baharis Baharis merged commit 9e2e04a into instamatic-dev:main Sep 19, 2025
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support numpy >=2
2 participants