refactor: update sasview api for test_sas.py #126

zmx27 · 2025-07-30T21:46:10Z

Addresses #100

zmx27 · 2025-07-30T21:51:36Z

@sbillinge here are my initial edits to update our code to use the new sasmodel/sasdata api. These changes introduced some warnings that I don't quite understand. Now that the tests in test_characteristicfunctions.py are no longer being skipped, I also added the new updated api there, but it introduced a new error that one of the functions that we were testing is no longer implemented. Here are the pytest results:

======================================== test session starts =========================================
platform darwin -- Python 3.13.5, pytest-8.4.1, pluggy-1.6.0
rootdir: /Users/zhimingxu/BillingeGroup/diffpy.srfit
configfile: pyproject.toml
plugins: env-1.1.5, cov-6.2.1
collected 111 items

tests/test_builder.py ......                                                                   [  5%]
tests/test_characteristicfunctions.py FFFF                                                     [  9%]
tests/test_constraint.py .                                                                     [  9%]
tests/test_contribution.py ..........                                                          [ 18%]
tests/test_diffpyparset.py sss                                                                 [ 21%]
tests/test_equation.py ..                                                                      [ 23%]
tests/test_fitrecipe.py ....                                                                   [ 27%]
tests/test_fitresults.py ...                                                                   [ 29%]
tests/test_literals.py .........                                                               [ 37%]
tests/test_objcrystparset.py sssssssssss                                                       [ 47%]
tests/test_parameter.py ...                                                                    [ 50%]
tests/test_parameterset.py .                                                                   [ 51%]
tests/test_pdf.py ..ssssss                                                                     [ 58%]
tests/test_profile.py ......                                                                   [ 63%]
tests/test_profilegenerator.py ...                                                             [ 66%]
tests/test_recipeorganizer.py ...............                                                  [ 80%]
tests/test_restraint.py .                                                                      [ 81%]
tests/test_sas.py ...                                                                          [ 83%]
tests/test_sgconstraints.py sss                                                                [ 86%]
tests/test_tagmanager.py ....                                                                  [ 90%]
tests/test_version.py .                                                                        [ 90%]
tests/test_visitors.py ....                                                                    [ 94%]
tests/test_weakrefcallable.py ......                                                           [100%]

============================================== FAILURES ==============================================
...
========================================== warnings summary ==========================================
tests/test_sas.py::test_generator
tests/test_sas.py::testGenerator2
  /Users/zhimingxu/miniconda3/envs/srfit-env/lib/python3.13/site-packages/pyopencl/cache.py:496: CompilerWarning: Non-empty compiler output encountered. Set the environment variable PYOPENCL_COMPILER_OUTPUT=1 to see more.
    _create_built_program_from_source_cached(

tests/test_sas.py::test_generator
tests/test_sas.py::testGenerator2
  /Users/zhimingxu/miniconda3/envs/srfit-env/lib/python3.13/site-packages/pyopencl/cache.py:500: CompilerWarning: Non-empty compiler output encountered. Set the environment variable PYOPENCL_COMPILER_OUTPUT=1 to see more.
    prg.build(options_bytes, devices)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
====================================== short test summary info =======================================
FAILED tests/test_characteristicfunctions.py::testSphere - NotImplementedError: ER function is no longer available.
FAILED tests/test_characteristicfunctions.py::testSpheroid - NotImplementedError: ER function is no longer available.
FAILED tests/test_characteristicfunctions.py::testShell - NotImplementedError: ER function is no longer available.
FAILED tests/test_characteristicfunctions.py::testCylinder - NotImplementedError: ER function is no longer available.
======================== 4 failed, 84 passed, 23 skipped, 4 warnings in 1.98s ========================

I'll get to this issue later.

zmx27

Please see comments

zmx27 · 2025-07-30T21:52:46Z

src/diffpy/srfit/sas/sasparser.py

@@ -22,7 +22,6 @@

 from diffpy.srfit.exceptions import ParseError
 from diffpy.srfit.fitbase.profileparser import ProfileParser
-from diffpy.srfit.sas.sasimport import sasimport


I deleted the import from this file as not used but I haven't deleted sasimport yet because the sas package in sasview still exists, and we still need to import from it.

zmx27 · 2025-07-30T21:53:40Z

src/diffpy/srfit/sas/sasparser.py

        loader = Loader()

+        # Convert Path object to string if needed


load seems to expect a string now because it's calling .lower() on filename. In the source code, the traceback leads to the lookup() function in sasdata.data_util.registry, which calls:

path_lower = path.lower()

zmx27 · 2025-07-30T21:54:12Z

src/diffpy/srfit/sas/sasparser.py

@@ -118,7 +122,16 @@ def parseFile(self, filename):
        self._meta["filename"] = filename
        self._meta["datainfo"] = data

-        self._banks.append([data.x, data.y, data.dx, data.dy])
+        # Handle case where loader returns a list of data objects
+        if isinstance(data, list):


The loader returns a list now. From source code:

def load(self, file_path_list: Union[List[Union[str, Path]], str, Path], format: Optional[Union[List[str], str]] = None ) -> List[Union[Data1D, Data2D]]:

we presumably don't need the conditional then. If it returns a list then just treat a list. We don't have to backwards compatible because we are only supporting recent versions of all dependencies.

zmx27 · 2025-07-30T22:07:22Z

tests/test_characteristicfunctions.py

    model = EllipsoidModel()
-    model.setParam("radius_a", prad)


I could not find any mention of the use of radius_a and radius_b in the documentation for sasview, even ones that dated back to version 4.x (the latest release is version 6.1.0). I can only assume that suitable replacements are radius_polar and radius_equatorial, which are the parameters that the ellipsoid model now uses.

EllipsoidModel must have a major and a minor axis, so these presumably refer to this? What are these axes called in EllipsoidModel?

zmx27 · 2025-07-30T22:10:34Z

tests/test_sas.py

    gen.background.value = 0.01

    y = gen(profile.xobs)
    diff = profile.yobs - y
    res = numpy.dot(diff, diff)
-    assert 0 == pytest.approx(res)
+    assert 0 == pytest.approx(res, abs=1e-3)


Here are the test results: assert 0 == 0.00011723342...0113
The value of res is not approximately 0 with the default tolerance that pytest.approx() provides. To make the test pass, I increased the tolerance level. This is probably because the radius_polar and radius_equatorial were incorrectly set earlier.

Please can you try and get to the bottom of why the test is failing with the default tolerance?

I will look into this

zmx27 · 2025-07-31T19:32:05Z

I looked into the source code for sas, and it seems like the calculate_ER function is indeed deprecated. Here is the function definition from sas.sascalc.calculator.BaseComponent:

def calculate_ER(self):
        """
        Calculate effective radius
        """
        return NotImplemented

Should I look into ways to somehow calculate the effective radius manually? How else should I modify the source code that the tests are testing? Or should I just deal with this in another PR?

sbillinge · 2025-08-01T00:53:13Z

I looked into the source code for sas, and it seems like the calculate_ER function is indeed deprecated. Here is the function definition from sas.sascalc.calculator.BaseComponent:
def calculate_ER(self):
        """
        Calculate effective radius
        """
        return NotImplemented
Should I look into ways to somehow calculate the effective radius manually? How else should I modify the source code that the tests are testing? Or should I just deal with this in another PR?

this appears not to be deprecated, but it is not implemented. I will have to look at the test

sbillinge

good work. Progress is being made. Please can you see my comments?

sbillinge · 2025-08-01T00:54:50Z

news/fix-sasmodels.rst

+
+**Changed:**
+
+* Refactored code utilizing sasmodels to use the new sasview api.


please move to fixed. This is a bug fix not a change. Reserve changes for changes in behavior that a user might need to know about.

sbillinge · 2025-08-01T00:56:50Z

src/diffpy/srfit/sas/sasparser.py


-        Loader = sasimport("sas.dataloader.loader").Loader
+        Loader = ld.Loader


please could you make ld more readable. maybe sas_dataloader?

sbillinge · 2025-08-01T00:58:02Z

src/diffpy/srfit/sas/sasparser.py

        loader = Loader()

+        # Convert Path object to string if needed
+        if not isinstance(filename, str):


why not just data = loader.load(str(filename)). I don't think we have to wrap this in a conditional.

sbillinge · 2025-08-01T00:59:25Z

src/diffpy/srfit/sas/sasparser.py

@@ -118,7 +122,16 @@ def parseFile(self, filename):
        self._meta["filename"] = filename
        self._meta["datainfo"] = data

-        self._banks.append([data.x, data.y, data.dx, data.dy])
+        # Handle case where loader returns a list of data objects


remove comment. Just make you code as readable as possible.

sbillinge · 2025-08-01T01:00:24Z

src/diffpy/srfit/sas/sasparser.py

-        self._banks.append([data.x, data.y, data.dx, data.dy])
+        # Handle case where loader returns a list of data objects
+        if isinstance(data, list):
+            # If it's a list, iterate through each data object


remove gratuitous comments

sbillinge · 2025-08-01T01:05:08Z

tests/test_characteristicfunctions.py

@@ -19,8 +19,10 @@
 import numpy
 import pytest

+# Use the updated SasView model API to load models


please remove comments

sbillinge · 2025-08-01T01:06:03Z

tests/test_characteristicfunctions.py

@@ -34,7 +36,7 @@ def testSphere(sas_available):
        pytest.skip("sas package not available")
    radius = 25
    # Calculate sphere cf from SphereModel
-    SphereModel = sasimport("sas.models.SphereModel").SphereModel
+    SphereModel = _make_standard_model("sphere")


it seems a bit odd that we are importing a private function. Are we sure this is the way we are supposed to be using the API?

I'll keep looking into this

sbillinge · 2025-08-01T01:09:05Z

tests/test_characteristicfunctions.py

    model = EllipsoidModel()
-    model.setParam("radius_a", prad)


EllipsoidModel must have a major and a minor axis, so these presumably refer to this? What are these axes called in EllipsoidModel?

sbillinge · 2025-08-01T01:09:23Z

tests/test_characteristicfunctions.py

    model = EllipsoidModel()
-    model.setParam("radius_a", prad)
-    model.setParam("radius_b", erad)
+    model.setParam("radius_polar", prad)


ah, I think this answers my question.....

sbillinge · 2025-08-01T01:13:00Z

tests/test_sas.py

    gen.background.value = 0.01

    y = gen(profile.xobs)
    diff = profile.yobs - y
    res = numpy.dot(diff, diff)
-    assert 0 == pytest.approx(res)
+    assert 0 == pytest.approx(res, abs=1e-3)


Please can you try and get to the bottom of why the test is failing with the default tolerance?

zmx27 · 2025-08-01T17:41:19Z

@sbillinge I'm sorry, I couldn't figure out why the tests aren't passing with the default tolerance for test_generator2(). I initially thought it may be because the list item that the load() function returns could have multiple elements, but a IndexError occurs when I changed it to return index 1 of the list, telling me that there is only one item in the list. And how should I proceed regarding the calculate_ER() function? Should I attempt to try to make a function that calculates the effectively radius? Could you please look into these when you have time and give me some guidance? Or should some of this work be done in another PR? Thanks!

sbillinge · 2025-08-02T19:29:48Z

@zmx27 I had a look at the sas models test and it is hard to figure out why it is off so much. I suggest that we

make an issue to look at this again in a future release. It would require some forensics like plotting the data in .txt file and plotting the generated curve on top of each other to see what is wrong.
leave the test as is in your dedit since it is now passing but add a comment line `# FIXME: go back to default tolerance when we figure out why the models are not identical"

There are a few other issues with the sasview integration (it is not working beyond python 3.11) and I am not sure who is using it, if anyone. So revisiting later seems like the best bet,

I will take a look at the characteristic function test that are failing now.....

sbillinge · 2025-08-02T19:48:56Z

Should I look into ways to somehow calculate the effective radius manually? How else should I modify the source code that the tests are testing? Or should I just deal with this in another PR?

I don't see where this is being called. It seems that test_sphere and test_spheroid and so on are failing becaue they call something that calls something insde sasview. Sot I feel that it is something inside sasview that is failing. Since spherical and spheroidal models are really common, I imagine that we are trying to call them in a way that has been deprecated. Please could you try and read the sasview/sasmodels documentation and see if there are examples of how to call these models in the latest versions? Then we can try and figure out if we are calling them in the right way. If we find we are not and there is a new way to invoke them or they have a new name, for example, then this could be a quick fix. If we are calling them correctly, this will take a bit longer.

Let's give it one more go before we punt it to a later release, but we can do that if we have to

sbillinge

@zmx27 please see my inline comment.

sbillinge · 2025-08-02T21:23:50Z

tests/test_characteristicfunctions.py

@@ -34,7 +36,7 @@ def testSphere(sas_available):
        pytest.skip("sas package not available")
    radius = 25
    # Calculate sphere cf from SphereModel
-    SphereModel = sasimport("sas.models.SphereModel").SphereModel
+    SphereModel = find_model("sphere")


this line might be the problem here. I think find_model() returns an instance of the model and not the model class. Does it work if you simply try

model = find_model("sphere")

and go from there?

zmx27 added 2 commits July 30, 2025 17:45

refactor: update sasview api for test_sas.py

96e1ff3

chore: add news file

6112cdd

zmx27 commented Jul 30, 2025

View reviewed changes

sbillinge reviewed Aug 1, 2025

View reviewed changes

chore: remove comments and use public function to load models

c68f981

sbillinge reviewed Aug 2, 2025

View reviewed changes


		Changed:

		* Refactored code utilizing sasmodels to use the new sasview api.


		Loader = sasimport("sas.dataloader.loader").Loader
		Loader = ld.Loader

refactor: update sasview api for test_sas.py #126

Are you sure you want to change the base?

refactor: update sasview api for test_sas.py #126

Conversation

zmx27 commented Jul 30, 2025

Uh oh!

zmx27 commented Jul 30, 2025

Uh oh!

zmx27 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zmx27 commented Jul 31, 2025

Uh oh!

sbillinge commented Aug 1, 2025

Uh oh!

sbillinge left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zmx27 commented Aug 1, 2025

Uh oh!

sbillinge commented Aug 2, 2025

Uh oh!

sbillinge commented Aug 2, 2025

Uh oh!

sbillinge left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!