TREXIO + MCSCF wavefunction #92

NastaMauger · 2024-12-20T20:58:26Z

Hello,

In this PR, you will find modifications that allow users to register an MCSCF function in the TREXIO format.

I am uncertain if additional data must be stored for TREXIO to function correctly with other software and to return the corresponding wave function produced by PySCF before registering it into TREXIO. However, the corresponding HDF5 file generated using the following code:

mcscf = mcscf.casci_symm.CASCI(rhf, norb, nelec)
mcscf.fcisolver = selected_ci_spin0_symm.SelectedCI(rhf)

mcscf.kernel()
trexio_file = 'data_mcscf.hdf5'
pyscf_trexio.to_trexio(mcscf, trexio_file)

pyscf_trexio.det_to_trexio(mcscf, norb, nelec, trexio_file)
eri = mol.intor('int2e')
pyscf_trexio.write_eri(eri, trexio_file)

seems to register the data correctly.

Please let me know if further modifications or improvements are necessary. I am also uncertain whether the one-electron MO group is essential, as mentione here. I would also appreciate any feedback on whether the write_eri function needs to be modified to handle CASCI/CASSCF, particularly for the corresponding active space

Best

q-posev · 2025-01-02T11:07:10Z

Thank you @NastaMauger !

I noticed that you do not register the CI determinants as part of your new mcscf_to_trexio. Is it intended?

q-posev · 2025-01-02T12:31:27Z

I guess the best test of the validity of the produced multi-configurational TREXIO wavefunction is via an attempt to import and use it in some software that handles such wave functions (e.g. QP2)? @NastaMauger @scemama

q-posev · 2025-01-02T20:49:41Z

pyscf/tools/trexio.py

@@ -434,10 +464,6 @@ def det_to_trexio(mcscf, norb, nelec, filename, backend='h5', ci_threshold=0., c
    with trexio.File(filename, 'u', back_end=_mode(backend)) as tf:
        if trexio.has_determinant(tf):
            trexio.delete_determinant(tf)
-        trexio.write_mo_num(tf, mo_num)
-        trexio.write_electron_up_num(tf, len(a))


Why is this removed? We need the number of up and down electrons to perform the CI determinants "correctness" check before writing them in the file.

This is now handled by the _mcscf_to_trexio function, which calls _mol_to_trexio.

Reshape write_eri function to ensusre conssitency between other codes mcscf_function directly register the determinant list now

NastaMauger · 2025-02-17T02:31:01Z

@q-posev

I noticed that you do not register the CI determinants as part of your new mcscf_to_trexio. Is it intended?
Done
I guess the best test of the validity of the produced multi-configurational TREXIO wavefunction is via an attempt to import and use it in some software that handles such wave functions (e.g. QP2)? @NastaMauger @scemama
There is still an issue due to the normalization factor, which needs to be resolved.
The write-eri function might still lack consistency between what the original function in pyscf-forge returns and what QP2 (or other software) might expect.

q-posev · 2025-02-17T09:29:34Z

@NastaMauger

For the ERIs - I agree, @kgasperich submitted a draft PR which takes care of the AO/MO inconsistency.

NastaMauger · 2025-03-01T04:22:07Z

Hello,

I am following this issue as it's a feature I really want to add.

After discussing with @q-posev and @scemama, we observed that although we correctly registered all necessary data into TREXIO, there is still a difference between the CASSCF energy from PySCF and the exported TREXIO file. Upon examining the convergence of the energy with respect to the CAS space, it seems there may be a hidden selected CI procedure within the CASSCF routine.

@sunqm Could you please confirm and provide guidance on how to correctly register the CASCI wavefunction if some selected CI is implicitly performed?

Best

MatthewRHermes · 2025-03-02T19:02:51Z

Hello,

I am following this issue as it's a feature I really want to add.

After discussing with @q-posev and @scemama, we observed that although we correctly registered all necessary data into TREXIO, there is still a difference between the CASSCF energy from PySCF and the exported TREXIO file. Upon examining the convergence of the energy with respect to the CAS space, it seems there may be a hidden selected CI procedure within the CASSCF routine.

@sunqm Could you please confirm and provide guidance on how to correctly register the CASCI wavefunction if some selected CI is implicitly performed?

Best

There is no selected CI procedure in CASSCF that I am aware of. There is a procedure to approximate the CI step during the orbital optimization, but that should have no effect on the evaluation of the CASCI energy, which is entirely determined by the MO coefficients, one- and two-electron integrals, and CI vectors. Could you elaborate on what you are seeing?

sunqm · 2025-03-08T17:25:24Z

Hello,

I am following this issue as it's a feature I really want to add.

After discussing with @q-posev and @scemama, we observed that although we correctly registered all necessary data into TREXIO, there is still a difference between the CASSCF energy from PySCF and the exported TREXIO file. Upon examining the convergence of the energy with respect to the CAS space, it seems there may be a hidden selected CI procedure within the CASSCF routine.

@sunqm Could you please confirm and provide guidance on how to correctly register the CASCI wavefunction if some selected CI is implicitly performed?

Best

Are you able to reproduce the CASSCF energy for small systems like H2 molecule with (2,2) CAS?

NastaMauger · 2025-07-03T13:56:06Z

I think the differences I got stems for different version I was using. Now it should be ok

scemama

I am not an expert of PySCF, but on the TREXIO side everything looks OK.

q-posev

@NastaMauger thank you!
However, test_trexio is failing in the CI. Have you tried to run pytest locally on your branch? Maybe it's because of the modifications in the mol_from_trexio?
In general, did you manage to reproduce the CI energy between QP2 and PySCF via TREXIO using the code in this branch? That could be a good test to add to test_trexio.py here if you or @scemama have a small CIPSI wavefunction in the TREXIO format.

q-posev · 2025-07-04T19:10:49Z

pyscf/tools/trexio.py

@@ -329,6 +365,12 @@ def write_eri(eri, filename, backend='h5'):
        idx[:,:,2:] = idx_pair[None,:,:]
        idx = idx[np.tril_indices(npair)]

+    idx=idx.reshape((num_integrals,4))


Why do you need to reshape the integrals here?

This is due to the difference between chemist and physicist notation.
QP2 uses the physicist's notation, whereas PySCF uses the chemist's.

There is a PR where Kevin is modifying this.

In fact, I think there should be a possibility with TREXIO to register data in one format or another, and to have an HDF5 field indicating which format the data is stored in.

How did you test this part?
I am asking because

I see that you swap the indices to go from one notation into another, but you do not swap the eri array accordingly (which remains in the original chemist notation of PySCF).

I see that read_eri part is untouched. So when reading the ERI's in physicist notation, one may encounter issues as PySCF may expect chemist notation. So the the index swap might be needed in read_eri too.

I guess one of these points leads to tests failing in the CI. But maybe I missed something?

@q-posev This is something I worked on with Anthony during a Zoom call a while ago because we noticed some weird behavior. I'm not sure if it's still necessary, so I left it here as a comment just in case.

Regarding your point 2: exactly. That's why I think something should be done in the TREXIO format to ensure that the formats are properly registered and can be used seamlessly across different software.

I never really had the opportunity to finish this, and I know Kevin is working on something similar on the QP2 side.

I added a small comment in case someone wants to follow this. If it turns out to be useless, I guess it can easily be removed by either you or Anthony in the next PR

Even if we enforce one format or another in TREXIO, the user can still confuse them and accidentally export the wrong info. But we have trexio-tools package which can be used to validate some TREXIO fields and people can contribute more robust tests to the repo (e.g. computing the HF energy from the one- and two-electron integrals)

In fact, I think there should be a possibility with TREXIO to register data in one format or another, and to have an HDF5 field indicating which format the data is stored in.

The whole point of TREXIO is to have a single and consistent way of doing things so the data is easy to interpret. We chose physicist's notation because chemist's notation can't generalize to more than 2-electron integrals. Reordering indices from 1,2,3,4 to 1,3,2,4 is not really a big issue...

@q-posev This is something I worked on with Anthony during a Zoom call a while ago because we noticed some weird behavior. I'm not sure if it's still necessary, so I left it here as a comment just in case.

In this zoom call we realized we had to reorder the indices because we checked the enrgy with quantum package. So we fixed the ordering for the export. I don't recall we checked importing integrals into pyscf. The reordering should also be done in the reading part.

q-posev · 2025-07-04T19:20:31Z

pyscf/tools/trexio.py

    from trexio_tools.group_tools import determinant as trexio_det

    mo_num = norb
+    ncore = mcscf.ncore
    int64_num = int((mo_num - 1) / 64) + 1


We can do int64_num = trexio.get_int64_num(trexio_file) here, the way @scemama did in PR #152

If the user registers the determinant in a new/different HDF5 file, this will not work.

It is possible that in previous versions of trexio you could bypass the need of storing the number of MOs in the trexio file, but it was because some internal consistency checking was lacking in TREXIO. In the current version, you can't write determinants if you don't write the number of MOs in the MO group to ensure that the determinants you store are consistent. So it should be totally equivalent.

I would rather use the library function call than compute it, because it will always return the value expected by the library. Here it is possible that user makes a mistake, and that norb is not the same as the value stored in the mo group.

So I guess, I need to update TREXIO on my own laptop because int64_num = trexio.get_int64_num(trexio_file) does not work on my end

NastaMauger · 2025-07-04T20:15:41Z

I have no clue why this CI failed. pytest succeeds on my laptop.

q-posev

@NastaMauger thanks! Indeed, the CI was failing before because of the broken ERI import. Now it's fixed and the CI failed only for Python 3.12 for the reasons unknown to me and unrelated to TREXIO, which is good news :-)

So if you ask me, I think this PR can be merged, even though the ERIs are exported in the chemist notation and not in the physicist one at the moment. I believe this issue can be addressed in a separate PR.

What do you think @sunqm @MatthewRHermes @scemama ?

NastaMauger · 2025-07-04T21:35:39Z

Ok @scemama, this is exactly what I was thinking. The correct ERIs are missing, and without them, it's useless to have a QP2 (or any other) test export.

I'm a bit confused about this part: "set the importing of ERIs as NotImplementedError".
Are you asking to create a function called import_eri that simply raises an error?

scemama · 2025-07-04T21:52:22Z

pyscf/tools/trexio.py

+#    x = idx[:,0]*(idx[:,0]+1)//2 + idx[:,1]
+#    y = idx[:,2]*(idx[:,2]+1)//2 + idx[:,3]


OK, NotImplementedError breaks the tests. So it was not a good idea.
You can reactivate the read_eri as it was before, replacing those two lines with

x = idx[:,0]*(idx[:,0]+1)//2 + idx[:,2] y = idx[:,1]*(idx[:,1]+1)//2 + idx[:,3]

pyscf/tools/trexio.py

NastaMauger · 2025-07-05T19:35:17Z

@scemama, , are the modifications you requested included in my PR? For some reason, I can’t see your comments. :(

q-posev · 2025-07-05T19:59:24Z

TREXIO CI is green now! I think this PR is ready to be merged

q-posev · 2025-07-13T11:25:17Z

@sunqm We believe that this PR is finally ready to be merged. Would it be possible to get another round of review from one of the pyscf-forge maintainers?

NastaMauger · 2025-07-17T23:17:45Z

@sunqm I have no clue why version 3.12 is having issues with the check, since the error arises in a part I haven't modified at all.

kousuke-nakano · 2025-07-27T05:40:28Z

@NastaMauger Hi. I would appreciate it if you could let me know the status of this pull request. I am working on a related topic. #153 Thank you!

NastaMauger · 2025-07-27T05:44:18Z

@kousuke-nakano In fact, this PR is ready to be merged. It is currently blocked for an unknown reason
@sunqm ?

kousuke-nakano · 2025-07-27T05:53:10Z

pyscf/tools/trexio.py

@@ -410,17 +452,18 @@ def get_occsa_and_occsb(mcscf, norb, nelec, ci_threshold=0.):

    return occsa_sorted, occsb_sorted, ci_values_sorted, num_determinants

-def det_to_trexio(mcscf, norb, nelec, filename, backend='h5', ci_threshold=0., chunk_size=100000):
+def det_to_trexio(mcscf, norb, nelec, trexio_file, ci_threshold=0., chunk_size=100000):
    from trexio_tools.group_tools import determinant as trexio_det


Hi @NastaMauger, in my opinion, we should not import trexio_tools; otherwise, pyscf has an additional dependency on trexio_tools. In other words, when we make a pull request to the main repository in the future, we should ask the committee member of pyscfto add not only trexio but also trexio_tools in the setup.cfg, which is not a good idea. We should hardcode it. What do you think?

I want to know your thoughts, @q-posev and @scemama.

Would it be possible to accept this PR as it is and open a new one with your idea? I agree that your suggestion would help reduce dependencies, but since this has already been validated by everyone and the only thing preventing the final merge is unrelated to the current modification, I would really appreciate if it could be merged

To be honest, I’m not sure how much work this would require, especially since this PR has been open for a while simply because I had forgotten about it

I want to know your thoughts, @q-posev and @scemama.

I agree. We should not import trexio tools.
Also, I recently removed the déterminant module from trexio tools because it was redundant with native functions of trexio.

kousuke-nakano · 2025-07-27T06:06:56Z

@NastaMauger Yes, I agree with you, and I will create another pull request regarding the dependency issue.

scemama · 2025-07-27T08:43:51Z

You can fix the dependency like this:
TREX-CoE/trexio_tools@739317f
It is in a pending PR of trexio_tools

kousuke-nakano · 2025-07-27T23:43:02Z

You can fix the dependency like this: TREX-CoE/trexio_tools@739317f It is in a pending PR of trexio_tools

Thank you @scemama! Once this pull request is merged, I will create another one to address this issue.

scemama · 2025-07-31T08:28:45Z

@NastaMauger I made a PR on your fork to fix the dependency issue with trexio_tools. So no other PR to pyscf-forge will be necessary.

sunqm approved these changes Dec 22, 2024

View reviewed changes

NastaMauger mentioned this pull request Dec 30, 2024

Merge on PySCF-Force github TREX-CoE/trexio_tools#54

Closed

q-posev reviewed Jan 2, 2025

View reviewed changes

MCSCF object can now be directly registered using TREXIO capabilities

dc61e9a

NastaMauger force-pushed the master branch from 4bbb732 to dc61e9a Compare January 30, 2025 10:28

Update occsa and occsb to also register core orbitals

c639fd4

Reshape write_eri function to ensusre conssitency between other codes mcscf_function directly register the determinant list now

NastaMauger mentioned this pull request Mar 1, 2025

[WIP] clearer trexio eri interface #105

Draft

NastaMauger requested review from sunqm and q-posev July 3, 2025 13:56

scemama approved these changes Jul 3, 2025

View reviewed changes

NastaMauger added 6 commits July 3, 2025 11:04

Missing term in det_to_trexio

00f97c3

Correct formatting

c4ac993

Correct formatting-Again

3fe27fd

Correct formatting-Again

58a1332

Correct formatting-Trailing spaces

21ce806

Correct formatting-Trailing spaces

243da68

q-posev reviewed Jul 4, 2025

View reviewed changes

NastaMauger added 2 commits July 4, 2025 15:38

Put ERI in physicist format as a comment.

38b91b8

Add comment

c48fb69

q-posev approved these changes Jul 4, 2025

View reviewed changes

NastaMauger added 2 commits July 4, 2025 17:38

Physicist format on

83c1a0d

read_eri raise error

04550af

NastaMauger requested a review from scemama July 4, 2025 21:43

scemama suggested changes Jul 4, 2025

View reviewed changes

NastaMauger added 2 commits July 4, 2025 17:54

read_eri in physicist notation

19bec15

Remove useless mo_num in det_to_trexio

177e46c

q-posev reviewed Jul 5, 2025

View reviewed changes

pyscf/tools/trexio.py Outdated Show resolved Hide resolved

Correct read_eri

1f74dd3

NastaMauger requested a review from scemama July 5, 2025 19:30

scemama approved these changes Jul 5, 2025

View reviewed changes

NastaMauger mentioned this pull request Jul 8, 2025

Problem export CASSCF wavefunctions produce by PySCF QMCPACK/qmcpack#5563

Open

sunqm added this to the v1.0.4 milestone Jul 13, 2025

sunqm approved these changes Jul 17, 2025

View reviewed changes

sunqm added 2 commits July 20, 2025 12:53

Adjust unittest due to Scipy 1.16 compatibility issue

aa6b613

scipy for python3.8

7e798f8

kousuke-nakano reviewed Jul 27, 2025

View reviewed changes

q-posev mentioned this pull request Jul 30, 2025

TODO list for TREX-IO: to make a pull request to the main repository in the fugure. #153

Open

NastaMauger requested a review from sunqm July 31, 2025 13:32

sunqm merged commit b4da987 into pyscf:master Aug 17, 2025
5 of 6 checks passed

		# x = idx[:,0]*(idx[:,0]+1)//2 + idx[:,1]
		# y = idx[:,2]*(idx[:,2]+1)//2 + idx[:,3]

TREXIO + MCSCF wavefunction #92

TREXIO + MCSCF wavefunction #92

Uh oh!

Conversation

NastaMauger commented Dec 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

q-posev commented Jan 2, 2025

Uh oh!

q-posev commented Jan 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NastaMauger commented Feb 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

q-posev commented Feb 17, 2025

Uh oh!

NastaMauger commented Mar 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MatthewRHermes commented Mar 2, 2025

Uh oh!

sunqm commented Mar 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NastaMauger commented Jul 3, 2025

Uh oh!

scemama left a comment

Choose a reason for hiding this comment

Uh oh!

q-posev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NastaMauger Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NastaMauger Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

q-posev Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

q-posev Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

q-posev Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NastaMauger Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NastaMauger commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

q-posev left a comment

NastaMauger commented Dec 20, 2024 •

edited

Loading

NastaMauger commented Feb 17, 2025 •

edited

Loading

NastaMauger commented Mar 1, 2025 •

edited

Loading

sunqm commented Mar 8, 2025 •

edited

Loading

NastaMauger Jul 4, 2025 •

edited

Loading

NastaMauger Jul 4, 2025 •

edited

Loading

q-posev Jul 4, 2025 •

edited

Loading

q-posev Jul 4, 2025 •

edited

Loading

q-posev Jul 4, 2025 •

edited

Loading

NastaMauger Jul 4, 2025 •

edited

Loading

NastaMauger commented Jul 4, 2025 •

edited

Loading

NastaMauger commented Jul 4, 2025 •

edited

Loading