Skip to content

Conversation

@manodeep
Copy link
Collaborator

@manodeep manodeep commented Nov 24, 2025

Testing for compiler flag updates to SPRs - do not merge (at least yet)


🚀 The latest prerelease access-esm1p6/pr170-22 at f32e5e8 is here: #170 (comment) 🚀

@manodeep manodeep marked this pull request as draft November 24, 2025 01:10
@github-actions
Copy link

The model version in the ./spack.yaml has not been updated.
You can update it manually.

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-1 with commit 4c7edae

🖥️ Gadi Deployment ❌

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-1 with commit 4c7edae

🖥️ Gadi Deployment ✔️

Usage Instructions

access-esm1.6, defined in ./spack.yaml, will be deployed to Gadi as:

  • 2025.11.001 as a Release (when merged).
  • pr170-1 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p6/pr170-1

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-esm1p6-pr170-1 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR.

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-2 with commit 40e0025

🖥️ Gadi Deployment ✔️

Usage Instructions

access-esm1.6, defined in ./spack.yaml, will be deployed to Gadi as:

  • 2025.11.001 as a Release (when merged).
  • pr170-2 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p6/pr170-2

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-esm1p6-pr170-2 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR.

@manodeep
Copy link
Collaborator Author

Both the previous two deployments passed !test repro on the config repo - here, and here

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-3 with commit 2fed6dc

🖥️ Gadi Deployment ✔️

Usage Instructions

access-esm1.6, defined in ./spack.yaml, will be deployed to Gadi as:

  • 2025.11.001 as a Release (when merged).
  • pr170-3 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p6/pr170-3

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-esm1p6-pr170-3 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR.

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-4 with commit c73873c

🖥️ Gadi Deployment ✔️

Usage Instructions

access-esm1.6, defined in ./spack.yaml, will be deployed to Gadi as:

  • 2025.11.001 as a Release (when merged).
  • pr170-4 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p6/pr170-4

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-esm1p6-pr170-4 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR.

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-5 with commit 959b758

🖥️ Gadi Deployment ❌

@manodeep
Copy link
Collaborator Author

!redeploy

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-6 with commit 959b758

🖥️ Gadi Deployment ✔️

Usage Instructions

access-esm1.6, defined in ./spack.yaml, will be deployed to Gadi as:

  • 2025.11.001 as a Release (when merged).
  • pr170-6 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p6/pr170-6

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-esm1p6-pr170-6 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR.

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-7 with commit 6a1f7b4

🖥️ Gadi Deployment ✔️

Usage Instructions

access-esm1.6, defined in ./spack.yaml, will be deployed to Gadi as:

  • 2025.11.001 as a Release (when merged).
  • pr170-7 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p6/pr170-7

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-esm1p6-pr170-7 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR.

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-8 with commit 82be501

🖥️ Gadi Deployment ✔️

Usage Instructions

access-esm1.6, defined in ./spack.yaml, will be deployed to Gadi as:

  • 2025.11.001 as a Release (when merged).
  • pr170-8 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p6/pr170-8

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-esm1p6-pr170-8 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR.

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-14 with commit c85b3ed

🖥️ Gadi Deployment ✔️

Usage Instructions

access-esm1.6, defined in ./spack.yaml, will be deployed to Gadi as:

  • 2025.11.001 as a Release (when merged).
  • pr170-14 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p6/pr170-14

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-esm1p6-pr170-14 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR.

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-15 with commit b0ec7c0

🖥️ Gadi Deployment ✔️

Usage Instructions

access-esm1.6, defined in ./spack.yaml, will be deployed to Gadi as:

  • 2025.11.001 as a Release (when merged).
  • pr170-15 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p6/pr170-15

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-esm1p6-pr170-15 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR.

@manodeep
Copy link
Collaborator Author

The last build with flto still crashes with a segfault (see this).

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-16 with commit 82657d9

🖥️ Gadi Deployment ✔️

Usage Instructions

access-esm1.6, defined in ./spack.yaml, will be deployed to Gadi as:

  • 2025.11.001 as a Release (when merged).
  • pr170-16 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p6/pr170-16

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-esm1p6-pr170-16 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR.

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-17 with commit cd3f8bc

🖥️ Gadi Deployment ✔️

Usage Instructions

access-esm1.6, defined in ./spack.yaml, will be deployed to Gadi as:

  • 2025.11.001 as a Release (when merged).
  • pr170-17 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p6/pr170-17

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-esm1p6-pr170-17 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR.

@manodeep
Copy link
Collaborator Author

manodeep commented Nov 26, 2025

Interestingly, this last build (with these changes to UM7 SPR) crashed with this error:

gc_abort (Processor   192): over-writing due to dim_e_out size
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 192 in communicator MPI_COMM_WORLD
  Proc: [[62655,1],192]
  Errorcode: 9

<snip>

forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
libpthread-2.28.s  000015336D23D990  Unknown               Unknown  Unknown
libopen-pal.so.80  000015336A5B9494  opal_progress         Unknown  Unknown
libmpi.so.40.40.7  000015336A69FEC2  Unknown               Unknown  Unknown
libmpi.so.40.40.7  000015336A704481  Unknown               Unknown  Unknown
libmpi.so.40.40.7  000015336A7463EA  Unknown               Unknown  Unknown
libmpi.so.40.40.7  000015336A6BD824  PMPI_Bcast            Unknown  Unknown
libmpi_mpifh_Inte  000015336DE19765  PMPI_BCAST            Unknown  Unknown
um_hg3.exe         0000000001202FFC  mpl_bcast                  59  mpl_bcast.F90
um_hg3.exe         0000000001202E4B  gcg_rvecsumr              218  gcg_rvecsumr.F90

Since the source file only contains 128 lines, and the traceback shows a line number of 218, I made an educated guess that the offending line is this one. (The traceback shows the call to MPL_BCAST after gcg_rvecsumr, and the only such call in gcg_rvecsumr is in that line). edit Of course, the bug could also be elsewhere and the execution just happened to be at that point within gcg_rvecsumr on a different rank

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-18 with commit 1fcd9c5

🖥️ Gadi Deployment ✔️

Usage Instructions

access-esm1.6, defined in ./spack.yaml, will be deployed to Gadi as:

  • 2025.11.001 as a Release (when merged).
  • pr170-18 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p6/pr170-18

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-esm1p6-pr170-18 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR.

@manodeep
Copy link
Collaborator Author

Judging by the size of the output file, build 18 has gone past the stage where build 17 crashed.

@manodeep
Copy link
Collaborator Author

All three runs finished with deployment 18, the performance is worse, and the results are not bitwise identical. -fp-model precise is going back in.

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-19 with commit 1b7cf02

🖥️ Gadi Deployment ✔️

Usage Instructions

access-esm1.6, defined in ./spack.yaml, will be deployed to Gadi as:

  • 2025.11.001 as a Release (when merged).
  • pr170-19 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p6/pr170-19

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-esm1p6-pr170-19 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR.

@manodeep
Copy link
Collaborator Author

Somewhat surprisingly, deployment 19 is by far the slowest of all the working deployments (throughput ~20.1 years/day). Going back to deployment 12 - which builds with the compiler pragma updates to UM7; and then adding in the source update for disabling bitwise-repro in GCOM4 (this branch which sets GC_FORCE_BITREP to 0 - which the documentation says should have been the case anyway)

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-20 with commit 618fc8a

🖥️ Gadi Deployment ❌

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-21 with commit 3fc5c5f

🖥️ Gadi Deployment ✔️

Usage Instructions

access-esm1.6, defined in ./spack.yaml, will be deployed to Gadi as:

  • 2025.11.001 as a Release (when merged).
  • pr170-21 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p6/pr170-21

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-esm1p6-pr170-21 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR.

@github-actions
Copy link

🚀 Attempted to deploy access-esm1p6 Prerelease pr170-22 with commit f32e5e8

🖥️ Gadi Deployment ✔️

Usage Instructions

access-esm1.6, defined in ./spack.yaml, will be deployed to Gadi as:

  • 2025.11.001 as a Release (when merged).
  • pr170-22 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p6/pr170-22

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.22/spack in the access-esm1p6-pr170-22 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

If the above was not what was expected, commit changes to config/versions.json in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants