WIP: use HPE Python build and OS Perl build on Polaris

[carns]

I'm not sure of the overall pros and cons or side effects of doing this vs. using Spack-built packages and/or the buildcache. The PR is for discussion.

It trims off some dependencies to save build time (if not using buildcache) or downloads/installs (if using buildcache).

@mdorier @GueroudjiAmal

[mdorier]

Perl is probably fine since it's in /usr so unlikely to change, and it's almost always a build dependency, not a a run-time one. Python, I'd push back on that, because it's a run-time dependency (so it may prevent a proper use of the build cache) and it's a module that can be updated/removed by the admins at any time, hindering reproducibility.

[carns]

Unfortunately we have several other packages on Polaris that must be external (libfabric, compilers, mpich, etc.) because the source either isn't available or is prohibitively complicated to build; just pointing out that we already can't guarantee reproducibility over time for a platform like this.

Can you explain the build cache issue?

Its possible this PR is a moot point now that I can reliably use the build cache on Polaris (by using --fresh with concretize), but I need to set aside some time to measure what's going on. It still seems to take me a surprisingly long time to get to the level of the stack of things like quintain and bedrock.

[mdorier]

You may not in fact have problems with the build cache. In the past I've had a lot of problems when using a platform-provided Python because Spack makes some assumptions about where Python libraries are, and on some platforms that assumption is broken. For instance on Ubuntu the Python packages aren't installed in the same directory as they would be on Debian. That's why I tend to avoid using a platform-provided Python.

[carns]

gotcha, ok.