OpenBLAS 0.3.24 fails on FreeBSD port

[eshibusawa]

Description

OpenBLAS 0.3.24 fails to build scipy during configure and some of octave-forge package at runtime on some CPUs.
The failure is caused when I used CLANG and enabled OPENMP, even if OpenBLAS's regression test passed.

A FreeBSD committer says about this issue:

E.g. such an error has been reported for MacPorts: #4239.
This is not exactly the same one, but there are some common points. In their case it seems caused by a combination of some versions of clang × some version of the linker.

Environment

uname -a
FreeBSD blizzard 13.2-RELEASE-p4 FreeBSD 13.2-RELEASE-p4 GENERIC amd64

clang --version
FreeBSD clang version 14.0.5 (https://github.com/llvm/llvm-project.git llvmorg-14.0.5-0-gc12386ae247c)
Target: x86_64-unknown-freebsd13.2
Thread model: posix
InstalledDir: /usr/bin

gfortran12 --version
GNU Fortran (FreeBSD Ports Collection) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Other information

• bug report on FreeBSD bugzilla
  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273219
• bug report about scipy build crash
  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=274284

[rgommers]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=274284 seems resolved, and was a pretty simple issue of pybind11 not being detected during the SciPy build. That seems unrelated to OpenBLAS.

The second one is possibly due to OpenBLAS. It's a pretty long thread, but this seems to be the relevant comment (copy of https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273219#c22):

As requested by antoine (for portmgr), this update has been reverted as it broke py-scipy during configure and octave at runtime on some CPUs. Please check the issue and resubmit with corrections. We will then do an exp-run to ensure that the issue has indeed been resolved.

Known bad CPU:

CPU: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz (2600.15-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406f1  Family=0x6  Model=0x4f  Stepping=1
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended
Features=0x21cbfbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,PQE,RDSEED,ADX,SMAP,PROCTRACE>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics

Known good CPU:

CPU: Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz (2200.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x50657  Family=0x6  Model=0x55  Stepping=7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended
Features=0xd39ffffb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,MPX,PQE,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PROCTRACE,AVX512CD,AVX512BW,AVX512VL>
  Structured Extended Features2=0x818<PKU,OSPKE,AVX512VNNI>
  Structured Extended
Features3=0xbc000400<MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x2b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics

Looks like the new openblas is either built with too many CPU features enabled or the dispatch logic (if there is any) doesn't work correctly.

[rgommers]

It's still very well possible that that second issue is also related to how SciPy is built within FreeBSD, since OpenBLAS tests alone pass and SciPy in FreeBSD was just switched from the distutils to the meson based build.

Although another comment says: On some amd64 CPUs, this change broke py-scipy during configure, and octave at runtime. In the end, octave kills the package builders.

There's also mentions of armv7 higher up, but the comment above this one is about Intel CPUs. It'd be great to find a way to narrow this down. OpenBLAS has reasonable CI coverage of FreeBSD and that all works.

[rgommers]

The two FreeBSD CI jobs OpenBLAS has are GCC 12.2 on x86-64 with pthreads, see https://github.com/OpenMathLib/OpenBLAS/wiki/CI-jobs-overview. So the Clang + OpenMP + FreeBSD combo isn't tested in CI on this repo.

[martin-frbg]

Strange how this would occur with a Haswell-generation cpu but not with Skylake, and also not cause a hang in OpenBLAS' own tests. And I wonder what scipy's configure job does at the point that leads to the hang - I'd think it would only need to check for the presence of some kind of BLAS and maybe go as far as linking some test source against it, but are you actually executing something more elaborate than openblas_get_config at that point ?

[rgommers]

And I wonder what scipy's configure job does at the point that leads to the hang ...

It uses pkg-config or cmake to detect OpenBLAS. There's no attempt to link anything.

[eshibusawa]

Hi,

Thank you for quick reply.

And I wonder what scipy's configure job does at the point that leads to the hang ...

It is unclear for me however it seems to compile and link some test source.

creating /tmp/tmpdwafuaq4/tmp
creating /tmp/tmpdwafuaq4/tmp/tmpdwafuaq4
INFO: compile options: '-c'
INFO: cc: /tmp/tmpdwafuaq4/source.c
INFO: cc /tmp/tmpdwafuaq4/tmp/tmpdwafuaq4/source.o -L/usr/local/lib -L/usr/local/lib/gcc12/gcc/x86_64-portbld-freebsd13.2/12.2.0/../../../ -lopenblas -lgfortran -o /tmp/tmpdwafuaq4/a.out
INFO:   FOUND:
INFO:     libraries = ['openblas', 'gfortran', 'openblas', 'gfortran']
INFO:     library_dirs = ['/usr/local/lib', '/usr/local/lib/gcc12/gcc/x86_64-portbld-freebsd13.2/12.2.0/../../../']
INFO:     language = c
INFO:     define_macros = [('HAVE_CBLAS', None)]
INFO: 
INFO:   FOUND:
INFO:     libraries = ['openblas', 'gfortran', 'openblas', 'gfortran']
INFO:     library_dirs = ['/usr/local/lib', '/usr/local/lib/gcc12/gcc/x86_64-portbld-freebsd13.2/12.2.0/../../../']
INFO:     language = c
INFO:     define_macros = [('HAVE_CBLAS', None)]
INFO: 

The CPU of my PC is the following:

CPU: Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz (2900.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0xa0655  Family=0x6  Model=0xa5  Stepping=5
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x29c67af<FSGSBASE,TSCADJ,SGX,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
  Structured Extended Features2=0x40000018<PKU,OSPKE,SGXLC>
  Structured Extended Features3=0xbc000400<MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x2b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics

A comment says:

I see python process tightly looping doing nothing but calling sched_yield(),
the call comes from libomp.so.1. Most likely cause is the following function:

void __kmp_infinite_loop(void) {
  static int done = FALSE;

  while (!done) {
    KMP_YIELD(TRUE);
  }
}

I do not know anything about OpenMP runtime to understand more, and binaries
do not have symbols.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=274284#c3

[brada4]

https://github.com/freebsd/freebsd-ports/blob/main/math/openblas/Makefile

It will build cpu-specific non-portable package if built interactively. I suspect locally built package was brought to lower grade CPU and is SIGILL-failing.
Just my 1c

[eshibusawa]

I suspect locally built package was brought to lower grade CPU and is SIGILL-failing.

I'm sorry I do not think so.
The bug reporters did not use prebuilt binary package but did build from source using package build tool called "poudriere".
In the default build settings on FreeBSD DYNAMIC_ARCH is enabled so if some user uses prebuilt package SIGILL-failing will not occur as long as CPU architecture detection works.

[martin-frbg]

he has a point in that any thread getting killed for whatever reason would probably lead to a hang, but I think the signal would also be recorded in the log

[martin-frbg]

I cannot reproduce this in a VM with 4 cpus, and unfortunately the only "real" FreeBSD machine with >10 cores that I have access to does not have either gcc or pkg installed and I am only an unprivileged user there. Is there any way to run pkg_add as unprivileged user to install packages locally (rather than having to bootstrap gcc12 there myself) ?

[brada4]

It is quite clear that "working machine" has avx512 while "broken machine" not ad there is ifdef that batch build is DYNAMIC_ARCH while casual interactive build not. Since no info on procedence of openblas package I suspect build machine changed down the road.

[eshibusawa]

Thank you for your effort for trying reproduce the problem. As far as I know there is no way to pkg install under the unprivileged user. However I will investigate that way and some method to reproduce the problem on a 4 CPU VM.

Since no info on procedence of openblas package I suspect build machine changed down the road.

I think the casual interactive build was not used on any system reported the problem. Because both Xeon system are FreeBSD package builder and CometLake is my PC where I used poudriere. As mentioned above poudriere is package build tool so it runs batch build.

but I think the signal would also be recorded in the log

The poudriere killed the python process building scipy after the 7200 seconds time out. This event was not recorded in my PC's kernel debug message.

I think somewhat strange, on my PC, why octave-forge problem is not reproduced while scipy problem is reproduced.

[brada4]

It is one on one poll... can you make sure 3 packages you talk about are built on the same machine?

[eshibusawa]

can you make sure 3 packages you talk about are built on the same machine?

YES. I make sure 3 packages I talk about are built on the same machine. Because poudriere builds ALL DEPENDENCIES FROM SOURCE and installs them to chroot based system (called jail) before building some port. For example before scipy build poudriere builds 118 dependencies (including gcc, gfortran, OpenBLAS and so forth) using base system compiler. Please refer the brief document about poudriere package build system.
https://man.freebsd.org/cgi/man.cgi?poudriere
Is there anything for supporting for your investigation for finding a way to narrow this couse? I'm really having trouble solving this problem...

In addition, I think that AVX instructions are irrelevant for this problem because it is disabled in batch build settings except in the case it is enabled explicitly.