Frequent random segfaults after Ubuntu 21.10 upgrade

[jkaye2012]

Description

HLS has been working flawlessly for me for months. Today, I upgraded my system from 21.04 to 21.10, and HLS is segfaulting. I'm not sure what could cause this, though I'm guessing it could be related to the glibc toolchain upgrade (to 2.34, https://discourse.ubuntu.com/t/impish-indri-release-notes/21951).

I have tried reinstalling HLS, nuking all stack caches, fully rebuilding, and even tried a few hand-written cradles to debug locally. I have been unable to identify anything that could be causing this.

Your environment

Which OS do you use:
Ubuntu 21.10

Which LSP client (editor/plugin) do you use:
Emacs + lsp-mode

Describe your project (alternative: link to the project):
Stack project, basic auto-detected configuration (no explicit hie.yaml)

Steps to reproduce

1. Install hls via ghcup
2. Run the language server

Expected behaviour

The language server should analyze the code.

Actual behaviour

The language server segfaults more than 90% of the time.

Include debug information

Please let me know what other specific information I can provide to aid debugging.

Here are three subsequent runs (no changes made in the interim):

First run, partial completion:

jkaye@jkaye-linux ~/git/adaptive (master) $ haskell-language-server-8.10.7 src
haskell-language-server version: 1.5.1.0 (GHC: 8.10.7) (PATH: /home/jkaye/.ghcup/bin/haskell-language-server-8.10.7~1.5.1) (GIT hash: 745ef26f406dbdd5e4a538585f8519af9f1ccb09)
 ghcide setup tester in /home/jkaye/git/adaptive.
Report bugs at https://github.com/haskell/haskell-language-server/issues

Step 1/4: Finding files to test in /home/jkaye/git/adaptive
Found 16 files

Step 2/4: Looking for hie.yaml files that control setup
Found 1 cradle
  ()

Step 3/4: Initializing the IDE

Step 4/4: Type checking the files
2022-01-16 17:20:59.703536477 [ThreadId 94] INFO hls:	Consulting the cradle for "src/Adaptive/Utility/Error.hs"
2022-01-16 17:20:59.703669999 [ThreadId 94] WARNING hls:	No [cradle](https://github.com/mpickering/hie-bios#hie-bios) found for src/Adaptive/Utility/Error.hs.
 Proceeding with [implicit cradle](https://hackage.haskell.org/package/implicit-hie).
You should ignore this message, unless you see a 'Multi Cradle: No prefixes matched' error.
2022-01-16 17:21:00.377424647 [ThreadId 105] INFO hie-bios:	The following GHC options are incompatible with GHCi and have not been passed to it: -O2
2022-01-16 17:21:00.377509719 [ThreadId 105] INFO hie-bios:	Configuring GHCi with the following packages: adaptive
2022-01-16 17:21:00.834241945 [ThreadId 112] INFO hie-bios:	/home/jkaye/git/adaptive/.stack-work/install/x86_64-linux-tinfo6/7030a16b185eb149ddf197c83e7f7fa816846865081aabf24f9a0b5da8c2d2ff/8.10.7/pkgdb:/home/jkaye/.stack/snapshots/x86_64-linux-tinfo6/7030a16b185eb149ddf197c83e7f7fa816846865081aabf24f9a0b5da8c2d2ff/8.10.7/pkgdb:/home/jkaye/.stack/programs/x86_64-linux/ghc-tinfo6-8.10.7/lib/ghc-8.10.7/package.conf.d
2022-01-16 17:21:01.909749557 [ThreadId 94] INFO hls:	Using interface files cache dir: /home/jkaye/.cache/ghcide/main-b5a7c46a8060146ee8744a52fe9b8a42c793ec8b
2022-01-16 17:21:01.909958407 [ThreadId 94] INFO hls:	Making new HscEnv[main]
2022-01-16 17:21:03.872761806 [ThreadId 245] INFO hls:	File:     /home/jkaye/git/adaptive/src/Adaptive/Utility.hs
Hidden:   no
Range:    2:5-2:34
Source:   typecheck
Severity: DsWarning
Message:  The export item ‘module Adaptive.Utility.Error’ exports nothing
File:     /home/jkaye/git/adaptive/src/Adaptive/Utility.hs
Hidden:   no
Range:    6:1-6:30
Source:   typecheck
Severity: DsWarning
Message: 
  The import of ‘Adaptive.Utility.Error’ is redundant
  except perhaps to import instances from ‘Adaptive.Utility.Error’
  To import instances alone, use: import Adaptive.Utility.Error()
Segmentation fault (core dumped)

Second run, segfault with no analysis:

jkaye@jkaye-linux ~/git/adaptive (master) <1> $ haskell-language-server-8.10.7 src
haskell-language-server version: 1.5.1.0 (GHC: 8.10.7) (PATH: /home/jkaye/.ghcup/bin/haskell-language-server-8.10.7~1.5.1) (GIT hash: 745ef26f406dbdd5e4a538585f8519af9f1ccb09)
 ghcide setup tester in /home/jkaye/git/adaptive.
Report bugs at https://github.com/haskell/haskell-language-server/issues

Step 1/4: Finding files to test in /home/jkaye/git/adaptive
Found 16 files

Step 2/4: Looking for hie.yaml files that control setup
Found 1 cradle
  ()

Step 3/4: Initializing the IDE

Step 4/4: Type checking the files
2022-01-16 17:19:11.645225456 [ThreadId 100] INFO hls:	Consulting the cradle for "src/Adaptive/Utility.hs"
2022-01-16 17:19:11.645401312 [ThreadId 100] WARNING hls:	No [cradle](https://github.com/mpickering/hie-bios#hie-bios) found for src/Adaptive/Utility.hs.
 Proceeding with [implicit cradle](https://hackage.haskell.org/package/implicit-hie).
You should ignore this message, unless you see a 'Multi Cradle: No prefixes matched' error.
2022-01-16 17:19:12.325040762 [ThreadId 104] INFO hie-bios:	The following GHC options are incompatible with GHCi and have not been passed to it: -O2
2022-01-16 17:19:12.325127715 [ThreadId 104] INFO hie-bios:	Configuring GHCi with the following packages: adaptive
2022-01-16 17:19:12.777375439 [ThreadId 112] INFO hie-bios:	/home/jkaye/git/adaptive/.stack-work/install/x86_64-linux-tinfo6/7030a16b185eb149ddf197c83e7f7fa816846865081aabf24f9a0b5da8c2d2ff/8.10.7/pkgdb:/home/jkaye/.stack/snapshots/x86_64-linux-tinfo6/7030a16b185eb149ddf197c83e7f7fa816846865081aabf24f9a0b5da8c2d2ff/8.10.7/pkgdb:/home/jkaye/.stack/programs/x86_64-linux/ghc-tinfo6-8.10.7/lib/ghc-8.10.7/package.conf.d
2022-01-16 17:19:13.811424401 [ThreadId 100] INFO hls:	Using interface files cache dir: /home/jkaye/.cache/ghcide/main-b5a7c46a8060146ee8744a52fe9b8a42c793ec8b
2022-01-16 17:19:13.811657489 [ThreadId 100] INFO hls:	Making new HscEnv[main]
Segmentation fault (core dumped)

Third run, success:

jkaye@jkaye-linux ~/git/adaptive (master) $ haskell-language-server-8.10.7 src
haskell-language-server version: 1.5.1.0 (GHC: 8.10.7) (PATH: /home/jkaye/.ghcup/bin/haskell-language-server-8.10.7~1.5.1) (GIT hash: 745ef26f406dbdd5e4a538585f8519af9f1ccb09)
 ghcide setup tester in /home/jkaye/git/adaptive.
Report bugs at https://github.com/haskell/haskell-language-server/issues

Step 1/4: Finding files to test in /home/jkaye/git/adaptive
Found 16 files

Step 2/4: Looking for hie.yaml files that control setup
Found 1 cradle
  ()

Step 3/4: Initializing the IDE

Step 4/4: Type checking the files
2022-01-16 17:19:53.421240657 [ThreadId 100] INFO hls:	Consulting the cradle for "src/Adaptive/Utility.hs"
2022-01-16 17:19:53.421438859 [ThreadId 100] WARNING hls:	No [cradle](https://github.com/mpickering/hie-bios#hie-bios) found for src/Adaptive/Utility.hs.
 Proceeding with [implicit cradle](https://hackage.haskell.org/package/implicit-hie).
You should ignore this message, unless you see a 'Multi Cradle: No prefixes matched' error.
2022-01-16 17:19:54.106772535 [ThreadId 104] INFO hie-bios:	The following GHC options are incompatible with GHCi and have not been passed to it: -O2
2022-01-16 17:19:54.106854736 [ThreadId 104] INFO hie-bios:	Configuring GHCi with the following packages: adaptive
2022-01-16 17:19:54.546994286 [ThreadId 112] INFO hie-bios:	/home/jkaye/git/adaptive/.stack-work/install/x86_64-linux-tinfo6/7030a16b185eb149ddf197c83e7f7fa816846865081aabf24f9a0b5da8c2d2ff/8.10.7/pkgdb:/home/jkaye/.stack/snapshots/x86_64-linux-tinfo6/7030a16b185eb149ddf197c83e7f7fa816846865081aabf24f9a0b5da8c2d2ff/8.10.7/pkgdb:/home/jkaye/.stack/programs/x86_64-linux/ghc-tinfo6-8.10.7/lib/ghc-8.10.7/package.conf.d
2022-01-16 17:19:55.591332999 [ThreadId 100] INFO hls:	Using interface files cache dir: /home/jkaye/.cache/ghcide/main-b5a7c46a8060146ee8744a52fe9b8a42c793ec8b
2022-01-16 17:19:55.591548518 [ThreadId 100] INFO hls:	Making new HscEnv[main]
2022-01-16 17:19:57.52640668 [ThreadId 258] INFO hls:	File:     /home/jkaye/git/adaptive/src/Adaptive/Utility.hs
Hidden:   no
Range:    2:5-2:34
Source:   typecheck
Severity: DsWarning
Message:  The export item ‘module Adaptive.Utility.Error’ exports nothing
File:     /home/jkaye/git/adaptive/src/Adaptive/Utility.hs
Hidden:   no
Range:    6:1-6:30
Source:   typecheck
Severity: DsWarning
Message: 
  The import of ‘Adaptive.Utility.Error’ is redundant
  except perhaps to import instances from ‘Adaptive.Utility.Error’
  To import instances alone, use: import Adaptive.Utility.Error()
2022-01-16 17:19:57.899411774 [ThreadId 240] INFO hls:	File:     /home/jkaye/git/adaptive/src/Adaptive/Interpreter/InMemory.hs
Hidden:   no
Range:    49:32-53:111
Source:   compile
Severity: DsWarning
Message: 
  Pattern match(es) are non-exhaustive
  In a case alternative:
  Patterns not matched:
  UpdateFile _ _ _ _
  DeleteFile _ _ _
2022-01-16 17:19:57.921917721 [ThreadId 132] INFO hls:	finish: User TypeCheck (took 2.29s)
2022-01-16 17:19:57.988142033 [ThreadId 1276] INFO hls:	finish: GetHie (took 0.07s)
2022-01-16 17:19:57.988956306 [ThreadId 1348] INFO hls:	finish: GenerateCore (took 0.00s)

Completed (16 files worked, 0 files failed)

Because I don't have debug symbols, gdb output is not particularly useful:

Thread 3 "haskell-langu:w" received signal SIGSEGV, Segmentation fault.
[Switching to LWP 74675]
0x00007ffff4fe48ef in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) info threads
  Id   Target Id                   Frame 
  1    LWP 74673 "haskell-languag" 0x00000000063a8883 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7335098) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  2    LWP 74674 "ghc_ticker"      0x00000000063aa1d4 in __libc_read (fd=fd@entry=3, buf=buf@entry=0x7ffff7a86cf8, nbytes=nbytes@entry=8) at ../sysdeps/unix/sysv/linux/read.c:27
* 3    LWP 74675 "haskell-langu:w" 0x00007ffff4fe48ef in ?? () from /lib/x86_64-linux-gnu/libc.so.6
  4    LWP 74676 "haskell-langu:w" 0x000000000647d229 in poll ()
  5    LWP 74677 "haskell-langu:w" 0x00000000062be027 in base_GHCziList_zdwbreak_info ()
  6    LWP 74678 "haskell-langu:w" 0x00000000063a8883 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fffec000bac) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  7    LWP 74933 "haskell-langu:w" 0x000000000647fc7f in epoll_wait ()
(gdb) info locals 
No symbol table info available.
(gdb) where
#0  0x00007ffff4fe48ef in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff50676ca in open64 () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x0000000041696da6 in ?? ()
#3  0x00000042070a3428 in ?? ()
#4  0x0000000000000000 in ?? ()

[pepeiborra]

See https://haskell-language-server.readthedocs.io/en/latest/troubleshooting.html#static-binaries

[jneira]

Maybe related with #2186
The next release will use static binaries produced in alpine linux using musl, so maybe the might help to this issue.

[jkaye2012]

Building from source (with cabal) seems to have resolved the issue for me as well. I'm going to close this since the related issue is mentioned here. Thanks for the tips!