cmd/compile: generics add significant build time and build size overhead

[xaurx]

Go version

go 1.20.7 linux/amd64 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/user/Library/Caches/go-build'
GOENV='/Users/user/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/user/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/user/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.21.3/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.21.3/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.21.3'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/03/7b5nrjcj4b31k3mvrqswnxxm0000gn/T/go-build1457705092=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

We have a large DB-like solution in golang. This project has a fancy b-tree implementation with around 10x B-tree instances working with different key/value structs using interface{}.
To save some memory and improve the performance we recently switched to generics, so B-Tree is now a parametrized struct with key/value types.

What did you see happen?

We found the following important issues after switching to generics:

1. Cold CI build time has increased from 6 to 16 minutes
2. CLI tool executable size has increased from 48 to 58 MB.
3. go build cache has increased from 760MB to 4.4GB (after clean build)

All of those seem rather bad and I hope someone would be interested looking into it.

What did you expect to see?

Not so significant build time and go build cache size changes.
Executable size is not so important, though is very surprising as well.

[randall77]

Can you show us your code? Without that, it will be hard for us to look into it.

[xaurx]

It's possible to show code privately under NDA. Works for you?

[randall77]

No, I'm not signing an NDA.

[xaurx]

Let me check. Maybe I can collect/provide some info which would be helpful?

[randall77]

It would be interesting to see the output of go tool nm on the binaries before and after.

[xaurx]

ok. Can we send them to email in your github profile?

[randall77]

Sure.

[randall77]

Likely closely related to #51957 .

[randall77]

@mdempsky This issue would at least be partially fixed by moving away from self-contained object files. Thin object files might help with the cpu, and almost certainly would help with the build cache size.
(They have a large package that then gets imported up the dependency tree several steps.)