docker run can fail with bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid argument

[cpuguy83]

Description

docker run fails with:

error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid argument: unknown

On kernel 4.9
Strangely this is supposed to be on Ubuntu 22.04, I'll follow up with the user on why they are on such an old kernel on 22.04.
In any case it seems like the detection logic here is not quite right.

It is failing in this call

runc/libcontainer/cgroups/devices/ebpf_linux.go

Lines 161 to 164 in 2685116

	oldProgs, err := findAttachedCgroupDeviceFilters(dirFd)
	if err != nil {
	return nilCloser, err
	}

At this system call:

runc/libcontainer/cgroups/devices/ebpf_linux.go

Lines 45 to 48 in 2685116

	_, _, errno := unix.Syscall(unix.SYS_BPF,
	uintptr(unix.BPF_PROG_QUERY),
	uintptr(unsafe.Pointer(&query)),
	unsafe.Sizeof(query))

Which happens before haveBpfProgReplace()

Steps to reproduce the issue

Likely this is due to an old kernel not supporting the bpf query while also configured for cgroup2.

This has been reported a few places and unfortunately just asked users to upgrade the kernel to fix the issue.

See:

• bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid argument: unknown docker/cli#4273
• Container dont start: 'setting cgroup config for procHooks process caused: bpf_prog_query(BPF_CGROUP_DEVICE) failed: operation not permitted: unknown' docker/cli#3481

Describe the results you received and expected

I believe this should hit the fallback code, but I'm not an expert in this area.

What version of runc are you using?

From the users docker version output:

 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061

Host OS information

From user's docker info output

 Kernel Version: 4.9.337-11
 Operating System: Ubuntu 22.04.2 LTS
 Architecture: aarch64

Host kernel information

From user's docker info output

Kernel Version: 4.9.337-11

[kolyshkin]

The docstring for LoadAttachCgroupDeviceFilter says:

  // Requires the system to be running in cgroup2 unified-mode with kernel >= 4.15 .

So, this configuration is not supported.

Also, https://github.com/opencontainers/runc/blob/main/docs/cgroup-v2.md says:

Kernel

• Recommended version: 5.2 or later

• Minimum version: 4.15

Kernel older than 5.2 is not recommended due to lack of freezer.

Notably, kernel older than 4.15 MUST NOT be used (unless you are running containers with user namespaces), as it lacks support for controlling permissions of devices.

So, the only thing we could have done better here is make the error message less cryptic.

[cpuguy83]

Thanks @kolyshkin!

I'm going to go ahead and close this issue since it seems there is nothing actionable on our end.

[blabno]

I'm on kernel 6.6.21 and still getting this error. Which kernel config flag do I need?

[blabno]

I'm on kernel 6.6.21 and still getting this error. Which kernel config flag do I need?

In my case it was CONFIG_BPF, CONFIG_BPF_SYSCALL, CONFIG_CGROUP_BPF.