cAdvisor e2e failing 100% on core OS

[timstclair]

Error:

F0620 11:09:44.381011   18340 runner.go:290] Error 0: error on host e2e-cadvisor-coreos-beta: command "godep" ["go" "test" "--timeout" "15m0s" "github.com/google/cadvisor/integration/tests/..." "--host" "e2e-cadvisor-coreos-beta" "--port" "8080" "--ssh-options" "-i /home/jenkins/.ssh/google_compute_engine -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o CheckHostIP=no -o StrictHostKeyChecking=no"] failed with error: exit status 1 and output: godep: WARNING: Godep workspaces (./Godeps/_workspace) are deprecated and support for them will be removed when go1.8 is released.
godep: WARNING: Go version (go1.6) & $GO15VENDOREXPERIMENT= wants to enable the vendor experiment, but disabling because a Godep workspace (Godeps/_workspace) exists
--- FAIL: TestDockerContainerSpec (0.98s)
    assertions.go:150: 

    Location:   docker_test.go:225

    Error:      Not equal: "0" (expected)
                    != "" (actual)

    Messages:   Cpu mask should be "0", but is ""

FAIL
FAIL    github.com/google/cadvisor/integration/tests/api    29.639s
ok      github.com/google/cadvisor/integration/tests/healthz    0.012s
godep: go exit status 1

failing line

This has been failing 100% on e2e-cadvisor-coreos-beta since #1333 was merged.

@Random-Liu @pwittrock @euank

NOTE: cadvisor-pull-build-test-e2e coreos-beta VM disabled -- re enable once this issue is resolved.

[timstclair]

@pwittrock do you know when you restarted the jenkins CI VMs? At first glance this doesn't look like the PR it started failing at would have caused it.

[timstclair]

/cc @dchen1107

[timstclair]

Failing since kubekins build #4096 (sorry, Google internal only)

Actually, ignore my comment about #1333 - many builds succeeded on that revision prior to the failures starting.

I'm guessing the core-os VM auto-updated and is no longer setting the CPU mask. I'll ping the pr-builder jenkins to see if it's having the same issue.

[vishh]

@timstclair Were able to root cause this issue?

[timstclair]

No, I haven't tracked it down yet. It looks like this has started affecting the build jobs as well (it didn't yesterday). I think this lends credence to it being caused by a core OS update.

[timstclair]

Let me see if I can reproduce on a new GCE coreos instance.

[vishh]

We can disable CoreOS node from the e2e suite until this issue is resolved.
WDYT?

On Wed, Jun 22, 2016 at 10:42 AM, Tim St. Clair notifications@github.com
wrote:

Let me see if I can reproduce on a new GCE coreos instance.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#1344 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AGvIKMebpzqb8hCjGqUgzMRqXjP98252ks5qOXPzgaJpZM4I7Hq7
.

[timstclair]

I'll disable it for the builder.

[euank]

Do you have more information on how these images are setup and so on (sorry, not familiar with the cadvisor test setup)? For reference, node_e2e disable updates now with this.

If the cadvisor tests fail on newer CoreOSs post-update, that sounds like something to triage and fix..

[timstclair]

Ok, I was able to reproduce this on a fresh coreOS beta image (VERSION_ID=1068.2.0). I just did:

1. Install go to home directory
2. go get godep & cadvisor
3. I had trouble building cadvisor, so I copied the binary built on another machine
4. run build/integration.sh, which surfaced the same error:

$ build/integration.sh 
>> starting cAdvisor locally
Waiting for cAdvisor to start ...
>> running integration tests against local cAdvisor
godep: WARNING: Godep workspaces (./Godeps/_workspace) are deprecated and support for them will be removed when go1.8 is released.
godep: WARNING: Go version (go1.6) & $GO15VENDOREXPERIMENT= wants to enable the vendor experiment, but disabling because a Godep workspace (Godeps/_workspace) exists
--- FAIL: TestDockerContainerSpec (0.51s)
        Location:       docker_test.go:225
    Error:      Not equal: "0" (expected)
                    != "" (actual)
    Messages:   Cpu mask should be "0", but is ""

FAIL
FAIL    github.com/google/cadvisor/integration/tests/api    29.321s
ok      github.com/google/cadvisor/integration/tests/healthz    0.004s
godep: go exit status 1
Integration tests failed
>> stopping cAdvisor
build/integration.sh: line 45:  5341 Killed                  sudo ./cadvisor --docker_env_metadata_whitelist=TEST_VAR

@euank can you take it from here?

[euank]

I'll take a guess that this is related to systemd >= 226's change in cgroup hierarchy, but not sure yet.

I was able to reproduce on a machine with systemd 226 and docker 1.11 launched with --exec-opt native.cgroupdriver=systemd.

The machine in question is gentoo, but I expect it'll reproduce broadly in that configuration. I'll dig further...

[euank]

It finds a cpuset root at both /sys/fs/cgroup/cpuset/system.slice/docker-de6460b5039fa64f505cf383c15dc96d515bbc507c76ec0f7c06a00a5115002f.scope and /sys/fs/cgroup/cpuset/init.scope/system.slice/docker-de6460b5039fa64f505cf383c15dc96d515bbc507c76ec0f7c06a00a5115002f.scope (note the init.scope bit), then tries to use the latter and doesn't find the files it expects and returns "". The first one would be correct. I think this is moby/moby#16256 (comment) .. I'm giving a quick go at patching this similarly to as suggested to be sure.

[euank]

Upstream bug to point to as well: opencontainers/runc#931

Our options are, I think:

1. Wait for runc + docker-on-coreos to be fixed (we might backport the fix)
2. Switch the CoreOS+docker test setup to use cgroupfs driver (non-default)
3. Switch to CoreOS stable where it's still using systemd 225 and thus shouldn't be affected
4. modify cadvisor to handle runc's bad behaviour and do the "right thing" when there are multiple cgroup paths for one container

My preference is 3 to put off having to get a better solution, and hope that 1 happens in the meanwhile. Sound reasonable?