[CI]: Epic: fix everything

[apostasie]

Currently known flaky tests (could be a test issue or a nerdctl issue)

New entrants:

• [CI]: TestNetworkInspect is flaky in windows #4322
• [CI]: Alma extended attributes getting lost (with nydus convert) (help needed) #4332
• [CI]: TestContainerRmIptables fails binding on port #4347
• [CI] Canary is hanging-up #4363
• [CI] CI runs should output containerd / buildkitd logs #4365
• [CI] Run a local rekor instead of depending on availability of public service #4366

Third-party:

• P0: advanced: [CI]: Windows failure on pull: UtilityVM: Incorrect function.: unknown #3524
• P0: [CI]: buildkit Internal: not found #4142

Possibly EL8 / slow environments specific:

• P1: [LEGACY] [CI]: ipfs /root/.ipfs/api not available #4118
• P1: [LEGACY] [CI]: TestRestartWithSignal does not seem to (always?) work correctly (EL8 only?) #4068
• P1: [LEGACY] [CI]: NoneNetworkManager is presumably flaky wrt hostname #3971

Kube specific:

• P1 (minor, but annoying): [CI]: Kube failure on commit + save #4106
  • to be confirmed this is still happening with containerd v2

Test currently disabled (link to documented issue):

• [URGENT] Regression? TestRunWithTtyAndDetached on windows appears broken on main #3437
• Is IPFS+estargz working? #3475
• see TestNetworkInspect (none and host inspection)

"Retry-them" tests that are clearly broken:

• [CI]: Syslog is flaky #4141
• [CI]: OCI runtime start failed: cannot start a container that has stopped: unknown #4143
• [CI]: TestComposeUpWithExternalNetwork network failure #4146
• [CI]: TestLogsWithoutNewlineOrEOF flaky #4147

Others:

• P0: [CI]: TestRunStargz is racy #4066

To confirm:

• [CI]: failure docker - TestBuildWithBuildArg/EmptyArgValueOverridesDefault #3727

CI structural issues

• P1: [CI]: github cache is too aggressive on limiting requests (429) #3733
• P2: Tests that require not-rootless are getting skipped with target=docker (where the cli is not run as root) #3453

Tests to add

• Test login, push (and authenticated pull) against docker hub #3494
• Expert: Add testing for OCI hooks #3156
• -v SRC:DST:rro,rprivate should be tested in the CI #1989
• CI: test the CRI log viewer #1957
• CI: add test for compose + restart #178

Enhancements: tigron

• [CI/tigron]: investigate file descriptor leak detection as a testing feature #4029
• [CI/tigron]: investigate go routine leak detection as a testing feature #4028
• [Tigron]: roadmap and tracking epic for work on test tooling #3448

Enhancements: generic

• dockerfile specific: [CI]: git clone revamp, part II #4031
• dockerfile specific: [CI] Epic: possible enhancements to current Dockerfile #4021
• [CI]: Discussion: should we drop testing inside a container? #3858
• [CI]: ghcr.io/containerd/stargz-snapshotter:0.15.1-kind needs an arm64 variant #3845
• We should have a publish scripts for the testing images we need, and we should not depend on third party images #3480
• [CI]: Disable test retries #3303
• Add harbor to our integration testing suite? #3257
• fix unit tests so that we can race them

Performance tracking

• Latest major disruption: [CI]: State of the CI #3924
• [CI]: Tracking status #3577

Ongoing work

Evergreen playground:

• [FUTURE] [CI]: Evergreen / Everred #4129

CI work in progress:

• [WINDOWS] updated busybox image & registry image #3636
• [Experimental] Windows registry #3579

[apostasie]

@AkihiroSuda could we label area CI and help wanted?

[apostasie]

@AkihiroSuda the following are ready (in order), at your convenience.

• [CI]: quality of life: minor debugging output fixes #4111
• [CI]: fix internal command flaky test (window) #4114
• [Tigron]: allow per-command environment override #4116
• Update containerd to v2.0.5 #4125
• [PRIORITY] [CI]: Fix EL8 tests execution #4127
• [Tigron]: revised Data interface #4081 <- this one is bigger

[apostasie]

New round of CI PRs:

• [CI]: fix use of non unique identifier #4250
• [CI]: re-enable tests following fixes #4248
• [CI]: tests cleanup #4247
• [CI]: rewrite logs tests #4209
• [CI]: rewrite some compose tests #4203

[Shubhranshu153]

Can we add this issue to the epic: #4322

[apostasie]

Can we add this issue to the epic: #4322

Done.