Skip to content

ci(k8s): record why the uwsgi container restarted when the deployment test fails - #16398

Merged
Maffooch merged 1 commit into
devfrom
ci/k8s-uwsgi-restart-diagnostics
Oct 17, 2026
Merged

Maffooch merged 1 commit into
devfrom
ci/k8s-uwsgi-restart-diagnostics

Conversation

@Maffooch

@Maffooch Maffooch commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Shortcut: sc-16687

Description

The test-k8s / Kubernetes Deployment job failed 10 times in the merge queue over the last three weeks with curl exit code 7 (once 28). The logs show the same pattern each time:

  • the pod passes kubectl wait and the login page returns 200;
  • the first POST /api/v2/api-token-auth/ gets upstream prematurely closed connection from nginx;
  • uwsgi starts again and the pod shows a restart (sometimes CrashLoopBackOff).

So the uwsgi container dies while it handles the token request. One guess is an OOM kill during password hashing (the container is limited to 512Mi and runs 4 processes × 4 threads), but nothing confirms it: the failure output never says why the container terminated.

This PR changes no behavior. It only adds two things to the Failed Logs step:

  • each django container's restart count and lastState (this shows OOMKilled or the exit code/signal);
  • the last 100 lines of the previous uwsgi container's logs.

The next failure should show the cause, and the real fix (resources, uwsgi settings, or code) can follow from that.

Test results

The workflow YAML parses, and the label selector defectdojo.org/component=django and container name uwsgi match helm/defectdojo/templates/django-deployment.yaml. The new commands only run in the failure path, so they have not run in CI yet.

Documentation

None needed.

🤖 Generated with Claude Code

… test fails

The k8s deployment job intermittently fails its API auth check because the
uwsgi container restarts on the first POST to /api/v2/api-token-auth/. The
failure logs never showed why it terminated. Print each django container's
last termination state and the previous uwsgi container's logs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Maffooch
Maffooch requested a review from blakeaowens as a code owner October 17, 2026 17:00
@Maffooch Maffooch added this to the 3.4.200 milestone Oct 17, 2026
@Maffooch
Maffooch enabled auto-merge October 17, 2026 17:02
@Maffooch
Maffooch requested a review from devGregA October 17, 2026 17:02
@Maffooch
Maffooch added this pull request to the merge queue Oct 17, 2026
Merged via the queue into dev with commit 96c08b1 Oct 17, 2026
49 checks passed
@Maffooch
Maffooch deleted the ci/k8s-uwsgi-restart-diagnostics branch October 17, 2026 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants