[no-relnote] Add E2E documentation

Signed-off-by: Carlos Eduardo Arango Gutierrez <[email protected]>

Author: ArangoGutierrez

tests/e2e/README.md

@@ -0,0 +1,140 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# NVIDIA Container Toolkit – End‑to‑End (E2E) Test Suite
+
+---
+
+## 1  Scope & Goals
+This repository contains a **Ginkgo v2 / Gomega** test harness that exercises an
+NVIDIA Container Toolkit (CTK) installation on a **remote GPU‑enabled host** via
+SSH.  The suite validates that:
+
+1. CTK can be installed (or upgraded) head‑less (`INSTALL_CTK=true`).
+2. The specified **container image** runs successfully under `nvidia-container-runtime`.
+3. Errors and diagnostics are captured for post‑mortem analysis.
+
+The tests are intended for continuous‑integration pipelines, nightly
+compatibility runs, and pre‑release validation of new CTK builds.
+
+---
+
+## 2  Execution model
+* The framework **does not** spin up a Kubernetes cluster; it drives a single
+  host reachable over SSH.
+* All commands run in a Ginkgo‑managed context (`ctx`) so they abort cleanly on
+  timeout or Ctrl‑C.
+* Environment discovery happens once in `TestMain` → `getTestEnv()`; parameters
+  are therefore immutable for the duration of the run.
+
+---
+
+## 3  Prerequisites
+
+| Item | Version / requirement |
+|------|-----------------------|
+| **Go toolchain** | ≥ 1.22 (for building Ginkgo helper binaries) |
+| **GPU‑enabled Linux host** | Running a supported NVIDIA driver; reachable via SSH |
+| **SSH connectivity** | Public‑key authentication *without* pass‑phrase for unattended CI |
+| **Local OS** | Linux/macOS; POSIX shell required by the Makefile |
+
+---
+
+## 4  Environment variables
+
+| Variable | Required | Example | Description |
+|----------|----------|---------|-------------|
+| `INSTALL_CTK` | ✖ | `true` | When `true` the test installs CTK on the remote host before running the image. When `false` it assumes CTK is already present. |
+| `TOOLKIT_IMAGE` | ✔ | `nvcr.io/nvidia/cuda:12.4.0-runtime-ubi9` | Image that will be pulled & executed. |
+| `SSH_KEY` | ✔ | `/home/ci/.ssh/id_rsa` | Private key used for authentication. |
+| `SSH_USER` | ✔ | `ubuntu` | Username on the remote host. |
+| `REMOTE_HOST` | ✔ | `gpurunner01.corp.local` | Hostname or IP address of the target node. |
+| `REMOTE_PORT` | ✔ | `22` | SSH port of the target node. |
+
+> All variables are validated at start‑up; the suite aborts early with a clear
+> message if any are missing or ill‑formed.
+
+---
+
+## 5  Build helper binaries
+
+Install the latest Ginkgo CLI locally so that the Makefile can invoke it:
+
+```bash
+make ginkgo  # installs ./bin/ginkgo
+```
+
+The Makefile entry mirrors the pattern used in other NVIDIA E2E suites:
+
+```make
+bin/ginkgo:
+	GOBIN=$(CURDIR)/bin go install github.com/onsi/ginkgo/v2/ginkgo@latest
+```
+
+---
+
+## 6  Running the suite
+
+### 6.1  Basic invocation
+```bash
+INSTALL_CTK=true \
+TOOLKIT_IMAGE=nvcr.io/nvidia/cuda:12.4.0-runtime-ubi9 \
+SSH_KEY=$HOME/.ssh/id_rsa \
+SSH_USER=ubuntu \
+REMOTE_HOST=10.0.0.15 \
+REMOTE_PORT=22 \
+make test-e2e
+```
+This downloads the image on the remote host, installs CTK (if requested), and
+executes a minimal CUDA‑based workload.
+
+---
+
+## 7  Internal test flow
+
+| Phase | Key function(s) | Notes |
+|-------|-----------------|-------|
+| **Init** | `TestMain` → `getTestEnv` | Collects env vars, initializes `ctx`. |
+| **Connection check** | `BeforeSuite` (not shown) | Verifies SSH reachability using `ssh -o BatchMode=yes`. |
+| **Optional CTK install** | `installCTK == true` path | Runs the distro‑specific install script on the remote host. |
+| **Runtime validation** | Leaf `It` blocks | Pulls `TOOLKIT_IMAGE`, runs `nvidia-smi` inside the container, asserts exit code `0`. |
+| **Failure diagnostics** | `AfterEach` | Copies `/var/log/nvidia-container-runtime.log` & dmesg to `${LOG_ARTIFACTS_DIR}` via `scp`. |
+
+---
+
+## 8  Extending the suite
+
+1. Create a new `_test.go` file under `tests/e2e`.
+2. Use the Ginkgo DSL (`Describe`, `When`, `It` …). Each leaf node receives a
+   `context.Context` so you can run remote commands with deadline control.
+3. Helper utilities such as `runSSH`, `withSudo`, and `collectLogs` are already
+   available from the shared test harness (see `ssh_helpers.go`).
+4. Keep tests **idempotent** and clean any artefacts you create on the host.
+
+---
+
+## 9  Common issues & fixes
+
+| Symptom | Likely cause | Fix |
+|---------|--------------|-----|
+| `Permission denied (publickey)` | Wrong `SSH_KEY` or `SSH_USER` | Check variables; ensure key is readable by the CI user. |
+| `docker: Error response from daemon: could not select device driver` | CTK not installed or wrong runtime class | Verify `INSTALL_CTK=true` or confirm CTK installation on the host. |
+| Test hangs at image pull | No outbound internet on remote host | Pre‑load the image or use a local registry mirror. |
+
+## 10  License
+Distributed under the terms of the **Apache License 2.0** (see header).
+