windows: route OS-shutdown SIGTERM through SCM to unblock svc.Run

[ablankz]

Description of the issue

Related issue: Fixes #2214

At OS shutdown, when the collector runs as a Windows service child (spawned by start-amazon-cloudwatch-agent.exe), csrss delivers CTRL_SHUTDOWN to the child (mapped to SIGTERM by the Go runtime) but the SCM does not route SERVICE_CONTROL_SHUTDOWN to it (that goes to the SCM-registered launcher). kardianos/service v1.2.1's non-interactive Run() waits only on the SCM channel and does not watch SIGTERM, so svc.Run never returns, main never returns, and the runtime's ctrlHandler is parked in block(). csrss meanwhile waits for the process to exit; the whole system shutdown deadlocks (~4–5 min → Event ID 6008 + Kernel-Power 41). sc stop / Stop-Service (SCM STOP via ControlService) is unaffected.

The issue is scoped to non-container Windows service installs: Windows containers take the -console true path at cmd/start-amazon-cloudwatch-agent/path_windows.go:43 and exit cleanly, and non-Windows builds call reloadLoop directly from main, both of which are unaffected. kardianos already accepts SERVICE_CONTROL_SHUTDOWN via cmdsAccepted at service_windows.go:182, so the handler is present — the deadlock in stage 1 (SIGTERM in block()) simply prevents stage 2 (the SCM shutdown control) from ever being reached.

Description of changes

Minimal, scoped, and no-op outside the affected case. When a non-SIGHUP terminating OS signal reaches reloadLoop's signal goroutine and we're running as a Windows service, this PR issues SERVICE_CONTROL_STOP against our own SCM entry via x/sys/windows/svc/mgr, then lets kardianos's normal STOP path (prg.Stop → close(stop)) drive svc.Run to return cleanly. Falls back to os.Exit(0) only if the SCM path is unavailable.

Why not kardianos/service.Service.Stop() directly: kardianos looks up the service by its own Config.Name (*fServiceName, default "telegraf" here), which is not registered in the SCM. The collector's real SCM registration is AmazonCloudWatchAgent (or whatever the operator installed under --service-name), and its ProcessId in the SCM points to the collector (SCM tracks the collector's PID, not the launcher's, after the launcher's SetServiceStatus handoff). We discover this entry by enumerating SCM services and matching ProcessId against os.Getpid() — no hardcoded name, so custom --service-name installations continue to work.

cmd/amazon-cloudwatch-agent/amazon-cloudwatch-agent.go (2 hunks / +6 lines):

• In reloadLoop's signal goroutine, on a non-SIGHUP terminating signal, call handleTerminatingSignalDispatch(stop, stopWaitTimeout). It tries SCM STOP first and blocks on <-stop (bounded by stopWaitTimeout = 30s). On failure or timeout, sets terminatingSignalReceived. Then the existing cancel() runs.
• In (*program).run(), after reloadLoop returns, call handleTerminatingSignal() — the fallback which os.Exit(0)s iff the flag was set.

New files under cmd/amazon-cloudwatch-agent/:

• shutdown_signal.go — package-level state (stopWaitTimeout, terminatingSignalReceived atomic.Bool, requestSCMStopFn test seam) and the cross-platform handleTerminatingSignalDispatch(stopCh, timeout).
• shutdown_signal_windows.go — the Windows implementation: minimal scmManager / scmService interfaces (so tests can substitute fakes) with concrete adapters over winmgr.Mgr / winmgr.Service, four test seams (exitFunc, isWinService, ownProcessID, scmConnectFunc), findOwnSCMServiceName(m) (enumerate + match ProcessId, skips ACCESS_DENIED and Query errors), requestSCMStop() (Connect → find → Control(Stop)), and handleTerminatingSignal() (fallback exitFunc(0)).
• shutdown_signal_notwindows.go — requestSCMStop() { return false } and handleTerminatingSignal() {} under //go:build !windows.
• shutdown_signal_test.go — cross-platform tests: 3 atomic-flag tests + 3 dispatch tests using the requestSCMStopFn seam (success path, SCM-unavailable path, SCM-accepted-but-timeout path).
• shutdown_signal_windows_test.go — Windows-only tests using fake scmManager / scmService: 5 findOwnSCMServiceName scenarios (match-by-PID / no-match / ListServices error / skips-OpenService-errors / skips-Query-errors), 5 requestSCMStop scenarios (not-a-service / Connect fail / no-matching-service / Control fail / success), and 3 handleTerminatingSignal scenarios (flag-unset / not-a-service / flag+service → exit).

Follow-up: main waits for teardown before ExitProcess

Once SCM STOP unblocks svc.Run, main returns and the Go runtime calls ExitProcess. But otelcol.Shutdown runs on a parallel goroutine (from otelcol's own SIGTERM handler); if it hasn't finished by the time main returns, its teardown is truncated — the final "Shutdown complete." log line and any pending flushes are dropped silently. In practice on this codebase Shutdown is intrinsically sub-second so the race is rarely observable, but a slow-shutdown plugin, a network flush, or any future teardown work would surface it.

The second commit (15cb73d) adds a done-channel signalled by (*program).run and waited on by main after s.Run() returns:

• shutdown_signal.go (+30 lines): runCompleteChan (chan struct{}) plus sync.Once guard, runCompleteTimeout = 30s, signalRunComplete(), waitRunComplete().
• amazon-cloudwatch-agent.go (+4 lines): (*program).run defers signalRunComplete so it fires on every normal return path. main's Windows service branch waits on runCompleteChan after s.Run returns.
• shutdown_signal_test.go (+66 lines): 3 new tests — idempotent signal, wait-then-signal, wait-then-timeout.

kardianos still reports SERVICE_STOPPED to SCM as soon as prg.Stop returns, so the OS shutdown of other services proceeds in parallel. Only this process is held back, and only until its own teardown finishes — capped at runCompleteTimeout so a stuck teardown cannot indefinitely hold up the OS. The Windows-fallback os.Exit(0) path is unaffected: defer doesn't run on os.Exit, but the process terminates immediately anyway, which is the intended behaviour of that fallback.

Behaviour matrix:

Path	requestSCMStop returns	Signal-goroutine action	handleTerminatingSignal	Effect
SCM STOP (sc stop → prg.Stop → close(stop))	not called (<-stop branch of outer select)	cancel()	no-op (flag unset)	unchanged
SIGHUP reload	not called (SIGHUP branch)	reload, cancel()	no-op (flag unset)	unchanged
Non-Windows	false (compile-time no-op)	terminatingSignalReceived.Store(true); cancel()	no-op (build tag)	unchanged (same as pristine)
Windows interactive / console (windowsRunAsService()==false)	false	same as above	early-return	unchanged
Windows service + OS shutdown, SCM STOP delivered	true	dispatch → <-stop fires → cancel()	no-op (flag NOT set)	kardianos's normal STOP path drives svc.Run to return; main exits normally
Windows service + SCM path unavailable (Connect / ListServices / no match / OpenService / Control failed)	false	flag set, cancel()	os.Exit(0)	fallback, OS shutdown proceeds
Windows service + SCM accepted but prg.Stop did not fire within 30s	true then timeout	flag set at timeout, cancel()	os.Exit(0)	fallback, OS shutdown proceeds

No new module dependencies (golang.org/x/sys is already in go.mod; the workaround uses its windows/svc and windows/svc/mgr subpackages).

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

• Unit tests (new) — 22 tests in total, all passing.

  Cross-platform (shutdown_signal_test.go, 9 tests):

  • TestTerminatingSignalReceived_DefaultUnset / _SetLoad / _Race: package-level atomic.Bool semantics (Race launches 64 goroutines, safe under -race).
  • TestDispatch_SCMUnavailable_SetsFallbackFlag: requestSCMStopFn=false → fallback flag set.
  • TestDispatch_SCMSuccess_NoFallbackFlag: requestSCMStopFn=true + close(stop) before timeout → fallback flag NOT set.
  • TestDispatch_SCMAcceptedButTimeout_SetsFallbackFlag: requestSCMStopFn=true + timeout expires → fallback flag set.
  • TestSignalRunComplete_Idempotent: signalRunComplete uses sync.Once so a double invocation does not panic on a closed channel.
  • TestWaitRunComplete_ReturnsWhenSignaled: waitRunComplete returns promptly once signalRunComplete fires.
  • TestWaitRunComplete_TimesOut: waitRunComplete falls through after runCompleteTimeout when no signal arrives.

  Windows-only (shutdown_signal_windows_test.go, 13 tests) using fake scmManager / scmService + seams:

  • TestFindOwnSCMServiceName_MatchByPID / _NoMatch / _ListServicesError / _SkipsOpenErrors / _SkipsQueryErrors: enumeration finds the entry by ProcessId; robust against per-service ACCESS_DENIED and Query errors; every opened service is Close()d.
  • TestRequestSCMStop_NotAService / _ConnectFail / _NoMatchingService / _ControlSTOPFail / _Success: each failure mode returns false and cleans up (Disconnect, Close); the success path issues winsvc.Stop.
  • TestHandleTerminatingSignal_FlagUnset_NoExit / _NotAService_NoExit / _FlagSetAndService_Exits: exitFunc(0) is called only when both preconditions hold.

  Result on the validation host (go test -v -count=1 ./cmd/amazon-cloudwatch-agent -run '^(TestTerminatingSignalReceived|TestDispatch|TestSignalRunComplete|TestWaitRunComplete|TestFindOwnSCMServiceName|TestRequestSCMStop|TestHandleTerminatingSignal)'):

  --- PASS: TestTerminatingSignalReceived_DefaultUnset (0.00s)
  --- PASS: TestTerminatingSignalReceived_SetLoad (0.00s)
  --- PASS: TestTerminatingSignalReceived_Race (0.00s)
  --- PASS: TestDispatch_SCMUnavailable_SetsFallbackFlag (0.00s)
  --- PASS: TestDispatch_SCMSuccess_NoFallbackFlag (0.01s)
  --- PASS: TestDispatch_SCMAcceptedButTimeout_SetsFallbackFlag (0.02s)
  --- PASS: TestSignalRunComplete_Idempotent (0.00s)
  --- PASS: TestWaitRunComplete_ReturnsWhenSignaled (0.01s)
  --- PASS: TestWaitRunComplete_TimesOut (0.02s)
  --- PASS: TestFindOwnSCMServiceName_MatchByPID (0.00s)
  --- PASS: TestFindOwnSCMServiceName_NoMatch (0.00s)
  --- PASS: TestFindOwnSCMServiceName_ListServicesError (0.00s)
  --- PASS: TestFindOwnSCMServiceName_SkipsOpenErrors (0.00s)
  --- PASS: TestFindOwnSCMServiceName_SkipsQueryErrors (0.00s)
  --- PASS: TestRequestSCMStop_NotAService (0.00s)
  --- PASS: TestRequestSCMStop_ConnectFail (0.00s)
  --- PASS: TestRequestSCMStop_NoMatchingService (0.00s)
  --- PASS: TestRequestSCMStop_ControlSTOPFail (0.00s)
  --- PASS: TestRequestSCMStop_Success (0.00s)
  --- PASS: TestHandleTerminatingSignal_FlagUnset_NoExit (0.00s)
  --- PASS: TestHandleTerminatingSignal_NotAService_NoExit (0.00s)
  --- PASS: TestHandleTerminatingSignal_FlagSetAndService_Exits (0.00s)
  PASS
  ok  github.com/aws/amazon-cloudwatch-agent/cmd/amazon-cloudwatch-agent  2.477s
  

• go test -race — same 22 tests, executed with CGO_ENABLED=1 and gcc from MinGW-w64 14.2.0 for the race detector. All 22 PASS, no data races reported.

• Static checks — executed against a fresh clone of main + this PR's commit on Windows Server 2022 with Go 1.25.8. Tools installed via the same go install / release-download commands the Makefile uses.

  • go fmt ./... — clean.
  • goimports -l -local github.com/aws/amazon-cloudwatch-agent on the six touched/new files — all clean.
  • impi --local github.com/aws/amazon-cloudwatch-agent --scheme stdThirdPartyLocal on the six files — all clean.
  • addlicense -check -l MIT -c "Amazon.com, Inc. or its affiliates." on the six files — all rc=0.
  • go vet ./cmd/amazon-cloudwatch-agent reports one finding at line 120 (sigchanyzer: misuse of unbuffered os.Signal channel as argument to signal.Notify, from signals := make(chan os.Signal)). This finding is present on main at HEAD~1 as well and is unrelated to this PR.
  • golangci-lint run --timeout 5m ./cmd/amazon-cloudwatch-agent (v2.12.2 — the Makefile version) reports 7 findings on this PR and 7 findings on main at HEAD~2. Compare-Object between the two runs: NEW findings: 0, REMOVED findings: 0. The three lines that appear in the raw diff (G706 269→273, G114 523→530, revive superfluous-else 635→642) are the same rules on the same statements, shifted downward by the lines these two commits add above each finding (+4, +7, +7 respectively; total +10 lines added to amazon-cloudwatch-agent.go). All findings are on code paths this PR does not touch.

• End-to-end (Windows Server 2022, EC2 m5.xlarge) — same host used for the reproduction, with WaitToKillServiceTimeout=300000 to expose the deadlock. Wrapper left pristine; only the collector rebuilt.

  Five scenarios were run, isolating the effect of each commit:

  #	Collector build	otelcol.Shutdown	"Shutdown complete." logged	OS shutdown
  1	Pristine main	<1 s	YES	290 s → 6008 + Kernel-Power 41
  2	main + a6a1a93	<1 s	YES	48 s, no 6008/41
  3	main + a6a1a93 + 10 s sleep injected inside Service.Shutdown (validation only, not part of PR)	10 s	NO — truncated	78 s, no 6008/41
  4	main + a6a1a93 + 15cb73d + the same 10 s sleep injected inside Service.Shutdown	10 s	YES — full sequence preserved	78 s, no 6008/41
  5	main + a6a1a93 + 15cb73d (this PR head)	<1 s	YES	33 s, no 6008/41

  Rows 3 → 4 form the direct A/B: same teardown-slowing patch, only difference is the wait mechanism from commit 15cb73d. In row 3 the process was terminated during otelcol.Shutdown (Event 7036 recorded the CWA "Stopped" transition at the same millisecond the sleep started); in row 4 main blocked in waitRunComplete until the full teardown sequence completed and only then returned.

  Row 5 confirms the wait mechanism adds no observable overhead in the normal case: every teardown log entry lands in the same second, SCM records the "Stopped" transition ~ms later, and total OS shutdown is well within the same range as row 2 (both are within observed EC2 platform variance for this instance).

  Agent log around shutdown in scenario 4:

  ... "Received signal from OS","signal":"terminated"
  ... "Starting shutdown..."
  ... "[RACETEST] injected sleep 10s starts"
  ... Windows service: SCM Control(Stop) accepted on "AmazonCloudWatchAgent"; waiting for prg.Stop
  ... Profiler is stopped during shutdown
  ... [10 seconds elapse]
  ... "[RACETEST] injected sleep 10s done"
  ... "Stopping extensions..."
  ... "Pod to Service Environment Mapping TTL Cache stopped"
  ... "Shutdown complete."
  

  In scenario 3 (same test without the wait) everything after "SCM Control(Stop) accepted" was cut off.

  Agent log around shutdown in scenario 5 (production representative — both commits, no injected delay):

  ... "Received signal from OS","signal":"terminated"
  ... "Starting shutdown..."
  ... "Stopping extensions..."
  ... "Pod to Service Environment Mapping TTL Cache stopped"
  ... "Shutdown complete."
  ... Windows service: SCM Control(Stop) accepted on "AmazonCloudWatchAgent"; waiting for prg.Stop
  ... Profiler is stopped during shutdown
  

  System event log in scenarios 2/4/5: 1074 (shutdown initiated) → 6006 (EventLog stopped normally) → 7036 (AmazonCloudWatchAgent transitioned to Stopped). No 6008, no 41. sc stop unchanged (~0.3 s).

Requirements

Before commiting your code, please do the following steps.

1. Run make fmt and make fmt-sh
2. Run make lint

---

Integration Tests

To run integration tests against this PR, add the ready for testing label.

[ablankz]

FYI: PR #2215 was revised to route the terminating signal through an SCM SERVICE_CONTROL_STOP on the collector's own service entry (discovered by matching os.Getpid() against SCM ProcessId), rather than calling os.Exit(0) directly. The result is that kardianos' normal STOP path (prg.Stop → close(stop)) drives svc.Run to return cleanly, and OS shutdown proceeds via the same code path as sc stop. os.Exit(0) is retained only as a fallback when the SCM path is unavailable (Connect/ListServices/OpenService/Control fails, or prg.Stop does not fire within 30 s).
E2E on Windows Server 2022 (WaitToKillServiceTimeout=300000): OS shutdown completes in 48 s with Event ID 6006 (no 6008/41). Agent log confirms the SCM path executed via Windows service: SCM Control(Stop) accepted on "AmazonCloudWatchAgent"; waiting for prg.Stop.

[ablankz]

Follow-up (commit 6359247): added a done-channel so main waits for (*program).run to finish before returning. Closes a race where svc.Run returns as soon as SCM STOP is processed, letting main reach ExitProcess before otelcol.Shutdown (running on a parallel goroutine from otelcol's own SIGTERM handler) can complete — the final "Shutdown complete." log line and any pending log/metric flushes would be dropped silently.

(*program).run defers signalRunComplete on every return path, and main's Windows service branch blocks on waitRunComplete after s.Run returns. Wait bounded at 30 s so a stuck teardown cannot indefinitely hold up the OS. kardianos still reports SERVICE_STOPPED to SCM as soon as prg.Stop returns, so the OS shutdown of other services proceeds in parallel.

Validated on Windows Server 2022 by injecting a 10 s sleep inside otelcol.Service.Shutdown:

• Without the wait (52bfdca only): the process was terminated during the sleep — Event 7036 recorded the CWA "Stopped" transition at the same millisecond the sleep started, and "Shutdown complete." was never logged.
• With the wait (52bfdca + 6359247): the full teardown sequence including "Shutdown complete." was preserved; total OS shutdown +10 s over baseline (all attributable to the injected delay).

[github-actions]

This PR was marked stale due to lack of activity.