Fail in-flight invocations when worker channel shuts down

[satvu]

Issue describing the changes in this PR

resolves #10936

Pull request checklist

IMPORTANT: Currently, changes must be backported to the in-proc branch to be included in Core Tools and non-Flex deployments.

• Backporting to the in-proc branch is not required
  • Otherwise: Link to backporting PR
• My changes do not require documentation changes
  • Otherwise: Documentation issue linked to PR
• My changes should not be added to the release notes for the next release
  • Otherwise: I've added my notes to release_notes.md
• My changes do not need to be backported to a previous version
  • Otherwise: Backport tracked by issue/PR #issue_or_pr
• My changes do not require diagnostic events changes
  • Otherwise: I have added/updated all related diagnostic events and their documentation (Documentation issue linked to PR)
• I have added all required tests (Unit tests, E2E tests)

Additional information

Additional PR information

[satvu]

Prior to this change, calling TryFailExecutions (replaced by Shutdown) would not fail executions without an exception being passed in. This test makes sure that in-flight invocations are failed whether or not an exception is passed in.

[Copilot]

Pull Request Overview

This PR modifies the worker channel shutdown behavior to better handle in-flight invocations when a worker channel encounters a fatal error. The key change is replacing the TryFailExecutions method with a more descriptive Shutdown method that properly fails in-flight invocations with a specific exception type.

Key changes:

• Replace TryFailExecutions method with Shutdown method across all worker channel implementations
• Introduce WorkerShutdownException to provide more specific error information for failed invocations
• Update HTTP proxy service to properly handle worker shutdown scenarios during request retries

Reviewed Changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/WebJobs.Script/Workers/Rpc/IRpcWorkerChannel.cs	Updates interface to replace TryFailExecutions with Shutdown method
src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs	Implements new Shutdown method with WorkerShutdownException
src/WebJobs.Script/Exceptions/WorkerShutdownException.cs	Adds new exception type for worker shutdown scenarios
src/WebJobs.Script/Http/RetryProxyHandler.cs	Updates HTTP retry logic to handle worker shutdown exceptions
src/WebJobs.Script/Http/ScriptInvocationRequestTransformer.cs	Adds transformer to pass script invocation context through HTTP proxy
src/WebJobs.Script/Http/DefaultHttpProxyService.cs	Integrates new transformer and adds invocation context to HTTP requests

[Copilot]

This check is performed on every retry attempt inside the loop. Consider moving this check outside the retry loop or caching the Task.IsFaulted result to avoid repeated property access.

Suggested change

	for (int attemptCount = 1; attemptCount <= MaxRetries; attemptCount++)
	{
	try
	{
	if (resultSource is not null && resultSource.Task.IsFaulted)
	var isTaskFaulted = resultSource?.Task.IsFaulted ?? false;
	for (int attemptCount = 1; attemptCount <= MaxRetries; attemptCount++)
	{
	try
	{
	if (isTaskFaulted)

[satvu]

This needs to be checked before every retry.

[kshyju]

A thought: Should Shutdown also dispose this instance? The (worker) process is removed/killed in Dispose method code, and I see the callers of Shutdown is explicitly calling Dispose after calling Shutdown. Wondering that should be part of Shutdown method itself.