[QUERY] Processor stop flow and time constraints.

[engenb]

Library name and version

Azure.Messaging.ServiceBus 7.17.0

Query/Question

This question pertains to application lifecycle and message processing. I'll do my best to lay out some context.

I have an application with an api that receives requests to run background jobs. the api validates the request, adds some system metadata, and queues the job via an azure ServiceBus queue.

a BackgroundService / IHostedService subscribes to this queue, pulls messages off, and processes them via a ServiceBusProcessor.

this application is hosted in Azure Container Apps.

when the application undergoes a scaling operation, particularly a scale-in, my application receives a request (SIGTERM) to gracefully shut down. as a result, StopAsync is invoked on my IHostedService. it's here that expected behavior gets a little murky for me. The cancellation token provided as an argument to StopAsync represents (via docs) when graceful shutdown is canceled and will now be forced. I would expect this to occur when the shutdown timeout has elapsed. I'm aware there are a few layers here. aspnetcore's generic host has its ShutdownTimeout and ACA has one as well. fwiw I've cranked both of these to an hour (ACA max) while troubleshooting this.

during BackgroundService.StopAsync, I indicate the ServiceBusProcessor should stop processing messages:

public override async Task StopAsync(CancellationToken cancellationToken)
{
    await _processor.StopProcessingAsync(cancellationToken);
}

according to the docs for StopProcessingAsync - it will stop receiving new messages but allow messages currently being processed to complete - as long as they complete within message TryTimeout. this makes sense, except that's not the behavior I'm seeing.

what I am currently observing is, when StopProcessingAsync is invoked, cancellation is requested of any messages currently being processed. am I just misunderstanding the docs? I would expect cancellation would not be requested of message processors until cancellation was requested via the token provided by StopAsync indicating we are no longer gracefully shutting down.

I'd be happy to provide a console app to reproduce this if that'd help illustrate the point. a simple console app with a "long" Task.Delay(...) in ProcessMessageAsync will do the trick.

I'm fully ready to concede that I'm misunderstanding something. or, maybe there really is a bug here. either way, thanks for taking a look!

Environment

No response

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[JoshLove-msft]

what I am currently observing is, when StopProcessingAsync is invoked, cancellation is requested of any messages currently being processed. am I just misunderstanding the docs

This is correct - the token is signaled so that message handlers have the option to either honor the token for any processing they are doing or ignore it.

[jsquire]

Hi @engenb. Thanks for reaching out and we regret that you're experiencing difficulties. As Josh mentioned, when you request that the processor stop, it will signal the token passed to your handler in the ProcessMessageEventArgs instance (those docs discuss the behavior). You application is free to honor that token or ignore it, as makes sense for your scenario.

The processor itself has no limit on the length of time that it will wait for processing to complete. The TryTimeout does not govern the end-to-end stop operation - only the service requests made. For example, your logic could take 30 minutes to process a message without issue - but when you make a call to complete the message, that call must finish within the TryTimeout period, or the underlying transport treats it as a failed request and the implicit retry logic is applied.

In a hosted service, the bigger consideration is - as you noted - the shutdown timeout from the Background Service and that of the host environment itself. So long as you've got those configured to give you the time that you need to finish processing, there's nothing special that you'd need to do for the processor itself.

[github-actions]

Hi @engenb. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[engenb]

thank you @JoshLove-msft and @jsquire - that was very helpful - I appreciate your concise answers and added context!

[github-actions]

Hi @engenb, since you haven’t asked that we /unresolve the issue, we’ll close this out. If you believe further discussion is needed, please add a comment /unresolve to reopen the issue.