ShutdownSignal not sent to consumers when broker blocks publisher and channel is closing

[siewp]

Hello there,

when the broker has insufficient disk space and is blocking publishers, a publisher confirm may reach the timeout and the channel closes. However, a consumer on the same channel never gets the ShutdownSignal in this case because this line is never executed. Besides Publisher Confirm, this also applies if you call channel.abort() when the broker is blocking. The ShutdownSignal is not broadcasted to consumers. Is that behaviour intended?

I'm asking because we are using the vertx-rabbitmq-client which uses only one channel per connection and restores a consumer when a ShutdownSignal is broadcasted which is not working when the case described above happens.

Thank you!

[michaelklishin]

A blocked publishing connection in and of itself won't be closed. It can be closed after a period of time due to missed heartbeats but in modern RabbitMQ series, the server will pause heartbeat tracking for such connections. The client, however, won't IIRC.

In any case, the shutdown signals is not what you are looking for. There is a separate notification that this client has supported since day one of that protocol extension, for over a decade.

[siewp]

Thanks for your quick response. You are right, channels of a publishing connections won't be closed in general, but when Publisher Confirms are used in combination with waitForConfirmsOrDie() that may happen.
Anyways, this seems to be more an issue of the Vert.x RabbitMQ Client. So, I will open an issue there.

[acogoluegnes]

@siewp I added a test that confirms the channel shutdown listener is called even if the close method times out. It also confirms the consumer shutdown listener is not called, because the line your pointed out does not execute. This should happen rarely, but it is worth fixing.

I'm not sure which parts belong to your application or to the Vert.x client but I'd recommend:

• using different connections for consumers and publishers with AMQP 0.9.1.
• avoiding the usage of Channel#waitForConfirmOrDie altogether in favor of the asynchronous listener (strategy 3 in the publish confirm tutorial).

[acogoluegnes]

Follow-up fix: #1656.

[ivu-mawi]

Hey, thanks so much for the lightning fast fix!

I have a local reproducer where I set the Low Disk Space limit of the broker and then publish a message with Publisher Confirms. I'm able to reproduce this issue very reliably.
With 5.26, I can no longer reproduce it (finishProcessShutdownSignal is now called, through the finally block).

Could I ask you for a guess as to whether this fix also impacts/improves clients that don't use publisher confirms? For example, is there any other situation where the client closes the channel (I assume this is what triggers this code path) which since 5.26 also correctly sends the shutdown signal? I'm asking because we have also (much more rarely) observed the missing consumer re-creation in other systems that do not use publisher confirms. (All while the low disk space warning of our broker is triggered intermittently, fairly regularly).

[acogoluegnes]

It could happen whenever Channel#close() or one of its variants is called and the client does not get a response from the broker. An example would be an explicit close() call when the connection is blocked, so no need to use publisher confirms.

Thanks for testing before/after the fix and confirming it works.