Unable to upgrade from 3.12.13 (erlang 26.2.5) to 3.13.3 - incompatible_feature_flags #11441

pmcevoy · 2024-06-12T16:21:24Z

pmcevoy
Jun 12, 2024

I'm trying to do a rolling upgrade of a 3.12.13 cluster to 3.13.3 (both with erlang 26.2.5) but new 3.13.3 nodes are failing to join with the error incompatible_feature_flags

My existing 3 node cluster has the following versions with all flags enabled

Running Nodes

[email protected]
[email protected]
[email protected]

Versions

[email protected]: RabbitMQ 3.12.14 on Erlang 26.2.5
[email protected]: RabbitMQ 3.12.14 on Erlang 26.2.5
[email protected]: RabbitMQ 3.12.14 on Erlang 26.2.5

... 

Feature flags

Flag: classic_mirrored_queue_version, state: enabled
Flag: classic_queue_type_delivery_support, state: enabled
Flag: direct_exchange_routing_v2, state: enabled
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: feature_flags_v2, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: listener_records_in_ets, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: restart_streams, state: enabled
Flag: stream_queue, state: enabled
Flag: stream_sac_coordinator_unblock_group, state: enabled
Flag: stream_single_active_consumer, state: enabled
Flag: tracking_records_in_ets, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled

Note that as per recommendation, all feature flags are enabled.

The new 3.13.3 node has the following feature flags out-of-the-box from the RPM from github download - I did nothing to enable extra features.

Running Nodes

[email protected]

Versions

[email protected]: RabbitMQ 3.13.3 on Erlang 26.2.5
...
Feature flags

Flag: classic_mirrored_queue_version, state: enabled
Flag: classic_queue_type_delivery_support, state: enabled
Flag: detailed_queues_endpoint, state: enabled
Flag: direct_exchange_routing_v2, state: enabled
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: feature_flags_v2, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: khepri_db, state: disabled
Flag: listener_records_in_ets, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: message_containers, state: enabled
Flag: message_containers_deaths_v2, state: enabled
Flag: quorum_queue, state: enabled
Flag: quorum_queue_non_voters, state: enabled
Flag: restart_streams, state: enabled
Flag: stream_filtering, state: enabled
Flag: stream_queue, state: enabled
Flag: stream_sac_coordinator_unblock_group, state: enabled
Flag: stream_single_active_consumer, state: enabled
Flag: stream_update_config_command, state: enabled
Flag: tracking_records_in_ets, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled

Relevant rabbitmq log on the node failing to join:

2024-06-12 16:10:44.514418+00:00 [info] <0.255.0> Peer discovery: will try to unlock with peer discovery backend rabbit_peer_discovery_aws
2024-06-12 16:10:46.006409+00:00 [info] <0.255.0> Peer discovery: node '[email protected]' selected for auto-clustering
2024-06-12 16:10:46.006498+00:00 [info] <0.255.0> Peer discovery: will try to lock with peer discovery backend rabbit_peer_discovery_aws
2024-06-12 16:10:46.006581+00:00 [info] <0.255.0> Will try to lock connecting to nodes ['[email protected]',
2024-06-12 16:10:46.006581+00:00 [info] <0.255.0>                                       '[email protected]']
2024-06-12 16:10:46.007416+00:00 [info] <0.255.0> DB: checking if `[email protected]` can join cluster using remote node `[email protected]`
2024-06-12 16:10:46.007493+00:00 [notice] <0.255.0> Feature flags: checking nodes `[email protected]` and `[email protected]` compatibility...
2024-06-12 16:10:46.012214+00:00 [warning] <0.255.0> Feature flags: nodes `[email protected]` and `[email protected]` are incompatible
2024-06-12 16:10:46.012312+00:00 [warning] <0.255.0> Peer discovery: could not auto-cluster with node '[email protected]': {error,incompatible_feature_flags}

I'm at a loss how to proceed or resolve. I have successfully managed to incrementally upgrade all the way from rabbit 3.8.2/erlang 22.2.7 to rabbit 3.12.14/erlang 26.2.5 without (much) issues, but this last jump has me baffled.

Sincerely
Pete

Answered by lukebakken

Dec 7, 2024

I verified that upgrading from 3.12.13 to the latest 3.x RabbitMQ works correctly:

https://github.com/lukebakken/rabbitmq-server-11441

See upgrade.sh for steps I used to upgrade this docker compose environment.

To see it in action:

direnv allow # optional
make up
./upgrade.sh

View full answer

lukebakken · 2024-12-07T16:56:20Z

lukebakken
Dec 7, 2024
Maintainer

I verified that upgrading from 3.12.13 to the latest 3.x RabbitMQ works correctly:

https://github.com/lukebakken/rabbitmq-server-11441

See upgrade.sh for steps I used to upgrade this docker compose environment.

To see it in action:

direnv allow # optional
make up
./upgrade.sh

0 replies

malsssh · 2025-03-13T15:02:57Z

malsssh
Mar 13, 2025

@pmcevoy,wondering how this ended ?

@lukebakken, I executed the same commands you used in the script and got an error. However, our environments differ - you used Docker while i am on RHEL with RPM. Perhaps there are some leftover files on the server that are missing in the new images which is why it worked for you.

rabbit@rabbit-01: RabbitMQ 3.12.14 on Erlang 26.2.5.5
rabbit@rabbit-02: RabbitMQ 3.12.14 on Erlang 26.2.5.5
rabbit@rabbit-03: RabbitMQ 3.12.14 on Erlang 26.2.5.5

Feature flags

Flag: classic_mirrored_queue_version, state: enabled
Flag: classic_queue_type_delivery_support, state: enabled
Flag: direct_exchange_routing_v2, state: enabled
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: feature_flags_v2, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: listener_records_in_ets, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: restart_streams, state: enabled
Flag: stream_queue, state: enabled
Flag: stream_sac_coordinator_unblock_group, state: enabled
Flag: stream_single_active_consumer, state: enabled
Flag: tracking_records_in_ets, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled

I did:
rabbitmqctl enable_feature_flag all
rabbitmq-upgrade drain
dnf install rabbitmq-server-0:3.13.7-1.el8.noarch erlang-0:26.2.5.8-1.el9.x86_64

After that it crashed.

Feature flags: nodes rabbit@rabbit-01 and rabbit@rabbit-02 are incompatible
Feature flags: nodes rabbit@rabbit-01 and rabbit@rabbit-03 are incompatible
Error during startup: {error,incompatible_feature_flags}

There is a difference in the flags on upgraded node. There are additional ones that are not present on other nodes:

classic_mirrored_queue_version
detailed_queues_endpoint
khepri_db
message_containers
message_containers_deaths_v2
quorum_queue_non_voters
stream_filtering
stream_update_config_command

7 replies

lukebakken Mar 13, 2025
Maintainer

Perhaps there are some leftover files on the server that are missing in the new images

My project uses a mounted volume for the data, so the data persists between version upgrades.

If I have time, I'll try out starting from 3.11 to see if that's the cause.

lukebakken Mar 13, 2025
Maintainer

OK, I modified my project to start at 3.11, and still don't see issues. I'll keep trying "farther back" starting points to see if I can trigger this issue.

malsssh Mar 13, 2025

https://groups.google.com/g/rabbitmq-users/c/nUPlVZYMGqI

Here's a similar case, he's upgrading with RPM's from 3.7.

malsssh Mar 13, 2025

I managed to upgrade the node. I removed node from cluster, deleted mnesia dir, added var RABBITMQ_FEATURE_FLAGS containing flags from version 3.12.14

pmcevoy Mar 18, 2025
Author

@malsssh Apologies for delayed reply.

In the end, I did not proceed past 3.12.13 - we are actively migrating away from RabbitMQ in our applications (to AWS SQS/SNS) and given we were on 3.8 for several years past it's sell-by date, we can live with being on 3.12.13 until we no longer need rabbit.

However, to be clear, my intial methodology to establish the version upgrade path from 3.8 to 3.12 was starting from a clean, empty cluster with no existing queues or resources present or even active clients. I did this in a clean-room AWS account, and repeatedly restarted from scratch which was possible as all VM installs were scripted in to UserData scripts part of Launch Templates.

My process was grow-then-shrink upgrade as we could not afford any downtime on the cluster - and grow/shrink reflects our attitude to OS patching as well. In each upgrade step, I only every updated Rabbit or Erlang - never both. There was one step where I also needed to upgrade from Amazon Linux 2 to Amazon Linux 2023.

After every component upgrade (whether Erlang or Rabbit), once all the old cluster nodes were removed and the new ones running, I ensured to enable all feature flags. However as documented above, I could not enable feature flags to 3.13.3 (which at evaluation time was the latest 3.13 available).

I'm happy to share our scripts and version hopping info if required

Pete

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to upgrade from 3.12.13 (erlang 26.2.5) to 3.13.3 - incompatible_feature_flags #11441

{{title}}

Replies: 2 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Unable to upgrade from 3.12.13 (erlang 26.2.5) to 3.13.3 - incompatible_feature_flags #11441

pmcevoy Jun 12, 2024

Replies: 2 comments · 7 replies

lukebakken Dec 7, 2024 Maintainer

malsssh Mar 13, 2025

lukebakken Mar 13, 2025 Maintainer

lukebakken Mar 13, 2025 Maintainer

malsssh Mar 13, 2025

malsssh Mar 13, 2025

pmcevoy Mar 18, 2025 Author

pmcevoy
Jun 12, 2024

Replies: 2 comments 7 replies

lukebakken
Dec 7, 2024
Maintainer

malsssh
Mar 13, 2025

lukebakken Mar 13, 2025
Maintainer

lukebakken Mar 13, 2025
Maintainer

pmcevoy Mar 18, 2025
Author