api/nodes reports incorrect limit of filedescriptors

[rstrlcpy]

Describe the bug

# http GET http://......./api/nodes
        .....
        "fd_total": 1024,     <<<<<<<<<<
       ....
        "fd_used": 4070,     <<<<<<<<<<<
        .....

# rabbitmqctl eval 'erlang:system_info(check_io).'
[[{name,erts_poll},        
  {primary,select},               <<<<<<<<<<<<<<<<<<<
  {kernel_poll,false},
  {memory_size,13448},  
  {total_poll_set_size,1},  
  {lazy_updates,true},
  {pending_updates,0},
  {batch_updates,false},
  {concurrent_updates,false},
  {fallback,true},                      <<<<<<<<<<<<<<<<<<<
  {max_fds,1024},                    <<<<<<<<<<<<<<<<<<<
  {active_fds,0},
  {poll_threads,0}],
 [{name,erts_poll},
  {primary,epoll},
  {kernel_poll,epoll},
  {memory_size,56},
  {total_poll_set_size,7},
  {lazy_updates,false},
  {pending_updates,0},
  {batch_updates,false},
  {concurrent_updates,true},
  {fallback,false},
  {max_fds,1048576},
  {active_fds,0},
  {poll_threads,0}],
 [{name,erts_poll},
  {primary,epoll},
  {kernel_poll,epoll},
  {memory_size,458808},
  {total_poll_set_size,4039},      <<<<<<<<<<<<<<<<<<<
  {lazy_updates,false},
  {pending_updates,0},
  {batch_updates,false},
  {concurrent_updates,true},
  {fallback,false},
  {max_fds,1048576},               <<<<<<<<<<<<<<<<<<<
  {active_fds,0},
  {poll_threads,1}]]

WebUI: File descriptors on Overview is red. That confuses.

Reproduction steps

Deploy a test server using rabbitmq:4.0.4-management-alpine (official docker-image).
Run test using rabbitmq-perf-test:

java -jar perf-test-latest.jar --uri amqp://rabbitmq-for-test --rate 1 --producers 4000 --consumers 1

Expected behavior

It would be better to show the numbers from the hot-path pollset not from fallback.

Additional context

4.0.3 and 4.0.2 are not affected because they were build somehow differently:
docker-library/rabbitmq#749

[michaelklishin]

@rstrlcpy your definition of "incorrect" assumes we use erlang:system_info/1. We don't. Those metrics used to come from a subsystem called file_handle_cache which was nearly completely removed from modern RabbitMQ.

See rabbit_mgmt_external_stats

rabbitmq-server/deps/rabbitmq_management_agent/src/rabbit_mgmt_external_stats.erl

Line 356 in f413880

State = #state{fd_total = file_handle_cache:ulimit(),

.

This metric has a chance of following its socket count counterparts that were completely removed because this is a kernel-level metric and should be monitored as such, that is, not by RabbitMQ itself.

[KianAnbarestani]

The bug arises because RabbitMQ 4.0.4 now reports fallback pollset metrics (with a low max_fds of 1024) in the HTTP API and WebUI, instead of using the hot-path pollset values. This causes file descriptor usage to appear red/confusing even when the actual FD capacity is high. Earlier versions (4.0.2, 4.0.3) used a different build that exposed the correct values. This mismatch was noted in docker-library/rabbitmq#749 and is misleading for users monitoring FD usage.

[michaelklishin]

@KianAnbarestani you are welcome to submit a PR that would do the right thing. Thank you.

[KianAnbarestani]

Thanks, @michaelklishin ! I’ll go ahead and prepare a PR to address the fallback metrics issue and ensure the correct pollset values are exposed. I’ll reference the behavior differences across versions and the context from #749 in the PR as well. Appreciate the opportunity!

[mkuratczyk]

I'm pretty sure this issue has been solved already, I just don't remember how (an Erlang/OTP change?). I think @lukebakken found this