Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: 开启健康检测的upstream,有流量的情况下调整权重会导致错误日志产生,并有概率导致500错误 #11897

Open
Lewisyixin opened this issue Jan 7, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@Lewisyixin
Copy link

Current Behavior

When modify nodes weight, error log will be recorded " all upstream nodes is unhealthy"

Expected Behavior

no error log

Error Logs

image

2025/01/07 18:04:23 [error] 6714#6714: *276732 [lua] balancer.lua:83: fetch_health_nodes(): failed to get health check target status, addr: 10.58.94.168:80, host: nil, err: target not found, client: 127.0.0.1, server: _, request: "GET /l HTTP/1.1", host: "l.com" 2025/01/07 18:04:23 [error] 6714#6714: *276732 [lua] balancer.lua:83: fetch_health_nodes(): failed to get health check target status, addr: 10.58.32.145:80, host: nil, err: target not found, client: 127.0.0.1, server: _, request: "GET /l HTTP/1.1", host: "l.com" 2025/01/07 18:04:23 [warn] 6714#6714: *276732 [lua] balancer.lua:89: fetch_health_nodes(): all upstream nodes is unhealthy, use default, client: 127.0.0.1, server: _, request: "GET /l HTTP/1.1", host: "l.com" 2025/01/07 18:04:24 [error] 6716#6716: *276683 [lua] balancer.lua:83: fetch_health_nodes(): failed to get health check target status, addr: 10.58.32.145:80, host: nil, err: target not found, client: 127.0.0.1, server: _, request: "GET /l HTTP/1.1", host: "l.com" 2025/01/07 18:04:24 [error] 6716#6716: *276683 [lua] balancer.lua:83: fetch_health_nodes(): failed to get health check target status, addr: 10.58.94.168:80, host: nil, err: target not found, client: 127.0.0.1, server: _, request: "GET /l HTTP/1.1", host: "l.com" 2025/01/07 18:04:24 [warn] 6716#6716: *276683 [lua] balancer.lua:89: fetch_health_nodes(): all upstream nodes is unhealthy, use default, client: 127.0.0.1, server: _, request: "GET /l HTTP/1.1", host: "l.com"

Steps to Reproduce

  1. create a route and upstream with 2 or more nodes. Upstream need turn on healthcheck
    image
    image
  2. run a wrk command to make requests to the server continuously.
    image
  3. change node weight continuously(such as keep executing the following command)
    curl -XPATCH -s -H "x-api-key: $token" http://192.168.20.226:9180/apisix/admin/upstreams/547982406942985081 -d '{"nodes":{"10.58.32.145:80": 20}}'
  4. and we will get error log
    image
    5.In extreme cases, client will get a 500 internal error(but this cannot be reproduced stably)
    image

Environment

  • APISIX version (run apisix version): 3.5
  • Operating system (run uname -a): 3.10.0-514.26.2.el7.x86_64
  • OpenResty / Nginx version (run openresty -V or nginx -V): openresty/1.21.4.2
  • etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 3.5
  • APISIX Dashboard version, if relevant:
  • Plugin runner version, for issues related to plugin runners:
  • LuaRocks version, for installation issues (run luarocks --version): 2.3.0
@github-project-automation github-project-automation bot moved this to 📋 Backlog in Apache APISIX backlog Jan 7, 2025
@dosubot dosubot bot added the bug Something isn't working label Jan 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: 📋 Backlog
Development

No branches or pull requests

1 participant