Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bitnami/redis-cluster] nodes never ready if using REDIS_PASSWORD_FILE instead of REDIS_PASSWORD #35982

Open
byErikas opened this issue May 31, 2023 · 5 comments · Fixed by #35992
Assignees
Labels
on-hold Issues or Pull Requests with this label will never be considered stale redis-cluster tech-issues The user has a technical issue about an application

Comments

@byErikas
Copy link
Contributor

byErikas commented May 31, 2023

Name and Version

bitnami/redis-cluster:7.0

What architecture are you using?

amd64

What steps will reproduce the bug?

I'm trying to run a docker stack on a swarm enviroment for the redis cluster, this is how the docker-compose looks:

version: '3.8'

services:
## REDIS
  redis-node-0:
    image: bitnami/redis-cluster:7.0
    volumes:
      - volume-redis-0:/bitnami/redis/data
    environment:
      REDIS_PASSWORD_FILE: /run/secrets/redis_password
      REDIS_NODES: "redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"
    configs:
      - source: redis_conf
        target: /opt/bitnami/redis/mounted-etc/overrides.conf
    secrets:
      - redis_password
    networks:
      - network-stack

  redis-node-1:
    image: bitnami/redis-cluster:7.0
    volumes:
      - volume-redis-1:/bitnami/redis/data
    environment:
      REDIS_PASSWORD_FILE: /run/secrets/redis_password
      REDIS_NODES: "redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"
    configs:
      - source: redis_conf
        target: /opt/bitnami/redis/mounted-etc/overrides.conf
    secrets:
      - redis_password
    networks:
      - network-stack

  redis-node-2:
    image: bitnami/redis-cluster:7.0
    volumes:
      - volume-redis-2:/bitnami/redis/data
    environment:
      REDIS_PASSWORD_FILE: /run/secrets/redis_password
      REDIS_NODES: "redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"
    configs:
      - source: redis_conf
        target: /opt/bitnami/redis/mounted-etc/overrides.conf
    secrets:
      - redis_password
    networks:
      - network-stack

  redis-node-3:
    image: bitnami/redis-cluster:7.0
    volumes:
      - volume-redis-3:/bitnami/redis/data
    environment:
      REDIS_PASSWORD_FILE: /run/secrets/redis_password
      REDIS_NODES: "redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"
    configs:
      - source: redis_conf
        target: /opt/bitnami/redis/mounted-etc/overrides.conf
    secrets:
      - redis_password
    networks:
      - network-stack

  redis-node-4:
    image: bitnami/redis-cluster:7.0
    volumes:
      - volume-redis-4:/bitnami/redis/data
    environment:
      REDIS_PASSWORD_FILE: /run/secrets/redis_password
      REDIS_NODES: "redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"
    configs:
      - source: redis_conf
        target: /opt/bitnami/redis/mounted-etc/overrides.conf
    secrets:
      - redis_password
    networks:
      - network-stack

  redis-node-5:
    image: bitnami/redis-cluster:7.0
    volumes:
      - volume-redis-5:/bitnami/redis/data
    environment:
      REDIS_PASSWORD_FILE: /run/secrets/redis_password
      REDISCLI_AUTH_FILE: /run/secrets/redis_password
      REDIS_CLUSTER_CREATOR: "yes"
      REDIS_CLUSTER_REPLICAS: 1
      REDIS_CLUSTER_SLEEP_BEFORE_DNS_LOOKUP: 30
      REDIS_NODES: "redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"
      BITNAMI_DEBUG: 1
    configs:
      - source: redis_conf
        target: /opt/bitnami/redis/mounted-etc/overrides.conf
    secrets:
      - redis_password
    networks:
      - network-stack

## END OF SERVICES ----------------------
## CONFIGS / SECRETS / VOLUMES / NETWORKS
configs:
  redis_conf:
    external: true
    name: web_vast-redis_conf

secrets:
  redis_password:
    external: true
    name: web_vast-redis_password

volumes:
  volume-redis-0:
    name: web_vast-redis-0
  volume-redis-1:
    name: web_vast-redis-1
  volume-redis-2:
    name: web_vast-redis-2
  volume-redis-3:
    name: web_vast-redis-3
  volume-redis-4:
    name: web_vast-redis-4
  volume-redis-5:
    name: web_vast-redis-5

networks:
  network-stack:
    driver: overlay
    name: web_vast-local

I've been trying to follow the setup based on the README.md over here: https://github.com/bitnami/containers/blob/main/bitnami/redis-cluster/README.md, however I need the redis password to be a secret that gets mounted on the container for redis, and I didn't see anything related to being able to use files for variables in the readme, so I went to dig around, and found that there seems to be some support for the "_FILE" variables over in this piece of the initial code https://github.com/bitnami/containers/blob/main/bitnami/redis-cluster/7.0/debian-11/rootfs/opt/bitnami/scripts/redis-cluster-env.sh#L68, however this doesn't seem to work.

I know that the "_FILE" support isn't in the documentation, and is likely not supported, bus is there any way I could set it up to read the password from a docker secret?

What is the expected behavior?

I had hopes that the cluster would start and initiate the same way it does if you simply use REDIS_PASSWORD in the docker-compose, as it works without any issues when configured using that, however it doesn't.

What do you see instead?

What seems to happen is that either the nodes, or the cluster are never ready, looking at the logs for the container that I use to initialize the cluster I get this:

redis-cluster 10:23:12.17 

redis-cluster 10:23:12.17 Welcome to the Bitnami redis-cluster container

redis-cluster 10:23:12.18 Subscribe to project updates by watching https://github.com/bitnami/containers

redis-cluster 10:23:12.18 Submit issues and feature requests at https://github.com/bitnami/containers/issues

redis-cluster 10:23:12.18 

redis-cluster 10:23:12.18 INFO  ==> ** Starting Redis setup **

redis-cluster 10:23:12.21 DEBUG ==> Validating settings in REDIS_* env vars..

redis-cluster 10:23:12.21 INFO  ==> Initializing Redis

redis-cluster 10:23:12.22 DEBUG ==> Ensuring expected directories/files exist

redis-cluster 10:23:12.24 INFO  ==> Setting Redis config file

Storing map with hostnames and IPs

redis-cluster 10:23:27.48 INFO  ==> ** Redis setup finished! **

140:C 31 May 2023 10:23:27.526 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

140:C 31 May 2023 10:23:27.526 # Redis version=7.0.11, bits=64, commit=00000000, modified=0, pid=140, just started

140:C 31 May 2023 10:23:27.526 # Configuration loaded

140:M 31 May 2023 10:23:27.527 * monotonic clock: POSIX clock_gettime

Node redis-node-0 not ready, waiting for all the nodes to be ready...

140:M 31 May 2023 10:23:27.528 * No cluster configuration found, I'm 2511621d9161c971e47c4317a5cea80ec8daed66

140:M 31 May 2023 10:23:27.530 * Running mode=cluster, port=6379.

140:M 31 May 2023 10:23:27.530 # Server initialized

140:M 31 May 2023 10:23:27.530 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

140:M 31 May 2023 10:23:27.531 * Creating AOF base file appendonly.aof.1.base.rdb on server start

140:M 31 May 2023 10:23:27.531 * Creating AOF incr file appendonly.aof.1.incr.aof on server start

140:M 31 May 2023 10:23:27.532 * Ready to accept connections

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

Node redis-node-0 not ready, waiting for all the nodes to be ready...

And it just keeps going forever, while if I inspect the node named redis-node-0 I see this:

redis-cluster 10:24:19.52 

redis-cluster 10:24:19.52 Welcome to the Bitnami redis-cluster container

redis-cluster 10:24:19.52 Subscribe to project updates by watching https://github.com/bitnami/containers

redis-cluster 10:24:19.53 Submit issues and feature requests at https://github.com/bitnami/containers/issues

redis-cluster 10:24:19.53 

redis-cluster 10:24:19.53 INFO  ==> ** Starting Redis setup **

redis-cluster 10:24:19.56 INFO  ==> Initializing Redis

redis-cluster 10:24:19.58 INFO  ==> Setting Redis config file

Changing old IP 10.0.82.21 by the new one 10.0.82.21

Changing old IP 10.0.82.13 by the new one 10.0.82.13

Changing old IP 10.0.82.17 by the new one 10.0.82.17

Changing old IP 10.0.82.19 by the new one 10.0.82.19

Changing old IP 10.0.82.23 by the new one 10.0.82.23

Changing old IP 10.0.82.9 by the new one 10.0.82.9

redis-cluster 10:24:19.77 INFO  ==> ** Redis setup finished! **

1:C 31 May 2023 10:24:19.813 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

1:C 31 May 2023 10:24:19.813 # Redis version=7.0.11, bits=64, commit=00000000, modified=0, pid=1, just started

1:C 31 May 2023 10:24:19.813 # Configuration loaded

1:M 31 May 2023 10:24:19.814 * monotonic clock: POSIX clock_gettime

1:M 31 May 2023 10:24:19.814 * Node configuration loaded, I'm 6ef7ffa00ff1632636efee1d17426d53eeacf111

1:M 31 May 2023 10:24:19.815 * Running mode=cluster, port=6379.

1:M 31 May 2023 10:24:19.815 # Server initialized

1:M 31 May 2023 10:24:19.815 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

1:M 31 May 2023 10:24:19.816 * Reading RDB base file on AOF loading...

1:M 31 May 2023 10:24:19.816 * Loading RDB produced by version 7.0.11

1:M 31 May 2023 10:24:19.816 * RDB age 53 seconds

1:M 31 May 2023 10:24:19.816 * RDB memory usage when created 1.51 Mb

1:M 31 May 2023 10:24:19.816 * RDB is base AOF

1:M 31 May 2023 10:24:19.816 * Done loading RDB, keys loaded: 0, keys expired: 0.

1:M 31 May 2023 10:24:19.816 * DB loaded from base file appendonly.aof.1.base.rdb: 0.000 seconds

1:M 31 May 2023 10:24:19.816 * DB loaded from append only file: 0.000 seconds

1:M 31 May 2023 10:24:19.816 * Opening AOF incr file appendonly.aof.1.incr.aof on server start

1:M 31 May 2023 10:24:19.816 * Ready to accept connections

Additional information

The only thing that is in my redis_conf that gets put into overrides.conf is this:

maxmemory-policy volatile-lru

so I don't think that this is a source of any issues

@byErikas byErikas added the tech-issues The user has a technical issue about an application label May 31, 2023
@github-actions github-actions bot added the triage Triage is needed label May 31, 2023
@byErikas
Copy link
Contributor Author

I've managed to narrow it down to a problem with the REDISCLI_AUTH variable, please take a look at #35992 for a fix that seems to work atleast for my enviroment.

@carrodher
Copy link
Member

Thanks for creating this issue and the associated PR. The team will review it and provide feedback. Once merged the PR, this issue will be automatically closed.

@joancafom
Copy link
Contributor

Hi @byErikas

I've recently reverted the changes introduced to support REDISCLI_AUTH from a file (see #39867) due to them causing some problems when redis was configured without a password.

I've created an internal task to evaluate the feasibility of this feature and the best way to tackle it, so we'll keep the thread updated when we have more news.

@joancafom joancafom reopened this Jul 4, 2023
@bitnami-bot bitnami-bot removed the solved label Jul 4, 2023
@joancafom joancafom added on-hold Issues or Pull Requests with this label will never be considered stale solved and removed triage Triage is needed labels Jul 4, 2023
@joancafom joancafom removed the solved label Jul 4, 2023
@github-actions github-actions bot added triage Triage is needed on-hold Issues or Pull Requests with this label will never be considered stale and removed on-hold Issues or Pull Requests with this label will never be considered stale triage Triage is needed labels Jul 4, 2023
@davama
Copy link

davama commented Jul 3, 2024

Good Day,

Dont mean to hijack this post but trying something similar to what the OP is doing.

Trying to understand how to set a password when ALLOW_EMPTY_PASSWORD=no (default).

Here's my compose file:

services:
  redis-base: &redis-base
    image: docker.io/bitnami/redis-cluster:latest
    environment:
      #ALLOW_EMPTY_PASSWORD: yes
      REDIS_MASTER_PASSWORD: myredispassword
      REDIS_PASSWORD: myredispassword
      REDIS_PASSWORD_FILE: /opt/redis/password.txt
      REDIS_NODES: redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5
    volumes:
      - ./pass:/opt/redis

  redis-node-0:
    <<: *redis-base
    container_name: redis-node-0

  redis-node-1:
    <<: *redis-base
    container_name: redis-node-1

  redis-node-2:
    <<: *redis-base
    container_name: redis-node-2

  redis-node-3:
    <<: *redis-base
    container_name: redis-node-3

  redis-node-4:
    <<: *redis-base
    container_name: redis-node-4

  redis-node-5:
    <<: *redis-base
    container_name: redis-node-5
    depends_on:
      - redis-node-0
      - redis-node-1
      - redis-node-2
      - redis-node-3
      - redis-node-4
    environment:
      #ALLOW_EMPTY_PASSWORD: yes
      REDIS_MASTER_PASSWORD: myredispassword
      REDIS_PASSWORD: myredispassword
      REDIS_PASSWORD_FILE: /opt/redis/password.txt
      REDIS_NODES: redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5
      REDIS_CLUSTER_REPLICAS: 1
      REDIS_CLUSTER_CREATOR: yes

redis-node-5 logs shows

Node redis-node-0 not ready, waiting for all the nodes to be ready...

And all nodes are role:master, nobody is a slave.

I have no issue when changing the env variables as below:

      ALLOW_EMPTY_PASSWORD: yes
      #REDIS_MASTER_PASSWORD: myredispassword
      #REDIS_PASSWORD: myredispassword
      #REDIS_PASSWORD_FILE: /opt/redis/password.txt

What am i missing?
Any input is appreciated!

Best,
Dave

@MarcusElevait
Copy link

MarcusElevait commented Oct 17, 2024

Any progress on this? Have the same problem with a local redis-cluster setup:

  redis:
    image: bitnami/redis-cluster:7.4
    environment:
      REDIS_NODES: redis redis-node-1 redis-node-2
      REDIS_CLUSTER_CREATOR: yes
      REDIS_CLUSTER_REPLICAS: 0
      REDIS_PASSWORD: "admin"
    ports:
      - "6379:6379"
    depends_on:
      - redis-node-1
      - redis-node-2

  redis-node-1:
    image: bitnami/redis-cluster:7.4
    environment:
      REDIS_PASSWORD: "admin"
      REDIS_NODES: redis redis-node-1 redis-node-2
    expose:
      - 6379

  redis-node-2:
    image: bitnami/redis-cluster:7.4
    environment:
      REDIS_PASSWORD: "admin"
      REDIS_NODES: redis redis-node-1 redis-node-2
    expose:
      - 6379

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
on-hold Issues or Pull Requests with this label will never be considered stale redis-cluster tech-issues The user has a technical issue about an application
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants