[BUG] for "suse:sles-sap-15-sp6:gen2:latest" "playbook_04_00_00_db_install.yaml" hangs on "TASK [roles-os/1.17-generic-pacemaker : 1.17 Generic Pacemaker - Enable Stonith] ***"

[lilyeyes]

Describe the bug
This playbook execution passed on "suse:sles-sap-15-sp5:gen2:latest" public cloud image but failed on "suse:sles-sap-15-sp6:gen2:latest" the playbook hangs on TASK [roles-os/1.17-generic-pacemaker : 1.17 Generic Pacemaker - Enable Stonith] ***.

I am using the latest version v3.13.0.1 to deploy but the playbook_04_00_00_db_install.yaml hangs as following (-vvvv):

TASK [roles-os/1.17-generic-pacemaker : 1.17 Generic Pacemaker - Enable Stonith] ***
task path: /tmp/Azure_SAP_Automated_Deployment_305404/sap-automation/deploy/ansible/roles-os/1.17-generic-pacemaker/tasks/1.17.2.0-cluster-Suse.yml:186
<192.168.58.86> ESTABLISH SSH CONNECTION FOR USER: azureadm
<192.168.58.86> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/tmp/Azure_SAP_Automated_Deployment_305404/WORKSPACES/SYSTEM/PRD-WEEU-305404-QES/sshkey"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="azureadm"' -o ConnectTimeout=10 -o StrictHostKeyChecking=no -o ServerAliveInterval=60 -o ServerAliveCountMax=120 -o 'ControlPath="/home/azureadm/.ansible/cp/21ab2d74ac"' 192.168.58.86 '/bin/sh -c '"'"'echo ~azureadm && sleep 0'"'"''
<192.168.58.86> (0, b'/home/azureadm\n', b"OpenSSH_8.4p1, OpenSSL 1.1.1d-fips  10 Sep 2019\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 20: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.58.86 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/azureadm/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/azureadm/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8598\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
<192.168.58.86> ESTABLISH SSH CONNECTION FOR USER: azureadm
<192.168.58.86> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/tmp/Azure_SAP_Automated_Deployment_305404/WORKSPACES/SYSTEM/PRD-WEEU-305404-QES/sshkey"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="azureadm"' -o ConnectTimeout=10 -o StrictHostKeyChecking=no -o ServerAliveInterval=60 -o ServerAliveCountMax=120 -o 'ControlPath="/home/azureadm/.ansible/cp/21ab2d74ac"' 192.168.58.86 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/azureadm/.ansible/tmp `"&& mkdir "` echo /home/azureadm/.ansible/tmp/ansible-tmp-1733399954.4499755-21068-131765141398929 `" && echo ansible-tmp-1733399954.4499755-21068-131765141398929="` echo /home/azureadm/.ansible/tmp/ansible-tmp-1733399954.4499755-21068-131765141398929 `" ) && sleep 0'"'"''
Read vars_file 'vars/ansible-input-api.yaml'
skipping: [qesdhdb01l1e8] => {
    "changed": false,
    "skip_reason": "Conditional result was False"
}
<192.168.58.86> (0, b'ansible-tmp-1733399954.4499755-21068-131765141398929=/home/azureadm/.ansible/tmp/ansible-tmp-1733399954.4499755-21068-131765141398929\n', b"OpenSSH_8.4p1, OpenSSL 1.1.1d-fips  10 Sep 2019\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 20: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.58.86 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/azureadm/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/azureadm/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8598\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
Using module file /opt/ansible/venv/2.11/lib/python3.6/site-packages/ansible/modules/command.py
<192.168.58.86> PUT /home/azureadm/.ansible/tmp/ansible-local-17224hxk_j7aq/tmpdn9siti5 TO /home/azureadm/.ansible/tmp/ansible-tmp-1733399954.4499755-21068-131765141398929/AnsiballZ_command.py
<192.168.58.86> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/tmp/Azure_SAP_Automated_Deployment_305404/WORKSPACES/SYSTEM/PRD-WEEU-305404-QES/sshkey"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="azureadm"' -o ConnectTimeout=10 -o StrictHostKeyChecking=no -o ServerAliveInterval=60 -o ServerAliveCountMax=120 -o 'ControlPath="/home/azureadm/.ansible/cp/21ab2d74ac"' '[192.168.58.86]'
<192.168.58.86> (0, b'sftp> put /home/azureadm/.ansible/tmp/ansible-local-17224hxk_j7aq/tmpdn9siti5 /home/azureadm/.ansible/tmp/ansible-tmp-1733399954.4499755-21068-131765141398929/AnsiballZ_command.py\n', b'OpenSSH_8.4p1, OpenSSL 1.1.1d-fips  10 Sep 2019\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 20: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.58.86 is address\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts\' -> \'/home/azureadm/.ssh/known_hosts\'\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts2\' -> \'/home/azureadm/.ssh/known_hosts2\'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8598\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug2: Server supports extension "[email protected]" revision 2\r\ndebug2: Server supports extension "[email protected]" revision 2\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug2: Unrecognised server extension "[email protected]"\r\ndebug2: Unrecognised server extension "[email protected]"\r\ndebug2: Unrecognised server extension "copy-data"\r\ndebug2: Unrecognised server extension "home-directory"\r\ndebug2: Unrecognised server extension "[email protected]"\r\ndebug3: Sent message fd 3 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/azureadm size 0\r\ndebug3: Looking up /home/azureadm/.ansible/tmp/ansible-local-17224hxk_j7aq/tmpdn9siti5\r\ndebug3: Sent message fd 3 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/azureadm/.ansible/tmp/ansible-tmp-1733399954.4499755-21068-131765141398929/AnsiballZ_command.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:29664\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 29664 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<192.168.58.86> ESTABLISH SSH CONNECTION FOR USER: azureadm
<192.168.58.86> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/tmp/Azure_SAP_Automated_Deployment_305404/WORKSPACES/SYSTEM/PRD-WEEU-305404-QES/sshkey"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="azureadm"' -o ConnectTimeout=10 -o StrictHostKeyChecking=no -o ServerAliveInterval=60 -o ServerAliveCountMax=120 -o 'ControlPath="/home/azureadm/.ansible/cp/21ab2d74ac"' 192.168.58.86 '/bin/sh -c '"'"'chmod u+x /home/azureadm/.ansible/tmp/ansible-tmp-1733399954.4499755-21068-131765141398929/ /home/azureadm/.ansible/tmp/ansible-tmp-1733399954.4499755-21068-131765141398929/AnsiballZ_command.py && sleep 0'"'"''
<192.168.58.86> (0, b'', b"OpenSSH_8.4p1, OpenSSL 1.1.1d-fips  10 Sep 2019\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 20: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.58.86 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/azureadm/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/azureadm/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8598\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
<192.168.58.86> ESTABLISH SSH CONNECTION FOR USER: azureadm
<192.168.58.86> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/tmp/Azure_SAP_Automated_Deployment_305404/WORKSPACES/SYSTEM/PRD-WEEU-305404-QES/sshkey"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="azureadm"' -o ConnectTimeout=10 -o StrictHostKeyChecking=no -o ServerAliveInterval=60 -o ServerAliveCountMax=120 -o 'ControlPath="/home/azureadm/.ansible/cp/21ab2d74ac"' -tt 192.168.58.86 '/bin/sh -c '"'"'sudo -H -S -n  -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-qortjuxsbfqujaooqyiitxlltckkobdz ; /usr/bin/python3.6 /home/azureadm/.ansible/tmp/ansible-tmp-1733399954.4499755-21068-131765141398929/AnsiballZ_command.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
!!!!!!!!!!!!!!!!!
(At here I waited +1 hours but still no any outputs, it hangs ....)
!!!!!!!!!!!!!!!!!

To reproduce
List of executed playbooks:

pb_get-sshkey.yaml
playbook_00_validate_parameters.yaml
playbook_01_os_base_config.yaml
playbook_02_os_sap_specific_config.yaml
playbook_03_bom_processing.yaml
playbook_04_00_00_db_install.yaml
playbook_04_00_01_db_ha.yaml
playbook_07_00_00_post_installation.yaml
playbook_08_00_00_post_configuration_actions.yaml

Expected behavior
This playbook passed on suse:sles-sap-15-**sp5**:gen2:latest so it should be passed on sp6.

Additional context
Add any other context about the problem here.

[hdamecharla]

@lilyeyes , thank you for reporting this to us.

Could you kindly let me know which ansible version was installed on the controller when you encountered this bug?

[lilyeyes]

controller

Hi hdamecharla,
Thanks for the quick response.
From the output of playbook playbook_00_validate_parameters I got:

[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the 
controller starting with Ansible 2.12. Current version: 3.6.15 (default, Sep 23
 2021, 15:41:43) [GCC]. This feature will be removed from ansible-core in 
version 2.12. Deprecation warnings can be disabled by setting 
deprecation_warnings=False in ansible.cfg.
ansible-playbook [core 2.11.12] 
  config file = None
  configured module search path = ['/home/azureadm/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/ansible/venv/2.11/lib/python3.6/site-packages/ansible
  ansible collection location = /opt/ansible/collections
  executable location = /opt/ansible/bin/ansible-playbook
  python version = 3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]
  jinja version = 3.0.3
  libyaml = True
No config file found; using defaults