Skip to content

Conversation

@azrle
Copy link

@azrle azrle commented Feb 21, 2020

MHA can be hung when doing DBI->connect and reading from server if we don't have a mysql_read_timeout.

Here are procedures to reproduce the problem:

  • Server
    We block 4th and following output packets from MySQL server.
    It simulate some cases when the server suddenly goes down.
sudo iptables -I OUTPUT -p tcp --destination $CLIENT_IP --source-port 3306 -m recent --rdest --name block-mysql --set

sudo iptables -A OUTPUT -p tcp --source-port 3306 -m recent --name block-mysql --rdest --rcheck --hitcount 3 -j DROP

sudo iptables -L -v
Chain INPUT (policy ACCEPT 1339 packets, 118K bytes)
 pkts bytes target     prot opt in     out     source               destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
Chain OUTPUT (policy ACCEPT 1167 packets, 92928 bytes)
 pkts bytes target     prot opt in     out     source               destination
   88  6708            tcp  --  any    any     anywhere             CLIENT_IP            tcp spt:mysql recent: SET name: block-mysql side: dest mask: 255.255.255.255
    0     0 DROP       tcp  --  any    any     anywhere             anywhere             tcp spt:mysql recent: CHECK hit_count: 3 name: block-mysql side: dest mask: 255.255.255.255

# clear counter
echo / | sudo tee /proc/net/xt_recent/block-mysql

cat /proc/net/xt_recent/block-mysql
# empty
  • Client
    When you run the following client script, it will hang there. And after adding mysql_read_timeout, the problem will be solved.
use DBI;

my $SERVER = '10.0.0.2';
my $dbh = DBI->connect(
    "DBI:mysql:;host=$SERVER;"
      . "port=3306;mysql_connect_timeout=5"
      # . ";mysql_read_timeout=5"
    , 'root', 'rootpass',
    { PrintError => 0, RaiseError => 0 }
);
die "failed to connect: $DBI::errstr" unless $dbh;
print "OK\n";

Value $connect_timeout is used because it's shorter than $wait_timeout and we hope MHA can fail faster in this case.

cc: @hirose31

MHA can be hung when doing `DBI->connect` and reading from server if we don't have a mysql_read_timeout.
Value connect_timeout is used because it's shorter than wait_timeout and we hope MHA can fail faster in this case.
@azrle azrle requested review from grypyrg and yoshinorim February 21, 2020 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant