The connection to MySQL failed because LOCK_thread_cache was not published.

description of the problem

The connection to MySQL failed and too many connections are removed. But after that, the port is open, mysql has not responded to the packages, including the version and the password plugins. And never recover again.

mysql error information

  • version: percona-server-5.7.24-27
  • max_connections = 5000
  • MySQL error-log
  • tcpdump while mysql is trying to connect the server.
  • strace info (strace -tt -T -T -v -f -p 15179 -o output.log)
17264 19: 48: 42.143662 set_robust_list (0x7fa5a22c39e0, 24 
15179 19: 48: 42.143673 <... clone resumed> plug <0.001116>
17264 19: 48: 42.144164 <... set_robust_list resumed> ) = 0 <0.000497>
17264 19: 48: 42.144535 gettid ( 
17264 19: 48: 42.144879 <... gettid resumed> ) = 17264 <0.000337>
17264 19: 48: 42.145227 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
  • strace -f -p 15179> 20minutes.log
9107 18: 50: 39.688588 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
8926 18: 50: 39.688703 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
8738 18: 50: 39.688719 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
8492 18: 50: 39.688730 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
8203 18: 50: 39.688742 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
8106 18: 50: 39.688752 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
7983 18: 50: 39.688763 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
7753 18: 50: 39.688774 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
7353 18: 50: 39.688785 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
7339 18: 50: 39,688796 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
7255 18: 50: 39.688806 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
7205 18: 50: 39.688817 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
7187 18: 50: 39.688827 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
7157 18: 50: 39.688836 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
7104 18: 50: 39.688846 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
7055 18: 50: 39.688856 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
7028 18: 50: 39.688871 futex (0x1e01e80, FUTEX_WAIT_PRIVATE, 2, NULL 
  • lsof -p 15179> lsof.15179.log, 4410 lines include "protocol: TCP"
mysqld 15179 mysql 5254u sock 0.7 0t0 protocol 111363206: TCP
mysqld 15179 mysql 5255u sock 0.7 0t0 protocol 111357243: TCP
mysqld 15179 mysql 5256u sock 0.7 0t0 protocol 111357244: TCP
mysqld 15179 mysql 5257u sock 0.7 0t0 protocol 111363207: TCP
mysqld 15179 mysql 5258u sock 0.7 0t0 protocol 111363208: TCP
mysqld 15179 mysql 5259u sock 0.7 0t0 protocol 111353396: TCP
mysqld 15179 mysql 5260u sock 0.7 0t0 protocol 111356603: TCP
mysqld 15179 mysql 5261u sock 0.7 0t0 protocol 111356604: TCP
mysqld 15179 mysql 5262u sock 0.7 0t0 protocol 111359747: TCP
mysqld 15179 mysql 5263u sock 0.7 0t0 protocol 111356606: TCP
mysqld 15179 mysql 5264u sock 0.7 0t0 protocol 111357250: TCP
mysqld 15179 mysql 5265u sock 0.7 0t0 protocol 111359748: TCP
mysqld 15179 mysql 5266u sock 0.7 0t0 protocol 111360201: TCP
mysqld 15179 mysql 5267u sock 0.7 0t0 protocol 111360202: TCP
mysqld 15179 mysql 5268u sock 0.7 0t0 protocol 111357251: TCP
mysqld 15179 mysql 5269u sock 0.7 0t0 protocol 111363211: TCP
mysqld 15179 mysql 5270u sock 0.7 0t0 protocol 111362371: TCP
mysqld 15179 mysql 5271u sock 0.7 0t0 protocol 111354590: TCP
mysqld 15179 mysql 5272u sock 0.7 0t0 protocol 111363212: TCP
mysqld 15179 mysql 5273u sock 0.7 0t0 protocol 111354591: TCP
  • netstat includes many close_wait connections;

  • top -Hbp 15179 -n1 | wc -l, 5000+ son of mysqld

source code analysis

LOCK_thread_cache is not published as usual. mysql_mutex_lock (& ‚Äč‚ÄčLOCK_thread_cache); but I do not know why.