haproxy is auto restarted every few minutes in ubuntu

my haproxy is auto restarted every few minutes, is it by design? from syslog:

    Aug 23 04:24:31 localhost haproxy(64543): (NOTICE) 235/042431 (64543) : New worker #1 (64544) forked
Aug 23 04:24:38 api-lb systemd(1): haproxy.service: Succeeded.
Aug 23 04:51:11 api-lb systemd(1): haproxy.service: Main process exited, code=exited, status=137/n/a
Aug 23 04:51:11 api-lb systemd(1): haproxy.service: Failed with result 'exit-code'.
Aug 23 04:51:11 api-lb systemd(1): haproxy.service: Scheduled restart job, restart counter is at 1.
Aug 23 04:52:17 api-lb systemd(1): haproxy.service: Main process exited, code=exited, status=137/n/a
Aug 23 04:52:17 api-lb systemd(1): haproxy.service: Failed with result 'exit-code'.
Aug 23 04:52:18 api-lb systemd(1): haproxy.service: Scheduled restart job, restart counter is at 2.
Aug 23 04:56:15 api-lb systemd(1): haproxy.service: Main process exited, code=exited, status=137/n/a
Aug 23 04:56:15 api-lb systemd(1): haproxy.service: Failed with result 'exit-code'.
Aug 23 04:56:16 api-lb systemd(1): haproxy.service: Scheduled restart job, restart counter is at 3.
Aug 23 04:56:36 api-lb systemd(1): haproxy.service: Main process exited, code=exited, status=137/n/a
Aug 23 04:56:36 api-lb systemd(1): haproxy.service: Failed with result 'exit-code'.
Aug 23 04:56:36 api-lb systemd(1): haproxy.service: Scheduled restart job, restart counter is at 4.
Aug 23 04:59:19 api-lb systemd(1): haproxy.service: Main process exited, code=exited, status=137/n/a
Aug 23 04:59:19 api-lb systemd(1): haproxy.service: Failed with result 'exit-code'.
Aug 23 04:59:19 api-lb systemd(1): haproxy.service: Scheduled restart job, restart counter is at 5.
Aug 23 05:00:20 api-lb systemd(1): haproxy.service: Main process exited, code=exited, status=137/n/a
Aug 23 05:00:20 api-lb systemd(1): haproxy.service: Failed with result 'exit-code'.
Aug 23 05:00:21 api-lb systemd(1): haproxy.service: Scheduled restart job, restart counter is at 6.
Aug 23 05:01:15 api-lb systemd(1): haproxy.service: Main process exited, code=exited, status=137/n/a
Aug 23 05:01:15 api-lb systemd(1): haproxy.service: Failed with result 'exit-code'.
Aug 23 05:01:15 api-lb systemd(1): haproxy.service: Scheduled restart job, restart counter is at 7.
Aug 23 05:04:15 api-lb systemd(1): haproxy.service: Main process exited, code=exited, status=137/n/a
Aug 23 05:04:15 api-lb systemd(1): haproxy.service: Failed with result 'exit-code'.
Aug 23 05:04:15 api-lb systemd(1): haproxy.service: Scheduled restart job, restart counter is at 8.
Aug 23 05:05:23 api-lb systemd(1): haproxy.service: Main process exited, code=exited, status=137/n/a
Aug 23 05:05:23 api-lb systemd(1): haproxy.service: Failed with result 'exit-code'.
Aug 23 05:05:24 api-lb systemd(1): haproxy.service: Scheduled restart job, restart counter is at 9.

it’s running under Ubuntu 20.04.1 LTS with HA-Proxy version 2.0.13-2 2020/04/01 – https://haproxy.org/, main config as:

    global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    maxconn 80000
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    log global
    mode    http
    option  httplog
    option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http
    maxconn 80000
    retries 3

any thoughts? We just do not want to lose the stats b/c each restart will ease the stats.
Thanks

haproxy – PostgreSQL High Memory consumption

We have been getting a lot of issues pertaining memory consumption in Postgres. I am thinking of deploying PgBouncer to resolve this issue, but wanted to know what could be the cause for this.

ps aux --sort=-%mem | head -30

USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
postgres  23438 41.0  6.2 78075200 8258220 ?    Ssl  Aug09 525:40 postgres: xxxxx: xxxxx(60028) idle
postgres   4225 31.7  5.9 77797888 7903220 ?    Ssl  Aug09 438:58 postgres: xxxxx: xxxxx(45012) idle
postgres 118999 39.3  4.8 76290020 6395136 ?    Ssl  Aug09 306:52 postgres: xxxxx: xxxxx(50026) idle

free -g
              total        used        free      shared  buff/cache   available

Mem:            125          36          35           3          54          84
Swap:            19           0          19



System RAM : 128 GB
CPU(s) : 32
Shared_buffer : 32 GB
DB Size : 24 GB
Max Connections : 200
Average Connections: 50
Effective_Cache_Size : 64 GB
Work_mem : 41MB
PG Version : 11
OS : Ubuntu

A lot of the idle connections are getting reused despite it being closed from the app side. I have tuned the database by changing Shared_buffer, work_mem to values as per PostgreSQL expectations, but still getting these issues.

PS : We have setup HAProxy between App and DB.

centos – Haproxy 2.1.4 too many SSL Handshake failures

Hi we are using haproxy 2.1.4 as SSL terminator between our own client and server machines(High load machines, always busy) and also requests will be a mix of http/1.1 and http/2.0. We are facing lots of SSL handshake failure in front end. I have enabled proxy logs using rsyslog and get following errors,

Aug  5 18:55:35 localhost haproxy(40308): 127.0.0.1:55442 (05/Aug/2020:18:55:35.364) frontend/1: SSL handshake failure
Aug  5 18:56:20 localhost haproxy(40308): 204.xx.xx.xx:45474 (05/Aug/2020:18:56:16.761) frontend/1: Connection closed during SSL handshake
Aug  5 18:56:22 localhost haproxy(40308): 204.xx.xx.xx:52088 (05/Aug/2020:18:56:19.403) frontend/1: Connection closed during SSL handshake
Aug  5 18:56:33 localhost haproxy(40308): 127.0.0.1:42470 (05/Aug/2020:18:56:33.933) frontend/1: SSL handshake failure
Aug  5 18:56:33 localhost haproxy(40308): 127.0.0.1:42472 (05/Aug/2020:18:56:33.944) frontend/1: SSL handshake failure

Few of the requests have source Ip as 127.0.0.1 but we are doing a plain text connection between proxy and backend as a proxy is SSL terminator here, I could not get detailed logs out of haproxy, my configurations are as follows,

global
   log         127.0.0.1 local2
   chroot /var/lib/haproxy
   maxconn 200000
   user test
   group testsending
   daemon

tune.ssl.cachesize 200000
#tune.h2.max-concurrent-streams 10
ssl-dh-param-file /etc/haproxy/dhparam.pem

#Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private

#Obtained from https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy

ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11

ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11

defaults
    log     global
    maxconn 20000
    mode    http
    option httplog
    option dontlog-normal
    option logasap
    retries 3
    retry-on all-retryable-errors
    option log-separate-errors
    timeout connect     5s
    timeout client     60s
    timeout server    450s

frontend    frontend_haproxy
     option forwardfor
     capture request header MONITORID len 64
     capture response header MONITORID len 64
     log-format "%ci:%cp (%t) %f %b/%s %Tq/%Tw/%Tc/%Tr/%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq Reqid:%hr Resid:%hs %{+Q}r %sslv %sslc"
     bind    *:8088  ssl crt /etc/haproxy/haproxy.pem alpn h2,http/1.1
     default_backend backend_eumagent

 backend     backend_eumagent
     timeout server  420000
     fullconn 2000
     server tomcat localhost:9099 check

and output of haproxy -vv as,

 HA-Proxy version 2.1.4 2020/04/02 - https://haproxy.org/
 Status: stable branch - will stop receiving fixes around Q1 2021.
 Known bugs: http://www.haproxy.org/bugs/bugs-2.1.4.html
 Build options :
   TARGET  = linux-glibc
   CPU     = generic
   CC      = gcc
   CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
   OPTIONS = USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1
 
 Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
 
 Default settings :
   bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
 
 Built with multi-threading support (MAX_THREADS=64, default=6).
 Built with OpenSSL version : OpenSSL 1.1.1c  28 May 2019
 Running on OpenSSL version : OpenSSL 1.1.1c  28 May 2019
 OpenSSL library supports TLS extensions : yes
 OpenSSL library supports SNI : yes
 OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
 Built with network namespace support.
 Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
 Built with PCRE version : 8.32 2012-11-30
 Running on PCRE version : 8.32 2012-11-30
 PCRE library supports JIT : no (USE_PCRE_JIT not set)
 Encrypted password support via crypt(3): yes
 Built with zlib version : 1.2.7
 Running on zlib version : 1.2.7
 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
 
 Available polling systems :
       epoll : pref=300,  test result OK
        poll : pref=200,  test result OK
      select : pref=150,  test result OK
 Total: 3 (3 usable), will use epoll.
 
 Available multiplexer protocols :
 (protocols marked as <default>      cannot be specified using 'proto' keyword)
               h2 : mode=HTTP       side=FE|BE     mux=H2
             fcgi : mode=HTTP       side=BE        mux=FCGI
        <default>      : mode=HTTP       side=FE|BE     mux=H1
        <default>      : mode=TCP        side=FE|BE     mux=PASS
 
 Available services : none
 
 Available filters :
  (SPOE) spoe
  (CACHE) cache
  (FCGI) fcgi-app
  (TRACE) trace
  (COMP) compression

We send requests to haproxy from apache(http1.1) and jetty(http2) httpclients using java 8 and our backend is an apache tomcat 9.0.30+ running using java 11. All our machines are centos 7.x versions.

I have also asked help in haproxy community

Kindly help me to debug this issue. Thanks in advance

configuration – HAProxy not sending health checks to backup servers

In my haproxy.cfg file, I have the following config for the backend

backend app
    balance     roundrobin

    server  s1 127.0.0.1:8000 check
    server  s2 135.111.29.95:80 check

    server  b1 135.111.29.96 check backup

Yet, when I check the HAProxy log, no health checks are being done on the backup server.

Am I just missing some line in the configuration?

Screencap of HAProxy backend stats

HAProxy + uwsgi all connection “Connection reset by peer”

I’ve a problem with UWSGI + HAProxy.
for all the connections that get closed, i see this log in uwsgi file (uwsgi-http key: client_addr: 192.168.173.250 client_port: 57002) hr_read(): Connection reset by peer (plugins/http/http.c line 917)

i can easily reproduce it with this

>>> for i in range(10):
...     requests.get('http://my-ha-url')
...
<Response (200)>
..
<Response (200)>

i see this

(uwsgi-http key:  client_addr: 192.168.173.250 client_port: 35534) hr_read(): Connection reset by peer (plugins/http/http.c line 917)
...
(uwsgi-http key:  client_addr: 192.168.173.250 client_port: 59134) hr_read(): Connection reset by peer (plugins/http/http.c line 917

BUT If i call directly without passing trough HA (calling the endpoing) there’s no log in uwsgi.

Has anyone idea of what can be?
is HA persisting the connection and at some point cutting it off?

rate limiting – HAProxy ratelimit with delay

Is there a way to set a limit per minute (e.g. 150 r/m) and not just have a maximum threshold of connection after which connections are blocked?

I’ve investigated a bit and I can’t find anything. I’ve created a lua script to mimic this behaviour but I’m not sure it’s efficent or correct: This is the gist: https://gist.github.com/esseti/890b96259e74cc0308735ab5f88b6c28

The problem that I’ve is that this is loop that stays there and repeat until the condition (the rate of requests < threshold). Thus, it can run into problem if people keeps sending requests. And i’m not sure it’s the correct/best way to do it.

Thus:

  • is there a way to put back the request to the queue in order to have haproxy reprocess it?
  • can the lua script return an error as response or trigger a 429 in some way?
  • is it possible in a LUA script to access/modify the sticky table? if yes, how can i do it?
  • a lua script that has a global matrix (matrix = {} {}) is it shared with all the other instances/processes of haproxy?
  • how does lua/haproxy cope with threads sleeping? (efficeny/costs)

centos – Haproxy service won’t start when keepalived service is running

I am trying to configure haproxy with keepalived but if I start keepalived service first, haproxy fails to start. When keepalived service is stopped, haproxy starts without any problems. I get the error below when I attempt to start haproxy manually while keepalived is running:
Proxy ‘webserver’: unable to find local peer ‘VIP01.domain.com’ in peers section ‘mypeer’.
Fatal errors found in configuration.
Control process exited, code=exited status=1
Failed with result ‘exit-code’.
Failed to start HAProxy Load Balancer.

The system is Centos 8 and I found the article below and checked if net.ipv4.ip_nonlocal_bind sysctl is set to 1 but it still doesn’t start when keepalived is running.
https://cbonte.github.io/haproxy-dconv/1.8/management.html#11

I would greatly appreciate any help.

load balancing – How can I get haproxy to fail over to another backend?

TLDR: What changes should I make to my HAProxy config to ensure that requests fail over to a working backend when a server goes down?

This seems like a simple question and may be a duplicate, but everything I’ve found online isn’t working and I may be having difficulty formulating the question so I can find relevant results…

I’ve been following this tutorial for my setup.

I’m putting together an application with several (identical, stateless) API applications running in Docker containers with an HAProxy load balancer in front. The HAProxy config is below:

global
    maxconn 4096
    daemon

defaults
    log     global
    mode    http

frontend api_gateway
    bind 0.0.0.0:80
    default_backend api_backend

backend api_backend
    balance roundrobin
    option httpchk GET /check # returns status 200 and text "OK" if server is up
    option redispatch
    server apiservice_1 192.168.xxx.yyy:61001 check inter 2s downinter 5s fall 1 rise 2 observe layer4 error-limit 1 on-error mark-down
    server apiservice_2 192.168.xxx.yyy:61002 check inter 2s downinter 5s fall 1 rise 2 observe layer4 error-limit 1 on-error mark-down

I’m trying to write an automated deployment script that will take down one API server at a time & update it with zero downtime. I set up my servers and the haproxy container and I’m using wget to monitor the requests every second using watch. I’m stopping & starting the backends to test the proxy’s behaviour.

The problem I’m finding is that occasionally the proxy will send back a 502 Bad Gateway if I hit the gateway as the server is going down. It’s not hard to do this – I just have to time it right.

It seems to me that it should be possible to configure the proxy to try a different backend without rejecting the request if the first backend it tries fails. But I cannot get this to work.

From my reading of the documentation for option redispatch it sounds like this is the option that should handle it. But I should note that I’m using a bearer token for authentication and not using cookies, which may mean that this option does not apply?

The other point that may be giving me problems with the notion of what it means for the server to be down. Much of the documentation I read for various options seems to reply on HTTP status codes to determine if the backend is down – what happens when the backend is just unresponsive? It seems to me that an unresponsive server is the paradigm case of “server down”, but I’m finding the documentation to be a bit unclear on this.

ssl – What happens to existing tcp sessions when a certificate is changed and haproxy is reloaded?

There may be two major reasons for changing a certificate.
1. Ordinary rotation, for example, due to expiration. In this case I would like to have a 100% high availiability. Existing sessions would use the old keys, the new sessions will use new encryption keys.
2. My private keys had been stolen and I want to drop all the exsting connections and reestablish those with the new keys.

How does the haproxy behave?

Display the TCP client port of the haproxy backend in the logs

The context:
I have a HAProxy configured to proxy TCP traffic for websockets. So a TCP redirect session looks like this:

client app --> ((frontend) haproxy (backend)) --> server app

From the server point of view, the TCP client that establishes the TCP connection is the backend of HAProxy. When this new connection is established, it is possible for me to know the port of the TCP client (in other words, the port of the TCP client connection of the HAProxy backend).

Question:
Is there a way to display the port of this client in the logs? Not the client application port, but really the backend TCP client port. Basically, I wish I had a way to correlate the socket I have on the server with a session on haproxy.

What I have tried:
I looked in the doc to customize the TCP log format and tried to find a field that would give me the information I need, but nothing seems to be fine.