working configuration for haproxy with the force-persist setting

Question

I'm not sure if I'm missing some key configuration piece or am just fundamentally misunderstanding the purpose of force-persist in haproxy (using version 1.5.11 on Ubuntu 14.04). From the documentation:

The "force-persist" statement allows one to declare various ACL-based conditions which, when met, will cause a request to ignore the down status of a server and still try to connect to it. That makes it possible to start a server, still replying an error to the health checks, and run a specially configured browser to test the service.

This sounds like exactly the behaviour I want, where I could put all app servers into "maintenance mode" for a code rollout, yet allow certain IPs to still connect in order to test it out, post-rollout, before giving everyone access again. Here is the configuration I have set up:

global
  log /dev/log  local0
  log /dev/log  local1 notice
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin
  stats timeout 30s
  user haproxy
  group haproxy
  daemon

  # Default SSL material locations
  ca-base /etc/ssl/certs
  crt-base /etc/ssl/private

  # Default ciphers to use on SSL-enabled listening sockets.
  # For more information, see ciphers(1SSL).
  ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
  ssl-default-bind-options no-sslv3

defaults
  log   global
  mode  http
  option    httplog
  option    dontlognull
  timeout connect 300s
  timeout client  50000
  timeout server  50000
  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http

listen mysite
  bind *:80
  bind *:443 ssl crt mysite.pem
  http-request set-header X-Forwarded-Port %[dst_port]
  http-request add-header X-Forwarded-Proto https if { ssl_fc }
  redirect scheme https if !{ ssl_fc }
  balance roundrobin
  option forwardfor
  option httpchk HEAD /haproxy_health_check.php
  acl whitelist src -f /etc/haproxy/whitelist.lst
  force-persist if whitelist
  server app-1 10.1.4.32:80 maxconn 20 check inter 10000

With this configuration, I think I should be able to issue the following command:

echo "disable server mysite/app-1" | socat /run/haproxy/admin.sock stdio

taking the single app server out of rotation, and so long as I'm coming from an IP address listed in /etc/haproxy/whitelist.lst, I should still be able to see the website as if the server was still enabled. However, what I end up seeing is the 503 error page, which I'd expect normally if I was a regular user, but not coming from the whitelisted IP. To remove the possibility that I was incorrectly specifying the IP addresses, or using the acl command incorrectly, I tried a variation, where I simply set:

force-persist if TRUE

From my reading of the documentation, I'd think this would act as if I'd never disabled the server no matter what IP I was coming from. Unfortunately I still get the 503.

There are clunkier ways, involving passing extra configuration in and reloading haproxy, that I could use to get this working, but "force-persist" along with the handy ability to disable via the command line seems like a much more elegant approach, and I'd definitely prefer it if I could make it work.

Has anyone else tried to get haproxy to work in this way? Am I wrong to interpret "force-persist" this way? Do I need some extra bit of configuration to make it work?

Answer 1

There is no rule to indicate what to persist on in this configuration. Usually you'll use this combined with cookie-based persistence. For example :

cookie SERVERID insert indirect nocache
server srv1 1.1.1.1:80 cookie s1 check
server srv2 2.2.2.2:80 cookie s2 check
acl from_management_net src 10.0.0.0/8
force-persist if from_management_net

Then on your client, visit the first server, get the cookie assigned, disable the server, and visit it again, and you'll continue to go there. Usually people implement a specific HTML page listing all servers and their respective cookies, that help choosing the servers from the client by just clicking on them.