Target specific Azure Web App Instance with Header instead of a Cookie

Question

I have an architecture where I have multiple instances but I want to maximize cache hits.

Users are defined in groups and I want to make sure that all users that belong to the same group hit the same server as much as possible.

The application is fully stateless, but having users from the same group hitting the same server will dramatically increase performance and memory load on all instances.

When loading the main page I already know which server I would like to send this user to on the XHR call.

Using the ARRAffinity cookies are not really great is almost impossible in this scenario (cross domain, have to make server call first etc) and I would strongly prefer sending a hint myself through a custom header.

I'm trying manually to do some workarounds with deleting the cookies and assigning them, but it feels very hacky and I don't get it fully working yet. and it doesn't work for XHR calls.

Question:

Is it possible to direct to a specific instance through a header, url or domain instead of a cookie?

Notes

• Distributed cache does not work for me in this case. I need the performance of memory cache without extra network hops and serialization/deserialization.
• This seems to be possible with Application Gateway, but it seem to need a lot of extra infrastructure and moving parts while all my problems would be fixed by sending the "right" header.
• I could fix this by duplicating the web app in its entirety and assigning a different hostname. Also this feels like adding a lot of extra moving parts that can break. Also maintenance will be harder and more confusing, I loss autoscale, etc.
• Maybe this can be fixed easily by Kubenetes/Docker Swarm type of architecture (no experience), but as this is a large legacy project and I have a pretty strict deadline I am very cautious of making such a dramatic switch last minute.

Answer 1

If I am understanding correctly you want to set a custom header via a client application and based on that proxy over the connection to some other backend server.

I like to use HAProxy for that, you can also look into Nginx as well for that. You can install HAProxy on linux from the distribution's package manager, or you can use the available HAProxy docker container. An example of installing it on ArchLinux:

sudo pacman -S haproxy 
sudo systemctl start haproxy

Once its installed you can find out where your haproxy.cfg config file is located and then copy the haproxy.cfg config snippet that I posted here below instead of the existing default config.

In my case haproxy.cfg was in /etc/haproxy/haproxy.cfg

To achieve what you want in HAProxy you would set all clients to communicate with this main HAProxy server, which would then forward the connection to the different backend servers you have based on the value of the custom header that you can set client side, for example "x-mycustom-header: Server-one". As a bonus, you can also enable sticky sessions as well if needed on HAProxy, but it is not mandatory to achieve what you are looking for.

Here is a simple example setup for the HAProxy config file (haproxy.cfg) with only 2 backend servers, but you can add more.

The logic here is that all the clients would make http requests to the HAProxy server listening on port 80, then HAProxy would check the value of the custom header called 'x-mycustom-header' that the clients added and based on that value, it will forward the client to either backend_server_one or backend_server_two.

For testing purposes both HAProxy and the two backends are on the same box but listening on different ports. HAProxy on port 80, server1 is on 127.0.0.1:3000 and server2 is on 127.0.0.1:4000.

 cat haproxy.cfg
#---------------------------------------------------------------------
# Example configuration.  See the full configuration manual online.
#
#   http://www.haproxy.org/download/1.7/doc/configuration.txt
#
#---------------------------------------------------------------------

global
    maxconn     20000
    log         127.0.0.1 local0
    user        haproxy
    chroot      /usr/share/haproxy
    pidfile     /run/haproxy.pid
    daemon

frontend  main
    bind :80
    mode                 http
    log                  global
    option               httplog
    option               dontlognull
    option               http_proxy
    option forwardfor    except 127.0.0.0/8
    maxconn              8000
    timeout              client  30s
    use_backend backend_server_one if { req.hdr(x-mycustom-header) server-one }
    use_backend backend_server_two if { req.hdr(x-mycustom-header) server-two }
    default_backend      backend_server_one #when the header is something else default to the first backend

backend backend_server_one
    mode        http
    balance     roundrobin
    timeout     connect 5s
    timeout     server  5s
    server      static 127.0.0.1:3000 #change this ip to your 1st backend ip address

backend backend_server_two
    mode        http
    balance     roundrobin
    timeout     connect 5s
    timeout     server  30s
    server      static 127.0.0.1:4000 #change this ip to your 2nd backend ip address

To test that this works you can open two netcat listeners, one on port 3000, and then the other on port 4000, run them on differnt screens or different ssh sessions.

nc -l 3000 # in the first screen
nc -l 4000 # in a second screen 

Then after you do a sudo systemctl reload haproxy to make sure that HAProxy is reloaded with your new config file, you can make an http GET request on port 80 and provide the "x-mycustom-header: Server-one" header.

You will be able to see the request in the output of the netcat instance that is listening on port 3000. Now change the header to "x-mycustom-header: Server-two" and make a second GET request, and you will see that the request reached to the second netcat instance this time, which is listening on port 4000, which indicates that this works.

Tested on ArchLinux