Posted on by Frederic Mohr

HA-Proxy for the win

I finally found time to take a closer look at HA-Proxy. It is a high-availability load balancer and (reverse-) proxy server and fully open source.

Attention: This is me testing stuff – I have not taken care of settings like no-sslv3, etc. So if you use this in production, make sure to read up on this first! Also, since I’m new to HA-Proxy, I might have misconfigured or missed a few options so don’t blame me if things aren’t perfect regarding either security or performance!

Goals:

My goals utilizing HA-Proxy included

  • Testing TLS offloading and passthrough capabilities
  • Moving client authentication from backend servers to HA-Proxy
  • Replacing Apache as reverse proxy
  • Increasing availability by adding loadbalancing to servers
  • Increasing availability through HA-Proxy failover setup
  • Learn lot’s of new stuff and share it! 🙂

First off I installed the current stable version 1.5.6 from the haproxy repositories, since Ubuntu 14.04 server still uses the old stable 1.4, which has a lot of features still missing – such as native SSL support.

sudo apt-get install software-properties-common
sudo add-apt-repository ppa:vbernat/haproxy-1.5
sudo apt-get update
sudo apt-get install haproxy

If you want to know more about the parameters used, check out the documentation here: http://cbonte.github.io/haproxy-dconv/configuration-1.5.html (from now on referenced to as $dokulink)

Edit the conf /etc/haproxy/haproxy.cfg to look like this.

global
    log 127.0.0.1 local0 notice                 # log to local rsyslog daemon
    maxconn 2000                                # Number of concurrent connections allowed
    user haproxy
    group haproxy
    tune.ssl.default-dh-param 2048				# DHE max size of parameters for key exchange - $dokulink#tune.ssl.default-dh-param

defaults
    log     global
    mode    http
    #option  httplog							# this option messed with my SSL passthrough, so I disabled it
    option  dontlognull
    retries 3
    option redispatch                           # redistribute sessions if a node goes down (no session stickiness)
    timeout connect  5000                       # minimum time to wait until timeout
    timeout client  10000                       # timeout received from client
    timeout server  10000                       # timeout received from server

To add a farm, you should first know a bit about the config structure.
There are three other config types besides global and defaults, named frontend, backend and listen.

The first two are used to configure the the interface that will be addressed by visitors (frontend) and the farm and loadbalancing settings (backend). The third one (listen) is actually just the combination of the first two, which takes less lines for the same config, but on the downside has a negative impact on the readability. Since I’m fairly new to HA-Proxy, I will use frontend and backend but it’s really up to you which path you’ll choose.

SSL Passthrough

Here haproxy doesn’t terminate the SSL connection but passes it right through to the internal server. This also means that you can’t mess with the traffic, add header options and so on. But it’s an easy way to loadbalance servers that already have SSL enabled without much effort.

frontend https_passthrough
  bind *:443
  option tcplog
  mode tcp
  default_backend apache01

backend apache01
  mode tcp
  option ssl-hello-chk
  # balance roundrobin		# Since I only have one server atm, I don't need a balance option
  server apache01.lan 10.0.3.4:443 check

SSL Offloading / Termination

The nice thing here is, that you can have either HTTP or HTTPS traffic internally, as it gets terminated by HA-Proxy and than sent out to the user over the secured connection which has been established between HA-Proxy and the user.

One possible reason to do this, is to use TLS certificates signed by a private CA in the internal network and only deploy the official “trusted” certificate to HA-Proxy. This makes it easier to switch certificates internally as you have full control over the CA and can create certificates as much as you want for any internal domain. If you want to renew your websites official certificate, you just have to deploy it onto HA-Proxy and restart the service. Or you can have HTTP traffic internally, in case one of your applications isn’t capable of TLS and send encrypt the traffic between the user and your loadbalancer.

frontend https_termination
  bind *:443 ssl crt /etc/ssl/private/officialcert.pem
  mode http
  option httpclose
  option forwardfor
  reqadd X-Forwarded-Proto:\ https
  default_backend ghostblog

backend ghostblog
  mode http
  server ghostblog.lan 10.0.3.4:2368 check

Note that the HA-Proxy TLS certificate format is actually a combined file of the .crt and the .key file. To create the file, just run

cat sitecert.crt sitecert.key > sitecert_haproxy.pem

So much for testing TLS passthrough and termination. Let’s move on to client certificate authentication.

Client Certificate Authentication

In the TLS passthrough section, client certificate auth will still work if it was enabled on the internal apache server, as everything get’s just passed through including the request for the user to authenticate.

But I’d rather have one, highly available place to do all the config and not care about deploying authentication onto every internal webserver.

Enabling it is pretty straight forward, just append the ca-file and verify parameter to the bind option in your TLS termination section. (Note: $certs/ == /etc/ssl/private/)

bind *:443 ssl crt $certs/officialcert.pem ca-file $certs/private_ca.crt verify required

Now, users are required to show a certificate that has been signed by myca.crt in order to fully establish the TLS connection. However, right now no one without a valid cert can visit my blog.

frontend https_termination
  bind *:8080  ssl crt $certs/officialcert.pem ca-file $certs/private_ca.pem verify optional
  mode http
  
  #Update - 17.11.2014
  #redirect location / if { path_beg /ghost/ } ! { ssl_fc_has_crt }
  redirect location / if { path_beg /ghost/ } ! { ssl_c_used }
  default_backend ghost-htsec

backend ghost-htsec
  mode http
  server ghost-htsec01 10.0.3.57:2368 check

Setting verify optional basically means that we don’t care if a visitor provides a certificate or not. Adding the redirect line adds the additional security for the subfolder we want to protect. Now visitors can browse my blog, but only those with a valid cert can go to /ghost/login/ or /ghost/signup/. The best part is, that those who try to login without a certificate don’t get an error message but instead are redirected to the root homepage /.

This adds another tiny bit of security by obscuring the login and register pages. Security by obscurity is nothing bad as long as you don’t solely rely on it for protection!

Since this is already quite a bit of haproxyness to take in, I’m going to stop here and publish what I have learned so far. Stay tuned for another post on HA-Proxy, where I will try to tackle my remaining goals.

UPDATE: 17.11.2014

I experienced huge problems with ssl_fc_has_crt in the past couple of days. The first connection to my host with a valid certificate would always be handled correctly, but a reload of the same page resulted in the redirect that should only be applied to users without a certificate. After going through the HA-Proxy documentation, I found that ssl_c_used is the better choice. Quote from the (http://www.haproxy.org/download/1.5/doc/configuration.txt)[docs]:

ssl_fc_has_crt : boolean
  Returns true if a client certificate is present in an incoming connection over
  SSL/TLS transport layer. Useful if 'verify' statement is set to 'optional'.
  Note: on SSL session resumption with Session ID or TLS ticket, client
  certificate is not present in the current connection but may be retrieved
  from the cache or the ticket. So prefer "ssl_c_used" if you want to check if
  current SSL session uses a client certificate.

ssl_c_used : boolean
  Returns true if current SSL session uses a client certificate even if current
  connection uses SSL session resumption. See also "ssl_fc_has_crt".

Update: 21.11.2014
I noticed, that a brand new Firefox profile as well as Firefox mobile on my Android where greeting me with this message instead of my website.

www.hashtagsecurity.com uses and invalid security certificate.
The certificate is not trusted becauzse no issuer chain was provided.
(Error code: sec_error_unknown_issuer)

I found that a little bit strange, since I bought a valid certificate at a Comodo reseller. This is due to Firefox being a bit more strict then other browsers when it comes to TLS implementation.

The fix is to add the ca-bundle certificates to your webserver config, which contains the TLS-Chain certificates.

# In Apache, add this line
SSLCertificateChainFile /etc/ssl/private/servername.ca-bundle
# In Lighttpd, add this line
ssl.ca-file = "/etc/ssl/private/servername.ca-bundle"

In NGINX and HAPROXY, you don’t change the config file. Just add the content of the ca-bundle.crt to your original certificate.

cd /wherever/your/certs/are/
cat servername.ca-bundle >> servername.crt

For HAPROXY, your certificate should look like this:

-----BEGIN CERTIFICATE-----
long-server-cert-string
-----END CERTIFICATE-----
-----BEGIN PRIVATE KEY-----
long-server-key-string
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
long-ca-cert-string-01    
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
long-ca-cert-string-02
-----END CERTIFICATE-----

After restarting HAPROXY, the error message went away and Firefox displayed the website just like any other browser.

Links:
Here are a few links that helped me come this far.

haproxy.com – they have a lot of stuff but I’m not always sure if it’s still accurate!