I finally found time to take a closer look at HA-Proxy. It is a high-availability load balancer and (reverse-) proxy server and fully open source.
Attention: This is me testing stuff – I have not taken care of settings like no-sslv3
, etc. So if you use this in production, make sure to read up on this first! Also, since I’m new to HA-Proxy, I might have misconfigured or missed a few options so don’t blame me if things aren’t perfect regarding either security or performance!
Goals:
My goals utilizing HA-Proxy included
- Testing TLS offloading and passthrough capabilities
- Moving client authentication from backend servers to HA-Proxy
- Replacing Apache as reverse proxy
- Increasing availability by adding loadbalancing to servers
- Increasing availability through HA-Proxy failover setup
- Learn lot’s of new stuff and share it! 🙂
First off I installed the current stable version 1.5.6 from the haproxy repositories, since Ubuntu 14.04 server still uses the old stable 1.4, which has a lot of features still missing – such as native SSL support.
sudo apt-get install software-properties-common
sudo add-apt-repository ppa:vbernat/haproxy-1.5
sudo apt-get update
sudo apt-get install haproxy
If you want to know more about the parameters used, check out the documentation here: http://cbonte.github.io/haproxy-dconv/configuration-1.5.html (from now on referenced to as $dokulink)
Edit the conf /etc/haproxy/haproxy.cfg
to look like this.
global
log 127.0.0.1 local0 notice # log to local rsyslog daemon
maxconn 2000 # Number of concurrent connections allowed
user haproxy
group haproxy
tune.ssl.default-dh-param 2048 # DHE max size of parameters for key exchange - $dokulink#tune.ssl.default-dh-param
defaults
log global
mode http
#option httplog # this option messed with my SSL passthrough, so I disabled it
option dontlognull
retries 3
option redispatch # redistribute sessions if a node goes down (no session stickiness)
timeout connect 5000 # minimum time to wait until timeout
timeout client 10000 # timeout received from client
timeout server 10000 # timeout received from server
To add a farm, you should first know a bit about the config structure.
There are three other config types besides global
and defaults
, named frontend
, backend
and listen
.
The first two are used to configure the the interface that will be addressed by visitors (frontend
) and the farm and loadbalancing settings (backend
). The third one (listen
) is actually just the combination of the first two, which takes less lines for the same config, but on the downside has a negative impact on the readability. Since I’m fairly new to HA-Proxy, I will use frontend and backend but it’s really up to you which path you’ll choose.
SSL Passthrough
Here haproxy doesn’t terminate the SSL connection but passes it right through to the internal server. This also means that you can’t mess with the traffic, add header options and so on. But it’s an easy way to loadbalance servers that already have SSL enabled without much effort.
frontend https_passthrough
bind *:443
option tcplog
mode tcp
default_backend apache01
backend apache01
mode tcp
option ssl-hello-chk
# balance roundrobin # Since I only have one server atm, I don't need a balance option
server apache01.lan 10.0.3.4:443 check
SSL Offloading / Termination
The nice thing here is, that you can have either HTTP or HTTPS traffic internally, as it gets terminated by HA-Proxy and than sent out to the user over the secured connection which has been established between HA-Proxy and the user.
One possible reason to do this, is to use TLS certificates signed by a private CA in the internal network and only deploy the official “trusted” certificate to HA-Proxy. This makes it easier to switch certificates internally as you have full control over the CA and can create certificates as much as you want for any internal domain. If you want to renew your websites official certificate, you just have to deploy it onto HA-Proxy and restart the service. Or you can have HTTP traffic internally, in case one of your applications isn’t capable of TLS and send encrypt the traffic between the user and your loadbalancer.
frontend https_termination
bind *:443 ssl crt /etc/ssl/private/officialcert.pem
mode http
option httpclose
option forwardfor
reqadd X-Forwarded-Proto:\ https
default_backend ghostblog
backend ghostblog
mode http
server ghostblog.lan 10.0.3.4:2368 check
Note that the HA-Proxy TLS certificate format is actually a combined file of the .crt and the .key file. To create the file, just run
cat sitecert.crt sitecert.key > sitecert_haproxy.pem
So much for testing TLS passthrough and termination. Let’s move on to client certificate authentication.
Client Certificate Authentication
In the TLS passthrough section, client certificate auth will still work if it was enabled on the internal apache server, as everything get’s just passed through including the request for the user to authenticate.
But I’d rather have one, highly available place to do all the config and not care about deploying authentication onto every internal webserver.
Enabling it is pretty straight forward, just append the ca-file and verify parameter to the bind option in your TLS termination section. (Note: $certs/ == /etc/ssl/private/
)
bind *:443 ssl crt $certs/officialcert.pem ca-file $certs/private_ca.crt verify required
Now, users are required to show a certificate that has been signed by myca.crt in order to fully establish the TLS connection. However, right now no one without a valid cert can visit my blog.
frontend https_termination
bind *:8080 ssl crt $certs/officialcert.pem ca-file $certs/private_ca.pem verify optional
mode http
#Update - 17.11.2014
#redirect location / if { path_beg /ghost/ } ! { ssl_fc_has_crt }
redirect location / if { path_beg /ghost/ } ! { ssl_c_used }
default_backend ghost-htsec
backend ghost-htsec
mode http
server ghost-htsec01 10.0.3.57:2368 check
Setting verify optional
basically means that we don’t care if a visitor provides a certificate or not. Adding the redirect
line adds the additional security for the subfolder we want to protect. Now visitors can browse my blog, but only those with a valid cert can go to /ghost/login/
or /ghost/signup/
. The best part is, that those who try to login without a certificate don’t get an error message but instead are redirected to the root homepage /
.
This adds another tiny bit of security by obscuring the login and register pages. Security by obscurity is nothing bad as long as you don’t solely rely on it for protection!
Since this is already quite a bit of haproxyness to take in, I’m going to stop here and publish what I have learned so far. Stay tuned for another post on HA-Proxy, where I will try to tackle my remaining goals.
UPDATE: 17.11.2014
I experienced huge problems with ssl_fc_has_crt
in the past couple of days. The first connection to my host with a valid certificate would always be handled correctly, but a reload of the same page resulted in the redirect that should only be applied to users without a certificate. After going through the HA-Proxy documentation, I found that ssl_c_used
is the better choice. Quote from the (http://www.haproxy.org/download/1.5/doc/configuration.txt)[docs]:
ssl_fc_has_crt : boolean
Returns true if a client certificate is present in an incoming connection over
SSL/TLS transport layer. Useful if 'verify' statement is set to 'optional'.
Note: on SSL session resumption with Session ID or TLS ticket, client
certificate is not present in the current connection but may be retrieved
from the cache or the ticket. So prefer "ssl_c_used" if you want to check if
current SSL session uses a client certificate.
ssl_c_used : boolean
Returns true if current SSL session uses a client certificate even if current
connection uses SSL session resumption. See also "ssl_fc_has_crt".
Update: 21.11.2014
I noticed, that a brand new Firefox profile as well as Firefox mobile on my Android where greeting me with this message instead of my website.
www.hashtagsecurity.com uses and invalid security certificate.
The certificate is not trusted becauzse no issuer chain was provided.
(Error code: sec_error_unknown_issuer)
I found that a little bit strange, since I bought a valid certificate at a Comodo reseller. This is due to Firefox being a bit more strict then other browsers when it comes to TLS implementation.
The fix is to add the ca-bundle certificates to your webserver config, which contains the TLS-Chain certificates.
# In Apache, add this line
SSLCertificateChainFile /etc/ssl/private/servername.ca-bundle
# In Lighttpd, add this line
ssl.ca-file = "/etc/ssl/private/servername.ca-bundle"
In NGINX and HAPROXY, you don’t change the config file. Just add the content of the ca-bundle.crt to your original certificate.
cd /wherever/your/certs/are/
cat servername.ca-bundle >> servername.crt
For HAPROXY, your certificate should look like this:
-----BEGIN CERTIFICATE-----
long-server-cert-string
-----END CERTIFICATE-----
-----BEGIN PRIVATE KEY-----
long-server-key-string
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
long-ca-cert-string-01
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
long-ca-cert-string-02
-----END CERTIFICATE-----
After restarting HAPROXY, the error message went away and Firefox displayed the website just like any other browser.
Links:
Here are a few links that helped me come this far.
- HA-Proxy Repository: http://haproxy.debian.net/
- SSL Termination: https://serversforhackers.com/editions/2014/07/29/haproxy-ssl-termation-pass-through/
- Loadbalancing with SSL: http://virtuallyhyper.com/2013/05/configure-haproxy-to-load-balance-sites-with-ssl/
- SSL Termination in Ubuntu 14.04: https://www.digitalocean.com/community/tutorials/how-to-implement-ssl-termination-with-haproxy-on-ubuntu-14-04
- Detailed example config: http://vmtoday.com/2013/07/updated-load-balancer-virtual-appliance/
- Why won’t SSL work?: http://seanmcgary.com/posts/using-sslhttps-with-haproxy
- HA Proxy Start Parameters: http://stackoverflow.com/questions/6154543/haproxy-which-configuration-files
- HA-Proxy Random Learnings: http://nbevans.wordpress.com/2011/03/03/cultural-learnings-of-ha-proxy-for-make-benefit/
- No ClientCert Redirects (Thanks Babtiste!): http://comments.gmane.org/gmane.comp.web.haproxy/14717
haproxy.com – they have a lot of stuff but I’m not always sure if it’s still accurate!
- SSL Client Certs http://blog.haproxy.com/2012/10/03/ssl-client-certificate-management-at-application-level/
- Native SSL: http://blog.haproxy.com/2012/09/04/howto-ssl-native-in-haproxy/
- Another native SSL: http://blog.haproxy.com/2012/09/10/how-to-get-ssl-with-haproxy-getting-rid-of-stunnel-stud-nginx-or-pound/