Load Balancing OpenSSH SFTP with HAProxy

In my previous post I described how we setup a Ubuntu Server (12.0.4) as an OpenSSH SFTP server.

In this post I am going to describe how I have load balanced 2 SFTP servers using HAProxy.

I will assume that we have 2 sftp Ubuntu servers with IP addresses of 192.168.10.1 & 192.168.10.2

We then need to spin up a new Ubunutu server and install the HAProxy package. This new server should have 2 NICs installed, one for management of the server and another for load balancing the SSH (port 22) connection. I should note here that I am using VMWare VM's for all of this work. In this example the management IP will be 192.168.10.100 and the IP address for the load balancing will be 192.168.10.50

apt-get install haproxy

once haproxy is installed there are a few configuration changes that need to be made for this to work. The first is in /etc/ssh/sshd_config where we need to ensure the ListenAddress is set to the management IP of 192.168.10.100 - if this is left at the default of 0.0.0.0 haproxy will not be able to bind to port 22.

sudo vi /etc/ssh/sshd_config
ListenAddress 192.168.10.100         

We also need to enable haproxy so that it starts automatically by editing the file shown below and ensuring 'Enabled' is set to 1

sudo vi /etc/default/haproxy

Enabled=1

Now we need to configure haproxy, edit the /etc/haproxy/haproxy.cfg file. I do this by deleting all content of this file and replacing it with my own, delete all content and then paste in the content shown below

sudo vi /etc/haproxy/haproxy.cfg
# config needs haproxy-1.1.28 or haproxy-1.2.1

global

        maxconn 4096

        daemon

defaults

        contimeout      5000

        clitimeout      50000

        srvtimeout      50000

listen  SSHLB 192.168.10.50:22

        mode tcp

        option tcplog

        balance roundrobin

        server  sftp01 192.168.10.1:22

        server  sftp02 192.168.10.2:22

At this stage I would reboot the LB server although you could probably get away with restarting haproxy and ssh.

Now if you make a connection to the LB address (192.168.10.50) over TCP port 22 your connection will be load balanced between the 2 servers using round robin. You will quickly see that this does not work as the 2 servers have different RSA Host Keys and once you have added a host to your known hosts file you will receive and error when this key changes i.e. when you are load balanced to another server.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Someone could be eavesdropping on you right now (man-in-the-middle attack)!

It is also possible that the RSA host key has just been changed.

The fingerprint for the RSA key sent by the remote host is

XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX.

Please contact your system administrator.

To overcome this issue you have to copy the private and public keys from one load balanced server to another. Copy the key files located in /etc/ssh/ and replace them in the other server. Make sure that the permissions are identical after copying as I found that this caused issues. Now both servers have the same Host Keys and the load balanced connection should operate without any issues. 

Please note that I am unsure if replacing these keys may cause other issues but so far the hosts appear to continue to operate normally. Further testing is required of this solution and I will update this post if any issues arise.