Load Balancing OpenSSH SFTP with HAProxy

In my previous post I described how we setup a Ubuntu Server (12.0.4) as an OpenSSH SFTP server.

In this post I am going to describe how I have load balanced 2 SFTP servers using HAProxy.

I will assume that we have 2 sftp Ubuntu servers with IP addresses of 192.168.10.1 & 192.168.10.2

We then need to spin up a new Ubunutu server and install the HAProxy package. This new server should have 2 NICs installed, one for management of the server and another for load balancing the SSH (port 22) connection. I should note here that I am using VMWare VM's for all of this work. In this example the management IP will be 192.168.10.100 and the IP address for the load balancing will be 192.168.10.50

apt-get install haproxy

once haproxy is installed there are a few configuration changes that need to be made for this to work. The first is in /etc/ssh/sshd_config where we need to ensure the ListenAddress is set to the management IP of 192.168.10.100 - if this is left at the default of 0.0.0.0 haproxy will not be able to bind to port 22.

sudo vi /etc/ssh/sshd_config
ListenAddress 192.168.10.100         

We also need to enable haproxy so that it starts automatically by editing the file shown below and ensuring 'Enabled' is set to 1

sudo vi /etc/default/haproxy

Enabled=1

Now we need to configure haproxy, edit the /etc/haproxy/haproxy.cfg file. I do this by deleting all content of this file and replacing it with my own, delete all content and then paste in the content shown below

sudo vi /etc/haproxy/haproxy.cfg
# config needs haproxy-1.1.28 or haproxy-1.2.1

global

        maxconn 4096

        daemon

defaults

        contimeout      5000

        clitimeout      50000

        srvtimeout      50000

listen  SSHLB 192.168.10.50:22

        mode tcp

        option tcplog

        balance roundrobin

        server  sftp01 192.168.10.1:22

        server  sftp02 192.168.10.2:22

At this stage I would reboot the LB server although you could probably get away with restarting haproxy and ssh.

Now if you make a connection to the LB address (192.168.10.50) over TCP port 22 your connection will be load balanced between the 2 servers using round robin. You will quickly see that this does not work as the 2 servers have different RSA Host Keys and once you have added a host to your known hosts file you will receive and error when this key changes i.e. when you are load balanced to another server.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Someone could be eavesdropping on you right now (man-in-the-middle attack)!

It is also possible that the RSA host key has just been changed.

The fingerprint for the RSA key sent by the remote host is

XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX.

Please contact your system administrator.

To overcome this issue you have to copy the private and public keys from one load balanced server to another. Copy the key files located in /etc/ssh/ and replace them in the other server. Make sure that the permissions are identical after copying as I found that this caused issues. Now both servers have the same Host Keys and the load balanced connection should operate without any issues. 

Please note that I am unsure if replacing these keys may cause other issues but so far the hosts appear to continue to operate normally. Further testing is required of this solution and I will update this post if any issues arise.

Configuring Ubuntu 12.0.4 with OpenSSH as an SFTP server.

We have a requirement to allow a customer to submit files securely to us via SFTP using Public-Key authentication, so.....

In this example I have given the Ubuntu server a hostname of testsftp and an IP address of 192.168.10.10/24. It is a standard install with the openssh-server package selected when prompted during setup. You can also run sudo apt-get install openssh-server when the server is up and running.

Make a copy of any files you are going to change and change the permissions on the copied file to read only.

sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.original

sudo chmod a-w /etc/ssh/sshd_config.original

View the /etc/ssh/sshd_config file and ensure the line shown below is present and has not been commented out.

PubkeyAuthentication yes

Create new local user that will use the SFTP service

sudo useradd -d /home/testuser -m testuser

sudo passwd testuser <password>

Generate the Public/Private key to be used with the user account. In my case I used the built in features within SecureCRT to generate an the Key Pair. The Unix utility ‘SSH-Keygen’ can also be used to generate the key pair. We will call the public key ‘public.pub’ and the private key ‘private’ 

Always ensure that the private key is protected with a very strong password.

ssh-keygen -b 1024 -t rsa

Next we need to copy the public key to the host

scp public.pub testuser@192.168.10.10:

password: ***********

Now we need to log in to the remote host as the ‘testuser’

ssh  testuser@192.168.10.10

password: ***********

Once we are connected to the host server we need to create some directories (if they don’t currently exist), set the appropriate permissions  and add the newly created public key to the authorised keys for the testuser.

testsftp$ mkdir -p ~/.ssh

testsftp$ chmod 700 ~/.ssh

testsftp$ cat public.pub >> ~/.ssh/authorized_keys     # Add public key to authorised key

testsftp$ chmod 600 ~/.ssh/authorized_keys

testsftp$ mv public.pub ~/.ssh      #Move public key to the .ssh folder

Now if you log back out and back in again using the same command as before you will be asked to enter the passphrase for the private key instead of your user password. Ensure your private key is in the correct location for whatever client you are using, with SecureCRT this is done through the ‘Manage Agent Keys’ tool.

ssh  testuser@192.168.10.10

Enter passphrase for key '/home/testuser/.ssh/private': *************

So now we have a local user who is able to authenticate against the testsftp server using a public private key pair. What we now want to do is restrict what this user can do to just sftp commands with their own directory and nothing else.

Modify /etc/ssh/sshd_config to include the line

             Subsystem       sftp    internal-sftp

At the end of the sshd_config file add the following content. This ensures any user who is a member of the sftponly group will not be able to leave their home directory and will be restricted to running the SFTP process.

Match Group sftponly

        ChrootDirectory %h

        ForceCommand internal-sftp

 AllowTcpForwarding no

Create the new user group and add testuser to that group

groupadd sftponly

usermod –g sftponly testuser

Deny the user Shell access

usermod –s /bin/false testuser

We now need to verify folder permissions. The users home directory must be owned by root as does the /home/ folder.

chown root:root /home/testuser/

Now when the user ‘testuser’ logs in using and SFTP client they will only be able to browse their home directories. Ensure that the ‘testuser’ account has valid permissions for the operations they need to perform when connected i.e. if they need to upload a file then they will need write permissions somewhere in their home directory.

Attempt to make a direct SSH connection to the host and you will see that the connection is refused.

References (thanks):

https://help.ubuntu.com/8.04/serverguide/user-management.html

http://archive.networknewz.com/networknewz-10-20030707AuthenticatingbyPublicKeyOpenSSH.html

http://www.romuloaraujo.com/2010/01/how-to-restrict-ssh-sftp-user-only.html