Linux Host Security - SSH

A series about server hardening…

This series is probably going to evolve as we progress through it, with modern methods of serving applications (containers), a series on how to secure an Apache host doesn’t really seem fitting at this stage.

For this chapter of the series we’ll start with SSH, and how we can secure our infrastructure.

SSH does an OK job at being secure out-of-the-box, but there are a number of things we can tweak - and it’s strongly advised to do so - to increase the overall security posture of your environment.

These steps alone, will not guarantee that your server won’t be attacked, although all these items together will be make things more difficult. Basically, your ‘fence’ needs to be taller than your neighbours'.

I’m running a CentOS 8 system, so keep in mind that SELinux is enabled and enforcing. We’ll dig into SELinux in a future chapter of this series.

No root login

First thing we do, is disable root login. It is enabled by default. Yes, root user doesn’t have a password set, by default - so remote users won’t be able to log in anyway.

But if root is given a password in future, we don’t want anyone to be able to log in remotely using the root user.

Default:

$ sudo grep -i permitroot /etc/ssh/sshd_config
PermitRootLogin yes

Changing this value:

• With a sudo user, we’re going to edit the /etc/ssh/sshd_config file
• and change the PermitRootLogin parameter to no, save and quit the text editor
• and restart the sshd service

$ sudo vim /etc/ssh/sshd_config

PermitRootLogin no
(save and quit vim)

$ sudo systemctl restart sshd

The following should go without saying; but ensure that you have your own user created - and you’re not logging in with - and using the root user to perform all your admin tasks. After performing the above steps, and once you disconnect, you won’t be able to connect with the root user again.

Change SSH port

By exposing ports to the internet, your server stands a chance of being attacked. An exposed port is an attackers way in. So taking a few extra steps to make things more difficult for attackers, can only be beneficial to our environment.

It’s common knowledge that specific services have default ports that they run on. By default, SSH runs on port 22. We can change this port, to a much higher value. Using non-default values, will increase the difficulty of penetrating our defences.

SSH is one of the primary ports that attackers use to gain access to your server. Check this out. Most cyber attacks focus on just three ports, and these are 22, 80, and 443. By default these ports are used for SSH, HTTP, and HTTPS.

Cool, so let’s change our default SSH port. We can do this in the same file as in the previous topic.

• With a sudo user, we’re going to edit the /etc/ssh/sshd_config file
• and change the #Port 22 parameter. We’re simply uncommenting the line by removing the ‘#’, and changing the 22, to a value of your choice. This port number should not be in use already, and preferrably a higher value than 10,000. Save and quit the text editor.
• Inform SELinux of this port change
• and restart the sshd service

$ sudo vim /etc/ssh/sshd_config

Port 61613
(save and quit vim)

$ sudo semanage port -a -t ssh_port_t -p tcp 61613

$ sudo systemctl restart sshd

If you don’t let SELinux know that SSH should be allowed to run on the new port, then it will prevent the sshd service from starting.

Taking care of the firewall

CentOS 8 now runs firewalld by default, and we’ll have to take care of this configuration as well. A firewall-cmd --list-all will show the current firewall rules, and we can see that ssh is allowed.

$ sudo firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens33
  sources:
  services: cockpit dhcpv6-client ssh
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

These services as we can see in the example above, are predefined, and are based on default service ports. We have two options here.

1. We can create and add our own service (with our own port), and remove the default one.
2. We can just add the new port number, and remove the predefined service.

Option1: Create our own firewalld service

It’s best to create your own services in the /etc/firewalld/services/ directory. Creating an xml file in the below format, in this directory, and reloading the firewalld service will add these rules/ports to the predefined services list. The XML file for our service needs to be in the following format:

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>$SHORTDESCRIPTION</short>
  <description>$DESCRIPTION</description>
  <port protocol="$PROTOCOL" port="$PORTNUMBER"/>
</service>

Completing the above parameters for our custom service, our XML file will look like this:

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>custom-SSH</short>
  <description>Custom SSH service to allow connections over SSH on the new port</description>
  <port protocol="tcp" port="61613"/>
</service>

After restarting our firewalld service (sudo systemctl restart firewalld) we can see our custom rule in the list of predefined services. Note: The name of the service is defined by the name of the xml file, I’ve named the file custom-ssh.xml.

$ sudo firewall-cmd --get-services | grep custom
[...]
custom-ssh
[...]

This service is added to the firewall config just like any other service. Remember that we should remove the default SSH service as well.

$ sudo firewall-cmd --permanent --add-service=custom-ssh

$ sudo firewall-cmd --permanent --remove-service=ssh

$ sudo firewall-cmd --reload

$ sudo firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens33
  sources:
  services: cockpit custom-ssh dhcpv6-client
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

Option2: Add the new port number, and remove the predefined service

• Add the new port number persistently
• Remove the predefined service
• Reload the firewall configuration

$ sudo firewall-cmd --permanent --add-port=61613/tcp

$ sudo firewall-cmd --permanent --remove-service=ssh

$ sudo firewall-cmd --reload

$ sudo firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens33
  sources:
  services: cockpit dhcpv6-client
  ports: 61613/tcp
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

That should be it. For both of these options, to test, don’t disconnect just yet. From another terminal window, ssh to the new port.

$ ssh -p 61613 <user>@<host>

Note: The <host> is a placeholder for an IP address.

SSH key-based authentication

Building on the previous two topics, we can also enable key-based authentication to our system. For this section, we’re going to use two systems, a ‘client’, and a ‘server’. The ‘server’ in my setup is the same node on which we’ve made the port, and firewall changes, as well as the no root login changes. This is the destination, the server we want to connect to, and the client is the node we’re using to connect to that server. In my setup, my client is running CentOS as well.

SSH keys

So an SSH client can authenticate to a server either using passwords, or SSH keys. Passwords are less secure and not recommend as the sole authentication method. This is because passwords can easily be shared, automated malicious bots will often launch brute-force attacks - This is when a malicious user or automated script repeatedly attempts to authenticate to password-based accounts.

SSH keys consist of a pair of cryptographic keys which can be used for authentication. The set is made up of a public and a private key. The public key is just that, public, and can be shared freely without it being a security issue. Although the private key must be kept safe and never shared.

Back to our client and server setup:

The client holds the private key, and the server has the public key.

1. The server will use the public key to encrypt a random string of characters and pass this encrypted string to the client.
2. The client will decrypt the message using their private key and joins the unencrypted string with a session ID - that was negotiated previously.
3. The client then generates an MD5 hash of this value (string+ID) and sends this message back to the server.
4. The server already had the original message and the session ID, so it can compare the MD5 hash generated by those values and if they match it obviously means that the client is valid, and has a copy of the private key.

Ok, so… We need to generate these keys for our setup.

Before elaborating on the commands for a specific section, I will specify if these are to be done on the client or the server.

We’re going to start on the client, where we will need to generate our SSH keys from. Afterwhich we will copy the public key to the server that we want to authenticate to.

Client

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/how2cloud/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/how2cloud/.ssh/id_rsa.
Your public key has been saved in /Users/how2cloud/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:udSFJ6G24Til1CwXz1vayxqORtiUlpFU/krh8s2ZMcI how2cloud@localhost

$ ls -lah ~/.ssh/ | grep how2cloud
-rw-------    1 how2cloud  how2cloud   2.5K Apr 13 12:23 id_rsa
-rw-r--r--    1 how2cloud  how2cloud   573B Apr 13 12:23 id_rsa.pub

With the keys generated, we can copy the public side of the key (the .pub file) to our server, for the specific user that we’ll be connecting to. Meaning the SSH keys are user-specific. This is important. As my keys for my user is separate from your user and keys. We can use the ssh-copy-id command which will add the key in the right place on the server-side.

$ ssh-copy-id -i /Users/how2cloud/.ssh/id_rsa.pub <user>@<host> -p 61613
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/Users/how2cloud/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Password: **********

Number of key(s) added:        1

Now try logging into the machine, with:   "ssh 'myuser@<host>'"
and check to make sure that only the key(s) you wanted were added.

From here we can log in again, specifying the private key in our command, and don’t forget the port number as well… We will be logged in without being prompted for our user’s password. If you added a password to your key-file, you will be prompted for this password when connecting. More on this in a bit…

$ ssh -i /Users/how2cloud/.ssh/id_rsa myuser@<host> -p 61613
[myuser@server ~]$

As you can see in the example, if we use the key-pair with the associated user, we can access our server without a password.

If the user is prompted for their user’s password as well, it will add another layer of protection. Usually we will disable password-based authentication when we enable key-based authentication, however for a more secure system, we’re going to need both.

Server

Add the following line, it doesn’t exist in a comment, we’ll have to add it.

sudo vim /etc/ssh/sshd_config
AuthenticationMethods publickey,password

After restarting sshd service we see the following behaviour

$ ssh -i /Users/how2cloud/.ssh/id_rsa myuser@<host> -p 61613
myuser@server's password: **********

[myuser@server ~]$ exit
logout

$ ssh myuser@<host>
myuser@server: Permission denied (publickey).

Specifying the ssh-key we’re still prompted for a password, and in the second attempt, we omit the key, and we’re denied access to the system.

Client

Something to note here; if the key-file has a password set, an initial prompt for that password is required before ssh will attempt to connect. The password required here: myuser@server's password: is the MYUSER’s password on the system we’re connecting to.

If we did add a password to our file the process will look like this:

$ ssh -i /Users/how2cloud/.ssh/id_rsa myuser@server -p 61613
Enter passphrase for key '/Users/how2cloud/.ssh/id_rsa': <key-file password goes here>
myuser@server's password: <user's password goes here>

Awesome… Now we have two ‘components’ that are required for a successful connection to our system. A password, that we should know (don’t write it down), and a key-file that is stored on our system. It’s up to you whether or not the key-file is protected by a password.

Ideally, we don’t want to specify the port number every time we connect, and if we have multiple identity files for multiple servers, we can specify each host with the SSH ports and identity files that accompany it. This is done on the client’s side, in their /etc/ssh/ssh_config file.

Host <host>
  IdentityFile ~/.ssh/id_rsa
  Port 61613

The <host> is either a DNS name or an IP address.

Quick Recap

Let’s have a quick recap; Up to this point, we’ve done a few things to secure our SSH connection. We have:

• We prevented the root user from logging in remotely
• We have changed the port that our SSH service is running on
• We’ve enabled key-based authentication

We can take our security even a step further; Multi-Factor Authentication

Multi-Factor Authentication

I have found that Google Authenticator is the easiest to work with on CentOS, and it provides everything you need in a multi-factor auth app. Before we carry on, just a little on what we’re going to do. Multi-Factor authentication, also sometimes referred to as 2-step auth, is the process of supplying another pin number or password, that is sent to you by the party/service/bank that you’re trying to auth to. This ensures that they know that you have access to a specific communication method (either an email address, or an SMS, or via a secure app). To elaborate on this, this is the same as when a bank, or e-commerce platform, would send an OTP to your phone to ensure that you have access to the phone number that they have on record.

Installing Google Authenticator

So how this is going to work is, we’re going to install the Google Authenticator service, our CentOS machine, but also the Google Authenticator app on our smartphone. Once this is installed on both components, we will sync the two, ensuring that our service is expecting the same code as what the app is giving us, and then we’re going to tell our pam service to prompt for this when we log in with SSH.

First, we install the EPEL repository on our CentOS machine, after which we can install the google-authenticator service.

$ sudo yum install epel-release

$ sudo yum install google-authenticator qrencode-libs

Browse your smart phone’s app store for the Google-Authenticator app, and install it.

Now, with Google-Authenticator installed, we can configure it. Something to note here, and this is something I struggled with; this service is, of course, user-specific. If you’re going to configure this with ‘sudo’ prepended to the command, it’s going to configure it for the root user, and if you’ve been following this post, we disabled that functionality earlier. So just execute the command, without sudo

$ google-authenticator    

This command will drop us into an interactive config menu. After answering yes to the first question, we’re going to link the app and google auth service. We’re presented with a URL and a secret key, this is an ‘either-or’ scenario. A quick, easy and less secure way, or a slightly longer and more secure method. The easy and less secure way, is we can either simply copy and paste this URL into a browser, this presents a QR code, and from the app we can tap this plus sign, and select ‘Scan Barcode’ option, scan the QR code in the browser, and we’re done. Or, for the slightly longer and more secure way, we can do a manual entry in the app, and enter the secret key that the interactive menu provides. Once that’s successfully entered, you need to enter the code that the app provides (676767 in my example) into the interactive menu. This will link the app and the service. The interactive menu will give us 5 emergency codes, these are used for offline auth. It’s probably best to save them somewhere.

Do you want authentication tokens to be time-based (y/n) y

Warning: pasting the following URL into your browser exposes the OTP secret to Google:
  https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/
  how2cloud@localhost.localdomain%3Fsecret%3XXMMXXNWVBU1Q6EXXXXXML26OIA%26issuer%3Dlocalhost.localdomain

[...QR CODE...]

Consider typing the OTP secret into your app manually.
Your new secret key is: XXMMXXNWVBU1Q6EXXXXXML26OIA
Enter code from app (-1 to skip): 676767
Code confirmed
Your emergency scratch codes are:
  11111111
  22222222
  33333333
  44444444
  55555555

Do you want me to update your "/home/how2cloud/.google_authenticator" file? (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y

By default, a new token is generated every 30 seconds by the mobile app.
In order to compensate for possible time-skew between the client and the server,
we allow an extra token before and after the current time. This allows for a
time skew of up to 30 seconds between authentication server and client. If you
experience problems with poor time synchronization, you can increase the window
from its default size of 3 permitted codes (one previous code, the current
code, the next code) to 17 permitted codes (the 8 previous codes, the current
code, and the 8 next codes). This will permit for a time skew of up to 4 minutes
between client and server.
Do you want to do so? (y/n) y

If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting? (y/n) y

A few things left to do is to configure pam to require a successful authentication from google-auth, tweak sshd service, and then restarting the SSHD service. I added this line to the top of the /etc/pam.d/sshd file. And this causes SSHD to request the ‘Verification code’ first, before the user’s password.

In the sshd config file, change the ChallengeResponseAuthentication parameter to yes, and change the AuthenticationMethods parameter to password publickey,keyboard-interactive (note: no comma between password and public key)

$ sudo vim /etc/pam.d/sshd
auth       required     pam_google_authenticator.so     ~/.google_authenticator
(save and quit vim)

$ sudo vim /etc/ssh/sshd_config
ChallengeResponseAuthentication yes
AuthenticationMethods password publickey,keyboard-interactive

$ sudo systemctl restart sshd

Don’t disconnect from your server, maybe open a new window and let’s test this out. If you’re getting errors, you can do a sudo tail -f /var/log/secure in another window (on the server) and see what error you’re getting.

Note that we still need the port and the ssh-keys in our command, if we didn’t make use of the ssh_config option as described earlier in this post.

$ ssh -p 61613 -i /Users/how2cloud/.ssh/id_rsa myuser@<serverIP>
Verification code:   <code from google auth app>
Password:    <password for 'myuser' user>

Conclusion

Cool… With these security implemented, added we can be sure that our system is more secure, and we should be able to sleep a bit better knowing that we’ve improved our system’s security posture.