When it comes to securing Linux systems, Secure Shell (SSH) has been one of the most dependable tools at a sysadmin's disposal. However, with every technology solution, there are potential pitfalls that need to be carefully managed for security. In the case of SSH, key mismanagement can cause immediate risk of mass compromise.

In this article, we will discuss some of the primary risks related with SSH keys not only on Linux, but any system using SSH for remote access. Those risks are:

• Stolen private keys

• Malicious keys

• Duplicate keys

• Orphan keys

• Excessive keys

• Unsecured private keys

• SSH key options

• authorized_keys2 files

Stolen Private Keys

SSH private keys are the crown jewels of remote access and a stolen key is a severe risk. Intruders can use stolen keys to impersonate users, access sensitive data, and take total control of a system. They also allow attackers to press lateral movement attacks and move onto other systems quickly and quietly.

We have seen many instances where poorly secured SSH keys can lead to compromise such as the following:

• Left in a directory with world readable permissions.

• Posted in chats like Slack.

• Sent in email.

• Lost/misplaced USB drives.

• Compromised administrator or developer systems.

• Leakage from system automation tooling leaving them behind in exposed directories.

The biggest problem with stolen keys is that there is no evidence they've been stolen. Copying data is completely silent. The only indication most organizations may see are some unusual logins if they are paying attention. Even worse, as SSH keys have no expiration themselves, they can work for quite some time unless they are rotated frequently.

Counteracting stolen keys is best handled by simply not using them. SSH key certificates are a superior solution as they can be set to have a short expiration time (e.g. a few minutes). A stolen certificate simply goes bad and won't work.

If you are using SSH keys (as most are), strategies such as periodic rotation and making sure they are securely handled is a must. If you can prevent personnel from handling them at all through the use of a credential vault this is even better. However, this level of care is difficult to manage in many organziations where keys hang around for many years. The best you can do is implement policies and measures to make sure keys are not allowed to exist beyond a certain age or move to SSH certificates (and here).

Finding Private Keys

First we'd recommend just looking on your systems for private keys hanging around where they shouldn't be. In particular, private keys in a user's home directory on a shared system is a major risk. If any account gets compromised, attackers will immediately search for private SSH keys lying about for all users.

Finding private keys can be as simple as this script below. The keys may be encrypted, but still it would be worth asking if the user requires their private key to be on a system, especially if that system is shared with multiple users. Sandfly users can enable a policy scan that will check for private keys for all users automatically.

#!/bin/bash

# This script needs to be run as root to be able to read all user's .ssh directories
if [ "$(id -u)" != "0" ]; then
   echo "This script must be run as root" 1>&2
   exit 1
fi

# Get list of all home directories
home_dirs=$(awk -F':' '{ print $6 }' /etc/passwd)

for dir in $home_dirs; do
    # Check if the .ssh directory exists
    if [ -d $dir/.ssh ]; then
        # Find all files in the .ssh directory that contain the word "PRIVATE"
        private_files=$(grep -lR "PRIVATE" $dir/.ssh 2>/dev/null)

        # If private_files is not empty, report
        if [ ! -z "$private_files" ]; then
            echo "User with home directory $dir has files in their .ssh directory that are likely private keys:"
            echo "$private_files"
        fi
    fi
done

Malicious Keys

Malicious SSH keys are inserted by adversaries to allow backdoor access. Logging onto a box over SSH by stuffing a new key into a user's authorized_keys file is simple and effective. Most users would never know a new key has shown up.

This script will locate any authorized_keys files modified in the past 24 hours, but it needs to be setup to run on all your systems. Other fleet management tools such as OSQuery can also be setup to search for recently modified files and should do so if you run it. Sandfly also has policy alerts that will sweep for this activity as well.

#!/bin/bash

# This script needs to be run as root to be able to read all user's .ssh directories
if [ "$(id -u)" != "0" ]; then
   echo "This script must be run as root." 1>&2
   exit 1
fi

# Get list of all home directories
home_dirs=$(awk -F':' '{ print $6 }' /etc/passwd)

# Get the current time
now=$(date +%s)

for dir in $home_dirs; do
    # Check if the authorized_keys file exists
    if [ -f $dir/.ssh/authorized_keys ]; then
        # Get the last modification time of the file
        mtime=$(stat -c %Y $dir/.ssh/authorized_keys)

        # Calculate the difference in seconds between now and the file's mtime
        diff=$((now - mtime))

        # If the file was modified in the last 24 hours (86400 seconds), report
        if [ $diff -le 86400 ]; then
            echo "User with home directory $dir has modified their authorized_keys file in the last 24 hours."
        fi
    fi
done

In whatever way you choose, monitoring for modified authorized_keys will find compromised credentials quickly that otherwise would go unnoticed.

Duplicate Keys

Look at this image below. It has a duplicate SSH key that will allow immediate login. How quickly can you spot it?

Actually in the image above, it has three duplicate keys. It's very difficult for a human to determine if a key has a duplicate by taking a quick look (let alone doing it across many systems and users).

A duplicate SSH key present in authorized_keys is what happens when a key is present two or more times. What that means is if you delete a key with the intent of disallowing future logins, the unnoticed duplicates will still allow the key holder to login to the host.

We see a number of customers that have duplicate keys detected by Sandfly. It is very easy to have happen with just some of the mistakes below:

• User pastes in a key that is already present but they didn't know.

• A system automation tool managing key files makes an error and inserts a key one or more times.

• A backup is restored that has an old authorized_keys file with a duplicate key present.

• A piece of malware inserts a backdoor key one or more times because they are not checking if they have already dropped their key.

Duplicate keys are a unique threat because unlike changing a password, removing a key may not prevent login if it still remains. Because of the difficulty in spotting this kind of situation it must be automated.

Sandfly looks for this activity by default, but this script will search for duplicate keys right now:

#!/bin/bash

# This script needs to be run as root to be able to read all user's .ssh directories
if [ "$(id -u)" != "0" ]; then
   echo "This script must be run as root." 1>&2
   exit 1
fi

# Get list of all home directories
home_dirs=$(awk -F':' '{ print $6 }' /etc/passwd)

for dir in $home_dirs; do
    # Check if the authorized_keys file exists
    if [ -f $dir/.ssh/authorized_keys ]; then
        # Sort keys, count duplicates, and print any with count > 1
        echo "Processing $dir/.ssh/authorized_keys."
        sort "$dir/.ssh/authorized_keys" | uniq -c | while read count key
        do
            if [ "$count" -gt 1 ]; then
                echo "$key is duplicated $count times"
            fi
        done
    fi
done

Orphan Keys

Orphan keys are those that remain on a system after the corresponding user has been removed, or their access is no longer required. These keys offer a backdoor entry point if they get compromised. This risk can be mitigated by setting up proper key de-provisioning procedures and regular audits to clean up any orphan keys.

If you have used SSH authorized_keys for any period of time you likely have orphan keys right now. The only way to know is to manually review your authorized_keys files and remove keys you know are no longer needed. It will work better in most cases to simply get a list of keys you know should be allowed access as a user and wipe out the older authorized_keys file and start fresh.

While looking at the timestamp on the authorized_keys file can show you the last time a key was added, it won't give you the dates of older keys so you cannot assign any time to when they first appeared. The only way to determine how long a key has been around is if you were tracking them centrally and can use the first time the key appeared as an anchor value across hosts (like Sandfly SSH Hunter does). File timestamps alone are not going to help you find older keys here.

This is a tedious task and is one of the biggest perils of authorized_keys for access management.

Excessive Keys

Users that have too many keys in the authorized_keys file are another potential attack vector. This causes a multitude of problems, the main one being that many people can log in as the same username and this makes auditing much harder. But also it leads to orphan keys (discussed above). Often nobody goes back to remove old keys and the list can grow and grow.

What is an excessive number of keys? Well, we think one key for one user is the limit. But in reality it just depends on your organization. We have seen usernames with nearly a hundred keys that allowed people to login as them (e.g. git or root). We feel this is very likely to lead to compromise. We recommend to keep the keys each user allows for login as short as you can.

This script below will search all local users home directories for authorized_keys files that have 10 or more keys present as a starting point.

#!/bin/bash

# This script needs to be run as root to be able to read all user's .ssh directories
if [ "$(id -u)" != "0" ]; then
   echo "This script must be run as root." 1>&2
   exit 1
fi

# Get list of all home directories
home_dirs=$(awk -F':' '{ print $6 }' /etc/passwd)

for dir in $home_dirs; do
    # Check if the authorized_keys file exists
    if [ -f $dir/.ssh/authorized_keys ]; then
        # Count the number of keys (lines) in the file
        num_keys=$(wc -l < $dir/.ssh/authorized_keys)

        # If 10 or more keys, report
        if [ $num_keys -ge 10 ]; then
            echo "User with home directory $dir has $num_keys keys in their authorized_keys file."
        fi
    fi
done

Unsecured Private Keys

A common mistake is not encrypting private keys. If private keys are not encrypted they can be easily abused if stolen. Keys should be encrypted at rest, and the storage medium (such as a filesystem) should be properly secured as well. The passphrase used to secure a key should be strong.

You may also want to increase the rounds used to check the passphrase to slow down brute force attempts (ssh-keygen -a flag) in case the key is stolen.

It's important to remember that passphrases are a stopgap measure. If an attacker has access to your system then you must assume they have stolen your passphrase with a keylogger or they can launch a brute force attack offline. However as an additional safety measure it does help to have them encrypted in case the keys are lost in other ways (such as stolen laptop, lost USB drive, etc.).

SSH Key Options

The use of SSH key options is another article in and of itself, but in general be wary of the following options and how they are being used. A full list of options is available in the man pages for OpenSSH, but these options in particular need to be closely watched:

• agent-forwarding - Can allow someone controlling the remote system to impersonate the user logging in and it should be configured correctly to be secure.

• command - Can allow someone to run commands as the user when they login with this key.

• environment - Can allow setting of environment variables for the user logging in with the key. This can cause security issues if the remote server allows them to be passed into the terminal session.

• port-forwarding - Can allow client system to forward ports bypassing firewalls and network controls.

• user-rc - Can allow system to run ~/.ssh/rc upon login to execute commands as the user.

• X11-forwarding - Permits X11 protocol to be forwarded over SSH, again bypassing firewall and network controls.

You can set the restrict option on all keys to automatically disable most of the above and other potentially risky options available in the future.

SSH authorized_keys2 Files

No, that's not a typo. There is an SSH authorized_keys2 file that works like authorized_keys but is deprecated. If this file exists, it may still allow login on older servers. Its presence is almost always a malicious attempt to conceal access.

If you see this file in a user's home directory you should be asking some pointed questions. If the user doesn't know why the file is there then the system should be immediately investigated. Again on newer systems it may not work, but it's still worth investigating if it is present because someone put it there thinking it may work.

This script will locate authorized_keys2 files under any local user's directory:

#!/bin/bash

# This script needs to be run as root to be able to read all user's .ssh directories
if [ "$(id -u)" != "0" ]; then
   echo "This script must be run as root." 1>&2
   exit 1
fi

# Get list of all home directories
home_dirs=$(awk -F':' '{ print $6 }' /etc/passwd)

for dir in $home_dirs; do
    # Check if the authorized_keys2 file exists
    if [ -f $dir/.ssh/authorized_keys2 ]; then
        echo "An authorized_keys2 file was found at: $dir/.ssh/authorized_keys2."
    fi
done

Automate SSH Threat Hunting on Linux

As you can see, there are a myriad of ways that SSH keys can be abused and often using scripts or other tools to check for these problems is tedious and prone to miss serious risks. This is why we have Sandfly and our SSH Hunter.

SSH Hunter can find the above attacks plus many others against your SSH infrastructure. Sandfly puts a very sharp focus on SSH keys because the abuse potential is extremely high. Being able to monitor this critical asset agentlessly means you can get a handle on your SSH security. If you want to see it in action you can watch the video below or contact us for a demo.

Conclusions

SSH is a massive step up in security over legacy protocols like Telnet. However, the management of SSH keys is notoriously difficult and open to theft. If your organization does not monitor SSH key usage, there is a high chance a compromise could happen at some point.

We strongly recommend monitoring SSH keys for abuse as it is a silent stalker for intruders wishing to get into Linux systems and remain undetected. Be sure to use the techniques listed above to check your systems, or contact Sandfly if you want a full trial of our agentless Linux security platform and have us find your SSH key problems for you instantly.