Tearing Apart a Datto Device

Datto devices are becoming a popular backup solution for small to medium sized businesses. They are easy to use and well equipped out of the box. We recently found ourselves in an engagement where one of these devices was accessible via the LAN. Gaining access to backups is a bit of a goldmine during an assessment; unrestricted access to file shares, configuration information, extracting hashes from the NTDS.dit file, and a multitude of other things. Anyone familiar enough with Datto devices knows you have a few useful restore options like mounting backups over open SMB shares or even starting your own VM . Let’s review what we found.

Initial Access

These Datto devices have quite a few attack surfaces out of the box, a web control panel for management, SSH is configured, and VNC is enabled by default. The authentication password for VNC, “Northern”, is well documented by Datto, so that didn’t take long to find. This allowed us to connect to the device remotely and luckily, the user was already logged in! However, VNC access isn’t very stealthy, and we wanted SSH access to the device instead.

Via VNC access we noticed the “root” user is managed by Datto. By looking through logs, it appears these accounts have their passwords changed frequently (once/day), so they aren’t prime targets for attacks directly. However, they do appear to get used so Datto must be tracking these changes somewhere. Hmmm…

Moving on, “backup-admin” is an account created for the client use. The password to this account can be viewed in the partner portal (https://auth.dattobackup.com/simplesaml), where clients can also view device status, backup progress, and initiate a remote web connection to the device. This might be an interesting attack vector as well, but we’ll focus on more local attacks.

Another account, “aurorauser” looks interesting and potentially promising. This account is an unprivileged user account, used for automatic login on boot…providing the easy VNC access. Unfortunately, the SSH password didn’t match the VNC password for this user. However, with a bit of research the default password to the account was found, “NorthernLight$”. Throw it into SSH and we’re in!

Pulling out the root

The Datto device is really just an Ubuntu Linux install with LAMP and bash scripts providing a web console and dashboard intelligence. SSH tunnels and rsync are used to connect back to Datto for off-site duplication. Running ‘uname -ar’ shows the OS is Ubuntu 10.04, released February, 2012 with an EOL May, 2013 for desktops and April, 2015 for servers. Privilege escalation was trivial thanks to a CVE-2012-0056. The exploit used was Mempodipper, courtesy of Jason Donenfeld (https://blog.zx2c4.com/749). Thankfully, Datto was kind enough to provide GCC, so simply pop open vim on the box, throw in the source, compile, and we’re root!

Building a backdoor

Where can we go from here? Our first goal to access the backup data was to obtain access to the web console. Poking through some files revealed “/datto/config/local/webaccess”, which is a text file containing MD5 hashes in a “user:hash:” format. Easy enough…just add our username and MD5 password to the file and we’re in! It’s also worth noting that while the newly added user credentials worked, the username did not appear in the list of users within the web console management interface.

With our purpose being to restore data from a backup (SAM file, sensitive files, etc.), we hopped into the “Restore” page in the web console, selected the DC, chose the backup date, and hit the mount button. Uh oh! Datto devices offer backup encryption for agents added to the device. While not enabled by default, in this case the customer enabled it manually requiring a decryption password prior to mounting or accessing any backup data.

Finding the keys

Let’s dig a little deeper into how this encryption system works. Dan Fuhry, an engineer from Datto has provided a nice little “Datto vs. NSA” paper here, which provides some insight. Here is the relevant section:

“My design for Datto’s encryption system provides unique encryption keys per agent, so there is no single key that can decrypt every dataset we have. The key used to encrypt your actual data is the master key, and that master key is only ever stored in an encrypted fashion. It’s completely random – not derived from a passphrase – and no human ever sees it. When you enter your passphrase, your Datto device does some number crunching on that passphrase and some additional data to get a user key. That user key is used to decrypt an encrypted copy of the master key. This gives you the ability to change your passphrase without having to re-encrypt the entire dataset, and have multiple valid passwords per agent.

The important thing to realize here is how vital your passphrase is to decrypting your data. Without it, the number crunching required to find your data is impossibly immense, even for the NSA. Datto doesn’t keep your passphrase anywhere. Therefore, no court can compel us to hand over your unencrypted data, because we don’t have it and can’t get it.”

This all sounds very promising. To summarize, you need to access the decrypted user key (requires passphrase), to then decrypt the master key, to then decrypt the backup data.

But wait, let’s think about this. If the Datto is backing up agents continuously, and you don’t have to type in your passphrase every time, the Datto must be storing this master key somewhere. This is where “sealing” comes in (Datto’s term). Essentially, when you first reboot a Datto, you need to login to the web console and “un-seal” your agents. This means typing in your passphrase so that the Datto can begin encrypting the backups its taking.

Your first thought might be to just dump the live memory from the device. The master key must be in there somewhere! Let’s first learn a little more about what this “un-sealing” process. As mentioned earlier, pretty much all of the functions on the Datto are controlled through either PHP scripts, or bash scripts. The PHP scripts are obfuscated using ion-cube, so we can’t just throw them into a text-editor. Luckily for us, a neat little decoder tool exists (https://phpdecode.blogspot.com/) that will decode the files to a more readable format. It’s not perfect, but it works great for our purposes. Digging through the source, you’ll notice a function  named “stashAgentKey()” in the encryptionFunctions.php file.

This snippet tells us that Datto is utilizing Linux shared memory at /dev/shm/ to store the master keys while the agent is unsealed. <3 u Datto. Here is an example dump of that file:

{"_stashSig":"6754d8e33202628d73abd270ca6d64d4acaa0ad4a7214aaa46c050faf5b86514","10.199.16.10":"PARTIALLYREDACTEDe332b4dec93617b98b820620e09285c29c02a4285fcff63e223bc767893e6305a812a1042046b08eba88d40de3e4e74944c5387100a0c779a990848d22","10.199.16.190":"PARTIALLYREDACTEDde2ef30759ffec62f7af649f5dca29c011526f0c3d70056440b46609a4e7ed600332af333454b4509b26a454d350fa5e0a7ca3f5ae7af39e998c05bab9",[...]}

Ahh, that’s much better. Edit: Now that we have the master keys for the agents, we should be able to decrypt the backups! However, that sounds like quite a bit of work. Based on Datto’s response, we learned we obtained the encrypted master key, not the actual master key. Here’s more detail taken from their response to this blog post.

If you haven’t read their response, it’s definitely worth while to look it over. It’s not every day that an organization is proactive and receptive to potential security vulnerabilities regarding their product.

While perusing the source, another potentially interesting rabbit hole…or, “rat” hole was found.

Did someone say RAT?

All of the Datto functions are available through a command line application called “snapctl”. Here are the documented arguments:

add                     [hostname]                              -Registers a new agent' . '
removeAgent             [hostname]                              -DESTROYS ALL DATA AND LOG FOR AN AGENT! DANGER!
restartAgent            [hostname]                              -Restarts the replay agent (Wait 30 seconds for agent to restart)
renameAgent             [old-hostname]  [new-hostname]          -Rename an agent, it's ZFS shares, and it's offsite shares
makeStorage             [hostname]                              -creates zpool for agent
command                 [hostname]      [command] [args] [dir]  -Runs a physical command on the host system
updateAgent             [hostname]                              -Updates the SnapToVM agent to latest version
setCompatibility        [hostname]      [on]                    -Set compatibility mode (1 = on 0 = off)
setWriteCaching         [hostname]      [on]                    -Set write caching enabled/disabled (1 = on 0 = off)
clearAllCompatibility                                           -Disables share compatibility for all agents.
list                                                            -Lists all agents protected by this core
dirtyUpgrade            [hostname]                              -This function updates the version of ShadowSnap regardless of previous version
cleanupAppassure        [hostname]                              -This function will clean up any left over Snap2VM files left in the agent directory
restartSystem           [hostname]                              -Reboots the system the agent is running on
setEngine               [hostname]      [engine]                -Allows you to set the backup engine

start                   [hostname]                              -Starts a backup for hostname
foregroundStart         [hostname]                              -Starts a backup on the command line for hostname
stop                    [hostname]                              -Stops the backup for hostname
forceFull               [hostname]                              -Forces the next backup to be full
forceDiffMerge          [hostname]                              -Forces the next backup to be diffmerge
setInterval             [hostname]      [interval]              -Sets the backup interval in minutes
getInterval             [hostname]                              -Prints the backup interval
status                  [hostname]                              -Prints the status of the hostnames backup
excludeVol              [hostname]      [mount]                 -Excludes a mountpoint from backup. Mountpoint must be a volume
includeVol              [hostname]      [mount]                 -Includes a mountpoint from backup. Mountpoint must be a volume
forceRemoveVol          [hostname]      [mount]                 -Removes an excluded mountpoint from the live dataset. Mountpoint must be a volume
checkVol                [hostname]      [mount]                 -Prints the Exclude status for volume
retention               [hostname]                              -Runs ZFS retention operations
dryRunRetention         [hostname]                              -Runs a dry run of ZFS retention operations, which will return the points that will be removed
offsiteRetention        [hostname]                              -Runs ZFS retention operations on offsite servers
setShareAccess          [hostname]      [username]              -Sets all future shares for this hostname to be password protected with access for specified user. A blank user value will disable.

createVM                [hostname]      (snap)       (suffix)   -Clones a snapshot of a host using an optional suffix. Defaults to the most recent snapshot and a suffix of -active.
startVM                 [hostname]                              -Starts a virtual machine for specified host
stopVM                  [hostname]                              -Stops a virtual machine for specified host
destroyVM               [hostname]      (keepClone)  (suffix)   -Destroys virtual machine and optionally keeps clone

setController           [hostname]      [controller]            -Sets the VBox Controller type (IDE or SCSI)
getController           [hostname]                              -Prints the controller type
updateVBox                                                      -Updates VirtualBox to the latest version
mountPoint              [hostname]      (snap)       (suffix)   -Mounts a cloned snapshot's volumes for file restore. Optional snapshot defaults to the most recent. Optional suffix defaults to "[SNAPSHOT#]-file"
unmountPoint            [hostname]      (suffix)                -Unmounts a cloned snapshot's volumes after file restore. Optional suffix defaults to "[SNAPSHOT#]-file"

shareVMDK               [hostname]      [snap]                  -Share RAW vmdk file
unshareVMDK             [hostname]      [snap]                  -Unmounts and de-share raw VMDK
shareVHD                [hostname]      [snap]                  -Share RAW vhd file
unshareVHD              [hostname]      [snap]                  -Unmounts and de-share raw VHD
convertVHD              [path]                                  -Converts a specified image or images in the specified directory to vhd
hostOverride            [hostname]      [newHostname]           -Overrides the hostname given to the agent for the path to the Samba share. Set blank value to disable.
agentSetConfig          [hostname]      [key]        [value]    -Sets an agent configuration variable given the correct key and a valid value. Host must be running.
agentGetConfig          [hostname]                              -Returns the current agent configuration for a given host. Host must be running.
screenshotOverride      [hostname]      [cpus]       [ram]      -Sets the resources to be used for the screenshot VM for this host. (RAM specified in MB)
hir                     [hostname]      [path]       (cowMerge) -Performs HIR on a target location using the hostname as a source of agent information. Optionally will merge the HIR with the images.
clearShares             [hostname]                              -Removes all mapped network drives that point to the datto from the agent machine
attachDMCrypt           [path]                                  -Attach an encrypted image to dm-crypt
detachDMCrypt           [path]                                  -Detach a dm-crypt device
decryptAgentKey         [hostname]                              -Enter a passphrase to unlock an agent's images
sealAgent               [hostname]                              -Discard an agent's master key from memory
addAgentPassphrase      [hostname]                              -Add an additional passphrase that can decrypt an agent
removeAgentPassphrase   [hostname]                              -Remove an existing passphrase from an agent
encryptAgent            [hostname]                              -Set an agent up for encryption. Any existing snapshots/live dataset will be destroyed.
uploadEncryption                                                -Force manual upload of encryption salt/mkey information
downloadEncryption                                              -Fetch encryption salt/mkey information and merge with local stash

screenshot              [hostname]      [delay]      (snap)     -screenshots the agent after delay in seconds using snap (optional)
addScreenshot           [hostname]      [snap]                  -adds snap to the screenshot queue
screenshotAll           [hostname]                              -Screenshot all hostnames on this machine that have not had a screenshot in the last 24 hours
killScreenshot          [hostname]                              -kills the screenshot operation
cleanupScreenshots      (hostname)                              -Cleans up screenshot remnants. Optionally, specify an agent whose screenshot remnants you want cleaned up.
debugScreenshot                                                 -Run the whole screenshot process in the foreground
testFilesystems         [hostname]      (path)                  -Test all backed up filesystems for an agent to verify that they can be mounted. Optionally, specify a path containing a clone of that hostname.

bmrInstall                                                      -Installs the latest network BMR environment.
kzfs                                                            -Upgrade the device to use kernel based zfs
r8169upgrade                                                    -Upgrades the realtek network drivers to version 6.015.00
largeFiles              [directory]     [minSize]               -Show all files in the target directory that are larger than minSize kilobytes
timestamp               [timestamp]                             -Convert a linux timestamp to human readable date
health                  (verbose)                               -Print a device health summary
device-key              show|hide                               -Show or hide the Encryption options section on the Admin page
runTask                                                         -Run a task from the Admin page

checkHost               [hostname]                              -Prints connectivity information about host
update                  [hostname]                              -Updates the agent info about the host
info                    [hostname]                              -Prints info about the host
viewLog                 [hostname]      [lines]                 -Prints lines number of lines from the end of the log file
printErrors             [hostname]                              -Echos the replay log (Should be used with TAIL)
getVMX                  [hostname]                              -Print the VMX Array data stored about the host
getWriters              [hostname]                              -Returns a list of installed VSS Writers
clearError              [hostname]                              -Clears last reported error from the system
truncateLogs            [hostname]                              -forces a truncate of the exchange and SQL logs
setTruncate             [hostname]      [time]                  -Sets the log truncate time out in minutes

updateOffSite                                                   -Updates which points are stored off-site
sendWeekly                                                      -Send the weekly backup digest
updateVols                                                      -Transfer sync/volume information offsite
syncNetworkConfig                                               -Updates /etc/network/interfaces with current state of physical interfaces
totalFreeRam                                                    -Gets the true total free memory available. Includes ZFS cache.
updateHardwareProfile                                           -Update the model of this device after a field upgrade.
updateVbox                                                      -Update VirtualBox to the latest supported version.
suppressUpdateNotification [hostname]   [hours]                 -Suppresses ShadowSnap update notification for given number of hours
agentUpdateStatus          [hostname]                           -Gets current agent update info as JSON data';

That’s a lot of functionality, and immediately you’ll notice some glaring issues. First off, the “command” option sends and executes an arbitrary command to any agent added to the Datto.

command                 [hostname]      [command] [args] [dir]  -Runs a physical command on the host system

Wow, remote command execution built into a backup device. It really is fully featured! Let’s run a quick test to try it out:

Perfect! We’re SYSTEM! Presumably this comes from the ShadowSnap agent that Datto requires you to install on agents backing up to the device. Based on the source code, the command doesn’t appear to be used that much. Moving past command execution, this utility has some other useful features. For one, the “screenshot” ability could be very useful for snooping.

screenshot              [hostname]      [delay]      (snap)     -screenshots the agent after delay in seconds using snap (optional)

Or how about restartSystem?

restartSystem           [hostname]                              -Reboots the system the agent is running on

Or possibly removeAgent if you are feeling dangerous?

removeAgent             [hostname]                              -DESTROYS ALL DATA AND LOG FOR AN AGENT! DANGER!

However, none of these are quite as interesting as two of the encryption commands.

addAgentPassphrase      [hostname]                              -Add an additional passphrase that can decrypt an agent
removeAgentPassphrase   [hostname]                              -Remove an existing passphrase from an agent

So let’s wrap this up. Datto states the because a master key is separate from the user key, that you can have multiple passphrases per agent. The web console doesn’t let you configure this. It requires you to enter the current passphrase in order to change it. But we know that the Datto device already has access to the decrypted master key while an agent is unsealed. So theoretically…

Perfect! We were able to add a temporary passphrase to an existing un-sealed agent, mount a backup with it, demount, and remove the temporary passphrase all without knowing or disturbing the current passphrase.

Now of course this is only scratching the surface of these devices. Here are a few other things we didn’t have time to fully explore:

• Datto downloads scripts over HTTP periodically and executes them blindly, some DNS modifications  or MiTM could make this very dangerous.
• Datto’s SSH syncing to the cloud might be vulnerable to different attacks including device traversal.
• How do user passwords get changed remotely?
• Can raw backup Datto be decrypted with the master keys obtained?
• How does Datto open remote HTTP tunnels from the partner site to enable remote login?

Hopefully this this post will be useful to anyone currently using or considering a Datto device.

Update: Datto wrote a great response to the findings reviewed above. They even straighten out some of our erroneous conclusions regarding identifying the encrypted master key (as opposed to the actual master key). The fixes are shown above. Their response can be found here.