Abusing Azure Hybrid Workers for Privilege Escalation – Part 2: An Azure PrivEsc Story

The NetSPI team recently discovered a set of issues that allows any Azure user with the Subscription Reader role to dump saved credentials and certificates from Automation Accounts. In cases where Run As accounts were used, this allowed for a Reader to Contributor privilege escalation path.

This is part two of a two-part blog series. In part one, we walked through a privilege escalation scenario by abusing Azure hybrid workers. In this blog, we’ll dig a little deeper and explain how we utilized an undocumented internal API to poll information about the Automation Account (Runbooks, Credentials, Jobs).

Note: The scope of this bug is limited to a subscription. A subscription Reader account is necessary to exploit this bug, and it is not a cross-tenant issue.

Background: Azure Hybrid Worker Groups

The genesis of this research stemmed from studying any potential abuse mechanisms from how Azure Automation handled authenticating Hybrid Worker nodes.

Azure Automation’s core feature is Runbooks, which are pieces of code that can be run on Azure’s Infrastructure or customer-owned Azure Virtual Machines (VMs). These are often used to run scheduled tasks or manage Azure resources. To accomplish this, the runbooks must be authenticated, which can be accomplished through several methods.  

Users can store credentials in Automation Accounts (AA) and access them via Runbooks. Automation Accounts can also use Run As accounts to create a Service Principal that will be used for authentication via a certificate stored in the Automation Account.  

The third option is using Managed Identities, which is what Microsoft is pushing users towards. Managed Identities allow the user to obtain a token at runtime to authenticate and eliminate the issue of stored credentials. The Get-AzPasswords script from the MicroBurst project supports dumping all three kinds of authentication, so long as you have Contributor access.  

Normally, a Runbook is executed in a sandbox on Azure’s infrastructure. However, this comes with certain constraints, namely processing power and execution time. Any long running or resource intensive code may be ill-suited to run in this manner.  

To bridge this gap, Azure offers Hybrid Worker Groups (HWG). HWGs offer users the ability to run Runbooks on their own Azure Virtual Machines, so they can run on more powerful machines for longer.  

Normally, this is accomplished by deploying a Virtual Machine Extension to the desired Virtual Machine to register the Virtual Machine as a HWG node. Then, the user can execute Runbooks on those Hybrid Worker nodes.  

There are also two types of HWGs: User and System. System HWGs are used for Update Management and don’t have the necessary permissions for what we’re interested in, so we’ll be focusing on User HWGs.

The First Set of Issues: Compromising Credentials

We began our research with a registered Hybrid Worker node. When you execute a runbook on the host, the HybridWorkerService process spawns the Orchestrator.Sandbox process. The command line for the latter is as follows.

Next, we focused on MSISecret. At first glance, it appears that the Hybrid Worker node must be able to use this to request an MSI token externally. After reversing the binary, this turned out to be true. 

Every Automation Account has a “Job Runtime Data Service” endpoint, or JRDS, which Hybrid Workers use to poll for jobs and request credentials. You can see the JRDS URL supplied in the command line above. Below is what the full path to request a token looks like in the binary. 

And here you can see this in action.

You can only get that MSI secret after receiving a job from the JRDS endpoint. This can be achieved by polling the /sandboxes endpoint. HWGs handle jobs in a first-come-first-serve fashion, so whichever node polls the endpoint first starts first. By default, nodes will poll every 60 seconds so if polled constantly, then we should almost always beat out the other nodes and get a job with a secret. However, this only works if Runbooks jobs are actively being run through the HWG.

Since we’re able to request Managed Identity tokens, it would make sense that we can request other forms of authentication. A quick grep through of the decompiled binary makes this apparent, and a quick request to these endpoints will yield results. 

The JSON Web Token (JWT) in these requests is for the System Assigned MI of the Virtual Machine, not a management token for Azure.

Requesting all certificates:

We were satisfied with these findings. We figured that this represented an escalation path from Virtual Machine Contributor to Subscription Contributor if Hybrid Worker nodes were in use and reported our findings to Microsoft.

Escalating Our Findings

After we had submitted our report, we found a recently published blog that detailed some of these same ideas, though their thesis was obtaining lateral movement after an administrator pushed a certificate to the Virtual Machine. The author also demonstrated that you could register a new Hybrid Worker node to an Automation Account using the Automation Account key and Log Analytics Workspace key. We wondered if we could abuse this route to escalate the severity of our previous findings.

To read Automation Account keys, a user only needs the Reader role. To exploit this, we hacked up some source code from Microsoft’s Desired State Configuration (DSC) repository.

The repository contained some scripts that are used to register a new Hybrid Worker node, so we bypassed some environment checks and created users/groups that are expected to exist. The registration process looks like this: 

1. Generate a new self-signed certificate or use an existing one
2. Create a payload with some details: HWG name, IP address, certificate thumbprint, etc.
3. Sign the payload with the AA key
4. Send a PUT request to the AA with all the above info 

This also does not require Hybrid Worker Groups to already be in use; we can supply an arbitrary group name and it will be created. After registering, we can use the certificate and key generated during this process to access the same endpoints that we identified earlier. You also don’t need a Log Analytics workspace key to register because not all AAs are linked to a workspace. 

From start to finish, this exploit works as follows: 

1. Attacker with Reader access reads the victim Automation Account key
2. Attacker uses this key to register their own Virtual Machine in their own tenant as a Hybrid Worker node
3. Attacker can dump any credentials or certificates from the victim AA and use them to authenticate 

We reported this issue to MSRC in a separate report. Below is the timeline for this case: 

• October 25, 2021: Initial report submitted 
• December 13, 2021: Second report submitted with details of full privilege escalation 
• December 31, 2021: $10k bounty awarded 
• March 14, 2022: Patch is applied 

Microsoft’s Response to the Azure Automation Account Vulnerabilities

After reporting our findings, Microsoft identified the Azure Automation customers vulnerable to this exploit and notified them through the Azure portal. A fix has been rolled out to all customers.

Additionally, Microsoft has updated their documentation with mitigation steps for customers. They’ve updated the Reader role so that it no longer has the ListKeys permission on Automation Accounts and can no longer fetch Automation Account keys. They recommend that customers switch to custom roles if they need a Reader to fetch the Automation Account keys.

Microsoft has also provided the following guidance for deploying Hybrid Workers:

Microsoft recommends installing Hybrid workers using the Hybrid Runbook Worker Virtual Machine extension – without using the automation account keys – for registration of hybrid worker. Microsoft recommends this platform as it leverages a secure Azure AD based authentication mechanism and centralizes the control and management of identities and other resource credentials. Refer to the security best practices for Hybrid worker role.

Conclusion

This issue allowed any user who could read Automation Account keys to extract any credentials or certificates from the affected Automation Account. This issue was not particularly technical or difficult to exploit, and only abused the intended methods for registration and credential retrieval. 

This is a good reminder that even low privileged role assignments such as Reader can have unintended consequences in your cloud environment. 

Want to learn more about cloud penetration testing? Explore our Azure cloud penetration testing service.