Extracting Managed Identity Certificates from the Azure Arc Service 

Initially announced in 2019, Microsoft created the Azure Arc service to bridge the gap between on-prem resources and the Azure cloud. While this service allows Azure administrators to easily integrate on-prem resources with their Azure cloud environment, it also brings along its own set of security concerns. 

TL;DR

1. Azure Arc uses a System-Assigned Managed Identity to authenticate enrolled systems to Azure
2. The authentication certificate associated with the Managed Identity is stored on the enrolled system: 
   1. Windows – “C:\ProgramData\AzureConnectedMachineAgent\Certs\myCert.cer” 
   2. Linux – “/var/opt/azcmagent/certs/myCert” 
3. This certificate can be extracted from the system (by local administrators) and used to authenticate as the Managed Identity away from the resource: 
   1. This authentication lacks important logging details (source IP, etc.) in Entra ID 
   2. This architecture breaks the fundamental model for Managed Identities and their credentials 
   3. Any Azure permissions/role applications for the Managed Identity are then available to the attacker. 
   4. These certificates are not controlled by the resource owner, so if one is compromised, the system needs to be removed from Arc and reenrolled to cycle the certificate. 
4. We wrote a tool to automate the process of extracting these certificates from Arc systems 

Azure Arc 

The core functionality of the Arc service allows Azure administrators to manage and integrate on-prem resources into the cloud. This includes the ability to run commands on systems, extend SQL access to web applications, and enroll systems from other cloud providers. While all this functionality is useful, the authentication model that’s used to integrate these systems into the Azure Arc service has a fundamental flaw. The authentication for the service utilizes a System-Assigned Managed Identity that stores its credential (a certificate) on the Arc system. These credentials are then available to anyone with local administrator access to the system. 

Certificate Storage 

Depending on the operating system, the credential (a PFX file) will be stored in one of two places: 

1. Windows – “C:\ProgramData\AzureConnectedMachineAgent\Certs\myCert.cer” 
2. Linux – “cat /var/opt/azcmagent/certs/myCert” 

*Note that neither file has a PFX file extension, but they are both PFX files, that do not have passwords

Accessing the certificate does require local administrator (or root) permissions, and Microsoft considers this security boundary to be the responsibility of the system owner. Microsoft acknowledges this in their documentation, noting that “The agent automatically applies an access control list to this directory, restricting access to local administrators and the “himds” account.”. 

It should be noted that the Arc service does not grant any default roles to the associated Managed Identities, but Microsoft’s own documentation does recommend adding roles to them. This is recommended to make it easier for the Arc systems to access Azure resources, like Key Vaults. Since there are no default role applications for the Managed Identities, there is no inherent privilege elevation risk associated with enrolling a system in Arc. That said, we have seen practical usage of Arc in several client environments, and the associated Managed Identities are often getting subscription (or resource) level permissions assigned.

Thanks to the functionality of the Arc platform, any user with “Write” permissions on a “HybridCompute” resource can create “run commands” on the systems that allow for command execution. Utilizing the command execution functionality, an attacker would be able to get the Base64 string of the private certificate and save it off to their attacking system. While the run command feature is not yet available in the Portal, it is possible to queue up commands with both the Az CLI and the Management REST APIs.

Extracting Managed Identity Certificates with MicroBurst 

As attackers, we typically want to automate as many tasks as we can. If we do get “HybridCompute” write permissions, an automated solution will help us gather the Managed Identity credentials as fast as possible. These credentials can then be used for persistence and (potentially) for privilege escalation, depending on the applied roles. 

To make this whole process easier, we have added a script (Get-AzArcCertificates) to the MicroBurst toolkit. Below is an overview of the script and its usage. 

Script Overview: 

1. Check the Az PowerShell module authentication 
2. Prompt for a Subscription to use 
3. Enumerate the available “Microsoft.HybridCompute/machines” resources in the subscription 
4. Select the systems to attack 
5. Loop through the selected systems 
   1. Create the Run Command instance for the resource 
   2. Wait for the command to execute 
   3. Write the certificate to a local file with the “AuthenticateAs” script 
   4. Delete the Run Command instance 

Usage – Import the function:

ipmo .\MicroBurst\Az\Get-AzArcCertificates.ps1

Usage – Run the function:

PS C:\MicroBurst> Get-AzArcCertificates -Verbose  
VERBOSE: Logged In as kfosaaen@example.com 
VERBOSE: Enumerating Azure Arc Resources in the "Sample Subscription" Subscription 
VERBOSE: 	1 Azure Arc Resource(s) enumerated in the "Sample Subscription" Subscription 
VERBOSE: 		Starting extraction on the i-001aab1bcba8519b1 system 
VERBOSE: 			The i-001aab1bcba8519b1 system is registered as a Windows system 
VERBOSE: 			Adding the SLTImRxhgyukwjE command to the i-001aab1bcba8519b1 system 
VERBOSE: 				Sleeping 10 seconds to allow the command to execute 
VERBOSE: 			Getting the command results from the i-001aab1bcba8519b1 system 
VERBOSE: 				Sleeping additional 5 seconds to allow the command to execute 
VERBOSE: 			Writing the certificate to C:\ MicroBurst\6843069d-5b5b-4618-86ac-0ccc8d6a6476.pfx 
VERBOSE: 				Run .\AuthenticateAs-6843069d-5b5b-4618-86ac-0ccc8d6a6476.ps1 (as a local admin) to import the cert and login as the Managed Identity for the i-001aab1bcba8519b1 system 
VERBOSE: 			Removing the SLTImRxhgyukwjE command from the i-001aab1bcba8519b1 system 
VERBOSE: Azure Arc certificate extraction completed for the "Sample Subscription" Subscription 

As noted in the script output, an “AuthenticateAs-*.ps1” script is generated, along with writing the PFX file to the current directory. In a local admin PowerShell session, this script can then be used to authenticate to the Az PowerShell module as the Managed Identity. 

At this point, we are authenticated as the Managed Identity and can utilize any of the roles/permissions applied to it. Since the normal tokens generated by a Managed Identity have a short lifetime, these certificates grant us significantly longer access to the identity. 

Detection and Hunting Opportunities 

There are several detection and hunting options for this attack. Keep in mind that an attacker may not use this specific tool, and they may just directly authenticate to the Arc system to extract the certificates. 

Detection Opportunity #1: Run command instance used on Azure Arc systems 

Data Source: Command Execution
Detection Strategy: Behavior
Detection Concept: Using Azure Activity Log, detect on when any of the following commands are run an an Arc System via Azure Credentials: 

• Microsoft.HybridCompute/machines/runCommands/write 
• Microsoft.HybridCompute/machines/runcommands/read 
• Microsoft.HybridCompute/machines/extensions/write 

Detection Reasoning: A threat actor can use the run command function to extract the certificate from an Arc system without having existing local access.  

The following commands are used by the Microburst tool: 

• Windows: gc C:\ProgramData\AzureConnectedMachineAgent\Certs\myCert.cer 
• Linux: cat /var/opt/azcmagent/certs/myCert 

Known Detection Consideration: None 

Detection Opportunity #2: Command run locally to extract certificate 

Data Source: Process Creation
Detection Strategy: Signature 
Detection Concept: Detect when a process is created with the following command line signatures: 

Windows

gc C:\ProgramData\AzureConnectedMachineAgent\Certs\myCert.cer

Linux

cat /var/opt/azcmagent/certs/myCert

Detection Reasoning: If a threat actor has local access to the Arc system they can extract the Managed Identity certificate using the previously noted commands. 
Known Detection Consideration: While the “get-content” (gc) and cat” commands are listed as examples, there are other commands that could be run to get access to the file, including using other binaries and protocols (HTTP, FTP, etc.) to exfiltrate the files. 

Detection Opportunity #3:  Managed Identity anomalous activities 

Data Source: Cloud Service 
Detection Strategy: Behavior 
Detection Concept: Look for any Managed Identities that are taking actions similar to what an attacker may do (listing resources, attempting to access unauthorized resources). Alternatively, you can start with expected activities for the Managed Identity (accessing specific resources) and then alert on events outside of that scope.  
Detection Reasoning: Given that the Arc System-Assigned Managed Identity should not be utilized for actions outside of its normal scope, there is potential to catch anomalous login behaviors. 
Known Detection Consideration: None

Hunting Opportunity #1: Azure VM File Inspection 

Data Source: File Creation  
Detection Strategy: Behavior 
Hunting Concept: Look for evidence of run commands on Arc Systems that would indicate this type of activity. 
Windows – The specific commands that were run can be found by investigating the script files in the “C:\Packages\Plugins\microsoft.cplat.core.runcommandhanderwindows\2.0.9\Downloads” directory. Note that the 2.0.9 denotes the version of the RunCommand extension and may differ on your Virtual Machine.

Linux – The specific commands that were run, can be found by investigating the temporary files in the “/var/lib/waagent/microsoft.cplat.core.runcommandhandlerlinux-1.3.7/config/$RUNCOMMANDNAME.0.settings ” file, where $RUNCOMMANDNAME is the name of the command. 

The settings file contains the following runtime settings, which include the executed command in the public settings field: 

{"runtimeSettings":[{"handlerSettings":{"protectedSettings":"[Truncated] ","protectedSettingsCertThumbprint":"runcommandhandlerlinux.FRQZhHOsNbwJLil","publicSettings":{"asyncExecution":false,"parameters":[],"source":{"script":"cat /var/opt/azcmagent/certs/myCert"},"timeoutInSeconds":0}}}]}

Detection Reasoning:
When using the run command action on an Arc system, there will be residual files left on the Arc system. 
Known Detection Consideration:
Note that the Windows extension will be removed after the run command instance is removed, so this data may be temporarily available. Additionally, the script files themselves may not be persistent in the directories, or they may get cleaned up by the attacker. 
Managed Identity Sign-in Log Addendum:
As an additional detection note, the logging of Managed Identity authentication will be virtually indistinguishable from the normal usage of the Managed Identity on the resource. Since Microsoft does not expect a Managed Identity credential to be used outside of the Azure resource it is assigned to, an IP address is not logged for the authentication event in Entra ID. 

This issue has been raised with MSRC in the past (See the Function Apps blog), but it was dismissed:

These credentials are not intended to be used externally by customers, so the logs aren't surfaced in customer logs. The fix to this issue is the fix to ensure that these credentials are not exposed.

Conclusions 

The accessibility of the certificates, in both Windows and Linux, was raised as two separate tickets to MSRC on April 23, 2024. Both tickets were dismissed on June 12, 2024 with the justification that this is expected behavior for the service. I also had one last call with MSRC and the Arc team on June 20, 2024 to clarify some final points (lack of logging, credential rotation, persistence) on the tickets. 

While I still don’t know if Managed Identities are the best route to enable authentication to an Entra ID tenant, I understand why the service team chose this option. Since these Arc systems do not actually live in Azure, where they could rely on internal cloud management fabric, they need to have some credentials to authenticate up to the cloud. Storing a key would have all the same problems as the Managed Identity certificates but would lack the ability to regularly rotate credentials that are included in the current configuration. The Managed Identity method of authentication allows the systems to get some form of access to the Azure tenant, but I do have concerns around storing credentials (regardless of the directory level protections) on the system. 

Previous Research 

Experimenting with Azure Arc – Matt Felton
https://journeyofthegeek.com/2021/06/12/experimenting-with-azure-arc/