This post reviews strategies for identifying and mitigating potential attack vectors against virtual machine (VM) services in the cloud. Organizations can use this information to understand the potential risks associated with their VM services and strengthen their defense mechanisms. This research focuses on VM services offered by three major cloud service providers (CSPs): Amazon Web Services (AWS), Azure and Google Cloud Platform (GCP).
VMs are among the most frequently deployed resources in every cloud environment. Their widespread use also makes them a prime target for attackers. Our research shows that 11% of cloud hosts exposed to the internet contain vulnerabilities rated Critical or High severity.
A compromised VM can provide attackers with access to not only the data within the VM instance but also the permissions assigned to it. As compute workloads like VMs are generally ephemeral and immutable, the risk posed by a compromised identity is arguably greater than that of compromised data within a VM.
It is important to note that all the attack paths discussed in this post are intended features with legitimate use cases, such as streamlining the configuration, updating, and monitoring of VMs across hybrid or multi-cloud environments, rather than vulnerabilities. However, if security best practices are not followed, accounts are not protected, and careful attention isn't given to the design of your architecture, malicious users could misuse these services or features. The responsibility of protecting and mitigating these attack paths falls on the cloud users and administrators.
Palo Alto Networks customers are better protected from the threats discussed above through the following products:
Related Unit 42 Topics | Cloud |
Summary of the VM Attack Paths
Understanding VMs in the Cloud
Vulnerability Exploitation
Startup Script Manipulation
AWS: Modify Startup Scripts in User Data
Azure: Modify Startup Scripts in Custom Data
GCP: Modify Startup Scripts in Metadata
SSH Key Push
AWS: Use EC2 Instance Connect to Push SSH Keys
Azure: Use VMAccess Extension to Push SSH Keys
GCP: Update Metadata to Push SSH Keys
GCP: Use OSLogin to Push SSH Keys
Direct Code Execution
AWS: Use SSM Run Command to Execute Code
Azure: Use Virtual Machine Run Command to Execute Code
Azure: Use a Custom Script Extension to Run Scripts
GCP: Use VM Manager to Execute Code
Run Pre-Patch or Post-Patch Scripts
Run Scripts in OS Policies
SSH Over Middleware
AWS: Use SSM Session Manager to Log into a VM
Serial Console Access
AWS: Login via Serial Ports
Azure: Login via Serial Ports
GCP: Login via Serial Ports
Conclusion
Palo Alto Networks Protection and Mitigation
Additional Resources
We explore the conditions and permissions for each attack path into a running VM instance to assist organizations in fine-tuning their detection and mitigation mechanisms. Table 1 provides an overview of all the attack paths we discuss.
AWS | Azure | GCP | |
Vulnerability Exploitation | Feasible: Yes
Complexity: Depends |
Feasible: Yes
Complexity: Depends |
Feasible: Yes
Complexity: Depends |
Startup Script Manipulation | Feasible: Yes
Feature: EC2 User Data Complexity: Low |
Feasible: only VM Scale Sets
Feature: VM custom data Complexity: Low |
Feasible: Yes
Feature: Metadata Startup Scripts Complexity: Low |
SSH Key Push | Feasible: Yes
Feature: EC2 Instance Connect Complexity: Low |
Feasible: Yes
Feature: VMAccess extension Complexity: Medium |
Feasible: Yes
Feature: Metadata, OSLogin Complexity: Low |
Direct Code Execution | Feasible: Yes
Feature: SSM Run Command Complexity: Medium |
Feasible: Yes
Feature: Run Command, Custom Script Extension Complexity: Low |
Feasible: Yes
Feature: VM Manager Complexity: Medium |
SSH Over Middleware | Feasible: Yes
Feature: SSM Session Manager Complexity: Low |
Feasible: No | Feasible: No |
Serial Console Access | Feasible: Yes
Feature: EC2 Serial Console Complexity: High |
Feasible: Yes
Feature: Azure Serial Console Complexity: High |
Feasible: Yes
Feature: Metadata/Serial Console Complexity: Low |
Table 1. Summary of VM attack paths.
VMs are among the oldest and most widely used infrastructure-as-a-service (IaaS) offerings across all cloud service providers. They offer a swift and straightforward method to “lift and shift” on-premises applications to the cloud, maintaining the same user experience at the operating system level and above. Modern VM services support a broad spectrum of operating systems, from Linux to Windows to macOS, enabling virtually any application to be deployed in the cloud.
While VMs might not be the most novel cloud technology today, they continue to host many vital cloud workloads. If a VM is compromised, attackers can not only exfiltrate sensitive data and hijack computational resources but also gain access to all the cloud permissions granted to the VM.
As the tactics, techniques and procedures (TTPs) employed by attackers in the cloud largely depend on the permissions they have managed to obtain, one common method of gaining more permissions is to compromise a compute resource, such as a VM, and hijack its workload identity. As a result, each VM instance can potentially serve as a stepping stone towards an attacker's goal, making it crucial to meticulously manage the VM's attack surface.
We define a VM attack path as a series of steps and conditions that could potentially allow an attacker to log in or execute commands in a VM instance. We assume that attackers possess basic information about the targeted VM, such as its unique identifier (UID), IP address, virtual private cloud (VPC) and region.
This information, which is not typically considered confidential, can be sourced from logs, code, or low-privileged read permissions. However, attackers do not possess the login credentials for VMs. The majority of the attack paths discussed in this post rely on control plane application programming interfaces (APIs) to gain access to a VM.
Subsequent sections will each cover a specific technique and explore the attack paths related to that technique in each CSP. We will outline the preconditions for each attack path, noting that while these conditions are necessary, they might not be sufficient. For instance, to exploit these attack paths, we assume the attackers have obtained the required permissions through means such as credential leaks or phishing in order to exploit a specific attack path.
We will focus on the most relevant permissions or configurations that result in these attack paths. Although most of the techniques described are not specific to any particular VM operating system, for simplicity, the references and examples provided will primarily be based on Linux systems.
Our research reveals that 11% of the cloud hosts exposed to the internet contain Critical or High severity vulnerabilities. Exploiting these vulnerabilities is one of the most common ways attackers use to gain initial access to cloud environments.
Given that modern applications are bundled with hundreds of dependent packages, the emergence of new vulnerabilities is accelerating faster than ever. Regardless of the instance type and cloud type, if attackers can identify a remotely exploitable vulnerability exposed by a VM, they could potentially compromise and take control of it.
Conditions:
Mitigations:
A startup script is a file that executes tasks during the initialization process of a VM instance. These scripts are typically used to set up the environment, download dependencies, initialize services and fetch updates. If attackers gain permissions to alter a VM's startup script, they could exploit this feature to inject malicious code into the VMs.
When launching a new EC2 instance, users can optionally pass parameters or scripts in user data. Any scripts in user data are run when the instance is launched. By default, the scripts are only executed during the first boot of the instance. However, it is possible to configure the cloud-init directives to force scripts to execute at every restart.
Conditions:
Mitigations:
The startup scripts are stored and passed to an Azure VM via its custom data. For a single VM, its custom data can only be set once at boot time and can’t be updated subsequently. However, custom data of a VM Scale Set, a group of VMs, can be updated.
Newly initiated VMs will receive the updated custom data. Existing VMs, on the other hand, need to be reimaged to receive the new custom data.
Conditions:
Mitigations:
Compute Engine’s metadata service offers a mechanism for storing and retrieving metadata in the form of key-value pairs, including startup/shutdown scripts, SSH keys and numerous feature flags. Metadata can be set at instance-level for each individual VM or project-wide level for all VMs within the project. Each VM is then configured according to its respective metadata.
The startup-script metadata key contains the commands that run when a VM instance boots.
Conditions:
Mitigations:
Given that each organization typically hosts various applications on hundreds (if not thousands) of VM instances in their cloud environments, managing the SSH keys for all these VMs can be a daunting task. To help streamline the process of credential management and access control, most CSPs offer features that allow for the easy insertion of SSH public keys into running VMs.
This process usually involves an agent running within a VM, fetching a public key from a cloud API endpoint, modifying the SSH daemon (sshd) configuration file and overwriting the authorized_keys file on the VM. If attackers gain permissions to push SSH keys, they could exploit this feature to gain unauthorized access to VMs.
EC2 Instance Connect provides a simple and secure method to manage SSH access to Linux VMs using identity and access management (IAM). When a user needs to SSH into a VM, Instance Connect pushes a temporary public key to the VM, allowing the user to authenticate with the SSH daemon.
Conditions:
Mitigations:
VM Extensions are small applications that facilitate post-deployment configuration and automation on VM instances. These extensions offer functions such as system configuration, system monitoring and system backup.
The VMAccess extension allows the management of administrative users on Linux VMs for tasks like setting a user’s password, pushing an SSH public key or creating a new sudo user. The az vm user command relies on the VMAccess extension to manage user accounts in a VM.
Conditions:
Mitigations:
Compute Engine’s metadata service offers a mechanism for storing and retrieving metadata in the form of key-value pairs. By updating the SSH keys metadata key, one can add SSH public keys to a VM instance.
Conditions:
Mitigations:
OSLogin automatically manages SSH keys in metadata and user accounts in VM instances using Google Cloud Identity (IAM) policies. This is the recommended way to manage SSH keys in VMs.
OSLogin can be enabled by updating the enable-oslogin metadata key in the metadata service. It is important to note that metadata-based SSH keys and OSLogin are two mutually exclusive features that can’t both be enabled.
Conditions:
Mitigations:
To streamline the management and configurations of a fleet of VM instances, most CSPs offer features that allow the execution of commands or scripts across a set of VMs. This eliminates the need for VMs to have exposed management ports, bastion hosts or even an active sshd running, increasing their security and cost-effectiveness.
These features usually rely on agents running in the VMs that fetch and execute commands from the cloud API endpoints. If attackers gain the necessary permissions to perform these actions, they could exploit these features to execute malicious code within the VMs.
The SSM Run Command allows users to execute commands on nodes where the System Manager is installed. The feature offers an easy way for performing one-time configurations or status checks across nodes in single-cloud, multi-cloud or hybrid cloud environments.
Conditions:
Mitigations:
The Run Command feature in Azure uses the VM agent within a VM to execute scripts. It can be used for application management, system diagnostics or troubleshooting when RDP or SSH service are unavailable.
Conditions:
Mitigations:
VM Extensions are small applications that can perform post-deployment configuration and automation on VM instances. The custom script extension allows for the downloading and execution of scripts within VMs.
Conditions:
Mitigations:
VM Manager is a suite of tools that can help manage a group of VMs. It is primarily used for applying patches, collecting OS information and installing or removing software packages.
The Patch feature can apply OS patches across a set of VM instances using OS package managers like the Advanced Packaging Tool (APT) and Yellowdog Updater, Modified (YUM). During the creation of a patch job, optional pre-patch or post-patch scripts can be executed to either prepare for or test the patch.
The OS Policy feature allows users to maintain a consistent configuration in OSes across multiple VMs. Each policy file contains the declarative configuration for resources such as packages, repositories or files. One way to configure resources in an OS is executing scripts.
Conditions:
Mitigations:
AWS SSM Session Manager provides a secure and auditable way to log into nodes using IAM. Nodes with Session Manager don’t need to have open inbound ports and users logging into the nodes don’t need to manage the private keys.
Session manager can also be configured on nodes in multi-cloud or hybrid cloud environments. If attackers gain permissions to perform the session manager’s actions, they could potentially abuse the feature to log into VMs with the session manager running.
Conditions:
Mitigations:
Most cloud service providers offer serial console access as a feature to troubleshoot boot and network configuration issues in VMs. This feature provides text-based console access to VMs, independent of the network and operating system state.
Because network-based access control does not apply to serial console access, attackers with serial console permissions could potentially abuse this feature to bypass network-based firewall restrictions and gain unauthorized access to VMs. It is important to note that the serial console access does not bypass the user authentication. Valid passwords or private keys are still needed to log into a VM.
Amazon EC2 Serial Console provides access to the serial port of EC2 instances.
Conditions:
Mitigations:
Azure Serial Console provides text-based console access to VM instances.
Conditions:
Mitigations:
GCP provides an alternative way to connect to a VM over a serial port. Compute Engine serial port access can be enabled in the metadata service by updating the serial-port-enable metadata key.
Conditions:
Mitigations:
This post provides an overview of potential attack paths into VMs and outlines the mitigation strategies that organizations can implement to enhance their cloud security. Maintaining the security posture of VMs in cloud environments is crucial.
Due to their widespread use and inherent permissions from workload identities, VMs are attractive targets for attackers. All the attack paths discussed throughout this post are based on the intended features of legitimate use cases. However, if these features are not properly secured, adversaries can abuse them with malicious intent. The responsibility of safeguarding these attack paths and mitigating potential risks lies with the cloud users.
IAM configuration plays a pivotal role in both enabling these attack paths and mitigating their associated risks. To ensure robust cloud security, it is vital to continuously identify these attack paths and monitor the use of risky permissions.
As cloud environments continue to evolve, so too will the TTPs employed by cyberattackers. Organizations must remain vigilant and proactive in their cloud security efforts, adapting their strategies to counter evolving threats.
Palo Alto Networks customers are better protected from the threats discussed above through the following products:
If you think you may have been compromised or have an urgent matter, get in touch with the Unit 42 Incident Response team or call:
Palo Alto Networks has shared these findings with our fellow Cyber Threat Alliance (CTA) members. CTA members use this intelligence to rapidly deploy protections to their customers and to systematically disrupt malicious cyber actors. Learn more about the Cyber Threat Alliance.
Palo Alto Networks
AWS
Azure
GCP
Others
Sign up to receive the latest news, cyber threat intelligence and research from us