The Unholy Marriage of AWS IAM Roles and Instance Profiles

TL;DR

EC2 instances continue to use role credentials for up to six hours—even after removing a role from the instance profile. Swapping roles can result in credential discrepancies of up to 54 minutes. Forcing a credential refresh requires deleting the role, revoking sessions for more than six hours, stopping/starting the instance, or associating a new instance profile and role. Instance profiles persist even after role deletion and can hold a role having a different name than that of the instance profile name.

Exploring AWS IAM Role & Instance Profile Quirks

Identity and access management (IAM) roles and instance profiles are important components for managing EC2 access to AWS resources. But what happens when an IAM role is removed from an instance profile that is still attached to an EC2 instance?

Why doesn’t removing a role from an instance profile prevent an EC2 instance from accessing the role's credentials via IMDS?

How does adding the same role back or rebooting the instance fail to refresh these credentials?

This post uncovers some unexpected behavior that occurs when modifying roles and profiles. We'll also look at how instance profiles remain even after the role has been deleted, and how any other role can be added to that instance profile.

AWS Concepts

Before getting into the details, you first need to understand certain AWS concepts.

IAM role

A role is like a set of permissions that determine what actions an AWS service or user can perform. Think of it as a virtual "key" that grants specific permissions to access certain AWS resources, akin to unlocking a door.

IAM roles are commonly used to allow EC2 instances or other AWS services to access resources. This is done without having to hardcode and store sensitive access keys or credentials. They provide an added layer of security by letting you define specific permissions for each role and can be easily managed through the IAM service.

Instance profile

An instance profile is a container that holds an IAM role. When you launch an EC2 instance with an associated profile, the instance can then use the permissions granted by the IAM role to access only those AWS services or resources the role permits—without having to manually manage credentials or access keys. This helps improve security and simplifies AWS resource management.

Instance metadata service (IMDS)

IMDS is a service that runs on every EC2 instance. It provides a way for applications running on the instance, or the local CLI, to retrieve information about a given instance. It's like a special website that only the EC2 instance can access; it contains data about the instance's configuration, network settings, attached role credentials (if attached), and other metadata.

Think of the instance metadata service as a built-in source of information for EC2 instances. It allows applications running on the instance, or the local CLI, to access important details about the instance's environment without the need for additional configuration or setup.

Observed Behavior

During my testing, I attached a role (RedRole) to an EC2 instance. Doing this through the AWS console automatically creates the instance profile and gives it the same name as the role. But for simplicity sake I’ve named it MyInPr—short for My Instance Profile. RedRole has permissions to write to S3 (s3:PutObject) and create an IAM user (iam:CreateUser). So now RedRole is attached to the MyInPr instance profile associated with my EC2 instance.

AWS IAM roles and instance profiles are crucial components for managing EC2 access to resources. Learn why they must be managed in tandem on EC2.

The application on the instance (or CLI) retrieves the temporary security credentials provided by the role from the IMDS (here called ASIA 123 for simplicity; access keys for temporary credentials always start with this prefix followed by randomly generated alphanumeric characters). The application and CLI are granted permissions for the actions and resources defined by associated security credentials for that role. Such credentials are only valid for six hours, after which new credentials are acquired by the IMDS.

But in removing the role from the instance profile (still attached to the EC2 instance), the application and CLI remain able to write to S3 and create a new IAM user. I’ve confirmed that the credentials served by the IMDS are still ASIA 123. Logging into the console doesn’t show this persistence, which is expected behavior according to AWS.

When a role is removed from an instance profile in AWS IAM, what happens in the EC2 to which it is assigned?

Upon reinstating RedRole with the instance profile, I verified IMDS continued to serve the original ASIA 123 credentials.

Testing to see what happens when a removed role is added back to EC2 via AWS IAM.

I then tested what would happen when I swapped RedRole for a different role (BlueRole). Almost immediately (3 seconds), the AWS console displayed BlueRole as being associated with the EC2 instance. But the CLI only confirmed the new ASIA 789 credentials after 54 minutes—a latency during which the older credentials remained usable.

Testing to see what happens when a new role is added in place of the original in an EC2 via AWS IAM.

To see how I could force IMDS to acquire new role credentials before waiting 54 minutes, I swapped an original YellowRole for the new OrangeRole. After rebooting the instance and regaining CLI access on the EC2 instance, I discovered YellowRole credentials were still being served. The only way to force EC2 to not use the original credentials was to either delete the role, revoke its sessions for up to six hours, or associate a new instance profile and role to the EC2 instance.

Tests revealed that instance profiles remain even after deleting corresponding roles through the AWS console. While I initially noted that IAM auto-generates instance profiles upon creating new roles, no such process occurs when roles are deleted. I successfully attached a deleted role profile to an EC2 instance and added a random role to that using CLI. As instance profiles can contain any role, their names might not match the internal roles.

Takeaways

AWS stated what I’ve reported as “expected behavior”
A recently emptied instance profile still attached to an EC2 instance continues to allow that instance to use the role’s credentials for up to six hours
Adding the original role back to the instance profile does not result in new credentials being acquired by the IMDS
Swapping out the original role with a new one in the instance profile leads to the original credentials persisting until eventual consistency occurs (up to 54 minutes in my testing)
When roles are swapped in the instance profile, the AWS console for EC2 and the CLI for the instance show different IAM roles being used for up to 54 minutes (in testing)
To force a credential refresh being served by IMDS, you need to either delete the role, revoke its session for more than six hours, stop/start the instance (not reboot), or associate a new instance profile and role to the EC2 instance
Even after role deletion in the AWS console, instance profiles remain and can be attached to EC2 instances, supporting various roles with potentially differing names

Commands Used

image (5)

To view these commands as a text file, click HERE.

Andre Rall

Andre brings a unique combination of strong cloud, cloud security, fraud, and risk knowledge to Uptycs. Andre joined Uptycs in 2022 as Director of Cloud Security to help shape the cloud security product offerings. Since joining Uptycs, Andre has influenced the CSPM and CIEM roadmap, and implemented thought leadership...