Most common security incidents
- Publicly accessible resources such as S3 buckets or ElasticSearch clusters.
- Leaked access keys. For example, access keys posted to GitHub.
- Compromised IAM Roles through SSRF or RCE against an EC2, resulting in access to the metadata service at 169.254.169.254.
AWS Security Maturity Roadmap
Stage 1: Inventory
- Identify all AWS accounts in the company and their points of contact.
- Integrate AWS accounts into AWS Organizations.
- Ensure all account root emails are on distribution lists.
- Opt-out of AI services using your data.
- Have an AWS account for Security.
- Create budget alarms.
If you don’t already have lots of AWS accounts, recognize that at this point you now have a minimum of 3 AWS accounts (an account that has your business stuff in it, the AWS Organization root account with nothing in it, and the Security account).
Stage 2: Have backups
- Create regular backups.
As you learn what data you have in AWS, you should ensure that data is backed up to a separate AWS account in a separate region from where the data comes from (or possibly even somewhere off of AWS)
Stage 3: Visibility and initial remediation
- Turn on CloudTrail, GuardDuty, and Access Analyzer for all accounts to send their logs and alerts to the Security account.
- Create an IAM role in every account that grants view access into the account from the Security account.
- Run a one-time scanning tool to identify tactical remediations.
- Turn on S3 Public Block Access.
- Develop an account initialization script and new account creation process.
Stage 4: Detection
- Send alerts to a ticketing system.
- Enable investigations to logs.
- Perform regular scanning of the accounts for security issues.
- Document your security guidelines for your company.
- Consider turning on other logging sources.
As a first step, most companies will create a CloudWatch Event rule that sends the alerts directly to an SNS that goes to email or Slack messages. They then quickly realize they need to filter this through a Lambda or something with logic in it in order to ignore certain types of alerts. As their alert response processes mature, they also find they need to send these to a ticketing system. This enables them to track metrics, have an audit history, and perform escalations.
You want to ensure you can minimally do the following:
- Receive a notification about a GuardDuty alert from any of your accounts and any region you enabled it in.
- Be able to alert on Access Denied errors from CloudTrail logs, specifically for your production accounts.
- Be able to search through CloudTrail logs to see all of the actions performed by a principal during a time period.
Consider turning on more logging sources, such as the following, plus any others you might be using (full list of known log sources on AWS is here):
- VPC DNS queries
- VPC Flow Logs (These are not as useful as many assume)
- CloudTrail S3 access and Lambda invokes
- S3 access logs
- Load balancers
- CloudFront
Stage 5: Secure IAM access
- Use SSO for access.
- Remove all IAM users.
- Remove all unused IAM roles.
- Reduce the privileges of service roles to necessary services.
- Implement pre-commit hooks for secret detection.
- Plan how accounts will be connected.
Stage 6: Reduce attack surface and mitigate compromises
- Apply SCPs.
- Have no publicly facing EC2s or S3 buckets.
- Enforce IMDSv2 on all EC2s.
SCPs (Service Control Policies) can be used to restrict actions an account can perform. See examples in my post AWS SCP Best Practices. You can make exceptions so only a special IAM role, such as that used for Stack Set deployments, can bypass the restrictions. You can apply different SCPs to different accounts and these can ensure that no one, not even the root user, can perform certain actions.
The SCPs to apply should include:
- Deny root user access.
- Allow only approved regions.
- Allow only approved services.
- Deny ability to create IAM access keys.
- Require the use of IMDSv2.
- Deny ability to leave Organization.
- Deny ability to make a VPC accessible from the Internet that isn’t already for specific accounts.
- Deny ability to disrupt GuardDuty, Access Analyzer, CloudTrail, S3 Public Block Access, and other security services.
- Deny ability to disrupt CloudWatch Event collection or other aspects of your monitoring and alerting pipeline.
- Deny ability to modify important IAM roles, such as one used for Stack Sets, incident response, or vendors performing monitoring.
Stage 7: Reproducibility and ownership
- Use Infrastructure as Code.
- Control AMI (Amazon Machine Images) and package sourcing.
- Apply tagging strategy
Once you control infrastructure changes through code, you can also use two-person rule deployments, where one person proposes the changes and another signs off in order for the change to be implemented. You can also use tools to scan these proposed changes before they are deployed. Options for doing this are described here.
With more people working in an AWS environment, you can lose track of what resources are associated with which applications or teams. When a problem is found with a resource, you want to be able to quickly identify who the owner is, without looking through the CloudTrail logs. This becomes especially important when a CI/CD system is solely responsible for creating resources, as you become less able to identify the owner. Therefore, you should have a tagging strategy. This can be enforced via SCPs or scanning or monitoring can be used to detect when required tags are not used
Stage 8: Enhance detection and least privilege refinement
- Implement real-time monitoring.
- Implement automated remediation.
- Refine IAM policies.
- Deploy honey tokens.
Stage 9: Secure network communications
- Move all non-public network resources into private subnets and proxy outbound requests so you can filter and block them.
- Restrict egress network traffic.
Stage 10: Incident preparation
- Limit the blast radius of incidents.
- Practice responding to incidents.
Consider breaking up services into pieces so that if one is compromised the blast radius is more limited.
Ensure there is a separation of duties so the security team can monitor the production accounts without being able to cause incidents there, and those with production access, cannot impact the security team.
If you are lucky enough to not have any security incidents, you should practice what happens when systems or IAM roles are compromised, either via a planned exercise, or red team engagement. How long does it take you to identify, understand, and react? Use the lessons learned to improve