Advanced

CloudTrail Monitoring for Agent Activity

Build a real-time monitoring and alerting pipeline that detects when AI agents attempt destructive actions on your AWS resources. CloudTrail, EventBridge, SNS, CloudWatch, and AWS Config working together.

Configuring CloudTrail for All Regions

CloudTrail is the foundation of all AWS monitoring. It records every API call made in your account. For AI agent safety, you need a multi-region trail that captures management events (which include all create, delete, and modify operations).

Bash — Create a multi-region CloudTrail trail
# Create an S3 bucket for CloudTrail logs
aws s3api create-bucket \
  --bucket company-cloudtrail-logs \
  --region us-east-1

# Apply bucket policy to allow CloudTrail to write logs
aws s3api put-bucket-policy \
  --bucket company-cloudtrail-logs \
  --policy '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "AWSCloudTrailAclCheck",
        "Effect": "Allow",
        "Principal": {"Service": "cloudtrail.amazonaws.com"},
        "Action": "s3:GetBucketAcl",
        "Resource": "arn:aws:s3:::company-cloudtrail-logs"
      },
      {
        "Sid": "AWSCloudTrailWrite",
        "Effect": "Allow",
        "Principal": {"Service": "cloudtrail.amazonaws.com"},
        "Action": "s3:PutObject",
        "Resource": "arn:aws:s3:::company-cloudtrail-logs/AWSLogs/*",
        "Condition": {
          "StringEquals": {"s3:x-amz-acl": "bucket-owner-full-control"}
        }
      }
    ]
  }'

# Create the multi-region trail
aws cloudtrail create-trail \
  --name ai-agent-audit-trail \
  --s3-bucket-name company-cloudtrail-logs \
  --is-multi-region-trail \
  --include-global-service-events \
  --enable-log-file-validation

# Start logging
aws cloudtrail start-logging --name ai-agent-audit-trail

# Verify the trail is active
aws cloudtrail get-trail-status --name ai-agent-audit-trail
💡
Log file validation: The --enable-log-file-validation flag creates a digest file that lets you verify CloudTrail logs have not been tampered with. This is critical if you need to investigate an incident and prove the logs are authentic.

EventBridge Rules for Destructive API Calls

EventBridge (formerly CloudWatch Events) can match CloudTrail events in near real-time (typically within 1-5 minutes of the API call). You can create rules that trigger when specific destructive API calls are made.

EventBridge Rule: Catch All Destructive EC2 Actions

JSON — EventBridge rule pattern for EC2 destructive actions
{
  "source": ["aws.ec2"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["ec2.amazonaws.com"],
    "eventName": [
      "TerminateInstances",
      "DeleteVpc",
      "DeleteSubnet",
      "DeleteSecurityGroup",
      "DeleteNatGateway",
      "DeleteVolume",
      "DeleteSnapshot",
      "DeleteKeyPair"
    ]
  }
}

EventBridge Rule: Catch All Destructive Actions Across Services

JSON — Comprehensive destructive action pattern
{
  "source": ["aws.ec2", "aws.rds", "aws.s3", "aws.lambda", "aws.eks",
             "aws.dynamodb", "aws.elasticache", "aws.cloudformation",
             "aws.iam", "aws.route53"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventName": [
      "TerminateInstances",
      "DeleteVpc",
      "DeleteSubnet",
      "DeleteSecurityGroup",
      "DeleteVolume",
      "DeleteDBInstance",
      "DeleteDBCluster",
      "DeleteDBSnapshot",
      "DeleteBucket",
      "DeleteObject",
      "DeleteFunction",
      "DeleteCluster",
      "DeleteNodegroup",
      "DeleteTable",
      "DeleteCacheCluster",
      "DeleteReplicationGroup",
      "DeleteStack",
      "DeleteRole",
      "DeletePolicy",
      "DeleteUser",
      "DeleteHostedZone"
    ]
  }
}
Bash — Create the EventBridge rule
# Create the rule
aws events put-rule \
  --name "detect-destructive-actions" \
  --description "Catches all destructive AWS API calls from any principal" \
  --event-pattern file://destructive-pattern.json \
  --state ENABLED

# Verify the rule
aws events describe-rule --name "detect-destructive-actions"

SNS Notifications for Destructive Events

Connect the EventBridge rule to an SNS topic that sends notifications to email, Slack, or PagerDuty.

Bash — Create SNS topic and subscribe
# Create the SNS topic
aws sns create-topic --name ai-agent-destructive-alerts

# Subscribe email
aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:ai-agent-destructive-alerts \
  --protocol email \
  --notification-endpoint security-team@company.com

# Subscribe a Slack webhook via HTTPS
aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:ai-agent-destructive-alerts \
  --protocol https \
  --notification-endpoint https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXX

# Grant EventBridge permission to publish to SNS
aws sns set-topic-attributes \
  --topic-arn arn:aws:sns:us-east-1:123456789012:ai-agent-destructive-alerts \
  --attribute-name Policy \
  --attribute-value '{
    "Version": "2012-10-17",
    "Statement": [{
      "Sid": "AllowEventBridgePublish",
      "Effect": "Allow",
      "Principal": {"Service": "events.amazonaws.com"},
      "Action": "SNS:Publish",
      "Resource": "arn:aws:sns:us-east-1:123456789012:ai-agent-destructive-alerts"
    }]
  }'

# Add SNS as the target for the EventBridge rule
aws events put-targets \
  --rule "detect-destructive-actions" \
  --targets "Id"="sns-target","Arn"="arn:aws:sns:us-east-1:123456789012:ai-agent-destructive-alerts"

CloudWatch Alarms for Unusual Activity Patterns

Beyond catching individual destructive calls, use CloudWatch to detect unusual patterns that might indicate an AI agent has gone rogue — such as a sudden spike in API calls or access denied errors.

Bash — CloudWatch metric filter for access denied events
# Create a metric filter on CloudTrail logs
aws logs put-metric-filter \
  --log-group-name "CloudTrail/DefaultLogGroup" \
  --filter-name "AccessDeniedCount" \
  --filter-pattern '{ $.errorCode = "AccessDenied" || $.errorCode = "Client.UnauthorizedAccess" }' \
  --metric-transformations \
    metricName=AccessDeniedCount,metricNamespace=AIAgentSecurity,metricValue=1

# Create an alarm that triggers when more than 10 access denied events occur in 5 minutes
aws cloudwatch put-metric-alarm \
  --alarm-name "ai-agent-access-denied-spike" \
  --alarm-description "More than 10 access denied events in 5 minutes - possible AI agent attempting restricted actions" \
  --metric-name AccessDeniedCount \
  --namespace AIAgentSecurity \
  --statistic Sum \
  --period 300 \
  --threshold 10 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:ai-agent-destructive-alerts

Alarm for API Call Volume Spike

Bash — Alarm for unusual API call volume from AI agent role
# Create metric filter for AI agent role API calls
aws logs put-metric-filter \
  --log-group-name "CloudTrail/DefaultLogGroup" \
  --filter-name "AIAgentAPICalls" \
  --filter-pattern '{ $.userIdentity.arn = "*AIAgentRole*" }' \
  --metric-transformations \
    metricName=AIAgentAPICallCount,metricNamespace=AIAgentSecurity,metricValue=1

# Alarm if AI agent makes more than 500 API calls in 10 minutes
aws cloudwatch put-metric-alarm \
  --alarm-name "ai-agent-api-call-spike" \
  --alarm-description "AI agent role making unusually high number of API calls" \
  --metric-name AIAgentAPICallCount \
  --namespace AIAgentSecurity \
  --statistic Sum \
  --period 600 \
  --threshold 500 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:ai-agent-destructive-alerts

AWS Config Rules for Compliance

AWS Config continuously evaluates your resource configurations against rules. Use it to ensure resource protection is always enabled:

Bash — AWS Config rules for resource protection compliance
# Ensure EC2 instances have termination protection enabled
aws configservice put-config-rule \
  --config-rule '{
    "ConfigRuleName": "ec2-termination-protection-enabled",
    "Description": "Checks that all EC2 instances have termination protection enabled",
    "Source": {
      "Owner": "AWS",
      "SourceIdentifier": "EC2_INSTANCE_DETAILED_MONITORING_ENABLED"
    },
    "Scope": {
      "ComplianceResourceTypes": ["AWS::EC2::Instance"]
    }
  }'

# Ensure RDS instances have deletion protection
aws configservice put-config-rule \
  --config-rule '{
    "ConfigRuleName": "rds-instance-deletion-protection-enabled",
    "Description": "Checks that all RDS instances have deletion protection enabled",
    "Source": {
      "Owner": "AWS",
      "SourceIdentifier": "RDS_INSTANCE_DELETION_PROTECTION_ENABLED"
    },
    "Scope": {
      "ComplianceResourceTypes": ["AWS::RDS::DBInstance"]
    }
  }'

# Ensure S3 buckets have versioning enabled
aws configservice put-config-rule \
  --config-rule '{
    "ConfigRuleName": "s3-bucket-versioning-enabled",
    "Description": "Checks that all S3 buckets have versioning enabled",
    "Source": {
      "Owner": "AWS",
      "SourceIdentifier": "S3_BUCKET_VERSIONING_ENABLED"
    },
    "Scope": {
      "ComplianceResourceTypes": ["AWS::S3::Bucket"]
    }
  }'

# Check compliance status
aws configservice get-compliance-details-by-config-rule \
  --config-rule-name rds-instance-deletion-protection-enabled \
  --compliance-types NON_COMPLIANT

Building the Complete Alert Pipeline

Here is the end-to-end architecture for monitoring AI agent activity:

1
CloudTrail — Records every API call across all regions. Logs are stored in S3 and optionally sent to CloudWatch Logs.
2
EventBridge — Matches CloudTrail events against your rule patterns in near real-time. Triggers within 1-5 minutes of the API call.
3
SNS — Fans out notifications to multiple subscribers: email, Slack webhook, PagerDuty, or a Lambda function for custom processing.
4
CloudWatch Alarms — Monitors aggregated metrics for unusual patterns: access denied spikes, API call volume anomalies, or off-hours activity.
5
AWS Config — Continuously verifies that protection settings (termination protection, deletion protection, versioning) remain enabled. Alerts if any resource becomes non-compliant.

Terraform: Complete Monitoring Pipeline

HCL — Terraform: EventBridge + SNS alert pipeline
# SNS Topic for alerts
resource "aws_sns_topic" "destructive_alerts" {
  name = "ai-agent-destructive-alerts"
}

resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.destructive_alerts.arn
  protocol  = "email"
  endpoint  = "security-team@company.com"
}

# SNS Topic Policy
resource "aws_sns_topic_policy" "allow_eventbridge" {
  arn = aws_sns_topic.destructive_alerts.arn
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Sid       = "AllowEventBridgePublish"
      Effect    = "Allow"
      Principal = { Service = "events.amazonaws.com" }
      Action    = "SNS:Publish"
      Resource  = aws_sns_topic.destructive_alerts.arn
    }]
  })
}

# EventBridge Rule
resource "aws_cloudwatch_event_rule" "destructive_actions" {
  name        = "detect-destructive-actions"
  description = "Detects destructive AWS API calls"

  event_pattern = jsonencode({
    source      = ["aws.ec2", "aws.rds", "aws.s3", "aws.lambda",
                   "aws.eks", "aws.dynamodb", "aws.cloudformation"]
    detail-type = ["AWS API Call via CloudTrail"]
    detail = {
      eventName = [
        "TerminateInstances", "DeleteVpc", "DeleteSubnet",
        "DeleteDBInstance", "DeleteDBCluster",
        "DeleteBucket", "DeleteObject",
        "DeleteFunction", "DeleteCluster",
        "DeleteTable", "DeleteStack"
      ]
    }
  })
}

# Connect EventBridge to SNS
resource "aws_cloudwatch_event_target" "sns" {
  rule      = aws_cloudwatch_event_rule.destructive_actions.name
  target_id = "sns-alert"
  arn       = aws_sns_topic.destructive_alerts.arn
}
💡
Slack integration: For Slack notifications with rich formatting, use a Lambda function as the SNS subscriber instead of a raw HTTPS webhook. The Lambda can format the CloudTrail event into a readable Slack message with resource names, the acting principal, and a direct link to the AWS Console.

Querying CloudTrail for AI Agent Activity

Use these queries to investigate AI agent behavior:

Bash — CloudTrail queries for AI agent investigation
# Find all actions by the AI agent role in the last 24 hours
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceType,AttributeValue=AWS::IAM::Role \
  --start-time $(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ) \
  --max-results 50 \
  --query 'Events[?contains(CloudTrailEvent, `AIAgentRole`)].[EventName,EventTime,Username]' \
  --output table

# Find all destructive events in the last 7 days
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=TerminateInstances \
  --start-time $(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ) \
  --output json | jq '.Events[] | {EventName, EventTime, Username}'

# Find all access denied events
aws cloudtrail lookup-events \
  --start-time $(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ) \
  --max-results 100 \
  --output json | jq '.Events[] | select(.CloudTrailEvent | fromjson | .errorCode == "AccessDenied") | {EventName: (.CloudTrailEvent | fromjson | .eventName), Principal: (.CloudTrailEvent | fromjson | .userIdentity.arn)}'