Detecting Security Threats and Investigating Incidents with AWS CloudTrail (2026 Edition): CIS Benchmark Monitoring, GuardDuty/Security Hub Integration, and Forensic Investigation in Practice

2 a.m. Suppose you're jolted awake by an alert that "an unfamiliar IAM user has been created in the production account." What's the first thing you should look at?

There's one answer. Is CloudTrail still alive. Because the standard first thing an attacker does is "stop the trail and erase their footprints." The logs after a trail is stopped don't capture the crucial act of compromise. Both detection and investigation start from here first.

I designed and led the reliability layer of a serverless (Lambda + DynamoDB) payment platform, and maintained 0 double charges during production operation. The core of that discipline is "designing from the start a state where correctness can be proven non-repudiably even after the fact." In a system where money moves, "probably fine" doesn't fly. Leaving in a tamper-proof form who, when, with which permission, from where, did what, and being able to pull it up in seconds when needed — that foundation is security operations centered on CloudTrail.

This article is a guide for assembling CloudTrail, not as "a place to put audit logs," but as a real-combat playbook for threat detection, fraud detection, and incident investigation (forensics). It handles end to end: the detection code, CIS-benchmark-compliant monitoring, integration with GuardDuty/Security Hub/Detective, Athena/Lake investigation queries, and evidence preservation.

This article's premise and division of labor: the basic setup of CloudTrail (multi-Region trail, encryption, enabling integrity validation, the mechanism of EventBridge/CloudWatch integration itself) is handled by the CloudTrail complete guide (the pillar article). On that foundation, this article concentrates on the deep dive into security operations and forensics. The specs, conditions, and finding names are all checked against the AWS official documentation (as of June 2026). Because pricing and the status of managed features are revised quickly, always confirm the latest official information before going to production. The account ID (111122223333), bucket names, and Region are illustrative.

0. Mental model: CloudTrail is the foundation of "detection" and "investigation," and the "evidence"

In the context of security operations, let me fix CloudTrail's role split into 3. This is the map of the whole article.

Detection = notice what's happening now. Real-time-ness is the lifeline. EventBridge / CloudWatch metric filters / GuardDuty.

Investigation = reconstruct what happened in the past with cross-cutting queries. Comprehensiveness is the lifeline. Athena / CloudTrail Lake.

Evidence = prove non-repudiably that "this credential really hit this API." Integrity is the lifeline. Log integrity validation / S3 Object Lock.

A CloudTrail event has "core fields" used repeatedly in both detection and investigation. Put these in your body and you can decipher all the later queries and filters (from the official userIdentity / event-record spec).

Note: the correct values of userIdentity.type include IdentityCenterUser, but a value IAMIdentityCenter doesn't exist. Don't confuse them when writing filters or queries.

1. The top-priority detection: an attacker comes to "stop the trail" first

Back to the opening scenario. The single most valuable thing in incident detection is, plainly, the immediate detection of "a tampering operation against CloudTrail itself." Because if this gets stopped, the premise of all subsequent detection and investigation collapses.

The eventSource to monitor is cloudtrail.amazonaws.com, and the central eventNames are the next 4.

"Look back at the logs later" is too slow for this. Because you need to detect in near real time, in seconds, the first choice is EventBridge → Lambda/SNS, not a CloudWatch metric filter (which has a buffer of several minutes). EventBridge can receive CloudTrail's API calls as detail-type: "AWS API Call via CloudTrail".

The EventBridge rule (Terraform)

# 証跡改ざん操作をニアリアルタイムで捕捉する EventBridge ルール
resource "aws_cloudwatch_event_rule" "cloudtrail_tampering" {
  name        = "detect-cloudtrail-tampering"
  description = "Detect attempts to stop, delete, or weaken CloudTrail logging"

  event_pattern = jsonencode({
    "detail-type" : ["AWS API Call via CloudTrail"],
    "detail" : {
      "eventSource" : ["cloudtrail.amazonaws.com"],
      "eventName" : ["StopLogging", "DeleteTrail", "UpdateTrail", "PutEventSelectors"]
    }
  })
}

resource "aws_cloudwatch_event_target" "tampering_to_sns" {
  rule      = aws_cloudwatch_event_rule.cloudtrail_tampering.name
  target_id = "notify-secops"
  arn       = aws_sns_topic.security_alerts.arn
}

# Lambda でトリアージ情報を整形してから通知したい場合は、こちらをターゲットに
resource "aws_cloudwatch_event_target" "tampering_to_lambda" {
  rule      = aws_cloudwatch_event_rule.cloudtrail_tampering.name
  target_id = "triage-lambda"
  arn       = aws_lambda_function.tampering_triage.arn
}

The triage Lambda (TypeScript)

Rather than just firing a notification, shape it into a form where "who, from where, which trail" they tried to stop is instantly readable. An alert has meaning only when it arrives in a "readable and judgable" state.

import type { EventBridgeEvent } from "aws-lambda";
import { SNSClient, PublishCommand } from "@aws-sdk/client-sns";

const sns = new SNSClient({});
const TOPIC_ARN = process.env.SNS_TOPIC_ARN!; // 環境変数のみ。ハードコード禁止

// CloudTrail 経由の API コール詳細（必要フィールドのみ）
interface CloudTrailDetail {
  eventName: string;
  eventSource: string;
  sourceIPAddress?: string;
  awsRegion?: string;
  userIdentity?: {
    type?: string;
    arn?: string;
    sessionContext?: { attributes?: { mfaAuthenticated?: string } };
  };
  requestParameters?: Record<string, unknown>;
}

export const handler = async (
  event: EventBridgeEvent<"AWS API Call via CloudTrail", CloudTrailDetail>,
): Promise<void> => {
  const d = event.detail;
  const id = d.userIdentity;

  const message = [
    "[CRITICAL] CloudTrail tampering detected",
    `eventName : ${d.eventName}`,
    `principal : ${id?.arn ?? "unknown"} (type=${id?.type ?? "unknown"})`,
    `mfa       : ${id?.sessionContext?.attributes?.mfaAuthenticated ?? "n/a"}`,
    `sourceIP  : ${d.sourceIPAddress ?? "unknown"}`,
    `region    : ${d.awsRegion ?? "unknown"}`,
    `time      : ${event.time}`,
    `params    : ${JSON.stringify(d.requestParameters ?? {})}`,
  ].join("\n");

  await sns.send(
    new PublishCommand({
      TopicArn: TOPIC_ARN,
      Subject: `CloudTrail tampering: ${d.eventName}`,
      Message: message,
    }),
  );
};

A design pressure point: consider prevention of "can't stop the trail" as a set too. Deny cloudtrail:StopLogging / cloudtrail:DeleteTrail with an SCP at the organization level, and a compromised principal in a member account can't stop the trail. The design of defense-in-depth with SCPs and multi-account auditing is detailed in the organization-trail / multi-account audit guide (not chased here). Detection (this chapter) and prevention (SCPs) are two wheels of a cart.

2. Detecting attack signs: a collection of CIS-benchmark-compliant alerts

The net to stretch next after trail tampering is monitoring alerts for "typical behaviors of compromise." Here, one most important honesty point.

The well-known monitoring alerts for "root usage," "unauthorized APIs," "sign-in without MFA," etc. are NOT the ones on the 'examples' page of CloudTrail's user guide. The metric filters CloudTrail officially lists as "examples" are actually only three.

The "CloudWatch alarm examples" page of CloudTrail's official documentation provides only these 3 examples.

Note that the filter pattern for console sign-in failure is, per the official docs, the following (usable as-is).

{ ($.eventName = ConsoleLogin) && ($.errorMessage = "Failed authentication") }

So, where do the standard alert groups widely spread as "CloudTrail monitoring best practices" — root usage, unauthorized APIs, sign-in without MFA, CMK disabling… — come from? The source is the CIS AWS Foundations Benchmark (and AWS Security Hub's CIS standard controls). To be precise, Security Hub's CloudWatch.1 – CloudWatch.14 controls correspond to each CIS recommendation (v1.2.0's 3.x / v1.4.0's 4.x).

Why being able to correctly cite the source is important. When asked in an audit or review "what is this monitoring compliant with," answering "it's CloudTrail's official examples" is inaccurate and damages trust. Being able to say "it's compliant with §3/§4 of the CIS AWS Foundations Benchmark and Security Hub's CloudWatch controls" is accountability for the design itself.

A list of CIS-compliant monitoring alerts (Security Hub CloudWatch controls)

A correspondence table based on the official Security Hub user guide. Each control is implemented with the same mechanism of "flow CloudTrail → CloudWatch Logs and place a metric filter + alarm."

Supplement (from the official note): these Security Hub controls become FAILED unless they exactly match "the precise metric filter the CIS prescribes." Add your own fields or phrases and it fails. On the other hand, only the remediation procedure examples for IAM-policy change (CloudWatch.4) and route-table change (CloudWatch.13) present a recommendation AWS intentionally changed from the CIS pattern (to "target only IAM/EC2 API calls"). For compliance purposes, use the CIS pattern strictly; for operational-noise-reduction purposes, use AWS's remediation examples — distinguish them.

Implementation example 1: detecting unauthorized APIs (AccessDenied surge) — CIS 3.1 / CloudWatch.2

An attacker who grabbed a compromised credential first probes "what can I do with this credential." In that process, AccessDenied / UnauthorizedOperation surges in a short time. This is an extremely useful signal of reconnaissance.

# CIS AWS Foundations Benchmark 3.1 / Security Hub CloudWatch.2 準拠
# 未認可API呼び出しの検知（CISが規定する正確なパターン）
resource "aws_cloudwatch_log_metric_filter" "unauthorized_api_calls" {
  name           = "CIS-UnauthorizedAPICalls"
  log_group_name = var.cloudtrail_log_group_name # CloudTrailの配信先ロググループ

  pattern = "{ ($.errorCode = \"*UnauthorizedOperation\") || ($.errorCode = \"AccessDenied*\") }"

  metric_transformation {
    name      = "UnauthorizedAPICalls"
    namespace = "CISBenchmark"
    value     = "1"
  }
}

resource "aws_cloudwatch_metric_alarm" "unauthorized_api_calls" {
  alarm_name          = "CIS-3.1-UnauthorizedAPICalls"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = 1
  metric_name         = "UnauthorizedAPICalls"
  namespace           = "CISBenchmark"
  period              = 300
  statistic           = "Sum"
  threshold           = 1
  alarm_description   = "CIS 3.1: Unauthorized API calls detected"
  alarm_actions       = [aws_sns_topic.security_alerts.arn]
  treat_missing_data  = "notBreaching"
}

Implementation example 2: detecting root-user usage — CIS 1.7/4.3 / CloudWatch.1

Because root has all permissions, its use in daily operation should in principle be zero. Pick up ones where userIdentity.type is Root and that aren't calls by an AWS service itself (invokedBy).

# CIS / Security Hub CloudWatch.1 準拠：root ユーザー利用の検知
resource "aws_cloudwatch_log_metric_filter" "root_usage" {
  name           = "CIS-RootAccountUsage"
  log_group_name = var.cloudtrail_log_group_name

  pattern = "{ $.userIdentity.type = \"Root\" && $.userIdentity.invokedBy NOT EXISTS && $.eventType != \"AwsServiceEvent\" }"

  metric_transformation {
    name      = "RootAccountUsage"
    namespace = "CISBenchmark"
    value     = "1"
  }
}

resource "aws_cloudwatch_metric_alarm" "root_usage" {
  alarm_name          = "CIS-RootAccountUsage"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = 1
  metric_name         = "RootAccountUsage"
  namespace           = "CISBenchmark"
  period              = 300
  statistic           = "Sum"
  threshold           = 1
  alarm_description   = "CIS: Root account used"
  alarm_actions       = [aws_sns_topic.security_alerts.arn]
  treat_missing_data  = "notBreaching"
}

Noise countermeasure: always set alarm_actions (SNS, etc.) on these alarms. Security Hub's CloudWatch.15 "alarms should have an action set" / CloudWatch.17 "alarm actions should be enabled" (these are NIST-family controls) hit exactly the typical hole of "created an alarm but there's no notification target / it's disabled." If the last meter of detection — the notification — isn't connected, the detector is the same as nonexistent.

3. Riding on managed detection: GuardDuty / Security Hub / Detective

A metric filter can pick up only "a pre-decided pattern." To catch unknown, anomalous behavior, the proper approach is to ride on managed detection services taking CloudTrail as input. Grasp the 3-way role split.

GuardDuty: representative findings derived from CloudTrail (the official IAM finding types)

GuardDuty, with CloudTrail's management events as the main data source, generates findings about IAM entities / access keys (the Resource Type is AccessKey). Let me list, classified, the representative findings taking CloudTrail as input that are documented officially.

The ML-anomaly-detection family (*:IAMUser/AnomalousBehavior) — the data source is CloudTrail management events

The signature / threat-intel family

The difference from CredentialAccess:IAMUser/CompromisedCredentials: this isn't ML anomaly detection but a High finding indicating that an access key judged "already leaked" by Amazon's threat intelligence was used. The data source is documented officially as "a feature included in Foundational Data Source Protection," and its generation basis differs from the AnomalousBehavior family above (ML of CloudTrail management events). Don't confuse them.

GuardDuty findings flow to EventBridge (detail-type: "GuardDuty Finding"). Using this, you can assemble "if a finding of severity 7 or higher appears, immediately SNS / an auto-containment Lambda" in the same way as Chapter 1. The defense-in-depth, multi-account design of GuardDuty alone (delegated administrator, EventBridge automated response, Terraform) is split into the GuardDuty deep-dive guide.

# GuardDuty の重大度の高い Finding を SecOps に即通知する
resource "aws_cloudwatch_event_rule" "guardduty_high_severity" {
  name = "guardduty-high-severity-findings"
  event_pattern = jsonencode({
    "source" : ["aws.guardduty"],
    "detail-type" : ["GuardDuty Finding"],
    # GuardDuty の severity は数値。7.0 以上 = High 以上を対象にする
    "detail" : { "severity" : [{ "numeric" : [">=", 7.0] }] }
  })
}

Security Hub: visualize "the holes in the detection net" with the CIS score

If GuardDuty is "dynamic detection," Security Hub is "static scoring of the posture." Enable the CIS / AWS FSBP standards including CloudWatch.1–17 from the previous chapter, and the holes in detection themselves — "no root-monitoring alert," "no monitoring of CloudTrail configuration changes" — are listed as FAILED. It's a device that prevents "you think you built a detector but didn't."

4. The incident-investigation playbook: forensics with Athena / Lake

After detection comes investigation. If detection is a "point," investigation is "lines and a plane" — reconstruct all of the compromised principal's actions chronologically and confirm the scope of the damage.

For a cross-cutting query over a period exceeding 90 days, or over a large volume of logs, Athena (SQL directly on CloudTrail logs on S3) is realistic. Because scan volume = billing, the iron rule is to narrow the date and Region with partition projection. CloudTrail Lake can query an immutable event data store directly with Trino SQL, but it ended new-customer onboarding as of May 31, 2026 (existing customers can continue). For a new build, Athena + S3 is the standard answer.

The Athena queries below assume that an external table for CloudTrail logs (with partition projection) has already been created. How to create the table definition (ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' and the partition-projection properties) is left to the pillar article.

The IR playbook: detection → triage → scope → containment → evidence preservation

1. 検知    : EventBridge / GuardDuty / CISアラームで気づく
2. トリアージ: 「本物か・誤検知か」「侵害プリンシパルは誰か(ARN)」を確定
3. スコープ : Athena/Lake で侵害ARNの全行動を時系列再構成、被害範囲を確定
4. 封じ込め : 鍵の無効化/失効、ポリシーのデタッチ、セッションの取り消し
5. 証拠保全 : 整合性検証(validate-logs)、ログの隔離コピー、否認不能化（第5章）

Let me list the queries you repeatedly hit in the field, used in step 3, "scoping."

Query 1: reconstruct "all actions" of the compromised principal chronologically

What, when, and from where the ARN identified in triage did. This is the starting point of all investigation.

-- 侵害が疑われるプリンシパルの全 API コールを時系列で並べる
SELECT
  eventtime,
  eventsource,
  eventname,
  sourceipaddress,
  awsregion,
  errorcode,
  readonly
FROM cloudtrail_logs
WHERE useridentity.arn = 'arn:aws:sts::111122223333:assumed-role/AppRole/suspicious-session'
  AND timestamp BETWEEN '2026/06/26' AND '2026/06/27' -- パーティション射影で絞る
ORDER BY eventtime ASC;

Query 2: signs of persistence — new access-key creation

An attacker creates access keys to solidify their foothold (corresponds to GuardDuty's Persistence:IAMUser/AnomalousBehavior).

-- 調査期間内に作成されたアクセスキー（誰が・誰に対して・どこから）
SELECT
  eventtime,
  useridentity.arn          AS actor,
  responseelements,         -- 作成されたキーIDやユーザー名が入る
  sourceipaddress
FROM cloudtrail_logs
WHERE eventname = 'CreateAccessKey'
  AND timestamp BETWEEN '2026/06/20' AND '2026/06/27'
ORDER BY eventtime ASC;

Query 3: signs of privilege escalation — IAM grant-system operations

-- 権限昇格に使われがちな IAM 操作を洗い出す
SELECT eventtime, useridentity.arn AS actor, eventname, requestparameters, sourceipaddress
FROM cloudtrail_logs
WHERE eventsource = 'iam.amazonaws.com'
  AND eventname IN (
    'AttachUserPolicy', 'AttachRolePolicy', 'AttachGroupPolicy',
    'PutUserPolicy', 'PutRolePolicy', 'PutGroupPolicy',
    'AddUserToGroup', 'CreatePolicyVersion', 'UpdateAssumeRolePolicy'
  )
  AND timestamp BETWEEN '2026/06/20' AND '2026/06/27'
ORDER BY eventtime ASC;

Query 4: signs of data exfiltration — snapshot sharing, bucket publication

Back up, in investigation, the control-plane operations Exfiltration:IAMUser/AnomalousBehavior captures.

-- スナップショットの外部共有・S3公開化など、持ち出しに繋がる制御操作
SELECT eventtime, useridentity.arn AS actor, eventname, requestparameters, sourceipaddress
FROM cloudtrail_logs
WHERE eventname IN (
    'ModifySnapshotAttribute',     -- スナップショットを他アカウントへ共有
    'CreateSnapshot',
    'PutBucketPolicy',             -- バケットポリシーの書き換え
    'PutBucketAcl',
    'PutBucketReplication',        -- 別バケットへの複製設定
    'DeleteBucketPublicAccessBlock'-- パブリックアクセスブロック解除
  )
  AND timestamp BETWEEN '2026/06/20' AND '2026/06/27'
ORDER BY eventtime ASC;

Query 5: tracking source IPs / sessions — grasping lateral movement

Is the compromise confined to "one ARN," or is it moving laterally? With a suspicious IP as the axis, surface all the principals that operated from that IP.

-- 特定の不審 IP から行動したすべてのプリンシパルと操作
SELECT
  useridentity.arn AS actor,
  useridentity.type AS actor_type,
  eventname,
  COUNT(*) AS calls,
  MIN(eventtime) AS first_seen,
  MAX(eventtime) AS last_seen
FROM cloudtrail_logs
WHERE sourceipaddress = '203.0.113.45'
  AND timestamp BETWEEN '2026/06/20' AND '2026/06/27'
GROUP BY useridentity.arn, useridentity.type, eventname
ORDER BY calls DESC;

Query 6: surfacing MFA-less sessions

-- MFA 認証されていないセッションでの書き込み系操作（侵害セッションの疑い）
SELECT eventtime, useridentity.arn AS actor, eventname, sourceipaddress
FROM cloudtrail_logs
WHERE useridentity.sessioncontext.attributes.mfaauthenticated = 'false'
  AND readonly = false
  AND timestamp BETWEEN '2026/06/26' AND '2026/06/27'
ORDER BY eventtime ASC;

A cost pitfall: always attach the WHERE timestamp BETWEEN ... partition-projection condition every time. Forget this and it full-scans years' worth, all-Regions logs, and billing spikes. Because Athena bills by the GB scanned, make "narrow by date first" a habit even in forensics, until it's second nature.

5. Evidence preservation and non-repudiation: prove "this credential really did it"

Once the scope is fixed in investigation, the last step is evidence preservation. Neglect this and the hard-won investigation result loses its power in an audit, legal proceedings, or internal explanation, because "it can't be proven untampered."

Log integrity validation: tamper detection and non-repudiation

CloudTrail's log-file integrity validation is built with hashing by SHA-256 and digital signing by SHA-256 with RSA. Every hour, a digest file containing references and hashes to the group of log files delivered in that hour is created, and each digest forms a chain by including the signature of the previous digest. With this, you can detect tampering, deletion, and forgery of the logs, and further can actively assert even "that not a single log was delivered in a certain period."

The official wording succinctly expresses this feature's value.

A validated log file lets you assert positively that "a particular user credential performed particular API activity" — this is the core of non-repudiation in security / forensic investigation.

Validation can be run with a single AWS CLI command.

# S3 上の指定期間のログを digest チェーンに照らして検証する
aws cloudtrail validate-logs \
  --trail-arn arn:aws:cloudtrail:ap-northeast-1:111122223333:trail/org-audit-trail \
  --start-time 2026-06-26T00:00:00Z \
  --end-time   2026-06-27T00:00:00Z

If you want to move the logs to another location (an isolation bucket, etc.) and validate there, because the CLI validation presupposes the original location where the logs were delivered, implement custom validation (the official docs have the procedure). If you make an isolated copy of the evidence, weave this point into the design.

Storage design for non-repudiation

Even if the validation algorithm is robust, it's meaningless if the logs themselves are deleted later. You need storage design that ensures their value as evidence.

The design philosophy: ideally, separate "the account that generates logs" and "the account that stores logs." Even if the production account is completely compromised, the WORM-ized logs in the Log Archive account remain intact and function as non-repudiable evidence. "Creating from the start a state where correctness can be proven after the fact" on a payment platform is exactly this stance.

6. Summary: a detection → investigation → preservation cheat sheet

Let me fold into one sheet, at the end, the key points of elevating CloudTrail from "a log storehouse" to "the foundation of security operations."

The line you must not cross is this.

A detector becomes a detector only when the notification is connected (CloudWatch.15/17). Evidence becomes evidence only when the logs become tamper- and delete-proof (integrity validation + WORM). "An alarm exists but there's no notification target," "logs exist but can be deleted" are the same as nonexistent.

And let me repeat the most important honesty point. Standard alerts like root usage and unauthorized-API monitoring come from the CIS AWS Foundations Benchmark (Security Hub's controls), not CloudTrail's official 'examples.' Being able to correctly cite the source is accountability itself in an audit or review.

Security operations isn't "set it and forget it"; its value is decided by whether you can fully design the loop of detection → investigation → preservation into a state anyone can run. In the "money moves" domain of payments, I led the design of reliability / auditing that underpins 0 double charges. Dropping "a state where correctness can be proven non-repudiably after the fact" into code from the start — I can apply that experience to the security monitoring / incident-response design of your AWS environment.

If you're troubled by designing a threat-detection / forensics / IR posture centered on CloudTrail, feel free to consult me from contact. From surfacing the current detection holes, let's nail it down together.

Related deep-dive articles: for CloudTrail's basic design see the CloudTrail complete guide, for GuardDuty's multi-layer detection the GuardDuty threat-detection guide, for network-layer defense the AWS WAF defense-in-depth guide, and for data-store permission design the DynamoDB security guide.