# Detecting Infrastructure Drift Without Drowning in Alerts

Canonical: https://kaansystems.com/library/infrastructure-drift-detection
Topic: Platform · Published: 2026-05-12 · Updated: 2026-10-12
Summary: Drift detection at scale produces thousands of 'differences,' most of them noise. A tiered classifier routes the dangerous drift to PagerDuty and sends everything else to a weekly digest.

## Key takeaways

- Drift detection fails not because detection is hard but because the signal-to-noise ratio is brutal — out of roughly 200 drift entries, maybe two are worth a human's attention.
- Score drift into three tiers at detection time: Tier 1 (authorization, audit, availability) pages the on-call, Tier 2 (capacity, cost, consistency) goes to a weekly digest, and Tier 3 (known-good) is suppressed at the detector.
- Tier 1 covers IAM, security groups, KMS, CloudTrail, S3 policies, backup retention, and production load-balancer changes — drift there means either an attacker moving laterally or a bypassed change-control process.
- Suppressing Tier 3 is what makes the whole system usable; without it the digest is unreadable and the Tier 1 alerts get ignored.
- The classifier is regex over the terraform plan JSON — roughly 30 lines of Python mapping resource types to tiers, with no machine learning and no fancy heuristics.
- Watch how often Tier 1 fires: more than once a week on average signals a change-control gap, and less than once a quarter means the classifier is too lenient.

**How do you detect infrastructure drift without drowning in alerts?** Score every drift entry at detection time into three risk tiers: authorization, audit, and availability changes page the on-call; capacity and cost changes go to a weekly digest; and known-good changes get suppressed. The fix isn't better detection. It's classification, and a regex classifier does the routing.

The same plan signal shows up once you have a [multi-account landing zone](/library/multi-account-aws-landing-zone). The guides around it live on the [Terraform hub](/library/terraform). A drift scan is part of the [7-day Assessment](/assessment).

## Why does drift detection turn into alert noise?

Drift detection sounds great in principle and becomes useless in practice. Run `terraform plan` against production once a day and you'll get a delta. Run it across 80 accounts and you'll get a delta the length of a novel.

The teams that turn it on usually turn it off within a month. The signal-to-noise ratio is brutal: out of two hundred drift entries, maybe two represent something you'd actually want a human to look at. The other 198 are autoscaling resizing a node pool, a managed service updating its own backing IAM policy, or a tag your costing system added after the fact.

The fix isn't better detection. It's classification.

## How do you tier drift so only the dangerous changes page?

Treat drift as a continuously scored risk surface, not as a binary state mismatch. Score it at detection time into three tiers:

```mermaid
flowchart LR
  D[terraform plan<br/>detects drift]
  D --> CL{Classifier<br/>regex over plan.json}

  CL -- "IAM, SG, KMS,<br/>CloudTrail, S3 policy" --> T1[Tier 1<br/>Auth · audit · availability]
  CL -- "instance type, tags,<br/>ASG bounds, Lambda mem" --> T2[Tier 2<br/>Capacity · cost · consistency]
  CL -- "ASG counts, AWS-managed<br/>policies, service-linked roles" --> T3[Tier 3<br/>Known-good — suppressed]

  T1 --> PD[PagerDuty<br/>page the on-call · hourly]
  T2 --> SL[Slack digest<br/>weekly review · Mondays]
  T3 --> NULL["/dev/null"]

  classDef src    fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
  classDef gate   fill:#1F4A3E,stroke:#3FA585,stroke-width:2px,color:#E8F5EE;
  classDef hot    fill:#3FA585,stroke:#3FA585,stroke-width:2px,color:#0a1612;
  classDef warm   fill:#1F4A3E,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
  classDef cold   fill:#0a1612,stroke:#2E8B6F,stroke-width:1px,color:#A8B8B0;

  class D src;
  class CL gate;
  class T1,PD hot;
  class T2,SL warm;
  class T3,NULL cold;
```

### Tier 1: Authorization, audit, and availability drift

Changes that, left alone, change who can do what, what gets logged, or what happens when something fails. Every entry in this tier pages the on-call:

- IAM policies, roles, or trust relationships
- Security group rules (especially anything opening 0.0.0.0/0)
- KMS key policies or rotation settings
- CloudTrail or VPC Flow Log configuration
- S3 bucket policies, public access blocks, encryption settings
- Backup plan deletion or retention reduction
- Production load-balancer listener changes

Drift in any of these means either an attacker is moving laterally or a human bypassed your change-control process. Either is worth waking someone up.

### Tier 2: Capacity, cost, and consistency drift

Not dangerous on their own, but indicators of a process gap. These go to a weekly digest:

- Instance type changes (someone resized for cost)
- Tag additions or removals (cost-allocation drift)
- Auto-scaling group min/max adjustments
- RDS parameter group tweaks
- Lambda memory or timeout adjustments

Once a week, an engineer reviews the digest. If a pattern is real ("the data team is rightsizing RDS by hand every Friday"), the IaC catches up. If it's one-off ("someone changed a memory limit during an incident"), the engineer either backports or accepts.

### Tier 3: Known-good drift

Expected changes you don't want cluttering the digest. Suppress them at the detector:

- Autoscaling instance counts inside declared bounds
- AWS-managed IAM policy version updates
- Service-linked role updates
- Default route table associations created by service launches
- CloudFront distribution metric subscriptions

Suppressing Tier 3 is what makes the system usable. Without it, the digest is unreadable and Tier 1 alerts get ignored.

## Implementation

Tooling: `terraform plan` in CI on a schedule is the baseline. For scale, `driftctl` or Pulumi's `pulumi refresh --diff` produce the same signal with less compute.

Run the detector hourly for Tier 1 (cheap, narrow scope), daily for Tier 2 (broader plan), weekly for the full estate. Route Tier 1 to PagerDuty via a webhook that includes the resource ARN and the field that changed. Route Tier 2 to a Slack channel. Route Tier 3 to `/dev/null`.

The classifier is regex on the planned change. Five lines of Python wraps the terraform plan JSON output into a tier judgment. No machine learning. No fancy heuristics. A static table maps `aws_iam_*` and `aws_security_group_rule` and a few others to Tier 1; everything else gets reviewed by the digest.

## The Template

Below is the GitHub Actions workflow that runs the hourly Tier 1 check against an estate of AWS accounts. It uses an OIDC-federated role to assume into each account, runs `terraform plan -detailed-exitcode`, and routes anything matching the Tier 1 pattern to a webhook.

```yaml
# .github/workflows/drift-check.yml
name: Infrastructure drift · Tier 1
on:
  schedule:
    - cron: '0 * * * *'   # hourly
  workflow_dispatch:

permissions:
  id-token: write          # for OIDC role assumption
  contents: read

jobs:
  detect:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        account: [prod-us-east-1, prod-us-west-2, log-archive]
      fail-fast: false
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::${{ secrets.ACCOUNT_ID }}:role/DriftDetector
          aws-region: us-east-1

      - uses: hashicorp/setup-terraform@v3
        with: { terraform_version: 1.9.x }

      - name: terraform plan
        id: plan
        working-directory: ./environments/${{ matrix.account }}
        run: |
          terraform init -input=false
          terraform plan -input=false -lock=false \
            -out=plan.bin -detailed-exitcode || exit_code=$?
          terraform show -json plan.bin > plan.json
          echo "exit_code=${exit_code:-0}" >> $GITHUB_OUTPUT

      - name: Classify and route
        if: steps.plan.outputs.exit_code == '2'   # changes detected
        run: |
          python3 .ci/classify-drift.py plan.json > tier1.json
          if [ -s tier1.json ]; then
            curl -X POST -H 'Content-Type: application/json' \
              --data @tier1.json \
              ${{ secrets.PAGERDUTY_WEBHOOK }}
          fi
```

The `classify-drift.py` script is a 30-line filter over the JSON plan that emits Tier 1 entries only:

```python
import json, sys, re

TIER1_RESOURCES = (
    r'^aws_iam_',
    r'^aws_security_group',
    r'^aws_kms_key',
    r'^aws_cloudtrail',
    r'^aws_s3_bucket_policy$',
    r'^aws_s3_bucket_public_access_block$',
    r'^aws_backup_(plan|vault)',
)

plan = json.load(open(sys.argv[1]))
tier1 = []

for change in plan.get('resource_changes', []):
    if change['change']['actions'] == ['no-op']:
        continue
    if any(re.match(p, change['type']) for p in TIER1_RESOURCES):
        tier1.append({
            'resource': change['address'],
            'type': change['type'],
            'actions': change['change']['actions'],
        })

if tier1:
    json.dump({'alert': 'Tier-1 drift', 'changes': tier1}, sys.stdout)
```

## Operating Notes

Run the digest review on Mondays. Anything in Tier 2 that's been there three consecutive weeks becomes a Tier 1 candidate, or gets accepted and moved to Tier 3 with a comment explaining why. The tiers aren't static. They're the team's working classification of what matters, and they should evolve as you learn what your environment actually does.

The number to watch isn't drift incidents. It's how often Tier 1 fires. More than once a week on average and your change-control process has a gap. Less than once a quarter and the classifier is too lenient. Someone is changing security-relevant settings without it surfacing. Drift detection is one guardrail in the broader [platform engineering foundation](/library/platform-engineering-for-regulated-startups) a regulated startup runs; treat this classifier as that foundation's early-warning system.

## Frequently asked questions

**Q: How do you stop drift detection from becoming alert noise?**

Classify drift by risk at detection time instead of treating every difference as an alert. Route dangerous changes (auth, audit, availability) to PagerDuty, send capacity and cost drift to a weekly digest, and suppress known-good drift entirely. The detection tooling is mostly solved; the classifier is the actual work.

**Q: What counts as Tier 1 drift that should page the on-call?**

Changes to IAM policies and roles, security group rules (especially anything opening 0.0.0.0/0), KMS key policies, CloudTrail or VPC Flow Log config, S3 bucket policies and public-access blocks, backup retention, and production load-balancer listeners. Drift in any of these means an attacker is moving laterally or someone bypassed change control.

**Q: How often should you run infrastructure drift detection?**

Run the narrow Tier 1 check hourly because it's cheap, run a broader Tier 2 plan daily, and run the full estate weekly. Route Tier 1 to PagerDuty, Tier 2 to a Slack digest reviewed on Mondays, and Tier 3 to /dev/null.

**Q: Do you need machine learning to classify infrastructure drift?**

No. The classifier is regex over the terraform plan JSON — about 30 lines of Python that map resource types like aws_iam_* and aws_security_group to Tier 1 and send everything else to the digest. No machine learning, no fancy heuristics.

**Q: How do you know if your drift classifier is tuned correctly?**

Watch how often Tier 1 fires. More than once a week on average means your change-control process has a gap; less than once a quarter means the classifier is too lenient and security-relevant changes are slipping through unsurfaced.

**Q: What tools can detect infrastructure drift?**

terraform plan on a schedule in CI is the baseline. At scale, driftctl or Pulumi's pulumi refresh --diff produce the same signal with less compute. The template here uses a GitHub Actions workflow with an OIDC-federated role and terraform plan -detailed-exitcode.

Template: "Terraform drift-check GitHub Actions workflow" — branded PDF at https://kaansystems.com/templates/infrastructure-drift-detection.pdf

Source: https://kaansystems.com/library/infrastructure-drift-detection · Work with Kaan Systems: https://kaansystems.com/contact