Detecting Infrastructure Drift Without Drowning in Alerts
Drift detection at scale produces thousands of 'differences,' most of them noise. A tiered classifier routes the dangerous drift to PagerDuty and sends everything else to a weekly digest.
Terraform drift-check GitHub Actions workflow
Copy as markdown to paste into your repo, or download a branded PDF for sharing with non-technical stakeholders.
Frequently asked questions
How do you stop drift detection from becoming alert noise?
Classify drift by risk at detection time instead of treating every difference as an alert. Route dangerous changes (auth, audit, availability) to PagerDuty, send capacity and cost drift to a weekly digest, and suppress known-good drift entirely. The detection tooling is mostly solved; the classifier is the actual work.
What counts as Tier 1 drift that should page the on-call?
Changes to IAM policies and roles, security group rules (especially anything opening 0.0.0.0/0), KMS key policies, CloudTrail or VPC Flow Log config, S3 bucket policies and public-access blocks, backup retention, and production load-balancer listeners. Drift in any of these means an attacker is moving laterally or someone bypassed change control.
How often should you run infrastructure drift detection?
Run the narrow Tier 1 check hourly because it's cheap, run a broader Tier 2 plan daily, and run the full estate weekly. Route Tier 1 to PagerDuty, Tier 2 to a Slack digest reviewed on Mondays, and Tier 3 to /dev/null.
Do you need machine learning to classify infrastructure drift?
No. The classifier is regex over the terraform plan JSON — about 30 lines of Python that map resource types like aws_iam_* and aws_security_group to Tier 1 and send everything else to the digest. No machine learning, no fancy heuristics.
How do you know if your drift classifier is tuned correctly?
Watch how often Tier 1 fires. More than once a week on average means your change-control process has a gap; less than once a quarter means the classifier is too lenient and security-relevant changes are slipping through unsurfaced.
What tools can detect infrastructure drift?
terraform plan on a schedule in CI is the baseline. At scale, driftctl or Pulumi's pulumi refresh --diff produce the same signal with less compute. The template here uses a GitHub Actions workflow with an OIDC-federated role and terraform plan -detailed-exitcode.
Get the next one.
New operating guides as we publish them — practical, infrequent, unsubscribe in one click.
A drift scan is part of the 7-day Assessment.
Kaan Systems does platform engineering and Terraform for funded SaaS, health-tech, and fintech. Compliance evidence is the by-product of that work. The Assessment includes a drift scan: which changes are dangerous, and which are noise. The 7-day Assessment is read-only. Book a call.