The Problem
Audits, customer security reviews, and regulatory inquiries all ask the same question in slightly different words: show me that this control was operating during the period under review.
The naive first-cycle answer is “let me put together a spreadsheet.” Three engineers, two weeks, screenshot evidence pulled from a dozen consoles, manually attested to. The auditor accepts it because the alternative is missing the audit window. Their report dings the company for “compensating controls” or “manual evidence collection.”
Six months later, the next cycle starts. The spreadsheet has to be rebuilt because the underlying data rotated out of console retention. Three engineers, two more weeks.
That’s work automation should be doing.
The Approach
Treat evidence as a continuously-generated output of the system, not a quarterly project. Five categories of evidence cover roughly 80% of what any audit or customer review will ask for:
1. Identity events
Every authentication, every authorization decision, every privileged action.
Sources: Cognito / Okta / Auth0 logs for human auth; CloudTrail (or equivalent) for API-level IAM events; Kubernetes audit log for cluster-level RBAC decisions.
What you’ll be asked to prove: “Show me the privileged actions performed by terminated employees in the 30 days before their termination.” Without an identity-events pipeline, this question takes a week to answer. With one, it’s a query.
2. Change events
Every change to configuration, infrastructure, or production code.
Sources: GitHub / GitLab events (PRs merged), Terraform Cloud / Atlantis logs (infrastructure applies), AWS Config (resource configuration changes), CI/CD pipeline events.
What you’ll be asked to prove: “Show me every production change in the period, who authored it, and who approved it.” If your change-management process is mature, this is one query against three tables.
3. Configuration snapshots
Point-in-time snapshots of the security-relevant configuration of every resource.
Sources: AWS Config for AWS resources; Conftest / OPA snapshots for Kubernetes; vendor APIs for SaaS configurations.
What you’ll be asked to prove: “Show me that on December 31, every S3 bucket holding PHI had encryption enabled and public access blocked.” Config snapshots are exactly the right shape for this.
4. Data-access events
Every read of sensitive data.
Sources: S3 server access logs (or CloudTrail data events for the buckets that matter), RDS audit logs, application-layer audit logging.
What you’ll be asked to prove: “Show me everyone who accessed the patient_records table in the period, what they queried, and what the business purpose was.” Application-layer audit logging is what makes the third part of that question answerable; without it, you can prove access but not purpose.
5. Secrets and credential lifecycle events
Creation, rotation, access, and revocation of every credential.
Sources: AWS Secrets Manager / HashiCorp Vault audit log; certificate authority audit log for mTLS; SSH key rotation logs if applicable.
What you’ll be asked to prove: “Show me that every production credential in this scope has been rotated within the last 90 days, and that no terminated employee retained access.” A secrets pipeline answers this in one query.
Implementation
The architectural shape is the same for all five categories:
- Source. Wherever the events are natively emitted.
- Collector. A pipeline that pulls or receives the events and writes them to durable storage. For AWS, typically CloudTrail → Kinesis Firehose → S3, or EventBridge → SQS → Lambda → S3.
- Storage. S3 with Object Lock (compliance mode) for raw evidence, plus a partitioned Parquet representation for query speed.
- Hash chain or signature. Optional but worth doing: each batch is signed or hashed-and-chained so tampering is detectable. The simple version is daily SHA-256 manifests checked into a separate tamper-evident store.
- Query layer. Athena, BigQuery, Snowflake, or whatever your team already uses. Auditors get a query interface, not a console screenshot.
The lifecycle policy is important. Raw evidence stays in S3 Object Lock for the full retention period (often seven years for HIPAA-aligned work). The hot, queryable layer holds 18 months. Anything older is rehydrated from cold storage on demand.
The Template
The map below ties controls to evidence sources. Run through it for your environment; anywhere you can’t fill in a source, query, or retention value is a gap to close before your next audit.
| Control area | Evidence type | Source | Sample query | Retention |
|---|---|---|---|---|
| Identity — privileged access | Auth + assumption events | CloudTrail AssumeRole, Identity Center login | WHERE eventName='AssumeRole' AND user_arn LIKE '%admin%' | 7 years |
| Identity — terminations | Account deactivation events | Identity Center, HR system join | WHERE event='UserDeactivated' AND timestamp > <termination_date> | 7 years |
| Change management — production | Merged PR + approver list | GitHub Audit Log | WHERE action='pull_request.merged' AND repo IN <prod_repos> | 7 years |
| Change management — infrastructure | Terraform apply events | TF Cloud / Atlantis | WHERE event='terraform.apply' AND workspace LIKE 'prod-%' | 7 years |
| Configuration — S3 encryption | Resource config snapshots | AWS Config | SELECT resourceId WHERE configuration.encryption.status='Disabled' | 7 years |
| Configuration — SG rules | Resource config snapshots | AWS Config | SELECT resourceId WHERE configuration.ipPermissions[*].cidrIp='0.0.0.0/0' | 7 years |
| Data access — PHI tables | DB audit log | RDS audit, application audit | WHERE table_accessed IN <phi_tables> GROUP BY user | 7 years |
| Data access — S3 PHI buckets | Object-level events | CloudTrail data events | WHERE eventSource='s3.amazonaws.com' AND requestParameters.bucketName=<phi_bucket> | 7 years |
| Secrets — rotation | Secret rotation events | Secrets Manager | WHERE event='SecretRotated' AND age_days > 90 | 7 years |
| Secrets — credential revocation | Credential deletion events | IAM access keys, app secrets | WHERE event='AccessKeyDeleted' | 7 years |
Fill in the queries with your environment’s specifics. Fill in the retention column based on your regulatory mix. The format is what matters: control mapped to source mapped to query mapped to retention.
What This Buys You
The first audit after this is in place feels different. Instead of three engineers and two weeks, it’s one engineer and two days. The auditor stops asking “can you produce this?” and asks “show me the query.” You demo. They check the retention. Move on.
The second audit feels like maintenance. The third audit, the auditor stops asking because the answers have all been pre-generated.
The same pipeline doubles as your incident-investigation substrate. The next security incident, your team doesn’t reconstruct the timeline from console screenshots. They query it.