# Evidence Pipelines 101: How to Stop Reconstructing Audit Artifacts Quarterly

Canonical: https://kaansystems.com/library/evidence-pipelines-101
Topic: Compliance · Published: 2026-05-12 · Updated: 2026-10-12
Summary: Most teams reconstruct audit evidence from scratch every quarter. That's weeks of senior engineering time spent producing artifacts a running system could generate itself. The pipeline below does it automatically.

## Key takeaways

- Audits ask you to prove a control was operating across the entire period under review, not just designed once — and that operational evidence is exactly what teams reconstruct from scratch every cycle.
- Rebuilding evidence by hand runs about three engineers for two weeks each cycle, and it recurs because the underlying data rotates out of console retention before the next audit.
- Five categories of events cover roughly 80% of what any audit or customer review asks for: identity, change, configuration snapshots, data-access, and secrets and credential lifecycle.
- The architecture is identical for all five categories: source, a collector, durable storage in S3 Object Lock, an optional hash or signature, and a query layer auditors use directly.
- Keep raw evidence in S3 Object Lock for the full retention period (often seven years for HIPAA-aligned work), hold 18 months in a hot queryable layer, and rehydrate older data on demand.
- Once the pipeline runs, auditor questions become queries and the same store doubles as your incident-investigation timeline instead of a screenshot reconstruction.

**What is an evidence pipeline?** It's the system that generates audit artifacts continuously from your running infrastructure, so you stop rebuilding them under deadline every quarter. Collect five categories of events into durable, tamper-evident storage behind a query layer, and the next audit becomes one engineer for two days, not three for two weeks.

## Why do teams rebuild audit evidence every quarter?

Audits, customer security reviews, and regulatory inquiries all ask the same question in slightly different words: **show me that this control was operating during the period under review.**

The naive first-cycle answer is "let me put together a spreadsheet." Three engineers, two weeks, screenshot evidence pulled from a dozen consoles, manually attested to. The auditor accepts it because the alternative is missing the audit window. Their report dings the company for "compensating controls" or "manual evidence collection."

Six months later, the next cycle starts. The spreadsheet has to be rebuilt because the underlying data rotated out of console retention. Three engineers, two more weeks.

That's work automation should be doing.

[SOC 2 readiness](/library/soc2-for-startups-complete-guide) is where this bites first: the audit doesn't want to see the control once, it wants evidence it operated across the entire period, and that evidence is exactly what teams end up reconstructing every cycle.

## How do Terraform and CI produce that evidence instead?

Treat evidence as a continuously generated output of Terraform, CI, and CloudTrail, not a quarterly project. Those three streams already answer the change and configuration questions. Five categories of events still cover roughly 80% of what any audit or customer review will ask for. The same pattern is [drift detection](/library/infrastructure-drift-detection) for the plan diff, [GitOps](/library/gitops-progressive-delivery) for the deploy record, and the [platform engineering guide](/library/platform-engineering-for-regulated-startups), which is the Terraform hub. The [7-day Assessment](/assessment) includes a one-page evidence map of what those systems already prove.

### 1. Identity events

Every authentication, every authorization decision, every privileged action.

Sources: Cognito / Okta / Auth0 logs for human auth; CloudTrail (or equivalent) for API-level IAM events; Kubernetes audit log for cluster-level RBAC decisions.

What you'll be asked to prove: "Show me the privileged actions performed by terminated employees in the 30 days before their termination." Without an identity-events pipeline, this question takes a week to answer. With one, it's a query.

### 2. Change events

Every change to configuration, infrastructure, or production code.

Sources: GitHub / GitLab events (PRs merged), Terraform Cloud / Atlantis logs (infrastructure applies), AWS Config (resource configuration changes), CI/CD pipeline events.

What you'll be asked to prove: "Show me every production change in the period, who authored it, and who approved it." If your change-management process is mature, this is one query against three tables.

### 3. Configuration snapshots

Point-in-time snapshots of the security-relevant configuration of every resource.

Sources: AWS Config for AWS resources; Conftest / OPA snapshots for Kubernetes; vendor APIs for SaaS configurations.

What you'll be asked to prove: "Show me that on December 31, every S3 bucket holding PHI had encryption enabled and public access blocked." Config snapshots are exactly the right shape for this.

### 4. Data-access events

Every read of sensitive data.

Sources: S3 server access logs (or CloudTrail data events for the buckets that matter), RDS audit logs, application-layer audit logging.

What you'll be asked to prove: "Show me everyone who accessed the patient_records table in the period, what they queried, and what the business purpose was." Application-layer audit logging is what makes the third part of that question answerable; without it, you can prove access but not purpose.

### 5. Secrets and credential lifecycle events

Creation, rotation, access, and revocation of every credential.

Sources: AWS Secrets Manager / HashiCorp Vault audit log; certificate authority audit log for mTLS; SSH key rotation logs if applicable.

What you'll be asked to prove: "Show me that every production credential in this scope has been rotated within the last 90 days, and that no terminated employee retained access." A secrets pipeline answers this in one query.

## Implementation

The architectural shape is the same for all five categories:

```mermaid
flowchart LR
  subgraph SRC[Sources]
    direction TB
    S1[CloudTrail / Okta<br/>identity events]
    S2[GitHub / Terraform<br/>change events]
    S3[AWS Config<br/>config snapshots]
    S4[RDS audit / app audit<br/>data-access events]
    S5[Secrets Manager<br/>credential lifecycle]
  end

  COL[Collector<br/>EventBridge → SQS → Lambda<br/>or Kinesis Firehose]

  subgraph STORE[Durable storage]
    direction TB
    RAW[(S3 raw<br/>Object Lock · 7yr)]
    PARQ[(Parquet partitioned<br/>hot · 18 mo)]
    HASH[Daily SHA-256 manifest<br/>tamper-evident]
  end

  Q[Query layer<br/>Athena · BigQuery · Snowflake]
  AUDITOR((Auditor<br/>or incident<br/>investigator))

  S1 & S2 & S3 & S4 & S5 --> COL
  COL --> RAW
  COL --> PARQ
  RAW --> HASH
  PARQ --> Q
  RAW -.->|rehydrate on demand| Q
  Q --> AUDITOR

  classDef src   fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
  classDef pipe  fill:#1F4A3E,stroke:#3FA585,stroke-width:2px,color:#E8F5EE;
  classDef store fill:#0a1612,stroke:#2E8B6F,stroke-width:1.5px,color:#E8F5EE;
  classDef out   fill:#3FA585,stroke:#3FA585,stroke-width:2px,color:#0a1612;
  classDef band  fill:#2E8B6F0A,stroke:#2E8B6F,stroke-width:1px;

  class S1,S2,S3,S4,S5 src;
  class COL,HASH pipe;
  class RAW,PARQ store;
  class Q,AUDITOR out;
  class SRC,STORE band;
```

1. **Source.** Wherever the events are natively emitted.
2. **Collector.** A pipeline that pulls or receives the events and writes them to durable storage. For AWS, typically CloudTrail → Kinesis Firehose → S3, or EventBridge → SQS → Lambda → S3.
3. **Storage.** S3 with Object Lock (compliance mode) for raw evidence, plus a partitioned Parquet representation for query speed.
4. **Hash chain or signature.** Optional but worth doing: each batch is signed or hashed-and-chained so tampering is detectable. The simple version is daily SHA-256 manifests checked into a separate tamper-evident store.
5. **Query layer.** Athena, BigQuery, Snowflake, or whatever your team already uses. Auditors get a query interface, not a console screenshot.

The lifecycle policy is important. Raw evidence stays in S3 Object Lock for the full retention period (often seven years for HIPAA-aligned work). The hot, queryable layer holds 18 months. Anything older is rehydrated from cold storage on demand.

## The Template

The map below ties controls to evidence sources. Run through it for your environment; anywhere you can't fill in a source, query, or retention value is a gap to close before your next audit.

| Control area | Evidence type | Source | Sample query | Retention |
|---|---|---|---|---|
| Identity — privileged access | Auth + assumption events | CloudTrail `AssumeRole`, Identity Center login | `WHERE eventName='AssumeRole' AND user_arn LIKE '%admin%'` | 7 years |
| Identity — terminations | Account deactivation events | Identity Center, HR system join | `WHERE event='UserDeactivated' AND timestamp > <termination_date>` | 7 years |
| Change management — production | Merged PR + approver list | GitHub Audit Log | `WHERE action='pull_request.merged' AND repo IN <prod_repos>` | 7 years |
| Change management — infrastructure | Terraform apply events | TF Cloud / Atlantis | `WHERE event='terraform.apply' AND workspace LIKE 'prod-%'` | 7 years |
| Configuration — S3 encryption | Resource config snapshots | AWS Config | `SELECT resourceId WHERE configuration.encryption.status='Disabled'` | 7 years |
| Configuration — SG rules | Resource config snapshots | AWS Config | `SELECT resourceId WHERE configuration.ipPermissions[*].cidrIp='0.0.0.0/0'` | 7 years |
| Data access — PHI tables | DB audit log | RDS audit, application audit | `WHERE table_accessed IN <phi_tables> GROUP BY user` | 7 years |
| Data access — S3 PHI buckets | Object-level events | CloudTrail data events | `WHERE eventSource='s3.amazonaws.com' AND requestParameters.bucketName=<phi_bucket>` | 7 years |
| Secrets — rotation | Secret rotation events | Secrets Manager | `WHERE event='SecretRotated' AND age_days > 90` | 7 years |
| Secrets — credential revocation | Credential deletion events | IAM access keys, app secrets | `WHERE event='AccessKeyDeleted'` | 7 years |

Fill in the queries with your environment's specifics. Fill in the retention column based on your regulatory mix. The format is what matters: control mapped to source mapped to query mapped to retention.

## What This Buys You

The first audit after this is in place feels different. Instead of three engineers and two weeks, it's one engineer and two days. The auditor stops asking "can you produce this?" and asks "show me the query." You demo. They check the retention. Move on.

The second audit feels like maintenance. The third audit, the auditor stops asking because the answers have all been pre-generated.

The same pipeline doubles as your incident-investigation substrate. The next security incident, your team doesn't reconstruct the timeline from console screenshots. They query it.

## Frequently asked questions

**Q: Why do teams rebuild audit evidence from scratch every quarter?**

The first-cycle answer is usually a manually assembled spreadsheet of screenshots pulled from a dozen consoles. Six months later the underlying data has rotated out of console retention, so the spreadsheet has to be rebuilt — three engineers, two more weeks — every cycle.

**Q: What is an evidence pipeline?**

It treats audit evidence as a continuously-generated output of your running systems rather than a quarterly project, collecting security-relevant events into durable, tamper-evident storage behind a query layer. Auditors get a query interface instead of a console screenshot.

**Q: What kinds of evidence do audits and security reviews actually ask for?**

Five categories cover roughly 80% of the requests: identity events, change events, configuration snapshots, data-access events, and secrets and credential lifecycle events. Each maps to specific log sources like CloudTrail, GitHub, AWS Config, RDS audit logs, and Secrets Manager.

**Q: How long do you have to retain audit evidence?**

Keep raw evidence in S3 Object Lock for the full retention period, often seven years for HIPAA-aligned work. A hot, queryable layer holds 18 months, and anything older is rehydrated from cold storage on demand.

**Q: How do you make the evidence tamper-evident?**

Store raw evidence in S3 Object Lock in compliance mode, and sign or hash-and-chain each batch so tampering is detectable. The simple version is a daily SHA-256 manifest checked into a separate tamper-evident store.

**Q: How much does a working pipeline actually save at audit time?**

The first audit after it's in place drops from roughly three engineers and two weeks to one engineer and two days. By the third cycle the auditor stops asking because the answers are pre-generated, and the same pipeline becomes your incident-investigation substrate.

Template: "Evidence map (control → log source → query → retention)" — branded PDF at https://kaansystems.com/templates/evidence-pipelines-101.pdf

Source: https://kaansystems.com/library/evidence-pipelines-101 · Work with Kaan Systems: https://kaansystems.com/contact