# Securing PHI in Observability Data

Canonical: https://kaansystems.com/library/securing-phi-observability
Topic: Compliance · Published: 2026-05-12 · Updated: 2026-10-12
Summary: Modern observability pipelines collect everything. That's the value, and it's also the problem. PHI leaks into logs, traces, and metrics through routes most teams don't audit. Defense in depth, layer by layer.

## Key takeaways

- PHI leaks into observability data through routes most teams never audit: URL parameters, verbose application logs, database query strings, error stack traces, and distributed trace spans.
- Any PHI sitting in your monitoring stack is almost certainly outside the HIPAA scope you architected for production data stores, which makes it a breach.
- Defense in depth means four layers, each cheaper to fix than the next one down: don't generate the PHI, scrub it at the ingestion pipeline, lock down storage, and treat tracing as its own special case.
- Scrub aggressively at the pipeline layer. False positives are nearly free in observability data because the real value still lives in your application database, where it belongs.
- Lock down the observability store like a production database: encrypt at rest, gate access with SSO plus MFA plus a scoped role, turn on the query audit log, and keep hot retention to about 30 days.
- Skipping tracing entirely for PHI-touching services is a reasonable trade against breach risk, and pattern-based scrubbing will never catch a structured leak like an extra={'patient_id': ...} field — only code review will.

**How do you keep PHI out of your observability data?** Defense in depth across four layers: don't generate the PHI at the application, scrub what leaks through at the ingestion pipeline, lock down the storage that still catches some, and treat tracing as the special case it is. Each layer is cheaper to fix than the one below it.

The scrub belongs on the same path as an [evidence pipeline](/library/evidence-pipelines-101). The account split that keeps logs away from the workload is [minimum-viable HIPAA on AWS](/library/minimum-viable-hipaa-aws). If prompts are a second PHI store, [LLM feature auditability](/library/llm-feature-auditability) is the trace shape that does not keep the content.

## How does PHI end up in logs and traces?

Observability stacks exist to capture as much as possible about what your systems are doing. Datadog, New Relic, Honeycomb, an in-house ELK setup, doesn't matter. That's their value. It also makes them PHI minefields in any healthcare-adjacent environment.

The leaks happen through routes most teams don't audit:

- **Request URL parameters.** `/api/patients?ssn=...` writes the SSN into every access log and trace span.
- **Verbose application logs.** "Loaded user 12345 (John Smith, DOB 1980-04-12) from RDS." Written during an incident two years ago, never removed, indexed in Elasticsearch ever since.
- **Database query strings.** Slow-query logs that include the query text capture every WHERE clause that touched PHI columns.
- **Error stack traces.** An unhandled exception in a function processing patient data often captures the entire request body in the error message.
- **Distributed traces.** A trace span across services naturally includes the entity IDs and frequently the entity contents at each hop.

Any of these put protected health information into a system that's almost certainly outside the HIPAA scope you've architected for production data stores. That's a breach.

## How do you keep observability out of a breach?

Defense in depth. Four layers, each cheaper to fix than the next one down:

```mermaid
flowchart TB
  subgraph APP[Layer 1 · Application — don't generate it]
    direction LR
    A1[no_log_body decorator]
    A2[Opaque IDs in URLs]
    A3[before_send hooks<br/>strip PHI from errors]
    A4[DB query logger<br/>no parameter values]
  end

  subgraph PIPE[Layer 2 · Pipeline — scrub at ingestion]
    direction LR
    B1[SSN · MRN · DOB regex]
    B2[Email · phone · CC regex]
    B3[Bearer · API-key regex]
  end

  subgraph STORE[Layer 3 · Storage — contain the blast radius]
    direction LR
    C1[Encrypt at rest · CMKs]
    C2[SSO + MFA + scoped role]
    C3[Query audit log on]
    C4[Hot retention ≤ 30d]
  end

  subgraph TRACE[Layer 4 · Tracing — treat specially]
    direction LR
    D1[Sampling exclusion]
    D2[Span attribute filter]
    D3[Service-level data class tag]
  end

  APP --> PIPE --> STORE --> TRACE
  TRACE --> SAFE((PHI<br/>contained))

  classDef l1   fill:#3FA585,stroke:#3FA585,stroke-width:2px,color:#0a1612;
  classDef l2   fill:#2E8B6F,stroke:#3FA585,stroke-width:2px,color:#0a1612;
  classDef l3   fill:#1F4A3E,stroke:#3FA585,stroke-width:2px,color:#E8F5EE;
  classDef l4   fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
  classDef safe fill:#0a1612,stroke:#3FA585,stroke-width:2.5px,color:#E8F5EE;
  classDef band fill:#2E8B6F0A,stroke:#2E8B6F,stroke-width:1px;

  class A1,A2,A3,A4 l1;
  class B1,B2,B3 l2;
  class C1,C2,C3,C4 l3;
  class D1,D2,D3 l4;
  class SAFE safe;
  class APP,PIPE,STORE,TRACE band;
```

### Layer 1: Don't generate it

The cheapest log to scrub is the one you never wrote. Engineering hygiene at the source:

- **Never log full request bodies for PHI-bearing endpoints.** Log the request shape (method, path, status, latency) but not the payload. Endpoints that touch PHI should have a `@no_log_body` decorator or equivalent enforced.
- **Use opaque identifiers in URL paths, not PHI.** `/api/patients/PAT-7891234` is fine. `/api/patients/?ssn=...` is not. Most teams need an internal-ID layer for this.
- **Sanitize error contexts.** Wrap every PHI-touching function in an exception handler that strips PHI from the stack trace before it propagates. Most observability SDKs support `before_send` hooks for exactly this.
- **Set the database query logger to not include parameter values.** Most ORMs default to including parameters in slow-query logs. Override that.

### Layer 2: Scrub at the pipeline

For everything you couldn't prevent at the source, the next defense is at the ingestion layer.

Every major observability vendor supports a "pre-send" or "scrubber" filter that runs against payloads before they leave the agent. Datadog calls it a Sensitive Data Scanner; Splunk has CIM-based redaction; Elastic has ingest pipeline processors; New Relic has attribute filters.

The patterns to scrub are well-known. The template below covers the common ones (SSN, MRN, DOB, email, phone, credit card). Configure them once at the agent level so they apply across every service.

The common mistake here is scrubbing too narrowly. Scrub *aggressively*. False positives are nearly free in observability data. A log line with "[REDACTED]" where a value used to be is fine for debugging context; the actual value is in your application database, where it belongs.

### Layer 3: Lock down storage

Even with scrubbing, some PHI inevitably leaks. The storage layer assumes that and contains the blast radius:

- **Encrypt the observability data store at rest.** With customer-managed keys if you're large enough to care about key access audit.
- **Restrict access via SSO + MFA + an explicit access role.** Engineers reach for observability tools more often than production databases; the access bar should be similar.
- **Audit who queries what.** Most observability vendors offer an audit log of queries run; turn it on. The next incident that involves "someone accessed PHI in logs," that audit log is your evidence.
- **Set short retention on raw logs.** Hot retention (queryable, fast) of 30 days for application logs is enough for ~95% of debug cases. Anything older goes to cold archive with stricter access.

### Layer 4: Treat tracing carefully

Traces are the hardest case. A trace span captures the context flowing across services, and in a microservices environment touching PHI, the "context" naturally includes patient identifiers, sometimes more.

Three patterns help:

- **Sampling exclusion.** Configure the tracer to not sample traces from PHI-touching services for the storage layer. They still emit metrics; they just don't generate trace data that lives in the trace store.
- **Span attribute filtering.** Every major tracer supports attribute filtering before send. Strip any span attribute matching a PHI pattern.
- **Service-level data classification.** Tag services as PHI-touching at deploy time. The agent reads the tag and applies stricter filters to traces from those services.

A reasonable choice is to skip tracing entirely for PHI-touching services. The observability cost is real, but it's an acceptable trade-off against the breach risk.

## The Template

A scrubber config with regex patterns for the most common PHI types. Apply at the observability-agent layer (format shown for Datadog's Sensitive Data Scanner; patterns are vendor-independent).

| Field | Pattern | Replacement |
|---|---|---|
| SSN | `\b\d{3}-\d{2}-\d{4}\b` | `[REDACTED-SSN]` |
| SSN (no dashes) | `\b\d{9}\b` (context-dependent) | `[REDACTED-SSN]` |
| MRN (common formats) | `\b(MRN[-:\s]*\d{6,10}|\d{8})\b` | `[REDACTED-MRN]` |
| DOB | `\b(0[1-9]|1[012])[-/](0[1-9]|[12]\d|3[01])[-/](19|20)\d\d\b` | `[REDACTED-DOB]` |
| Email | `\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z\|a-z]{2,}\b` | `[REDACTED-EMAIL]` |
| Phone | `\b(\+\d{1,2}\s?)?\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4}\b` | `[REDACTED-PHONE]` |
| Credit card | `\b(?:\d{4}[-\s]?){3}\d{4}\b` | `[REDACTED-CC]` |
| Auth bearer token | `Bearer\s+[A-Za-z0-9._~+/=-]+` | `Bearer [REDACTED]` |
| API key (generic) | `(sk|api)[-_][A-Za-z0-9]{20,}` | `[REDACTED-KEY]` |

For Datadog specifically, this lives at `Logs → Configuration → Pipelines → Sensitive Data Scanner`. The same patterns apply (sometimes with vendor-specific escape rules) to:

- **Elastic ingest pipelines** (`gsub` processors)
- **New Relic log obfuscation rules**
- **Splunk SEDCMD transforms**
- **Cloudflare Logpush filters**

Test the patterns against your actual logs before turning them on in production. Start in "tag-only" mode where matches are flagged but not modified, then flip to "replace" once you've verified no false-positive damage to debugging.

## Operating Notes

Add an audit step to your engineering checklist for new PHI-touching code: before merge, search the diff for likely log-leak candidates. A regex for `logger.info` or `console.log` followed by a variable reference catches most of them.

You will have leaks. Recovery is to immediately rotate the scrubber pattern to cover the new shape, then run the storage-layer access audit for the period the leak was live. If no one accessed the affected logs, you have a finding to document. If someone did, you have a more serious finding to escalate.

The hardest leak to catch is the structured one: a developer adds an `extra={'patient_id': ...}` field to a log call intending it to be opaque, but the field naming convention propagates patient identifiers into the index. Pattern-based scrubbing won't catch that. Only code review will.

## Frequently asked questions

**Q: How does PHI end up in logs and traces?**

It leaks in through routes most teams don't audit: SSNs and other identifiers in request URL parameters, verbose application logs that print names and dates of birth, database slow-query strings, request bodies captured in error stack traces, and entity data carried across distributed trace spans.

**Q: Is PHI in my monitoring stack actually a HIPAA breach?**

Yes. Your observability system is almost certainly outside the HIPAA scope you architected for production data stores, so any protected health information that lands there is PHI in an unauthorized system — a breach.

**Q: How do I keep PHI out of my observability data?**

Use defense in depth across four layers: don't generate the PHI at the application (no full request bodies, opaque IDs in URLs, sanitized error contexts), scrub what leaks at the ingestion pipeline, lock down the storage layer, and handle tracing specially. The cheapest log to scrub is the one you never wrote.

**Q: Should I scrub PHI patterns narrowly or aggressively?**

Aggressively. False positives are nearly free in observability data — a log line showing [REDACTED] where a value used to be is fine for debugging, because the actual value still lives in your application database. Scrubbing too narrowly is the common mistake.

**Q: How long should I keep observability logs that might contain PHI?**

Keep hot, queryable retention short — around 30 days for application logs covers roughly 95% of debug cases. Anything older should move to a cold archive with stricter access.

**Q: Should I trace services that handle PHI?**

Tracing is the hardest case because span context naturally carries patient identifiers. Options are sampling exclusion, span attribute filtering, and service-level data classification — but a reasonable choice is to skip tracing entirely for PHI-touching services, accepting the observability cost against the breach risk.

Template: "Log-scrubber regex config" — branded PDF at https://kaansystems.com/templates/securing-phi-observability.pdf

Source: https://kaansystems.com/library/securing-phi-observability · Work with Kaan Systems: https://kaansystems.com/contact