Securing PHI in Observability Data

The Problem

Observability stacks exist to capture as much as possible about what your systems are doing. Datadog, New Relic, Honeycomb, an in-house ELK setup, doesn’t matter. That’s their value. It also makes them PHI minefields in any healthcare-adjacent environment.

The leaks happen through routes most teams don’t audit:

• Request URL parameters. /api/patients?ssn=... writes the SSN into every access log and trace span.
• Verbose application logs. “Loaded user 12345 (John Smith, DOB 1980-04-12) from RDS.” Written during an incident two years ago, never removed, indexed in Elasticsearch ever since.
• Database query strings. Slow-query logs that include the query text capture every WHERE clause that touched PHI columns.
• Error stack traces. An unhandled exception in a function processing patient data often captures the entire request body in the error message.
• Distributed traces. A trace span across services naturally includes the entity IDs and frequently the entity contents at each hop.

Any of these put protected health information into a system that’s almost certainly outside the HIPAA scope you’ve architected for production data stores. That’s a breach.

The Approach

Defense in depth. Four layers, each cheaper to fix than the next one down:

Layer 1: Don’t generate it

The cheapest log to scrub is the one you never wrote. Engineering hygiene at the source:

• Never log full request bodies for PHI-bearing endpoints. Log the request shape (method, path, status, latency) but not the payload. Endpoints that touch PHI should have a @no_log_body decorator or equivalent enforced.
• Use opaque identifiers in URL paths, not PHI. /api/patients/PAT-7891234 is fine. /api/patients/?ssn=... is not. Most teams need an internal-ID layer for this.
• Sanitize error contexts. Wrap every PHI-touching function in an exception handler that strips PHI from the stack trace before it propagates. Most observability SDKs support before_send hooks for exactly this.
• Set the database query logger to not include parameter values. Most ORMs default to including parameters in slow-query logs. Override that.

Layer 2: Scrub at the pipeline

For everything you couldn’t prevent at the source, the next defense is at the ingestion layer.

Every major observability vendor supports a “pre-send” or “scrubber” filter that runs against payloads before they leave the agent. Datadog calls it a Sensitive Data Scanner; Splunk has CIM-based redaction; Elastic has ingest pipeline processors; New Relic has attribute filters.

The patterns to scrub are well-known. The template below covers the common ones (SSN, MRN, DOB, email, phone, credit card). Configure them once at the agent level so they apply across every service.

The common mistake here is scrubbing too narrowly. Scrub aggressively. False positives are nearly free in observability data. A log line with “[REDACTED]” where a value used to be is fine for debugging context; the actual value is in your application database, where it belongs.

Layer 3: Lock down storage

Even with scrubbing, some PHI inevitably leaks. The storage layer assumes that and contains the blast radius:

• Encrypt the observability data store at rest. With customer-managed keys if you’re large enough to care about key access audit.
• Restrict access via SSO + MFA + an explicit access role. Engineers reach for observability tools more often than production databases; the access bar should be similar.
• Audit who queries what. Most observability vendors offer an audit log of queries run; turn it on. The next incident that involves “someone accessed PHI in logs,” that audit log is your evidence.
• Set short retention on raw logs. Hot retention (queryable, fast) of 30 days for application logs is enough for ~95% of debug cases. Anything older goes to cold archive with stricter access.

Layer 4: Treat tracing carefully

Traces are the hardest case. A trace span captures the context flowing across services, and in a microservices environment touching PHI, the “context” naturally includes patient identifiers, sometimes more.

Three patterns help:

• Sampling exclusion. Configure the tracer to not sample traces from PHI-touching services for the storage layer. They still emit metrics; they just don’t generate trace data that lives in the trace store.
• Span attribute filtering. Every major tracer supports attribute filtering before send. Strip any span attribute matching a PHI pattern.
• Service-level data classification. Tag services as PHI-touching at deploy time. The agent reads the tag and applies stricter filters to traces from those services.

A reasonable choice is to skip tracing entirely for PHI-touching services. The observability cost is real, but it’s an acceptable trade-off against the breach risk.

The Template

A scrubber config with regex patterns for the most common PHI types. Apply at the observability-agent layer (format shown for Datadog’s Sensitive Data Scanner; patterns are vendor-independent).

For Datadog specifically, this lives at Logs → Configuration → Pipelines → Sensitive Data Scanner. The same patterns apply (sometimes with vendor-specific escape rules) to:

• Elastic ingest pipelines (gsub processors)
• New Relic log obfuscation rules
• Splunk SEDCMD transforms
• Cloudflare Logpush filters

Test the patterns against your actual logs before turning them on in production. Start in “tag-only” mode where matches are flagged but not modified, then flip to “replace” once you’ve verified no false-positive damage to debugging.

Operating Notes

Add an audit step to your engineering checklist for new PHI-touching code: before merge, search the diff for likely log-leak candidates. A regex for logger.info or console.log followed by a variable reference catches most of them.

You will have leaks. Recovery is to immediately rotate the scrubber pattern to cover the new shape, then run the storage-layer access audit for the period the leak was live. If no one accessed the affected logs, you have a finding to document. If someone did, you have a more serious finding to escalate.

The hardest leak to catch is the structured one: a developer adds an extra={'patient_id': ...} field to a log call intending it to be opaque, but the field naming convention propagates patient identifiers into the index. Pattern-based scrubbing won’t catch that. Only code review will.