Adding AI Features Without Breaking Your Compliance Posture
Every healthcare-adjacent SaaS is racing to add AI features. Most haven't traced what that does to their HIPAA or SOC 2 posture. Four leak points, one pre-flight checklist, no surprises at the next audit.
AI-feature compliance pre-flight checklist
Copy as markdown to paste into your repo, or download a branded PDF for sharing with non-technical stakeholders.
Frequently asked questions
Do I need a BAA to use an LLM with PHI?
Yes. If protected health information goes into a prompt, the model vendor is processing PHI on your behalf, and under HIPAA that requires a Business Associate Agreement — the same as it does for AWS or your database host. The major providers offer BAAs, but only on request and not on every tier; the free ChatGPT product doesn't qualify.
What if I can't get a BAA from the model vendor?
You have two honest options: de-identify the data before it leaves your systems (Safe Harbor or Expert Determination), or don't build the feature that way. 'We'll get the BAA later' is how PHI ends up in a training corpus.
Where does PHI leak when you add an AI feature?
Four places: the prompt (the context you assemble and send to the vendor), the vendor itself (a new sub-processor), the logs (prompts and responses captured for debugging, evals, and observability), and the output (a hallucinated dosage or wrong eligibility answer). None show up in a demo; all show up in an audit, a security review, or a breach investigation.
How should we log prompts and responses for a PHI-bearing feature?
Don't log full prompts or responses to general application logs — log the shape (token counts, latency, model version, a request ID), not the content. If you must retain prompts for evals, keep them in a scoped, encrypted store inside your HIPAA boundary with a deliberately set retention period, not a default.
Does AI output need a human reviewer?
For anything consequential, yes — the more consequential the output, the more the human has to be in the decision, not just notified of it. Summarizing a note for a clinician to review is fine; auto-sending a prior-auth denial with no human review is not. Draft, don't decide.
How often should we re-check the model vendor's terms?
Quarterly. AI providers update their data-use and retention terms far more often than a database host does, and a default that flips from 'no training' to 'training unless you opt out' won't send you an email. Run the pre-flight per feature and per data flow too — a BAA for one vendor doesn't cover the next.
Get the next one.
New operating guides as we publish them — practical, infrequent, unsubscribe in one click.
The model vendor is now in your audit scope.
Kaan Systems does platform engineering and Terraform for funded SaaS, health-tech, and fintech. Compliance evidence is the by-product of that work. The Teardown runs the pre-flight on the feature you are about to ship: sub-processor, data store, and the path out. The Architecture and Compliance Teardown is a written read in about five business days. Book a call.