A chatbot’s history sidebar is a records system with someone else’s retention policy
OpenAI’s March 2023 bug was minor as breaches go. It showed health systems that staff conversations with AI assistants are stored data held by a vendor. HIPAA’s questions follow: what goes in, how long it stays, who sees it.
Looking Back revisits one week at a time, from January 2020 to today, and asks what it meant for the banks, health systems and public agencies we serve. The series was written in 2026, with the benefit of hindsight, and each article says which week it looks back on.
The week
On March 20, 2023, OpenAI took ChatGPT offline after users reported seeing the titles of other people’s conversations in their history sidebar. On March 24 the company explained that a bug in an open-source library used for caching had caused some users to see data belonging to others. It said that during a window of about nine hours, payment-related details of about 1.2 percent of ChatGPT Plus subscribers active at the time may have been visible to other users. Those details included name, email address, payment address, and the last four digits and expiry date of a card, but not full card numbers. OpenAI said it had fixed the bug and contacted affected users. On March 30 Italy’s data protection authority ordered a temporary limit on ChatGPT’s processing of Italian users’ data, citing the incident among other concerns.
By the standards of 2023’s breaches, this one was minor. It was fixed quickly, the data involved was limited, and the vendor explained it within days. Its significance for health systems lay elsewhere. Many staff thought of a chatbot as a tool, like a calculator: ask a question, get an answer, move on. The bug showed that each conversation was stored, indexed and displayed back. It was a record on a vendor’s servers. Treating AI conversations as a data store raises three governance questions for any health organization: what goes in, how long it stays, and who can see it.
What goes in
The first question is classification. Clinicians and staff use AI assistants with whatever helps them work. That can mean a draft patient letter, a referral summary, a denial appeal, a spreadsheet of scheduling data or a block of clinical notes to tidy up. The titles exposed by the bug were generated from conversation content. Even the metadata could reveal what someone was working on, and in a hospital that can include a patient’s condition.
Under HIPAA, entering protected health information into a consumer service with no business associate agreement is, in most cases, a disclosure the Privacy Rule does not permit. In early 2023, consumer chatbots offered no such agreement. Many staff did not see a chatbot as a cloud service at all. A policy forbidding PHI in unapproved cloud services did not obviously apply, in their minds, to a chat window.
Closing that gap means naming AI assistants in acceptable use and data handling policies. Those policies should state which data classes may go into which approved tools. They should also give practical examples of what de-identification means, because “remove the name” falls well short of HIPAA’s standards. Technical controls back the policy: web filtering that steers users to approved tools, and data loss prevention rules tuned for medical record numbers and clinical content. Training should be specific to roles. Revenue-cycle staff drafting appeals face different temptations from physicians summarizing charts.
How long it stays
The second question is retention. Consumer AI services kept conversation histories by default so users could return to them. At the time, OpenAI also used consumer conversations to improve its models unless users opted out. Information typed into a prompt could therefore persist in at least two forms: as a stored conversation and as training data.
Health organizations face retention pressure in both directions. Privacy principles and state laws push toward deleting data when it is no longer needed. Medical records rules, litigation holds and HIPAA’s documentation requirements push toward keeping certain material. A consumer tool controlled by the individual user and governed by the vendor’s terms meets neither need. The organization cannot delete what it does not control, and it cannot produce what it never captured.
Enterprise agreements, which began to appear later in 2023, changed the picture. They offered commitments on retention periods, exclusion from model training and, for some vendors and products, a business associate agreement. Those are procurement questions, and they belong in the standard contract checklist for any AI tool. The checklist should cover where conversations are stored and for how long, and whether the customer can set retention. It should also ask what logs the vendor keeps for abuse monitoring, and whether the vendor will sign a BAA for the specific product in use, not just for its platform generally.
There is also a records question inside the organization. If a clinician uses an approved assistant to draft part of a note or a patient message, some of that work may belong in the designated record set. Health information management leaders should decide which AI-assisted outputs are records, and make sure the final versions land in the EHR rather than only in the chat history.
Who can see it
The third question is access. The bug was a failure of the vendor’s own separation between users. Ordinary access risks matter more day to day. Shared logins, unmanaged personal accounts and conversations that persist after an employee leaves are common in consumer tools. An organization cannot revoke access to an account it does not know exists.
The controls are familiar from other software. Require single sign-on so access follows employment and ends with it. Prefer tools that support audit logs the organization can review. Restrict sharing features that create public links to conversations. Treat the AI vendor as a business associate where PHI is involved, with the security review and incident notice terms that implies. The vendor’s own isolation between customers also deserves questions. A caching bug that crossed user boundaries in a consumer product could, in principle, cross tenant boundaries in an enterprise one. Contracts should require prompt notice of any such failure.
Regulators showed they would treat these stores as personal data in the full sense. Italy’s order a week later cited the incident alongside broader concerns about legal basis, transparency and age verification. For US health systems, the parallel authority was their own HIPAA program, applied to a new kind of system.
With hindsight
Italy’s regulator lifted its restriction at the end of April 2023 after OpenAI made changes, then in December 2024 fined the company €15 million. Among its findings was a failure to notify the authority of the March 2023 breach. OpenAI said it would appeal. Enterprise AI agreements soon routinely offered no-training commitments, configurable retention and single sign-on, and some vendors began offering business associate agreements for specific products. In May 2025, in the New York Times copyright case, a federal court ordered OpenAI to preserve consumer chat logs it would otherwise have deleted. That obligation was largely lifted that autumn. The episode underlined that a vendor’s retention can be set by a court the customer has no part in. The three questions now extend to assistants connected to EHRs and email, where what goes in includes whatever the tool can reach.