A hallucination rate is a sales claim, and Texas enforced it like one
Texas’s settlement with Pieces Technologies applied an old consumer protection statute to a generative AI error rate. Health system buyers can test any such number on three dimensions: what it counts, where it was measured, and how it stays true after updates.
Looking Back revisits one week at a time, from January 2020 to today, and asks what it meant for the banks, health systems and public agencies we serve. The series was written in 2026, with the benefit of hindsight, and each article says which week it looks back on.
The week
On September 18, 2024, the Texas Attorney General announced a settlement with Pieces Technologies, a Dallas company whose generative AI product summarized patient information and drafted clinical documentation for hospital staff. The office alleged that Pieces had made false and misleading statements about its product’s accuracy, including advertised metrics for its rate of severe “hallucinations”, in violation of the Texas Deceptive Trade Practices Act. It described the case as a first-of-its-kind settlement involving generative AI in healthcare. Pieces denied wrongdoing. Under the agreement, the company committed to disclose clearly how any advertised accuracy metric was defined and calculated. It also agreed to inform customers of known or reasonably knowable harmful or potentially harmful uses of its products. The settlement carried no financial penalty. A week later, the Federal Trade Commission announced “Operation AI Comply”, a set of actions against deceptive AI claims.
The settlement created no new AI rule. It took a statute written for misleading advertising and applied it to a number that describes how often a generative model gets things wrong. Once a state enforcer treats that number as a representation that must be accurate and explained, any health system that relied on one in a purchasing decision has a reason to look at it again. Most states have similar consumer protection laws, so the theory travels.
A clinical accuracy claim has three components, and each can fail independently. The first is the definition: what counts as an error. The second is the measurement: on which records, by whom, under what conditions. The third is maintenance: whether the number still describes the product after it changes. A claim is only as reliable as the weakest of the three, and a buyer who examines all three has examined the whole claim.
The definition: what counts as an error
A hallucination rate sounds precise until someone asks what a hallucination is. A summary that omits a medication, one that invents a lab value and one that places an event on the wrong day are different failures with different clinical consequences. So is an omission compared with a fabrication. A vendor can report a very low rate simply by counting only the most severe category, or by excluding omissions.
Buyers should ask for the error taxonomy in writing: the categories, the severity scale, the rules for scoring ambiguous cases, and examples of each. They should ask for rates for every category, not only the headline one. They should also ask who did the scoring. Clinicians independent of the product team give a different answer from the engineers who built it.
The definition must also fit the intended use. A rate measured on discharge summaries says little about nursing handoff notes, prior authorization letters or patient-facing messages, where the cost of a given error differs.
The measurement: where and on whom
An error rate measured on one hospital’s records, in one specialty and one documentation style may not hold elsewhere. Patient complexity, note templates, local abbreviations, interpreters and languages, and the configuration of the electronic health record all affect performance.
The questions here concern sampling. How many cases were reviewed, and were they drawn at random or chosen? Did the sample include complex patients with many conditions? Pediatric cases? Patients with limited English proficiency? Was the evaluation repeated after model updates, and are the results from each round available? A vendor that cannot answer has a marketing figure, not a measurement.
The measurement also depends on the reference standard. If reviewers compared the draft only with the source note, errors already present in that note go uncounted. If they compared it with the full chart, the rate will be higher and more honest. Neither is wrong, but the buyer needs to know which one produced the number, and whether the comparison was blinded to which drafts came from the tool.
A health system can close part of this gap itself with a structured local pilot. A practical design is a fixed sample of local encounters, drafted by the tool and reviewed by clinicians against the source record using the vendor’s own taxonomy. That produces a local rate the organization can compare with the advertised one. It also produces something more useful: a list of the error types the tool makes in this environment, which shapes training and review policy.
The maintenance: keeping the number true
Generative AI products change often. Vendors update underlying models, prompts and retrieval components, sometimes without visible version changes. A number measured when the contract was signed may not describe the product six months later.
The settlement’s second commitment, disclosure of known harmful uses, implies an ongoing duty, and buyers can mirror it in contracts. Useful terms include notice of material model changes, re-measurement of advertised metrics after significant updates, a defined channel for reporting errors with response times, and disclosure of harmful uses or incidents identified at other customers. Renewal is the moment to enforce these terms, because that is when the buyer has the most bargaining power.
Internally, maintenance means a low-friction way for clinicians to flag errors from inside the workflow, and a named owner who reviews the reports, spots patterns and escalates to the vendor. It also means monitoring how much clinicians edit drafts. A sudden fall in edits may mean the tool improved, or that reviewers stopped reading carefully. Only sampling against the source record tells the two apart.
Vendor enforcement does not shift the provider’s own responsibility. A health system that deploys a documentation tool remains responsible for the accuracy of the medical record and for patient safety, whatever the vendor advertised. Counsel can help allocate risk through warranties and indemnities, but no contract term replaces knowing what the number means.
With hindsight
The Pieces settlement became a reference point for later state activity on AI in healthcare. In September 2024 California enacted AB 3030, which from January 2025 required disclosure when generative AI is used in certain patient communications. In 2025 Texas passed two relevant laws. SB 1188, effective September 2025, requires practitioners to review AI-generated records and recommendations before relying on them. The Texas Responsible AI Governance Act (HB 149), effective January 2026, requires providers to disclose AI use in treatment to patients. Federal enforcement priorities on AI shifted after January 2025, which made state law relatively more important. Ambient documentation tools spread quickly through 2025 and 2026, and the review burden on AI-drafted notes fell, as the settlement implied it would, on clinicians and their organizations.