Why o1’s reasoning summaries added work for clinical AI committees
OpenAI’s o1 narrated its reasoning just as federal rules on decision support transparency were taking hold. A fluent explanation answers a clinician’s question, not the vendor or the regulator’s, and health systems had to supply three different kinds of transparency themselves.
Looking Back revisits one week at a time, from January 2020 to today, and asks what it meant for the banks, health systems and public agencies we serve. The series was written in 2026, with the benefit of hindsight, and each article says which week it looks back on.
The week
On September 12, 2024, OpenAI released o1-preview and o1-mini, the first widely available models built to spend more computation reasoning through a problem before answering. The company reported strong results on mathematics, coding and science benchmarks. It chose not to show users the model’s raw chain of thought, displaying a generated summary instead. The release arrived as health systems prepared for two federal rules on clinical algorithms. The ONC HTI-1 final rule required developers of certified health IT to support “source attributes” for decision support interventions by the end of 2024. The HHS Section 1557 nondiscrimination rule, finalized in May 2024, required covered entities to identify and mitigate discrimination risks in patient care decision support tools, with that provision’s compliance date in May 2025.
A model that explains itself sounds like what clinical AI governance had been asking for. o1 produced fluent, step-by-step accounts of how it reached an answer. For a clinician at the point of care, that is reassuring. For a governance committee, it created more work. A plausible explanation is not an accurate account of what a model did, and OpenAI itself said users were seeing a summary, not the underlying reasoning.
The release blurred three kinds of transparency that have different audiences and different owners. Transparency at the point of use tells a clinician why this answer, for this patient. Product transparency tells the deploying organization what the model is, how it was built and where it fails. Deployment transparency tells patients, regulators and auditors how the organization chose, tested and monitors the tool. Every disclosure a health system makes about clinical AI belongs to one of these three. o1 improved only the first.
Point-of-use transparency: an explanation is output, not evidence
A reasoning summary is generated text, produced by the same system whose answer is under review. It may faithfully describe the path to the answer or it may be a tidy rationalization. Research on chain-of-thought faithfulness had already shown that stated reasoning does not always reflect the factors that drove a model’s output.
Explanations still have value. They can help a clinician spot an obvious error, such as a misread lab value or a wrong unit. The risk is the opposite effect: a confident, well-structured rationale lowers a reader’s guard precisely where vigilance matters. Dosing, contraindications, pediatric and renal adjustments, and rare presentations are the cases where an articulate wrong answer does the most harm.
Health systems can manage this with design choices. Interfaces can show the explanation only after the clinician has formed a view. Training can teach staff to check the cited facts rather than the flow of the argument. Testing can deliberately include cases where the model is known to be wrong, to see whether reviewers catch the error when the explanation is persuasive.
Product transparency: what the rules ask vendors to disclose
HTI-1 placed obligations on developers of certified health IT. For predictive decision support interventions the developer supplies, users must be able to see source attributes such as intended use, development data, known risks, validation and fairness information. An earlier article in this series showed how narrow that duty was: for models a hospital built or bought elsewhere, the EHR only had to hold the information, and the hospital had to produce it.
General-purpose reasoning models fit awkwardly here. A model used through a chat interface for clinical questions is often not a decision support intervention inside certified health IT at all. It arrives with general documentation, such as a system card, that describes benchmark results and safety testing but says nothing about a specific hospital’s patients or workflows.
The gap has to be closed in procurement. A health system can require a vendor to state the permitted clinical uses and the model versions it will deploy. The contract should also cover notice of model changes and whether customer inputs are retained or used for training. Before any protected health information is entered, a business associate agreement must be in place. Where clinicians are already using consumer chat tools on personal accounts, as many were in 2024, product transparency means little until the approved tool is easier to use than the unapproved one.
Deployment transparency: what only the health system can show
The Section 1557 rule put its decision support duty on covered entities, the providers and plans themselves. They must make reasonable efforts to identify tools that use race, color, national origin, sex, age or disability as input variables, and to mitigate the risk of discrimination. That duty sits with the deployer, whoever built the tool. Counsel should advise on how it applies to particular deployments, including general-purpose models.
Meeting it requires records only the health system can create: an inventory of decision support tools in use, the evidence behind each approval, local validation results on the organization’s own population, and monitoring that would detect drift or uneven error rates. For reasoning models, the record should also state which uses are permitted. Drafting patient letters and summarizing literature carry different risks from suggesting a diagnosis or a dose.
Accountability needs a name. A committee can approve a use, but a specific person must receive error reports, review monitoring data and hold authority to suspend the tool. Without that role, the documentation describes a control that nobody runs. The same person should own the link between clinician error reports and the vendor, so that a pattern of bad answers reaches the people who can change the model or its configuration.
Seen this way, o1 made one layer of clinical AI transparency more convincing and left the other two unchanged. That is a reason to invest in the two layers the model cannot supply.
With hindsight
Reasoning models became standard within a year, and every major provider shipped them. The concern about explanations held up. In 2025 researchers at model developers, including Anthropic, published work showing that reasoning models often omit factors that influenced their answers. Model providers cautioned against treating stated reasoning as a full account. On regulation, the two layers diverged. HTI-1’s source attribute requirements took effect for certified developers. In late December 2025, however, the federal health IT office proposed the HTI-5 rule, which would remove those AI “model card” requirements from the decision support certification criterion. As of this writing that proposal has not been finalized. The Section 1557 decision support duty for deployers took effect in May 2025 and became the more durable obligation. Product transparency, in other words, shifted further toward contracts, and deployment transparency stayed with the health system.