Trust nobody owned: the expired root certificate and hospital legacy clients

The week

On September 30, 2021, the IdenTrust DST Root CA X3 certificate expired. Let’s Encrypt, the free certificate authority behind a large share of the web’s TLS certificates, had relied on a cross-signature from that root so that older devices would trust its certificates. Meanwhile its own root, ISRG Root X1, spread into operating system and browser trust stores. The date had been announced well in advance, and Let’s Encrypt had arranged a chain that kept most older Android devices working. Even so, clients with outdated trust stores or older TLS libraries, including OpenSSL 1.0.2 and older Apple operating systems, began failing to connect to sites and APIs using Let’s Encrypt certificates. Reports of broken integrations, scripts and devices circulated over the following days.

Certificate incidents are usually server stories: someone forgot to renew. This one ran the other way. The servers held valid certificates, renewed on schedule. What failed were clients that had not refreshed their list of trusted roots in years, and in a health system those clients are everywhere. Each one carries a trust decision that somebody made once, at build time, and nobody has revisited. We look at the episode in three parts: the clients that broke, the inventories that could not see them, and the owners who had to act.

The clients that broke

The failures showed up in places that rarely get an architecture review. Interface engines pulling lab results or eligibility data from partner APIs. Scripts that move files to a clearinghouse every night. Appliances in the data center that check licenses or send telemetry over HTTPS. Tablets and handhelds that can no longer take operating system updates. Medical devices and their gateways that call a manufacturer’s cloud service for updates or remote support.

Each of these carries its own trust store, usually frozen on the day its firmware or base image was built. A Linux appliance shipped in 2016 trusts the roots that existed in 2016. An integration written in Java may trust whatever came with the runtime bundled into the vendor’s installer, not the operating system’s current list. When the trust anchor moved, these clients had no way to follow.

Health systems have more of them than most industries. Clinical devices are validated as configured, which discourages routine updates. Laboratory, imaging and pharmacy platforms often run for a decade. Integration work is spread across many teams and vendors. The failures were also hard to diagnose. A TLS handshake error at 2 a.m. looks like a network fault or an outage at the partner, and the first hours went on the wrong theory.

The inventories that could not see them

Organizations that tracked certificates at all tracked the ones they issued: the certificates on their own websites and servers, with renewal dates in a spreadsheet or a management tool. That answers the question “when do our certificates expire.” It does not answer “which of our clients will stop trusting someone else’s certificate.” The second question depends on software versions, operating system builds and embedded libraries.

The facts usually existed in pieces. The configuration management database knew a server’s operating system version. Biomedical engineering knew a device’s firmware level. The vendor knew which TLS library it shipped. Nobody held a view that joined those facts to an outside event: on this date, this root expires, and these clients will reject these services. Without that join, months of advance notice from Let’s Encrypt produced little action, because no one could name the affected systems with confidence.

Building that view is not exotic work. It means adding a few attributes to the asset record for anything that initiates outbound TLS. Record the TLS library and version, where its trust store comes from (operating system, runtime, or bundled), and how that store gets updated. It also means recording which external endpoints each integration calls, since those are the dependencies that change on someone else’s schedule. A health system with that data can turn the next root change into a query instead of a war room.

The owners who had to act

For a server certificate, the owner is clear: the team that runs the server. For a client trust store, ownership is spread across four parties. The operating system vendor decides when a new root ships. The application vendor may bundle its own copy of a library and its own root list. The device manufacturer controls updates through its own validation process. The hospital may have pinned a version for stability and then lost track of why.

When the failures arrived, the fixes were varied and sometimes awkward. Update the trust store. Update the library. Ask the partner to serve a different chain. For devices that could not change quickly, route their traffic through a proxy or integration layer that could. Each fix needed someone with authority to make the change, and often that someone sat in a vendor’s support queue. Health systems that already had escalation paths and named contacts at device and application vendors resolved issues in hours. Those negotiating contact details mid-incident took days.

The durable response is contractual and architectural at once. Procurement can ask, before purchase, how a product updates embedded roots and how fast it can ship a trust store change outside a full release. Biomedical engineering can add that answer to device risk assessments alongside patching and end-of-support dates. Integration teams can keep long-lived clients behind a small number of central brokers whose trust stores are maintained by the health system itself. That shifts the dependency from dozens of frozen images to a few systems the hospital controls. The trade-off is concentration: a misconfigured broker breaks many feeds at once, so it needs change control and monitoring to match.

A simple test shows whether the ownership question has been answered. Pick ten critical outbound integrations and ask who would change each trust store, and how long it would take. If the answers are names and days, the next trust change will be routine. If the answers are “the vendor, probably,” the health system has a dependency it does not manage.

With hindsight

September 2021 turned out to be an early rehearsal for a run of trust changes. In 2024, Google announced that Chrome would stop trusting new certificates from Entrust, a long-established authority, which forced replacements on a deadline customers did not choose. In 2025, the CA/Browser Forum approved a schedule that shortens the maximum lifetime of public TLS certificates in steps, starting at 200 days in March 2026 and reaching 47 days by 2029. Shorter lifetimes make manual renewal impractical and push every organization toward automated certificate management. For health systems, the clients that could not follow the 2021 root change are often the ones that struggle with automation now. Their trust stores and renewal processes are still fixed at build time. The organizations in the best position are those that recorded what their clients trust, not only what their servers present.