A weekend of public red-teaming showed agencies that AI testing never finishes

The week

Over the weekend of August 10 to 13, 2023, the DEF CON security conference in Las Vegas hosted a public red-teaming exercise against large language models from several leading AI developers. It was organized by the conference’s AI Village with partners and with support from the White House Office of Science and Technology Policy. Participants had timed sessions to make the models produce harmful, false or biased output, or to break their safeguards, with results collected for later analysis. Earlier that week, at the Black Hat conference, DARPA announced the AI Cyber Challenge, a multi-year competition to build AI systems that find and fix software vulnerabilities. Both followed the voluntary commitments the White House secured from major AI companies in July, which included internal and external security testing of models before release.

The DEF CON exercise showed something useful about large language models. Ordinary people, given an hour and some prompts, could make leading models misbehave in ways their developers had not anticipated. For agencies, the more practical message was about workload. A model that can be broken in new ways by new people cannot be certified once and filed. Testing is a recurring activity, and every recurring activity in government needs budget, staff and authority. We break that workload into three stages, which together cover an AI system’s life in an agency: before award, before deployment, and in operation.

Before award

Procurement is where an agency has the most influence over an AI vendor and usually the least testing capacity. Evaluation panels typically score written proposals and scripted demonstrations. Neither reveals how a model behaves when users push it, misunderstand it or try to abuse it.

An agency can do better without building a laboratory. Solicitations can require vendors to describe their own red-teaming, including who performed it, what categories of harm it covered and what was fixed as a result. Evaluations can include a hands-on test period in which agency staff, including frontline users, try the system against realistic tasks and known failure patterns. Contracts can require notice of model changes and access for agency testing after award.

Each of these needs people who understand both the mission and the technology. A contracting officer cannot design a test of a benefits chatbot alone. A program office cannot judge a vendor’s red-team report without someone who knows what good testing looks like. Agencies that lack that person end up accepting vendor assurances by default.

There is a market effect too. When agencies ask the same testing questions in every solicitation, vendors learn to answer them well and to invest in the testing behind the answers. When each agency asks differently, or not at all, vendors optimize for demonstrations. Shared solicitation language across agencies, maintained by a central team, is cheaper than each agency writing its own and gives buyers a common baseline for comparing bids.

Before deployment

A model tested by its developer has been tested in general. An agency deploys it in a specific context: particular users, particular data, particular consequences of error. A chatbot that answers tax questions, a tool that summarizes case files and a system that drafts eligibility letters each fail in different ways.

Pre-deployment testing should therefore be built around the agency’s own use. That means test sets drawn from real questions and documents, cases representing groups the system might treat unfairly, and adversarial attempts by people who know how the program is gamed. Fraud investigators, caseworkers and call center staff often find failures that technologists miss, because they know what real users and bad actors do.

Testing also needs a decision attached. Before testing starts, the agency should decide what results would stop deployment, what would require mitigation, and who signs off. Without that, test findings become a report that nobody is obliged to act on. The authorizing official for the system, or an equivalent senior owner, should accept residual risk in writing, as already happens for security authorizations.

Test data raises its own issues. Realistic tests need realistic records, and those records often contain personal information. Agencies should decide early whether testing can use synthetic or de-identified data, and where it cannot, run tests inside the same controlled environment as production. The privacy office should sign off on the test plan, not only on the deployment.

In operation

The DEF CON results arrived after the exercise, and they described models that would soon be updated. That is the normal state of AI products. Vendors change underlying models, adjust safety filters and add features, sometimes monthly. Each change can fix one failure and introduce another. User behavior also changes once the public learns what a system will and will not do.

Operational testing has three parts. Monitoring watches outputs in production for known failure signs, such as refusals, complaints, escalations to human staff and unusual patterns of use. Regression testing reruns the agency’s test set whenever the vendor announces a model change. Periodic red-teaming brings in fresh testers, internal or external, to look for new failure modes. Findings need a route back to the vendor, with contract terms that require a response, and a route to the program owner, who decides whether to pause use.

All three require ongoing staff time, and that is where most agency plans in 2023 were thin. AI teams were small, and the people who understood both models and missions were in demand across government and industry. A realistic plan budgets testing as an operating cost for each deployed system, names the people who do it, and shares capacity across agencies where possible. Shared test sets for common uses, such as public-facing chatbots, reduce duplication. Without that capacity, testing requirements written into policy will be met on paper and missed in practice.

Operational findings also need an incident path. When a public-facing system gives harmful advice, the agency needs to decide quickly whether to suspend it, correct it or warn users, and to know who has authority to do each. Treating serious AI failures as incidents, with the same logging, escalation and after-action review as a security incident, gives the testing program teeth and gives leaders evidence about where to invest.

With hindsight

Red-teaming moved into federal policy quickly. The October 2023 executive order on AI directed NIST to develop red-teaming guidance, and NIST’s generative AI profile for its AI Risk Management Framework, published in July 2024, built testing into its recommended practices. OMB’s March 2024 guidance required pre-deployment testing and ongoing monitoring for rights- and safety-impacting uses. The replacement guidance issued in April 2025 kept pre-deployment testing and monitoring for high-impact AI even as other parts of federal AI policy changed. DARPA’s AI Cyber Challenge ran its final round at DEF CON in August 2025. Capacity did not keep pace. Federal technology teams were cut during 2025, including the closure of GSA’s 18F digital services team, which left testing requirements in place with fewer people available to carry them out.