How to evaluate FOIA and public records redaction software

What this software does

Redaction software helps staff review records that are responsive to a request and remove the parts that are exempt from release. Most products remove duplicates, group email threads, find likely sensitive information such as Social Security numbers, and let reviewers apply redactions labeled with an exemption. The output is a release set with the redactions permanently applied, and a log of what was withheld and why.

Request tracking (intake, fees, correspondence and deadlines) and collection (searching email and file systems) are often separate tools.

The rules that shape the work

The Freedom of Information Act (5 U.S.C. 552) applies to federal agencies. States, counties and cities follow their own public records laws, which set different deadlines, exemptions and fee rules, so check your own statute alongside the points below. For federal agencies, several provisions bear directly on redaction tools:

• Segregation and marking. Any reasonably segregable portion of a record must be released after the exempt portions are deleted. The amount of information deleted, and the exemption under which it was deleted, must be indicated on the released record, at the place of the deletion if technically feasible, unless that indication would itself harm the protected interest.
• Foreseeable harm. An agency may withhold information only if it reasonably foresees that disclosure would harm an interest protected by an exemption, or if disclosure is prohibited by law. It must consider partial disclosure and take reasonable steps to segregate and release what is not exempt.
• Nine exemptions. Subsection (b) lists nine exemptions. Some depend on context that software cannot reliably judge, such as the deliberative process privilege under Exemption 5.
• Deadlines. An agency determines whether to comply with a request within 20 working days. In unusual circumstances that limit can be extended by written notice, generally by no more than 10 working days.
• Formats and posting. Records must be provided in the form or format requested if they are readily reproducible in it, and records released that have been requested three or more times must be made available electronically.
• Reporting. Each agency submits an annual FOIA report to the Attorney General and the Office of Government Information Services.

The Department of Justice’s Office of Information Policy publishes the Guide to the Freedom of Information Act and government-wide guidance on applying the exemptions. Two other rules often come up. Requests from individuals for records about themselves can also involve the Privacy Act of 1974. And under 32 CFR 2002.44, marking information as controlled unclassified information (CUI) does not decide a FOIA request: the decision rests on the content and the FOIA exemptions, whether or not the information is marked.

Who should be on the evaluation team

• FOIA analysts and reviewers, who can judge whether proposals save time or create rework.
• The FOIA officer or program manager, who tracks the backlog and reports on it.
• Agency counsel, who advises on exemptions and defends withholdings if a request is litigated.
• The records manager, for collections and retention.
• The privacy officer, for Privacy Act requests and the handling of personal information in the tool itself.
• IT security and accessibility staff, to review hosting, access control and the accessibility of both the tool and the files it produces.

Questions to ask and things to test

Test with a closed request of your own, with a mix of email, scans and spreadsheets.

• Compare proposals with your analysts’ decisions. What did the tool flag that they did not, and what did it miss?
• Try your most common record types, including poor scans, long email threads with attachments, and spreadsheets with hidden rows or columns.
• Look at duplicate grouping. How are near-duplicates grouped, and how easily can an analyst split a group that should not have been merged?
• Check how each redaction is labeled. Does every redaction carry its exemption at the place of deletion, and can a reviewer see why it was proposed?
• Inspect the released file. Open it outside the tool and try to copy the text under a redaction, search for a redacted word, and read the metadata, comments and tracked changes. Nothing redacted should be recoverable.
• Review the withholding log with counsel. Does it record each exemption, page and reviewer in a form that would help prepare a Vaughn index if the request is litigated?
• Ask how CUI markings are treated. Flagging marked content for review is useful; treating the marking itself as a reason to withhold is not.
• Walk through second-level review, consultations and referrals.
• Ask how it connects to your collection and request tracking tools.

Common limits to probe

• Who decides. Automated detection proposes redactions. Exemption decisions belong to the agency; no record should be releasable without human review.
• What it misses. Context-dependent material, such as deliberative content or information that identifies someone only in combination with other facts, is hard to detect. Analysts stay responsible for a complete review.
• Input quality. Handwriting, faint scans and unusual file types reduce detection accuracy and need closer human review.
• Media. Audio and video redaction is often not included, or is sold separately.
• Classified information. Most commercial tools are not designed for it; such records are handled under your agency’s own procedures.
• Hosting and authorization. Ask what authorization the service holds for federal use, and what documentation the vendor provides to support your agency’s own authorization decision.

Implementation risks

• Recoverable redactions. The most damaging failure is a release in which a black box covers text that can still be copied, or in which metadata, comments or earlier versions survive. Check released files every time.
• Automation bias. Reviewers may come to accept proposals without reading the page. Keep sampling and second-level review in place, and measure what reviewers change.
• Inconsistent decisions. Term lists, exemption lists and reviewer guidance need an owner.
• The backlog during transition. Decide which open requests finish in the old process and which move, so no statutory deadline falls between two systems.
• Sensitive data in a new place. The tool will hold some of the agency’s most sensitive records before they are redacted. Review access, retention and deletion settings, and how request files will be kept under your records schedule and the Federal Records Act.
• Accessibility. Federal agencies must make the information and communication technology they develop, procure, maintain or use accessible under Section 508. Ask for an Accessibility Conformance Report for the reviewer interface, and check that released files work with assistive technology.

Related reading on this site

Requirement pages: Freedom of Information Act, Privacy Act of 1974, Controlled Unclassified Information (CUI), Federal Records Act, State public records laws, Section 508.

In the glossary: FOIA, Privacy Act of 1974, CUI, Section 508.

This guide describes the category, not any one product. It is general information, not legal advice: for what a rule requires of your organization, read the rule itself or ask your counsel.