The Forensic Gap in AI Safety Laws
A trio of state-level laws has expanded the reach of government regulators over the United States’ artificial intelligence (AI) industry. California’s Senate Bill (SB) 53, New York’s Responsible AI Safety and Education Act, and Illinois’s SB 315, or the Artificial Intelligence Safety Measures Act, all require frontier developers to report certain safety incidents to state authorities. The laws also require the largest developers—those with over $500 million in gross annual revenue and whose models use high levels of compute to train—to publish AI frameworks that detail their safety standards.
Illinois’s law, the most recent of the three, includes a mandatory annual independent audit provision. California’s legislature passed two bills in August—AB 1405 and SB 813—under which Sacramento would establish an AI auditor registry and designate qualified evaluators; California Governor Gavin Newsom has until the end of September to sign them.
The regulatory regime these bills have created—incident reporting, increased transparency, and independent audits—helps close the knowledge gap between developers and government.
Although a step forward, the laws do not outline what comes after the incident report, as Mackenzie Arnold and Stephan Llerena have noted—an omission that matters if they are meant to produce safety comparable to other high-risk industries such as aviation and nuclear energy. In those industries, safety improvements rely on a cumulative forensic process: Investigators determine what happened in a specific incident, and why, and they propose corrective measures that can prevent it from recurring. Scaled up across an industry, that process drives safety improvements over time.
The omission of a clear investigatory regime from today’s AI laws thus matters for any risky technology, but AI poses special challenges. Frontier models are opaque enough that developers cannot reliably say why a model produced the outputs it did. Investigators may struggle to reconstruct an accident given the complexity of the evidence needed. And developers retain most of the data, technical access, and expertise required for reconstruction. Together, these obstacles may leave regulators dependent on developers to reconstruct what happened and limit their ability to translate incident investigations into systematic safety improvements.
Investigative authority alone may therefore be insufficient for a robust AI safety regime. Unless the developer preserves the right evidence and investigators have the technical capacity to interpret it, investigators may not be able to reconstruct the event in the first place. The oversight models proposed currently—developer-retained independent third-party auditors, a Financial Industry Regulatory Authority (FINRA)-style organization, or a Federal Aviation Administration (FAA)-style regulator—would all face this problem: Unless the evidence has been preserved and external authorities have the expertise needed to interpret it independently, investigators will remain dependent on developers.
AI may therefore need an investigatory regime that does not follow the standard playbook. Two preconditions for such a regime can be established now, before a major accident happens.
First, regulators can require developers to preserve data so that incidents may be reconstructed. None of the three state bills stipulates this level of retention, but without it, investigators may not have the data they need to construct a causal account.
Second, regulators can require large frontier developers to form dedicated units tasked with preserving incident evidence and helping investigators to access and interpret it. This would help investigators decrease their epistemic dependence—their reliance on the developer not only for the evidence but also for their interpretation of what that evidence means—when conducting investigations.
The Reporting Regime
The incident reporting statute at the core of Illinois, California, and New York’s AI laws is substantially the same: Developers are required to report “critical safety” incidents to state authorities within a specified time period. The laws all use the same definition of critical safety incidents as one of four events: unauthorized access to model weights causing death or bodily injury; harm from catastrophic risk; loss of model control causing death or injury; or deceptive model behavior that subverts developer control and demonstrates materially increased catastrophic risk.
The laws also define “catastrophic risk” as a “foreseeable and material risk” that a “developer’s development, storage, use, or deployment of a frontier model” will “materially contribute to” the death or serious injury of more than 50 people or more than $1 billion of loss or damage to property in a single incident; the laws enumerate several risk pathways.
Under the scheme, developers must provide “basic information” about the incident: when it happened, why it qualifies, and a “short and plain” description of incident specifics. Large developers must report quarterly self-assessments of catastrophic risk from internal model use. The reporting deadlines are 15 days in California, 72 hours in New York and Illinois, and a 24-hour rule in all three for incidents that pose imminent risk of death or serious injury.
Finally, the laws differ on penalty specifics and the report-receiving bodies (emergency management agencies in Illinois and California, and a new body under the Department of Financial Services in New York). California’s reporting mechanism took effect on Jan. 1 of this year; New York’s and Illinois’s will take effect Jan. 1, 2027.
The scheme’s definitions, conditions, and arguably high reporting thresholds exclude many important cases. The recent Hugging Face breach by OpenAI agents is instructive. Despite prompting federal attention and disclosures by other developers, the breach might not qualify as a statutory critical safety incident: It carried no death, serious injury, or billion-dollar property loss, leaving only the deception pathway, where the evidence is ambiguous.
Unlike other provisions in the regulatory laws, the scheme asks developers to act mostly after the fact: Developers report only after harm has accrued or, in the case of deception, after a model has already done something to demonstrate “materially increased catastrophic risk.” The parallel reporting system requires only large developers to report their assessments—not that they must conduct assessments, or what they look like.
What happens after a report is made is less clear. New York and Illinois require their receiving bodies to review it, but none require the developer to investigate its own incident, nor assign investigatory powers to another actor. Developers therefore retain substantive control over what’s reported and what happens afterward, while state authorities have few tools to investigate. If SB 53 and its companion laws are to improve AI safety over time, the reporting scheme will need a mechanism to generate corrective measures and safety recommendations.
The Forensic Problem
Investigatory regimes in other safety-critical domains rely on evidence, access, and expertise to reconstruct an accident and make recommendations. In aviation, for example, operators must preserve a wide swath of evidence—crash wreckage, cargo, recorders, and records—until handed over to the National Transportation Safety Board (NTSB). The NTSB, in turn, has wide powers to find out what caused the accident and make recommendations that can prevent its recurrence.
Current U.S. AI laws provide none of these. This gap matters because AI presents an unusual forensic challenge to any investigatory regime, ahead of whatever form it takes.
A First Challenge Is Model Opacity
Frontier AI models are described as “black boxes.” Developers control much over a model’s creation process—the training data, inputs, and model weights—but the process’s complexity makes it difficult to understand how a model produces a particular output. The extent of this interpretability problem is often misunderstood: Developers have made real gains in explaining how a model works under the hood by identifying a model’s internal features that are associated with specific outputs—a process called mechanistic interpretability. But limits exist as to how much these approaches can tell an investigator, particularly about corrective measures.
With complete information and access, a developer could provide an investigator with forensically important information: what the model did; what in its training or posttraining process likely contributed to the output; and what input likely triggered the incident. Investigators could derive these from an exact system reconstruction (the model, hardware, operating environment, weights), incident logs, decision traces, training records, testing history, and any relevant organizational data, including design documents and risk assessments. Investigators could replay the event—running the model again using the same inputs and under the same conditions to see if it produces the same result. They could also remove or alter parts of the model—called ablation—to see if this changes the result. And investigators could ask mechanistic interpretability engineers to find which features were implicated in the error.
What these techniques cannot do is point to a stuck valve—a malfunctioning or broken part that can be fixed, swapped out, or replaced. Interpretability engineers can connect features to outputs, and “clamp” a feature to suppress an output. But the feature cannot be pulled out and replaced akin to a part.
A developer has a number of tools to alter a model’s output—fine-tuning, changed prompts, retraining, new safeguards—to see if the error recurs. But unlike swapping out a part, these techniques cannot foreclose the failure mode in a verifiable way. The model’s input space—the enormous range of inputs and contexts it may encounter—is vast and unstructured. (In technical terms, general-purpose AI models are often not given the narrow specified operational design domains, such as those that power self-driving cars.)
A comparison with aviation can illustrate the difference. In the Boeing Maneuvering Characteristics Augmentation System (MCAS) scandal that downed two planes and killed 346 people, the system’s operations were opaque to the pilots, who couldn’t figure out what was happening in time to recover from failure. But the chain of events was transparent to investigators, who were able to identify a bounded component-level sequence: The MCAS system relied on a single angle-of-attack sensor, and that sensor malfunctioned, causing it to repeatedly put the aircraft’s nose down despite the pilots’ efforts to pull up.
The investigation yielded specific fixes: MCAS would read two sensors instead of one, and it would shut down if they disagreed. Engineers could test this fix against MCAS’s real-world operating conditions, such as whether an airplane’s flaps were up or down, because they were both known ahead of time and limited. Engineers could thus be confident the fix would foreclose the MCAS failure mode that caused the two crashes. In the case of an AI accident, investigators can sample how a retrained AI model responds to a range of similar conditions. But they can’t test it against the full range of conditions it will encounter: The model could respond to similar or new conditions in unanticipated ways that reintroduce the error. And a retrained model might open novel pathways to failure the original model did not have.
A Second Challenge Is Evidence
To accurately reconstruct an accident, investigators need as complete a record of the execution environment as possible. The inputs and conditions of the run that produced the accident would have to be preserved, as well as the surrounding compute stack, given that a high-fidelity reconstruction may require producing the same framework, kernel libraries, hardware, and serving configuration, not just the model weights. If the model was deployed agentically, the relevant evidence expands to include system and user prompts, permissions, memory, any safeguard around the model, and so on.
AI’s evidentiary configuration is distributed and mutable: A model’s output is the result of many components and processes, all or most of which can be altered or replaced after the model error occurs. Models are updated frequently, or swapped out for new models. Updates made by chip developers and library vendors—not the AI developers—can change model outputs without a model update or swap.
Even with all these held constant, a model with the same input can give different answers, even when controlled to eliminate randomness, because of engineering choices such as server batching: Servers group model inputs together for efficiency, and the batch’s size changes the result, as Thinking Machines Lab has shown. And recreating the model’s operational condition may be a challenge if it involves long strings of prompts, multiple models interacting with one another, or other environmental complexities stemming from AI’s wide and varied use-contexts.
Contrast once again with aviation. Aircraft have a distributed but relatively stable configuration: They are assembled from thousands of parts made by hundreds of designers and manufacturers in different jurisdictions. Once a component is installed, the manufacturer generally cannot alter it without logging the change. When a plane crashes, investigators thus usually have a detailed record of the aircraft’s configuration and extensive documentation about the design and function of those components. An aircraft’s components wear down over time, but they do not change their story.
The Third Challenge Is the AI Firm Itself
The challenges outlined above are technical problems with existing, if sometimes incomplete, technical solutions. But the actors most able to carry them out—in terms of capability, expertise, and access—are the firms themselves. Firms possess the lion’s share of the relevant data (such as the model versions and operating environment) and the workers needed to make sense of it, such as interpretability expertise.
Put simply, having “grown,” trained, and run their own models, firms know their products better than outsiders do. Firms also have commercial reasons to conceal much of their development, introducing an organizational layer of opacity on top of the technical ones. In the case of internally deployed models, firm employees may be the only actors who know the systems exist. This gives firms a default epistemic advantage—not just control of the data but knowledge over how to interpret it—over outsiders.
A final contrast with the MCAS scandal. Boeing concealed the problem for commercial reasons: The 737 Max would be less competitive if it required additional pilot training. Nonetheless, aviation’s investigatory machinery arguably worked to prevent further accidents with the same failure mode. This is because the evidence not only was technically transparent but also could be understood by experts outside of Boeing.
After the second crash, more than 50 flight jurisdictions grounded their 737 Max fleets, some citing the similarities. Indonesian and Ethiopian investigators were able to match the flights’ near-identical MCAS patterns. Boeing’s concealment soon fell apart under investigations from multiple bodies—in Indonesia, Ethiopia, and the United States. Despite the tragedy of the two crashes, the system worked before more occurred.
Had the crashes involved a frontier model, much of the underlying forensic record would remain at the developer’s discretion to disclose. A developer might undermine its commercial interests if it reveals information about a model it did not want competitors or users to access. And it would leave open the question of which institutions, if any, could act on what that record showed.
Building for Investigation
The reporting regime created by California and its companion laws marks an advance in AI governance. All three states' reporting mechanisms, plus Illinois’s third-party audit, will help regulators collect incident data, allowing them to look for patterns and develop better pictures of their states’ respective AI risk landscapes—as reported by developers themselves. The penalty provision may deter developers from failing to report. These mechanisms do not rely on forensic investigation to increase AI safety. And OpenAI and others’ recent disclosures suggest developers are willing to cooperate, if only partly, on AI safety measures configured around self-reporting and transparency.
But this regime may struggle to deliver safety improvements over time. After MCAS, investigators suggested not only component-level fixes but also revisions to pilot training, the component certification process, and Boeing’s and the FAA’s expectations for pilot performance under pressure. Two accidents led to industry-wide changes.
Several current proposals seek to strengthen independent oversight over AI beyond the status quo. Under Illinois’s SB 315 audit provision, third-party evaluators can access “all materials reasonably necessary,” but developers can restrict that access on security grounds or stipulate on-site reviews. California’s pending bills would create an auditor registry and certify qualified risk assessors, but neither requires developers to undergo audits. In June, Anthropic proposed an independent evaluation ecosystem wherein evaluators receive access to model data and unredacted risk reports—expanding evidentiary access beyond current law—backed by a government agency tasked with enforcing the results. Google and others have meanwhile proposed self-regulatory organization (SRO) schemes modeled on FINRA, in which a private, industry-funded body under government supervision regulates AI.
All come with trade-offs: An SRO could combine expertise hired at industry wages with oversight independence, for example, strengthening regulators’ ability to interpret evidence and sidestepping the issue of government agency independence—but it may also be vulnerable to industry capture.
What they do not address directly are the forensic preconditions outlined here: Developers are currently not required to preserve evidence in any jurisdiction for later reconstruction, nor create an internal unit responsible for preserving that evidence and a developer’s interpretation of it. The evidentiary ball would remain in the developer’s court.
The gap between a robust, aviation-style investigatory regime and current AI regulations is thus wide, whatever oversight architecture emerges. But these forensic preconditions for better regulation need not wait.
Regulators can require that developers preserve what investigators need to reconstruct an accident: model weights, inputs and outputs, relevant logs, and operational conditions. Firms may already possess this data and store it for internal purposes. Regulators can stipulate a proactive data retention provision under which developers retain this data for a specified period. This would preserve the relevant data even when an incident takes time to discover. (The Hugging Face breach was discovered within a few days of its occurrence, but the OpenAI agents that executed it began planning the breach almost two months earlier.) A triggering event (for example, those outlined in SB 53 or its companion laws) would activate a forensic information retention requirement that would give investigators adequate time to access and/or take control of the evidence.
Regulators can also require large frontier developers to form dedicated units for managing this evidence and helping investigators access it. Anthropic, OpenAI, and Google DeepMind already have internal safety officers, teams, or frameworks—though public information about the extent of their authority and independence from management is difficult to parse. A statutory unit would provide investigators with an institutional interface inside the firms they are investigating. Ideally, these units would function independently of management, with protected channels of communication, to prevent conflicts of interest in how they collect and interpret evidence.
If a formal investigatory regime emerges from the laws now on the books or introduced into Congress, it will need to carefully and creatively address AI’s forensic challenges—beginning with the technology’s high levels of opacity compared to other safety-critical domains. The two provisions suggested above would help ensure that both the evidence and expertise needed for such a regime are available when an accident occurs—and hopefully before a major one.
