Advice and Consent for Major Governmental AI Deployments
On March 26, Judge Rita Lin of the U.S. District Court for the Northern District of California granted a preliminary injunction for Anthropic in its ongoing dispute with the Department of Defense over its designation of Anthropic as a supply chain risk. The Anthropic affair has intensified a debate over how law should constrain the executive branch’s use of artificial intelligence (AI). Much of this debate centers on the types of substantive rules at the heart of the Anthropic dispute, such as: Should large-language models (LLMs) be used for lethal autonomous weapons? What rules should govern the use of AI to process intelligence? What exactly do “any lawful use” provisions allow, and who decides?
While Congress should resolve many of these matters, its ability to prospectively regulate governmental AI systems is limited. Developing and deploying AI systems in government settings requires myriad choices, many of which could impact safety, security, and constitutional values. Congress cannot easily anticipate all of these choices, much less decide on rules for them in advance. This creates a dilemma: A complete legislative specification of these design choices will remain impossible, but reliance on the good faith of the executive branch to govern its own AI deployments responsibly is equally unwise.
Additional oversight mechanisms are therefore necessary. This article proposes a new approach: Congress should require the executive branch to secure affirmative congressional approval before deploying AI in certain high-risk domains. Here’s how this could work. Congress would first designate certain governmental domains as “Protected Use Cases”: areas of use where, in its judgment, additional congressional oversight is warranted. Congress would then enact a default rule prohibiting the use of (specific types of) AI for Protected Use Cases.
To overcome this prohibition, the president would be required to submit to Congress a detailed proposal for deploying AI in a Protected Use Case. This would include, for example, a particularized description of the AI system, planned guardrails, authorized and prohibited use cases, authorized users, oversight affordances, and permitted modifications. After back-and-forth negotiations between the branches, Congress would vote on ordinary legislation to provide a specific exception for the proposed deployment, as amended by Congress during the legislative process. Once the bill becomes law, the specific deployment would be approved, but the background prohibition on Protected Use Cases would remain for other, nonapproved AI systems.
The resulting dynamics would loosely mirror the constitutional advice-and-consent process for principal officers, granting Congress an additional mechanism for subjecting powerful governmental decision-makers to public oversight and democratic legitimation of governmental AI systems, without the need to rely entirely on prospective legislation.
Scalable Subordinates
AI systems are increasingly indispensable tools of governance. Many of the central tasks of public administration—collecting and analyzing information, investigating possible offenses, and coordinating the organs of state toward substantive policy goals—seem ripe for augmentation, acceleration, and automation by AI systems.
But with this potential also comes great peril. The executive branch is checked in significant part by its human workforce. Soldiers and civil servants have diverse values and interests that typically diverge from the president’s to some degree. They have an independent obligation to obey the law and uphold the Constitution, while lacking many of the legal protections afforded to the commander in chief. This gives them powerful incentives to resist executive branch abuses.
Replacing human government workers with AI systems could unsettle these dynamics. Without specific requirements to the contrary, governmental AI systems could be built to be unwaveringly obedient. The systems would not fear criminal liability, worry about their post-government reputation, or suffer crises of conscience. They could act in unison at superhuman speeds, meticulously covering their tracks and frustrating attempts to hold them or their principals accountable.
Individual human employees or officers are also sometimes wicked, of course. But the downside risk from any given individual is limited. AI systems, by contrast, are highly scalable: Eventually, an AI system could perform the work of entire bureaucracies.
This structural shift suggests a case for recalibration of Congress’s checking function. One of Congress’s most significant powers is the constitutional requirement that the Senate confirms the executive branch’s principal officers. Limiting this requirement to principal officers was a prudent response to the realities of human administration. If the Senate needed to consent to the appointment of every officer or employee, it would have little time for other business. It therefore made sense to focus senatorial scrutiny on the top of the organizational chart. Indeed, the number of Senate-confirmed positions is arguably already too large.
But if a small number of executive branch AI systems could perform nearly all the work of entire departments, the design of those systems would be at least as necessary as the selection of a department head. This suggests that congressional involvement in the selection and design of those systems could be similarly appropriate.
Red Lines and Gray Areas
A congressional approval requirement makes sense for another reason: It would be unwise for Congress to rely solely on prospective legislation to govern the design, deployment, and use of executive branch AI.
The prospect of a highly automated executive branch has previously motivated me to argue that governmental AI systems should be designed to follow the law. Bright-line rules governing the procurement, deployment, behavior, and instruction of governmental AI systems will be crucial to guarding against their misuse.
But as any law student knows, it is impossible to identify and agree on rules for every imaginable scenario an actor might face. AI systems themselves might reduce the costs of the legislative process, but the complete specification of desirable behavior will always elude us. This presents a significant limitation of our prior proposal to require that governmental AI systems be “law-following AIs.” The law will always have gaps, and clever AIs will be good at finding them. Law-following AI, on its own, cannot prevent AI systems from exploiting such loopholes, because they remain legal by definition.
Some recent trends in frontier AI development vividly illustrate this point. Frontier AI companies rely on documents called “model specifications” or “constitutions” to shape the behavioral propensities of their AI systems. These documents contain bright-line rules, such as “never [p]rovide serious uplift to those seeking to create biological, chemical, nuclear, or radiological weapons with the potential for mass casualties.” But the documents also rely on a sort of Aristotelian virtue ethics that cannot be formalized as legal requirements. In other words, it may be important that executive branch AI systems have good “character” that dictates how they should navigate moral or legal gray areas. While a general legislative definition of “good character” would be unwise and difficult to pin down, assessment of whether a particular AI system has good character mirrors the familiar confirmation hearing process, where holistic assessment of a nominee’s past conduct and disposition is fair game.
The impossibility of a unilaterally specified code for AI behavior is an instance of a more general issue: Safe and socially beneficial AI behavior is the product of a large number of design choices. Congress could not—and should not attempt to—prospectively dictate how the executive branch must make these design choices. Similarly, evaluation of AI systems is far from a mechanistic science: Proper testing involves making a large number of context-specific judgment calls that would elude legislative prespecification. However, assessing claims about specific AI systems is significantly easier. A congressional approval regime would enable such flexible assessments and allow Congress to reject proposed deployments on the basis of considerations that could never have been operationalized in prospective legislation.
Statutory Advice and Consent
To enact this policy, Congress would first need to define the set of executive branch deployments that require specific congressional approval. These are the Protected Use Cases. Careful demarcation of the Protected Use Cases is critical: Too narrow a scope could enable executive-branch misfeasance, while overbreadth would hamper state capacity.
Congressional preapproval is most important for systems that will exercise certain functions that pose a significant risk to the constitutional order, and for which after-the-fact remedies are limited. Imagine, for example, an AI system that integrates existing sources of lawfully collected information across multiple agencies, including purchased data, open-source information from the internet, intelligence sources, public security camera footage, and information already reported to the government (such as mortgage applications). The AI system then identifies possible crimes, obtains warrants for further information, and decides whether to recommend them for prosecution.
It seems likely that such a system would identify thousands of previously unidentified crimes. Perhaps, in isolation, this could be a valuable way for the government to catch more criminals. But the opportunities for abuse are obvious. Absent technical safeguards, it would be easy for a malicious government to use such a system to scrutinize political opponents systematically while ignoring political allies. Given the difficulty of successfully arguing a defense of selective or vindictive prosecution, this could be a powerful tool for political repression.
A full enumeration of candidate Protected Use Cases is beyond the scope of this piece. However, uses that might warrant inclusion include criminal investigations, domestic intelligence-gathering and analysis, domestic military deployments, and prosecutions. While this would include a wide range of governmental functions, it would also leave many core functions unimpeded: The work of many federal agencies could be entirely exempted.
Congress would then establish by statute that AI systems could not be used in a Protected Use Case without explicit statutory authorization. This basic structure is similar to familiar statutes such as the Posse Comitatus Act, which prohibits the use of regular military forces for domestic law enforcement without express congressional or constitutional authorization.
Congress should then statutorily preapprove certain deployments within the Protected Use Cases, meaning additional congressional approval would not be necessary. At least two categories of preapproval seem prudent. The first set of preapprovals would be for classes of AI systems that seem unlikely to pose the severe risks to liberty that motivate this proposal. One candidate class would be so-called narrow AI systems. While there are serious risks associated with these technologies, their adoption seems unlikely to displace human civil servants to a destabilizing degree.
Congress should also statutorily preapprove testing and development of AI systems for Protected Use Cases. This statutory scheme depends on the executive’s ability to propose well-specified deployments to Congress for approval. Departments and agencies cannot do this unless they can develop and test AI systems for Protected Use Cases without prior congressional approval. Of course, it would be important to carefully define “testing and development” so that it could not be abused to circumvent the general preapproval requirement. However, this is a manageable legislative drafting task.
Any governmental deployment of AI in a Protected Use Case not covered by a preapproval provision would be prohibited by default. Federal employees who willfully violate this prohibition would be subject to civil and criminal penalties, as would vendors who willfully aid in illegal deployment. This enforcement mechanism would parallel (and supplement) the Antideficiency Act, which prohibits federal employees from expending federal resources in excess of statutory appropriations. The executive branch would therefore need congressional approval for deployments for Protected Use Cases not covered by statutory preapprovals. Here’s how I envision that process working. Congress would give the executive branch a generous budget to develop and test prototypes for Protected Use Cases. This generous funding would help entice vendors to participate in prototype development, notwithstanding the prospect of Congress rejecting a proposed deployment. And the executive branch, cognizant of the need for congressional approval, would conduct the development and testing phase to maximize the likelihood that Congress signs off. They would, for example, brief the relevant committees about development progress and seek input on which tests should be conducted.
When the time is right, the executive branch would submit to Congress a report for proposed deployment of an AI system for a Protected Use Case. Obviously, since the approval requirement and process are entirely statutory, Congress cannot bindingly dictate the scope of future congressional approvals. But the policy goals of this process would be best advanced if the proposed deployment were fairly specific. It should include, for example, a technical description of the system being deployed, such as the AI models that power it and any accompanying technical safeguards. It should also include the results from the testing process.
But the proposed deployment need not be limited to technical information. For example, Congress might establish accompanying rules regarding who is allowed to operate the system and the purposes for which it may be used. It might also insist on reporting requirements, which could possibly be automated by other AI systems. Indeed, the design space for a proposed deployment would be large and multidimensional: Congress could create technical, legal, informational, and organizational safeguards that are tailored to the risks the system poses. In particularly risky cases, they could, for example, make liberal use of sunset provisions, impose demanding reporting requirements, and provide that failure to comply with any of the required safeguards voids the approval entirely.
However, it is equally important for Congress to enable the continuous development of governmental technologies. To facilitate this, Congress could explicitly greenlight certain modifications to an approved deployment while explicitly prohibiting others. Changes to maintain core functionality, improve user interfaces, and bolster cybersecurity could be allowed expressly, while adaptation of the software for new, riskier use cases could be expressly prohibited. The scale of the deployment could also be carefully managed, with Congress expanding the permitted scope of deployment over time as the product matured and Congress gained more confidence in it.
After back and forth negotiations and modifications, Congress would authorize the proposed deployment—including required safeguards and permitted alterations—through ordinary legislation. Once signed into law (or enacted over a presidential veto), the proposed deployment would be authorized. Congress would then use its oversight functions (both those specifically mandated in the proposed deployment and its general oversight powers) to ensure that the government did not exceed the scope of its authorization.
Congress Should Do Its Job
Perhaps the most compelling objection to this proposal is that it tasks Congress with forming nuanced views about the safety and security of proposed AI deployments. At a time when Congress is less popular than ever and hobbled by gridlock, is trusting Congress with more responsibility for complex technical matters really wise? More pointedly: Wouldn’t the dysfunctional and highly polarized nature of the legislative process risk kneecapping state capacity?
These are all reasonable concerns. Yet it’s difficult to imagine how the technological transformation of the executive branch could be anything other than disastrous without more aggressive congressional oversight. The Madisonian vision of legislative ambition checking presidential ambition has not played out as the framers imagined, but it is nevertheless an indispensable part of our system.
Item-specific control over executive branch resourcing decisions may not be the norm, but it has precedent. Early Congresses “exercised fine-grained control over all funding decisions,” including in military matters, down to “the precise numbers of troops [and] their alotted [sic] daily rations.” Today, Congress is still intimately involved in major procurement decisions, particularly in defense. Congressional oversight of specific AI procurements could be similarly critical.
If we begin to rely on Congress to evaluate executive branch technology, we should provide it with the necessary resources. It is cliché to argue that Congress should revive (something like) the Office of Technology Assessment (OTA), the now-defunded agency that assessed the likely impacts of emerging technologies for Congress. But it remains true: Congress will need in-house expertise to help it understand the technologies the executive branch requests. While the Government Accountability Office and Congressional Research Service make admirable efforts to provide Congress with reliable information about AI and other emerging technologies, the imagined policy here will require a dedicated organization for AI assessment. Nor, for separation of powers reasons, would it be wise for Congress to depend on information provided from executive branch bodies such as the Center for AI Standards and Innovation or USAi. Congress needs a body it alone controls and can trust. For now, let’s call it the Congressional AI Research Office (CAIRO).
CAIRO could act as a hybrid between a congressional support agency (such as the OTA) and committee staff. Similar to congressional support agencies, CAIRO should have a deep, standing bench of technological experts. But like congressional committee staff, CAIRO should be explicitly divided into majority and minority staff, with some shared nonpartisan staff for work with less partisan valence. This would enable each party to receive trusted input on the issues they care about. And similar to committee staff, the majority party should be entitled to more funding, but with significant protections for the minority party. This balances the political incentives of the majority party with the ability to retain a stable core of experts across Congresses.
CAIRO staff would analyze proposed deployments for their respective principals. To do this, each side of CAIRO would need to develop extensive in-house expertise on AI systems. This would include not just measuring AI models’ capabilities and propensities but also having the ability to systematically assess all aspects of a proposed AI system, including the development process and other system-level characteristics that may affect the product’s functionality and security.
CAIRO staff might develop and perform their own evaluations but would also need to rely heavily on evaluations from third parties and the executive branch. Accordingly, the staff would need to be able to interrogate the scientific validity and integrity of those external evaluations. And it could also serve as a secure staging environment for members to interact with AI system prototypes or demonstrations. It would also include cleared staff and secure computing environments to assess classified proposed deployments.
These resources are unlikely to wake Congress from its current slumber. But they could represent a necessary first step toward reestablishing a healthier interbranch dynamic.
* * *
The future of U.S. state capacity requires ambitious government deployment of AI, and American liberty requires careful regulation of that deployment. Managing this tension will be a central task in the coming decades.
While no single mechanism will suffice, active congressional oversight of the executive branch’s AI use will remain indispensable. Given the speed and scale at which AI can act, prospective approval will be at least as important as retrospective review. Before the executive branch is allowed to use AI for its most significant functions—those that threaten to deprive the people of their life, liberty, and property—the specific consent of the people’s representatives should be obtained.
