When AI Models Can Continually Learn, Will Our Regulations Be Able to Keep Up?

Amid all the conversations about generative artificial intelligence (AI) competing with humans on an increasing number of tasks, there is still one hurdle that AI tools have yet to overcome: continual learning. An AI model may be trained on huge amounts of data, but once it is released as a finished product, it does not typically learn and grow as users give it more information or tasks to do. Instead, an AI model is limited to processing a fixed amount of information (its “context window”) within a given session or conversation, and once that session is over nothing is remembered for use in later applications of the tool. 

It turns out this is a major limitation, and one that computer scientists have been working hard to manage or overcome. For example, the coding-assistant company Cursor updates its Tab model multiple times a day, several AI labs have “memory” features that incorporate persistent information about the user into new sessions, and Anthropic’s Claude writes summaries for itself in long sessions to try and preserve some content across time. But no AI lab has released a general-purpose foundation model that can update itself as it acquires new information through everyday use in the way a human brain does.

They are trying, though—and if they succeed, what would that mean for ongoing efforts to regulate AI tools? Continual learning might seem just like a nice product feature, useful for users but not particularly significant from a regulatory perspective. But it turns out that the possibility of AI models that change over time raises some significant challenges to the still-nascent world of AI regulation. In particular, some approaches to AI regulation are built around the idea of an AI model as a finished product, in ways that might not adjust well to models that can change over time. And any approach to regulating AI will have to figure out how to apportion liability between different actors—which continual learning may complicate by blurring lines of responsibility and loosening the connection between AI expertise and AI control.

There will probably be ways of addressing all of these challenges, but they will face a different set of tradeoffs than the world of regulating static AI models, where regulation has already been hard enough. As a result, it is worth anticipating the possibility that AI tools will become ones that can learn, and considering that potential development when assessing policy proposals today. 

When AI Models Learn to Learn

In broad strokes, building an advanced AI model can be divided into two phases. First, there’s the “training” phase, in which a model is trained on a set of information and encodes what it learns in a set of weights that guide its response to future inputs. Then, there’s an execution phase that is sometimes called the “test-time” phase, where the model is deployed to interact with new inputs, like new prompts from users in ChatGPT.

“Continual learning,” which sometimes goes by the name “test-time training,” refers to the capacity of a model to update its encoded information after its initial training phase, based on information presented to it during its use. Computer scientists in the field of machine learning have been working on developing this capacity for years, but they have yet to figure out how to make it work in a robust enough way to integrate it into contemporary large language models. There are clearly strong incentives to figure this problem out.

Because continual learning isn’t a part of frontier AI models’ capacities yet, any predictions about how to address it down the road are necessarily speculative. It may be that test-time training won’t have an effect on AI tools in a way that maps onto major regulatory concerns. Maybe test-time training will just amount to something like better memory or a longer context window, allowing tools to incorporate more information about their users and their needs.

But test-time training could change models in at least a couple of ways that would raise concerns. For one thing, just as existing training methods can create dangerous capacities in models, test-time training could add to a model’s capacities, letting a model do things (like conduct a cyberattack) that it previously could not do. Or, for another, test-time training could change a model’s propensities within its existing capabilities: A developer might remove certain kinds of biases or behaviors in a model, only to have those problems reemerge through training that occurs via a user’s ongoing deployment of that model. Given the strong incentives to develop the technology, and the potential complications it could create for regulatory systems that are just now being built, it is worth thinking through some of the implications that continual learning might have for AI regulation. (Continual learning is also sometimes discussed in the context of AI agents that develop autonomy, but the regulatory challenges of that potential future are a different ball game.)

The Model as a Moving Target

If the development of continual learning does lead to a world where using a model meaningfully changes that model’s capacities or propensities, the first and most obvious challenge would be to regulations that treat models as identifiable products with fixed capacities.

The world of AI models is vast and varied, ranging from small models built by hobbyists to behemoth models at the frontier of our collective technical capacity. Some legislation has approached this variety by applying rules only to certain types of models, such as models that were trained using a certain amount of computing power (a number sometimes used as a proxy for a model’s overall capability). That is the approach taken by provisions in the EU’s AI Act or California’s recently passed SB 53, for instance.

Important limitations to such a model-based approach have already been identified. Test-time training might add one more: if models can gain capacities over time based on how an individual user deploys them, then proxies of a model’s capacity at training time, before it has been deployed by users, will become less reliable. As a result, test-time training will make it harder to identify which models are the ones worth regulating. This is probably unsurprising: if models start developing based on how individual users use them, then there will end up being at least as many models as there are users. That could be the nail in the coffin of model-based approaches to regulation.

But the difficulties of regulating a changing product do not stop there. Any rules or standards that require developers, regulators, or others to study or audit models for various features will become less effective, too. How capable is the new model at assisting a user with building a bioweapon? An audit could tell you where the model’s capacities are before it is sent out into the world. But what happens if that model is used in a pharmaceutical company or university lab in a way that slowly or inadvertently adds to its capacities? Or what if the model’s safety is premised on safeguards installed by a developer that a user is able to permanently erode, whether intentionally or unintentionally? Regulations that tie safety to models’ capabilities at the point of their initial marketing might be unable to address those kinds of developments. 

Who Is on the Hook When Control is Diffused?

In addition to the challenge of figuring out what to regulate in a world with continual learning, there will be new puzzles when it comes to figuring out who to hold liable when things go wrong.

Let’s say a model developer who makes foundation models takes effective precautions to prevent their model from discriminating on the basis of race in a variety of contexts. Then a software company that specializes in insurance tools builds a “wrapper” around that model, retraining it specifically in a way that adds some functionality for customers in the insurance industry. An insurance agent, using the specialized tool over time, unintentionally causes the model to change in ways that result in it discriminating illegally against clients. Who should be liable?

On one hand, the insurance agent is the one who, in some sense, caused the problem—a model that did not discriminate was turned into one that did discriminate because of the agent’s actions. On the other hand, that effect was not intended, and the insurance agent may be much less able to understand and safeguard against these risks than the model developer. Or maybe liability should fall on the intermediary, the company taking a general tool and adapting it to a particular industry with distinctive risks and regulations.

Right now, in a world without test-time training, the people with the most expertise in model development are often the same as the people who create the capacities and safeguards of frontier models: the AI labs. That makes those labs an ideal target for regulation. They are the ones in the best position to look for risks and implement safeguards, because they are creating the models and have the expertise.

Test-time training could erode the connection between knowledge of models and control over models, by allowing the most capable models to be developed and changed in meaningful, permanent ways by users. Almost all users are going to be much less knowledgeable than AI developers—less able to run tests and studies on their models, less informed about the broader technical landscape, and so on. So test-time training shifts some of the control over whether outcomes are good or bad from centralized, knowledgeable actors (the AI labs) to diffuse, less-skilled actors (users). Because developers will have lost some of their control over their models’ capabilities and tendencies, it may be harder to hold them liable for certain outcomes. 

This shift in control also poses a challenge for one of the paradigms that has received more emerging support recently: regulations that take AI companies themselves as the targets of regulation, rather than focusing on regulating those companies’ products directly. As it has become clearer that regulations focused on models themselves have important limitations, thoughtful commentators have advocated for “organization-level” or “entity-based” regulatory approaches that focus on the policies and practices of AI model developers. But if developments in continual learning mean that significant changes to models will happen after they are released to the world, that could undermine the efficacy of regulations that focus on courses of conduct that companies take in earlier stages when the models are first being developed. There are plenty of other reasons to support entity-based approaches to regulation, but continual learning may nonetheless be a challenge for them, should it arise. 

What Might New Regulatory Approaches Look Like?

These challenges don’t necessarily mean that effective regulation will be impossible. But they do suggest different potential paths, with different potential tradeoffs.

There might be some easy answers for some situations. It seems like there is a strong argument that a user who intentionally modifies a model to get it to do something illegal should be on the hook in some way for bad consequences that arise. But how on-the-hook should a model developer be in that context for building a tool that could be modified in that way? And what happens if ex post liability for such intentionally bad acting users doesn’t seem like it will be sufficient, either because those users will be hard to detect, or because they will be judgment proof and insufficiently disincentivized from bad acts?

It seems likely that continual learning would create regulatory pressure in favor of systems for controlling and scrutinizing AI tools as they change over time. If models can add new capacity, especially dangerous new capacity, some may advocate for systems of continuous monitoring. There may also be pressure for regulation to become increasingly attentive to the role of actors downstream from AI developers, up to and including individual users. If a developer creates a foundation model but many different users have their own instantiations of that model, and those instantiations change meaningfully over time, that may lead regulators to want toexamine AI tools at the level of individual instantiations. Or it may cause regulators to want to move upstream, regulating the ways that model developers allow their models to change in the first place.

Just how feasible and burdensome policies like this would be will likely depend on the specifics of the technological, legal, and market realities surrounding continual learning. Will it be possible to regulate that learning in a way to permit certain kinds of changes but not others? How easy or hard will it be for users to add bad capacities or tendencies to models? Will it be possible for users to add harmful capacities unintentionally or unknowingly? These are all questions that may depend heavily on the shape of the underlying technology.

The details of surveillance and enforcement will also depend heavily on the economics and infrastructure of these new types of models (which will in turn be shaped by regulation). Will models continue to be hosted centrally by AI companies, which will keep individualized instantiations with their own weights available for particular users via API or chatbot login? Or will that kind of individualization take place in some way on infrastructure that is more in the user’s control than the AI company’s, like on a user’s personal computer or on a server selected by the user? Or will learning develop in ways that facilitate the growth of intermediaries like the insurance software company discussed above? The answers to these questions are hard to predict in advance, as they likely depend on regulatory and technological details that don’t yet exist. 

But whatever those answers are, they may have serious implications for many facets of the market for AI tools and for users’ experiences. Continuous, individualized monitoring, for instance, could raise real privacy concerns. Restrictions on how models can learn could become a new front in the perpetual struggle to balance the benefits of regulation with the benefits of new technologies. And the potential expansion of liability to users and intermediaries could also have implications for the costs and benefits individuals face when deciding whether to adopt this technology in the first place.

* * * 

Continual learning may never happen. Or it may happen in ways that are benign. But the last few years of AI regulation counsel against the idea that “the technology will never get that good” as a dependable regulatory paradigm. Today, test-time training is more a topic in the worlds of computer science than in law and policy. But as policymakers build legal regimes around AI, it is worth considering how these regimes might or might not adapt well to its development. As with many facets of AI, the challenge to address is how to craft legal frameworks resilient enough to handle a future in which change is common and comes fast.