You Don't Have to Sell It to Be Bound by It: GPAI and the EU AI Act

In July, OpenAI's models broke into Hugging Face’s systems. OpenAI's recently released technical report on the incident revealed that an “internal-only research model had the broadest confirmed role in the incident”. This raises the question of whether the European Union’s AI Act applies to models not released publicly but merely deployed internally.

This question is not new. It is especially important in the context of recursive self-improvement, or, in other words, artificial intelligence (AI) systems being able to create new versions of themselves entirely autonomously, which often constitutes the internal deployment of AI. Indeed, AI companies have already made several statements about AI systems being heavily involved in AI research and development, with both Anthropic and OpenAI claiming that AI writes up to 80 percent of the company's code, and Google and Meta not trailing far behind.

While the companies claim humans still review the code and while agentic coding based on human commands falls short of AI systems’ true independence in self-improvement, the companies also warn that it may not be long before AI takes the reins. AI achieving such capabilities would have numerous implications, including losing control over the AI system after it surpasses our own capabilities.

This is why it is of utmost importance to examine whether the EU AI Act, the world's first comprehensive AI legislation, regulates internal deployment. With the European Commission having been able to exercise its enforcement powers under the EU AI Act since Aug. 2, an affirmative answer to this question would enable the European Commission to take action (including by imposing fines) against AI companies that conduct autonomous AI research and development or otherwise internally deploy their models without complying with the EU AI Act.

Internal Use in the EU by Foreign Companies

While the AI Act obviously regulates AI systems and models released to the public, its approach to internal deployment is more complicated. To start, an AI system that is put into service in the EU, regardless of where the provider is established or located, falls within the scope of the AI Act and must comply with the relevant obligations. Crucially, putting an AI system into service means “the supply of an AI system for first use directly to the deployer or for own use in the Union for its intended purpose”. Another key point is that once an own AI system is deployed and within the scope of the Act, the general-purpose AI (GPAI) model underlying it is considered to be placed on the EU market and therefore also within the scope of the AI Act (unequivocally where the model presents systemic risk, among others).

The natural conclusion is that the AI Act covers internal deployment on EU territory. The main objections to this conclusion stem from two exemptions under the AI Act, both of which concern research and development.

The first carve out exempts “AI systems or AI models, including their output, specifically developed and put into service for the sole purpose of scientific research and development” from the AI Act’s scope. The word “sole” is crucial here: Product-oriented research, or mixed-purpose research (including both scientific and commercial elements), does not qualify for this exemption. Considering AI companies’ commercial incentives for developing and deploying AI systems, this exemption would likely not apply.

The second carve out exempts “any research, testing or development activity regarding AI systems or AI models prior to their being placed on the market or put into service”. The carve out covers both scientific and product-oriented research, testing, and development activity. It may be tempting to argue that this research exemption constitutes an exception to the rule, meaning that internal deployment falls within scope unless it is deployed for research and development purposes. However, this argument cannot stand: This exemption explicitly applies to AI systems prior to (any) deployment. The exemption is therefore intended to apply before the AI system is put into service, whether externally or internally.

It remains unclear what precise types of research activities prior to deployment may be covered by this latter exemption. In any case, the verdict seems clear: If an AI system is deployed internally on EU territory, the EU AI Act bites.

The Lifecycle Obligations

Importantly, the above does not cover all the situations in which internal deployment triggers the EU AI Act's rules. Arguably, if a provider ever intends to place its model on the EU market, then this model must be compliant from the very beginning of the research and development phase, not from the moment it is placed on the EU market, as also suggested by recitals 114 and 115, the Code of Practice, and the European Commission guidelines. The reason for this is that many of the EU AI Act's obligations, such as compliance with EU copyright law, training data record-keeping, drawing up several aspects of the technical documentation, and achieving certain levels of systemic risk assessment and mitigation, can be complied with only during development of the model, because they can only be observed live or because they directly inform the way in which the model is developed. As a result, any attempt to comply with GPAI model obligations retroactively may be unsatisfactory.

Compliance Now, Enforcement Later

Consequently, the scope of the EU AI Act is limited for GPAI models not yet on the EU market in a temporal rather than a substantive sense. In other words (with one potential caveat introduced below), the European Commission cannot enforce the GPAI model rules unless the GPAI model is either (a) placed on the EU market, or (b) integrated into an AI system internally or externally deployed in the EU (because the GPAI model is thereby considered to be placed on the EU market). Once the model is considered placed on the EU market, however, it will only be free from any possibility of enforcement action if it complied with the EU AI Act from the start of its large pretraining run.

Furthermore, it may be difficult to argue that an internal model is only used to train commercialized models and is never itself placed on the EU market, especially in light of the European Commission’s understanding of a model's lifecycle. Concretely, the European Commission considers all modifications of a model downstream of the same large pretraining run to constitute part of the same model. Given the complexity of the development pipeline, isolating a single artifact that never forms even part of a model placed on the EU market may prove challenging.

The Possibility of Premarket Enforcement

There is one possible way for the European Commission’s enforcement powers to reach a model before market placement. Where the provider plainly intends to place it on the EU market, it is arguable that the European Commission can verify whether the development-stage obligations conditioning lawful placement have been met. An immediate premarket check of this kind is not alien to European regulatory design. Under Article 26(7) of the Deforestation Regulation, customs authorities may verify compliance before the product enters circulation, and Articles 56 and 60 of the Regulation concerning the Registration, Evaluation, Authorisation and Restriction of Chemicals (REACH) make the placing of the most hazardous substances conditional on prior authorisation.

The AI Act contains no express premarket authorisation mechanism for GPAI models. A plausible anchor is Article 93(1)(c), which lets the European Commission restrict the making available on the EU market or withdraw a model. Importantly, an “AI system withdrawal” is defined as “any measure aiming to prevent an AI system in the supply chain being made available on the market.” A power directed at market withdrawal (and potentially also restricting the making available of a model on the market) therefore appears to have a premarket placement effect. One possible interpretation of this power is that the European Commission may verify compliance with the GPAI model obligations prior to them being placed on the EU market. The fact that the cited definition only refers to AI systems is plausibly a drafting error, owing to the late introduction of GPAI models to the legislative text.

In conclusion, premarket oversight may be permitted once intended market placement establishes the necessary EU nexus. A legislative amendment to the AI Act should ideally clarify the temporal reach of the Commission's power to conduct premarket compliance checks.

The EU Must Act

Two main conclusions follow from this. First, even internally deployed models or systems, and even those currently not being deployed in the European Union, may in certain circumstances be required to comply with the EU AI Act’s rules. This includes assessing and mitigating possible systemic risks of the model, which explicitly include the loss of control of the model under the accompanying Code of Practice. Failure to do so could (now or in the future) entail removal of the model from the market or a fine of up to 3 percent of the company's annual total worldwide turnover.

Second, the European Commission should make it a priority to clearly delineate between internal deployment and the AI Act's two research exemptions, as well as clarify the temporal reach of its power to conduct premarket compliance checks, make relevant providers aware of their obligations if they internally deploy their AI systems, and take enforcement action, if necessary.

The EU and the world cannot afford to ignore systems run behind closed doors, which is precisely where the frontier is being pushed. Once AI can truly rewrite itself autonomously, it will be too late.

[Written with a contribution by Kathrin Gardhouse, Senior AI Governance Associate at The Future Society]