AI and Secure Code Generation

At the end of 2024, 25 percent of new code at Google was being written not by humans, but by generative large language models (LLMs)—a practice known as “vibe coding.” While the name may sound silly, vibe coding is a tectonic shift in the way software is built. Indeed, the quality of LLMs themselves is improving at a rapid pace in every dimension we can measure—and many we can’t. This rapid automation is transforming software engineering on two fronts simultaneously: Artificial intelligence (AI) is not only writing new code; it is also beginning to analyze, debug, and reason about existing human-written code.

As a result, traditional ways of evaluating security—counting bugs, reviewing code, and tracing human intent—are becoming obsolete. AI experts no longer know if AI-generated code is safer, riskier, or simply vulnerable in different ways than human-written code. We must ask: Do AIs write code with more bugs, fewer bugs, or entirely new categories of bugs? And can AIs reliably discover vulnerabilities in legacy code that human reviewers miss—or overlook flaws humans find obvious? Whatever the answer, AI will never again be as inexperienced at code security analysis as it is today. And as is typical with information security, we are leaping into the future without useful metrics to measure position or velocity.

Finding and Fixing Bugs

People often think of writing software as an iterative process; one might say it is composed of successive experiments. You write code, run the code, or—in a hacker’s case—read the code to find bugs. Then you fix the bugs with more code (or write an exploit). This creates another question: Is the AI good at looking at code to find bugs?

At some level, identifying previously unknown security flaws—known in the industry as zero-day vulnerabilities—in widely deployed, mature software is the one true unsaturated evaluation left for LLMs. A zero-day is a hidden flaw that attackers know about but defenders don’t, and it remains exploitable until discovered and patched. The world has a vast amount of existing software code, written over the past quarter-century, quietly powering critical infrastructure and personal devices worldwide. Uncovering these latent vulnerabilities has spawned an entire ecosystem—a digital Cold War—with technicians racing to either exploit or fix these hidden weaknesses. For example, when Microsoft disclosed the Exchange ProxyLogon vulnerabilities in 2021, defensive and offensive teams worldwide raced to patch or exploit them first, illustrating how quickly such vulnerabilities become focal points of global cybersecurity conflict. In this case, one could argue that the ecosystem of offensive teams got the jump on the defensive teams, and American businesses largely paid the price.

Information security technicians have a particular understanding built from years in the trenches: Whoever knows the system best, controls it. In this context, whether AI is a static pattern matcher or a machine that truly understands the code itself is of world-shaking consequence.

The top of the pyramid of unknowns is whether software bugs are sparse or dense. If they are sparse, then the time and treasure to find and fix them is well spent. If they are dense, then finding and fixing them wastes both time and treasure. Six take away one, for example, is a 15 percent improvement, but 6,000 take away one is unarguably irrelevant. Yet this may be a moot point if AI security tools begin to neutralize entire classes of vulnerabilities—not by eliminating the need for human expertise, but by shifting where that expertise is applied.

Increasingly, the work isn’t about just finding bugs, but understanding their significance in a vastly more complex, AI-shaped landscape. The center of gravity is moving: from flaw hunting to impact analysis, from static review to dynamic reasoning, from code inspection to system behavior. As AI takes on more of the search, humans may be tasked with the harder problem—deciding which bugs actually matter.

For example, LLMs can excel at rapidly uncovering vulnerabilities in the open-source libraries underlying modern software supply chains. But determining whether those vulnerabilities can realistically be exploited requires distinctly human expertise. Humans must assess if attacker-controlled data can reach the vulnerable code, consider the specific platforms and configurations involved, and carefully evaluate whether built-in security measures render potential threats irrelevant. Such nuanced analysis remains firmly beyond current AI capabilities.

There are many roadblocks preventing LLMs from eclipsing even an average human’s skill at reading code to find bugs. A significant challenge is their limited context size—they can reason over only so much information at once, typically equivalent to a few lengthy novels. While that may seem substantial, comprehending a large software system requires systematically maintaining context across many layers of abstraction, dependencies, and interactions. Current “reasoning models” manage this by breaking complex problems into smaller steps, iteratively exploring different reasoning paths, and generating intermediate summaries or metadata to extend their effective reach. But even these incremental reasoning approaches are still early stage—they can mimic focused human reasoning, but they don’t yet replicate the kind of deep, holistic understanding needed to fully analyze large-scale software systems.

Currently, humanity’s approach to measuring the capabilities of LLMs in identifying and addressing security bugs reveals more about our own methodological limitations than the strengths or weaknesses of AI. We remain stuck in simplistic frameworks: Should we treat AI purely as classifiers, marking code functions merely as “vulnerable” or “safe”? Should we expect models to pinpoint the exact lines of problematic code, or acknowledge that bugs might not reside neatly within any single line at all but, instead, emerge unpredictably from subtle interactions and implicit state conditions woven invisibly throughout the program (what Sergey Bratus and colleagues deemed a “weird machine”—functionality in the code that did not come from anywhere)?

Consider the Spectre vulnerability revealed in 2018: This flaw wasn’t a matter of identifying a single faulty line of code but, rather, understanding how multiple correct, seemingly innocuous software and hardware interactions combined to allow attackers to leak sensitive information through “speculative execution,” the chip designer’s way of trying to presume what the code was about to do.

Perhaps, rather than asking an LLM to pinpoint a specific faulty line, we should be challenging it to produce inputs capable of defeating parsers or circumventing logic—probing whether it grasps deeper, systemic vulnerabilities. Eric S. Raymond famously argued that “many eyes make all bugs shallow,” but even he didn’t foresee those eyes belonging not to insightful human collaborators but, instead, to the relentless gaze of artificial intelligence.

This ambiguity signals not just uncertainty, but a fundamental shift in our understanding of how vulnerabilities emerge, manifest, and persist within software systems. It’s clear that we cannot yet fully interpret the cognitive processes of these rapidly evolving AI models. But acknowledging this limitation doesn't diminish the inevitability that LLMs will increasingly influence software security practices. We might reasonably anticipate that the next evolution in AI capabilities will involve teaching models not merely how to classify code or locate bugs, but how to debug proactively, or even to autonomously generate robust specifications, accurate threat models, and detailed data-flow diagrams—all essential elements previously confined to human practitioners. In other words, as AI moves from assisting humans to acting autonomously, the nature of security itself shifts.

From N-Day to Every-Day Vulnerabilities

Historically, software vulnerabilities that were already discovered, documented, and patched—so-called N-days—faded quietly into the background. Patches were issued, release notes were published (often minimally), and life moved on, largely invisible to the wider world. But AI-driven code analysis is changing that dynamic. As LLMs and automated code-scanning tools become widespread, previously obscure vulnerability fixes will become immediately and transparently visible. A security patch that once quietly passed unnoticed—such as OpenAI’s recent discovery of a subtle but ultimately non-exploitable bug in OpenSSH—now rapidly attracts attention, scrutiny, and automated analysis by both defensive teams and adversaries. Even when harmless in isolation, these findings illustrate how quickly AI-driven code analysis can publicly surface vulnerabilities, amplifying their visibility and reshaping the vulnerability disclosure landscape.

This transparency has deep implications. Security teams accustomed to slow-moving disclosure processes and controlled vulnerability releases will suddenly face an environment of immediate exposure and urgency. Likewise, policymakers and institutions accustomed to treating N-days as safely resolved problems will discover they’ve become active battlefields. This new normal—where every vulnerability patch is effectively a public disclosure—demands new norms, new institutional agility, and an entirely new approach to managing vulnerability information.

Like all technological advances, the increased transparency around N-day vulnerabilities cuts both ways: Defensive teams gain better insights, but adversaries, now armed with AI-driven reverse engineering tools, can equally capitalize on rapid disclosures.

Autonomy Means Losing the Plot

AI experts have spent decades telling themselves that software security is about clarity: Understand your threat model, map your attack surface, and tame complexity into submission. Autonomy challenges that premise. Once AI-driven systems start independently defining their own objectives, rewriting their own code, and deciding what threats matter, clarity goes out the window. We won’t just lose track of the code; we’ll lose track of why the code even exists.

Previous work from Dan Geer and Glenn Gaffney warned us that autonomy inevitably produces emergent behaviors, unexpected outcomes that emerge unpredictably from complexity. But here’s the next leap forward: We’re now entering territory where even the threat models themselves become opaque. It’s not just “Why did the AI fix that bug?” It’s “Why does the AI think that’s a bug in the first place?”

In short, autonomy means losing the plot. Humans are about to find themselves on the outside looking in, facing security decisions made by AIs whose reasoning isn’t just complicated—it’s fundamentally inaccessible.

Hazards and Risks

When discussing software security, experts often overlook a crucial distinction: A vulnerability existing in the code doesn’t automatically mean it poses an immediate threat. For example, a buffer overflow—a flaw allowing memory to be overwritten—is a hazard, but it becomes a genuine risk only if an attacker can reliably exploit it to take control of a machine. Put another way, bugs themselves are abundant, but the ability and expertise needed to transform a bug into a successful attack are relatively rare—at least for now.

However, many current approaches oversimplify this distinction. Security tools frequently label code functions simply as “vulnerable” if they match patterns seen in past exploits, or “safe” if they don’t. This binary approach treats potential hazards as immediate risks, masking the deeper complexity of real-world software interactions.

Consider a common example: A web server might contain vulnerable code that, in isolation, looks like a serious flaw. But if there’s no practical way for attackers to deliver malicious data to that code—because of network configurations, application logic, or other protective layers—then the vulnerability poses no real-world risk. Our current methods rarely account for these nuanced scenarios, resulting in an incomplete and misleading picture of security.

Even children learn early to point and name—but knowing the word “dog” doesn’t reveal whether the animal might bite. In cybersecurity, we’ve built systems that similarly point and name vulnerabilities without understanding whether they’re truly dangerous. By embracing AI solely for pattern recognition, we’ve created a powerful “pointing machine” that identifies possible threats but does not comprehend their actual impact. What we need instead is a “knowing machine,” capable of understanding how code functions within complex, real-world environments, recognizing not just hazards but the full context of how and whether those hazards might become genuine risks.

Large language models offer a promising step in this direction. Unlike traditional static and dynamic code analysis tools, these models can simulate code execution more holistically, envisioning realistic usage scenarios and potential attacker behaviors. In many cases, they can foresee and address potential problems more effectively, rapidly, and inexpensively than humans alone. Leveraging these models properly could bridge the gap between abstract vulnerabilities and meaningful security—helping us move from merely pointing at problems to truly understanding and mitigating them.

Charting a New Security Landscape

Current cybersecurity frameworks—such as the National Institute of Standards and Technology’s Secure Software Development Framework or Open Worldwide Application Security Project guidelines—assume predictable human coders and defined accountability. These assumptions collapse beneath the weight of AI’s velocity, opacity, and autonomy. If the government intends to catch up, it must shift from periodic audits to continuous AI-powered monitoring, immediately flagging anomalies as they emerge. Provenance standards—akin to a “software bill of materials” but tracking models, prompts, and datasets—would be very useful for auditing the security of largely AI-written codebases.

Moreover, regulators will need dynamic, risk-based frameworks tailored explicitly to AI-generated code, applying stricter oversight to high-impact systems. This demands infrastructure that instantly processes AI-discovered vulnerabilities, rapidly assesses their exploitability, and automates defensive measures previously reliant on human pace.

Finally, and perhaps most urgently, government cybersecurity teams must swiftly become literate in AI-specific security. Adaptation isn’t optional—it’s overdue. AI-generated software is already reshaping our threat landscape, and institutional inertia is a luxury we can no longer afford.

Building Resilience in an Age of Autonomy

Understanding the implications of the wave about to hit the security space compels AI experts to ask whether we have grown complacent—whether our models and methods, evolved incrementally over decades, are fit to handle a paradigm shift driven by forces we barely comprehend. A state of security is the absence of unmitigable surprise—not no surprise, but no surprise without a workable mitigation. Yet we are now faced with uncertainty that traditional tools cannot dispel. To ensure that systems do not silently fail demands not just a pointing machine, but a knowing machine—one that understands not just hazards but the cascading complexities of risks, the subtle interdependencies hidden in plain sight, and the intricacies of code in execution.

To truly bend AI to our purpose means embracing a radical humility: recognizing that as we teach these machines, these machines can also teach us. In doing so, we cross a threshold where software security is no longer merely about finding bugs or fixing flaws but about jointly navigating a future defined not by the vulnerabilities we fear but by the resilience we build.

Building that resilience means asking questions that welcome rather than fear the autonomy and potential of AI: How can institutions evolve to embrace and guide increasingly capable AI-driven decision-making? What new regulatory approaches can harness AI’s unprecedented ability to identify and neutralize vulnerabilities before they are exploited, even crafting mitigations for surprises not yet seen? And ultimately, how might our norms and policies encourage collaborative innovation, ensuring we don’t merely cope with AI’s complexity, but thrive within it?