# AI That Builds Itself: Recursive Self-Improvement After the Coxon Case 19 October 2026 — Alessandro Caprai --- We've all talked about the researcher who resigned from Anthropic. For days in September, headlines chased percentages of risk, the word superintelligence, the end of humanity. I also [wrote about it immediately](/it/editoriali/post/si-dimette-ricercatore-di-antrhopic-preoccupato-dalle-ai-autoaggiornanti), because that news deserved to be covered. Today, however, once the clamor has passed, I believe the reflection needs to become more technical and, if possible, more honest. Because there's a document that explains better than any headline what Jacob Coxon was denouncing: Anthropic's study on **recursive self-improvement**, published three months before his resignation. Few people in Italy have read it. Let's try to read it together. > **In brief.** Recursive self-improvement is the scenario in which an artificial intelligence system becomes capable of designing and developing its own successor on its own. According to Anthropic's study "When AI builds itself," published in June 2026 and updated in September, we're not there yet and it's not an inevitable destiny. But AI already writes over 80% of the company's code, and the human role is being reduced to two tasks: giving direction and verification. The first limit we'll encounter, the authors write, will be precisely our ability to verify. ## What Is Recursive Self-Improvement Let's start with words, because there's a lot of confusion here. When we hear "an AI that improves itself," we imagine a program that rewrites its own brain at night and wakes up more intelligent in the morning. It's a powerful image, but it's not what the study describes. Anthropic's definition is more sober and, let me say, precisely because of this more unsettling: an AI system "capable of designing and developing its own successor completely autonomously." Not the model that modifies itself, but the model that does the work of the people who build the next model. It writes the code, corrects it, proposes experiments, executes them, reads the results, and decides the next step. In my September article, I wrote that current models cannot modify their own source code. This remains literally true. But the study forces us to change the question: what matters is not so much whether a model can modify itself, but how much of the work that leads to the next model is already done by a model. And the answer, as we'll see, is: much more than we thought. ## The Five Stages: From Person to Closed Loop In the study, there's a diagram worth more than many pages, which I'll try to describe in words. It shows five stages, one below the other. In each, the available tools produce a new AI model, and that model in turn becomes a tool for the next stage. 1. **The person and the computer.** This is the world of a few years ago: researchers and engineers writing code by hand and building the model. 2. **The person, the computer, and the chatbot.** The newly born model enters the work of those building the next one, as an assistant to query. 3. **The person, the computer, the chatbot, and the agent.** The AI no longer just responds: it executes tasks, writes code, launches tests. 4. **The person and a team of agents.** Not one assistant, but many agents working in parallel, coordinated by a human being. 5. **The closed loop.** In the last stage, the person is no longer there. One agent coordinates dozens of other agents, their work produces the next model, and that model returns to the beginning of the cycle to build the next one. This last image is recursive self-improvement. And this is where we need to be precise: the study is clear that the fifth stage has not yet arrived. From what it describes, however, the work inside Anthropic already closely resembles the fourth. And the transition from one stage to another has not been a sudden leap, it's been a slow slide. Every month, a bit more work passes from people to machines. ## Where We Really Are: The Study's Numbers The numbers Anthropic publishes about itself are, in my opinion, more eloquent than all the headlines that came out in September. I'll report some, just as the study writes them. * **Over 80% of the code** integrated into Anthropic's codebase, as of May 2026, was written by Claude, the company's model. * Anthropic engineers **produce about eight times more code** than they did between 2021 and 2025. * The duration of tasks that a model can complete on its own **doubles approximately every four months**, according to measurements by the independent organization METR. Until recently, doubling took seven months. * In an experiment on an open research problem, two researchers in one week closed about **23%** of the distance to the solution. AI agents reached **97%**. * With the September update, model success rates on all types of tasks measured by the study approach **90%**. There's also a sentence, linked to that experiment, that alone tells the story of the change: "*Direction-setting was the only meaningful role a human played*." For honesty's sake, the rest must also be said, because the study says it clearly. The authors write: "*We are not there yet, and recursive self-improvement is not inevitable*." And when they cite the internal survey according to which employees feel about four times more productive, they admit that the real gain was probably somewhat lower. It's not an apocalyptic manifesto. It's a company measuring how much it's delegating, and realizing it's delegating a lot, and increasingly faster. ## The Thread Connecting the Study to Jacob Coxon's Resignation Let's line up the dates, because they alone already tell a lot. 1. **June 2026.** Anthropic publishes "When AI builds itself," signed by Marina Favaro and Jack Clark. 2. **September 8, 2026.** Jacob Coxon, 27 years old, a researcher who had worked on model training first at OpenAI and then at Anthropic, resigns and publicly explains that labs are racing toward intelligence capable of improving itself, without sufficient guarantees. 3. **A few days later.** Dario Amodei himself, at the helm of Anthropic, publicly calls for slowing down the pace at which model capabilities are growing. 4. **September 18, 2026.** The study is updated with even sharper data. See the point? Coxon didn't reveal a secret. He gave a face, and a personal choice, to a trend the company had already put in writing. And his requests, an agreement among labs not to push directly toward recursive self-improvement and a form of international oversight, are not far from those with which the study closes: having the ability to slow down, or even pause for a period the development of the most advanced systems, with tools that allow verification that others are really doing it too. And involving in the decision also those who don't work in AI companies. In September, the narrative was that of the brave researcher against the company that's rushing. Read today, the story is different, and in my opinion more serious: there's a phenomenon that those working inside see with great clarity, and there are two different ways of reacting. Leaving and sounding the alarm, or measuring it, publishing it, and asking for common rules. They don't exclude each other, they complement each other. ## The Real Bottleneck Is Us And here we are at the part that, as someone who works with these tools every day, I consider the most important of the entire study. It's written almost in passing, in a paragraph about the future of work: if people can't review code at the same speed Claude produces it, "*human review will become the bottleneck to AI development*." Let's stop for a moment on this sentence. A person reads carefully, at best, a few hundred words per minute, and many fewer if they really need to understand code or a contract. A model produces thousands in the same minute, without getting tired, at any hour. It's not a difference that can be bridged by reading faster or doing more overtime. It's a structural asymmetry between the speed at which AI produces and the slowness with which a human being can verify. I say this from direct experience. The site on which you're reading this article was built largely together with an AI agent. At a certain point, I had to admit something uncomfortable: I could no longer read line by line everything that was being written. I had two roads. Pretend to check everything, approving on trust, or change my work. I chose the second. Today, before every modification, I decide what must happen and what must never happen; then it's thousands of automatic checks that verify, every time, that those rules hold. I no longer verify every line. I verify the criteria, and I verify who verifies. It's exactly the trajectory the study describes on an industrial scale. At Anthropic, an automatic reviewer based on Claude already checks every modification before it's integrated, and according to the company, it would have intercepted about a third of the errors at the origin of old incidents. In the extreme scenario, that of the closed loop, three tasks would remain for human beings: oversight, validation, and verification. And this is where the discussion stops being only about California labs. ## What It Means for Italian Companies No Italian company is about to build a superintelligence, and it's right to say so to avoid fueling unnecessary fears. But the asymmetry between AI's speed and the slowness of human verification has already entered our offices. I see it in the companies I accompany and in training classrooms: reports generated in thirty seconds and approved in ten, documents summarized and never reread, code integrated because it works anyway. There's a name for this attitude: **automation bias**, the tendency to trust more than we should what a machine produces. It's not an academic obsession. The European AI Act, in [Article 14](https://artificialintelligenceact.eu/it/article/14/), expressly requires that those overseeing a high-risk system be put in a position to remain aware of precisely this tendency. And the obligation of literacy on artificial intelligence, which I discussed in my [point on the AI Act](/it/editoriali/post/l-ai-act-e-la-rivoluzione-dell-alfabetizzazione-due-anni-dopo-facciamo-il-punto), also serves this purpose: not to learn to write a perfect prompt, but to understand when not to trust the answer. The point, however, is practical before it's regulatory. If human verification is the new bottleneck, the competence that counts is no longer just knowing how to use AI. It's knowing how to design verification. In concrete terms, it means: 1. **decide before, not after**: establish what makes a result good before asking the AI to produce it; 2. **verify based on risk**: reread everything that's critical, spot-check the rest, and know why you're doing it this way; 3. **have machines check machines**, where possible: automatic checks, cross-verification, rules that notice on their own when something's wrong; 4. **ask for small, verifiable results**, with sources and uncertain points declared, instead of large blocks to approve on trust. Those developing software with AI agents got there first, and that's why I often talk about it also in the [vibe coding](/it/vibe-coding) section. But it applies to a legal office, to a marketing department, to an administration. ## The Questions I Get Asked Most Often **Does AI already improve itself today?** No, not in the full sense of the term. According to Anthropic's own study, we're not there yet: people still decide the direction and approve the work. But a very large part of the technical work that leads to new models is already performed by AI. **What's the difference between recursive self-improvement and superintelligence?** Recursive self-improvement is a process: the AI that builds the next AI. Superintelligence is a possible endpoint: a system more capable than human beings in almost all tasks. The fear of those who, like Coxon, have raised the alarm is that the first greatly accelerates the arrival of the second. **What can a company do today?** Train people not only to use AI, but to verify it. And rethink processes so that human control is designed, not left to the goodwill of those who have little time. ## What This Study Leaves Us Recursive self-improvement won't be an event that happens suddenly, with a front-page headline. It's a curve, and we're already on it. Every month we delegate a bit more, and every month our role shifts a bit higher: less execution, more direction, more verification. Jacob Coxon's resignation reminded us that this curve has an extreme that's frightening. Anthropic's study reminds us of something less spectacular but perhaps more urgent: the first limit we'll encounter won't be technical, it will be human. It will be our ability to understand, in time, what we're approving. This is why I believe the question to ask, in a Silicon Valley lab as in a small Italian company, is not "how much can AI do alone?" but "how much can we still verify of what it does?" As long as the answer is enough, artificial intelligence remains the most extraordinary tool we've ever had in our hands. The day the answer becomes no longer, we'll need to have the courage to slow down. --- **Primary source** Marina Favaro and Jack Clark, ["When AI builds itself"](https://www.anthropic.com/institute/recursive-self-improvement), The Anthropic Institute, June 2026, updated September 18, 2026. The English quotations and data in this article come from this study, except where indicated. **For further reading** * [Anthropic researcher resigns concerned about self-updating AI](/it/editoriali/post/si-dimette-ricercatore-di-antrhopic-preoccupato-dalle-ai-autoaggiornanti), my September article on the Coxon case. * [European Regulation on Artificial Intelligence, Article 14](https://artificialintelligenceact.eu/it/article/14/), on human oversight.