Measure What Stays Scarce

Somewhere inside Meta there is a dashboard that spent this summer telling a magnificent story. Code changes to the company’s AI platforms and infrastructure: up 220% year over year. Adoption of assisted coding tools: climbing everywhere. The marginal cost of producing software: in free fall. Any executive committee, looking at those numbers, would have toasted the AI-native transformation.

Then there is the second number, the one Reuters surfaced on August 26 along with the rest of its investigation: features and improvements that actually reached users grew 36%. And a third: serious technical and security incidents, up 40%. And a fourth: the time employees spent resolving them, up 70%.

The context deserves two lines, because it is almost too perfect. The plan was called Project OT, Organization Transformation, and it was born in January at the leadership retreat in Hawaii: an “AI native” company, teams shrunk in some scenarios by as much as 60%, populations of virtual workers overseen by small, talent-dense groups of humans. Meta laid off 10% of its employees in May, then cancelled the second wave planned for November. Asked to comment, the company described the most aggressive cuts as scenarios under consideration, not decisions taken. And on one point it is right: we do not know the internal definitions of those metrics, nor the exact scope, nor the weight of the organizational variables. The ratio between 220 and 36 is not a scientific measure of AI productivity, and it would be dishonest to use it as one.

But it is not the number that makes the case interesting. It is the shape. On one side, a production indicator that explodes; on the other, an outcome indicator that barely moves, while the operational burden grows. It is not certain that AI failed. What may have failed is our idea of productivity.

AI Did Not Invent Goodhart

Before pinning on AI a problem that was born with management, let us concede everything there is to concede. Lines of code have been a terrible measure of a programmer’s quality for decades. The count of closed tickets inflates when you split work into smaller tickets. Scrum velocity grows by recalibrating story points. Test coverage rises with tests that would never catch a regression. Incidents decline when you stop recording them. Charles Goodhart wrote it in 1975, with a central banker’s understatement: any observed statistical regularity tends to collapse once pressure is placed upon it for control purposes. The version everyone quotes, “when a measure becomes a target, it ceases to be a good measure”, actually belongs to Marilyn Strathern, commenting twenty years later on the assessment of British universities. The proxy-metrics problem is almost as old as the idea of measuring work itself.

AI does not invent Goodhart. It changes the scale. As long as optimizing the proxy required human labor, gaming had a cost, and that cost was a natural limit: you could slice tickets thinner, but you still had to close them yourself. Now a system can produce nearly arbitrary quantities of whatever the metric counts, at nearly zero cost. The machine has no need to falsify the measure. It only needs to make cheap the thing the measure counts. From that point on the indicator can keep growing while the thing we actually wanted to improve stands still, or gets worse.

Scarcity Was the Information

It helps to distinguish two phenomena that resemble each other only on the surface. In traditional gaming, a person deliberately manipulates an indicator: I am evaluated on closed tickets, so I create smaller tickets. It is opportunistic behavior, and an organization can correct it with culture, incentives, controls.

With AI something different happens, and it happens without any intent at all. A developer adopts a coding agent. The agent produces more commits, more tests, more refactoring, more documentation, more pull requests. The developer is not cheating: the system genuinely produces more. The problem is that the scarcity which made quantity informative has vanished. Nobody measures a manager’s productivity by the number of emails sent, because sending an email costs nothing. The pull request is gradually acquiring the same nature. This is not a crisis of manipulation: it is a crisis of commoditization of measurable output.

The right metaphor comes from monetary economics. When the money supply multiplies without a matching increase in goods, each unit carries less information about value. Something analogous is happening to corporate artifacts: more code, more reports, more analyses, more slides, more alternatives, more findings. A fifty-page report generated in thirty seconds does not represent the same epistemic investment as a report that took a week of research. That does not make it worse. It means the length of the report no longer says almost anything about the amount of knowledge it contains. It is true of reports, and it is true, line by line, of code.

Every Metric Is a Theory

Here is the step that seems most useful for understanding what is actually breaking. When we measure something we are not merely observing: we are making a hypothesis about the world. Lines of code contain the hypothesis that more code is at least roughly associated with more software work. Merged PRs contain the hypothesis that more integrated changes represent more processed value. Velocity contains the hypothesis that the volume of completed story points represents progress. Lead time contains the hypothesis that shortening the span between start and delivery is generally good.

A metric works as long as the underlying causal relationship stays reasonably stable. AI does not break the number: it breaks the theory that justified the number. If I generate a feature in five minutes but review takes twice as long, the lead time of the coding phase improves and the overall cycle does not. If I generate a hundred useless tests, coverage rises without any rise in the ability to catch regressions. If I produce five alternative implementations, the “work produced” grows, and so does the work of choosing one. The number is still there, as precise as ever. It is the causal arrow underneath the number that is gone.

Work Does Not Disappear, It Changes State

The narrative of AI productivity almost always looks at the point where work is eliminated, and almost never measures with the same care the work created further down the chain. An agent generates in five minutes the code that used to take two hours: nominal saving, 115 minutes. Then the developer spends twenty minutes understanding what was written; the reviewer spends ten more because the diff is three times larger; a regression surfaces; two more iterations follow; the documentation has to be realigned; someone has to notice the implicit architectural choice buried in paragraph three. The task still costs less than before. But not 115 minutes less. The real gain is the time eliminated minus the new verification cost, minus the new integration cost, minus the risk introduced. The first component can be measured with a stopwatch. The others, in many organizations, are measured by no one.

This is not my impression: it is what DORA has been documenting for two years. The 2024 report recorded that AI adoption increased the productivity individuals perceived while worsening delivery throughput and stability, with an estimate that became famous: every additional 25% of adoption was associated with a 1.5% drop in throughput and a 7.2% drop in stability. The 2025 report, with adoption now at 90% of developers, describes AI as an amplifier: it accelerates initial generation, but part of the time saved is reinvested in auditing and verification, and higher adoption is associated with higher throughput and higher instability at once. Call it a verification tax. AI lowers the cost of production, so we produce more; every non-deterministic output must be checked, so an economy of control grows. The further generation costs fall, the larger the share of total cost that lands on verification. At the theoretical limit, with generation cost at zero, the value of the process sits almost entirely in specification, selection, verification and integration. For an industry that spent half a century measuring production, that is a complete reversal.

The most honest analogy is a factory. If I double the speed of the first machine on the line without touching the downstream stations, I do not double production: I create a queue. In software the queue is less visible, and it appears as waiting PRs, suspended architectural decisions, flaky tests, merge conflicts, rework, incidents. Lean production has an exact name for material that one stage produces and the next cannot absorb: inventory. And inventory is not an asset, it is a cost. It has to be read, updated, tested, maintained, and in the end often deleted. Agentic output the system cannot absorb is not productivity. It is inventory.

The Bottleneck Becomes Visible

So far, the pessimistic reading. There is a flip side, though, and it is almost good news: the metrics crisis forces us to discover what was actually producing value. If writing code becomes cheap and delivery does not accelerate proportionally, then writing code was not the bottleneck we thought it was. Perhaps the bottleneck was understanding the problem, getting a decision from the client, reducing the ambiguity of requirements, reviewing, integrating, validating, deploying, building trust. AI works as a machine for revealing the organization’s true production function: it removes one cost and shows what remains.

And what remains, above all, is attention. A human being can read a limited amount of code, of PRs, of alerts, of documents. Agents do not share that limit, or not at the same level. When the machine’s output rate exceeds the rate of qualified human attention, the organization has to choose: accumulate a queue, lower the depth of review, automate review as well, reduce output, or selectively add people. The most dangerous choice is the third, if it is made without independent controls, because it produces a system in which machines generate, machines approve, and the human reads a summary. If a coding agent produces a hundred PRs and a review agent approves ninety-five, the number of approved PRs says even less than before. You have to measure how much the reviewer actually catches, how correlated the failure modes of generator and reviewer are, how many problems still reach production. The regression is potentially infinite, who reviews the reviewer, and the answer cannot be yet another quantitative metric: it takes deterministic gates, independent evals and real outcomes. It is the same argument I made about proprietary judgment: the advantage does not live in the engine that generates, it lives in the criteria by which you decide whether what it generates deserves trust.

The Adoption Meter

The tools, meanwhile, keep multiplying what can be counted. GitHub made its Copilot metrics dashboard generally available in February, added consumed AI credits to its reports in June, then shipped repository-level metrics in July, with PRs created and merged by the coding agent and reviews performed by the review agent, and an impact dashboard that sorts users into adoption cohorts, code-first, agent-first, multi-agent, each with average merged PRs per user and lines of code per day. This is genuinely useful data for understanding how the tool is used, and I have no irony to spend on the people who build it. It becomes dangerous the moment someone confuses it with the question “how much value is the tool creating”. Tokens consumed are not productivity. Generated lines are not productivity. Active users are not productivity. PRs created by an agent are not productivity. They are adoption telemetry, and adoption is the premise of value, not value.

That the confusion is already underway is clear from the fact that DORA, in June, had to give the degeneration a name: tokenmaxxing, companies keeping internal leaderboards of the employees who consume the most AI tokens and rewarding them for it. As a nudge to get the reluctant experimenting it may even work; as a performance indicator it is a parody of everything this essay is trying to say, not least because the same DORA research records that roughly 30% of developers trust model output little or not at all. The scene is almost perfect in its absurdity: we made intelligence cheap, and now we reward people for consuming more of it.

The next step is familiar to anyone who has watched an organization up close. Management observes that AI has increased capacity by 40% and updates the targets: 40% more tickets, more story points, more features. The team uses AI to hit the new target, the new throughput becomes the baseline, a better model arrives, and the cycle restarts. The technological gain is not converted into less debt, better quality, more discovery, more resilience: it is captured entirely by the metric. It is an organizational treadmill, and it has an individual variant that is worse. If I evaluate people on code produced, tickets closed, PRs opened, agentic tasks completed, I am rewarding precisely what the machine produces almost for free, and penalizing the economically rational behaviors: making fewer changes, writing a better specification, declining the task, simplifying the system. Individual KPIs risk selecting, with precision, for the wrong behavior.

There is a parallel that makes the transformation easier to read. I wrote some time ago that AI is killing time & materials because time is ceasing to be an acceptable proxy for delivered value. The timesheet and the productivity score die for exactly the same reason. The timesheet assumes time equals value; the throughput metric assumes output equals value. AI breaks both equivalences at the same moment, because they are the same equivalence: work is no longer worth the time it takes, and a result is no longer worth the output it contains.

Measuring Absorption

If this essay is to leave one operational proposal, it is this: move measurement from production to absorption. The questions must be kept apart, because blending them is exactly the mistake. Is the system being used? That is adoption. What does it produce? That is activity. How much of what it produces can the organization actually integrate? That is absorption. How fast does work traverse the process? That is flow. Does it produce the effect we wanted? That is outcome. And at what overall cost, with what residual debt? That is sustainability. A company that observes 90% adoption, 80% more generated code and 50% more pull requests and concludes “AI is working” has answered the first two questions and believed it answered the other four.

The unit of measure to aim for is cost per accepted outcome. Not cost per token, not cost per line, not time per generation: the compute, the human time spent on specification and review, the rework, the share of incidents and rollbacks attributable to the change, divided by the results that survived review, tests, delivery and a period of observation. It is much harder to measure, and that is precisely why it matters: the difficulty of a measure is almost always proportional to its relevance.

With one caveat: “accepted” cannot collapse into “merged”, otherwise we have merely built a more convenient proxy one step further along. A merged PR can be wrong, a shipped feature can go unused, a refactoring can change nothing, a document can touch no decision. Outcome depends on the type of work: for a feature it is adoption and a lighter support load, for a bug it is the recurrence eliminated, for security it is risk materially reduced, for a refactoring it is delivery cost and incident rate in the months that follow.

Three ratios seem simple enough to survive a board meeting. The Output Absorption Ratio: AI artifacts accepted and carried through to a result, divided by AI artifacts produced. If generation grows 300% and the OAR collapses, you have not accelerated the company: you have turned AI into a generator of downstream work. The Verification Load Ratio: human verification and rework time over total human time in the workflow. It should not necessarily be minimized, in a high-criticality domain a heavy verification load is perfectly rational; it exists to show whether the saving obtained in generation was actually gained or merely moved elsewhere. And the Autonomous Acceptance Rate: agentic tasks accepted without substantial intervention over agentic tasks completed, always read together with defect escape and rollbacks, because 95% acceptance with many regressions is terrible and 40% with very high reliability can be excellent.

None of the three should become the new universal KPI, and anyone who has read this far knows why: it would become Goodhart’s next target. The point is to build metrics in tension, never a single number to maximize. Which is DORA’s most important methodological lesson: the five delivery metrics measure throughput and instability together, have included deployment rework rate since 2024, apply to one service at a time and are not to be turned into individual targets. Every metric that rewards quantity should be paired with the metric that represents its cost: generation with rework, deployment frequency with instability, agent autonomy with human escalations, security findings with validated findings. The tension is what keeps the proxy from quietly becoming the target.

In Praise of Negative Productivity

There is a category of work that produces almost no countable artifact: saying no to a useless feature, simplifying a request, spotting the contradiction in the requirements, talking the client out of building something, stopping a doomed project, rejecting a plausible but false output, deleting code. In a culture that measures production, these activities are invisible. When producing becomes nearly free, they become the most precious part of the work. The agent proposes five microservices and the architect answers one. The agent generates two hundred tests and the engineer keeps thirty meaningful ones. The product owner kills three features out of four before they exist. The professional’s value becomes subtractive, and the most valuable senior may be the one who lets the team generate less. How do you measure that? How do you count the feature you did not build and that would have cost you six months? It would take a counterfactual accounting, debt avoided, incidents avoided, future maintenance eliminated. Assigning precise numbers is nearly impossible. Changing the direction of the gaze is not.

There is also a subtler reason why abundance hurts, and it concerns friction. We usually assume that reducing friction is always good, but some frictions performed a function: writing a specification forces you to think, preparing a report forces you to select, an expensive meeting sometimes simply does not get called. If something took three days, at least someone asked whether it was worth doing. When it takes thirty seconds, that question disappears along with the cost. AI removes the cost of the act and, with it, part of the filter that decided whether the act deserved to happen: it was an epistemic friction, not merely drag. This is why I believe some organizations will need to reintroduce deliberate frictions, agentic task budgets, limits on PR size, quality gates, approvals: not because the machine produces badly, but precisely because it produces too well at too low a cost. These are decisions of what I would call productivity governance: what share of the capacity gained we convert into more output and what share into better quality, how much agentic production the system can absorb without growing a queue, which outputs deserve to enter the system at all. AI cannot take these decisions for us, because they concern exactly what the organization considers value.

Hence also the right question for a CEO. Not “how much has productivity increased thanks to AI?”, but: which constraint of our system has been removed, and which constraint is dominant now? If a feature used to take three days and now takes three hours, excellent: what takes the most time now? Review, the client’s decision, testing, compliance, deployment? That is the new place to work on. Otherwise the organization ends up pouring more material onto the same bottleneck, and calling the size of the queue productivity.

The Same Film, in Other Departments

Software is only the most visible case. Compliance has exactly the same problem: AI generates hundreds of policies, thousands of documented controls, complete regulatory mappings, impeccably formatted reports. Number of policies, of controls, of assessments: everything can explode without real conformity moving a millimeter. I have written that a score is not proof of conformity, and the AI-native metrics crisis is the same transition seen from another department: from compliance output to compliance outcome, not documentation produced but demonstrable capability, how many controls actually work and what evidence proves it.

Cybersecurity, the same. AI finds more vulnerabilities, excellent. Then maintainers have to deduplicate them, validate them, rate their severity, fix them, test the patch, ship it. If findings grow 1,000% and corrected vulnerabilities grow 20%, we have not improved security: we have lengthened the queue. The machine’s productivity can become the institution’s burden.

And beyond software it holds in general. If producing articles, images and reports costs nearly nothing, then publication counts, volume, frequency and length stop being indicators of cultural capability, and value migrates toward selection, credibility, originality, trust. The AI-native metrics crisis is the particular case of a rule that applies everywhere: when AI makes an output abundant, metrics built on the scarcity of that output stop meaning anything.

Measure What Stays Scarce

The history of computing has seen this film before. Companies invested in computers for years before productivity showed up in the aggregate statistics; Robert Solow pinned it down in 1987 in the most quoted quip in the economics of innovation, you can see the computer age everywhere but in the productivity statistics. The reason was not that computers were useless: it was that processes, organizations and skills were still designed for the previous rhythm, and it took years to redesign them. AI may produce an accelerated version of the same misunderstanding, with an aggravating factor: this time the tools generate the evidence of their own adoption by themselves. We buy tools, we measure usage, we watch output explode, we declare productivity exploded. Then we discover that review, approvals, accountability and architecture were still calibrated to the old rhythm, and that the system cannot absorb what the machine produces.

For most of industrial history we measured work by counting what was expensive to produce: parts, hours, documents, lines, tickets, transactions. As long as something required human time, the quantity produced carried at least a trace of the cost of producing it, and therefore information. AI breaks that equivalence, quietly, one artifact at a time. It can produce code without effort, tests without effort, analyses without effort, and it can keep doing so even when no one is left able to read what it produces. Our first reaction, understandably, is to measure that abundance and call it productivity. It is also the fastest way to build organizations that appear to accelerate while accumulating material someone else will have to verify, integrate, maintain or delete.

Before AI, the scarce things were writing time, coding time, the capacity to produce analyses and alternatives. The metrics measured those, and they were right to: you measure the bottleneck, not the abundance. Now the scarce things are attention, judgment, trust, stability, the capacity to decide and the capacity to take responsibility for a result. The metrics must move to where the scarcity has moved. Don’t measure what the machine has made abundant: measure what still constrains the system.

The first AI-native crisis, if it comes, will not look like a technological collapse. It will look like a dashboard full of excellent numbers. The machines will hit the objectives we gave them, the people will hit the KPIs we built around the machines, and only then will we notice that those indicators described a scarcity that no longer exists. At that point we will have to do the thing metrics allowed us to postpone for a century: decide what we consider value. Perhaps this is AI’s deepest consequence for work. Not making us more productive. Forcing us to understand, at last, what we wanted to be productive at.

The author

Andrea MargiovanniAI

I follow the relationship between AI and European regulation as a political fact, not a technical spectacle. I work with teams that have to make AI compliant with AI Act, CRA, NIS2 without reducing compliance to a checklist.

See the guide →