What Is the Jellyfish AI Engineering Benchmark — and Why Should You Care?

The Jellyfish AI Engineering Benchmark is the largest continuous dataset of real-world AI coding behavior ever assembled: as of early 2026 it covers 1,000+ companies, 200,000 engineers, and 37 million pull requests analyzed over rolling quarters. Unlike survey-based studies that capture developer sentiment, Jellyfish pulls instrumented telemetry — actual PRs merged, code churn rates, token consumption logs, and review cycles — making it a ground-truth view of what AI coding tools actually produce rather than what developers believe they produce. The benchmark is updated quarterly and published at jellyfish.co/ai-engineering-trends.

What makes this benchmark unusually credible is the scale of the signal. Prior studies of AI coding productivity — including GitHub Copilot’s own research — relied on controlled experiments with hundreds of participants. Jellyfish’s dataset captures organic adoption across the industry’s full distribution: early adopters, laggards, mid-market teams, and Fortune 500 engineering organizations. That distribution exposes efficiency curves, plateau effects, and adoption divides that controlled experiments systematically miss. For engineering leaders making decisions about AI tool budgets in 2026, this dataset is the closest thing to a definitive industry baseline. The core finding — that productivity gains from AI are real but non-linear, and that token consumption is a poor proxy for business output — is reshaping how forward-thinking companies structure their AI ROI measurement.

The Tokenmaxxing Trap: When Token Consumption Became the Wrong Metric

Tokenmaxxing refers to the organizational behavior of treating AI token consumption as a proxy for developer productivity — rewarding engineers who burn the most tokens and structuring team incentives around AI usage volume rather than business outcomes. The term was popularized after a TechCrunch report revealed that Meta had ranked 85,000 employees on a token consumption leaderboard, with a “Token Legend” tier awarded to one engineer who consumed 281 billion tokens in 30 days. Uber reportedly exhausted its entire annual AI budget of $3.4 billion in four months. These are not isolated incidents — they are the predictable result of misaligned incentives at scale.

The Jellyfish Q1 2026 study of 7,548 engineers quantifies why tokenmaxxing is dangerous. The top 20% of token consumers averaged 23 merged PRs per quarter versus 11 for the bottom 20% — a genuine productivity signal. But the cost-per-PR in the top cohort was dramatically higher. Typical developers consumed roughly 51 million tokens per month; 90th percentile users consumed about 380 million tokens per month, a 7x multiplier. Per-developer token consumption rose approximately 18.6x in nine months, driven primarily by agentic tools like Claude Code and Cursor’s agent mode that run multi-step workflows without human checkpointing. The critical issue is that the productivity gains are real but sublinear: doubling tokens does not double output, and at the high end of the consumption curve the cost-per-PR climbs faster than throughput. When companies optimize for the metric (token volume) rather than the outcome (features shipped, defects reduced), they get exactly the metric — and miss the outcome.

Jellyfish Data Breakdown: The Token Efficiency Curve in Practice

The token efficiency curve describes the relationship between per-developer AI token consumption and productive output (measured in merged PRs, features shipped, or defects resolved). Jellyfish’s data shows a clear three-zone structure: a steep gains zone at low-to-moderate consumption (roughly 10M–80M tokens/month per developer), a diminishing returns zone in the 80M–200M range, and a cost escalation zone above 200M tokens/month where incremental productivity gains approach zero while costs continue climbing linearly.

In numerical terms from the Jellyfish Q1 2026 dataset: the bottom 20% of token spenders (below ~20M tokens/month) averaged 11 merged PRs per quarter. Moving from this baseline to moderate usage (20M–80M tokens/month) produces the clearest productivity signal — this cohort accounts for most of the genuine AI productivity gains visible in aggregated industry data. The top 20% (above ~150M tokens/month) averaged 23 PRs — twice the output — but the token cost to reach that 2x throughput was approximately 10x the token spend of low users. Agentic tools are the primary driver of this cost escalation: when an engineer delegates a full feature to Claude Code or a similar agent, the agent may consume 50–200M tokens in a single session to explore approaches, generate code, run tests, and revise based on failures. The business calculus only favors this expenditure if the resulting PR would have taken the developer three or more days to write manually. For routine tasks, agent-driven token consumption frequently exceeds the cost of the human-hours saved — a break-even that most teams are not currently measuring.

Harvard + Jellyfish: AI Makes Developers Faster — But Features Shipped Haven’t Moved

A joint study between Harvard Economics and Jellyfish analyzed 100,000 software engineers across 500 companies and reached an uncomfortable conclusion: AI is making individual developers measurably faster, but the organizations they work for are not shipping significantly more features. The individual-level gains are real and consistent — developers using AI coding tools complete assigned tasks faster, write more code per hour, and report higher confidence in their output. The organizational-level outcome, however, shows no statistically significant increase in features shipped or business deliverables completed in the same period.

This individual-to-organizational gap has a name in productivity research: the “Jevons paradox” of software development. When a task becomes faster, the time saved does not automatically convert into more output — it often converts into more thorough code review, more exploratory coding, longer AI prompt iteration cycles, or simply more meetings. Jellyfish’s instrumented data provides a concrete mechanism: 64% of developers report achieving at least a 25% productivity increase using AI tools, and they are not wrong about the speed of the tasks they’re tracking. But the tasks they track — individual coding sessions — are a fraction of the total engineering workflow. Planning, architecture, code review, debugging AI-generated errors, stakeholder communication, and deployment remain largely unaffected by AI assistance, which means the denominator of total engineering time is only partially compressed by AI speed gains. The Harvard-Jellyfish findings are consistent with the GitHub Copilot controlled experiment results from 2023: isolated task completion speed improves, but end-to-end feature delivery time does not improve at the same rate, because coding is rarely the bottleneck.

Code Churn: The Silent Productivity Killer Hidden in AI-Assisted Code

Code churn — the rate at which recently written code is revised, deleted, or reverted — is one of the most reliable leading indicators of technical debt accumulation and future maintenance cost. Jellyfish’s 2026 benchmark data shows a 5–11% increase in reverted code across teams with high AI adoption, a figure that understates the true churn effect because it only captures outright reverts, not substantive revisions within the same PR window. A separate Faros AI study measured an 861% increase in code churn under high AI adoption conditions. GitClear’s January 2026 analysis found that regular AI users experienced 9.4x higher code churn rates than their non-AI counterparts.

The mechanism is consistent across studies: AI coding tools are optimized for local coherence (the code compiles, the tests pass, the immediate requirement is satisfied) rather than systemic fit (the code integrates cleanly with existing architecture, follows team conventions, and remains maintainable). When AI generates code that passes immediate review but violates deeper architectural invariants, the churn materializes weeks or months later when the invariants are exposed — often during a different engineer’s feature work. The code acceptance rate problem amplifies this: AI tools report 80–90% acceptance rates in their own telemetry, but real-world post-revision acceptance (the rate at which AI-generated code survives two weeks without substantive modification) drops to 10–30%. Engineers who trust the high claimed acceptance rate and reduce their review rigor are accepting a churn liability. The engineering leaders who are handling this correctly are treating code churn rate as a first-class metric alongside PR throughput — refusing to declare AI productivity a success unless churn rates remain flat or decline.

The AI Adoption Divide — and What Separates Top Quartile Teams

The Jellyfish 37 million PRs benchmark reveals a widening gap between top and median AI adopters that is not primarily explained by which tools teams use, but by how they use them. Median AI adoption across Jellyfish’s dataset is 71% of developers using AI tools with a 27% AI code ratio (share of merged code that originated from AI suggestions). Top AI adopters achieve 2x PR throughput compared to low adopters and have autonomous agent PRs reaching 35% of all merged PRs — a signal that these teams have successfully integrated agentic workflows into production engineering, not just individual developer assistance.

What separates top quartile teams is a cluster of operational practices rather than tool selection. First, outcome instrumentation: top teams measure cost-per-PR, churn rate by AI cohort, and feature delivery time — not just token consumption or developer satisfaction. Second, workflow integration depth: rather than adding AI tools as standalone chat interfaces, top adopters have embedded AI into CI/CD pipelines, PR review automation, and structured planning workflows where the AI’s context is pre-loaded with architectural constraints. Third, human review calibration: top teams maintain or increase code review rigor for AI-generated code rather than relaxing it, which suppresses churn at the cost of some raw throughput. Fourth, task routing: high-performing teams are deliberate about which tasks are appropriate for full agentic delegation (greenfield feature scaffolding, test generation, documentation) versus which require human-led development with AI assistance (security-sensitive code, architectural decisions, cross-system integrations). Teams that apply agentic tools indiscriminately consume more tokens and produce higher churn than teams with explicit task-routing policies.

How to Measure Real AI Coding Productivity (Not Just Token Volume)

Real AI coding productivity measurement requires a dashboard that tracks both the speed dimension (what AI tool vendors optimize for) and the quality-and-cost dimension (what determines long-term ROI). Based on Jellyfish’s benchmark methodology and the Harvard-Jellyfish joint study, effective measurement frameworks include five categories of metrics: throughput indicators, quality signals, cost efficiency ratios, organizational output metrics, and adoption health signals.

Throughput indicators are what most teams already measure: PRs merged per developer per sprint, task completion time, and code volume. These are necessary but insufficient. Quality signals that AI adoption requires teams to add include: code churn rate by AI usage cohort (target: no increase above baseline), revert rate for AI-assisted PRs versus human-written PRs, and post-shipment defect density by code origin. Cost efficiency ratios are the metrics most engineering leaders are currently missing: cost-per-merged-PR (total AI tool spend divided by PRs merged), tokens-per-story-point, and the break-even threshold for agentic delegation (hours-saved versus token cost at current pricing). At $200–$2,000+ per engineer per month for agentic AI tools, teams that don’t instrument cost-per-PR are flying blind. Organizational output metrics are the Harvard-Jellyfish gap measure: features shipped per quarter, deployment frequency, and time-to-production for net-new features. If these numbers are not moving despite positive throughput and developer sentiment signals, the Jevons paradox is active in your organization. Adoption health signals include AI tool active usage rate (not license utilization, but actual sessions per week per developer), developer-reported confidence in AI output, and whether senior engineers are using AI tools — senior engineers abandoning AI tools is an early warning signal that churn and review burden are eroding the value proposition.

FAQ

What did the Jellyfish AI coding study find about token consumption and productivity?

The Jellyfish Q1 2026 study of 7,548 engineers found that top 20% token consumers produced 2x the PR throughput of the bottom 20%, but at 10x the token cost. The productivity gains are real but sublinear — more token consumption does not produce proportionally more output, and at high consumption levels the cost-per-PR escalates faster than throughput.

What is tokenmaxxing and why is it a problem?

Tokenmaxxing is the organizational behavior of treating AI token consumption as a proxy for developer productivity. It became prominent after reports that companies like Meta ranked employees on token consumption leaderboards and Uber burned through its entire annual AI budget in four months. It’s a problem because token volume is a measure of AI tool usage, not business output — teams optimizing for the metric miss the outcome.

How much does the typical developer consume in AI tokens per month?

According to Jellyfish’s data, the typical developer consumes approximately 51 million tokens per month. The 90th percentile consumes around 380 million tokens per month — roughly 7x more. Per-developer token consumption rose about 18.6x in nine months between 2025 and 2026, primarily driven by agentic tools running multi-step automated workflows.

What did the Harvard-Jellyfish joint study find about AI coding productivity?

The joint study analyzed 100,000 developers across 500 companies and found that AI makes individual developers measurably faster, but organizations are not shipping significantly more features as a result. Individual coding speed improved, but the organizational-level metrics — features shipped, deployment frequency, time-to-production — showed no statistically significant increase in the same period.

Why is code churn a hidden cost of AI-assisted development?

AI coding tools optimize for local correctness (code compiles, tests pass) rather than systemic fit with existing architecture and conventions. Code that passes immediate review but violates deeper architectural invariants accumulates as churn — revisions and reverts that materialize weeks later. Jellyfish found a 5–11% increase in reverted code under high AI adoption; GitClear found regular AI users experience 9.4x higher code churn than non-AI users. This hidden quality cost partially offsets the throughput gains that AI tools deliver.