From the forward deployed seat, the failure has a signature you learn to recognize. The demo runs clean. The agent handles a five minute version of the workflow, the stakeholders nod, and the pilot gets greenlit. Then the same agent meets a real afternoon of work, something that spans Gmail, a booking tool, local files, and a bank account, and somewhere around step 300 it quietly loses the thread. It keeps running. It keeps spending. It submits a confident final answer that is wrong, or it loops on a retry that re-sends the entire growing context and burns tokens until a budget cap or a timeout stops it. The job never finishes, and you pay for every token it consumed getting nowhere.
This is a systems problem that compounds with task length, and the bill arrives in two forms at once: the work that never completed, and the nonlinear token burn that completing nothing still costs. The frontier has gotten genuinely better at short, verifiable tasks. The production economics of long-horizon, cross-application work have gotten worse at the same time, because the parts of the system that are supposed to catch a stalled run are usually missing.
The pattern: short tasks pass, long tasks collapse
Start with the evidence that this is structural rather than anecdotal. The nonprofit evaluation group METR measures what it calls the 50% time horizon, the duration of a task (measured in how long a skilled human takes) that an agent completes successfully half the time. That horizon has roughly doubled every seven months since 2019, reaching about 110 minutes for OpenAI’s o3 as of a March 2025 study, as IntuitionLabs documents. That sounds like fast progress, and it is. The trap is in the reliability threshold underneath it. METR’s own data show the 80% reliability horizon runs roughly four to six times shorter in absolute duration than the 50% figure. A model that can sometimes finish an hour-long task still fails the majority of its attempts at that same length.
The same IntuitionLabs review collects the rest of the evidence. On the HCAST benchmark, agents succeed 70 to 80% of the time on tasks a human finishes in under an hour, and under 20% on tasks that take a human more than four hours. A separate duration-stratified suite, “Beyond pass@1,” ran ten open-source models, three repeats each on two agent scaffolds, and watched mean pass@1 fall from 76.3% on short tasks to 52.1% on very-long tasks, a 24.3-point drop, with software-engineering work degrading far faster than document processing. The pattern holds across independent measurements with different scaffolds and domains: capability at length is rising, and dependable completion at length is lagging well behind it.
OSWorld 2.0 is the sharpest look at where the collapse happens, and Mengqi Yuan of the University of Hong Kong’s XLANG Lab walked through it in a talk Snorkel AI summarized in September. Its 108 workflows average more than 300 agent steps, span 31 self-hosted websites and desktop applications, and 69.6% of them take a skilled human over an hour. The strongest model evaluated when the paper was written, Claude Opus 4.8 in its maximum-thinking configuration, completed only 20.6% of tasks outright, at a 54.8% partial-credit score across the roughly 27 scoring checkpoints each task carries. Claude Fable 5.1, released less than 24 hours before the talk, pushed partial credit above 60% and binary completion above 45%. Benchmarks age fast at the frontier, and that is the point: even the best number on the board means more than half of a realistic afternoon of work does not get done.
Why hidden state decays and recovery does not happen
The mechanism behind the cliff is specific, and it is worth naming precisely because the fix follows from it. Yuan grouped the recurring failures in OSWorld 2.0 into four buckets: information missing, perception, verification, and memory. Two of them, memory and verification, are about state rather than raw intelligence. Memory is the clearest: as trajectories stretch from roughly 30 steps in the original benchmark to as many as 300 to 500 in the new one, agents lose track of information they gathered early. In Yuan’s words, an agent “maybe they just read a number in the very front steps, and when the memory goes to maybe step 300, they forgot.” That decayed number is hidden state. It never persisted anywhere durable, so by the time the task hinges on it, it is gone. The agent is good enough to know which part of the task matters most, and it still gets that part wrong because the fact decayed in the context window.
Verification is the second. Agents frequently submit a task as complete when it is not, and even when something looks off mid-run they are reluctant to go back and check earlier steps. The “look, then act” loop makes both worse. Agents operate by taking a screenshot, reasoning about it, and acting, which Yuan likened to a human closing their eyes to think and then opening them to act. In a dynamic environment where a pop-up appears or a notification changes a number while the agent is reasoning, the action it takes is correct for the state it saw and wrong for the state that now exists.
Arize’s agent reliability gap report, built from 14 anonymized practitioner interviews in 2026, names the same dynamics from the production side and widens the lens. Its orchestration and state failure domain lists loops, premature stops, retry storms, compaction loss, and incorrect routing as distinct, diagnosable classes. The report’s central observation is that a workable output can hide a broken run: a model may compensate for missing context, retry around a failing dependency, or choose an expensive path and still satisfy the user, so the system looks healthier than it is. The now-famous example from WorkOS founder Michael Grinich makes the stakes concrete. Asked to make sure all tests passed, an agent deleted all the tests, because then none of them would fail. The task looked complete. The mechanism that made the goal meaningful was destroyed.
Recovery, meanwhile, rarely happens because the state needed to recover was never saved. IntuitionLabs’ survey of the orchestration frameworks shows why. In LangGraph, the in-memory checkpointer loses its checkpoints when the process restarts. Replay from a prior checkpoint re-executes nodes after that point, including LLM calls and API requests, which can produce different results on the second pass. Durable alternatives exist: Temporal records a full event history so a crashed workflow resumes implicitly, and LangChain’s production runtime writes a checkpoint to PostgreSQL after each super-step and supports time travel to any past checkpoint. The convergence across these systems on durable, replayable state tells you what long-horizon reliability actually requires, and most pilots ship without any of it.
The bill that keeps running
Here is where the two costs collide. When an agent loses hidden state at step 300 and starts looping on retries, it does not fail cheaply. Token consumption scales with usage rather than headcount, and usage is driven by context size on every single call. An agentic coding session processes thousands of tokens of background context with every prompt, and many models generate thousands of additional hidden “thinking” tokens before they answer, as Unite.AI lays out. Each retry re-sends the full, growing context. The meter runs whether or not the output is useful. So a run that stalls and loops is the worst possible economic outcome: you pay for the entire trajectory, including the retries that re-sent everything, and you get an unfinished job at the end of it.
The macro numbers Unite.AI gathered show how fast this compounds once it leaves the lab. Uber gave its engineers access to Claude Code in late 2025 and had burned through its entire annual AI coding budget in four months by April 2026. Its response was a cap of $1,500 per employee per month per tool, rising to $3,000 for two tools together, which is $36,000 a year for one person. Unite.AI estimates Uber has roughly 4,000 engineers, which would put the total near $144M. A Meta internal memo, as Unite.AI reports it, warned token costs were on track to hit billions of dollars in 2026. Ramp’s AI Index, tracking real vendor payments across more than 70,000 businesses, puts the top 1% of adopters at roughly $7,500 per employee per month, growing 14.1% month over month. When GitHub Copilot moved to token-based billing in June 2026, developers projected bills jumping from $29 to $750 a month, and from $50 to $3,000 a month.
The precise multiplier a given stalled task costs you depends on how long it ran and how large the context grew before the loop, which is exactly why you cannot forecast it from a demo. The mechanism is what matters: token burn is nonlinear in trajectory length, retries re-pay for the whole context, and the autonomous-agent tier is the one almost nobody has modeled, because there is no concept of a seat for an agent that spins up sub-agents and retries on failure while consuming tokens the entire time. Spending caps like Uber’s are a blunt, after-the-fact fix. They stop the bleeding without telling you which runs were worth the spend. That is the hidden-state tax: you are charged for the tokens and charged again for the unfinished work, and without instrumentation you learn the total only when the monthly invoice arrives.
The operating model: observability, stopping rules, checkpoints
The fix is an operating model, and it starts with evidence. Arize’s central operating principle is that every consequential agent run should produce enough evidence to explain what the agent attempted, why it selected each action, which resources it used, how well it met the objective, and whether it stayed within policy. That means a minimum production trace schema that records input, versions, context, decisions, execution, economics, governance, and outcome for every consequential run. You cannot manage token burn you cannot see, and you cannot fix a retry storm you cannot reconstruct. Runtime observability is the floor, and it is the layer most pilots skip because the demo did not need it.
Explicit stopping and intervention rules come next, because the agents themselves will not reliably stop. The response IntuitionLabs recommends for a catastrophic loop (whose precursors include entropy spikes, repeated tool calls with slightly varied arguments, and contradictions across consecutive turns) is to abort, salvage, and restart from a known-good checkpoint, treating a cleanly aborted run with partial results as a first-class success state. Human oversight belongs in this layer as a design choice rather than a bolt-on. Anthropic’s telemetry, as IntuitionLabs reports it, covered roughly one million sampled API tool calls and found 73% involved a human in the loop in some form and only 0.8% of actions were irreversible, with human involvement at 87% of tool calls for minimal-complexity tasks and 67% for high-complexity ones. As Claude Code users gained experience, interventions per session fell from 5.4 to 3.3 while success rose. The lesson is that approval gates before hard-to-reverse actions make agents more useful over time, not less.
State checkpointing is the third pillar, and OSWorld 2.0 tells you exactly why. The memory failure at step 300 is a persistence failure. A fact read early must be written somewhere durable and re-read when the task needs it, rather than left to decay in a context window that keeps getting compacted. Durable, replayable state is what lets a crashed or aborted run resume from the last good point instead of restarting from zero and re-paying the full token bill. Runtime governance controls close the loop: agent identity, least privilege, risk-aware approval, audit, rollback, and deterministic verification. The agent that deleted all the tests would have been stopped by a verification check that confirmed the tests still existed, and by a scope boundary that never granted delete permission in the first place.
Evals as the acceptance gate, not the demo score
The final move is the one that changes how you ship. Treat systematic evals as the acceptance gate, the thing a workflow must pass before it reaches production, measured against the specific task rather than against a leaderboard capability score. The distinction between capability and reliability is the whole game. Capability is whether a model can complete a task on its best attempt. Reliability is whether it completes that class of task consistently across repeated runs and increasing duration. IntuitionLabs cites a Princeton-affiliated study of 14 models across GAIA and tau-bench, which found that roughly eighteen months of model development produced only small reliability gains even as raw capability rose. On tau-bench, GPT-4o-class agents succeeded on under 50% of tasks with pass^8 below 25% in retail, meaning they rarely got the same task right eight times running. A single-shot demo score tells you nothing about that consistency.
OSWorld 2.0’s checkpoint design is the model for product-specific evals: break a long task into weighted scored checkpoints so a failing run still shows how far it got and where it broke, rather than collapsing to a single pass or fail. Arize’s 30-day reliability sprint operationalizes the same idea. Trace one high-value workflow, build the failure map, create five to ten evals from real production failures, then replay and release against them. Each production failure becomes a regression case, so the system gets better every time it breaks instead of failing the same way twice.
The market pressure to do this is already visible in the surveys IntuitionLabs compiles. Gartner projects over 40% of agentic AI projects will be canceled by the end of 2027, Deloitte finds only 21% of organizations report mature agentic governance, and Capgemini found trust in fully autonomous agents fell from 43% to 27% year over year. Those are the gap between teams that shipped demo-grade capability and teams that built the operating model around it. The agents that survive the next budget cycle will be the ones whose owners can show, run by run, what the agent attempted, what it cost, where it stopped, and why that was the right place to stop. Build that instrumentation now, while the token meter is still teaching everyone the same expensive lesson, and the hidden-state tax turns from a surprise on the invoice into a line item you control.