When the Prompt Isn’t the Problem Anymore

Everyone who ends up developing enterprise level systems with a coding agent hits the same wall. In the beginning, a loose prompt is plenty. “Add a login flow.” “Fix this bug.” “Refactor this component.” The agent fills in the gaps intelligently, the code works, you ship.

Then the codebase grows. More files, more services, more decisions baked into the architecture that nobody wrote down. And the same loose prompting style that worked at 5,000 lines starts producing subtly wrong answers at 50,000. Code that compiles but ignores an existing pattern. A fix that solves the symptom in one file while breaking an assumption in another. A “refactor” that quietly drops an edge case.

The instinct is to blame the model. The instinct after that – the one I had, and the one most builders seem to land on – is to prompt harder. Longer prompts. More careful phrasing. More context pasted in up front.

That second instinct is aimed at the wrong layer, and there is now enough research to say why.

It stopped being prompting a while ago

Anthropic’s applied AI team put a name to the shift in a September 2025 engineering post. As AI use moves from one-shot chat toward agents operating over many turns and long time horizons, the discipline that matters stops being “write a good prompt” and becomes context engineering: curating the entire, constantly-changing set of tokens available to the model at each step, not just the opening instruction.

The distinction is worth being precise about, because a fair number of people on Hacker News have called it a rebrand. Prompt engineering is about the words in a single request. Context engineering is about the system prompt, the tools, the message history, the file contents, and everything else competing for a finite attention budget on every single turn. If you only ever send one message, they are the same thing. If you are 90 turns into a session, they are not remotely the same thing.

Which is exactly the situation a growing codebase puts you in. It isn’t that your prompts need to get cleverer. It’s that the volume and structure of context the agent needs in order to reason correctly grows with the system, and past a certain point, adding more information makes the reasoning worse rather than better.

Why “just paste in more” backfires

There is measured evidence behind the feeling that the agent gets flakier the more you throw at it. Chroma’s 2025 research, run across 18 frontier models including the Claude 4 family, found that accuracy on retrieval and reasoning tasks degrades as input length grows – well before the stated context-window limit, and even on simple tasks. They called it context rot. It isn’t a cliff you fall off. It’s a slope you’re always on.

Anthropic’s post cites the same research and frames an agent’s usable attention as a finite, depleting resource, similar to human working memory. That’s a consequence of the architecture: every token attends to every other token, so the cost of holding a large context in focus grows faster than the context does. Million-token windows move where the degradation shows up. They don’t remove it.

The 2026 benchmarks make the long-horizon version of this concrete. SlopCodeBench, which measures agents on iterative tasks where the agent’s own earlier code carries forward and the spec evolves at each checkpoint, saw the strongest models tested score 11% and 17% strict pass rates. Separate work summarized on arxiv finds frontier models near-perfect on tasks a skilled human finishes in under four minutes, and succeeding under 10% of the time on tasks that take a human more than four hours. The gap isn’t intelligence. It’s what happens to reasoning when state accumulates.

So when builders describe having to “refine my prompting finer and finer,” one of two things is usually happening. Either the task genuinely needs more context than fits cleanly, and stuffing it all in degrades the reasoning you were trying to buy. Or the session has accumulated stale, irrelevant, redundant context – dead-end file reads, abandoned approaches, old tool output – that is crowding out the signal. Neither one is fixed by a longer prompt.

The part where the standard advice falls apart

Here’s where I have to argue with the version of this post I originally set out to write.

The single most repeated piece of advice in this genre is: write a good CLAUDE.md. A project-root memory file the agent loads every session, holding your commands, conventions, architecture, and gotchas so you don’t restate them constantly. It’s Anthropic’s own recommendation, it’s what I do, and it’s the first thing I’d have told you to fix.

The empirical evidence for it is bad.

A controlled two-agent ablation across Claude Code and Codex, using 17 real tasks from 3 repositories over 288 evaluated runs, found that context strategy did not measurably move correctness on either agent. Related work reports that context files can reduce task success rates relative to giving the agent no repository context at all, while adding more than 20% to inference cost. And a study of 2,303 context files from 1,925 repositories found them long, complex, and scoring at a “very difficult” readability level – the band usually reserved for dense legal and academic text.

Read those three findings together and a fairly obvious explanation shows up. Most CLAUDE.md files in the wild are themselves context rot. They’re the place people dump everything they were afraid the agent would forget, they get appended to and never pruned, and they burn attention budget on every turn whether or not any of it is relevant to the task at hand. The file isn’t a bad idea. The unbounded-append habit around it is.

The correction isn’t to delete it. It’s to treat it with the same budget discipline as everything else: short, high-signal, load-bearing facts the agent can’t derive by looking, and nothing else. If a line in your context file could be discovered by the agent running grep, it’s costing you more than it returns.

What actually seems to hold up

Strip away the file-writing advice and the durable techniques are the ones that change the shape of context rather than its volume.

Just-in-time retrieval is the big one. Rather than pre-loading the relevant codebase, let the agent explore on demand, using paths, grep results, and directory structure as lightweight pointers and pulling actual file contents only when it needs them. This mirrors how people work – nobody memorizes a codebase, we use naming conventions and folder layout as an index and retrieve as we go. It also means your directory structure is context engineering, which is a more useful thing to invest an afternoon in than another thousand words of CLAUDE.md.

Compaction is the second. Claude Code does this natively: as a session approaches its limit, the history gets summarized, keeping architectural decisions, open bugs, and implementation details while dropping redundant tool output, and the agent continues with a fresh window plus the summary. Doing it deliberately – clearing a session once the exploration is done and starting the implementation clean – is more effective than letting the automatic version rescue you at the boundary.

Then there’s memory outside the window. An agent keeping a running NOTES.md or task list that persists on disk and gets read back in later preserves state across long work without holding it in the active window the whole time. And for genuinely large tasks, delegation: a lead agent handing focused sub-tasks to sub-agents with their own clean contexts, each of which might burn tens of thousands of tokens exploring and report back a 1,000-token summary. Total work scales up; the lead agent’s context stays tight.

The last one operates above all of this. Spec-driven development treats a written specification, not the prompt and not the code, as the primary artifact, with generated code as a regenerable output of that spec. GitHub’s Spec Kit and AWS’s Kiro both build workflows around it. Adoption claims in industry coverage are self-reported rather than controlled, so read them as directional – but the underlying logic is sound, because a spec is the one piece of context that stays accurate while the session churns underneath it.

What we changed

Across our own projects, the shift has been away from prompt craft and toward the system around the prompt. Context files got shorter, not longer, and we prune them when a convention becomes discoverable in the code itself. Long sessions get cut deliberately at the seam between exploring and building. Large changes get decomposed into sub-agent-sized units with clean context instead of one heroic freeform session. And anything non-trivial gets a lightweight spec first, so “what to build” isn’t being re-inferred from a paragraph every time.

None of that is about phrasing a cleverer sentence. It’s engineering practice applied to a resource – attention – that most teams still treat as free. If you’re doing AI-assisted development at any real scale, that resource is the one that’s actually scarce.

---

At EchoForgeX, we build AI-powered tools and help businesses integrate AI into their workflows. Get in touch to learn how we can help your team work smarter with AI.