Context Engineering Is Not Prompt Engineering : Simple Design Hacks

ByNilay Shah 6 min read
Featured image for "Context Engineering Is Not Prompt Engineering" a dark blue gradient graphic showing the post title beside a stack of outlined layers, representing context engineering as a layered data architecture for AI agents.

Every few months our industry renames a discipline and pretends it is new. Context engineering is not one of those cases: it marks a real shift in where the hard work of building AI agents actually lives, and most of that work turns out to be data architecture, not prompt writing.

I have watched this play out in our own agent initiatives. The team spends two weeks polishing a system prompt, the demo works, and then the agent goes near production data and starts answering confidently from a stale table nobody remembered was deprecated. The prompt was never the problem. The information architecture around the agent was.

The thesis

Prompt engineering optimizes a sentence. Context engineering optimizes the entire supply chain of information that surrounds that sentence: what gets retrieved, from where, in what shape, with what metadata, and at what moment in the agent’s loop. Anthropic frames it as curating the optimal set of tokens during inference from a constantly evolving universe of possible information. That is not a wording problem. That is a data problem, and it belongs to the people who have been doing retrieval design, metadata management, and governance for years: data architects.

Almost everything written about context engineering so far comes from AI and ML engineers, and it reads that way. It is about token budgets, compaction, and scratchpads. All useful. But the enterprise version of this problem sits one layer lower, in how your platforms expose meaning to machines. That is the angle this post takes.

Where prompt engineering ends

A prompt is one input among many. When an agent runs, its context window carries the system prompt, tool definitions, retrieved documents, message history, and long-term memory, and each of those is fed by an upstream pipeline. The prompt is the only part you can fix in a text editor. Everything else is architecture.

The context window: what the model actually sees context window contents System prompt prompt engineering lives here Retrieved data fed by retrieval pipelines Tool definitions fed by platform interfaces Message history fed by compaction strategy Long-term memory fed by storage design Everything upstream context engineering lives here One box is a writing exercise. The other five are data architecture.

Signals worth your attention

A handful of findings from the past year make the case sharper than any opinion could.

Long context does not mean usable context. Chroma’s context rot research tested 18 frontier models, including GPT-4.1, Claude, and Gemini, and found every single one degrades as input length grows. Accuracy can drop 30 to 50 percent well before the advertised window limit, and it drops non-uniformly.

The middle of your context is a dead zone. The lost-in-the-middle effect means models attend well to the beginning and end of context but poorly to the middle. Where you place retrieved information matters as much as whether you retrieved it.

Practical budgets are far below marketed limits. Teams running high-accuracy workloads on models with million-token windows report safe operating budgets closer to 150K to 400K tokens. The window is a ceiling, not a target.

Distractors are worse than absence. Chroma’s work shows that semantically similar but irrelevant content actively degrades answers. A retrieval pipeline that returns ten plausible documents when two are correct is doing damage, not adding safety margin.

The strategies are converging. LangChain’s framework groups every context engineering technique into four moves: write context out of the window, select what comes in, compress what stays, and isolate across agents. Three of those four are retrieval and storage problems.

Enterprises are already voting. In Atlan’s survey work, 95 percent of data leaders said context engineering is important to running agents at scale. The market has decided this is an infrastructure concern.

Production failures are context failures. Gartner attributes 85 percent of AI model failures to deployment and data issues, not model capability. The model was fine. What it was shown was not.

Deep breakdown: context as a data architecture stack

Here is the reframe I find most useful. Strip away the AI vocabulary and context engineering decomposes into four layers that any platform architect will recognize, because we have built versions of all of them before.

The context architecture stack Agent runtime assembles the window each turn: prompt, history, memory Selection layer retrieval, ranking, filtering, placement: where quality is won or lost Semantic and metadata layer definitions, lineage, ownership, freshness: what things mean Data platform warehouses, lakes, vector stores, operational systems Most teams tune the top layer. Most failures originate in the middle two.

The data platform layer is where the raw material lives, and agents inherit every one of its existing sins. Stale tables, duplicate metrics, undocumented datasets. An agent does not know your revenue table was superseded last quarter unless something tells it.

The semantic and metadata layer is where meaning lives, and it is the layer I would argue matters most. When an agent retrieves a number, the number alone is nearly useless. Is it net or gross? Which currency? When was it last refreshed? Who owns it? A semantic layer answers the definitional questions; active metadata answers the trust questions. DataHub’s framing is blunt and correct: without lineage and freshness signals, agents reason about a world that no longer exists. If you have invested in a semantic layer for BI consistency, you have already built half of your agent context layer. Most organizations just have not connected the two.

The selection layer is where the research findings bite. Given context rot and distractor sensitivity, the job is not “retrieve relevant documents” but “retrieve the minimum sufficient set, ranked, deduplicated, and placed deliberately.” This is precision-over-recall as an architectural principle. Every document you add past sufficiency is not free; it is actively corrosive.

The agent runtime is the layer the AI engineering community writes about: compaction, scratchpads, memory files, sub-agent isolation. Important work. But it is the assembly line, not the supply chain. If the layers below feed it ambiguous, stale, or unranked material, no amount of clever window management recovers the loss.

The practical consequence: staff your agent effort accordingly. A team of prompt specialists without a metadata practice will plateau early, a pattern I covered in good architect, bad architect. “The organizations that get agents to production reliably are the ones treating context as a governed data product with owners, contracts, and freshness guarantees.”.

There is also a cost argument hiding here. Every token that enters the window is paid for twice, once in API spend and once in latency, and both compound across every turn of an agent loop. Disciplined context engineering is the cheapest FinOps lever most AI programs have not pulled yet: retrieving less, but better, cuts inference bills and improves answer quality at the same time.

The contrarian take

Bigger context windows will not dissolve this problem, and I would go further: they make it worse. The comfortable assumption is that context engineering is a temporary tax we pay until windows get large enough to dump everything in. The evidence points the opposite way. Every model tested degrades with length, distractors do measurable harm, and the gap between marketed window size and usable window size is growing, not shrinking. A ten-million-token window fed by an ungoverned data estate is simply a larger container for confusion. Selectivity is not a workaround for small windows. It is a permanent architectural requirement, the same way indexing did not become obsolete when storage got cheap.

If you are wondering where to start, begin with an audit rather than a framework. Trace one production agent answer back to its sources and ask three questions at each hop: who owns this, when was it refreshed, and why was it selected. In my experience the first pass of context engineering is uncomfortable, because most teams cannot answer even the ownership question.

One tool worth attention

Keep an eye on the Model Context Protocol (MCP). It is doing for agent context what JDBC did for database access: standardizing how agents discover and consume tools, data, and metadata from enterprise systems. The interesting part for architects is not the protocol mechanics but the design pressure it creates. Once your catalog, semantic layer, and warehouse all speak MCP, the question “how do we expose governed context to agents” gets a concrete, testable answer instead of a bespoke integration per team. If each team is currently building its own retrieval pipeline and access shim, MCP is the consolidation point to evaluate first.

The rule of thumb

Treat the context window like a production dashboard, not a data lake: everything in it must earn its place, carry its metadata, and have an owner who can explain why it is there. If you cannot say who governs what your agent sees, you have not built an AI system. You have built an incident that has not happened yet.

References

Similar Posts