Two terms keep colliding in infrastructure AI: semantic layer and context layer. They're not the same thing, and conflating them is how teams end up with agents that understand vocabulary but have no idea what's actually running — or agents that know the environment but still guess wrong about what the data means.
The problem starts with naming
You have a Kubernetes cluster. The LLM already knows what a pod, a deployment, or an ingress is — that’s standard. It doesn’t need explaining.
Then you add custom resource definitions. A CRD named after a Cisco router, a firewall policy, or — because someone thought it was funny — “easy-ci-v11.” The moment you extend the platform with your own types, the model has no idea what it’s looking at. It starts guessing. Developers are notorious for naming things in ways that mean something different from what the thing actually does, and a model trained on public data has never seen your internal vocabulary.
That’s the problem a semantic layer solves. And it’s a different problem from what a context layer solves.
Semantic layer: meaning
A semantic layer sits on top of your data and explains what things mean — in terms a model can use. Tables, columns, custom resource types, internal acronyms. If your whole engineering org uses “BAD” as a shorthand and everyone knows what it means, the model very likely thinks it means something else. Someone has to write that down.
The alternative is handing the model your entire database on every question. That’s both too slow and too expensive. So you build a layer of meaning it can consult instead — defined once, used every time.
A word on the sales pitch: conversational-analytics tools are often sold as “just ask your data, no analytics team needed.” The real risk is an agent that gives wrong answers in fluent, confident language to people who no longer have the expertise to catch it. You still need the team. Someone has to build and maintain the layer.
Context layer: what’s actually running
A semantic layer tells the agent what things mean. A context layer tells it what’s actually there — a live map of your environment: which services exist, where they run, how they connect, what depends on what.
The distinction matters. An agent that knows vocabulary but not topology will still reconstruct everything from scratch on every session. An agent with live context can skip the archaeology.
A useful context layer isn’t a dump of everything — that would overflow a context window immediately. It has query and filtering capabilities. The agent asks a specific question, gets back a targeted slice, and starts working. Infrastructure questions are relationship-shaped, so the layer has to carry dependencies, not just names.
What this looks like in practice
You have a service called “pizza-ordering” running across dev, staging, and prod AWS accounts. You type: check if there are any errors for pizza-ordering production.
Without a context layer: the agent starts from scratch. It knows you’ve used AWS before, so it lists accounts, lists services, guesses what you mean, runs cascading searches through files. Maybe 100,000 tokens later it finds the right account — only to discover the logs are in Datadog, not CloudWatch. This happens every single session unless you maintain hand-written specs and keep them updated as the environment changes.
With B.O.R.I.S: one or two MCP calls. B.O.R.I.S tells the agent it’s an ECS service in a specific account, logs reachable this way, dependencies already resolved. The agent goes straight to querying. Faster answer, most of the context window preserved for actual work.
The manual version of the same task — mapping an AWS Transit Gateway network — took four sessions: one to call the API and dump a map, a second for more data, a third, then a fourth an hour later to open all the files and finally make sense of them. Collecting context manually is what nearly everyone is doing today, whether they call it that or not.
You don’t need a bigger model. You need better context.
Research shows a smaller model with a good semantic and context layer can outperform a larger model working blind. dbt Labs’ 2026 benchmark found that in-scope questions through a semantic layer hit near-perfect accuracy, and out-of-scope questions returned a clean error rather than a plausible wrong answer. Raw inference without a layer tends to fail by returning something confident and incorrect.
The practical implication: before you upgrade the model, fix the context.
Stale memory is a different problem
Many agent harnesses now ship with memory. The risk: memory goes stale. It isn’t revalidated against reality, and it quietly biases every session. Your agent’s behavior gets tilted by the residue of past interactions rather than the current state of your infrastructure.
We’ve seen this directly — using an AI assistant until accumulated memory polluted sessions badly enough to require “don’t look at the memory, treat this as a raw session” in every prompt. It still drifted. The fix was moving to a fresh-session agent where you control exactly what context loads.
For DevOps the stakes are higher. An agent making decisions about an AWS estate with tens of thousands of resources, based on what things looked like three weeks ago, is making decisions on wrong information.
The healthier default: fresh sessions that load only current, relevant context each time. Live data checks memory — not the other way around. The missing piece used to be that nobody maintained internal documentation consistently enough for this to work. An agent that keeps the context map current changes that.
The productivity trap
Kent Beck put it plainly: typing got faster, thinking not.
The speed gains from AI coding are real. The trap: you can feel productive while spending the majority of a session gathering context, and still land on the wrong answer. The fix isn’t a faster model — it’s giving the agent the right information to start with, so it doesn’t spend your token budget on archaeology.
As teams ship more code faster — much of it written by agents and forgotten within weeks — no single person can hold the full picture of what’s running. A context layer that stays current is how you keep the system legible when the team can’t.
What to do
- Build the semantic layer first for anything with internal naming, custom types, or domain-specific vocabulary. Don’t assume the model will infer it.
- Add the context layer to stop agents from rediscovering your topology on every session. If your agent is listing AWS accounts from scratch each time, that’s the symptom.
- Prefer fresh sessions with loaded context over persistent memory for production workloads. Stale memory is harder to debug than missing memory.
- Don’t upgrade the model to fix a context problem. They’re different levers.
Resources
- Episode #11 — Base of Record for Intelligent Systems — the direct predecessor: graph-RAG, live truth, and why B.O.R.I.S is a context layer.
- Episode #7 — When Agent Memory Helps and When It Hurts — the case against persistent memory and what to do instead.
- Episode #2 — The Tool Layer: What Makes Agentic AI Possible — why the context window is finite and why rediscovery is expensive.
- Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update (dbt Developer Blog) — near-perfect in-scope accuracy and safe failure out of scope.
- Effective context engineering for AI agents (Anthropic) — context as a finite resource with diminishing returns.
- Model Context Protocol — the interface agents use to consume tools, semantic layers, and context layers.