AI Agent Infrastructure: The Stack Explained

Everything that makes an AI agent work in production is AI agent infrastructure, and most of it has nothing to do with the model. The model is one endpoint in a larger system built from job queues, databases, auth layers, tracing pipelines, and permission controls that most engineers already know how to build. Frameworks like LangGraph, the Claude Agent SDK, and the OpenAI Agents SDK define how an agent reasons and loops; infrastructure is what runs beneath them and survives when the process dies.

Key Takeaways

• Model access is the entry point to agentic infrastructure — not the model itself, but the endpoint, routing, and fallback layer around it.
• Memory, durable run state, and a job queue are what keep an agent alive across crashes and long-running background tasks.
• Identity, permissions, and observability are the three components that make an agent safe to hand to a customer.
• Completion notifications for background agent runs are lossy by design — the structural fix is a manifest file, not better monitoring.
• Short, direct answers to the most common questions about building and running agent infrastructure.

The Agentic Infrastructure Layer Nobody Talks About: Model Access and the Tool Layer

Agentic infrastructure starts before the first token is generated. Before an agent can reason, it needs a stable path to a model, and that path is more complex than a single API call. Model access means choosing between a hosted inference endpoint from Anthropic, OpenAI, or Google, and a self-hosted server like vLLM or llama.cpp that exposes an OpenAI-compatible API. Either way, you need routing logic that falls back to a secondary model when the primary is down, and per-request token budgets that stop a runaway call from draining your quota.

Most teams should start with a hosted API. It removes GPU procurement, model serving, and version management from your plate. Self-hosting makes sense when privacy requirements, call volume, or offline operation demand it. Not before.

What Is the Tool Layer and Why Does It Matter?

The tool layer is the set of typed interfaces an agent calls to act on the world. Without it, an agent produces text and nothing else. With it, an agent can read a database, call an API, write a file, or publish a post. Good tool definitions include a JSON schema for every input and output, authentication scoped to that tool rather than a shared credential, and idempotent writes so a retry does not create a duplicate record.

The Model Context Protocol (MCP) is the open standard for exposing tools to agents over HTTP. It gives any agent a consistent way to find and call tools regardless of which framework runs the reasoning loop. Nova's MCP server, available in early access, runs at mcp.nova.host over Streamable HTTP. Claude Code, claude.ai, Codex, and Cursor can all connect to it and manage a WordPress site through it. Access uses a per-account API key or OAuth 2.1. The server exposes 23 tools in four groups, each with a typed schema and scoped credentials.

A framework like LangGraph decides when to call a tool and what to do with the result. Infrastructure decides whether that call is authenticated, logged, and retried on failure. Those are separate concerns. Conflating them is one of the most common mistakes teams make when moving from a demo to a production system.

The AI Agent Infrastructure Stack: Memory, State, and Orchestration

The AI agent infrastructure stack has two paired components in the middle: memory and state, then orchestration and execution. Skip either one and you find the gap when an agent crashes mid-task with no way to resume, or when a background job finishes and nothing is notified.

Memory and State

Agent infra for memory starts with context-window management. Every model has a finite window, and a long-running agent will fill it. You need a clear rule for what stays in the window and what moves to a longer-term store. Start with Postgres and an embeddings column. Add a vector database only when query patterns and data volume make Postgres slow, not before.

Durable run state is a different problem. If an agent crashes after step three of seven, it should pick up at step four. That means writing state to a persistent store at each step boundary, not keeping it in memory. Without this, every crash resets the agent to the beginning. Long tasks become unreliable no matter how capable the model is.

Orchestration, Execution, and the Notification Problem

Orchestration is the job queue and workers that run agent steps. Each worker needs retry logic with exponential backoff, per-step timeouts, and concurrency limits so a traffic spike does not block other work. Steps that run code need sandboxed environments so the agent cannot touch the host system.

Here is the part most teams learn the hard way: completion notifications for background runs are lossy. A run finishes and the caller is never told. The fix is not better monitoring. Treat the notification as a hint and the file on disk as the truth. Every run writes a manifest to a fixed location. That manifest moves through four states: launched, running, finished, and collected. A sentinel process polls those manifests and re-raises any finished run that nobody has collected. The sentinel catches what the notification missed.

Nova's background work runs on a Redis-backed job queue. Services are written in Go on Google Kubernetes Engine behind Traefik. The manifest-plus-sentinel pattern came from a real failure, not a design exercise.

Infrastructure for AI Agents: Permissions, Safety, and Cost Control

When an agent can publish content, modify records, or trigger operations, a missing permission check is not a bug. It is a liability. This is the layer most teams underinvest in until something goes wrong.

Identity and Permissions

Every agent needs its own credentials, separate from the human who set it up. Per-agent API keys limit damage if a credential leaks: it cannot reach other agents' resources or gain new permissions. Scoped keys go further by restricting which operations that credential can run at all.

One-shot delegation tokens are the right model for agents that call downstream services. When Nova's MCP server calls a backend on an agent's behalf, it uses a token signed with a separate secret. That token is bound to one tenant, valid for one use, and consumed atomically in Redis. A leaked token is already spent or already expired. It cannot be reused.

Nova gives each WordPress site owner six permission toggles: content read, content write, publish, site data read, operations, and destructive. The owner turns on only what the agent needs. Destructive operations stay off unless the owner turns that toggle on, and even then the riskiest ones, such as a restore, use a two-step prepare-then-execute flow with a short-lived token. That gate is not optional. It is what separates an agent that helps from one that causes incidents.

Safety at the Action Layer

Publishing through the agent does not skip quality checks. Nova's MCP server runs the same internal-link quality gate used across the rest of the content pipeline. It also runs a compare-and-publish check that records the timestamp and sha256 digest of the post before writing. If the post changed since the agent last read it, the write is rejected. The agent cannot overwrite edits it never saw.

Every tool call goes into an audit log: what the agent called, what arguments it passed, what the system returned. That log is the only reliable way to reconstruct what happened after a failure.

Observability and Cost

Trace every model call and every tool call. Link them by run ID so you can replay the full sequence after a failure. Standard application logs are not enough for this. Agent runs are non-deterministic and multi-step. You need traces, not just logs.

Token metering belongs at the platform level. Nova meters AI tokens per account across all AI services and enforces a daily spend breaker. When the platform hits its daily ceiling, further model calls stop until the next reset. That prevents a misbehaving agent or a misconfigured billing rule from generating an unbounded bill.

An evaluation harness closes the loop. When you change a prompt or swap a model, you need a way to measure whether outputs got better or worse. Without one, every model change is a guess.

Build Versus Rent

Rent inference. Use a hosted API. Start your data layer with Postgres and a queue. Add an MCP tool layer to give agents typed access to your systems. Self-host models only when privacy, volume, or offline requirements make hosted APIs unworkable.

Own metering, permissions, and tracing first. These are not components you add later. They are what make an agent safe to give to a customer. Ship without them and you are not running an agent in production. You are running a demo with production consequences. For teams that want help building this layer, managed infrastructure services and fractional CTO services are available. See pricing and Kubernetes WordPress hosting for platform details.

The Bottom Line

The model is the easy part. The job queue, manifest files, permission toggles, delegation tokens, and trace pipeline are what determine whether an agent holds up on real systems or breaks the first time something goes wrong. Most of that work is backend engineering you already know how to do. The teams that ship reliable agents fastest are the ones who treat this as a software problem, not an AI problem. To go deeper on any layer of this stack, a technology consulting engagement is the place to start.

FAQs

What is AI agent infrastructure?

AI agent infrastructure is everything that surrounds a model and lets it act on real systems safely. It includes the inference endpoint, tool layer, memory store, job queue, permission system, and tracing pipeline. The model is one part of that system. The rest is what makes it work in production.

What is the difference between an agent framework and agent infrastructure?

A framework controls how an agent reasons: when to call a tool, how to handle the result. Infrastructure is what runs underneath: the queue, database, auth layer, and tracing system that keep working when the process dies. Both are required. Neither replaces the other.

Do I need a GPU to run AI agents?

No. Start with a hosted inference API from Anthropic, OpenAI, or Google. Self-hosting on a GPU only makes sense when privacy requirements, call volume, or offline operation make hosted APIs unworkable. Everything else in the agent stack runs on standard backend infrastructure.

What is MCP and why does it matter for agent infrastructure?

MCP (Model Context Protocol) is an open standard for exposing tools to agents over HTTP. It gives agents a consistent way to find and call tools regardless of which framework drives the reasoning loop. Nova's MCP server uses this standard to let AI agents manage WordPress sites.

How do you stop an AI agent from taking destructive actions?

Set per-capability permission toggles so the agent can only access what it needs, and require human approval before any destructive operation runs. Back that with one-shot delegation tokens that expire after a single use and an audit log of every tool call. An agent that cannot escalate its own permissions cannot cause unbounded damage.