AI Systems Architecture

Retrieval-Augmented Generation (RAG)

RAG grounds an LLM's response in retrieved external data rather than relying solely on what the model memorized during training. At query time: embed the query, retrieve the most relevant chunks from a vector store (or hybrid search), insert them into the prompt as context, then generate an answer constrained to that context.

What you can actually build

• A support assistant that answers from your actual product docs and ticket history instead of generic training knowledge
• Compliance-aware Q&A that cites the specific policy section an answer came from
• Systems that stay current without retraining the model — update the index, not the weights

Beyond naive RAG

• Hybrid search — combine dense vector search with sparse keyword (BM25) matching for exact-term accuracy (SKUs, codes, names)
• Re-ranking — retrieve a larger candidate set, then use a cross-encoder to re-rank for precision before passing to the LLM
• Query rewriting / decomposition — break a complex question into sub-queries before retrieving
• Self-correcting RAG — grade retrieved docs for relevance and re-retrieve if they fail the check, instead of generating from weak context

Model Context Protocol (MCP)

MCP is an open protocol (introduced by Anthropic) standardizing how LLM applications connect to external tools, data sources, and systems. Instead of every application writing custom, one-off integration code for each tool, MCP defines a common interface: servers expose tools/resources, and any MCP-compatible client (Claude, an IDE, a custom agent) can discover and call them the same way.

What you can actually build

• An MCP server that exposes your internal systems (a database, a ticketing system, an EDI processing pipeline) as standardized tools any compatible LLM client can call
• A single agent that connects to many MCP servers (Slack, GitHub, a CRM) without bespoke integration code per service
• Reusable tool servers shared across multiple applications, instead of re-implementing the same API wrapper repeatedly

// minimal MCP server tool definition (conceptual)
server.tool("lookup_claim_status", {
  description: "Look up the status of an insurance claim by ID",
  inputSchema: { claimId: "string" },
  handler: async ({ claimId }) => {
    const result = await db.query(
      "SELECT status FROM claims WHERE id = $1", [claimId]
    );
    return result.rows[0];
  }
});

MCP is moving quickly; verify current spec details, security guidance, and ecosystem maturity against docs.claude.com before treating specifics as settled.

LLM Orchestration

Orchestration is the control layer that coordinates everything else: sequencing LLM calls, routing between models, managing memory/state across turns, calling tools, handling retries and fallbacks, and deciding what happens next. LangChain/LangGraph from the prior guide are orchestration frameworks; this section is the underlying concept independent of any specific tool.

What you can actually build

• Multi-step pipelines: classify intent → route to the right handler → retrieve if needed → generate → validate output
• Model routing: cheap/fast model for simple queries, escalate to a stronger model only when needed
• Fallback and retry logic when a model call fails, times out, or returns malformed output
• Memory management across a multi-turn conversation, deciding what context to keep, summarize, or drop

Common orchestration patterns

Agentic Workflows

An agentic workflow is one where the model itself decides the next action — which tool to call, whether to retrieve more information, when the task is complete — rather than following a developer-defined fixed sequence. This sits at the dynamic end of a spectrum that starts with a single LLM call and ends with a fully autonomous multi-step agent.

What you can actually build

• A research agent that decides which sources to search, how many times to search, and when it has enough information to answer
• A coding agent that writes code, runs tests, reads the failure, and iterates without a human specifying each step
• A claims-processing agent that gathers data from multiple systems, checks rules, and only escalates to a human when confidence is low

Production guardrails for agents

• Hard iteration/step limits to prevent runaway loops
• Scoped, least-privilege tool permissions — an agent shouldn't have write access it doesn't need
• Human-in-the-loop approval gates before irreversible or high-stakes actions
• Full logging of every tool call and decision for auditability

AI Pipeline Design

Pipeline design is the architecture-level discipline of deciding how data moves through an AI system end to end: ingestion, processing, storage, retrieval, generation, validation, and monitoring — and where each failure mode is caught. This is the layer where EDI/ETL background transfers most directly.

A typical production RAG pipeline, stage by stage

Chunking Strategies

Chunking is how you split source documents into retrievable units before embedding. It is the single highest-leverage decision in most RAG systems — the right chunk size and strategy can double retrieval quality; the wrong one silently cripples it. Most RAG failures trace back here, not to the model or the prompt.

Chunking strategies compared

The parent-child (hierarchical) pattern

Index small chunks (128–256 tokens) for high-precision retrieval. When a small chunk matches, return its parent chunk (512–1024 tokens) to the LLM for richer context. This separates the precision of retrieval from the context richness of generation.

from llama_index.core.node_parser import HierarchicalNodeParser
from llama_index.core.retrievers import AutoMergingRetriever

# Build hierarchy: 2048 → 512 → 128 token chunks
parser = HierarchicalNodeParser.from_defaults(
    chunk_sizes=[2048, 512, 128]
)
nodes = parser.get_nodes_from_documents(documents)

# Retriever: match on small chunks, return parent for context
base_retriever = index.as_retriever(similarity_top_k=12)
retriever = AutoMergingRetriever(base_retriever, storage_context)

Chunk size guidelines

Multi-Agent Systems

A multi-agent system uses multiple LLM-powered agents working together — one orchestrating, others specialising — to complete tasks too complex or broad for a single agent. Each sub-agent has a focused role, its own tools, and communicates results back to the orchestrator. The architecture trades simplicity for capability.

Core multi-agent patterns

Supervisor agent in LangGraph

from langgraph.graph import StateGraph, END
from typing import Literal

AGENTS = ["claims_agent", "policy_agent", "billing_agent"]

def supervisor(state):
    """Route to the right specialist agent based on query."""
    prompt = f"Route this query to one of {AGENTS}: {state['query']}"
    route = llm.invoke(prompt).content.strip()
    return {"next_agent": route}

def should_continue(state) -> Literal["claims_agent","policy_agent","billing_agent","END"]:
    return state.get("next_agent", "END")

graph = StateGraph(dict)
graph.add_node("supervisor", supervisor)
graph.add_node("claims_agent", claims_agent_fn)
graph.add_node("policy_agent", policy_agent_fn)
graph.add_node("billing_agent", billing_agent_fn)

graph.add_conditional_edges("supervisor", should_continue)
[graph.add_edge(a, "supervisor") for a in AGENTS]  # loop back
graph.set_entry_point("supervisor")
app = graph.compile()

Communication patterns between agents

Production guardrails for multi-agent systems

AI System Deployment Patterns

Deploying an AI system is not the same as deploying a standard API. The system has non-deterministic outputs, external model dependencies, retrieval pipelines that can go stale, and quality metrics that degrade silently over time. Each of these requires specific production patterns beyond what a standard deployment checklist covers.

Deployment architecture options

CI/CD for AI systems

# Example GitHub Actions gate for a RAG pipeline change
# Runs on every PR that touches prompts, retrieval, or chain logic

name: RAG Quality Gate
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run retrieval eval
        run: python eval/run_ragas.py --min-faithfulness 0.80
                                      --min-context-recall 0.75
      - name: Run DeepEval suite
        run: deepeval test run tests/test_chain.py
      - name: Check for prompt regressions
        run: promptfoo eval --config promptfoo.yaml --ci

Canary and shadow deployment

Production monitoring checklist

Production-Level Interview Q&A

Click a question to expand the answer.