18 — Architecture Recommendations

What Was Researched

Synthesized architecture recommendations for building a modern, model-agnostic agent harness, based on findings from all 17 preceding research directories and 10 local codebase studies.

Which Sources Were Used

All previous research directories (01–15, 17) and all 10 local codebases.

Recommended Architecture: The Composite Approach

This is not a merge of reference codebases. Hermes, Codex, Pi, LangGraph, OpenClaw, LiteLLM, assistant-ui, LibreChat, and others were studied to extract patterns. The recommendation is to build one new harness — pick the best idea per layer and connect them via standard interfaces (OpenAI-compat API, MCP, SSE, SKILL.md). Do not fork and glue every project into a single repo.

Core Principle: Narrow Waist, Rich Edges

Adopt Hermes's "narrow core, capability at edges" principle as the foundational design constraint.

┌──────────────────────────────────────────────────────────────┐
│                    APPLICATION LAYER                         │
│  Coding Agent │ Personal Assistant │ Research Agent │ Custom │
├──────────────────────────────────────────────────────────────┤
│                    GATEWAY / ORCHESTRATION                   │
│  Multi-channel routing │ Session management │ Cron scheduler │
├──────────────────────────────────────────────────────────────┤
│                    AGENT RUNTIME                             │
│  Agent loop │ Tool dispatch │ Memory │ Skills │ Subagents    │
├──────────────────────────────────────────────────────────────┤
│                    AI SDK (NARROW WAIST)                     │
│  Provider translation │ Streaming │ Tool calling │ Caching   │
├──────────────────────────────────────────────────────────────┤
│                    MODEL PROVIDERS                           │
│  OpenAI │ Anthropic │ Google │ xAI │ Ollama │ OpenRouter │ … │
└──────────────────────────────────────────────────────────────┘

Layer 1: AI SDK (Narrow Waist)

Recommendation: Standardize on an OpenAI-compatible client (base_url + api_key). LiteLLM is optional — not required.

Layer 2: Agent Runtime

Recommendation: While-loop agent with LangGraph-inspired state management

Layer 3: Gateway / Orchestration

Recommendation: OpenClaw-inspired gateway with Hermes features

Layer 4: Application Layer

Recommendation: Modular applications built on the runtime

Technology Choices

Primary Stack

Model Routing Strategy (5-Tier)

From 13_model_agnostic_harness_architecture/model_landscape_june_2026.md:

Security Model

File Convention Standards

Critical Design Constraints

1. Prompt Caching is Sacred (Design for Byte-Stability)
   • What/Why: Modern frontier models (Anthropic, DeepSeek, OpenAI) charge up to 90% less for cached input tokens. Prompt caching works by storing prefix spans in memory. Any modification to a prefix invalidates the entire cache downstream.
   • When to Use: Essential for multi-turn cognitive loops where the system instruction and tool definitions remain static, but message histories accumulate.
   • How to Design:
     • Order Matters: Place volatile variables (such as the current date/time, user query, and dynamic database context) at the absolute tail of the message array. Keep the system prompt, guidelines, and tool schemas at the head.
     • Byte-Stability: Clean whitespaces, sort tools alphabetically by name, and standardize date strings to UTC-day-only formats.
     • Tailored Slicing: Ensure the prefix hits model-defined cache boundaries (e.g., Anthropic's 1024-token minimum or 2048-token increments).
2. Minimize Regex & Deterministic Logic (LLM-First and AST Parsing)
   • Constraint: Restrict the use of regular expressions to trivial boundary validation (e.g., prefix checks like text.startsWith('/')). Never use regex to parse tool call payloads, JSON configurations, or nested structures.
   • Rationale: Regex parsing is brittle to formatting variations, markdown wrapping, escaping, and presents a risk of catastrophic backtracking (ReDoS).
   • Alternatives:
     • Programmatic Parsers: Use standard JSON/JSON5 parsers or AST builders to parse code blocks or JSON payloads.
     • LLM-First Parsing: Use model-native structured output schemas (JSON schema validation) or delegate parsing to a fast, dedicated LLM helper.
3. Tool footprint matters — Every core tool costs tokens on every API call.
4. Context discipline — Implement both caps and compression.
5. Budget tracking from day one — Token + cost + iteration budgets.
6. Message role alternation — Never two same-role messages in a row.
7. Skills as user messages — Don't inject into system prompt.

Tracing & Observability Architecture

To debug agent loops, analyze performance bottlenecks, and monitor production runs, the harness must support unified tracing.

1. Tracing Standard: OpenTelemetry (OTel)

• Recommendation: Use OpenTelemetry Semantic Conventions for GenAI to decouple tracing from specific backend vendors.
• Implementation: Instrument LLM clients and tool dispatch functions with OTel Spans. Trace attributes should follow standard keys:
  • gen_ai.system (e.g., openai, anthropic)
  • gen_ai.request.model and gen_ai.response.model
  • gen_ai.usage.input_tokens and gen_ai.usage.output_tokens
  • gen_ai.request.temperature

2. Tracing Tool Integrations

• LangSmith (Commercial/Enterprise):
  • Use Case: Deep run nesting visualizer, run feedback loops (attaching feedback scores to runs), and exporting failing agent runs directly into test datasets.
  • Integration: Seamlessly enabled via environment flags (LANGCHAIN_TRACING_V2=true) and standard callback handlers.
• Langfuse (Open-Source/Self-Hostable):
  • Use Case: Zero-dependency self-hosted tracing. Offers visual traces, cost estimation, prompt version management, and SDK middleware.
  • Integration: Connect via @langfuse/typescript or langfuse Python SDK using decorators or client wrappers.
• Arize Phoenix (Open-Source/Local-First):
  • Use Case: Local Jupyter notebook evaluations and zero-config local tracer servers. Excellent for checking prompt drift and conducting cosine-similarity evals on agent retrievals.
  • Integration: Run phoenix.server.start() locally and export OTel traces directly to the local collector endpoint (http://localhost:6006/v1/traces).

3. Decoupled Middleware Pattern (Callbacks)

• Do not hardcode vendor-specific tracing code directly inside the agent loop. Instead, implement a Lifecycle Callback Registry in the runtime:

  interface AgentLifecycleCallbacks {
    onLlmStart?: (runId: string, prompt: Message[]) => void;
    onLlmEnd?: (runId: string, response: LlmResponse) => void;
    onToolStart?: (runId: string, toolName: string, args: any) => void;
    onToolEnd?: (runId: string, toolName: string, result: any) => void;
    onException?: (runId: string, error: Error) => void;
  }
  

  This callback interface broadcasts events to active tracers (LangSmith, Langfuse, OTel) without leaking tracking libraries into cognitive reasoning files.

Multi-Model Deliberation & Fusion Recommendations

When to Use Multi-Model Deliberation

Recommended Implementation

1. Gateway-Level Fusion Tool: Register harness__fusion as a gateway-injected tool. The primary model decides when to invoke deliberation — it is not forced on every request CLAIM-145.
2. Budget Panels First: Start with 3× Flash-tier panel + 1 frontier judge. Budget panels outperform standalone frontier models on DRACO at ~50% cost CLAIM-157.
3. Structured JSON Judge Output: Judge must produce structured analysis (consensus, contradictions, blind spots) — not freeform merge CLAIM-145.
4. Anonymize Panel Responses: Strip model identifiers before judge evaluation to prevent lab-bias CLAIM-150.
5. Recursion Protection: Depth headers prevent infinite nested fusion calls CLAIM-146.
6. Promise.allSettled() for Panel Dispatch: Partial panel failures should not abort the entire deliberation.

Framework Selection for Deliberation

For detailed research including taxonomy, self-hosted implementation code, anti-patterns, benchmarks, and decision matrices, see multi_model_deliberation_and_swarms.md.

Generative UI & MCP UI Recommendations

Core Recommendations for Dynamic UIs

Framework Selection for Agent UIs

• Vercel AI SDK (AI SDK UI): Use for progressive JSON token stream rendering, letting client components render layouts immediately as parameter nodes materialize CLAIM-172.
• CopilotKit: Use for active state updates (AG-UI protocol) syncing client components directly with background agent states CLAIM-173.
• Mastra AI & mcp-use: Use to construct and expose rich visual tools on local/remote MCP servers CLAIM-176.

For detailed security guidelines, postMessage schemas, and Tasks lifecycles, see the dedicated document: mcp_apps_and_ui.md.

Human-in-the-Loop & Execution Control Recommendations

Core HITL Design Guidelines

For detailed code paradigms, cascading cancellation fixes, and framework implementations, see the dedicated document: human_in_the_loop_steering.md.

Agent Scratchpad & Graph-Based Session Memory Recommendations

Core Memory & Scratchpad Design Guidelines

For detailed research covering scratchpad patterns, auto-memory logs, and knowledge graphs, see: agent_scratchpads_and_session_memory.md.

Agent Self-Improvement & Curation Recommendations

Core Self-Improvement & Curation Guidelines

For detailed research covering curation engines, preferences, and RISE/TT-SI loops, see: self_improving_agents_and_learning_loops.md.

Open Questions for Implementation

1. Python or TypeScript for agent core? — Python has ecosystem, TS has full-stack advantage.
2. Monolithic or distributed? — Hermes monolith is simpler, OpenClaw gateway scales better.
3. Which model backend first? — Ollama for local dev, OpenRouter for quick multi-model access, LiteLLM when you need self-hosted auth/budgets at scale.
4. Graph-based or while-loop? — While-loop for v1, graph for v2?
5. Which memory providers to support first? — Start with SQLite + one cloud provider?