Paladin Documentation

Welcome to the Paladin documentation! Paladin is a Rust-based enterprise multi-agent orchestration framework built with Hexagonal Architecture and Domain-Driven Design principles.

🚀 Getting Started

New to Paladin? Start here:

1. Quickstart Guide - Get your first Paladin agent running in 15 minutes
2. Installation - Detailed setup instructions for all platforms
3. Examples Gallery - Working code examples for common use cases

📚 User Guides

Learn how to build with Paladin:

• Autonomous Agent Features - Auto-planning, prompt generation, dynamic temperature, and agent handoffs
• Battalion Orchestration - Multi-agent coordination with orchestration patterns
• Maneuver Flow DSL - Declarative workflows with Flow DSL syntax
• Tool Integration (Arsenal) - Integrate external tools via MCP protocol
• Memory Management (Garrison) - Conversation context and persistence
• Output Formatting (Herald) - Format and stream agent responses
• WarEngine: Superstep Execution - Battlefield state, Waypoint checkpointing, and cyclic graph execution
• Control Flow - Dynamic routing and subgraphs
• Parley & Chronicle - Pause, resume, history and graceful shutdown
• Aegis: Fault Tolerance - Retry, timeout, error handlers, model fallback and node caching
• Agent Runtime - Middleware, context management, vault memory, structured output and the reasoning agent
• Eval Harness - Evaluate Paladin agent quality and catch regressions
• CLI Usage Guide - Complete command-line interface reference

🏗️ Architecture

Understand Paladin's design:

• Architecture Overview - Three-layer hexagonal architecture
• Hexagonal Design - Port/adapter pattern implementation
• Domain Model - DDD entities and relationships
• Commissary - Input-side, per-call window-rationing officer
• Design Patterns - Patterns used throughout Paladin

🧭 Deployment Topologies

Choose how to run Paladin:

• Choosing a Topology - Compare embedded library, orchestrated Battalion, HTTP service host, queue/worker and sidecar deployments

🚢 Deployment

Deploy Paladin to production:

• Docker - Containerized deployment
• Kubernetes - Cloud-native orchestration
• CI/CD - Automated pipelines with GitHub Actions
• Production Best Practices - Security, scaling, and reliability
• Platform API - Runs, threads, assistants, schedules and webhooks REST surface
• Versioning Policy - Lockstep versioning rules and transition criteria
• Release Checklist - Dependency-aware release and publish workflow

🔧 Operations

Monitor and maintain Paladin:

• Observability - Traces, sinks and persistence
• Logging - Structured logging configuration
• Monitoring - Metrics and dashboards
• Troubleshooting - Common issues and solutions
• Performance Tuning - Optimize for throughput and latency

🤝 Contributing

Extend and improve Paladin:

• Contribution Guide - How to contribute
• Adapter Development - Create custom adapters
• Testing Guide - Testing requirements and patterns

📖 API Reference

Comprehensive API documentation is available via rustdoc:

cargo doc --open

Or browse online at: https://docs.rs/paladin (when published)

🎯 Key Concepts

Medieval Military Theme

Paladin uses a consistent Medieval Military naming convention. This is a short excerpt — see Domain Model for the complete ubiquitous-language table of every Medieval-Military term Paladin uses:

Architecture Layers

Paladin follows hexagonal (ports and adapters) architecture:

• Core Layer - Pure domain logic, no external dependencies
• Application Layer - Use cases and port definitions (interfaces)
• Infrastructure Layer - Adapter implementations for external systems

Dependencies flow inward only: Infrastructure → Application → Core

💡 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: You're reading it!

📄 License

See LICENSE for details.

Installation

This guide covers adding Paladin to an existing Rust project or setting up the Paladin workspace for development.

Prerequisites

Required

Why Rust >= 1.88? Paladin uses edition 2024 features. Verify your toolchain:

rustc --version   # should print >= 1.88.0

Update with rustup update stable.

Optional (for Docker-based services)

• Docker + Docker Compose v2 -- required for the built-in Redis, MinIO, and MySQL services (see Docker Guide)

Installing Rust

# Install rustup and the stable toolchain
curl --proto =https --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env

# Build tools -- Linux (Ubuntu/Debian)
sudo apt-get install -y build-essential pkg-config libssl-dev

# Build tools -- macOS (via Homebrew)
brew install openssl pkg-config

Windows users should use rustup-init.exe or WSL 2.

Adding Paladin to a Rust Project

Cargo.toml -- choose your crates

Paladin v0.10.0 is published as a workspace of focused crates. Add only what you need:

[dependencies]
# Core framework -- always required
paladin-ai-core   = "0.10.0"
paladin-ports     = "0.10.0"

# LLM providers (pick one or more)
paladin-llm       = { version = "0.10.0", features = ["llm-openai"] }

# Multi-agent orchestration (optional)
paladin-battalion = "0.10.0"

# Memory / Garrison (optional)
paladin-memory    = "0.10.0"

# Storage adapters (optional)
paladin-storage   = "0.10.0"

# Async runtime (required)
tokio = { version = "1", features = ["full"] }

Umbrella crate

The paladin-ai umbrella crate (v0.10.0) re-exports everything and accepts workspace feature flags:

[dependencies]
paladin-ai = { version = "0.10.0", features = ["redis-queue", "s3-storage"] }
tokio      = { version = "1", features = ["full"] }

Feature Flag Profiles

A minimal profile for the common getting-started case -- the three default LLM providers plus the adapters most guides exercise:

Full feature inventory

Regenerated from the facade Cargo.toml [features] block -- every shipped flag, the crate it forwards into, and what it gates. default = ["llm-openai", "llm-anthropic", "llm-deepseek"].

The paladin-cli binary carries required-features = ["cli"] in Cargo.toml, so it is not built by a default cargo build -- pass --features cli (or --bin paladin-cli --features cli) to build it.

Verification

cargo check

No errors means all selected features resolved correctly.

Cloning the Source for Development

# 1. Clone
git clone https://github.com/DF3NDR/paladin-dev-env.git
cd paladin-dev-env

# 2. Build the workspace
cargo build

# 3. Run unit tests
cargo test --workspace --lib

# 4. (Optional) Start backing services
make services-up   # Redis, MinIO, MySQL via Docker Compose

See Development Setup for the full contributor workflow.

Environment Variables for LLM Keys

Paladin reads API keys exclusively from environment variables -- never put keys in config files.

# Set at least one provider key before running
export OPENAI_API_KEY="sk-..."       # OpenAI
export DEEPSEEK_API_KEY="sk-..."     # DeepSeek
export ANTHROPIC_API_KEY="sk-..."    # Anthropic

Copy .env.example to .env for local development (.env is git-ignored).

Next Steps

• Quickstart -- write your first Paladin agent in minutes
• Configuration -- full config.yml schema reference
• User Guides -- in-depth agent patterns

Quickstart

Get a Paladin agent running in under 15 minutes.

Prerequisites

Complete Installation first and set your LLM API key:

export OPENAI_API_KEY="sk-..."

Create a New Project

cargo new my-paladin-agent
cd my-paladin-agent

Add Paladin to Cargo.toml:

[dependencies]
paladin-ai    = "0.10.0"
paladin-ports = "0.10.0"
paladin-llm   = { version = "0.10.0", features = ["openai"] }
tokio         = { version = "1", features = ["full"] }

Your First Paladin Agent

Replace src/main.rs with the following:

// src/main.rs -- Hello, Paladin!
use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin::application::services::paladin::paladin_execution_service::PaladinExecutionService;
use paladin::infrastructure::resilience::circuit_breaker::CircuitBreaker;
use paladin_ports::output::llm_port::LlmPort;
use paladin_llm::openai::OpenAIAdapter;
use std::sync::Arc;
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // 1. Create an LLM adapter (reads OPENAI_API_KEY from env)
    let llm_port: Arc<dyn LlmPort> = Arc::new(OpenAIAdapter::from_env()?);

    // 2. Build the Paladin using the fluent builder
    let paladin = PaladinBuilder::new(llm_port.clone())
        .system_prompt("You are a concise and helpful assistant.")
        .name("HelloPaladin")
        .model("gpt-4")
        .temperature(0.7)
        .max_loops(1)
        .build()
        .await?;

    // 3. Create an execution service (the circuit breaker guards against cascading failures)
    let circuit_breaker = Arc::new(CircuitBreaker::new(3, 2, Duration::from_secs(30)));
    let service = PaladinExecutionService::new(llm_port, circuit_breaker, None, None);

    // 4. Execute with a prompt
    let result = service.execute(&paladin, "Say hello in one sentence.").await?;

    println!("Output : {}", result.output);
    println!("Tokens : {}", result.usage.total_tokens);
    println!("Time   : {}ms", result.execution_time_ms);

    Ok(())
}

Run it:

cargo run

Expected output (exact wording varies):

Output : Hello! I am your AI assistant, ready to help.
Tokens : 18
Time   : 342ms

Running the Built-in Examples

The Paladin workspace ships with ready-to-run examples:

# Clone the workspace if you haven't already
git clone https://github.com/DF3NDR/paladin-dev-env.git
cd paladin-dev-env

# Start backing services (Redis, MinIO) -- optional for basic examples
make services-up

# Run the basic Paladin example
cargo run --example basic_paladin

# Sequential multi-agent pipeline
cargo run --example formation_sequential

# Concurrent multi-agent execution
cargo run --example phalanx_parallel

Understanding the Output

PaladinExecutionService::execute returns a PaladinResult with these fields:

What's Next?

Configuration

Paladin is configured via a YAML file (config.yml by default) and environment variables. Environment variables take precedence over file values and use the APP_ prefix format shown throughout this guide.

Loading Configuration

// Load from the default config.yml in the current directory
let settings = paladin_ai_core::config::ApplicationSettings::load()?;

// Or specify a path
let settings = paladin_ai_core::config::ApplicationSettings::from_file("config.yml")?;

LLM Provider

Nine providers are supported: openai, anthropic, deepseek, kimi, qwen, grok, ollama, gemini, and a generic operator-configured openai-compatible provider for any other OpenAI-compatible endpoint. Only the providers compiled in via the matching llm-<provider> Cargo feature are usable at runtime; the compiled default remains openai + anthropic + deepseek (see Feature Flags).

llm:
  default_provider: "openai"   # openai | anthropic | deepseek | kimi | qwen | grok | ollama | gemini | openai-compatible

  openai:
    base_url: "https://api.openai.com/v1"
    default_model: "gpt-4"
    default_temperature: 0.7
    timeout_seconds: 300
    max_retries: 3

  deepseek:
    base_url: "https://api.deepseek.com/v1"
    default_model: "deepseek-chat"
    default_temperature: 0.7
    timeout_seconds: 300
    max_retries: 3

  anthropic:
    base_url: "https://api.anthropic.com/v1"
    default_model: "claude-3-5-sonnet-20241022"
    default_temperature: 0.7
    timeout_seconds: 300
    max_retries: 3

  # Six providers added alongside the original three. Each carries its own dated
  # verification status below rather than one blanket disclaimer — see "Live
  # verification status" further down for what was confirmed and when.
  #
  # Kimi (Moonshot AI). Live-verified 2026-08-22 (plan 17-19, closing G-17-4b): GET
  # /models and a generate() round trip both succeeded against api.moonshot.ai.
  kimi:
    base_url: "https://api.moonshot.ai/v1"
    default_model: "kimi-k3"
    timeout_seconds: 60

  # Qwen (Alibaba DashScope). Live-verified 2026-08-23 (plan 17-21 gap closure): GET
  # /models returned a 162-model catalog at the endpoint below, including the default
  # model, and a generate() round trip succeeded.
  #
  # DashScope API keys are scoped to the Model Studio region that issued them and are
  # REJECTED by every other region's endpoint. `base_url` below is Singapore, the
  # shipped default. If your workspace is in the US or on the mainland, you MUST
  # set `DASHSCOPE_BASE_URL` to your own region's endpoint:
  #   - Singapore      (shipped default): https://dashscope-intl.aliyuncs.com/compatible-mode/v1
  #   - US (Virginia):                    https://dashscope-us.aliyuncs.com/compatible-mode/v1
  #   - China (mainland):                 https://dashscope.aliyuncs.com/compatible-mode/v1
  qwen:
    base_url: "https://dashscope-intl.aliyuncs.com/compatible-mode/v1"
    default_model: "qwen-plus"
    timeout_seconds: 60

  # Grok (xAI). Live-verified 2026-08-22 (plan 17-18, closing G-17-4a): GET /models and
  # a generate() round trip both succeeded against api.x.ai.
  grok:
    base_url: "https://api.x.ai/v1"
    default_model: "grok-4.6"
    timeout_seconds: 60

  # Ollama (self-hosted) requires no api_key at all (D-12) — omit the field entirely.
  # Not applicable to live-vendor verification: self-hosted, no vendor endpoint to
  # verify. Its live exercise is the Docker Tier 2 suite (UAT test 3), passed on a
  # GitHub Actions runner 2026-08-19.
  ollama:
    base_url: "http://localhost:11434/v1"
    default_model: "llama3"
    timeout_seconds: 60

  # Gemini uses a bespoke `generateContent` protocol, not OpenAI-compatible.
  # Live-verified: GET /models and a generate() round trip both succeeded against
  # generativelanguage.googleapis.com. default_model was refreshed from
  # gemini-2.5-flash (retired for new users) to gemini-3.6-flash per the live catalog.
  gemini:
    base_url: "https://generativelanguage.googleapis.com/v1beta"
    default_model: "gemini-3.6-flash"
    timeout_seconds: 60

  # Generic adapter for ANY OpenAI-compatible endpoint not named above (self-hosted
  # vLLM/LiteLLM, Groq, Together, Mistral, Fireworks, Bedrock's OpenAI-compat mode, ...).
  # base_url and default_model are REQUIRED here — there is no vendor default.
  openai-compatible:
    base_url: "https://your-endpoint.example.com/v1"
    default_model: "your-model-name"
    timeout_seconds: 60

API keys are read exclusively from environment variables:

Security: Never put API keys in config.yml. Use environment variables or a secrets manager (AWS Secrets Manager, HashiCorp Vault, Kubernetes Secrets).

Live verification status

Per-vendor, not one blanket disclaimer — each provider's base_url and default_model above carry their own dated status:

Running against a local Ollama server (RT-FR-22)

Ollama is the only provider in the table above that requires no vendor API key (D-12) — it is also the only one you can run entirely on your own machine. Two commands get a model serving locally:

ollama serve                 # starts the local server (default: http://localhost:11434)
ollama pull llama3           # pulls the model named in the Ollama config block above

Point Paladin at it with the Ollama configuration block already shown above under LLM Provider — base_url and default_model there are exactly what ollama serve/ollama pull produce by default. To override the base URL without editing config.yml (a different port, a remote host, a container), set OLLAMA_BASE_URL; it follows the same environment-variable-overrides-file precedence as every other provider on this page.

A minimal reasoning_agent preset call against a local Ollama server, once it is running (the exact signature per 26-CONTEXT.md D-35 — paladin::presets and ReasoningAgent do not exist in the tree yet as of this plan; the runnable, doc-tested version of this snippet arrives with the agent-runtime user guide, plan 26-21):

use paladin::presets::{reasoning_agent, ReasoningAgentOptions};

// `llm` is any Arc<dyn LlmPort> -- for example the adapter built from the Ollama
// configuration block above (base_url "http://localhost:11434/v1", model "llama3").
let agent = reasoning_agent(llm, arsenal, ReasoningAgentOptions::default())?;
let result = agent.run("What is the capital of France?").await?;

Verifying the adapter against a real Ollama server is an existing suite, not a new one (D-32). tests/integration/ollama_docker_test.rs already is the "ignored-by-default integration test gated on an env var" RT-FR-22 asks for: it is required-features-gated on integration-tests + llm-ollama, every test in it independently probes OLLAMA_TEST_URL before doing anything else, and it prints a named SKIP: reason and returns early — never panics or hangs — when the service is unreachable. Run it with:

cargo test --test ollama_docker --features integration-tests,llm-ollama

Locally this will almost always print SKIP: ... and pass without exercising anything, because OLLAMA_TEST_URL is unset by default — that is the intended, documented behavior for a plain cargo test, not a failure. The suite is exercised for real only in CI's ollama-integration job (.github/workflows/ci.yml, the ollama-integration job near line 748), which brings up a Docker Compose ollama-test service, waits for it to report healthy, pulls qwen2.5:0.5b, runs this exact suite against it, and separately fails the job if any test took the SKIP: path — so a green run there is proof the live server was actually exercised, not proof-by-absence. Do not add a second Ollama integration test file — this is the one, and none should be added.

A rejected credential now announces itself

Every provider above except Ollama shares one underlying protocol engine (CompatEngine), so this applies uniformly to all of them — an operator debugging a self-hosted OpenAI-compatible endpoint gets the same signal as one debugging DashScope, Moonshot or xAI (2026-08-22, plan 17-22, closing G-17-4d).

Before this change, a rejected credential and an offline vendor looked identical: the model-list fetch silently fell back to a curated list with nothing above a debug log line, in either case. This is what let a genuine credential/region mismatch go undiagnosed for five days during this phase's own live verification (.planning/WINDOWS.md gap history). Now, when the configured endpoint rejects the request (an authentication failure), a warn-level line is emitted naming the endpoint and stating that the returned list is the curated fallback, not the vendor's own catalog — for example:

[WARN] configured endpoint https://dashscope-intl.aliyuncs.com/compatible-mode/v1 rejected
the request while listing models (Authentication failed: ...); the returned model list is
the curated fallback, not this vendor's own catalog — a credential scoped to a different
account or region is the usual cause

An endpoint that is simply unreachable — a self-hosted Ollama that has not started yet, a network blip, a slow response — stays at debug, exactly as before: being offline is a supported state (D-13/D-14), not a misconfiguration, and this diagnostic does not fire for it. Seeing the warning at all means the fix is the same one described throughout this page: check the configured base_url/*_BASE_URL override against the credential you are using.

Environment variables

The full LLM environment-variable surface — every credential, base-URL, model, timeout and (for the generic openai-compatible provider) capability/temperature override the adapters read — is documented as a first-class configuration path, alongside the YAML above, in .env.example at the repository root. Copy it to .env and fill in the credentials you need; unset variables fall back to the defaults shown in this guide.

In the devcontainer, these credentials arrive from ~/.config/paladin/ (one file per secret, filename = the lowercased variable name — e.g. ~/.config/paladin/xai_api_key → XAI_API_KEY) via .devcontainer/paladin-env.sh, sourced automatically into interactive shells by ~/.bashrc. A genuinely-exported non-empty value always wins over the file. A non-interactive shell (a script, a CI step, an agent's Bash tool) does not run ~/.bashrc and therefore does not source paladin-env.sh automatically — it must be sourced explicitly: set -a; . .devcontainer/paladin-env.sh; set +a.

Garrison (Short-term Memory)

The Garrison stores conversation context between Paladin turns.

garrison:
  garrison_type: "in_memory"       # in_memory | sqlite
  # path: "./garrison.db"          # Required when garrison_type = "sqlite"
  max_entries: 100                  # Max conversation turns to retain
  max_tokens: 4000                  # Context-window token budget
  tokenizer: "gpt-4"               # Model name for token counting
  eviction_strategy: "importance_based"  # importance_based | fifo | sliding_window
  preserve_recent_count: 10        # Always keep at least N recent entries

Env vars: APP_GARRISON_TYPE, APP_GARRISON_PATH, APP_GARRISON_MAX_ENTRIES, APP_GARRISON_MAX_TOKENS, APP_GARRISON_EVICTION_STRATEGY, APP_GARRISON_PRESERVE_RECENT_COUNT

Sanctum (Long-term Vector Memory)

Sanctum stores semantic memories in a vector database for RAG.

sanctum:
  enabled: false
  adapter_type: "in_memory"        # in_memory | qdrant

  qdrant:                          # Required when adapter_type = "qdrant"
    url: "http://localhost:6334"
    collection_name: "paladin_memories"
    vector_dimension: 1536         # Must match your embedding model

rag:
  top_k: 5                         # Results to retrieve
  min_similarity: 0.7              # Score threshold (0.0-1.0)
  max_tokens: 2000                 # Max tokens to inject from RAG -- enforced by the
                                    # Commissary (Phase 33): shed memories are recorded
                                    # and a marker naming the omitted count and the
                                    # budget is appended to the injected context.
  timeout_seconds: 5

memory_extraction:
  enabled: true
  strategy: "on_completion"        # every_turn | on_completion | manual

Env vars: APP_SANCTUM_ENABLED, APP_SANCTUM_ADAPTER_TYPE, APP_SANCTUM_QDRANT_URL, APP_SANCTUM_QDRANT_COLLECTION_NAME, APP_SANCTUM_QDRANT_VECTOR_DIMENSION

See Sanctum Vector Memory for detail.

Arsenal (Tool System / MCP)

The Arsenal connects Paladins to external tools via the Model Context Protocol.

arsenal:
  default_timeout_seconds: 30
  max_concurrent_tools: 5
  mcp_servers:
    # STDIO server (command-line process)
    - name: "web_search"
      server_type: "stdio"
      command: "uvx"
      args: ["mcp-web-search"]

    # Streamable-HTTP server (remote, optionally authenticated)
    - name: "code_analyzer"
      server_type: "streamable_http"
      endpoint: "http://localhost:8080/mcp"
      # NAMES the env var holding the bearer token -- never a literal
      # secret in this file. Omit entirely for an unauthenticated server.
      auth_token_env: "CODE_ANALYZER_TOKEN"

Env vars: APP_ARSENAL_DEFAULT_TIMEOUT_SECONDS, APP_ARSENAL_MAX_CONCURRENT_TOOLS

See Arsenal & Tools for full integration guide.

Citadel (State Persistence)

Citadel saves Paladin state to disk for crash recovery and resumption.

citadel:
  enabled: false
  state_dir: "./paladin-states"
  autosave_enabled: false          # Save state after each execution
  cleanup_enabled: false           # Delete old state files automatically
  max_state_age_days: 30

Env vars: APP_CITADEL_ENABLED, APP_CITADEL_STATE_DIR, APP_CITADEL_AUTOSAVE_ENABLED, APP_CITADEL_CLEANUP_ENABLED, APP_CITADEL_MAX_STATE_AGE_DAYS

Battalion (Multi-agent Orchestration)

battalion:
  default_timeout_seconds: 300     # Per-battalion execution timeout
  error_strategy: "fail_fast"      # fail_fast | continue_on_error | retry_then_continue
  max_concurrent_paladins: 10      # Phalanx concurrency limit
  metadata_output_enabled: false   # Write execution metadata to files

  retry:                           # Used when error_strategy = retry_then_continue
    max_attempts: 3
    exponential_backoff: true
    jitter: true
    base_delay_ms: 100
    max_delay_seconds: 10

  maneuver:                        # Flow DSL (Maneuver pattern)
    error_strategy: "fail_fast"    # fail_fast | continue_parallel | ignore_errors
    output_format: "combined_text" # combined_text | structured_json
    pass_output_as_input: true
    timeout_seconds: 300
    collect_timing_metrics: true
    max_agents: 30
    max_depth: 5

Env vars: APP_BATTALION_DEFAULT_TIMEOUT_SECONDS, APP_BATTALION_ERROR_STRATEGY, APP_BATTALION_MAX_CONCURRENT_PALADINS, APP_BATTALION_RETRY_MAX_ATTEMPTS, etc.

See Battalion Patterns for Formation, Phalanx, Campaign, and Chain of Command details.

Herald (Output Formatting)

herald:
  default_formatter: "json"        # json | markdown | table

  json:
    pretty: true
    include_metadata: true

  markdown:
    include_colors: true
    heading_level: 2

  table:
    max_column_width: 60
    border_style: "rounded"        # ascii | rounded | modern | sharp | none

Env vars: APP_HERALD_DEFAULT_FORMATTER, APP_HERALD_JSON_PRETTY, APP_HERALD_MARKDOWN_INCLUDE_COLORS, APP_HERALD_TABLE_BORDER_STYLE

Token Budget Terminology

Paladin uses max_tokens in four independent, non-overlapping senses:

Any future spend-governance cap uses a distinct key, allowance, never max_tokens.

Autonomous Features

All autonomous features are opt-in (disabled by default). Uncomment sections in config.yml to enable:

autonomous:
  planning:
    enabled: false          # Decompose complex tasks into subtasks
    max_subtasks: 10

  prompt_generation:
    enabled: false          # Auto-generate system prompts from description
    description: null       # e.g. "Expert data analyst"

  dynamic_temperature:
    enabled: false          # Adjust temperature per task type
    min: 0.1
    max: 0.9

  handoffs:
    enabled: false          # Delegate to specialist Paladins
    strategy: "automatic"   # automatic | explicit | {threshold: 0.8}
    max_depth: 5

Env vars: APP_AUTONOMOUS_PLANNING_ENABLED, APP_AUTONOMOUS_PLANNING_MAX_SUBTASKS, APP_AUTONOMOUS_PROMPT_GENERATION_ENABLED, APP_AUTONOMOUS_DYNAMIC_TEMPERATURE_ENABLED, APP_AUTONOMOUS_HANDOFFS_ENABLED, APP_AUTONOMOUS_HANDOFFS_STRATEGY

Multi-Environment Pattern

Keep a config.yml for defaults and override per environment:

# Development
export APP_LLM_DEFAULT_PROVIDER=openai
export APP_GARRISON_TYPE=in_memory

# Staging
export APP_GARRISON_TYPE=sqlite
export APP_GARRISON_PATH=/data/garrison.db
export APP_SANCTUM_ENABLED=true

# Production
export APP_GARRISON_TYPE=sqlite
export APP_SANCTUM_ENABLED=true
export APP_SANCTUM_ADAPTER_TYPE=qdrant
export APP_CITADEL_ENABLED=true
export APP_CITADEL_AUTOSAVE_ENABLED=true

Complete Example (config.yml)

llm:
  default_provider: "openai"
  openai:
    default_model: "gpt-4"
    default_temperature: 0.7

garrison:
  garrison_type: "sqlite"
  path: "./garrison.db"
  max_entries: 200
  max_tokens: 8000

arsenal:
  default_timeout_seconds: 30
  max_concurrent_tools: 5
  mcp_servers:
    - name: "web_search"
      server_type: "stdio"
      command: "uvx"
      args: ["mcp-web-search"]

battalion:
  error_strategy: "retry_then_continue"
  max_concurrent_paladins: 10
  retry:
    max_attempts: 3
    exponential_backoff: true

herald:
  default_formatter: "markdown"

See Also

• Installation
• Quickstart
• Garrison Memory
• Arsenal & Tools
• Sanctum Vector Memory
• Battalion Patterns

Paladin Agents

A Paladin is Paladin AI's core autonomous agent entity — an LLM-powered reasoner that operates a configurable reasoning loop, maintains conversation memory via Garrison, executes external tools via Arsenal, and optionally leverages autonomous features like task planning, auto-generated prompts, and dynamic temperature.

Ready to run a number of agents? See Deployment Topologies for how to choose between embedding, hosting, queue/worker, and sidecar models.

Table of Contents

1. Quick Start
2. PaladinBuilder API
3. Execution Model
4. PaladinResult Fields
5. StopReason Variants
6. Autonomous Features
7. Memory — Garrison
8. Tools — Arsenal
9. Output Formatting — Herald
10. Configuration Reference
11. Error Handling
12. Best Practices

Quick Start

Add the paladin-ai crate and enable any desired feature flags:

[dependencies]
paladin-ai = { version = "0.10.0", features = ["llm-openai"] }
tokio = { version = "1", features = ["full"] }

Build and execute a Paladin:

use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin_ports::output::llm_port::LlmPort;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Construct an LLM adapter (e.g., OpenAI)
    let llm_port: Arc<dyn LlmPort> = Arc::new(openai_adapter());

    // Build the Paladin
    let paladin = PaladinBuilder::new(llm_port)
        .system_prompt("You are a helpful assistant.")
        .name("Assistant")
        .model("gpt-4o")
        .temperature(0.7)
        .max_loops(3)
        .timeout_seconds(120)
        .build()
        .await?;

    // Execute
    let result = paladin
        .execute("Explain the Rust ownership model in one paragraph.")
        .await?;

    println!("{}", result.output);
    println!("Tokens used: {}", result.usage.total_tokens);
    println!("Stop reason: {:?}", result.stop_reason);
    Ok(())
}

PaladinBuilder API

PaladinBuilder is located at src/application/services/paladin/paladin_builder.rs. All methods are fluent (return Self). Call .build().await? at the end.

Core Configuration

Integrations

Autonomous Features

Execution Model

A Paladin's inner reasoning loop:

┌─────────────────────────────────────────────────────────────────┐
│  1. Build Prompt                                                 │
│     System prompt + Garrison history + User input               │
├─────────────────────────────────────────────────────────────────┤
│  2. LLM Call (via LlmPort)                                       │
│     Generate response from the configured model                  │
├─────────────────────────────────────────────────────────────────┤
│  3. Check Stop Conditions                                        │
│     • Stop word detected in output?  → StopWord(word)           │
│     • loop_count ≥ max_loops?        → MaxLoops                 │
│     • Elapsed > timeout?             → Timeout                  │
├─────────────────────────────────────────────────────────────────┤
│  4. Tool Execution (if Arsenal attached)                        │
│     Parse tool-call JSON in response → invoke via ArsenalPort   │
│     Append tool result to context                                │
├─────────────────────────────────────────────────────────────────┤
│  5. Update Garrison                                              │
│     Store assistant turn and any tool results                    │
├─────────────────────────────────────────────────────────────────┤
│  6. Loop or Complete                                             │
│     If no stop condition: loop_count++ → back to step 1         │
│     Otherwise: build PaladinResult and return                   │
└─────────────────────────────────────────────────────────────────┘

PaladinResult Fields

Returned by execute() and streamed by execute_stream().

Check completeness:

if result.stop_reason.is_successful() {
    println!("Complete output: {}", result.output);
} else {
    println!("Partial output ({}): {}", result.stop_reason, result.output);
}

StopReason Variants

Autonomous Features

All autonomous features are opt-in (disabled by default) to maintain backward compatibility.

Autonomous Planning (MaxLoops::Auto)

When enabled, the Paladin uses an LLM call to decompose the user's task into subtasks before executing them sequentially.

use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin::core::platform::container::paladin::MaxLoops;

let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are a research assistant.")
    .enable_autonomous_planning(true)
    // max_loops controls the subtask cap when using auto planning:
    .max_loops(10)
    .build()
    .await?;

The PaladinResult.plan field contains the TaskPlan with each subtask's description and result.

Auto-Generated System Prompts

Instead of writing a system prompt manually, provide an agent description and let the LLM generate an optimized prompt:

let paladin = PaladinBuilder::new(llm_port)
    .agent_description("Expert in Rust async programming and tokio runtime")
    .enable_autonomous_prompts(true)
    .build()
    .await?;

Tip: Calling .system_prompt(...) on the same builder disables auto-generation for that instance — the manual prompt always takes precedence.

Dynamic Temperature

Temperature increases linearly from the configured base value toward 1.0 over the reasoning loops. This encourages broader exploration in later iterations when the agent may be stuck:

let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are a problem solver.")
    .temperature(0.3)          // Start temperature
    .max_loops(5)
    .enable_dynamic_temperature(true)  // Reaches ~1.0 by loop 5
    .build()
    .await?;

Agent Handoffs

A Paladin can delegate sub-tasks to specialist agents at runtime using the Arsenal handoff tool. Register specialist agents on the builder via with_handoffs, which takes the whole Vec<Arc<Paladin>> at once — there is no per-call chainable specialist-registration method:

use paladin_core::platform::container::garrison::GarrisonConfig;
use paladin_memory::garrison::in_memory_garrison::InMemoryGarrison;
use paladin_ports::output::garrison_port::GarrisonPort;

/// Attach an `InMemoryGarrison` (the one-argument `new(config)` constructor — there is
/// no zero-argument form) for conversation memory, and register specialist agents for
/// delegation via `with_handoffs`, which takes the whole `Vec<Arc<Paladin>>` at once —
/// there is no per-call chainable `with_specialist` method.
pub async fn attach_garrison() -> Result<Paladin, Box<dyn std::error::Error>> {
    let llm_port: Arc<dyn LlmPort> = Arc::new(MockLlmAdapter::new());

    let garrison_config = GarrisonConfig::default();
    let garrison: Arc<dyn GarrisonPort> = Arc::new(InMemoryGarrison::new(garrison_config));

    let code_reviewer = PaladinBuilder::new(llm_port.clone())
        .system_prompt("You review Rust code for correctness and style.")
        .name("CodeReviewer")
        .build()
        .await?;
    let security_auditor = PaladinBuilder::new(llm_port.clone())
        .system_prompt("You audit code for security vulnerabilities.")
        .name("SecurityAuditor")
        .build()
        .await?;

    let paladin = PaladinBuilder::new(llm_port)
        .system_prompt("You are a memory-enabled coordinator.")
        .with_garrison(garrison)
        .with_handoffs(vec![Arc::new(code_reviewer), Arc::new(security_auditor)])
        .build()
        .await?;

    Ok(paladin)
}

Delegation records appear in PaladinResult.handoff_history.

Memory — Garrison

Attach a Garrison adapter to give the Paladin persistent conversation memory. InMemoryGarrison takes a required GarrisonConfig argument — there is no zero-argument constructor:

use paladin_core::platform::container::garrison::GarrisonConfig;
use paladin_memory::garrison::in_memory_garrison::InMemoryGarrison;
use paladin_ports::output::garrison_port::GarrisonPort;

/// Attach an `InMemoryGarrison` (the one-argument `new(config)` constructor — there is
/// no zero-argument form) for conversation memory, and register specialist agents for
/// delegation via `with_handoffs`, which takes the whole `Vec<Arc<Paladin>>` at once —
/// there is no per-call chainable `with_specialist` method.
pub async fn attach_garrison() -> Result<Paladin, Box<dyn std::error::Error>> {
    let llm_port: Arc<dyn LlmPort> = Arc::new(MockLlmAdapter::new());

    let garrison_config = GarrisonConfig::default();
    let garrison: Arc<dyn GarrisonPort> = Arc::new(InMemoryGarrison::new(garrison_config));

    let code_reviewer = PaladinBuilder::new(llm_port.clone())
        .system_prompt("You review Rust code for correctness and style.")
        .name("CodeReviewer")
        .build()
        .await?;
    let security_auditor = PaladinBuilder::new(llm_port.clone())
        .system_prompt("You audit code for security vulnerabilities.")
        .name("SecurityAuditor")
        .build()
        .await?;

    let paladin = PaladinBuilder::new(llm_port)
        .system_prompt("You are a memory-enabled coordinator.")
        .with_garrison(garrison)
        .with_handoffs(vec![Arc::new(code_reviewer), Arc::new(security_auditor)])
        .build()
        .await?;

    Ok(paladin)
}

Available Garrison adapters (in crates/paladin-memory/):

See Garrison Memory for full documentation.

Tools — Arsenal

Attach an Arsenal registry backed by MCP (Model Context Protocol) servers:

use paladin_ports::output::arsenal_port::ArsenalRegistry;

// Registry pre-loaded from config.yml arsenal.mcp_servers section
let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are a web researcher with tool access.")
    .with_arsenal_registry(arsenal_registry)
    .build()
    .await?;

See Arsenal Tools for MCP server configuration and custom tool implementation.

Output Formatting — Herald

Format execution results using a Herald adapter:

use paladin::infrastructure::adapters::herald::JsonHerald;
use paladin_core::platform::container::herald::Herald;

let herald: Arc<dyn Herald> = Arc::new(JsonHerald::default());

let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are an API assistant.")
    .with_herald(herald)
    .build()
    .await?;

See Herald Output for available formatters.

Configuration Reference

All builder values can also be set through config.yml:

paladin:
  default_model: "gpt-4o"
  default_temperature: 0.7
  default_max_loops: 3
  timeout_seconds: 300
  retry_attempts: 3

autonomous:
  planning:
    enabled: false
    max_subtasks: 10
  prompt_generation:
    enabled: false
  dynamic_temperature:
    enabled: false
  handoffs:
    enabled: false
    max_depth: 3

See Configuration for the full schema.

Error Handling

PaladinError variants from paladin_core::platform::container::paladin_error:

Best Practices

• Always set a system prompt that clearly defines the agent's role and constraints.
• Set timeout_seconds appropriate for your task; defaults to 300s.
• Use add_stop_word for structured output tasks so the agent knows when it is done.
• Enable Garrison for any multi-turn conversation to maintain context.
• Check stop_reason.is_successful() before consuming result.output in production.
• Prefer execute_stream() for tasks > 30s so the caller can render output incrementally.
• Use autonomous features sparingly — they add LLM overhead; profile before enabling in loops.

Battalion Orchestration Patterns

The Battalion system in crates/paladin-battalion/ coordinates multiple Paladin agents through eight distinct execution patterns, plus the Commander strategy router that can select a pattern automatically.

Table of Contents

1. Overview
2. Quick Start
3. The Eight Patterns
   • Formation — Sequential
   • Phalanx — Concurrent
   • Campaign — Graph/DAG
   • Chain of Command — Hierarchical
   • Conclave — Mixture of Experts
   • Council — Collaborative Discussion
   • Grove — Semantic Routing
   • Maneuver — Flow DSL
4. Commander — Strategy Router
5. Error Handling
6. Performance Notes
7. Best Practices

Overview

All services require only Arc<dyn PaladinPort> (from paladin-ports) — they never import LLM provider libraries directly.

Quick Start

[dependencies]
paladin-ai = { version = "0.10.0", features = ["llm-openai"] }
tokio = { version = "1", features = ["full"] }

use paladin_battalion::formation_service::FormationExecutionService;
use paladin_core::platform::container::battalion::formation::Formation;
use paladin_core::platform::container::battalion::BattalionConfig;
use std::sync::Arc;

// Each Paladin is built with PaladinBuilder (see paladin-agents.md)
let paladins = vec![analyzer, processor, summarizer];
let config = BattalionConfig::default();
let formation = Formation::new(paladins, config)?;

let service = FormationExecutionService::new(paladin_port);
let result = service.execute(&formation, "Analyze the Q3 earnings report").await?;
println!("{}", result.output);

The Eight Patterns

Formation — Sequential

Source: crates/paladin-battalion/src/formation_service.rs

Output from each Paladin feeds the input of the next. Ideal for multi-step data transformation pipelines.

use paladin_battalion::formation_service::FormationExecutionService;
use paladin_core::platform::container::battalion::formation::Formation;

let formation = Formation::new(vec![extractor, analyzer, writer], config)?;
let service = FormationExecutionService::new(paladin_port);
let result = service.execute(&formation, "Raw data...").await?;

Configuration keys: sequential timeout, error strategy.

Phalanx — Concurrent

Source: crates/paladin-battalion/src/phalanx_service.rs

All Paladins receive the same input and execute concurrently via tokio tasks. Results are aggregated according to the AggregationStrategy.

use paladin_battalion::phalanx_service::PhalanxExecutionService;
use paladin_core::platform::container::battalion::phalanx::{AggregationStrategy, Phalanx};

let phalanx = Phalanx::new(
    vec![security_auditor, performance_analyst, style_checker],
    AggregationStrategy::Concatenate,
    config,
)?;

let service = PhalanxExecutionService::new(paladin_port);
let result = service.execute(&phalanx, "Review this Rust code...").await?;

AggregationStrategy variants: Concatenate, FirstSuccess, Majority, Custom.

Concurrency is bounded by a tokio::sync::Semaphore (configurable via max_concurrency in BattalionConfig).

Campaign — Graph/DAG

Source: crates/paladin-battalion/src/campaign_service.rs

Paladins are arranged in a directed acyclic graph. Execution is topologically sorted so upstream agents complete before downstream agents begin.

use paladin_battalion::campaign_service::CampaignExecutionService;
use paladin_core::platform::container::battalion::campaign::Campaign;

let campaign = Campaign::builder()
    .add_node("ingest", ingest_paladin)
    .add_node("analyze", analyze_paladin)
    .add_node("report", report_paladin)
    .add_edge("ingest", "analyze")
    .add_edge("analyze", "report")
    .config(config)
    .build()?;

let service = CampaignExecutionService::new(paladin_port);
let result = service.execute(&campaign, "Start").await?;

Independent branches execute concurrently; the service enforces dependency order.

Chain of Command — Hierarchical

Source: crates/paladin-battalion/src/chain_of_command_service.rs

A commander Paladin decomposes the task and routes sub-tasks to specialist Paladins, then synthesizes their outputs.

use paladin_battalion::chain_of_command_service::ChainOfCommandExecutionService;
use paladin_core::platform::container::battalion::chain_of_command::ChainOfCommand;

let chain = ChainOfCommand::new(
    commander_paladin,
    vec![backend_dev, frontend_dev, qa_engineer],
    config,
)?;

let service = ChainOfCommandExecutionService::new(paladin_port);
let result = service.execute(&chain, "Build a login feature").await?;

Conclave — Mixture of Experts

Source: crates/paladin-battalion/src/conclave_execution_service.rs

Multiple expert Paladins process the same task in parallel; an aggregator Paladin synthesizes their outputs into a final response.

use paladin_battalion::conclave_execution_service::ConclaveExecutionService;
use paladin_core::platform::container::battalion::conclave::Conclave;

let conclave = Conclave::new(
    vec![legal_expert, technical_expert, business_expert],
    synthesis_paladin,
    config,
)?;

let service = ConclaveExecutionService::new(paladin_port);
let result = service.execute(&conclave, "Should we adopt microservices?").await?;
// result.aggregated_output contains the synthesized response
// result.successful_expert_count() shows how many experts contributed

Council — Collaborative Discussion

Source: crates/paladin-battalion/src/council_service.rs

Paladins take turns responding to each other in a structured discussion, building toward a shared conclusion or consensus.

use paladin_battalion::council_service::CouncilService;
use paladin_core::platform::container::battalion::council::Council;

let council = Council::new(
    vec![optimist_paladin, skeptic_paladin, moderator_paladin],
    config,  // includes discussion_rounds
)?;

let service = CouncilService::new(paladin_port);
let result = service.execute(&council, "Evaluate adopting async Rust").await?;

Grove — Semantic Routing

Source: crates/paladin-battalion/src/grove_service.rs

The Grove routes the input to the most semantically appropriate Paladin from the registered specialists, using LLM-based capability matching.

use paladin_battalion::grove_service::GroveExecutionService;
use paladin_core::platform::container::battalion::grove::Grove;

let grove = Grove::new(
    vec![python_expert, rust_expert, go_expert],
    config,
)?;

let service = GroveExecutionService::new(paladin_port);
let result = service.execute(&grove, "Help me with Rust lifetimes").await?;
// Routes to rust_expert automatically

Maneuver — Flow DSL

Source: crates/paladin-battalion/src/maneuver/

Maneuver is a declarative flow DSL that lets you compose multiple Battalion patterns in a single workflow definition. See Maneuver Flow DSL for full syntax and examples.

Commander — Strategy Router

Source: crates/paladin-battalion/src/commander.rs

The Commander provides a single entry-point that automatically selects the optimal pattern based on input analysis and the number/capabilities of Paladins provided. Build it through CommanderBuilder (the same builder Orchestration uses) and run it with the live single-argument execute method — the direct Commander::new constructor takes five positional arguments (strategy, paladins, config, aggregator, paladin_port), and execute takes only the input string, never a separate strategy/config pair per call:

use paladin_battalion::commander::CommanderBuilder;
use paladin_core::platform::container::battalion::BattalionStrategy;

/// Build a Commander through `CommanderBuilder` (the builder `orchestration.md` also
/// uses) and run it with the live single-argument `execute` method. The direct
/// `Commander::new` constructor takes five positional arguments (strategy, paladins,
/// config, aggregator, paladin_port) — there is no two-argument form, and `execute`
/// takes only the input string, never a separate strategy/config pair per call.
pub async fn run_commander() -> Result<(), Box<dyn std::error::Error>> {
    let paladin_port = mock_paladin_port();

    let commander = CommanderBuilder::new(paladin_port)
        .strategy(BattalionStrategy::Auto)
        .paladins(vec![
            create_paladin("Analyzer"),
            create_paladin("Processor"),
            create_paladin("Synthesizer"),
        ])
        .build()?;

    let result = commander
        .execute("Analyze and summarize this report")
        .await?;

    println!("Strategy selected: {:?}", result.strategy_used);
    if let Some(reason) = &result.strategy_selection_reasoning {
        println!("Reasoning: {reason}");
    }
    println!("Output: {}", result.final_output);
    Ok(())
}

Force a specific strategy by passing a different BattalionStrategy variant to .strategy() on the builder instead of BattalionStrategy::Auto.

Auto Mode Heuristics

Error Handling

All services use ErrorStrategy from paladin_core::platform::container::battalion:

use paladin_core::platform::container::battalion::{BattalionConfig, ErrorStrategy};

let config = BattalionConfig {
    error_strategy: ErrorStrategy::ContinueOnError,
    max_concurrency: Some(4),
    timeout_seconds: 120,
    ..Default::default()
};

BattalionResult fields: final_output: String, paladin_results: Vec<PaladinResult> (each entry carries its own usage: TokenUsage, D-07), status: BattalionStatus, per_paladin_tokens: HashMap<String, TokenUsage> (the per-Paladin split), and total_tokens: u64 (the derived aggregate, D-08). There is no execution_time_ms or token_usage field on BattalionResult itself.

Performance Notes

• Phalanx concurrency is capped by BattalionConfig::max_concurrency (default: unbounded). Set this to avoid overloading upstream LLM rate limits.
• Formation adds one LLM call per Paladin sequentially — keep chains short (<6) for latency-sensitive workloads.
• Campaign parallelises independent branches automatically; no manual coordination needed.
• The Commander auto-router adds a small analysis overhead (~50ms); negligible for most tasks.

Best Practices

• Formation: Keep the chain ≤5 agents and design each stage to produce clean hand-off text.
• Phalanx: Set max_concurrency to stay within LLM provider rate limits.
• Campaign: Validate your DAG has no cycles before deployment (Campaign::build() checks this).
• Conclave: Ensure expert agents have distinct, non-overlapping system prompts for better synthesis.
• Council: Include a moderator Paladin to keep discussions on track.
• Grove: Write precise capability descriptions in each Paladin's agent_description field.
• Commander Auto: Test your routing decisions with representative inputs before production.

Orchestration

The Battalion runtime in crates/paladin-battalion/ coordinates multiple Paladin agents through a family of orchestration patterns, a strategy router (Commander), a cron-style job scheduler, and an event/trigger system. This guide is the comprehensive reference for choosing a pattern and wiring it up.

For a quick pattern-by-pattern cheat sheet see Battalion Patterns; for the declarative flow language see Maneuver Flow DSL; for how agents and workflows call each other see the Agent ↔ Orchestrator Bridge. For how a Battalion fits among the ways to run agents, see Deployment Topologies.

Every code example targets the current v0.10.0 workspace. The substantive examples are real, compiled code pulled from the paladin-doc-examples crate via mdBook {{#include}}, so they are checked against the live API; a few illustrative fragments are marked rust,ignore. The API forms are verified against crates/paladin-battalion/ and crates/paladin-ports/.

Table of Contents

1. Workflow Patterns Overview
2. Formation — Sequential
3. Phalanx — Parallel
4. Campaign — Graph / DAG
5. Chain of Command — Hierarchical
6. Commander — Dynamic Strategy Routing
7. Job Scheduling
8. Event and Trigger System
9. Configuration Reference
10. See Also

Workflow Patterns Overview

All orchestration services depend only on Arc<dyn PaladinPort> (from paladin-ports) — they never import an LLM provider crate directly. Pick a pattern by the shape of the work:

Conclave (mixture-of-experts), Council (turn-taking discussion), and Grove (semantic routing) are additional patterns documented in Battalion Patterns. The declarative Maneuver flow DSL has its own guide: Maneuver Flow DSL.

Decision Flowchart

flowchart TD
    start([Have a task + several Paladins]) --> q1{One fixed order of steps?}
    q1 -->|Yes| formation[Formation — sequential]
    q1 -->|No| q2{Steps independent, run together?}
    q2 -->|Yes| phalanx[Phalanx — parallel]
    q2 -->|No| q3{Explicit dependencies / branches?}
    q3 -->|Yes| campaign[Campaign — DAG]
    q3 -->|No| q4{A lead agent should delegate?}
    q4 -->|Yes| chain[Chain of Command]
    q4 -->|No| q5{Pattern varies per request?}
    q5 -->|Yes| commander[Commander — auto-route]
    q5 -->|No| formation

Formation — Sequential

Source: crates/paladin-battalion/src/formation_service.rs

Each Paladin's output becomes the next Paladin's input. Ideal for refinement pipelines (extract → analyze → write). If a stage fails, the configured ErrorStrategy decides whether the chain short-circuits (FailFast, the default) or continues.

#![allow(unused)]
fn main() {
use paladin_battalion::formation_service::FormationExecutionService;
use paladin_core::platform::container::battalion::formation::Formation;
use paladin_core::platform::container::battalion::{BattalionConfig, ErrorStrategy};

/// Run three Paladins in sequence; each one's output feeds the next.
pub async fn run_formation() -> Result<(), Box<dyn std::error::Error>> {
    let paladin_port = mock_paladin_port();
    let extractor = create_paladin("Extractor");
    let analyzer = create_paladin("Analyzer");
    let writer = create_paladin("Writer");

    let config = BattalionConfig {
        error_strategy: ErrorStrategy::FailFast, // first failure aborts the chain
        ..Default::default()
    };
    let formation = Formation::new(vec![extractor, analyzer, writer], config)?;

    let service = FormationExecutionService::new(paladin_port);
    let result = service
        .execute(&formation, "Raw Q3 earnings data...")
        .await?;

    println!("Final output: {}", result.final_output);
    Ok(())
}
}

Error handling / short-circuit: with ErrorStrategy::FailFast the first failing stage stops the Formation and returns the error. With ContinueOnError, a failed stage is skipped and its input is passed through to the next stage. Keep chains short (≤5) for latency-sensitive paths — each stage is one sequential LLM round-trip.

Phalanx — Parallel

Source: crates/paladin-battalion/src/phalanx_service.rs

Every Paladin receives the same input and runs concurrently on tokio tasks. Results are combined according to an AggregationStrategy, and concurrency is bounded by Phalanx::with_max_concurrency so you don't exceed LLM rate limits.

#![allow(unused)]
fn main() {
use paladin_battalion::phalanx_service::PhalanxExecutionService;
use paladin_core::platform::container::battalion::phalanx::{AggregationStrategy, Phalanx};

/// Fan the same input out to several Paladins concurrently, then aggregate.
pub async fn run_phalanx() -> Result<(), Box<dyn std::error::Error>> {
    let paladin_port = mock_paladin_port();
    let security = create_paladin("SecurityAuditor");
    let perf = create_paladin("PerformanceAnalyst");
    let style = create_paladin("StyleChecker");

    let phalanx = Phalanx::new(vec![security, perf, style], BattalionConfig::default())?
        .with_aggregation(AggregationStrategy::CollectAll)
        .with_max_concurrency(4); // cap concurrent Paladins

    let service = PhalanxExecutionService::new(paladin_port);
    let result = service
        .execute(&phalanx, "Review this Rust module...")
        .await?;

    println!("Aggregated: {}", result.final_output);
    Ok(())
}
}

AggregationStrategy variants: CollectAll (gather all outputs), FirstSuccess (first to finish wins), Majority (consensus), and Custom(String).

Campaign — Graph / DAG

Source: crates/paladin-battalion/src/campaign_service.rs

Paladins are arranged in a directed acyclic graph. The service topologically sorts the graph so every upstream node completes before its downstream nodes start; independent branches run concurrently. Campaign::build() rejects cycles.

#![allow(unused)]
fn main() {
use paladin_battalion::campaign_service::CampaignExecutionService;
use paladin_core::platform::container::battalion::campaign::{
    Campaign, CampaignEdge, EdgeCondition,
};

/// Arrange Paladins as a DAG: `ingest → analyze → report`.
pub async fn run_campaign() -> Result<(), Box<dyn std::error::Error>> {
    let paladin_port = mock_paladin_port();

    let mut campaign = Campaign::new(BattalionConfig::default());
    let ingest = campaign.add_paladin(create_paladin("Ingest"));
    let analyze = campaign.add_paladin(create_paladin("Analyze"));
    let report = campaign.add_paladin(create_paladin("Report"));

    // Edges define dependencies; `EdgeCondition::Always` is unconditional.
    campaign.add_edge(CampaignEdge::new(ingest, analyze, EdgeCondition::Always))?;
    // A conditional edge only traverses when the upstream output matches:
    campaign.add_edge(CampaignEdge::new(
        analyze,
        report,
        EdgeCondition::Contains("ready".to_string()),
    ))?;
    campaign.set_entry_point(ingest)?;

    let service = CampaignExecutionService::new(paladin_port);
    let result = service.execute(&campaign, "Start").await?;

    println!("Campaign output: {}", result.final_output);
    Ok(())
}
}

Paladins are added with add_paladin (returning a Uuid), wired with add_edge using CampaignEdge::new(source, target, condition), and the graph's start is set with set_entry_point. Use EdgeCondition::Contains/Regex for conditional branching; validate() (called by execute) rejects cycles.

Chain of Command — Hierarchical

Source: crates/paladin-battalion/src/chain_of_command_service.rs

A commander Paladin decomposes the task, routes sub-tasks to specialist (subordinate) Paladins, and synthesizes their outputs into a final answer.

#![allow(unused)]
fn main() {
use paladin_battalion::chain_of_command_service::ChainOfCommandExecutionService;
use paladin_core::platform::container::battalion::chain_of_command::ChainOfCommand;

/// A commander Paladin delegates to specialists and synthesizes their work.
pub async fn run_chain_of_command() -> Result<(), Box<dyn std::error::Error>> {
    let paladin_port = mock_paladin_port();
    let commander = create_paladin("Commander");
    let specialists = vec![
        create_paladin("BackendDev"),
        create_paladin("FrontendDev"),
        create_paladin("QaEngineer"),
    ];

    let chain = ChainOfCommand::new(commander, specialists, BattalionConfig::default())?;

    let service = ChainOfCommandExecutionService::new(paladin_port);
    let result = service.execute(&chain, "Build a login feature").await?;

    println!("Selected specialists: {:?}", result.selected_specialists);
    println!("Reasoning: {}", result.reasoning);
    for output in &result.outputs {
        println!("- {output}");
    }
    Ok(())
}
}

The service returns a DelegationResult with selected_specialists, reasoning, and the specialists' outputs. Give each subordinate a distinct agent_description so the commander can route accurately.

Commander — Dynamic Strategy Routing

Source: crates/paladin-battalion/src/commander.rs

The Commander is a single entry-point that selects a pattern automatically (Auto mode) based on the input text and the number/capabilities of the Paladins, or runs an explicit strategy you name. It also collects rich telemetry and can export execution metadata to JSON.

Auto mode

#![allow(unused)]
fn main() {
use paladin_battalion::commander::CommanderBuilder;
use paladin_core::platform::container::battalion::BattalionStrategy;

/// Let the Commander auto-select the best pattern for the input.
pub async fn run_commander_auto() -> Result<(), Box<dyn std::error::Error>> {
    let paladin_port = mock_paladin_port();

    let commander = CommanderBuilder::new(paladin_port)
        .strategy(BattalionStrategy::Auto)
        .paladins(vec![
            create_paladin("Analyzer"),
            create_paladin("Processor"),
            create_paladin("Synthesizer"),
        ])
        .build()?;

    let result = commander
        .execute("Analyze and summarize this report")
        .await?;

    println!("Strategy selected: {:?}", result.strategy_used);
    if let Some(reason) = &result.strategy_selection_reasoning {
        println!("Reasoning: {reason}");
    }
    println!("Output: {}", result.final_output);
    Ok(())
}
}

Explicit strategy

let commander = CommanderBuilder::new(paladin_port)
    .strategy(BattalionStrategy::Formation) // force a specific pattern
    .paladins(pipeline_paladins)
    .build()?;
let result = commander.execute(input).await?;

Auto-mode heuristics (first match wins)

Maneuver is explicit-only and is never chosen by Auto mode. Strategy selection typically adds ~0–5 ms of overhead; the decision is reported in result.strategy_selection_reasoning.

Metadata export

Point the Commander at a directory and it writes one JSON file per execution ({strategy}_{timestamp}_{uuid_short}.json, where uuid_short is the first 8 characters of the Battalion's UUID, e.g. formation_20250715_143022_a1b2c3d4.json) for audit, cost, and performance analysis.

use paladin_core::platform::container::battalion::BattalionConfig;
use std::path::PathBuf;

let config = BattalionConfig::new("audited_battalion")
    .with_metadata_dir(PathBuf::from("./battalion_metadata"));

let commander = CommanderBuilder::new(paladin_port)
    .strategy(BattalionStrategy::Auto)
    .paladins(paladins)
    .config(config)
    .build()?;

let result = commander.execute(input).await?;
// Metadata written to ./battalion_metadata/{strategy}_{timestamp}_{uuid}.json

Each file records battalion_id, strategy_used, duration_ms, total_tokens, per-Paladin paladin_results (output, execution_time_ms, usage: TokenUsage, stop_reason), per_paladin_times, per_paladin_tokens, and strategy_selection_reasoning.

Job Scheduling

Source: crates/paladin-ports/src/output/scheduler_port.rs and queue_port.rs

The scheduler runs jobs on a 6-field cron schedule; the queue ports manage asynchronous work items. A Redis-backed implementation is gated behind the root redis-queue feature.

Prerequisites: the Redis-backed queue requires the redis-queue feature and a running Redis instance. Run make dev to start it (alongside MinIO, MySQL, Qdrant).

Scheduling a recurring job

JobSpec carries a human label, a cron expression, and arbitrary metadata. SchedulerPort returns a JobId you can use to query status or cancel.

#![allow(unused)]
fn main() {
use paladin_ports::output::scheduler_port::{JobSpec, JobStatus, SchedulerPort};

/// Schedule a recurring job with a 6-field cron expression.
pub async fn run_scheduling() -> Result<(), Box<dyn std::error::Error>> {
    let scheduler: Arc<dyn SchedulerPort> = mock_scheduler();
    scheduler.start().await?;

    // 6-field cron: sec min hour day month weekday
    let spec = JobSpec::new("daily-digest", "0 0 9 * * *") // every day at 09:00:00
        .with_metadata("workflow", "news-digest");

    let job_id = scheduler.schedule_job(spec).await?;

    let status: JobStatus = scheduler.get_job_status(&job_id).await?;
    println!("job {job_id:?} is {status:?}");

    // Later: scheduler.cancel_job(&job_id).await?;
    Ok(())
}
}

JobStatus lifecycle: Scheduled → Running → Completed (or Failed(String) / Cancelled). JobInfo (from get_job_info) adds created_at, last_run, next_run, run_count, and failure_count.

Queue management, retry, and timeouts

The FullQueuePort trait composes enqueue/dequeue, batch, priority, and management operations (pause_queue, resume_queue, retry_item, purge_failed, get_queue_stats). Retry and timeout behavior for battalion execution is controlled by the battalion.retry and battalion.default_timeout_seconds configuration (see Configuration Reference).

use paladin_ports::output::queue_port::{FullQueuePort, QueueStats};

let stats: QueueStats = queue.get_queue_stats("news-digest").await?;
println!("pending: {}, processing: {}", stats.pending_items, stats.processing_items);

// Retry a failed item or purge the dead-letter set
queue.retry_item("news-digest", item_id).await?;
let purged = queue.purge_failed("news-digest").await?;

Event and Trigger System

Source: crates/paladin-core/src/platform/container/trigger.rs

A Trigger binds an incoming event to an action when a TriggerCondition matches. Events are matched by event_type_pattern, optional source_pattern, payload conditions, minimum priority, and optional TimeCondition windows (active hours/days and a cooldown).

Defining a condition and firing an event

Build a TriggerCondition (and TriggerConfig), then fire a matching event through the orchestrator bridge. fire_event returns an EventDispatchResult reporting how many triggers matched and their IDs.

#![allow(unused)]
fn main() {
use paladin_core::base::entity::message::MessagePriority;
use paladin_core::platform::container::trigger::{TimeCondition, TriggerCondition, TriggerConfig};
use paladin_ports::output::orchestrator_port::{FireEventRequest, OrchestratorPort};

/// Build a trigger condition and fire a matching event.
pub async fn run_events() -> Result<(), Box<dyn std::error::Error>> {
    let condition = TriggerCondition {
        event_type_pattern: "critical_finding".to_string(),
        source_pattern: Some("security-*".to_string()),
        payload_conditions: vec![],
        min_priority: Some(MessagePriority::High),
        time_conditions: Some(TimeCondition {
            active_hours: Some((9, 17)),            // only 09:00–17:00
            active_days: Some(vec![1, 2, 3, 4, 5]), // Mon–Fri
            cooldown_seconds: Some(300),            // at most once per 5 min
        }),
    };

    let config = TriggerConfig {
        max_retries: 3,
        timeout_seconds: 60,
        preserve_after_completion: false,
        ttl_seconds: 3600,
        processing_priority: MessagePriority::High,
    };

    // Fire an event through the orchestrator bridge.
    let orchestrator = mock_orchestrator();
    let result = orchestrator
        .fire_event(FireEventRequest {
            event_type: "critical_finding".to_string(),
            payload: serde_json::json!({ "severity": "high", "cve": "CVE-2025-0001" }),
            source: "security-scanner".to_string(),
        })
        .await?;

    println!(
        "{} trigger(s) fired: {:?}",
        result.triggered_count, result.trigger_ids
    );
    Ok(())
}
}

A matched trigger initiates the bound workflow (e.g. scheduling a job or queuing a Paladin run). See the Agent ↔ Orchestrator Bridge for end-to-end recipes that combine events, triggers, and agent execution.

Configuration Reference

Battalion execution behavior is configured programmatically through BattalionConfig (paladin_core::platform::container::battalion::BattalionConfig) — there is no battalion: section in config.yml and no APP_BATTALION_* environment-variable convention; ApplicationSettings does not carry a battalion field at all. Set behavior per-Battalion with the builder:

use paladin_core::platform::container::battalion::{BattalionConfig, ErrorStrategy, RetryPolicy};

let config = BattalionConfig::new("audited_battalion")
    .with_timeout(300)                             // seconds; matches the built-in default
    .with_error_strategy(ErrorStrategy::FailFast)  // FailFast | ContinueOnError | RetryThenContinue
    .with_retry_policy(RetryPolicy::default());     // max_attempts: 3, base_delay: 100ms,
                                                     // max_delay: 10s, exponential_backoff + jitter: true

Phalanx concurrency is set separately via Phalanx::with_max_concurrency (see Phalanx — Parallel) — it is not a BattalionConfig field.

BattalionResult (returned by the Formation/Phalanx/Campaign/Commander services) exposes: final_output: String, paladin_results: Vec<PaladinResult>, status: BattalionStatus, strategy_used: BattalionStrategy, total_tokens: u64, per_paladin_times, and per_paladin_tokens. (Chain of Command returns a DelegationResult instead.)

See Also

• Agent ↔ Orchestrator Bridge — agents triggering workflows and workflows invoking agents, with use-case recipes.
• Battalion Patterns — concise cheat sheet for all eight patterns including Conclave, Council, and Grove.
• Maneuver Flow DSL — declarative composition of multiple patterns.
• Content Processing — feeding a content pipeline into agent analysis.
• Crate Map — where paladin-battalion and paladin-ports sit in the workspace.

Content Processing

The paladin-content crate (crates/paladin-content/) ingests content from external sources, runs it through aggregation/analysis use cases, hands it to a Paladin agent for AI enrichment, and delivers the result. This guide covers the ingestion adapters, the processing use cases, the content → agent bridge, and delivery — documenting only what is wired into the compiled crate today.

Every code example targets the current v0.10.0 workspace. The substantive examples are real, compiled code pulled from the paladin-doc-examples crate via mdBook {{#include}} (a few illustrative fragments are rust,ignore). The API forms are verified against crates/paladin-content/src/.

Feature flags. Content processing lives behind the root content-processing feature, which enables paladin-content. Within the crate, news-api enables the News API fetcher and llm enables LLM-powered analysis. See the Crate Map for the full flag table.

Table of Contents

1. Content Ingestion Sources
2. Aggregation and the Processing Pipeline
3. Content → Agent Bridge
4. Content Delivery
5. Capabilities and Limitations
6. See Also

Content Ingestion Sources

Every fetcher produces a ContentItem (paladin_core::platform::container::content::ContentItem), the common currency of the pipeline. Sources are constructed and configured programmatically (there is no dedicated content: section in config.yml yet — see Limitations).

PDF / documents — PdfExtractor

PdfExtractor parses a PDF (from a path or raw bytes) into a Document. DocumentAdapter wraps document parsing for the pipeline.

#![allow(unused)]
fn main() {
use paladin_content::adapters::document::pdf_extractor::PdfExtractor;
use std::path::Path;

/// Extract a PDF (from a path or raw bytes) into a `Document`.
pub fn ingest_pdf() -> Result<(), Box<dyn std::error::Error>> {
    let extractor = PdfExtractor::new();
    let document = extractor.extract(Path::new("./reports/q3-earnings.pdf"))?;
    // Or from bytes already in memory:
    // let document = extractor.extract_bytes(&pdf_bytes)?;
    Ok(())
}
}

HTTP endpoints — HttpContentFetcher

HttpContentFetcher fetches a URL and returns a ContentItem. It implements the ContentFetchingService trait, so it can be driven directly or through the FetchContent use case.

#![allow(unused)]
fn main() {
use paladin_content::adapters::input::http_content_fetcher::HttpContentFetcher;
use paladin_content::services::content_fetching_service::{ContentFetchingService, FetchContent};

/// Fetch a URL into a `ContentItem`, directly and via the `FetchContent` use case.
pub fn ingest_http() -> Result<(), Box<dyn std::error::Error>> {
    let fetcher = HttpContentFetcher::new();
    // Direct use:
    let item = fetcher.fetch_content("https://example.com/article")?;

    // Or wrapped in the use case (same trait, swappable adapter):
    let fetch = FetchContent::new(HttpContentFetcher::new());
    let item = fetch.execute("https://example.com/article")?;
    Ok(())
}
}

News / feeds — NewsApiFetcher (feature news-api)

NewsApiFetcher polls a News API endpoint. It takes an API key and reuses an HttpContentFetcher for transport.

#![allow(unused)]
fn main() {
use paladin_content::adapters::input::news_api_fetcher::NewsApiFetcher;

/// Construct a News API fetcher (feature `news-api`).
pub fn ingest_news() {
    let fetcher = NewsApiFetcher::new("YOUR_NEWS_API_KEY".to_string())
        .with_content_fetcher(HttpContentFetcher::new());
}
}

Files — FileContentFetcher

For local ingestion and testing, FileContentFetcher reads a file from disk and infers its content type from the extension. Unlike the HTTP fetcher, it implements ContentIngestionPort (paladin_ports::input): its fetch_content takes a ContentItem describing the source path and returns a populated ContentItem. (It is an internal #[doc(hidden)] adapter; the primary documented ingestion paths are HTTP, PDF, and the News API above.)

Aggregation and the Processing Pipeline

Once items are fetched, the use cases combine and analyze them. Each use case is generic over a trait, so adapters are swappable.

flowchart LR
    src[(Sources: PDF / HTTP / News / File)] --> fetch[FetchContent]
    fetch --> agg[AggregateContent]
    agg --> sum[ContentSummarizer]
    sum --> ai[LlmContentAnalyzer]
    ai --> deliver[DeliverContentUseCase]
    deliver --> out[(Destinations)]

Aggregation

AggregateContent wraps a ContentListService and merges a vector of JSON values into a single aggregated value — useful for collapsing multiple fetched sources before analysis.

#![allow(unused)]
fn main() {
use paladin_content::services::content_aggregator_service::AggregateContent;

/// Merge JSON from several sources into one aggregated value.
pub fn aggregate() {
    // `MockListService` implements the `ContentListService` trait.
    let aggregator = AggregateContent::new(MockListService);
    let source_a = serde_json::json!({ "title": "A" });
    let source_b = serde_json::json!({ "title": "B" });
    let aggregated = aggregator.execute(vec![source_a, source_b]);
}
}

Summarization

ContentSummarizer produces summaries and keywords without an LLM call (deterministic text processing), returning a ContentSummary plus ContentMetadata.

#![allow(unused)]
fn main() {
use paladin_content::services::content_summarizer_service::ContentSummarizer;

/// Summarize a `ContentItem` and extract keywords (no LLM call).
pub fn summarize() {
    let item = text_content_item("A long article body about quarterly earnings...");
    let summarizer = ContentSummarizer::new();
    let summary = summarizer.summarize_content(&item, 500); // max 500 chars
    let keywords = summarizer.extract_keywords(&item);
}
}

Content → Agent Bridge

The llm feature enables LlmContentAnalyzer, which passes a ContentItem plus a prompt to a Paladin LLM analysis service for AI enrichment. This is the seam where the content pipeline meets the agent layer.

LlmContentAnalyzer::analyze_with_prompt_async takes an LlmContentAnalysisInput (prompt: PromptItem, content: ContentItem) and an LlmContentAnalysisConfig (model, retries, timeout, max_content_length), and returns the analysis as JSON.

#![allow(unused)]
fn main() {
use paladin_content::services::content_llm_analysis_service::{
    LlmContentAnalysisConfig, LlmContentAnalysisInput, LlmContentAnalyzer,
};
use paladin_llm::llm_analysis_service::LlmAnalysisService;
use paladin_llm::mock::MockLlmAdapter;
use paladin_ports::output::llm_port::LlmPort;

/// Pass content + a prompt to a Paladin LLM service for AI enrichment.
pub async fn content_to_agent() -> Result<(), Box<dyn std::error::Error>> {
    // In production this is a real provider (e.g. OpenAIAdapter); here a mock.
    let llm: Arc<dyn LlmPort> =
        Arc::new(MockLlmAdapter::new().with_response("{\"summary\":\"...\"}"));
    let llm_service = Arc::new(LlmAnalysisService::new(llm));

    let analyzer = LlmContentAnalyzer::new(llm_service);
    let input = LlmContentAnalysisInput {
        prompt: text_prompt_item("Summarize the key risks in this article."),
        content: text_content_item("Latest article body..."),
    };
    let config = LlmContentAnalysisConfig::default(); // gpt-3.5-turbo, 3 retries, 30s timeout

    let analysis = analyzer
        .analyze_with_prompt_async(&input, &config)
        .await
        .map_err(|e| -> Box<dyn std::error::Error> { e.into() })?;
    println!("{}", serde_json::to_string_pretty(&analysis)?);
    Ok(())
}
}

Use the async method (analyze_with_prompt_async). The sync analyze_with_prompt is a compatibility stub that returns an error directing callers to the async path.

For richer agent interactions — an agent that triggers a workflow, or a workflow step that invokes a full Paladin agent loop — see the Agent ↔ Orchestrator Bridge.

Content Delivery

DeliverContentUseCase sends processed content to a destination through the ContentDeliveryService port (paladin_ports::output::content_delivery_port). It takes a DeliveryRequest and returns a DeliveryResponse (with a DeliveryStatus).

#![allow(unused)]
fn main() {
use paladin_content::services::content_delivery_service::DeliverContentUseCase;
use paladin_ports::output::content_delivery_port::{
    ContentPayload, DeliveryMethod, DeliveryPriority, DeliveryRequest,
};

/// Deliver processed content through a `ContentDeliveryService`.
pub fn deliver() -> Result<(), Box<dyn std::error::Error>> {
    let delivery = DeliverContentUseCase::new(MockDeliveryAdapter);

    let request = DeliveryRequest {
        recipient_id: "ops-team".to_string(),
        delivery_method: DeliveryMethod::Email {
            to: "ops@example.com".to_string(),
            subject: "Daily digest".to_string(),
        },
        content_payload: ContentPayload::SingleItem(text_content_item("Digest body...")),
        priority: DeliveryPriority::Normal,
        scheduled_time: None,
        metadata: None,
    };

    let response = delivery.execute(request)?;
    println!("delivery status: {:?}", response.status);
    Ok(())
}
}

For push/email/system notification of delivered content, wire the delivery adapter to the notification adapters (paladin-notifications) or fire a notification through the orchestrator bridge — see the bridge recipes.

Capabilities and Limitations

The crate's manifest declares some features whose adapters are not yet implemented in v0.10.0. To keep this guide honest:

For web-scraping and RSS today, fetch the raw resource with HttpContentFetcher and parse it in your own adapter. Filtering/dedup must likewise be done in caller code until the content_filtering_service module is completed and re-enabled.

See Also

• Agent ↔ Orchestrator Bridge — end-to-end recipes combining content ingestion with agent analysis and notification.
• Orchestration — running the analysis Paladin inside a Battalion workflow.
• Paladin Agents — building the Paladin that performs the AI enrichment.
• Crate Map — paladin-content exports and feature flags.

Agent ↔ Orchestrator Bridge

Paladin agents and Battalion workflows interact bidirectionally:

• An agent can trigger orchestration — schedule a job, enqueue an item, fire an event, or send a notification — through a narrow, policy-guarded port.
• A workflow can invoke an agent — run a single Paladin or a whole Battalion as a step and feed its output back into the workflow.

This guide covers both directions, how to configure the bridge safely, and four end-to-end recipes. It builds on the Orchestration and Content Processing guides.

Every example targets the current v0.10.0 workspace. The substantive examples are real, compiled code pulled from the paladin-doc-examples crate via mdBook {{#include}} (one illustrative fragment is rust,ignore). API forms are verified against crates/paladin-ports/src/output/orchestrator_port.rs, paladin_executor_port.rs, battalion_port.rs, and the concrete OrchestratorBridgeAdapter in src/application/services/orchestration/.

Table of Contents

1. Agents Triggering Orchestration
2. Orchestration Invoking Agents
3. Configuring the Bridge
4. Use-Case Recipes
5. See Also

Agents Triggering Orchestration

The seam is OrchestratorPort (crates/paladin-ports/src/output/orchestrator_port.rs). It exposes exactly four actions, mirrored by the BridgeAction enum:

The concrete adapter, OrchestratorBridgeAdapter, wraps an Arc<Orchestrator> and a BridgePolicy. It enforces the policy before performing any underlying call, so an agent can never exceed the actions or per-execution caps it was granted.

sequenceDiagram
    participant Agent as Paladin agent (tool call)
    participant Bridge as OrchestratorBridgeAdapter
    participant Policy as BridgePolicy
    participant Orch as Orchestrator

    Agent->>Bridge: fire_event(FireEventRequest)
    Bridge->>Policy: is_allowed(FireEvent)?
    Policy-->>Bridge: true
    Bridge->>Policy: cap_for(FireEvent)
    Policy-->>Bridge: 3
    Bridge->>Orch: dispatch event (within cap)
    Orch-->>Bridge: EventDispatchResult
    Bridge-->>Agent: Ok(EventDispatchResult)

Tool-based invocation from an agent loop

Expose the bridge to a Paladin as a tool. When the agent decides to act, the tool implementation calls the relevant OrchestratorPort method. The agent never touches the Orchestrator directly — only the policy-guarded port.

#![allow(unused)]
fn main() {
use paladin_ports::output::orchestrator_port::{
    BridgeAction, BridgePolicy, FireEventRequest, OrchestratorBridgeError, OrchestratorPort,
};

/// An agent fires a domain event through the policy-guarded bridge.
pub async fn agent_triggers_orchestration() -> Result<(), Box<dyn std::error::Error>> {
    // Grant ONLY the actions this agent should perform, with explicit caps.
    let mut allowed = HashSet::new();
    allowed.insert(BridgeAction::FireEvent);
    let policy = BridgePolicy::new(allowed, 0, 0, 5, 0); // up to 5 events, nothing else

    // In production this is an `OrchestratorBridgeAdapter`; here a mock stands in.
    let bridge: Arc<dyn OrchestratorPort> = mock_orchestrator();
    let _ = &policy; // the real adapter is constructed as `::new(orchestrator, policy)`

    match bridge
        .fire_event(FireEventRequest {
            event_type: "critical_finding".to_string(),
            payload: serde_json::json!({ "severity": "high" }),
            source: "security-agent".to_string(),
        })
        .await
    {
        Ok(result) => println!("fired; {} trigger(s) matched", result.triggered_count),
        Err(OrchestratorBridgeError::ActionNotAllowed(_)) => {
            eprintln!("policy forbids this action")
        }
        Err(OrchestratorBridgeError::QuotaExceeded { .. }) => {
            eprintln!("per-execution cap reached")
        }
        Err(e) => return Err(e.into()),
    }
    Ok(())
}
}

OrchestratorBridgeError distinguishes ActionNotAllowed (the policy doesn't grant the action) from QuotaExceeded (the per-execution cap is reached), so an agent can react sensibly instead of failing opaquely.

Orchestration Invoking Agents

The reverse direction uses the executor ports:

• PaladinExecutorPort (paladin_executor_port.rs) — run a single Paladin: async fn execute(&self, paladin: &Paladin, input: &str) -> Result<PaladinResult, PaladinError>.
• BattalionPort (battalion_port.rs) — run/monitor a whole Battalion by id: execute(battalion_id) -> BattalionResult, plus status and cancel.

A workflow step builds the input string (passing context from earlier steps), calls the executor, and reads the result back out.

sequenceDiagram
    participant WF as Workflow step
    participant Exec as PaladinExecutorPort
    participant Paladin as Paladin agent

    WF->>Exec: execute(&paladin, input_with_context)
    Exec->>Paladin: run agent loop
    Paladin-->>Exec: PaladinResult { output, usage, ... }
    Exec-->>WF: Ok(PaladinResult)
    Note over WF: feed result.output into the next step

#![allow(unused)]
fn main() {
use paladin_core::platform::container::paladin::Paladin;
use paladin_ports::output::paladin_executor_port::PaladinExecutorPort;

/// A workflow step runs a single Paladin, passing context via the input string.
pub async fn orchestration_invokes_agent(
    analyst: &Paladin,
) -> Result<(), Box<dyn std::error::Error>> {
    let executor: Arc<dyn PaladinExecutorPort> = mock_executor();

    let upstream = "Q3 revenue rose 12% QoQ; churn fell to 2.1%.";
    let input = format!("Summarize the key risks given this context:\n{upstream}");

    let result = executor.execute(analyst, &input).await?;

    println!("agent said: {}", result.output);
    println!(
        "tokens: {}, stop reason: {:?}",
        result.usage.total_tokens, result.stop_reason
    );
    Ok(())
}
}

PaladinResult carries output, usage (the full TokenUsage split), execution_time_ms, loop_count, and stop_reason — everything the workflow needs to decide what to do next. To invoke a whole Battalion instead of a single agent, use BattalionPort::execute(battalion_id) and read the BattalionResult (see Orchestration → Configuration Reference).

Configuring the Bridge

Bridge behavior is configured programmatically through BridgePolicy — there is no dedicated config.yml bridge section in v0.5.0. A policy is two things: the set of allowed actions, and a per-execution cap for each action.

#![allow(unused)]
fn main() {
/// Build least-privilege and default bridge policies.
pub fn configure_bridge() {
    use paladin_ports::output::orchestrator_port::{BridgeAction, BridgePolicy};

    // Explicit, least-privilege: allow scheduling + notifications only,
    // with caps of (jobs=2, queue=0, events=0, notifications=5).
    let mut allowed = HashSet::new();
    allowed.insert(BridgeAction::ScheduleJob);
    allowed.insert(BridgeAction::SendNotification);
    let policy = BridgePolicy::new(allowed, 2, 0, 0, 5);

    // Builder-style: start from caps and add actions.
    let policy = BridgePolicy::new(HashSet::new(), 1, 1, 1, 1)
        .allow(BridgeAction::FireEvent)
        .allow(BridgeAction::QueueItem);

    // Conservative-but-usable default: all four actions, cap 3 each.
    let policy = BridgePolicy::default();
}
}

The three forms shown are: an explicit least-privilege policy, the builder-style .allow(..), and the conservative-but-usable Default (all four actions, cap 3 each). Prefer an explicit least-privilege policy for agents you don't fully trust.

Tip: because the adapter enforces the policy before every call, tightening a policy is a safe, local change — you don't have to audit the agent's prompt to constrain what it can do.

Use-Case Recipes

1. News monitoring pipeline with AI analysis

NewsApiFetcher → AI summarization (LlmContentAnalyzer) → notification via the bridge.

#![allow(unused)]
fn main() {
use paladin_ports::output::orchestrator_port::SendNotificationRequest;

/// Recipe: notify the result of an AI summary through the bridge.
pub async fn recipe_news_notification(
    bridge: &Arc<dyn OrchestratorPort>,
    summary: &str,
) -> Result<(), Box<dyn std::error::Error>> {
    bridge
        .send_notification(SendNotificationRequest {
            channel: "email".to_string(),
            recipient: "ops@example.com".to_string(),
            subject: "Daily news digest".to_string(),
            body: summary.to_string(),
        })
        .await?;
    Ok(())
}
}

See Content Processing for the ingestion/analysis half and Orchestration → Job Scheduling to run this on a cron.

2. Research workflow

A web/HTTP tool gathers sources, a Paladin synthesizes them, and a Formation assembles the final report.

// 1. Agent gathers sources via an HTTP tool (Arsenal), producing notes.
// 2. Synthesis Paladin run as a workflow step:
let synthesis = executor.execute(&synthesizer, &collected_notes).await?;
// 3. Formation assembles intro → body → conclusion from the synthesis.
let report = formation_service.execute(&report_formation, &synthesis.output).await?;

3. Scheduled batch enrichment (job queue)

A recurring job enqueues items; a worker drains the queue and runs each through a Paladin.

#![allow(unused)]
fn main() {
use paladin_core::platform::container::schedule::Schedule;
use paladin_ports::output::orchestrator_port::{QueueItemRequest, ScheduleJobRequest};

/// Recipe: schedule a recurring batch job and enqueue an item.
pub async fn recipe_scheduled_batch(
    bridge: &Arc<dyn OrchestratorPort>,
    content_id: &str,
) -> Result<(), Box<dyn std::error::Error>> {
    bridge
        .schedule_job(ScheduleJobRequest {
            name: "nightly-enrichment".to_string(),
            description: "Enrich the day's content with AI tags".to_string(),
            schedule: Schedule::Daily(2, 0), // 02:00 daily
        })
        .await?;

    bridge
        .queue_item(QueueItemRequest {
            queue_name: "enrichment".to_string(),
            payload: serde_json::json!({ "content_id": content_id }),
        })
        .await?;
    Ok(())
}
}

4. Trigger-initiated agent run

An agent fires a domain event; a registered Trigger matches it and initiates a Paladin run — fully event-driven, no polling.

#![allow(unused)]
fn main() {
/// Recipe: an agent fires an event that a Trigger turns into a Paladin run.
pub async fn recipe_trigger_initiated(
    bridge: &Arc<dyn OrchestratorPort>,
) -> Result<(), Box<dyn std::error::Error>> {
    let dispatch = bridge
        .fire_event(FireEventRequest {
            event_type: "anomaly_detected".to_string(),
            payload: serde_json::json!({ "metric": "latency_p99", "value": 920 }),
            source: "monitor-agent".to_string(),
        })
        .await?;

    println!("{} trigger(s) initiated", dispatch.triggered_count);
    Ok(())
}
}

See Also

• Orchestration — the Battalion patterns, job scheduler, and trigger system the bridge drives.
• Content Processing — the ingestion/analysis pipeline used in recipes 1 and 3.
• Paladin Agents — building the agents on both sides of the bridge.
• Crate Map — where OrchestratorPort and the executor ports live.

Arsenal Tools

The Arsenal system (crates/paladin-ports/src/output/arsenal_port.rs) gives Paladins access to external tools and services through the Model Context Protocol (MCP). Tools are called Armaments; the registry that holds them is the Arsenal.

Table of Contents

1. Concepts
2. Quick Start — STDIO Server
3. Streamable-HTTP Server Configuration
4. config.yml Reference
5. ArsenalPort Trait
6. ArsenalRegistry Trait
7. Attaching Arsenal to a Paladin
8. Custom Armaments (Direct Rust Tools)
9. Handoff Tool
10. Error Handling
11. Best Practices

Concepts

Quick Start — STDIO Server

STDIO servers are the most common MCP transport. The process is spawned and communicated with via newline-delimited JSON on stdin/stdout.

1. Configure in config.yml

arsenal:
  mcp_servers:
    - name: web_search
      type: stdio
      command: uvx
      args: ["mcp-server-brave-search"]
      env:
        BRAVE_API_KEY: "${BRAVE_API_KEY}"
    - name: filesystem
      type: stdio
      command: npx
      args: ["-y", "@modelcontextprotocol/server-filesystem", "/workspace"]

2. Build a Paladin with the Arsenal

use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin_ports::output::llm_port::LlmPort;
use paladin_ports::output::arsenal_port::ArsenalRegistry;
use std::sync::Arc;

// Arsenal registry is built from config.yml automatically when using
// PaladinBuilder::from_config() or can be constructed manually.
let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are a research assistant with web search access.")
    .with_arsenal_registry(arsenal_registry)
    .build()
    .await?;

let result = paladin.execute("Find the latest Rust release notes").await?;
println!("{}", result.output);

The Paladin will automatically detect tool-call JSON in LLM responses, invoke the tool via the Arsenal, and feed results back into the reasoning loop.

Streamable-HTTP Server Configuration

Remote MCP servers are reached over HTTP(S) using the Streamable-HTTP transport (D-02/D-03). This is the real, currently-implemented remote transport, replacing the retired MCPSseAdapter (which was never actually SSE — just a mislabeled, unauthenticated plain-HTTP-POST adapter):

arsenal:
  mcp_servers:
    - name: my_api_server
      type: streamable_http
      endpoint: "http://localhost:8080/mcp"
      # NAMES the env var holding the bearer token -- never a literal secret
      # in this file. Omit entirely for an unauthenticated server.
      auth_token_env: "MY_API_SERVER_TOKEN"

MCPStreamableHttpAdapter builds the connection and delegates to MCPClient::connect_streamable_http, which performs the full initialize -> notifications/initialized handshake:

use paladin::infrastructure::adapters::arsenal::mcp_streamable_http_adapter::MCPStreamableHttpAdapter;

let adapter = MCPStreamableHttpAdapter::new("http://localhost:8080/mcp")
    .with_bearer_token(std::env::var("MY_API_SERVER_TOKEN")?); // never hardcode the token
let client = adapter.connect().await?;
let tools = client.discover_tools().await?;

config.yml Reference

arsenal:
  mcp_servers:
    - name: <identifier>              # Unique name used in logs and errors
      type: stdio | streamable_http   # Transport type ("sse" is retired --
                                       # fails loud with a migration message)
      # STDIO fields:
      command: <executable>           # e.g. python3, npx, uvx
      args: [<arg>, ...]              # Command-line arguments
      # Streamable-HTTP fields:
      endpoint: <url>                 # Full URL of the remote MCP endpoint
      auth_token_env: <ENV_VAR_NAME>  # NAMES the env var holding the bearer
                                       # token -- never a literal secret here

ArsenalPort Trait

Defined in crates/paladin-ports/src/output/arsenal_port.rs:

#[async_trait]
pub trait ArsenalPort: Send + Sync {
    /// List all available armaments from this MCP server
    async fn list_armaments(&self) -> Vec<Armament>;

    /// Invoke an armament with the given arguments
    async fn invoke(&self, call: ArmamentCall) -> Result<ArmamentResult, ArsenalError>;

    /// Validate call arguments against the armament's JSON schema
    fn validate_call(&self, call: &ArmamentCall) -> Result<(), ArsenalError>;
}

Direct usage:

use paladin_core::platform::container::arsenal::ArmamentCall;
use serde_json::json;
use std::collections::HashMap;

let mut args = HashMap::new();
args.insert("query".to_string(), json!("Rust 2024 edition features"));

let call = ArmamentCall::new("web_search", args);
arsenal_port.validate_call(&call)?;

let result = arsenal_port.invoke(call).await?;
if result.success {
    println!("{}", result.output.unwrap());
}

ArsenalRegistry Trait

Defined alongside ArsenalPort:

#[async_trait]
pub trait ArsenalRegistry: Send + Sync {
    /// Register a new armament in the registry
    async fn register(&self, armament: Armament);

    /// Remove an armament by name
    async fn remove(&self, name: &str);

    /// Get all registered armament descriptors
    async fn list(&self) -> Vec<Armament>;

    /// Look up a specific armament by name
    async fn get(&self, name: &str) -> Option<Armament>;
}

Attaching Arsenal to a Paladin

use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin_ports::output::arsenal_port::ArsenalRegistry;
use std::sync::Arc;

let paladin = PaladinBuilder::new(llm_port)
    .system_prompt(
        "You are a coding assistant. Use the filesystem tool to read files when needed."
    )
    .with_arsenal_registry(Arc::new(my_registry))
    .build()
    .await?;

Custom Armaments (Direct Rust Tools)

Implement ArsenalPort to expose any Rust function as a tool. Armament carries a parameters JSON Schema (not input_schema) plus a required_params list; the live ArmamentResult has five fields — call_id, success, output, error and execution_time_ms — and a call's arguments are read from the arguments map on ArmamentCall, not an args field:

use async_trait::async_trait;
use paladin_core::platform::container::arsenal::{
    Armament, ArmamentCall, ArmamentResult, ArsenalError,
};
use paladin_ports::output::arsenal_port::ArsenalPort;

/// Implement `ArsenalPort` to expose any Rust function as a tool.
pub struct CalculatorTool;

#[async_trait]
impl ArsenalPort for CalculatorTool {
    async fn list_armaments(&self) -> Vec<Armament> {
        vec![Armament {
            name: "calculate".to_string(),
            description: "Evaluate a mathematical expression".to_string(),
            parameters: serde_json::json!({
                "type": "object",
                "properties": {
                    "expression": { "type": "string" }
                },
                "required": ["expression"]
            }),
            required_params: vec!["expression".to_string()],
        }]
    }

    async fn invoke(&self, call: ArmamentCall) -> Result<ArmamentResult, ArsenalError> {
        // Arguments live on the `arguments` map, not an `args` field.
        let expr = call
            .arguments
            .get("expression")
            .and_then(|v| v.as_str())
            .unwrap_or_default();
        // ... evaluate `expr` ...
        Ok(ArmamentResult {
            call_id: call.call_id,
            success: true,
            output: Some(serde_json::json!(42)),
            error: None,
            execution_time_ms: 1,
        })
    }

    fn validate_call(&self, call: &ArmamentCall) -> Result<(), ArsenalError> {
        match call.arguments.get("expression") {
            Some(v) if v.is_string() => Ok(()),
            Some(_) => Err(ArsenalError::InvalidArguments(
                "expression must be a string".into(),
            )),
            None => Err(ArsenalError::InvalidArguments(
                "expression is required".into(),
            )),
        }
    }
}

Handoff Tool

The handoff_tool in crates/paladin-core/src/platform/container/arsenal/handoff_tool.rs is a built-in Armament that allows a Paladin to delegate sub-tasks to specialist agents at runtime. Register specialist agents on the builder via with_handoffs, which takes the whole specialist list at once — there is no per-call chainable registration method:

/// Register specialist agents on the builder so the built-in handoff Armament can
/// delegate to them at runtime — `with_handoffs` takes the whole specialist list at
/// once, there is no per-call chainable registration method.
pub async fn build_coordinator_with_handoffs() -> Result<(), Box<dyn std::error::Error>> {
    let llm_port: Arc<dyn LlmPort> = Arc::new(MockLlmAdapter::new());

    let code_paladin = PaladinBuilder::new(llm_port.clone())
        .system_prompt("You review code changes.")
        .name("CodeReviewer")
        .build()
        .await?;
    let test_paladin = PaladinBuilder::new(llm_port.clone())
        .system_prompt("You write and run tests.")
        .name("TestEngineer")
        .build()
        .await?;

    let coordinator = PaladinBuilder::new(llm_port)
        .system_prompt("You are a coordinator. Delegate to specialists when needed.")
        .with_handoffs(vec![Arc::new(code_paladin), Arc::new(test_paladin)])
        .build()
        .await?;

    let _ = coordinator;
    Ok(())
}

The LLM will emit a tool-call for handoff when it determines a specialist is more appropriate. Delegation records appear in PaladinResult.handoff_history.

Error Handling

ArsenalError variants (from paladin_core::platform::container::arsenal):

Best Practices

• Validate before invoking — call validate_call() to catch argument errors early.
• Set timeouts — all MCP servers should have timeout_seconds to avoid blocking the reasoning loop indefinitely.
• Describe tools well — the Armament description is what the LLM reads to decide whether to call the tool; make it precise.
• Namespace tool names — use server_name.tool_name convention to avoid collisions when registering multiple servers.
• Test with mock — implement a MockArsenalPort in tests to avoid spawning real subprocesses.

Graph Visualization

Since: v0.10.0 (Phase 28, PRD 07)

Two CLI commands and one admin-gated dev page turn a WarGraphDoc/WarGraph, an execution history, or both, into a diagram: paladin-cli graph export renders a graph's static shape, paladin-cli run export renders a thread's execution overlay on top of that shape, and the dev-ui inspector page renders the same overlay in a browser.

Exporting a graph's static shape

paladin-cli graph export --format mermaid path/to/graph.json

renders a WarGraphDoc (JSON or YAML) to Mermaid on stdout — pipe-friendly, no colour, byte-exact to what to_mermaid produces. --format dot renders the same shape as a Graphviz digraph instead. Either an explicit file or a stored assistant works:

paladin-cli graph export --format mermaid --assistant my-workflow@3
paladin-cli graph export --format dot my-graph.yaml --out diagram.dot

--assistant <id>[@<version>] resolves a Workflow-kind assistant's stored document through the configured RunStoreConfig backend (SQLite locally, Postgres by URL) — the same store the server resolves against, reached through ports only, never HTTP. --out <path> writes to a file with a short confirmation instead of stdout.

Exporting a run's execution overlay

paladin-cli run export --thread <thread-id>

renders the SAME graph shape plus an execution overlay: which nodes actually ran, how many times, with what outcome, and which edges actually fired versus were merely evaluated. Two optional flags narrow or redirect resolution:

paladin-cli run export --run <run-id>                      # derive thread + graph from a run row
paladin-cli run export --thread <thread-id> --waypoint <id> # cap history to one Waypoint
paladin-cli run export --thread <thread-id> --graph my-graph.json  # explicit graph document

The command prints two lines identifying the resolved overlay source and graph resolution ahead of the diagram, so you always know which of the two mixed-fidelity paths below produced what you're looking at.

Overlay source: Waypoints vs. persisted trace

• Waypoint history (always available) — fired edges are derived from each superstep's completed → next superstep's vanguard transition. Evaluated-but-not-fired edges are never visible on this path (there is nothing in a Waypoint recording "this edge was checked and lost").
• Persisted trace (the upgrade, requires trace.persist: true — see the observability page) — fired and evaluated-but-not-fired edges are read directly from TraceEvent::EdgeEvaluated, exact rather than derived.

run export prefers the trace source whenever non-empty rows exist for the thread, falling back to Waypoint history otherwise.

Graph resolution order

1. --graph <file>, if given.
2. The run's assistant version's stored WarGraphDoc, when the thread belongs to a known run.
3. Observed-only — no static graph resolves (no --graph, no run, or the assistant is Agent-kind). The diagram is built from only the nodes and edges actually seen, titled (observed nodes only — no graph document available). This is expected and acceptable: the acceptance question ("which branch fired and why did node X run 3 times") is answered by the overlay itself, not by unexecuted nodes.

The badge and outcome-colour legend

Every rendered node carries a guillemet-quoted kind badge («paladin», «function», «gate», «workflow», «worker»), frozen by the golden fixtures under crates/paladin-battalion/tests/golden/export/. Gate nodes render as a diamond; Workflow nodes render as a nested subgraph cluster; a worker-template node and a deferred (Muster aggregator) node both render dashed.

On an execution overlay, a visited node is additionally coloured by its last visit's outcome:

A node visited more than once carries a ×N badge; every visited node's label additionally carries <duration>ms · <tokens>tok — a cache-hit visit's duration/token figures are None by construction (never a stale or zero number) and render as a dash instead. A fired edge renders bold; an evaluated-but-not-fired edge (trace source only) renders dotted.

The dev-ui inspector page

GET /v1/dev-ui/threads/{id} renders the same RunInspectorPort::inspect view as run export, as a static, admin-gated HTML page: the diagram, a per-node visit summary, a fired/evaluated-edge list per superstep, and the full superstep table. It requires:

• The dev-ui feature (crates/paladin-web's first [features] section, off by default, absent from full) compiled in.
• An authenticated request satisfying the SAME require_auth + require_admin middleware pair the rest of the crate's admin routes use — it exposes state field names and the run's own structure, which is operator information.

The page has no build pipeline: one static HTML template (include_str!-ed), the InspectorView JSON embedded verbatim into a typed <script type="application/json"> element (with </ and <!-- escaped so the payload can never break out of its own tag), and an inline vanilla-JS renderer. It makes exactly one external request beyond the page itself — importing the Mermaid ESM module the diagram is rendered with.

mermaid_url for air-gapped hosts

Mermaid is loaded from web_server.dev_ui.mermaid_url, which defaults to the jsDelivr mermaid@11 ESM bundle CDN URL. It is deliberately not vendored into the crate — a multi-megabyte JS asset in a published crate's include list is a cost nobody asked for. An air-gapped operator points this at a local mirror instead:

web_server:
  dev_ui:
    mermaid_url: "https://internal-mirror.example.com/mermaid/11/mermaid.esm.min.mjs"

or via the environment:

export APP_WEB_SERVER_DEV_UI_MERMAID_URL="https://internal-mirror.example.com/mermaid/11/mermaid.esm.min.mjs"

If the configured URL cannot be loaded (a 5-second timeout, or a load failure), the diagram panel falls back to showing the raw Mermaid source text — the other three panels (node visits, fired edges, superstep table) render independently and are unaffected.

Eval Harness

Since: v0.10.0 (Phase 28, PRD 07)

paladin-eval is a published, composition-tier crate (classification recorded in .planning/decisions/0048-paladin-eval-composition-crate.md, ADR-0048 — not linked here as a clickable URL, since it lives outside docs/src and mdBook's linkcheck runs in strict warning-policy = "error" mode) for writing deterministic, scripted-LLM test scenarios against a real WarEngine run, evaluating them with a purpose-built assertion library, and running them either through cargo test or paladin-cli eval run.

The scenario file format

A scenario is a .eval.yaml (JSON also accepted) file carrying schema_version: "1", one target, and one or more cases:

schema_version: "1"
target:
  graph_doc: crates/paladin-battalion/tests/fixtures/graph_docs/approval_gate.json
store: in_memory
llm:
  writer:
    - text: "This looks great, ship it."
cases:
  - name: approve
    input:
      topic: "the quarterly report"
    parley_responses:
      - kind: approval
        value: true
    assertions:
      - kind: route_taken
        nodes: [writer, review]
      - kind: run_status
        status: completed

target is either { graph_doc: <path> } — a WarGraphDoc compiled through a named, host-registered EngineRegistries — or { registered: <name> } — a Rust closure the host test binary registered via ScenarioRunner::register_graph, for graphs that need Function nodes, worker templates, or custom edge evaluators no document can express. store defaults to in_memory; sqlite_temp is available for scenarios that need real persistence (crash/resume cases). llm scripts a ScenarioLlm — global and/or per-node sequences of text, tool_call, or error entries, plus prompt-substring match rules checked before the sequence, consumed one entry per call. Every case may set interrupt_after_superstep (drops the engine after that superstep, resumes over the same store — the simulated-crash technique the integration tests also use) and parley_responses (answers a raised Parley in order).

The JSON Schema

The scenario format's JSON Schema is schemars-derived from the Rust types, never hand-written, so it can never drift from what the runner actually accepts. The generated schema is checked in as a golden file at docs/schemas/eval-scenario.schema.json (repo-root-relative). Regenerate it after changing any scenario type:

UPDATE_EVAL_SCHEMA=1 cargo test -p paladin-eval --test schema_golden

The twelve assertions

Every assertion evaluates against exactly three inputs: the captured TraceRecord stream, the final Battlefield, and the RunOutcome — proving the trace model is sufficient without reaching into engine internals. Every failure renders an actionable message (expected vs. observed, the relevant seq range, and — for node_executed — a full visit table).

run_status/total_tokens_max/supersteps_max fail explicitly — never silently default — when the trace carries no RunFinished record at all. A custom(fn) assertion also exists, but only in the Rust API: it has no serde representation and can never be written into a scenario file.

Running scenarios

cargo test (the evals harness)

Scenarios are discovered and run through a libtest-mimic custom test harness — no proc macro, one runtime-discovered Trial per (file, case) pair:

// tests/evals.rs
paladin_eval::eval_scenarios!("evals/**/*.eval.yaml", |runner: &mut ScenarioRunner| {
    runner.register_graph("approval-gate", build_approval_gate_graph);
});

cargo test --test evals                    # every scenario
cargo test --test evals e2e-2::approve      # one case, by <file-stem>::<case>

paladin-cli eval run

The same ScenarioRunner reachable from the CLI, behind the cli feature:

paladin-cli eval run "evals/*.eval.yaml"

reports one line per case plus a summary, exiting non-zero on any failure.

paladin-cli eval run "evals/*.eval.yaml" --repeat 20

runs every case 20 times and reports a pass rate, exiting non-zero on any divergence across repeats — not merely a failure count. Nondeterminism under fully scripted mocks is treated as a bug to surface, not averaged away; the diverging seq range is named in the output.

paladin-cli eval run "evals/*.eval.yaml" --bless

(re)writes each final_state_snapshot case's blessed file from its case's own final Battlefield — the UPDATE_*=1 env-var bless idiom's CLI cousin.

Live mode: the promotion path

Every scenario above runs against ScenarioLlm, a scripted, deterministic LlmPort implementation — no network call, no real provider, safe for every CI run. --live promotes a scenario to a real provider:

PALADIN_EVAL_LIVE=1 paladin-cli eval run "evals/*.eval.yaml" --live

Live mode requires all three of --live, the PALADIN_EVAL_LIVE environment variable, and a configured provider credential (the same live-API-key handling every live-API test in this project follows) — missing any one refuses with a typed error naming which is missing, never silently falling back to scripted mocks. Content-bearing assertions (final_state_field_equals/_matches, field_json_path_equals, final_state_snapshot) are skipped under live mode — a real provider's non-deterministic output cannot be asserted byte-exact — unless the scenario opts in via live: { allow_content_assertions: true }; structural assertions (node_executed, route_taken, run_status, and the rest) always run.

This is the intended promotion path: every scenario is scripted-mocked in CI by default, and --live is reserved for a pre-release live smoke pass against a real provider, never entered by accident in ordinary CI.

Garrison Memory

The Garrison is Paladin AI's conversation memory system. When attached to a Paladin it stores and retrieves conversation history, giving the agent context across multiple reasoning loops and between invocations.

Garrison is defined in crates/paladin-ports/src/output/garrison_port.rs (the GarrisonPort trait) with adapter implementations in crates/paladin-memory/src/garrison/.

Table of Contents

1. Concepts
2. Quick Start
3. Garrison Adapters
4. GarrisonPort Trait
5. GarrisonConfig
6. Conversation Roles
7. Attaching to a Paladin
8. Long-Term Memory with Embeddings
9. config.yml Reference
10. Error Handling
11. Best Practices

Concepts

Quick Start

use paladin_memory::garrison::InMemoryGarrison;
use paladin_core::platform::container::garrison::{GarrisonConfig, GarrisonEntry, ConversationRole};
use paladin_ports::output::garrison_port::GarrisonPort;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let garrison = Arc::new(InMemoryGarrison::new(GarrisonConfig::default()));

    // Store a user message
    garrison.remember(GarrisonEntry::new(
        ConversationRole::User,
        "What is Rust's ownership model?".to_string(),
    )).await?;

    // Store the assistant reply
    garrison.remember(GarrisonEntry::new(
        ConversationRole::Assistant,
        "Rust's ownership model ensures memory safety without a GC...".to_string(),
    )).await?;

    // Recall last 10 entries
    let history = garrison.recall_recent(10).await?;
    for entry in &history {
        println!("{:?}: {}", entry.role, entry.content);
    }

    // Search by keyword
    let results = garrison.search("ownership", 5).await?;
    println!("Found {} messages about ownership", results.len());

    Ok(())
}

Garrison Adapters

Both adapters are in crates/paladin-memory/src/garrison/.

InMemoryGarrison

use paladin_memory::garrison::InMemoryGarrison;
use paladin_core::platform::container::garrison::GarrisonConfig;

let garrison = InMemoryGarrison::new(GarrisonConfig::default());

SqliteGarrison

use paladin_memory::garrison::SqliteGarrison;
use paladin_core::platform::container::garrison::GarrisonConfig;

let garrison = SqliteGarrison::connect(
    "./garrison.db",
    GarrisonConfig::default(),
    "my-paladin-id",
).await?;

SqliteGarrison::connect() creates the file and runs migrations automatically.

GarrisonPort Trait

Full interface defined in crates/paladin-ports/src/output/garrison_port.rs:

#[async_trait]
pub trait GarrisonPort: Send + Sync {
    /// Store a new conversation entry
    async fn remember(&self, entry: GarrisonEntry) -> Result<(), GarrisonError>;

    /// Retrieve the N most recent entries (newest last)
    async fn recall_recent(&self, limit: usize) -> Result<Vec<GarrisonEntry>, GarrisonError>;

    /// Full-text search across stored entries
    async fn search(&self, query: &str, limit: usize) -> Result<Vec<GarrisonEntry>, GarrisonError>;

    /// Clear all entries from this garrison
    async fn forget_all(&self) -> Result<(), GarrisonError>;

    /// Get storage statistics
    async fn stats(&self) -> Result<GarrisonStats, GarrisonError>;
}

GarrisonStats

pub struct GarrisonStats {
    pub entry_count: usize,   // Total stored entries
    pub total_tokens: u32,    // Cumulative token count
    pub size_bytes: Option<u64>, // Storage size (adapters may not support this)
}

GarrisonConfig

use paladin_core::platform::container::garrison::GarrisonConfig;

// max_entries: window size; max_tokens: token budget per context window
let config = GarrisonConfig::new(200, Some(8000));

Conversation Roles

use paladin_core::platform::container::garrison::ConversationRole;

ConversationRole::User       // Human turn
ConversationRole::Assistant  // Paladin / LLM turn
ConversationRole::System     // System instruction
ConversationRole::Tool       // Tool call result

Attaching to a Paladin

use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin_memory::garrison::SqliteGarrison;
use paladin_core::platform::container::garrison::GarrisonConfig;
use paladin_ports::output::garrison_port::GarrisonPort;
use std::sync::Arc;

let garrison: Arc<dyn GarrisonPort> = Arc::new(
    SqliteGarrison::connect("./memory.db", GarrisonConfig::default(), "agent-1").await?
);

let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are a persistent memory assistant.")
    .with_garrison(garrison)
    .build()
    .await?;

Once attached, the Paladin automatically:

1. Retrieves recent history before each LLM call.
2. Appends the user turn and assistant response after each loop.

Long-Term Memory with Embeddings

The LongTermGarrisonPort trait (also in garrison_port.rs) extends GarrisonPort with semantic similarity search using vector embeddings:

pub trait LongTermGarrisonPort: GarrisonPort {
    async fn remember_with_embedding(
        &self,
        entry: GarrisonEntry,
        embedding: Vec<f32>,
    ) -> Result<(), GarrisonError>;

    async fn search_similar(
        &self,
        query_embedding: Vec<f32>,
        limit: usize,
    ) -> Result<Vec<GarrisonEntry>, GarrisonError>;
}

For full vector-based semantic memory, consider Sanctum — see Sanctum Vector Memory.

config.yml Reference

garrison:
  type: sqlite        # "in_memory" or "sqlite"
  path: ./garrison.db # SQLite only
  max_entries: 100
  max_tokens: 8000    # Optional token budget
  eviction_strategy: oldest  # "oldest" (default)

Error Handling

GarrisonError variants:

Summaries (is_summary)

A GarrisonEntry can carry is_summary: true, marking it as a compressed stand-in for older history rather than a raw conversation turn — written by the SummarizationMiddleware described in the Agent Runtime guide once a conversation's history exceeds a configured token or message threshold. Summaries compound: the middleware builds each new summary from the newest existing summary plus the raw entries newer than it, so an unbounded conversation converges on one summary plus a bounded raw tail rather than growing forever.

GarrisonPort has no delete-by-id method (remember, recall_recent, search, forget_all and stats are the whole trait), so old summary entries are never removed from the store — this is by design, not an oversight. The effective history a consumer should use is computed by finding the newest entry with is_summary == true in a recalled window and taking that entry plus every raw entry newer than it; older summaries remain in the store as historical artifacts, latest-wins.

Best Practices

• Always use SqliteGarrison in production — InMemoryGarrison loses all history when the process restarts.
• Set max_tokens to stay within the LLM's context window; large histories degrade performance.
• Use one Garrison per Paladin — shared garrisons across multiple agents mix conversation contexts and confuse the LLM.
• Call forget_all() between sessions if context carry-over is undesirable (e.g., fresh chat sessions).
• Use search() to retrieve relevant past entries rather than dumping the full history into the prompt.

Sanctum Vector Memory

Sanctum is Paladin AI's long-term semantic memory system. It stores memories as vector embeddings, enabling similarity-based retrieval across sessions — unlike Garrison which stores sequential conversation history, Sanctum finds conceptually similar past experiences.

Sanctum is defined in crates/paladin-ports/src/output/sanctum_port.rs (the SanctumPort trait) with adapter implementations in crates/paladin-memory/src/sanctum/.

Table of Contents

1. Sanctum vs. Garrison
2. Quick Start
3. Sanctum Adapters
4. SanctumPort Trait
5. SanctumEntry and Memory Types
6. Searching with SanctumQuery
7. RAG — Retrieval-Augmented Generation
8. Attaching to a Paladin
9. Docker Setup (Qdrant)
10. config.yml Reference
11. Error Handling
12. Best Practices

Sanctum vs. Garrison

Quick Start

Prerequisite: A running Qdrant instance. Use make dev to start the Docker Compose stack, or docker run -p 6334:6334 qdrant/qdrant.

use paladin_memory::sanctum::QdrantSanctumAdapter;
use paladin_memory::services::rag_retrieval_service::RagRetrievalService;
use paladin_core::platform::container::sanctum::{Memory, MemoryType, SanctumEntry};
use paladin_ports::output::sanctum_port::{SanctumPort, SanctumQuery};
use paladin_ports::output::embedding_port::EmbeddingPort;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let sanctum = Arc::new(
        QdrantSanctumAdapter::new("http://localhost:6334", "memories", 1536).await?
    );
    let embedder: Arc<dyn EmbeddingPort> = Arc::new(openai_embedder());

    // Store a memory
    let content = "Rust's borrow checker prevents data races at compile time.";
    let embedding = embedder.embed_text(content).await?;

    let memory = Memory::builder("agent-1".to_string(), content.to_string())
        .memory_type(MemoryType::Semantic)
        .importance(0.9)
        .build()?;

    let entry = SanctumEntry {
        memory,
        embedding: embedding.vector.clone(),
        dimension: embedding.vector.len(),
    };

    sanctum.store(entry).await?;

    // Semantic search
    let query_vec = embedder.embed_text("memory safety in Rust").await?.vector;
    let results = sanctum.search(SanctumQuery {
        embedding: query_vec,
        limit: 5,
        filter: None,
    }).await?;

    for r in results {
        println!("[score: {:.3}] {}", r.score, r.entry.memory.content);
    }
    Ok(())
}

Sanctum Adapters

Both in crates/paladin-memory/src/sanctum/.

QdrantSanctumAdapter

Production-grade vector store with HNSW indexing.

use paladin_memory::sanctum::QdrantSanctumAdapter;

let sanctum = QdrantSanctumAdapter::new(
    "http://localhost:6334",  // Qdrant URL
    "paladin_memories",       // Collection name
    1536,                     // Vector dimension (match your embedding model)
).await?;

The collection is auto-created if it does not exist.

InMemorySanctum

Fast, ephemeral vector store for development and testing.

use paladin_memory::sanctum::InMemorySanctumAdapter;

let sanctum = InMemorySanctumAdapter::new(1536);

SanctumPort Trait

#[async_trait]
pub trait SanctumPort: Send + Sync {
    /// Store a single memory with its embedding
    async fn store(&self, entry: SanctumEntry) -> Result<(), SanctumError>;

    /// Store multiple memories in a single batch operation
    async fn store_batch(&self, entries: Vec<SanctumEntry>) -> Result<(), SanctumError>;

    /// Search for semantically similar memories
    async fn search(&self, query: SanctumQuery) -> Result<Vec<SanctumSearchResult>, SanctumError>;

    /// Delete a memory by its ID
    async fn delete(&self, id: &str) -> Result<bool, SanctumError>;
}

SanctumEntry and Memory Types

use paladin_core::platform::container::sanctum::{Memory, MemoryType, SanctumEntry};

let memory = Memory::builder("paladin-id".to_string(), "content here".to_string())
    .memory_type(MemoryType::Semantic)   // Semantic | Episodic | Procedural
    .importance(0.8)                      // 0.0–1.0
    .add_metadata("topic".to_string(), serde_json::json!("rust"))
    .build()?;

let entry = SanctumEntry {
    memory,
    embedding: vec![0.1_f32; 1536],  // Your embedding vector
    dimension: 1536,
};

MemoryType variants:

Searching with SanctumQuery

use paladin_ports::output::sanctum_port::{SanctumQuery, SanctumFilter};

// Basic similarity search
let results = sanctum.search(SanctumQuery {
    embedding: query_vec,
    limit: 10,
    filter: None,
}).await?;

// With metadata filter
let results = sanctum.search(SanctumQuery {
    embedding: query_vec,
    limit: 5,
    filter: Some(SanctumFilter {
        paladin_id: Some("agent-1".to_string()),
        memory_type: Some(MemoryType::Semantic),
        min_importance: Some(0.7),
        ..Default::default()
    }),
}).await?;

// Each result contains:
// result.entry   → the SanctumEntry
// result.score   → cosine similarity (0.0–1.0, higher = more similar)

RAG — Retrieval-Augmented Generation

The RagRetrievalService (camelCase Rag, not RAG) in crates/paladin-memory/src/services/rag_retrieval_service.rs automates memory retrieval and injection into the Paladin's prompt context:

use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin_ports::output::sanctum_port::SanctumPort;
use paladin_ports::output::embedding_port::EmbeddingPort;
use std::sync::Arc;

let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are a knowledgeable assistant.")
    .with_sanctum(sanctum_port)       // Vector store
    .with_embedding_port(embedder)    // Embedding provider
    .build()
    .await?;

When Sanctum and an embedding port are both attached, the Paladin will automatically:

1. Embed the user's input query.
2. Retrieve the top-K most similar memories from Sanctum.
3. Prepend retrieved context to the prompt before the LLM call.
4. Extract and store important information from the response.

The RAG retrieval config is controlled via config.yml:

rag:
  enabled: true
  top_k: 5
  min_score: 0.7
  inject_into_prompt: true

Calling RagRetrievalService Directly

Construct the service over a SanctumPort and EmbeddingPort, optionally inject an exact token counter via with_token_counter (the default is a heuristic estimator), and call the timeout-bounded retrieval entry point, retrieve_context_with_timeout:

use paladin_memory::sanctum::InMemorySanctum;
use paladin_memory::services::rag_retrieval_service::{
    RagConfig, RagRetrievalService, retrieve_context_with_timeout,
};
use paladin_memory::token_counter::HeuristicTokenCounter;
use paladin_ports::output::embedding_port::{Embedding, EmbeddingError, EmbeddingPort};
use paladin_ports::output::sanctum_port::SanctumPort;

/// A deterministic, no-network embedder — enough to drive the RAG example without a
/// real embedding provider.
struct MockEmbedder;

#[async_trait]
impl EmbeddingPort for MockEmbedder {
    async fn embed_text(&self, text: &str) -> Result<Embedding, EmbeddingError> {
        Ok(Embedding {
            vector: vec![0.0_f32; 8],
            model: "mock-embedder".to_string(),
            dimension: 8,
            token_count: Some(text.split_whitespace().count() as u32),
        })
    }

    async fn embed_batch(&self, texts: &[&str]) -> Result<Vec<Embedding>, EmbeddingError> {
        let mut out = Vec::with_capacity(texts.len());
        for text in texts {
            out.push(self.embed_text(text).await?);
        }
        Ok(out)
    }

    fn dimension(&self) -> usize {
        8
    }

    fn model_name(&self) -> &str {
        "mock-embedder"
    }
}

/// Build a `RagRetrievalService` (camelCase `Rag`, not `RAG`) over an in-memory
/// Sanctum, inject an exact token counter via `with_token_counter`, and call the
/// timeout-bounded retrieval entry point. Returns a `RagRetrievalResult` — the
/// retained memories plus the Commissary's `shed: Vec<ShedItem>` accounting for
/// anything dropped to fit the token budget.
pub async fn retrieve_with_timeout() -> Result<(), Box<dyn std::error::Error>> {
    let sanctum: Arc<dyn SanctumPort> = Arc::new(InMemorySanctum::new(1_000));
    let embedding: Arc<dyn EmbeddingPort> = Arc::new(MockEmbedder);

    let service = RagRetrievalService::new(sanctum, embedding, RagConfig::default())
        .with_token_counter(Arc::new(HeuristicTokenCounter));

    let result =
        retrieve_context_with_timeout(&service, "agent-1", "memory safety in Rust", 5).await?;

    println!(
        "retained={} shed={}",
        result.memories.len(),
        result.shed.len()
    );
    Ok(())
}

retrieve_context_with_timeout wraps RagRetrievalService::retrieve_context, which rations the ranked search results through the Commissary and returns a RagRetrievalResult:

pub struct RagRetrievalResult {
    pub memories: Vec<RagRetainedMemory>,  // retained, descending relevance order
    pub shed: Vec<ShedItem>,               // memories dropped entirely to fit the budget
    pub prompt_tokens: u32,
    pub allotted_tokens: u32,
    pub exact_tally: bool,
}

Every memory that does not survive rationing appears in shed, labelled by its memory UUID — nothing is dropped silently. Retrieval failures surface as a typed RagRetrievalError (Sanctum, Commissary, BudgetTooLarge, UnmatchedDispensedLabel, DuplicateMemoryId).

Rendering the Result — the Omission Marker

RagRetrievalService::format_for_prompt renders a RagRetrievalResult into prompt context — the same renderer the facade's PaladinExecutionService::format_retrieved_context mirrors, so the two can never drift — and appends a trailing omission-marker line whenever shed is non-empty, naming how many lower-relevance memories were dropped entirely and the token budget they were rationed against:

use paladin_memory::services::rag_retrieval_service::{
    RagRetrievalResult, RagRetrievalService as Service,
};

/// Render a `RagRetrievalResult` into prompt context exactly as
/// `RagRetrievalService::format_for_prompt` does — the same renderer the facade's
/// `PaladinExecutionService::format_retrieved_context` mirrors — appending the shared
/// RAG omission marker whenever memories were shed to stay within the token budget.
pub fn format_result(service: &Service, result: &RagRetrievalResult) -> String {
    service.format_for_prompt(result)
}

Attaching to a Paladin

use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin_memory::sanctum::QdrantSanctumAdapter;
use std::sync::Arc;

let sanctum = Arc::new(
    QdrantSanctumAdapter::new("http://localhost:6334", "memories", 1536).await?
);
let embedder = Arc::new(openai_embedder());

let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are a knowledge-augmented assistant.")
    .with_sanctum(sanctum)
    .with_embedding_port(embedder)
    .build()
    .await?;

Docker Setup (Qdrant)

The development Docker Compose stack includes Qdrant:

make dev           # Starts Redis, MinIO, MySQL, and Qdrant
# or individually:
docker run -p 6334:6334 -p 6333:6333 qdrant/qdrant

Default connection: http://localhost:6334 (gRPC) / http://localhost:6333 (REST dashboard).

config.yml Reference

sanctum:
  type: qdrant           # "qdrant" or "in_memory"
  url: "http://localhost:6334"
  collection: paladin_memories
  vector_dimension: 1536 # Must match embedding model output dimension

rag:
  enabled: true
  top_k: 5               # Number of similar memories to retrieve
  min_score: 0.7         # Minimum cosine similarity threshold
  inject_into_prompt: true

memory_extraction:
  enabled: true
  strategy: selective    # "all" or "selective"

Error Handling

SanctumError variants:

Best Practices

• Match dimensions — set vector_dimension to exactly the output size of your embedding model (OpenAI text-embedding-3-small = 1536, text-embedding-3-large = 3072).
• Use store_batch() when loading a knowledge base — it is significantly faster than individual store() calls.
• Set min_score in SanctumQuery to filter out low-quality matches; 0.7 is a good starting point.
• Separate collections per agent or per use-case to avoid cross-contamination in multi-agent systems.
• Use InMemorySanctum in tests to avoid requiring a running Qdrant instance.

Herald Output Formatting

The Herald system provides pluggable output formatters for Paladin and Battalion execution results. A Herald transforms a PaladinResult or BattalionResult into a human-readable or machine-readable string — JSON, Markdown, or ASCII table.

The Herald trait is defined in crates/paladin-core/src/platform/container/herald.rs. Adapters are in src/infrastructure/adapters/herald/.

Table of Contents

1. Overview
2. Available Heralds
3. Herald Trait
4. Attaching to a Paladin
5. Attaching to a Battalion Service
6. Custom Herald Implementation
7. Streaming Output
8. Error Handling

Overview

Available Heralds

JsonHerald

Serialises PaladinResult and BattalionResult to JSON.

use paladin::infrastructure::adapters::herald::JsonHerald;
use paladin::infrastructure::adapters::herald::json_herald::JsonHeraldConfig;

// Default: pretty = true, include_metadata = true
let herald = JsonHerald::new();

// Compact JSON without metadata
let herald = JsonHerald::with_config(JsonHeraldConfig {
    pretty: false,
    include_metadata: false,
});

let json_str = herald.format_paladin_result(&result)?;
// "usage" is the full TokenUsage split, serialized as a stable six-key object:
// {"output": "...", "usage": {"prompt_tokens": 100, "completion_tokens": 50,
//   "total_tokens": 150, "cache_read_tokens": null, "cache_write_tokens": null,
//   "reasoning_tokens": null}, "execution_time_ms": 1230, ...}

MarkdownHerald

Formats results with Markdown headings, status badges, and code blocks. Supports ANSI colour codes for terminal output.

use paladin::infrastructure::adapters::herald::MarkdownHerald;
use paladin::infrastructure::adapters::herald::markdown_herald::MarkdownHeraldConfig;

// Default: auto-detects terminal colour support
let herald = MarkdownHerald::new();

// Custom: force no colours, H1 headings
let herald = MarkdownHerald::with_config(MarkdownHeraldConfig {
    include_colors: false,
    heading_level: 1,
});

TableHerald

Renders results as ASCII tables using comfy-table.

use paladin::infrastructure::adapters::herald::TableHerald;
use paladin::infrastructure::adapters::herald::table_herald::TableHeraldConfig;

// Default configuration
let herald = TableHerald::default();

// Custom: 80-char column width, rounded borders
let herald = TableHerald::new(TableHeraldConfig {
    max_column_width: 80,
    border_style: "rounded".to_string(),
});

Herald Trait

The Herald trait has seven methods, not three — format_stream_chunk returns Result<Option<String>, HeraldError> where None means "buffering, not ready to emit yet", not an error:

pub trait Herald: Send + Sync {
    /// Format a completed Paladin result
    fn format_paladin_result(&self, result: &PaladinResult) -> Result<String, HeraldError>;

    /// Format a completed Battalion result
    fn format_battalion_result(&self, result: &BattalionResult) -> Result<String, HeraldError>;

    /// Format a streaming chunk. `Ok(None)` means "buffering -- not ready to emit yet".
    fn format_stream_chunk(&self, chunk: &StreamChunk) -> Result<Option<String>, HeraldError>;

    /// Finalize streaming output with metadata (tokens, timing) once the stream completes.
    fn finalize_stream(&self, metadata: &ExecutionMetadata) -> Result<String, HeraldError>;

    /// Format an error for display. Infallible -- never returns Err.
    fn format_error(&self, error: &PaladinError) -> String;

    /// Formatter identifier, e.g. "json", "markdown", "table".
    fn name(&self) -> &str;

    /// MIME type of the formatted output, e.g. "application/json".
    fn mime_type(&self) -> &str;
}

Attaching to a Paladin

use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin::infrastructure::adapters::herald::JsonHerald;
use paladin_core::platform::container::herald::Herald;
use std::sync::Arc;

let herald: Arc<dyn Herald> = Arc::new(JsonHerald::new());

let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are an API assistant.")
    .with_herald(herald)
    .build()
    .await?;

let result = paladin.execute("List all Rust 2024 edition features").await?;
// result.output is already formatted as JSON
println!("{}", result.output);

Attaching to a Battalion Service

Formation, Phalanx, and other services accept a Herald via .with_herald():

use paladin_battalion::phalanx_service::PhalanxExecutionService;
use paladin::infrastructure::adapters::herald::MarkdownHerald;
use std::sync::Arc;

let service = PhalanxExecutionService::new(paladin_port)
    .with_herald(Arc::new(MarkdownHerald::new()));

let result = service.execute(&phalanx, "Analyse this dataset").await?;
println!("{}", result.output);

Custom Herald Implementation

Implement the Herald trait — all seven methods — to create a bespoke formatter:

use paladin_core::platform::container::herald::{
    BattalionResult, ExecutionMetadata, Herald, HeraldError, PaladinError, PaladinResult,
    StreamChunk,
};

/// A bespoke CSV formatter implementing the full seven-method `Herald` trait — the
/// same seven methods `output-formatting.md` documents, not the three-method stale
/// shape this page previously showed.
pub struct CsvHerald;

/// Escape a field for inclusion in a comma-separated row. This bespoke example
/// deliberately keeps escaping minimal (commas only, no quoting/newline handling) --
/// it is not RFC 4180-complete -- but applies it consistently across every method
/// below so no field can silently break row alignment.
fn csv_escape(field: &str) -> String {
    field.replace(',', ";")
}

impl Herald for CsvHerald {
    fn format_paladin_result(&self, result: &PaladinResult) -> Result<String, HeraldError> {
        Ok(format!(
            "{},{},{},{}\n",
            csv_escape(&result.output),
            result.usage.total_tokens,
            result.execution_time_ms,
            csv_escape(&format!("{:?}", result.stop_reason)),
        ))
    }

    fn format_battalion_result(&self, result: &BattalionResult) -> Result<String, HeraldError> {
        Ok(format!("{}\n", csv_escape(&result.final_output)))
    }

    fn format_stream_chunk(&self, chunk: &StreamChunk) -> Result<Option<String>, HeraldError> {
        Ok(Some(chunk.content.clone()))
    }

    fn finalize_stream(&self, metadata: &ExecutionMetadata) -> Result<String, HeraldError> {
        Ok(format!(
            "# total_tokens={},duration_ms={}\n",
            metadata.token_usage.total_tokens,
            csv_escape(&format!("{:?}", metadata.duration_ms)),
        ))
    }

    fn format_error(&self, error: &PaladinError) -> String {
        format!("error,{}\n", csv_escape(&error.to_string()))
    }

    fn name(&self) -> &str {
        "csv"
    }

    fn mime_type(&self) -> &str {
        "text/csv"
    }
}

Streaming Output

Use format_stream_chunk() during execute_stream():

use paladin_ports::output::paladin_port::PaladinStreamChunk;
use paladin_core::platform::container::herald::Herald;

let mut stream = paladin.execute_stream("Generate a long report").await?;
let herald = MarkdownHerald::new();

while let Some(chunk_result) = stream.recv().await {
    match chunk_result {
        Ok(chunk) => {
            // chunk.text is raw text; wrap in a StreamChunk for the Herald
            if let Some(formatted) = herald.format_stream_chunk(&chunk.into())? {
                print!("{}", formatted);
            }
            if chunk.is_final { break; }
        }
        Err(e) => eprintln!("Stream error: {}", e),
    }
}

Error Handling

HeraldError variants from paladin_core::platform::container::herald_error:

Maneuver: Flow DSL Orchestration

Declarative multi-agent workflows with dynamic execution patterns

Table of Contents

1. Overview
2. Quick Start
3. Flow DSL Syntax
4. Execution Patterns
5. Configuration
6. CLI Commands
7. Visualization
8. Error Handling
9. Performance
10. Best Practices
11. API Reference
12. Troubleshooting

Overview

Maneuver is a declarative Battalion orchestration pattern that uses a Flow DSL (Domain-Specific Language) to define complex agent execution patterns. Unlike other Battalion patterns that require explicit code, Maneuver allows you to express workflows as simple text expressions.

Key Features

• Declarative Syntax: Define workflows as text expressions (agent1 -> agent2)
• Mixed Patterns: Combine sequential and parallel execution in a single flow
• Visual Feedback: ASCII and Mermaid.js visualization of flow graphs
• Type-Safe Parsing: Compile-time validation of flow expressions
• Commander Integration: Automatic pattern detection for "flow" keywords

Comparison with Other Patterns

Quick Start

Installation

Maneuver is included in paladin-battalion. Add it to your workspace:

[dependencies]
paladin-battalion = { version = "0.10.0", path = "crates/paladin-battalion" }
tokio = { version = "1.0", features = ["full"] }

Basic Example

use paladin_battalion::maneuver::service::ManeuverExecutionService;
use paladin_battalion::maneuver::Maneuver;
use paladin_battalion::maneuver::parser::FlowParser;
use std::collections::HashMap;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Define flow using DSL
    let flow = FlowParser::parse("analyzer -> summarizer -> reviewer")?;

    // Create Paladins
    let mut agents = HashMap::new();
    agents.insert("analyzer".to_string(), create_paladin("analyzer", "Analyze input"));
    agents.insert("summarizer".to_string(), create_paladin("summarizer", "Summarize"));
    agents.insert("reviewer".to_string(), create_paladin("reviewer", "Final review"));

    // Create Maneuver
    let maneuver = Maneuver::new("doc-workflow", agents, flow, Default::default())?;

    // Execute
    let service = ManeuverExecutionService::new(Arc::new(paladin_port));
    let result = service.execute(&maneuver, "Document to process").await?;

    println!("Final output: {}", result.final_output);
    Ok(())
}

CLI Quick Start

# Create a Maneuver configuration
paladin battalion new --name my-workflow --type maneuver -o workflow.yaml

# Visualize the flow
paladin maneuver visualize -c workflow.yaml --format ascii

# Validate configuration
paladin maneuver validate -c workflow.yaml --verbose

# Execute the workflow (there is no -i/--input flag on `battalion run` —
# it prompts "Enter input for maneuver" interactively on stdin)
paladin battalion run -c workflow.yaml -t maneuver

Flow DSL Syntax

The Flow DSL uses a simple, intuitive syntax for defining agent execution patterns.

Basic Syntax

Sequential Execution

agent1 -> agent2 -> agent3

Output from agent1 flows as input to agent2, then to agent3.

Parallel Execution

(agent1, agent2)

Both agent1 and agent2 execute concurrently with the same input.

Note: Use commas (,) for parallel, not pipes (|).

Nested Patterns

agent1 -> (agent2, agent3) -> agent4

1. agent1 executes first
2. Output flows to both agent2 and agent3 (parallel)
3. Combined output flows to agent4

Syntax Rules

Valid Examples

# Simple sequential
agent1 -> agent2

# Simple parallel
(agent1, agent2)

# Mixed nested
start -> (analyzer, reviewer) -> end

# Complex workflow
intake -> (technical, business, security) -> synthesis -> review

# Deep nesting
a -> (b -> (c, d), e) -> f

Invalid Syntax

# ❌ Pipe operator (use comma instead)
(agent1 | agent2)

# ❌ Missing parentheses for parallel
agent1 -> agent2, agent3

# ❌ Spaces in agent names
my agent -> another agent

# ❌ Empty groups
() -> agent1

# ❌ Trailing operators
agent1 ->

Execution Patterns

Sequential Pattern

Flow: agent1 -> agent2 -> agent3

Behavior:

1. Execute agent1 with initial input
2. Pass agent1 output to agent2 as input
3. Pass agent2 output to agent3 as input
4. Return agent3 output as final result

Use Cases:

• Data transformation pipelines
• Multi-stage analysis
• Progressive refinement

Example:

// Flow: "extractor -> translator -> formatter"
let flow = FlowParser::parse("extractor -> translator -> formatter")?;

// Input: "Extract data from: <raw_text>"
// extractor output: "Data: {...}"
// translator output: "Translated: {...}"
// formatter output: "Formatted report: {...}" (final)

Parallel Pattern

Flow: (agent1, agent2, agent3)

Behavior:

1. Execute all agents concurrently with same input
2. Wait for all to complete
3. Combine outputs (concatenation or custom logic)
4. Return combined result

Use Cases:

• Multi-perspective analysis
• Expert panel reviews
• Parallel processing

Example:

// Flow: "(tech_reviewer, business_reviewer, security_reviewer)"
let flow = FlowParser::parse("(tech_reviewer, business_reviewer, security_reviewer)")?;

// All receive: "Review this proposal: {...}"
// Output combines all three perspectives

Nested Pattern

Flow: agent1 -> (agent2, agent3) -> agent4

Behavior:

1. Execute agent1 with initial input
2. Pass output to both agent2 and agent3 (parallel)
3. Wait for both to complete
4. Combine their outputs
5. Pass combined result to agent4
6. Return agent4 output as final result

Use Cases:

• Divide-and-conquer workflows
• Multi-faceted analysis with synthesis
• Complex decision trees

Example:

// Flow: "analyzer -> (summarizer, translator) -> reviewer"
let flow = FlowParser::parse("analyzer -> (summarizer, translator) -> reviewer")?;

// 1. analyzer processes input
// 2. summarizer + translator work in parallel on analysis
// 3. reviewer synthesizes both outputs into final result

Execution Order Visualization

Sequential: agent1 → agent2 → agent3
           t₀      t₁      t₂

Parallel:   agent1
           ↙      ↘
       agent2    agent3
           ↘      ↙
         (combine)

Nested:     agent1
              ↓
          ┌───┴───┐
      agent2   agent3
          └───┬───┘
           agent4

Configuration

Maneuver Configuration

use paladin_battalion::maneuver::{ManeuverConfig, ErrorStrategy, OutputFormat};
use std::time::Duration;

let config = ManeuverConfig::new()
    .with_error_strategy(ErrorStrategy::ContinueParallel)
    .with_output_format(OutputFormat::Concatenate)
    .with_pass_output_as_input(true)
    .with_timeout(Duration::from_secs(300))
    .with_timing_metrics(true);

let maneuver = Maneuver::new("workflow", agents, flow, config)?;

Error Strategies

pub enum ErrorStrategy {
    /// Stop immediately on first error
    FailFast,

    /// Continue parallel branches but fail sequential chains on error
    ContinueParallel,

    /// Log errors but continue execution regardless
    IgnoreErrors,
}

When to Use:

• FailFast: Critical workflows where any failure invalidates the result
• ContinueParallel: Parallel sections can fail independently
• IgnoreErrors: Best-effort workflows, collect whatever partial results are available

Output Formats

pub enum OutputFormat {
    /// Concatenate all outputs with newlines (default)
    Concatenate,

    /// JSON array with each agent's output as an element
    JsonArray,
}

Example Outputs:

// Concatenate (default)
"Output from agent1\n---\nOutput from agent2\n---\nOutput from agent3"

// JsonArray
r#"["output from agent1", "output from agent2", "output from agent3"]"#

YAML Configuration

type: maneuver
name: "document-workflow"

# Flow expression using DSL
flow: "analyzer -> (summarizer, translator) -> reviewer"

# Available Paladins (must match names in flow)
paladins:
  - inline:
      name: "analyzer"
      system_prompt: "Analyze the input document"
      model: "gpt-4"
      temperature: 0.7
      provider:
        type: openai

  - inline:
      name: "summarizer"
      system_prompt: "Create a concise summary"
      model: "gpt-4"
      temperature: 0.5
      provider:
        type: openai

  - inline:
      name: "translator"
      system_prompt: "Translate to simple language"
      model: "gpt-4"
      temperature: 0.5
      provider:
        type: openai

  - inline:
      name: "reviewer"
      system_prompt: "Final review and synthesis"
      model: "gpt-4"
      temperature: 0.6
      provider:
        type: openai

# Optional: visualize before execution
visualize: "ascii"

CLI Commands

Create Maneuver Configuration

paladin battalion new --name my-workflow --type maneuver --output workflow.yaml

Creates a template YAML file with example flow and agents.

Visualize Flow

# ASCII tree visualization
paladin maneuver visualize -c workflow.yaml --format ascii

# Mermaid flowchart (for documentation)
paladin maneuver visualize -c workflow.yaml --format mermaid

# Save to file
paladin maneuver visualize -c workflow.yaml --format ascii -o flow.txt

Output Example (ASCII):

└─> analyzer
    ├─> [PARALLEL]
    │   ├─> summarizer
    │   └─> translator
    └─> reviewer

Output Example (Mermaid):

flowchart LR
    agent_analyzer
    agent_analyzer --> parallel_1[Parallel]
    parallel_1 --> agent_summarizer
    parallel_1 --> agent_translator
    parallel_1 --> agent_reviewer

Validate Configuration

# Basic validation
paladin maneuver validate -c workflow.yaml

# Verbose validation with detailed output
paladin maneuver validate -c workflow.yaml --verbose

Validates:

• Flow expression syntax
• All agents referenced in flow exist in config
• Paladin configuration structure
• Provider settings

Execute Maneuver

# Interactive execution — prompts "Enter input for maneuver" on stdin;
# `battalion run` has no -i/--input flag
paladin battalion run -c workflow.yaml -t maneuver

# Save output to file
paladin battalion run -c workflow.yaml -t maneuver -o result.json

# Verbose execution
paladin battalion run -c workflow.yaml -t maneuver -v

Visualization

ASCII Tree Format

Perfect for terminal output and debugging:

└─> intake
    ├─> [PARALLEL]
    │   ├─> technical
    │   ├─> business
    │   └─> security
    └─> synthesis
        └─> review

Features:

• Box-drawing characters (├─>, └─>, │)
• Clear hierarchy visualization
• Sequential and parallel markers
• Nested structure representation

Mermaid Flowchart Format

Ideal for documentation and presentations:

flowchart LR
    agent_intake
    agent_intake --> parallel_1[Parallel]
    parallel_1 --> agent_technical
    parallel_1 --> agent_business
    parallel_1 --> agent_security
    parallel_1 --> agent_synthesis
    agent_synthesis --> agent_review

Features:

• Web-ready visualization
• Integrates with GitHub/GitLab/documentation tools
• Professional diagram quality
• Exportable to SVG/PNG

Programmatic Visualization

use paladin_battalion::maneuver::visualizer::{FlowVisualizer, VisualizationFormat};

let flow = FlowParser::parse("a -> (b, c) -> d")?;

// ASCII visualization
let ascii = FlowVisualizer::to_ascii(&flow);
println!("{}", ascii);

// Mermaid visualization
let mermaid = FlowVisualizer::to_mermaid(&flow);
println!("{}", mermaid);

// Using format parameter
let viz = FlowVisualizer::visualize(&flow, VisualizationFormat::Ascii);

Error Handling

Validation Errors

use paladin_battalion::maneuver::parser::FlowParseError;

match FlowParser::parse("agent1 -> (agent2 | agent3)") {
    Ok(flow) => { /* Success */ },
    Err(FlowParseError::InvalidCharacter { position, character }) => {
        eprintln!("Invalid character '{}' at position {}", character, position);
        // Error: Invalid character '|' at position 17
    },
    Err(e) => eprintln!("Parse error: {}", e),
}

Execution Errors

use paladin_battalion::maneuver::ManeuverError;

match service.execute(&maneuver, input).await {
    Ok(result) => println!("Success: {}", result.final_output),
    Err(ManeuverError::AgentNotFound { agent_name, available_agents }) => {
        eprintln!("Agent '{}' not found. Available: {:?}", agent_name, available_agents);
    },
    Err(ManeuverError::ExecutionError(msg)) => {
        eprintln!("Execution failed: {}", msg);
    },
    Err(e) => eprintln!("Error: {}", e),
}

Error Recovery

// Configure error handling strategy
let config = ManeuverConfig::new()
    .with_error_strategy(ErrorStrategy::IgnoreErrors);

// Execution continues despite failures
let result = service.execute(&maneuver, input).await?;

// Check status
match result.status {
    ExecutionStatus::Success => println!("All agents succeeded"),
    ExecutionStatus::PartialSuccess => println!("Some agents failed"),
    ExecutionStatus::Failed => println!("Execution failed"),
}

// Inspect individual outputs
for (agent, output) in result.step_outputs {
    if output.is_empty() {
        println!("Agent {} failed", agent);
    }
}

Performance

Benchmarks

Based on battalion_benchmarks.rs:

Optimization Tips

1. Minimize Sequential Chains

❌ Slow: a -> b -> c -> d -> e -> f (6 sequential calls)

✅ Fast: a -> (b, c, d) -> e (3 stages total)

2. Use Parallel Where Possible

// Slow: Sequential when order doesn't matter
"tech_review -> security_review -> legal_review"

// Fast: Parallel independent reviews
"(tech_review, security_review, legal_review)"

3. Configure Timeouts

let config = ManeuverConfig::new()
    .with_timeout(Duration::from_secs(120))  // Per-agent timeout
    .with_error_strategy(ErrorStrategy::ContinueParallel);  // Don't wait for failures

4. Optimize Agent Prompts

• Keep system prompts concise
• Use lower max_loops values when possible
• Set appropriate temperature values

5. Monitor Timing Metrics

let config = ManeuverConfig::new()
    .with_timing_metrics(true);

let result = service.execute(&maneuver, input).await?;

if let Some(metrics) = result.timing_metrics {
    for (agent, duration) in metrics {
        println!("{}: {}ms", agent, duration.as_millis());
    }
}

Best Practices

1. Flow Design

Keep Flows Simple

// ✅ Good: Clear, easy to understand
"intake -> analyze -> decide"

// ❌ Bad: Too complex, hard to debug
"a -> (b -> (c, d -> (e, f)), g -> (h, i)) -> j"

Use Descriptive Names

// ✅ Good: Clear purpose
"document_analyzer -> sentiment_classifier -> report_generator"

// ❌ Bad: Cryptic names
"agent1 -> agent2 -> agent3"

2. Agent Configuration

Specialize Agents

Each agent should have a clear, focused responsibility:

- name: "analyzer"
  system_prompt: "Analyze technical feasibility only. Focus on implementation challenges."

- name: "risk_assessor"
  system_prompt: "Assess security and privacy risks only."

- name: "synthesizer"
  system_prompt: "Combine technical analysis and risk assessment into recommendation."

Use Consistent Naming

Match agent names in flow expression exactly:

// Flow uses: analyzer, summarizer, reviewer
flow: "analyzer -> summarizer -> reviewer"

// Paladins must use same names:
agents.insert("analyzer", ...);
agents.insert("summarizer", ...);
agents.insert("reviewer", ...);

3. Error Handling

Always Handle Errors

// ✅ Good: Explicit error handling
match service.execute(&maneuver, input).await {
    Ok(result) => process_result(result),
    Err(ManeuverError::AgentNotFound { agent_name, .. }) => {
        log_error!("Missing agent: {}", agent_name);
        return default_result();
    },
    Err(e) => {
        log_error!("Execution failed: {}", e);
        retry_with_fallback();
    },
}

// ❌ Bad: Unwrapping
let result = service.execute(&maneuver, input).await.unwrap();

Choose Appropriate Strategy

// Critical workflows: fail fast
let config = ManeuverConfig::new()
    .with_error_strategy(ErrorStrategy::FailFast);

// Best-effort workflows: collect partial results
let config = ManeuverConfig::new()
    .with_error_strategy(ErrorStrategy::IgnoreErrors);

4. Testing

Validate Flows Early

#[test]
fn test_workflow_validation() {
    let flow = FlowParser::parse("analyzer -> summarizer").unwrap();

    let mut agents = HashMap::new();
    agents.insert("analyzer".to_string(), create_test_agent("analyzer"));
    agents.insert("summarizer".to_string(), create_test_agent("summarizer"));

    let result = Maneuver::new("test", agents, flow, Default::default());
    assert!(result.is_ok());
}

Test Visualizations

#[test]
fn test_flow_visualization() {
    let flow = FlowParser::parse("a -> (b, c)").unwrap();
    let ascii = FlowVisualizer::to_ascii(&flow);

    assert!(ascii.contains("PARALLEL"));
    assert!(ascii.contains("a"));
    assert!(ascii.contains("b"));
    assert!(ascii.contains("c"));
}

5. Documentation

Document Complex Flows

# Flow explanation:
# 1. Intake agent validates and normalizes input
# 2. Three specialists analyze in parallel:
#    - Technical feasibility
#    - Business value
#    - Security implications
# 3. Synthesis agent combines all perspectives
# 4. Final review for quality assurance
flow: "intake -> (technical, business, security) -> synthesis -> review"

API Reference

Core Types

FlowParser

pub struct FlowParser;

impl FlowParser {
    /// Parse a flow expression from text
    pub fn parse(input: &str) -> Result<FlowExpression, FlowParseError>
}

FlowExpression

#[derive(Debug, Clone, PartialEq, Eq)]
pub enum FlowExpression {
    /// Single agent execution
    Agent(String),

    /// Sequential execution (agent₁ → agent₂ → ...)
    Sequential(Vec<FlowExpression>),

    /// Parallel execution (agent₁, agent₂, ...)
    Parallel(Vec<FlowExpression>),
}

impl FlowExpression {
    /// Get all agent names referenced in this expression
    pub fn agent_names(&self) -> Vec<String>
}

Maneuver

pub struct Maneuver {
    pub name: String,
    pub agents: HashMap<String, Paladin>,
    pub flow: FlowExpression,
    pub config: ManeuverConfig,
}

impl Maneuver {
    /// Create a new Maneuver with validation
    pub fn new(
        name: impl Into<String>,
        agents: HashMap<String, Paladin>,
        flow: FlowExpression,
        config: ManeuverConfig,
    ) -> Result<Self, ManeuverError>

    /// Validate that all flow agents exist
    pub fn validate(&self) -> Result<(), ManeuverError>
}

ManeuverConfig

pub struct ManeuverConfig {
    pub error_strategy: ErrorStrategy,
    pub output_format: OutputFormat,
    pub pass_output_as_input: bool,
    pub timeout: Option<Duration>,
    pub collect_timing_metrics: bool,
    pub detailed_observability: bool,
}

impl ManeuverConfig {
    pub fn new() -> Self
    pub fn with_error_strategy(self, strategy: ErrorStrategy) -> Self
    pub fn with_output_format(self, format: OutputFormat) -> Self
    pub fn with_timeout(self, timeout: Duration) -> Self
}

ManeuverResult

pub struct ManeuverResult {
    /// Final aggregated output
    pub final_output: String,

    /// Individual agent outputs
    pub step_outputs: HashMap<String, String>,

    /// Execution order
    pub execution_order: Vec<String>,

    /// Per-agent timing (if enabled)
    pub timing_metrics: Option<HashMap<String, Duration>>,

    /// Execution status
    pub status: ExecutionStatus,
}

ManeuverExecutionService

pub struct ManeuverExecutionService {
    paladin_port: Arc<dyn PaladinPort>,
}

impl ManeuverExecutionService {
    pub fn new(paladin_port: Arc<dyn PaladinPort>) -> Self

    pub async fn execute(
        &self,
        maneuver: &Maneuver,
        input: &str,
    ) -> Result<ManeuverResult, ManeuverError>
}

Visualization

FlowVisualizer

pub struct FlowVisualizer;

impl FlowVisualizer {
    /// Generate ASCII tree visualization
    pub fn to_ascii(flow: &FlowExpression) -> String

    /// Generate Mermaid flowchart
    pub fn to_mermaid(flow: &FlowExpression) -> String

    /// Generate visualization in specified format
    pub fn visualize(flow: &FlowExpression, format: VisualizationFormat) -> String
}

pub enum VisualizationFormat {
    Ascii,
    Mermaid,
}

Troubleshooting

Common Issues

1. Parse Error: Invalid Character '|'

Problem: Using pipe operator for parallel execution

// ❌ Wrong
let flow = FlowParser::parse("(agent1 | agent2)")?;

Solution: Use comma instead

// ✅ Correct
let flow = FlowParser::parse("(agent1, agent2)")?;

2. AgentNotFound Error

Problem: Agent name in flow doesn't match configured agents

// Flow references "analyzer"
let flow = FlowParser::parse("analyzer -> summarizer")?;

// But agent is named "Analyzer" (different case)
agents.insert("Analyzer".to_string(), paladin);

Solution: Use exact same names

// ✅ Correct - exact match
agents.insert("analyzer".to_string(), paladin);

3. Missing Parentheses for Parallel

Problem: Forgetting parentheses around parallel agents

// ❌ Wrong - will be parsed as "agent1 -> agent2", "agent3"
let flow = FlowParser::parse("agent1 -> agent2, agent3")?;

Solution: Always use parentheses for parallel

// ✅ Correct
let flow = FlowParser::parse("agent1 -> (agent2, agent3)")?;

4. Timeout Errors

Problem: Agents taking too long to execute

// Default timeout may be too short
let config = ManeuverConfig::default();  // 300s default

Solution: Increase timeout for slow workflows

// ✅ Longer timeout
let config = ManeuverConfig::new()
    .with_timeout(Duration::from_secs(600));  // 10 minutes

5. Partial Results from Parallel Execution

Problem: Some agents fail in parallel execution

Solution: Use appropriate error strategy

// Continue despite failures
let config = ManeuverConfig::new()
    .with_error_strategy(ErrorStrategy::ContinueParallel);

let result = service.execute(&maneuver, input).await?;

// Check which agents succeeded
for (agent, output) in result.step_outputs {
    if !output.is_empty() {
        println!("{} succeeded: {}", agent, output);
    }
}

Debugging Tips

1. Enable Verbose Logging

env_logger::init();  // In main()

// Set RUST_LOG=debug
// Will show detailed execution trace

2. Visualize Before Executing

paladin maneuver visualize -c config.yaml --format ascii

Visual inspection often reveals flow logic issues.

3. Validate Configuration

paladin maneuver validate -c config.yaml --verbose

Catches configuration mismatches before execution.

4. Check Timing Metrics

let config = ManeuverConfig::new()
    .with_timing_metrics(true);

let result = service.execute(&maneuver, input).await?;

if let Some(metrics) = result.timing_metrics {
    for (agent, duration) in metrics {
        if duration > Duration::from_secs(60) {
            println!("⚠️  {} took {}s", agent, duration.as_secs());
        }
    }
}

5. Inspect Individual Outputs

let result = service.execute(&maneuver, input).await?;

// Check each agent's output
for agent in result.execution_order {
    if let Some(output) = result.step_outputs.get(&agent) {
        println!("\n=== {} ===", agent);
        println!("{}", output);
    }
}

Getting Help

• Documentation: https://github.com/DF3NDR/paladin-dev-env/docs
• Issues: https://github.com/DF3NDR/paladin-dev-env/issues
• Discussions: https://github.com/DF3NDR/paladin-dev-env/discussions
• Examples: examples/ directory in repository

Advanced Topics

Custom Output Formatting

use paladin_battalion::maneuver::OutputFormat;

// Implement custom aggregation logic
let config = ManeuverConfig::new()
    .with_output_format(OutputFormat::JsonArray);

// Result will be JSON:
// {"agent1": "output1", "agent2": "output2"}

Integration with Commander

Commander automatically detects Maneuver patterns:

use paladin_battalion::commander::CommanderBuilder;
use paladin_core::platform::container::battalion::BattalionStrategy;

// Maneuver is explicit-only — it is NOT selected by Auto mode.
// You must explicitly set BattalionStrategy::Maneuver and provide a flow expression.
let commander = CommanderBuilder::new(paladin_port)
    .strategy(BattalionStrategy::Maneuver)
    .paladins(paladins)
    .flow("agent1 -> agent2 -> agent3".to_string())
    .build()?;

let result = commander.execute("Process this document").await?;

Performance Tuning

For high-throughput systems:

// Minimize overhead
let config = ManeuverConfig::new()
    .with_timing_metrics(false)  // Disable if not needed
    .with_detailed_observability(false)  // Reduce logging
    .with_error_strategy(ErrorStrategy::FailFast);  // Fast failure

// Use connection pooling for LLM providers
// Pre-validate flows at startup
// Cache parsed flow expressions

Last Updated: February 2026 Version: 0.10.0 Status: Production Ready

WarEngine: Battlefield State & Superstep Execution

Since: v0.10.0 (Phase 22) Crates: paladin-battalion (WarEngine, WarGraph), paladin-core (Battlefield, Waypoint), paladin-storage (the WaypointPort backends)

Every code example targets the current v0.10.0 workspace. The substantive examples are real, compiled code pulled from the paladin-doc-examples crate via mdBook {{#include}}, so they are checked against the live API; a few illustrative fragments are marked rust,ignore. The API forms are verified against crates/paladin-battalion/ and crates/paladin-core/.

The WarEngine is the superstep engine every Paladin Battalion pattern in this book ultimately runs on: it executes a WarGraph of nodes over a typed Battlefield, in bounded supersteps that checkpoint automatically after each one and resume with zero re-execution after a crash. Unlike the legacy Campaign graph, a WarGraph permits cycles — including self-loops — so iterative workflows (retry-and-refine, evaluate-optimize loops) are expressible directly. This page is the substrate the Control Flow, Parley & Chronicle, Aegis and Agent Runtime guides build on — those guides cover routing, pause/resume, fault tolerance and middleware respectively; this page does not re-explain any of them.

Table of Contents

1. Building a Graph: Battlefield State and Superstep Merge Semantics
2. Waypoint Checkpointing and Addressing
3. The Three WaypointPort Backends
4. EngineConfig, EngineLimits and Bounded Iteration
5. WaypointRetentionService
6. The Graph Fingerprint
7. Where to Go Next

Building a Graph: Battlefield State and Superstep Merge Semantics

A Battlefield is the typed shared state a WarGraph's nodes read and write. Its shape is declared once, as a BattlefieldSchema of FieldSpecs — each field names a DispatchRule (LastWrite, Append, MergeObject, Sum, or Custom) that decides how two nodes' concurrent writes to the same field within one superstep are merged. A node's contribution is a StateDelta: a set of field values, never a direct mutation — the engine merges every delta produced in a superstep into the Battlefield in one step, through each field's own dispatch rule, so the merge order is deterministic regardless of how many nodes ran concurrently.

The graph below is deliberately small and cyclic: one node self-loops over a (count, status) Battlefield a few times before falling out of the loop — the same shape that makes cyclic execution useful for retry-and-refine style workflows. build_graph takes the EngineLimits to construct the graph with — pass EngineLimits::default() for the built-in bounds, or the limits configure_limits derives from EngineConfig (see EngineConfig, EngineLimits and Bounded Iteration below).

use paladin_battalion::engine::graph::{EdgeSpec, EngineLimits, NodeSpec, WarGraph};
use paladin_battalion::engine::node::{NodeContext, StateNode, StateNodeError};
use paladin_core::platform::container::battalion::campaign::EdgeCondition;
use paladin_core::platform::container::battlefield::{
    Battlefield, BattlefieldSchema, DispatchRule, FieldName, FieldSpec, StateDelta,
};
use paladin_core::platform::container::directive::{Directive, NextStep};
use paladin_core::platform::container::waypoint::NodeId;

/// A pure `StateNode` that increments a `count` field each visit and
/// self-loops -- via `NextStep::Edges` and a `Contains("looping")` condition
/// on its own outgoing edge -- until `count` reaches `target`, then writes
/// `status = "done"` and falls out of the loop. The smallest shape that
/// demonstrates cyclic superstep execution, not a straight-line DAG.
struct LoopUntil {
    target: u64,
}

#[async_trait]
impl StateNode for LoopUntil {
    async fn run(
        &self,
        state: &Battlefield,
        _ctx: &NodeContext,
    ) -> Result<Directive, StateNodeError> {
        let count_field = FieldName::new("count").map_err(|e| StateNodeError(e.to_string()))?;
        let status_field = FieldName::new("status").map_err(|e| StateNodeError(e.to_string()))?;
        let current: u64 = state
            .get(&count_field)
            .map_err(|e| StateNodeError(e.to_string()))?
            .unwrap_or(0);
        let next = current + 1;

        let mut delta = StateDelta::new();
        delta
            .set(count_field, next)
            .map_err(|e| StateNodeError(e.to_string()))?;
        delta
            .set(
                status_field,
                if next >= self.target {
                    "done"
                } else {
                    "looping"
                },
            )
            .map_err(|e| StateNodeError(e.to_string()))?;

        Ok(Directive {
            delta,
            next: NextStep::Edges,
        })
    }
}

/// Build a small cyclic `WarGraph`: one `Function` node self-loops over a
/// `(count, status)` `Battlefield` a few times, then falls out of the loop
/// once `status` reads `"done"` -- a shape `WarGraph::validate` accepts
/// precisely because cycles, including self-loops, are legal (ENG-FR-02),
/// unlike the legacy Campaign graph's cycle-rejecting validation. Takes the
/// `EngineLimits` to construct the graph with -- see [`configure_limits`]
/// for how a deployment derives them from `EngineConfig`, or pass
/// `EngineLimits::default()` to accept the built-in bounds.
pub fn build_graph(limits: EngineLimits) -> Result<WarGraph, Box<dyn std::error::Error>> {
    let count = FieldName::new("count")?;
    let status = FieldName::new("status")?;
    let schema = BattlefieldSchema::new(vec![
        FieldSpec::new(
            count,
            DispatchRule::LastWrite,
            Some(serde_json::json!(0)),
            false,
        ),
        FieldSpec::new(status, DispatchRule::LastWrite, None, false),
    ]);

    let mut graph = WarGraph::new(schema, limits);
    let looper = NodeId::new("looper");
    graph.add_node(
        looper.clone(),
        NodeSpec::Function(Arc::new(LoopUntil { target: 3 })),
    );
    graph.add_edge(EdgeSpec {
        from: looper.clone(),
        to: looper.clone(),
        condition: Some(EdgeCondition::Contains("looping".to_string())),
    });
    graph.add_entry(looper);

    Ok(graph)
}

Waypoint Checkpointing and Addressing

Exactly one Waypoint is persisted automatically after every superstep — a full snapshot of the Battlefield as of that point, never an incremental diff. A Waypoint is addressed by the pair (ThreadId, WaypointId): ThreadId identifies the run, WaypointId identifies one checkpoint within it, and each Waypoint also carries parent_waypoint_id lineage back to the start of the thread. A Waypoint's vanguard field — a Vec<NodeId>, not a struct of its own — lists the nodes ready to execute in the next superstep; an empty vanguard after a superstep is what RunOutcome:: Completed means.

use paladin_battalion::engine::{RunOutcome, WarEngine};
use paladin_core::platform::container::waypoint::ThreadId;
use paladin_storage::waypoint::in_memory::InMemoryWaypointStore;

/// Build the graph, run it to completion over a fresh
/// `InMemoryWaypointStore`, and return the outcome plus the store and
/// thread so [`inspect_waypoints`] can read back the Waypoint history the
/// run left behind -- a `RunOutcome::Failed` carrying
/// `EngineError::RecursionLimitExceeded` is what a graph that never falls
/// out of its loop would produce once `EngineLimits::max_supersteps` is
/// exhausted.
pub async fn run_engine()
-> Result<(RunOutcome, Arc<InMemoryWaypointStore>, ThreadId), Box<dyn std::error::Error>> {
    let (limits, durability) = configure_limits()?;
    let graph = build_graph(limits)?;
    let store = Arc::new(InMemoryWaypointStore::new());
    let engine = WarEngine::new(mock_paladin_port(), store.clone()).with_durability(durability);
    let thread = ThreadId::new("superstep-engine-guide")?;
    let outcome = engine
        .start(&graph, thread.clone(), StateDelta::new())
        .await?;
    Ok((outcome, store, thread))
}

use paladin_core::platform::container::waypoint::WaypointId;
use paladin_ports::output::waypoint_port::WaypointPort;

/// Given the store and thread [`run_engine`] just used, read back the
/// persisted Waypoint via `WaypointPort::latest`, addressed by `(ThreadId,
/// WaypointId)`, and return its `vanguard` -- the nodes ready for the next
/// superstep the engine checkpointed after the run's final superstep.
pub async fn inspect_waypoints(
    store: &InMemoryWaypointStore,
    thread: &ThreadId,
) -> Result<Option<(ThreadId, WaypointId, Vec<NodeId>)>, Box<dyn std::error::Error>> {
    let waypoint = store.latest(thread).await?;
    Ok(waypoint.map(|wp| (wp.thread_id, wp.waypoint_id, wp.vanguard)))
}

The Three WaypointPort Backends

Every Waypoint write goes through the WaypointPort trait (paladin-ports), and three backends implement it, all passing the same shared contract test suite:

InMemoryWaypointStore needs no Cargo.toml feature flag beyond what crates/doc-examples already declares — paladin-storage's waypoint module is not feature-gated, unlike its sqlite/mysql/postgres submodules.

EngineConfig, EngineLimits and Bounded Iteration

Because a WarGraph permits cycles, every run needs bounds so it always terminates. EngineLimits (paladin-battalion) is what a WarGraph is constructed with; the app-facing EngineConfig (src/config/engine.rs) is what a deployment actually configures, and converts into EngineLimits via impl From<EngineConfig> for EngineLimits:

use paladin::config::engine::EngineConfig;
use paladin_battalion::engine::WaypointDurability;

/// Configure the engine's bounded-iteration limits and Waypoint durability
/// through the app-facing `EngineConfig` -- the same struct the
/// `APP_ENGINE_MAX_SUPERSTEPS`, `APP_ENGINE_MAX_NODE_VISITS`,
/// `APP_ENGINE_RUN_TIMEOUT_SECS`, `APP_ENGINE_WAYPOINT_DURABILITY` and
/// `APP_ENGINE_MAX_MUSTER_TASKS` environment overrides populate at boot --
/// then convert it into the `EngineLimits` a `WarGraph` is constructed with
/// (`waypoint_durability` stays on the source `EngineConfig` value itself;
/// it is not part of `EngineLimits` and is passed to
/// `WarEngine::with_durability` separately).
pub fn configure_limits() -> Result<(EngineLimits, WaypointDurability), Box<dyn std::error::Error>>
{
    let config = EngineConfig {
        max_supersteps: 20,
        max_node_visits: 10,
        run_timeout_secs: Some(60),
        waypoint_durability: WaypointDurability::Strict,
        max_muster_tasks: 50,
        ..EngineConfig::default()
    };
    config.validate()?;

    let durability = config.waypoint_durability;
    let limits: EngineLimits = config.into();
    Ok((limits, durability))
}

A graph that never falls out of its own loop hits EngineError::RecursionLimitExceeded once max_supersteps is exhausted — the same limit EngineLimits::default() sets to 50 and this page's own example graph would hit if its LoopUntil node never wrote status = "done".

WaypointRetentionService

Waypoint history grows without bound unless something prunes it. WaypointRetentionService (src/application/services/waypoint_retention.rs) is the application-layer policy: it defines the single project-wide rule for what may never be deleted — a thread's latest Waypoint, plus every Waypoint whose status is AwaitingInput — and drives the storage-layer prune() free function (crates/paladin-storage/src/waypoint/retention.rs) with that rule and a configured age/count bound. The storage layer itself carries no opinion about what "protected" means; it is handed the answer as a plain function argument.

The Graph Fingerprint

A WarGraph's content fingerprint — GRAPH_FINGERPRINT_VERSION (currently "v6") plus a blake3 hash over node ids, edge specs and schema field names — is compared on resume, so resuming a thread against a structurally different graph fails fast with GraphMismatch rather than silently replaying stale routing. The fingerprint is deliberately not computed over prompts or models (those may be hot-swapped without changing run semantics), and it excludes every EngineLimits field — raising max_supersteps to let a resumed run continue is a legitimate operator action, not a graph change. The version tag has bumped twice since Phase 22: v1 → v2 fixed a delimiter-collision encoding bug, and the current v6 covers the Aegis and structured-output-schema sections the fingerprint's canonical byte stream now includes.

Where to Go Next

This page covers the engine's own state, checkpointing and bounds — not what runs on top of it:

• Routing a node's output to the next node, including dynamic jumps and Muster fan-out — see Control Flow: Dynamic Routing & Subgraphs.
• Pausing and resuming a run, including human-in-the-loop Parleys and graceful shutdown — see Parley & Chronicle.
• Per-node fault tolerance — retry, timeout, error handlers, model fallback and caching — see Aegis: Retry, Timeout, Error Handlers, Model Fallback and Node Caching.
• Middleware, context management, Vault memory and structured output — see Agent Runtime.

Control Flow: Dynamic Routing, Fan-Out & Subgraphs

Node-authored routing, worker fan-out, subgraph composition and LLM-evaluated edges for the WarEngine, the superstep-based engine built on WarGraph, Battlefield typed state and Waypoint checkpointing.

Table of Contents

1. WarGraph in Three Sentences
2. Directives: Node-Authored Routing
3. DirectiveParser: Reading a Paladin's Output
4. Muster: Dynamic Fan-Out
5. Subgraphs: Composing Graphs
6. LLM-Evaluated Edges
7. Migrating from v0.9: M-B-01

WarGraph in Three Sentences

A WarGraph is a collection of nodes and static edges, executed by a WarEngine in supersteps: each superstep runs every currently-ready node concurrently, merges their Battlefield state deltas, and computes the next superstep's ready set. Unlike the legacy CampaignExecutionService's DAG, a WarGraph permits cycles — WarGraph::validate rejects only a node that could never become ready (unreachable from entry and not marked a dynamic target or worker template), not a cycle itself, and every run is bounded by EngineLimits (max_supersteps, max_node_visits). This page assumes that much and no more; for the full engine guide — Battlefield state, superstep merge semantics, Waypoint checkpointing, the WaypointPort backends, EngineConfig/EngineLimits and the graph fingerprint — see WarEngine: Battlefield State & Superstep Execution.

Directives: Node-Authored Routing

A StateNode::run returns a Directive — the StateDelta it contributes, plus a NextStep telling the engine how to route control next:

pub struct Directive {
    pub delta: StateDelta,
    pub next: NextStep,
}

pub enum NextStep {
    Edges,               // the default — evaluate this node's static outgoing edges
    Goto(Vec<NodeId>),    // enter these nodes directly next superstep
    Muster(Vec<MusterTask>), // fan out worker tasks (see below)
    End,                  // complete the run after this superstep's merge
    Parley(ParleyRequest), // suspend the run awaiting external input (see below)
}

NextStep::Edges is the default and the only variant a pre-CF-02 node ever produces (impl From<StateDelta> for Directive) — a graph that never opts in behaves identically to before this feature existed.

NextStep::Goto enters the named node(s) directly in the next superstep, bypassing the normal readiness check. Every target must be a declared node; a target reachable ONLY via Goto (no static incoming edge) must additionally be marked with WarGraph::mark_dynamic_target — an undeclared target fails the run with EngineError::GotoUnknownNode. A Goto target is still subject to EngineLimits::max_node_visits like any other entry, so a refine loop (writer → reviewer → Goto(writer) until satisfied) is legal and bounded, not an unconditional escape from the engine's own iteration guarantee. A node authoring its own routing and the graph's static edges never both fire in the same superstep — every non-Edges variant routes the emitting node's static edges NotFiring.

NextStep::End completes the run after the current superstep's merge. Peers in the same superstep still merge their own deltas normally; End takes precedence over a Goto emitted by another node in the same superstep.

NextStep::Parley suspends the run awaiting external input — fully implemented as of Phase 24 (HITL-01, HITL-02): the emitting node's static edges resolve NotFiring for that superstep, every ParleyRequest raised in the suspending superstep is collected onto one AwaitingInput Waypoint, and WarEngine::resume_with is the only path that advances a suspended thread. See Parley & Chronicle for the shipped pause/resume mechanism, building an approval gate, and raising a parley from a Paladin node.

DirectiveParser: Reading a Paladin's Output

A NodeSpec::Paladin node's raw string output is turned into a Directive by its DirectiveParser:

pub enum DirectiveParser {
    PlainOutput,                              // the default
    StructuredDirective { on_parse_error: OnParseError },
}

DirectiveParser::PlainOutput — the default — writes the raw output to output_field and routes via NextStep::Edges, byte-identical to a pre-CF-02 Paladin node.

DirectiveParser::StructuredDirective parses a JSON envelope out of the output and applies ONLY the envelope's delta — no implicit output_field write. See the DirectiveParser rustdoc (crates/paladin-battalion/src/engine/directive_parser.rs) for the authoritative envelope shape rather than duplicating it here, so the two cannot drift. on_parse_error resolves a failed extraction: OnParseError::FailRun (the default) fails the run with EngineError::DirectiveParseFailed; OnParseError::FallbackPlain degrades to PlainOutput semantics.

use paladin_battalion::engine::directive_parser::{DirectiveParser, OnParseError};

let parser = DirectiveParser::StructuredDirective {
    on_parse_error: OnParseError::FailRun,
};
// output: r#"{"delta": {"verdict": "approved"}, "next": "edges"}"#

Muster: Dynamic Fan-Out

A node returns NextStep::Muster(Vec<MusterTask>) to fan out N worker tasks, each dispatching to a worker template node — one registered via WarGraph::add_worker_template rather than add_node, so it can run only as a Muster task and never as a normal graph entry:

graph.add_worker_template(
    NodeId::new("summarize_chunk"),
    NodeSpec::Function(Arc::new(SummarizeChunk)),
);

Each MusterTask carries an isolated payload (never merged into the Battlefield — visible only to that worker) and a caller-chosen task_key, used to order worker results deterministically on aggregation and to reject a duplicate key within one Muster. A worker node reads its task's payload through NodeContext::muster — never through a Battlefield field — via the {muster.payload} / {muster.task_key} placeholders in an InputMapping template; graph validation rejects any schema field declared with the muster. prefix, so this namespace can never be shadowed. EngineLimits::max_muster_tasks (default 100, overridable via the APP_ENGINE_MAX_MUSTER_TASKS environment variable) bounds a single Muster directive, enforced at directive-receipt time before any task dispatches (EngineError::MusterTaskLimitExceeded) — raising it is a legitimate operator action, so it is excluded from the graph fingerprint. A run resumed mid-Muster picks up the outstanding tasks from the last progress Waypoint, which stores the superstep's unmerged delta snapshot rather than a partially merged Battlefield.

Subgraphs: Composing Graphs

NodeSpec::Battalion embeds a child WarGraph as a single node, running to completion within ONE parent superstep regardless of how many supersteps the child itself takes:

use paladin_battalion::engine::graph::{NodeSpec, StateMap};

let state_map = StateMap::new()
    .with_input(FieldName::new("topic")?, FieldName::new("child_topic")?)
    .with_output(FieldName::new("child_summary")?, FieldName::new("summary")?);

graph.add_node(
    NodeId::new("sub_workflow"),
    NodeSpec::battalion(Arc::new(child_graph), state_map),
);

StateMap is the complete contract for what crosses the parent/child boundary in either direction: inputs are (parent field, child field) pairs seeding the child's initial state from the parent's superstep snapshot; outputs are (child field, parent field) pairs returned as the Battalion node's own delta, merged under the parent's dispatch rules like any other node's delta. A child field not named in outputs never leaves the child — the child's own schema, nodes and edges stay entirely private, never visible in the parent's Battlefield, this node's delta, or the parent thread's Waypoint payload (even Debug output on NodeSpec::Battalion prints only the child's fingerprint and the two map sizes). A child run gets its own namespaced Waypoint thread (ThreadId::child, carrying checkpoint_ns) so its checkpoints never collide with the parent's or a sibling subgraph's history. restart_on_resume (default false) controls whether a resumed parent run restarts this node's child from scratch rather than continuing a partially-completed child thread.

LLM-Evaluated Edges

Register an LlmDecisionEvaluator under EdgeCondition::Custom("<decision name>"), exactly like any other evaluator through EdgeEvaluatorRegistry:

use std::sync::Arc;
use paladin_battalion::llm_decision::{LlmDecisionEvaluator, OnAmbiguous};
use paladin_core::platform::container::waypoint::NodeId;

let evaluator = LlmDecisionEvaluator::new(
    "route_urgency",
    llm.clone(),
    "gpt-4",
    "Is this urgent? Reply escalate or archive.\n\n{output}",
    vec![
        ("escalate".to_string(), NodeId::new("urgent_handler")),
        ("archive".to_string(), NodeId::new("archive_handler")),
    ],
)
.on_ambiguous(OnAmbiguous::Default("archive".to_string()));

engine.with_edge_evaluator("route_urgency", Arc::new(evaluator));

The model is asked once per decision per superstep — every outgoing edge sharing the same source node and rendered prompt consults one memoized answer, so N outgoing edges never become N independent (and possibly inconsistent) calls. A model answer matching no declared choice is resolved by on_ambiguous: OnAmbiguous::Fail (the default) fails the run; OnAmbiguous::Default treats it as a named fallback choice.

Commander's StrategySelection::Semantic applies the same idea to Battalion pattern selection: prompt a model with the strategy catalog and the run's input, parse the answer as a strategy name, and fall back to StrategySelection::Heuristic — deterministically, with the fallback and its cause recorded in BattalionResult::strategy_selection_reasoning — on any LLM error or an answer naming no catalog strategy.

use paladin_battalion::commander::{CommanderBuilder, StrategySelection};

let commander = CommanderBuilder::new(paladin_port)
    .strategy(BattalionStrategy::Auto)
    .paladins(paladins)
    .strategy_selection(StrategySelection::Semantic {
        llm: llm.clone(),
        model: "gpt-4".to_string(),
    })
    .build()?;

Both LlmDecision and StrategySelection::Semantic are off by default and reached only in code — no APP_* environment variable, cargo feature, or config-struct field can turn either on; a v0.9 configuration boots identically.

Egress boundary: an LlmDecisionEvaluator's prompt_template renders against live Battlefield state (or a legacy Paladin's raw output) and the rendered result is sent, verbatim, to a third-party model. Whatever the template's placeholders resolve to is exactly what leaves this process — if a workflow's schema carries secret-like data and the template references that field, it is sent to the model. This is the workflow author's control point, not something the evaluator can filter; neither its error paths nor its memoized state ever interpolate the rendered prompt or the model's raw response body.

Migrating from v0.9: M-B-01

Before this phase, an unregistered EdgeCondition::Custom(name) silently evaluated to true on every run — a bug, not a feature (BUG-01). It is now a fail-closed validation error: an unregistered Custom name fails graph/campaign validation before any node executes, naming every offender. If you have a v0.9 workflow using EdgeCondition::Custom, see MIGRATION.md §9.1, entry M-B-01, for the worked before/after example and the exact validation error text.

Parley & Chronicle: Pause, Resume, History and Graceful Shutdown

Human-in-the-loop approval gates, typed resume, an inspectable and forkable execution history, and cooperative shutdown for the WarEngine — built as thin, well-specified layers over the Waypoint substrate the Control Flow guide introduces.

Table of Contents

1. Building an Approval Gate
2. Raising a Parley from a Paladin Node
3. Resuming a Suspended Thread
4. Partial Answers
5. Expiry: on_expire Policies
6. Chronicle: History, Replay and Fork
7. Graceful Shutdown from the Embedder's Side
8. The HTTP Surface
9. Notifying a Human a Parley Is Waiting

Building an Approval Gate

A Parley is a suspension point: a node stops the run, a Waypoint with status: AwaitingInput is persisted, every task/timer/connection for the run is released, and the run returns RunOutcome::AwaitingInput. The primary building block is NodeSpec::Gate — a node with no run body of its own that always parleys on its first visit and writes the delivered value on the post-resume visit.

An approval gate is exactly one Gate node plus two conditional edges — no custom node code:

use std::time::Duration;

use paladin_battalion::engine::graph::{GateRequestTemplate, NodeSpec, WarGraph};
use paladin_battalion::engine::{EdgeCondition, InputMapping};
use paladin_core::platform::container::battlefield::FieldName;
use paladin_core::platform::container::parley::{OnExpire, ParleyKind};

let approved = FieldName::new("approved").unwrap();

let request = GateRequestTemplate::new(
    ParleyKind::Approval,
    InputMapping::new("Deploy build {build_id} to production?"),
)
.with_payload_template(InputMapping::new(r#"{"build_id": "{build_id}"}"#))
.with_expires_in(Duration::from_secs(24 * 60 * 60))
.with_on_expire(OnExpire::FailRun);

graph.add_node("approve", NodeSpec::gate(request, Some(approved.clone())));
graph.add_edge("approve", "deploy", Some(EdgeCondition::Contains(r#""approved":true"#.into())));
graph.add_edge("approve", "cancel", Some(EdgeCondition::Contains(r#""approved":false"#.into())));

A few things to note:

• output_field (approved here) is required for Approval/Choice/FreeText gates and must be None for a StateEdit gate — WarGraph::validate rejects every other combination, and checks the field exists in the schema with a type compatible with the gate's kind (Approval → Bool or String; Choice/FreeText → String).
• prompt_template/payload_template render from the Battlefield through the same InputMapping templating every other node uses.
• Approval values are normalised before delivery: JSON true/false or the case-insensitive strings "yes"/"no"/"approve"/"deny" all resolve to a JSON boolean (or "true"/"false" if output_field is a String field).
• A Gate's output_field is what edge evaluation reads for its source — exactly like a Paladin node's own output_field — so Contains/Regex/a registered Custom evaluator all work unchanged. Anchor a Contains needle to the full "field":value pair (as above), not a bare true/false — Contains/Regex match against the whole serialized Battlefield JSON, and every non-required schema field's own "required":false entry contains the bare word false.

Raising a Parley from a Paladin Node

A Paladin node can raise a parley without a declarative Gate, through the structured directive envelope's next.parley key:

{
  "delta": {},
  "next": {
    "parley": {
      "kind": "Approval",
      "prompt": "Deploy build #482 to production?",
      "payload": { "build": 482 },
      "expires_in_secs": 86400
    }
  }
}

The parser stamps parley_id, node_id and created_at, and computes expires_at from expires_in_secs — an author never supplies these. kind/prompt are required; payload, choices, expires_in_secs and on_expire are optional (on_expire defaults to FailRun).

On the post-resume re-run, InputMapping::render resolves the answer through a parley. namespace — resolved only from NodeContext, never the Battlefield, exactly like the muster. namespace the Control Flow guide documents:

// {parley.value}       -- the submitted/defaulted value
// {parley.prompt}      -- the originating request's own prompt
// {parley.kind}        -- "Approval" | "Choice" | "FreeText" | "StateEdit"
// {parley.responded_by} -- the responder's identity, or empty for a defaulted response

WarGraph::validate rejects any schema field named with the parley. prefix, so a graph's own state can never shadow this namespace. NodeContext::parley_response() gives the same data to ordinary Rust code:

if let Some(response) = ctx.parley_response() {
    // response.value, response.kind, response.responded_by, response.defaulted
}

Resuming a Suspended Thread

WarEngine::resume_with is the only path that advances a suspended thread:

use paladin_core::platform::container::parley::ParleyResponse;

let outcome = engine.resume_with(&graph, thread_id, responses).await?;

Every submitted ParleyResponse is validated totally before anything is persisted — an error on any response leaves the thread suspended with no Waypoint written:

A plain WarEngine::resume (no responses) against an AwaitingInput thread never guesses — it fails closed with EngineError::ThreadAwaitingInput, naming the still-outstanding parleys.

Partial Answers

When several nodes parley in the same superstep, all of their requests are recorded on one AwaitingInput Waypoint, and all must be answered before the run continues. Submitting a correct answer for only some of them persists a new AwaitingInput Waypoint at the same superstep, with responses extended and RunOutcome::AwaitingInput naming only the still-remaining requests:

// Two parleys outstanding; answer only one.
let outcome = engine.resume_with(&graph, thread_id.clone(), vec![first_response]).await?;
// outcome is RunOutcome::AwaitingInput { parleys: [second_request], .. } -- still suspended.

let outcome = engine.resume_with(&graph, thread_id, vec![second_response]).await?;
// outcome is RunOutcome::Completed { .. } (or whatever the graph reaches next).

This partially-answered state is a property of the persisted Waypoint, not process memory: it survives a full process restart and is queryable from a cold store handle. Responses are durably consumed only when the first post-resume Waypoint actually persists — if the process dies between validation and that write, the AwaitingInput Waypoint just read is still latest, and re-submitting the identical responses is safe.

Expiry: on_expire Policies

A ParleyRequest optionally carries expires_at, evaluated lazily at resume time — there is no background timer. Each request's own on_expire policy decides what happens once the clock has passed it, independent of whether the caller's own submission even names the expired request:

• OnExpire::FailRun (the default) — resume_with persists a Failed Waypoint (its reason naming the expired parley and node) and returns Err(EngineError::ParleyExpired). The thread is thereafter resumable only via replay/fork from an earlier Waypoint — a plain resume against it fails closed with EngineError::ThreadAlreadyFailed.
• OnExpire::ResumeWithDefault(value) — value (validated against the request's own kind at graph-validate time for a Gate, or at raise time for a directive-raised parley) is substituted as the response, with responded_by: None and defaulted: true so an audit trail can see the substitution happened. The run then proceeds exactly as if that value had been submitted.

Chronicle: History, Replay and Fork

ChronicleService is a thin, port-only read facade (no paladin-battalion dependency) over one thread's Waypoint history:

use std::sync::Arc;
use paladin::application::services::chronicle::ChronicleService;

let chronicle = ChronicleService::new(waypoint_port);

// Newest-first summaries, including every branch's lineage.
let page = chronicle.history(&thread_id, 20, None).await?;

// The full snapshot (Battlefield, vanguard, records, status) for one Waypoint.
let waypoint = chronicle.inspect(&thread_id, waypoint_id).await?;

// The newest summary on the branch rooted at `branch_root`, or `None`.
let latest = chronicle.latest_on_branch(&thread_id, branch_root).await?;

WarEngine::replay/WarEngine::fork re-enter the superstep loop from any past Waypoint, each producing a new branch while the original chain stays untouched:

// Re-run forward from an earlier Waypoint, unchanged.
let outcome = engine.replay(&graph, &thread_id, from_waypoint_id).await?;

// Re-run forward, but first merge an edit into the starting Battlefield --
// the "what-if" primitive. An edit naming an undeclared field fails closed
// and persists nothing, exactly like a real node's delta would.
let outcome = engine.fork(&graph, &thread_id, from_waypoint_id, edit).await?;

Immutability is a hard, byte-for-byte invariant: every mainline Waypoint serialises to identical bytes before and after a replay/fork, and calling either twice from the same Waypoint produces two independent branches, disturbing neither each other nor the mainline. A branch is a queryable attribute — Waypoint.fork_of/WaypointSummary.fork_of mark the branch root and every subsequent Waypoint on that branch inherits the same value — so the whole branch tree reconstructs from WaypointSummary alone, with no full-Waypoint loads.

Subgraph forks never share child Waypoints. A branch runs its NodeSpec::Battalion children under a thread id derived from both the parent thread and the branch root (ThreadId::child_on_branch), so a fork's subgraph child always starts fresh and WaypointPort::latest on that child thread never resolves the mainline child's own history.

Graceful Shutdown from the Embedder's Side

WarEngine::with_shutdown_grace(Duration) (default 30 s) configures how long a mid-superstep cancellation waits for the in-flight batch of node tasks before giving up on the stragglers:

use std::time::Duration;
use paladin_battalion::engine::WarEngine;
use paladin_battalion::engine::shutdown::ShutdownCoordinator;

let coordinator = ShutdownCoordinator::new();
let (token, _guard) = coordinator.register();

let engine = WarEngine::new(waypoint_port)
    .with_cancellation_token(token)
    .with_shutdown_grace(Duration::from_secs(30));

When the token fires while nodes are in flight, the engine keeps awaiting the WHOLE batch (never just the one that triggered cancellation) until the grace deadline; nodes still running at the deadline are aborted and recorded NodeOutcomeKind::Skipped { reason: "shutdown" }, their deltas discarded. Those nodes' ids are re-listed in the Halted Waypoint's vanguard alongside the normally computed next vanguard, so resume re-executes exactly them — exactly once — while every node that finished inside the grace window merges normally. shutdown_grace = Duration::ZERO aborts immediately.

An embedder that wants every in-flight run to drain on process shutdown constructs one ShutdownCoordinator, registers every WarEngine run with it (register() returns a child token plus an RAII RunGuard), and calls coordinator.cancel_and_wait(grace) from its own shutdown path:

// On SIGTERM/SIGINT:
let outcome = coordinator.cancel_and_wait(Duration::from_secs(30)).await;
if outcome.drained() {
    // every registered run finished inside the grace window
} else {
    // the deadline elapsed first; any straggler is Skipped and re-listed for resume
}

paladin-server and ServiceRunner both wire this into SIGTERM/SIGINT already — see Kubernetes deployment for the operator-facing terminationGracePeriodSeconds/env-var contract. shutdown_grace_secs/graceful_shutdown are runtime settings only: they are never hashed into the graph fingerprint and never affect resume's GraphMismatch check.

The HTTP Surface

paladin-web exposes three thread routes behind the same authentication middleware /v1/agents/* already uses; the one mutating route, POST /v1/threads/{id}/resume, additionally requires an admin-role credential:

POST .../resume never holds the connection open: it validates synchronously (typed 400/404/409 errors on rejection, nothing persisted) and, only for a valid and complete submission, either re-enqueues the same run_id onto the durable run server (Phase 27's Platform API, when a run row exists for the thread) or, for a pre-run-server thread with no run row, spawns the actual engine continuation as a background task registered with the process's ShutdownCoordinator (run_id: null on the response in that fallback case) — either way, 202 returns immediately. A client polls GET .../state, or GET /v1/runs/{run_id} when run_id is present, for the outcome. Two distinct 409 conflict codes share the same HTTP status: thread_not_awaiting_input (the thread is not suspended) and graph_not_registered (the thread's graph fingerprint has no WarGraph registered in this process). Every route answers 501 not_implemented, naming the config key to set, when no waypoint backend (APP_WAYPOINT_STORE_ BACKEND=sqlite|postgres) is wired.

The full run lifecycle — submission, cancellation, streaming, assistants, schedules and webhooks — is documented separately. This page covers the thread-level pause/resume/history surface; see Platform API — Runs, Threads, Assistants, Schedules, Webhooks for POST /runs, GET /runs/{run_id}/stream, and everything built on top of the resume mechanism this page describes.

Never template a secret or credential into a Gate's payload. GET /v1/threads/{id}/state returns that payload verbatim to any authenticated caller. A payload_template is author-controlled, rendered from the Battlefield exactly like a prompt — treat it with the same care you would a log line or an error message, never a place to carry an API key or a database credential.

Interim authorization posture. The two read routes (GET .../state, GET .../history) accept any authenticated caller, while answering an approval gate through POST .../resume requires an admin-role credential and answers 403 otherwise. This narrows the phase's own D-24 decision for the mutating route only, applied because neither ThreadId nor Waypoint carries an owner or principal to scope against yet (CR-01, commit 00b1e552, rationale recorded in thread_controller.rs's module rustdoc). This is a documented, accepted interim posture, not an oversight: every credential configured for the resume route must be treated as admin-equivalent until Phase 27's PLAT-06 supplies real per-thread ownership scoping.

Notifying a Human a Parley Is Waiting

Notifying someone that a parley is waiting is application code composed from the existing paladin-notifications port — this phase adds no new port for it. A typical composition, called right after a RunOutcome::AwaitingInput comes back from start/resume_with:

use std::sync::Arc;

use paladin_core::platform::container::notification::{
    Notification, NotificationChannel, NotificationContent, NotificationPriority,
    NotificationRecipient,
};
use paladin_ports::output::notification_port::NotificationDeliveryPort;

async fn notify_parley_waiting(
    notifications: Arc<dyn NotificationDeliveryPort>,
    approver_email: &str,
    prompt: &str,
) -> Result<(), Box<dyn std::error::Error>> {
    let notification = Notification::new(
        NotificationRecipient::Email(approver_email.to_string()),
        NotificationContent::new(
            "Approval needed".to_string(),
            prompt.to_string(),
            "parley".to_string(),
        ),
        NotificationChannel::Email,
        NotificationPriority::High,
    )?;

    notifications.deliver_notification(notification).await?;
    Ok(())
}

Call this from the same application code that calls WarEngine::start/resume_with — a RunOutcome::AwaitingInput { parleys, .. } names exactly which requests are now waiting, so the prompt and recipient can be derived from them directly. No engine or port change is required to add a second channel (Slack, SMS, a webhook): construct a different NotificationDeliveryPort adapter and call it the same way.

Aegis: Retry, Timeout, Error Handlers, Model Fallback and Node Caching

Per-node fault tolerance for the WarEngine — a typed error taxonomy, exact-backoff retry, wall-clock and progress-aware timeouts, saga-style compensation handlers, an ordered model fallback chain, and result caching — built as one opt-in policy attached beside a node, over the superstep engine the Control Flow and Parley & Chronicle guides introduce.

Every Rust sample on this page is compiled code pulled from the paladin-doc-examples crate via mdBook {{#include}}, so a sample cannot drift from the landed API.

Table of Contents

1. What an Aegis Is
2. The Error Taxonomy: Why transience() Exists
3. Retry
4. Timeouts: Wall Clock versus Idle
5. Error Handlers and Compensation
6. Model Fallback
7. Node Caching
8. The Node-Kind Support Matrix
9. Limitations You Should Know About
10. Security Notes

What an Aegis Is

An Aegis is the whole fault-tolerance policy for one node:

pub struct Aegis {
    pub retry: Option<RetryPolicy>,
    pub timeout: Option<TimeoutPolicy>,
    pub on_error: Option<ErrorHandlerSpec>,
    pub cache: Option<CachePolicy>,
}

Every field is independently optional, so a node can carry only a timeout, only a cache, or all four. An Aegis is not a field on NodeSpec — it attaches as a sidecar on the WarGraph, keyed by NodeId, through two builder methods:

• WarGraph::set_aegis(node_id, aegis) — this node's own policy.
• WarGraph::with_default_aegis(aegis) — the graph-wide fallback for every node with no entry of its own.

use paladin_battalion::engine::{EngineLimits, InputMapping, NodeSpec, WarGraph};
use paladin_core::platform::container::aegis::{Aegis, RetryPolicy, TimeoutPolicy};
use paladin_core::platform::container::battlefield::{
    BattlefieldSchema, DispatchRule, FieldName, FieldSpec,
};
use paladin_core::platform::container::waypoint::NodeId;

/// Attach an `Aegis` to one node, with a graph-wide default for the rest.
pub fn attach_an_aegis() -> Result<WarGraph, Box<dyn std::error::Error>> {
    let draft = FieldName::new("draft")?;
    let review = FieldName::new("review")?;
    let schema = BattlefieldSchema::new(vec![
        FieldSpec::new(draft.clone(), DispatchRule::LastWrite, None, false),
        FieldSpec::new(review.clone(), DispatchRule::LastWrite, None, false),
    ]);

    let mut graph = WarGraph::new(schema, EngineLimits::default());
    graph.add_node(
        NodeId::new("writer"),
        NodeSpec::paladin(
            create_paladin("Writer"),
            InputMapping::new("Draft a reply to: {draft}"),
            draft,
        ),
    );
    graph.add_node(
        NodeId::new("reviewer"),
        NodeSpec::paladin(
            create_paladin("Reviewer"),
            InputMapping::new("Review this draft: {draft}"),
            review,
        ),
    );

    // Every node with no entry of its own gets three attempts and a
    // 30-second per-attempt wall clock ...
    graph.with_default_aegis(Aegis {
        retry: Some(RetryPolicy::default()),
        timeout: Some(TimeoutPolicy {
            run_timeout: Some(Duration::from_secs(30)),
            idle_timeout: None,
        }),
        ..Default::default()
    });

    // ... while `writer`'s own entry wins WHOLESALE: it is retried up to
    // five times and, because this Aegis sets no `timeout`, it carries no
    // per-attempt bound at all -- the default's 30 s is NOT merged in.
    graph.set_aegis(
        NodeId::new("writer"),
        Aegis {
            retry: Some(RetryPolicy {
                max_attempts: 5,
                ..RetryPolicy::default()
            }),
            ..Default::default()
        },
    );

    Ok(graph)
}

A node's own entry wins wholesale. aegis_for(node) returns the node's own set_aegis entry if there is one, else the default, else nothing — it never merges the two field by field. In the sample above, writer sets retry but not timeout, so writer has no per-attempt timeout even though the default declares one. If you want the default's timeout plus a different retry policy, spell the timeout out on the node's own Aegis.

A node with no Aegis at all — the state every v0.9 graph is in — behaves byte-identically to before this feature existed: one attempt, no bound, no handler, no cache. Nothing here is on by default.

Validation is fail-closed and happens in WarGraph::validate, before any node runs: an Aegis on an undeclared node (EngineError::AegisOnUndeclaredNode), a RetryPolicy with max_attempts: 0 (RetryPolicyInvalid — never "unlimited", never a silent single attempt), a TimeoutPolicy with a zero duration (TimeoutPolicyInvalid), a Custom name nobody registered (UnregisteredRetryPredicate / UnregisteredErrorHandler), or a policy on a node kind that does not support it (AegisUnsupportedForNodeKind, see the matrix) each list every offender in one error.

The Error Taxonomy: Why transience() Exists

A configuration error and a 503 Service Unavailable must not be retried identically: resending a request the provider rejected as malformed burns every attempt for nothing, while giving up on an overloaded upstream after one try throws away a recovery that would have cost half a second. The engine therefore never decides "retry or not" from an error's message. Every error is classified into one of three values by reading its typed fields:

PaladinError::transience() and LlmError::transience() are table-driven, one arm per variant. Provider adapters no longer collapse an HTTP status into a string: every non-2xx response without a dedicated variant becomes LlmError::ProviderError { provider, status, message }, and status is what the table reads.

use paladin_core::platform::container::paladin_error::PaladinError;
use paladin_core::platform::container::transience::Transience;
use paladin_ports::output::llm_port::LlmError;

/// Classification is read from typed fields, never parsed out of a message.
pub fn classify() {
    // A 503 from a provider is worth retrying ...
    let overloaded = LlmError::ProviderError {
        provider: "openai".to_string(),
        status: 503,
        message: "upstream overloaded".to_string(),
    };
    assert_eq!(overloaded.transience(), Transience::Transient);

    // ... a rejected credential is not, however many times you resend it.
    let rejected = LlmError::AuthenticationError("invalid api key".to_string());
    assert_eq!(rejected.transience(), Transience::Permanent);

    // A configuration error is permanent by construction.
    let misconfigured = PaladinError::ConfigurationError("no model".to_string());
    assert_eq!(misconfigured.transience(), Transience::Permanent);

    // A bare, untyped message cannot be classified confidently.
    let opaque = PaladinError::ExecutionError("something happened".to_string());
    assert_eq!(opaque.transience(), Transience::Unknown);
}

When a node fails under an Aegis, the failure travels as one structured value rather than a string:

pub struct NodeError {
    pub node_id: NodeId,
    pub attempt: u32,          // 1-indexed; the attempt that produced it
    pub transience: Transience,
    pub source: NodeErrorSource, // Paladin { kind, message } | Llm { status, provider, .. }
                                 // | Function { message } | Timeout(TimeoutKind) | Cancelled
}

The same NodeError appears on the failed Waypoint (WaypointStatus::Failed.node_error), on RunOutcome::node_error(), inside EngineError::NodeFailed, and as BattalionError::Node at the crate boundary — so a caller, an operator reading Chronicle output and a compensation handler all see the identical value. The human-readable display line on the Waypoint is unchanged from v0.9.

Retry

A RetryPolicy retries a failed attempt inside the same superstep: no Waypoint is written between attempts, sibling nodes in the superstep are unaffected, and every attempt reads the same immutable Battlefield snapshot — a failed attempt's delta never reaches the merged state.

use paladin_core::platform::container::aegis::RetryPredicate;

/// A retry policy spelled out field by field (these are the defaults).
pub fn retry_policy() -> RetryPolicy {
    RetryPolicy {
        max_attempts: 3,                              // attempts, including the first
        initial_interval: Duration::from_millis(500), // the wait before attempt 2
        backoff_factor: 2.0,                          // 500 ms, 1 s, 2 s, 4 s, ...
        max_interval: Duration::from_secs(60),        // every computed wait is capped here
        jitter: true,                                 // + uniform [0, delay) on each wait
        retry_on: RetryPredicate::TransientOnly,      // Permanent and Unknown get one attempt
    }
}

The backoff formula. The delay before attempt n (for n >= 2) is

min(initial_interval × backoff_factor ^ (n − 2), max_interval)  +  jitter

where jitter, when enabled, adds a uniformly random duration in [0, delay). With the defaults and jitter off, the waits before attempts 2, 3, 4 and 5 are exactly 500 ms, 1 s, 2 s and 4 s. max_attempts counts attempts including the first: 3 means the node runs at most three times.

The predicate. retry_on gates every retry on the error's classification:

A Custom predicate sees the structured NodeError and the attempt about to run:

use async_trait::async_trait;
use paladin_battalion::engine::WarEngine;
use paladin_battalion::retry_predicate::{RetryPredicateError, RetryPredicateEvaluator};
use paladin_core::platform::container::node_error::{NodeError, NodeErrorSource};
use paladin_storage::waypoint::in_memory::InMemoryWaypointStore;

/// Retry only rate limits, and only twice.
struct RateLimitOnly;

#[async_trait]
impl RetryPredicateEvaluator for RateLimitOnly {
    async fn allows(&self, err: &NodeError, attempt: u32) -> Result<bool, RetryPredicateError> {
        let rate_limited = matches!(
            err.source,
            NodeErrorSource::Llm {
                status: Some(429),
                ..
            }
        );
        Ok(rate_limited && attempt <= 3)
    }
}

/// Register the predicate on the engine; a policy names it by string.
pub fn engine_with_custom_predicate() -> WarEngine<InMemoryWaypointStore> {
    WarEngine::new(mock_paladin_port(), Arc::new(InMemoryWaypointStore::new()))
        .with_retry_predicate("rate-limit-only", Arc::new(RateLimitOnly))
}

/// The policy that resolves to the registered predicate above.
pub fn policy_naming_the_predicate() -> RetryPolicy {
    RetryPolicy {
        retry_on: RetryPredicate::Custom("rate-limit-only".to_string()),
        ..RetryPolicy::default()
    }
}

What the record shows. NodeExecutionRecord.attempt is the attempt that finally succeeded (or the one that exhausted the budget), and NodeExecutionRecord.attempts lists every failed attempt in order as an AttemptRecord { attempt, started_at, duration_ms, error }. The trace sink receives one NodeStarted/NodeFinished pair per attempt, each carrying attempt, so an observer can tell three attempts from one.

Interceptors run once per attempt. A NodeInterceptor's before hook runs before every attempt (and its after hook only after the attempt that succeeded), because the retry loop wraps outside the interceptor chain. An interceptor's own Skip/Fail decision is never retried as if it were the node's fault.

Inside a Muster. Retry is per task: a failing mustered task retries in its own future with its own attempt counter while its siblings finish and are recorded normally. A progress Waypoint inside a Muster superstep lists only completed tasks.

A Parley is not a failure. A node that returns NextStep::Parley leaves the retry loop as a success; the post-resume re-run starts again at attempt 1 with a fresh budget.

Shutdown during backoff. The backoff wait races the engine's cancellation token, so a run cancelled while a node sleeps between attempts aborts at once (never burning the shutdown grace window). The node is recorded Skipped { reason: "shutdown" } and re-listed in the Halted Waypoint's vanguard, and resume re-executes it from attempt 1.

Tuning is not a graph change. retry and timeout are deliberately excluded from WarGraph::fingerprint(), so tightening either never makes resume fail with GraphMismatch. on_error and cache, which change what a run does, are hashed (fingerprint version v6 — Phase 26 bumped it again when output_schema on NodeSpec::Paladin joined the canonical byte stream; see The Graph Fingerprint for the full scheme).

Timeouts: Wall Clock versus Idle

A TimeoutPolicy bounds each attempt with up to two independent limits, both named by a typed TimeoutKind on the resulting NodeError — never inferred from message text:

The difference matters for streaming: a model that emits a chunk every 100 ms for two minutes is healthy, while one that goes silent for 30 s is stalled. A wall clock alone cannot tell them apart.

/// A per-attempt wall clock and a progress-aware idle bound, together.
pub fn timeout_policy() -> Aegis {
    Aegis {
        timeout: Some(TimeoutPolicy {
            // No single attempt may run longer than this, progress or not.
            run_timeout: Some(Duration::from_secs(120)),
            // ... but an attempt that reports no progress for 10 s is
            // stalled and is cut long before the wall clock would cut it.
            idle_timeout: Some(Duration::from_secs(10)),
        }),
        ..Default::default()
    }
}

A timed-out attempt is Transient, so under a RetryPolicy it is retried like any other transient failure, and its partial work is discarded exactly as any failed attempt's is — the attempt future is dropped on expiry, so a half-finished delta can never become a Directive.

What counts as progress. For a Paladin node, PaladinExecutionService beats the attempt's HeartbeatHandle after every LLM completion, every streamed chunk and every Armament invocation, through the new PaladinPort::execute_observed method. A Function node beats explicitly via ctx.heartbeat(). A Battalion node beats once per child superstep.

use paladin_battalion::engine::{NodeContext, StateNode, StateNodeError};
use paladin_core::platform::container::battlefield::{Battlefield, StateDelta};
use paladin_core::platform::container::directive::Directive;

/// A long-running Function node that reports progress as it goes.
struct BatchScorer;

#[async_trait]
impl StateNode for BatchScorer {
    async fn run(
        &self,
        state: &Battlefield,
        ctx: &NodeContext,
    ) -> Result<Directive, StateNodeError> {
        let mut delta = StateDelta::new();
        for chunk in 0..10 {
            // ... score one chunk ...
            ctx.heartbeat(); // each beat restarts the idle window
        }
        delta
            .set(
                FieldName::new("scored").map_err(|e| StateNodeError(e.to_string()))?,
                true,
            )
            .map_err(|e| StateNodeError(e.to_string()))?;
        Ok(delta.into())
    }
}

ctx.heartbeat() on a node with no idle_timeout is a free no-op, so a node can beat unconditionally.

The run-level budget. EngineLimits.run_timeout (from EngineConfig.run_timeout_secs / APP_ENGINE_RUN_TIMEOUT_SECS) bounds the whole run and nests outside every per-attempt bound: the effective per-attempt deadline is min(run_timeout, remaining engine budget), and the fired kind names whichever was tightest. An attempt the engine budget cuts records Timeout(EngineRun); that kind is never retried (the budget is gone), and the run ends with EngineError::RunTimeoutExceeded { elapsed, limit } through the same Failed-Waypoint path RecursionLimitExceeded and NodeVisitLimitExceeded take. The budget is measured per start/resume call — a resume restarts it — and a Battalion child measures its own budget against its own limits.

Error Handlers and Compensation

on_error runs only on a node's final failure — after retries are exhausted, or immediately when the predicate refuses the error — and never while retries remain. Three handler shapes exist:

use paladin_core::platform::container::aegis::ErrorHandlerSpec;

/// The three handler shapes an `Aegis.on_error` can carry.
pub fn handler_specs() -> Result<[ErrorHandlerSpec; 3], Box<dyn std::error::Error>> {
    // Route: write the structured NodeError into `booking_error` and run
    // `cancel` next, in place of the failed node's static successors.
    let route = ErrorHandlerSpec::Route {
        to: NodeId::new("cancel"),
        error_field: FieldName::new("booking_error")?,
    };

    // Absorb: merge this delta instead and carry on down the static edges.
    let mut fallback_delta = StateDelta::new();
    fallback_delta.set(FieldName::new("summary")?, "unavailable")?;
    let absorb = ErrorHandlerSpec::Absorb { fallback_delta };

    // Custom: hand the error to a handler registered by this name.
    let custom = ErrorHandlerSpec::Custom("escalate".to_string());

    Ok([route, absorb, custom])
}

The node's record reads Failed whichever handler ran — the node did fail; the handler decided what happens next.

A worked compensation chain. book calls a provider that rejects its credential (a Permanent failure). Under the default predicate the retry policy is consulted once and declines, so book executes exactly once; Route writes the structured error into booking_error and cancel runs in the next superstep, reading it:

/// PRD 04's compensation chain: `book` fails permanently, `cancel` runs.
pub fn compensation_chain(
    cancel: Arc<dyn StateNode>,
) -> Result<WarGraph, Box<dyn std::error::Error>> {
    let booking = FieldName::new("booking")?;
    let booking_error = FieldName::new("booking_error")?;
    let schema = BattlefieldSchema::new(vec![
        FieldSpec::new(booking.clone(), DispatchRule::LastWrite, None, false),
        // The error field must be declared, with any dispatch but `Sum`.
        FieldSpec::new(booking_error.clone(), DispatchRule::LastWrite, None, false),
    ]);

    let mut graph = WarGraph::new(schema, EngineLimits::default());
    graph.add_node(
        NodeId::new("book"),
        NodeSpec::paladin(
            create_paladin("Booker"),
            InputMapping::new("Book the flight"),
            booking,
        ),
    );
    // `cancel` is reachable ONLY through the handler: no static edge, no
    // `mark_dynamic_target` -- a Route target is eligible by declaration.
    graph.add_node(NodeId::new("cancel"), NodeSpec::Function(cancel));

    graph.set_aegis(
        NodeId::new("book"),
        Aegis {
            // A permanent failure (a rejected credential, say) takes ONE
            // attempt under the default predicate -- the handler runs at
            // once, not after three wasted retries.
            retry: Some(RetryPolicy::default()),
            on_error: Some(ErrorHandlerSpec::Route {
                to: NodeId::new("cancel"),
                error_field: booking_error,
            }),
            ..Default::default()
        },
    );
    Ok(graph)
}

Validation checks the wiring before the run: the Route target must be a declared node and not a worker template (RouteTargetUnknown / RouteTargetIsWorkerTemplate), error_field must be a declared field whose dispatch is not Sum (RouteErrorFieldUndeclared / RouteErrorFieldDispatchInvalid — a serialised error object cannot be summed), and an Absorb delta may write only declared fields (AbsorbDeltaSchemaInvalid). A Route target reachable only by routing needs no mark_dynamic_target — it is eligible by declaration.

Loops are bounded. A routed visit counts against max_node_visits through the one existing counter, so an a → b → a compensation cycle ends with NodeVisitLimitExceeded rather than spinning.

Custom handlers are registered on the engine, like edge evaluators, and may return any NextStep — including Parley, which suspends the run through the same human-in-the-loop path a node's own parley takes (one AwaitingInput Waypoint; the post-resume re-run is a fresh attempt 1 with ctx.parley_response() set, and no retry budget is consumed):

use paladin_battalion::error_handler::ErrorHandler;
use paladin_core::platform::container::directive::NextStep;

/// A handler that ends the run cleanly instead of failing it.
struct EndTheRun;

#[async_trait]
impl ErrorHandler for EndTheRun {
    async fn handle(&self, err: &NodeError, state: &Battlefield) -> Result<Directive, NodeError> {
        // `err` is the structured NodeError; `state` is the Battlefield as
        // it was BEFORE this superstep (the failed attempt wrote nothing).
        Ok(Directive {
            delta: StateDelta::new(),
            next: NextStep::End,
        })
    }
}

/// Register the handler; `ErrorHandlerSpec::Custom("end-the-run")` names it.
pub fn engine_with_custom_handler() -> WarEngine<InMemoryWaypointStore> {
    WarEngine::new(mock_paladin_port(), Arc::new(InMemoryWaypointStore::new()))
        .with_error_handler("end-the-run", Arc::new(EndTheRun))
}

On a worker template (a node mustered as a task) the rule is: Absorb, or a Custom handler that returns NextStep::Edges — a delta that becomes that task's aggregation contribution. A Route on a worker template is rejected at validation (HandlerNotAllowedOnWorkerTemplate), and a Custom handler returning Goto/End/Parley/Muster from inside a mustered task fails the run with MusterHandlerMustBeDeltaOnly { node, task_key, returned } before any routing side effect. Handle it at the aggregator instead.

Model Fallback

FallbackLlmAdapter composes an ordered chain of LlmPorts into one LlmPort, so it drops in wherever a single provider adapter would — no new trait, no policy type:

use paladin_llm::fallback::FallbackLlmAdapter;
use paladin_llm::mock::MockLlmAdapter;
use paladin_ports::output::llm_port::LlmPort;

/// An ordered chain: try `primary`, then `backup`, then `local`.
pub fn fallback_chain() -> Result<Arc<dyn LlmPort>, Box<dyn std::error::Error>> {
    let primary: Arc<dyn LlmPort> = Arc::new(MockLlmAdapter::new().with_provider_name("openai"));
    let backup: Arc<dyn LlmPort> = Arc::new(MockLlmAdapter::new().with_provider_name("anthropic"));
    let local: Arc<dyn LlmPort> = Arc::new(MockLlmAdapter::new().with_provider_name("ollama"));

    let chain = FallbackLlmAdapter::new(vec![primary, backup, local])?;
    assert_eq!(chain.get_provider_name(), "fallback");

    // Hand the chain to a PaladinBuilder / PaladinExecutionService exactly
    // where a single provider adapter would go -- it is a plain `LlmPort`.
    Ok(Arc::new(chain))
}

The hop rule. Every call starts at element 0. The chain moves to the next element only when the current one fails with an error whose transience() is Transient or Unknown; a Permanent error short-circuits, so a malformed prompt or a rejected credential is never re-sent to every provider. When the whole chain fails, the caller gets LlmError::AllProvidersFailed { attempts, last } with one summary per provider in chain order, and its own transience is exactly the last attempt's. No state survives a call: one call's hop never changes where the next — or a concurrent one — starts.

Streaming: the first-chunk rule. generate_stream falls through only if the call itself fails or the stream's first item is an error (the adapter peeks it and still delivers it). Once any chunk has been delivered, a later error propagates unchanged — a partial answer is never silently completed by a different model.

Observability. Each hop emits TraceEvent::FallbackHop { node_id: None, from_provider, to_provider } through FallbackLlmAdapter::with_trace_sink and a warn! log line naming both providers. A served response is stamped with the serving provider's name under LlmResponse.metadata["paladin.served_by"], which PaladinExecutionService copies into PaladinResult.served_by (None for a non-fallback port). Treat served_by as observability, not attestation.

Circuit breakers sit above the chain. The facade CircuitBreaker yields PaladinError::CircuitBreakerOpen above the port boundary and is invisible to the adapter. A breaker wrapping an individual LlmPort inside the chain must surface an LlmError the chain classifies Transient (a NetworkError, or a 503 ProviderError) for the chain to hop past it.

Node Caching

A CachePolicy lets a node's result be served from a cache instead of executed. Two things are needed — a backend on the engine and a policy on the node:

use paladin_core::platform::container::aegis::{CacheKeySpec, CachePolicy};
use paladin_core::platform::container::battlefield::CacheMarker;
use paladin_storage::node_cache::InMemoryNodeCache;

/// Wire a cache backend on the engine and a cache policy on one node.
pub fn cached_graph()
-> Result<(WarEngine<InMemoryWaypointStore>, WarGraph), Box<dyn std::error::Error>> {
    // 1. The backend is an engine concern. Without one, ANY `cache` policy
    //    in the graph fails validation before a node runs.
    let engine = WarEngine::new(mock_paladin_port(), Arc::new(InMemoryWaypointStore::new()))
        .with_node_cache(Arc::new(InMemoryNodeCache::new()));

    // 2. A field that must never be served from a cached delta opts out
    //    at the schema: here an `Append` log, which a fork would otherwise
    //    replay one more time on every hit.
    let question = FieldName::new("question")?;
    let answer = FieldName::new("answer")?;
    let audit_log = FieldName::new("audit_log")?;
    let schema = BattlefieldSchema::new(vec![
        FieldSpec::new(question.clone(), DispatchRule::LastWrite, None, false),
        FieldSpec::new(answer.clone(), DispatchRule::LastWrite, None, false),
        FieldSpec::new(audit_log, DispatchRule::Append, None, false).with_cache(CacheMarker::Deny),
    ]);

    // 3. The policy is a node concern: a TTL and a key composition.
    let mut graph = WarGraph::new(schema, EngineLimits::default());
    graph.add_node(
        NodeId::new("answerer"),
        NodeSpec::paladin(
            create_paladin("Answerer"),
            InputMapping::new("Answer: {question}"),
            answer,
        ),
    );
    graph.set_aegis(
        NodeId::new("answerer"),
        Aegis {
            cache: Some(CachePolicy {
                ttl: Duration::from_secs(15 * 60),
                // `Default` keys on the graph fingerprint, the node id, the
                // rendered input and the Paladin's own configuration;
                // `Fields(..)` adds (or, for a Function node, narrows to)
                // the named fields' values.
                key: CacheKeySpec::Fields(vec![question]),
            }),
            ..Default::default()
        },
    );

    Ok((engine, graph))
}

Key composition. The key is a versioned, collision-resistant digest over: the graph fingerprint, the node id, the node's input (a Paladin node's rendered InputMapping string; a Function node's full Battlefield snapshot, or the Fields(..) subset), the mustered task_key and payload when present, and the Paladin's own configuration fingerprint (model, system prompt, temperature, max_loops, stop words). Change the prompt, the model, the graph shape or the input and the key changes — a stale hit is impossible by construction, not by convention. CacheKeySpec::Fields naming an undeclared field is a validation error (CacheKeyFieldUndeclared), because a typo would otherwise silently narrow the key.

TTL. ttl is a hard expiry with a closed boundary: an entry read at exactly its expires_at instant is a miss. The engine re-checks expiry on every hit regardless of what the backend served.

What a hit and a miss look like. The lookup runs before attempt 1 and outside the interceptor chain. A hit merges the stored delta with zero executions — no port call, no interceptor — and is recorded as Succeeded at attempt 1 with cache_hit: true on both the NodeExecutionRecord and the NodeFinished trace event. A miss executes the node normally and stores the delta only after a successful, Edges-routed attempt: a failure, a handler outcome, or a Goto/End/Parley/Muster directive is never cached (replaying only the delta would drop the routing).

Best effort by construction. A backend get error is a miss; a put error is logged and the run continues. Without a backend, any cache policy anywhere in the graph — including inside a Battalion child — fails start/resume with CachePolicyWithoutCacheBackend before a node runs.

Backends. InMemoryNodeCache (always available; process-local) and RedisNodeCache behind the redis-cache cargo feature on paladin-storage (facade passthrough redis-cache), which stores each entry as JSON under {key_prefix}:{key} with a server-side expiry and invalidates by cursor-based scan. Both run the same contract suite. Operators configure the backend through NodeCacheConfig (src/config/node_cache.rs), off by default:

The per-node policies (retry, timeout, handlers, cache TTL and key) are code, never configuration — there is no APP_AEGIS_* variable, on purpose.

The Node-Kind Support Matrix

A Battalion node's child run is the attempt unit: timeout cancels the child through the inherited token and the parent attempt fails with Timeout; on_error compensates the child's failure. retry and cache are rejected because child Waypoints are durable state — attempt isolation and cache replay would need per-attempt child-thread namespacing and a resume rule for a failed child, which do not exist yet. A Gate node has no attempt to retry, time or cache; its expiry is the Parley on_expire policy. Any Aegis on a Gate, or retry/cache on a Battalion, is EngineError::AegisUnsupportedForNodeKind, naming every offender. (A cache policy inside a Battalion child's own nodes works, through the inherited backend.)

Limitations You Should Know About

1. idle_timeout degrades to a wall clock on a port that never beats. The default PaladinPort::execute_observed delegates to execute and correctly claims no progress. An idle_timeout on a node whose port does not override it therefore fires after the window elapses from the start of the attempt, whether or not work is happening — still a bound, never no bound. PaladinExecutionService reports progress; a custom port must call heartbeat.beat() from execute_observed to get the progress-aware behaviour.
2. Caching an Append field replays the append. A cached delta is merged exactly as the original was, so a hit on a node writing an Append-dispatch field appends again — and a Chronicle fork that hits the cache appends once more on the fork. This is allowed; the opt-out is the schema-level cache: Deny marker (FieldSpec::with_cache(CacheMarker::Deny)), which rejects a Paladin output_field marked Deny at validation and refuses to store any Function delta that touches a Deny field.
3. Author-visible state carries raw content. An error_field written by a Route handler holds a provider's (redacted, bounded) error text; a Waypoint payload holds whatever the Battlefield holds. Both inherit the M-B-04 warning in MIGRATION.md §9.1: a downstream node or an LLM call that reads them is reading provider-influenced text. Do not template either into a prompt without the same care you would give any untrusted input.
4. A hanging EdgeConditionEvaluator is still unbounded — R-23-01 remains an accepted risk. Per-attempt timeouts wrap node execution, not edge evaluation. A hostile or hanging evaluator is author-supplied in-process code with the same trust as a StateNode; EngineLimits::max_supersteps remains the run-level bound, and the new timeout machinery does not cover it. This page does not claim otherwise.

Security Notes

• Redact, then bound. Every provider response body that enters an error — ProviderError, LlmFailure, a NodeError — passes through map_http_status, which redacts credentials before truncating to the 512-character excerpt budget. The order is load-bearing: bounding first can slice a secret across the truncation boundary and leak the surviving tail. Adapter-specific pre-checks (an Anthropic 403, a Gemini error envelope) run ahead of the shared helper and never duplicate its table.
• Classification never reads a message. transience() on every variant reads typed fields; a status is a u16, never a substring.
• NodeCacheConfig.redis_password is never Debug-printed — the struct renders it as a fixed placeholder.
• A cached delta is trusted state. The key includes the graph fingerprint, the Redis key prefix is configuration, and no cross-graph collision is possible by construction; a backend shared with untrusted writers is a backend concern, not something the engine can defend against.
• served_by is observability, not provenance.

Agent Runtime: Middleware, Context Management, Vault Memory, Structured Output and the Reasoning Agent

An ordered ExecutionMiddleware chain wrapping PaladinExecutionService's reasoning loop, the built-in policies that ride it (call/token/tool budgets, guardrails, retry/fallback, summarization, Vault recall), cross-session Vault memory, typed structured output, and a reasoning_agent preset that assembles all of it into a runnable agent — built as an opt-in layer over the execution service the Paladin Agents guide introduces.

Every Rust sample on this page is compiled code pulled from the paladin-doc-examples crate via mdBook {{#include}}, so a sample cannot drift from the landed API.

Table of Contents

1. The Two-Layer Contract: NodeInterceptor vs ExecutionMiddleware
2. Installing a Chain
3. Built-in Middleware
4. Vault: Cross-Session Memory
5. Structured Output
6. Native response_format by Provider
7. The Tool-Call Protocol
8. The reasoning_agent Preset
9. The Ollama Recipe
10. Security Notes

The Two-Layer Contract: NodeInterceptor vs ExecutionMiddleware

Paladin has two, deliberately independent, hook layers. Confusing them is the most common mistake when adding cross-cutting behavior to a Paladin — use this table before reaching for either:

The two never share a registry and neither wraps the other's decision as if it were the node's own fault: the interceptor decides whether the node runs at all; the middleware chain runs inside the node's own execution, once the interceptor has already let it proceed. When a WarEngine dispatches a NodeSpec::Paladin node it always calls PaladinPort::execute_observed (superstep.rs:946) — a PaladinExecutionService carrying middleware applies its chain automatically as a node, with no engine-side change and no second registry. See paladin-battalion's NodeInterceptor and paladin-ports's PaladinPort rustdoc for the same table restated on the two traits themselves.

Installing a Chain

Middleware is stateless and Arc-shared; per-run mutable state (counters, scratch data) lives on the ModelCallContext/ToolCallContext the chain receives, never on the middleware value itself — so one Arc<dyn ExecutionMiddleware> is safe to reuse across concurrent runs.

service
    .with_middleware(Arc::new(ModelCallLimit::new(10)))
    .with_middleware(Arc::new(TokenBudget::new(50_000)));

// Or replace the whole chain at once:
service.with_middleware_chain(vec![limit, guardrail, recall]);

Every built-in is inert by default — a service with no middleware installed, or an AgentRuntimeConfig with every section enabled: false (the default), behaves byte-identically to v0.9: the rendered prompt bytes, the port call count and the PaladinResult are unchanged. A single grouped AgentRuntimeConfig::build_chain(&self, deps) assembles every enabled section in a fixed order (limits → guardrail → trimmer/summarizer → recall → protocol → resilience) from one config.yml block, so an operator never has to hand-assemble the chain in code.

Built-in Middleware

StopReason is #[non_exhaustive]; CallLimit and TokenBudget both report is_successful() == true (the run ended with the model's last answer intact) and is_limit() == true. Guardrail failures surface as the typed, structured PaladinError::GuardrailTripped { rule, target }.

Every built-in's configuration lives under one AgentRuntimeConfig (src/config/agent_runtime.rs) — every section enabled: false by default, every scalar field overridable through APP_AGENT_RUNTIME_<SECTION>_<FIELD>. A v0.9 config.yml with no agent_runtime: section boots identically to before.

Vault: Cross-Session Memory

Paladin ships three distinct "memory" concepts, and it is easy to reach for the wrong one:

Reach for the Vault when a fact needs to survive past the end of the conversation or run that produced it and be readable from a different thread later (the user's preferred language, set once and read by every future conversation). Reach for the Garrison for the transcript of the current conversation. Reach for a Waypoint only if you are the engine itself checkpointing execution state.

Adapters: InMemoryVault (always available), SqliteVault (feature sqlite), SemanticVault (composes an existing SanctumPort + EmbeddingPort — ungated, since it holds only trait objects).

Confinement is structural, not a convention. The host grants a namespace subtree through a RunScope (WarEngine::with_vault(vault, base) for every engine node, or PaladinExecutionService::execute_scoped for a direct call); the vault_get/vault_put Armaments take an absolute namespace argument, and ConfinedVault rejects any call whose namespace does not have the grant as a prefix with VaultError::NamespaceDenied — before the inner store is ever touched. A run with no grant gets no Vault tools listed at all, never a fallback to the root namespace. Namespace::is_prefix_of compares path segments, not raw strings, so a namespace user/alice2 is never treated as a descendant of user/alice.

Vault values are JSON, size-bounded (max_value_bytes, default 64 KiB) and returned to the model as data recalled from storage, not instructions — the same framing this page's Security Notes apply to every other model-facing string on this page.

Structured Output

execute_structured<T: DeserializeOwned + JsonSchema> runs through a new StructuredExecutorPort (deliberately not PaladinPort — a structured run is a distinct execution shape, not an execute variant). The schema comes from schemars::schema_for! at the application layer; the value comes from serde_json::from_value — serde's own deserialization is the typed validation. Structured<T> { value, raw: PaladinResult } preserves the underlying PaladinResult alongside the typed value.

A bounded repair loop drives every structured run: attempt 1 appends a documented instruction block to the input (and, where the provider supports it, also sets a native response_format — belt-and-braces, so correctness never depends on the native mode); a parse or shape failure re-prompts, up to max_repair_attempts (default 1), with the parse error and the offending output; exhaustion raises the structured, typed PaladinError::StructuredOutputInvalid { attempts, last_error, raw_output }, preserving the raw text for inspection rather than discarding it.

On a WarEngine, a NodeSpec::Paladin with output_schema set writes the parsed JSON value to its output_field through the same driver; a node declaring output_schema on an engine with no with_structured_executor (or naming an unregistered Registered schema) is a typed EngineError at graph validation, before any node runs.

The ExecutionMiddleware chain does not run on this path (WR-02, 26-REVIEW.md): execute_structured/execute_json_schema dispatch directly, bypassing run_before/run_after/ run_around_tool entirely, so Guardrail, VaultRecallMiddleware, ToolCallLimit, TokenBudget/ModelCallLimit, and any custom middleware installed on the same PaladinExecutionService are silently inert for a structured-output call — this is intentional (the bounded repair loop is not the multi-loop reasoning loop before_model/after_model model), but it means a Guardrail rule or a token budget installed for execute() gives no protection on execute_structured()/execute_json_schema().

Native response_format by Provider

An absent response_format leaves every request body byte-identical to before this phase; no ProviderCapabilities field was added for it (a structured-output capability flag is a deferred idea, tracked separately from tool-calling capability).

The Tool-Call Protocol

No shipped LlmPort adapter ever populates LlmResponse.function_call (ADR-0042's deferred, wire-level tool calling — unchanged by this phase). ToolCallProtocolMiddleware makes the reasoning loop's tool branch reachable for a shipped provider anyway, at the prompt level:

• before_model renders the arsenal's list_armaments() (name, description, parameter schema) plus the call-format instructions into a ## Tools prompt section.
• after_model, when the response carries no real function_call, runs the same JSON extraction the structured-output machinery uses over the response text. If it yields the documented envelope {"tool": "<name>", "arguments": {...}} naming a tool the arsenal actually has, after_model synthesizes a function_call so the reasoning loop's existing tool-dispatch branch fires unchanged. An unknown tool name in the envelope is never synthesized into a call.
• FinishOnPlainAnswerMiddleware finishes the run with StopReason::Completed as soon as a response carries no tool call — without it, the loop runs to max_loops and returns StopReason::MaxLoops even after answering, which is the wrong default for a tool-using agent.

Both middlewares are opt-in (installed by the reasoning_agent preset, not by default), and this is a prompt-level protocol, not a wire-level one: no LlmRequest.tools field exists, no adapter changed, and every adapter's tool-calling capability flag stays exactly as it was. ADR-0042's native, wire-level tool calling remains deferred with its trigger unchanged.

The reasoning_agent Preset

paladin::presets::reasoning_agent(llm, arsenal, opts) assembles an LlmPort, an executable Arc<dyn ArsenalPort> and a ReasoningAgentOptions into a runnable ReasoningAgent — a thin wrapper exposing run(&self, input) -> Result<PaladinResult, PaladinError> (and run_structured::<T>). The arsenal argument must be executable, not just a list of tool definitions — build one with InProcessArsenal (an in-process closure-backed ArsenalPort), an MCP-backed ArsenalExecutionService, or a CompositeArsenalPort combining several.

pub async fn reasoning_agent_example() -> Result<(), Box<dyn std::error::Error>> {
    let llm = MockLlmAdapter::new().with_responses(vec![
        r#"{"tool":"add","arguments":{"a":2,"b":2}}"#.to_string(),
        "The answer is 4".to_string(),
    ]);
    let arsenal = InProcessArsenal::new()
        .with_tool(add_armament(), |_args| async { Ok(json!({ "sum": 4 })) });
    let agent = reasoning_agent(Arc::new(llm), Arc::new(arsenal), Default::default())?;
    let result = agent.run("What is 2+2?").await?;
    assert!(result.output.contains('4'));
    assert_eq!(result.loop_count, 2);
    assert_eq!(result.stop_reason, StopReason::Completed);
    Ok(())
}

By default, a failed tool call is fed back into the model's context as sanitized text and the loop continues (tool_error_mode: FeedToModel) — this is v0.9's existing behavior, named rather than changed; FailRun is the new opt-in that raises a structured PaladinError::ArmamentFailed instead. Every reason fed back to the model is redacted (bearer tokens, provider API-key shapes, key=/token= query values, JWT-shaped triples) before it is bounded to an excerpt length — redact, then bound, never the other way around.

The Ollama Recipe

Ollama is verified through the same OpenAI-compatible path every other compat-engine provider uses, not as a separate adapter — see the Configuration guide for the config.yml block, OLLAMA_BASE_URL, and running the env-probed integration suite:

cargo test --test ollama_docker --features integration-tests,llm-ollama

Security Notes

• Recalled Vault content and fed-back tool errors are data, not instructions. Both are wrapped in a delimited section that states this explicitly; treat anything an agent vault_puts the same way you would treat any other provider-influenced or user-influenced text — never a place to carry a credential, and never templated into a prompt as if it were trusted.
• Redact, then bound, everywhere a model-facing or error-facing string is built. The order is load-bearing: bounding first can slice a secret across the truncation boundary and leak the surviving tail.
• Guardrail regex patterns compile once, at construction, under an explicit size bound — the regex crate is linear-time, so there is no ReDoS surface, and an oversized or invalid pattern is a typed construction error rather than a runtime surprise.
• No secret-shaped field exists in AgentRuntimeConfig. ModelFallbackConfig names providers by string; credentials still come from the existing provider-factory env/config path, never from this struct.
• A hanging ExecutionMiddleware hook is bounded only by the node's Aegis run_timeout or the service's per-run timeout — there is no per-hook timeout yet. This is a stated, accepted limitation (alongside R-23-01, the hanging EdgeConditionEvaluator), not a gap this page hides.

Paladin Configuration Guide

This guide covers how to configure Paladin agents for optimal performance, from basic setup to advanced tuning.

Table of Contents

• Basic Configuration
• System Prompt Best Practices
• Model Selection
• Temperature and Sampling
• Stop Words and Termination
• Timeout and Retry Settings
• Advanced Configuration

Basic Configuration

Minimal Setup

use paladin::prelude::*;

let paladin = PaladinBuilder::new(llm_adapter)
    .name("Assistant")
    .system_prompt("You are a helpful assistant.")
    .build().await?;

Common Configuration

let paladin = PaladinBuilder::new(llm_adapter)
    .name("DataAnalyst")
    .system_prompt("You are an expert data analyst. Provide clear, data-driven insights.")
    .model("gpt-4")
    .temperature(0.7)
    .max_loops(5)
    .timeout_seconds(120)
    .build().await?;

Full Configuration

PaladinBuilder has no add_armament() method; tool registration goes through .with_arsenal_registry(registry) (src/application/services/paladin/paladin_builder.rs:686) -- see Tool Integration for the registry-wiring detail.

let paladin = PaladinBuilder::new(llm_adapter)
    .name("ResearchAssistant")
    .system_prompt("You are a research assistant specializing in academic papers.")
    .user_name("Researcher")
    .model("gpt-4-turbo")
    .temperature(0.8)
    .max_loops(10)
    .add_stop_word("END").add_stop_word("STOP").add_stop_word("FINAL_ANSWER")
    .timeout_seconds(300)
    .retry_attempts(3)
    .with_garrison(garrison)
    .with_arsenal_registry(arsenal_registry)
    .build().await?;

System Prompt Best Practices

The system prompt defines your Paladin's behavior and capabilities. Follow these best practices:

1. Be Specific About Role

❌ Vague:

.system_prompt("You are helpful.")

✅ Specific:

.system_prompt("You are a senior software engineer specializing in Rust. \
                You provide code reviews focused on safety, performance, and idiomatic patterns.")

2. Define Output Format

.system_prompt("You are a JSON API. Always respond with valid JSON. \
                Structure: {\"status\": \"success|error\", \"data\": {...}, \"message\": \"...\"}  \
                Never include markdown code blocks or explanations outside the JSON.")

3. Set Boundaries

.system_prompt("You are a customer support agent for TechCorp. \
                - Only answer questions about our products and services \
                - Escalate billing questions to the finance team \
                - Do not provide medical, legal, or financial advice \
                - Be polite and professional at all times")

4. Include Examples (Few-Shot)

.system_prompt("You categorize customer feedback as: FEATURE_REQUEST, BUG_REPORT, or PRAISE. \
                \
                Examples: \
                Input: 'The app crashes when I upload large files' \
                Output: BUG_REPORT \
                \
                Input: 'It would be great to have dark mode' \
                Output: FEATURE_REQUEST \
                \
                Input: 'Love the new design!' \
                Output: PRAISE")

5. Specify Tone and Style

.system_prompt("You are a technical writer creating documentation for developers. \
                - Use clear, concise language \
                - Prefer active voice \
                - Include code examples \
                - Target audience: junior to mid-level developers \
                - Avoid jargon unless necessary")

Model Selection

Choose the right model for your use case:

OpenAI Models

// GPT-4 Turbo - Best for complex reasoning
.model("gpt-4-turbo")  // Latest turbo model
.model("gpt-4")        // Standard GPT-4

// GPT-3.5 - Fast and cost-effective
.model("gpt-3.5-turbo")  // Recommended for most tasks

When to use:

• GPT-4: Complex reasoning, code generation, detailed analysis
• GPT-3.5: Simple queries, classification, summarization

DeepSeek Models

// DeepSeek Chat - Strong coding capabilities
.model("deepseek-chat")

// DeepSeek Coder - Specialized for code
.model("deepseek-coder")

When to use:

• deepseek-chat: General purpose, good for multi-turn conversations
• deepseek-coder: Code generation, technical documentation

Anthropic Models

// Claude 3 Family
.model("claude-3-opus")    // Most capable
.model("claude-3-sonnet")  // Balanced
.model("claude-3-haiku")   // Fastest

When to use:

• Opus: Complex analysis, long documents, creative writing
• Sonnet: General purpose, good balance of speed and quality
• Haiku: Fast responses, simple queries, high throughput

Model Comparison

Temperature and Sampling

Temperature controls randomness in responses:

Temperature Scale

// 0.0 - Deterministic, focused (best for factual tasks)
.temperature(0.0)

// 0.3-0.5 - Slightly varied (good for classification)
.temperature(0.4)

// 0.7 - Balanced (general purpose)
.temperature(0.7)

// 0.9-1.0 - Creative, diverse (brainstorming, creative writing)
.temperature(0.9)

// >1.0 - Very random (experimental, not recommended)
.temperature(1.2)

Use Cases by Temperature

Example: Task-Specific Configuration

// Code Review - Deterministic
let code_reviewer = PaladinBuilder::new(llm_adapter)
    .system_prompt("Review Rust code for safety and best practices.")
    .temperature(0.2)
    .build().await?;

// Content Writer - Creative
let writer = PaladinBuilder::new(llm_adapter)
    .system_prompt("Write engaging blog posts about technology.")
    .temperature(0.9)
    .build().await?;

// Customer Support - Balanced
let support = PaladinBuilder::new(llm_adapter)
    .system_prompt("Help customers with product questions.")
    .temperature(0.7)
    .build().await?;

Stop Words and Termination

Control when a Paladin stops generating:

Basic Stop Words

let paladin = PaladinBuilder::new(llm_adapter)
    .add_stop_word("END").add_stop_word("STOP").add_stop_word("###")
    .build().await?;

Use Cases

1. Structured Output

// Stop at delimiter for parsing
.system_prompt("Generate a list of items. End with '---'")
.add_stop_word("---")

2. Multi-Step Reasoning

// Stop when final answer is reached
.system_prompt("Think step by step. When done, output FINAL_ANSWER: <answer>")
.add_stop_word("FINAL_ANSWER:")

3. Dialog Systems

// Stop at turn boundaries
.system_prompt("You are user A in a conversation. End each turn with [END_TURN]")
.add_stop_word("[END_TURN]")

Max Loops

Prevent infinite reasoning loops:

// Default: 3 loops
.max_loops(3)

// For simple tasks: 1 loop
.max_loops(1)

// For complex reasoning: 10+ loops
.max_loops(15)

What is a loop? A loop is one reasoning cycle: prompt → LLM → response → (optional tool calls) → repeat.

Timeout and Retry Settings

Timeout Configuration

use std::time::Duration;

let paladin = PaladinBuilder::new(llm_adapter)
    .timeout_seconds(60)  // 60 second timeout
    .build().await?;

Recommended Timeouts:

• Simple queries: 30 seconds
• Complex reasoning: 120 seconds
• With tool calls: 300 seconds

Retry Configuration

let paladin = PaladinBuilder::new(llm_adapter)
    .retry_attempts(3)                        // Retry up to 3 times
    .build().await?;

Error Handling

Reaching max_loops is not an error -- there is no PaladinError::MaxLoopsExceeded variant. Execution returns Ok(PaladinResult { stop_reason: StopReason::MaxLoops, .. }) instead (crates/paladin-core/src/platform/container/execution_result.rs:38); check stop_reason on the success path rather than matching an Err arm for it:

match paladin.execute(input).await {
    Ok(response) if response.stop_reason == StopReason::MaxLoops => {
        eprintln!("Max reasoning loops exceeded");
        // Increase max_loops or refine system prompt
    }
    Ok(response) => println!("Success: {}", response.content),
    Err(PaladinError::Timeout(secs)) => {
        eprintln!("Request timed out after {} seconds", secs);
        // Increase timeout or simplify prompt
    }
    Err(PaladinError::LlmError(msg)) => {
        eprintln!("LLM error: {}", msg);
        // Check API key, rate limits, model availability
    }
    Err(e) => eprintln!("Other error: {}", e),
}

Advanced Configuration

Configuration from File

There is no paladin: top-level section in config.yml, and no ApplicationSettings type or PaladinBuilder::from_config() method (src/config/settings.rs -- the real top-level type is Settings, with no paladin field). A single Paladin is always configured via the PaladinBuilder fluent API shown throughout this guide, not loaded from a config-file section.

The one config-driven path that does exist is different in shape and purpose: the HTTP service host (paladin-server) loads a list of agents from Settings.agents: Vec<AgentDefinition> (src/config/agents.rs:209), keyed by id rather than name, and with a narrower field set (id, model, system_prompt, optional provider/temperature/max_loops/timeout_seconds, stop_words, allowed_roles -- no retry_attempts):

use paladin::config::Settings;

let config = Settings::load_from_file("config.yml")?;
// config.agents: Vec<AgentDefinition> -- the HTTP service host's agent registry

config.yml:

agents:
  - id: "assistant"
    model: "gpt-4"
    system_prompt: "You are a helpful assistant."
    temperature: 0.7
    max_loops: 5
    timeout_seconds: 120
    stop_words:
      - "END"
      - "STOP"

Environment-Based Configuration

let model = std::env::var("PALADIN_MODEL").unwrap_or("gpt-3.5-turbo".to_string());
let temperature = std::env::var("PALADIN_TEMPERATURE")
    .ok()
    .and_then(|s| s.parse::<f32>().ok())
    .unwrap_or(0.7);

let paladin = PaladinBuilder::new(llm_adapter)
    .model(&model)
    .temperature(temperature)
    .build().await?;

Dynamic Configuration

PaladinBuilder::build() is async (paladin_builder.rs:1267), so a synchronous factory method cannot return its result directly -- both methods below need async fn and .await:

struct PaladinFactory;

impl PaladinFactory {
    async fn create_for_task(task_type: &str, llm_adapter: Arc<dyn LlmPort>) -> Result<Paladin, PaladinError> {
        match task_type {
            "code_review" => Self::create_code_reviewer(llm_adapter).await,
            "creative_writing" => Self::create_writer(llm_adapter).await,
            "data_analysis" => Self::create_analyst(llm_adapter).await,
            _ => Self::create_default(llm_adapter).await,
        }
    }

    async fn create_code_reviewer(llm_adapter: Arc<dyn LlmPort>) -> Result<Paladin, PaladinError> {
        PaladinBuilder::new(llm_adapter)
            .system_prompt("Expert Rust code reviewer")
            .temperature(0.2)
            .model("gpt-4")
            .build()
            .await
    }

    // ... other factory methods
}

Configuration Validation

Validation happens inside .build() (PaladinBuilder::validate, src/application/services/paladin/paladin_builder.rs:1116, private to the builder) -- build() itself is the only validation entry point. There is no public paladin.validate() on the built Paladin for re-validating after construction:

let paladin = PaladinBuilder::new(llm_adapter)
    .temperature(0.7)
    .build()
    .await;  // Validates configuration as part of build()

if let Err(e) = paladin {
    eprintln!("Invalid configuration: {}", e);
}

Configuration Checklist

Before deploying a Paladin, verify:

• System prompt is clear and specific
• Appropriate model selected for task
• Temperature suitable for use case (0.2 for factual, 0.9 for creative)
• Max loops set appropriately (1-3 for simple, 10+ for complex)
• Timeout configured (30-300 seconds)
• Retry logic in place for production
• Stop words defined if needed
• Error handling implemented
• Configuration tested with sample inputs

Performance Tuning

For Throughput

// Fast model, simple prompts
let paladin = PaladinBuilder::new(llm_adapter)
    .model("gpt-3.5-turbo")
    .temperature(0.7)
    .max_loops(1)
    .timeout_seconds(30)
    .build().await?;

For Quality

// Best model, detailed prompts
let paladin = PaladinBuilder::new(llm_adapter)
    .model("gpt-4")
    .temperature(0.5)
    .max_loops(10)
    .timeout_seconds(300)
    .build().await?;

For Cost Efficiency

// Cheaper model, efficient prompts
let paladin = PaladinBuilder::new(llm_adapter)
    .model("deepseek-chat")
    .temperature(0.7)
    .max_loops(3)
    .build().await?;

Next Steps

• Battalion Patterns - Multi-agent orchestration
• Tool Integration - Add capabilities with Arsenal
• Memory Management - Use Garrison for context
• Examples - See configuration in action

Related Documentation

• Quickstart Guide
• API Reference
• Performance Tuning

Memory Management Guide

This guide covers how to use the Garrison memory system to give your Paladins conversation context, long-term knowledge, and semantic search capabilities.

Table of Contents

• Overview
• Garrison Architecture
• In-Memory Garrison
• Persistent Garrison
• Memory Windowing
• Semantic Search
• Memory Types
• Best Practices
• Advanced Patterns
• Troubleshooting

Overview

The Garrison system provides Paladins with:

• Conversation Context: Maintain multi-turn dialogue history
• Memory Windowing: Manage token limits intelligently
• Persistence: Save and restore sessions across restarts
• Semantic Search: Retrieve relevant memories by meaning, not just keywords
• Embeddings: Vector-based similarity for long-term memory

Key Concepts:

• Garrison: Memory storage system for a Paladin
• GarrisonEntry: Single memory record (message, observation, fact)
• ConversationHistory: Ordered sequence of interactions
• Memory Window: Limited context size respecting token limits
• Long-Term Memory: Persistent storage with semantic retrieval

Garrison Architecture

Core Components

// Single memory entry
pub struct GarrisonEntry {
    pub id: Uuid,
    pub role: ConversationRole,
    pub content: String,
    pub timestamp: DateTime<Utc>,
    pub metadata: HashMap<String, serde_json::Value>,
    pub token_count: Option<u32>,
}

// Conversation roles
pub enum ConversationRole {
    System,    // System prompts
    User,      // User messages
    Assistant, // Paladin responses
    Tool,      // Tool execution results
}

// Memory interface
#[async_trait]
pub trait GarrisonPort: Send + Sync {
    async fn remember(&self, entry: GarrisonEntry) -> Result<(), GarrisonError>;
    async fn recall_recent(&self, limit: usize) -> Result<Vec<GarrisonEntry>, GarrisonError>;
    async fn search(&self, query: &str, limit: usize) -> Result<Vec<GarrisonEntry>, GarrisonError>;
    async fn forget_all(&self) -> Result<(), GarrisonError>;
    async fn stats(&self) -> Result<GarrisonStats, GarrisonError>;
}

// Extended port for long-term memory
#[async_trait]
pub trait LongTermGarrisonPort: GarrisonPort {
    async fn remember_with_embedding(
        &self,
        entry: GarrisonEntry,
        embedding: Vec<f32>
    ) -> Result<(), GarrisonError>;

    async fn search_similar(
        &self,
        query_embedding: Vec<f32>,
        limit: usize
    ) -> Result<Vec<(GarrisonEntry, f32)>, GarrisonError>;
}

Memory Flow

User Input → Garrison adds User entry
    ↓
Paladin retrieves relevant history (window or search)
    ↓
LLM generates response with full context
    ↓
Garrison adds Assistant entry
    ↓
(Optional) Tool calls → Garrison adds Tool entries
    ↓
Repeat for next interaction

In-Memory Garrison

Fastest option for short-lived sessions where persistence isn't needed.

Basic Usage

use paladin_memory::garrison::InMemoryGarrison;
use paladin_core::platform::container::garrison::{GarrisonEntry, ConversationRole, GarrisonConfig};
use paladin::prelude::*;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let llm_adapter = Arc::new(OpenAIAdapter::new().build()?);

    // Create in-memory garrison — max_entries and max_tokens are GarrisonConfig::new
    // constructor arguments, not with_max_entries()/with_max_tokens() builder calls.
    let garrison = Arc::new(InMemoryGarrison::new(
        GarrisonConfig::new(100, Some(4000))
    ));

    // Build Paladin with memory
    let paladin = PaladinBuilder::new(llm_adapter)
        .name("ChatBot")
        .system_prompt("You are a helpful assistant with memory of our conversation.")
        .with_garrison(garrison.clone())
        .build()?;

    // First interaction
    let response1 = paladin.execute("My name is Alice").await?;
    println!("Bot: {}", response1.content);

    // Second interaction - Paladin remembers
    let response2 = paladin.execute("What's my name?").await?;
    println!("Bot: {}", response2.content);  // Should say "Alice"

    // Check garrison statistics
    let stats = garrison.stats().await?;
    println!("Total memories: {}", stats.entry_count);
    println!("Total tokens: {}", stats.total_tokens);

    Ok(())
}

Configuration Options

// GarrisonConfig::new(max_entries, max_tokens) — there is no default-then-with_max_*
// builder chain; entry/token limits are constructor arguments, not builder methods.
let garrison = InMemoryGarrison::new(
    GarrisonConfig::new(100, Some(4000))
        // Eviction strategy when limits reached
        .with_eviction_strategy(EvictionStrategy::FIFO)  // First-in-first-out
        // Entries always kept regardless of eviction strategy
        .with_preserve_recent(10)
);

// Token counting is a separate concern from GarrisonConfig — GarrisonEntry.token_count
// is populated by a `TokenCounterPort` implementation (the exact `TiktokenCounter::new("gpt-4")`,
// or the ungated default `HeuristicTokenCounter`), there is no `GarrisonConfig::with_token_counter`
// method.

Eviction Strategies

The real type is EvictionStrategy (crates/paladin-core/src/platform/container/garrison.rs), not EvictionPolicy, and it has three variants — there is no Lru or Custom(..) variant:

pub enum EvictionStrategy {
    // Remove oldest entries first
    FIFO,

    // Preserve system prompts and recent messages, evict middle entries (the default)
    ImportanceBased,

    // Always keep only the most recent N entries
    SlidingWindow,
}

ImportanceBased (the default) already protects system-role entries and the preserve_recent_count most recent entries before evicting anything else — the effect the former Custom closure example was reaching for is the built-in behavior, not something you need to hand-write:

let garrison = InMemoryGarrison::new(
    GarrisonConfig::new(100, Some(4000))
        .with_eviction_strategy(EvictionStrategy::ImportanceBased)
        .with_preserve_recent(10)
);

Persistent Garrison

SQLite-backed storage for sessions that need to survive restarts.

Setup

use paladin_memory::garrison::SqliteGarrison;
use paladin_core::platform::container::garrison::GarrisonConfig;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create persistent garrison — the constructor is `connect`, not `new`, and it
    // takes the config and a paladin_id (the scoping key) directly; there is no
    // separate `.with_config(...)` builder step.
    let garrison = Arc::new(
        SqliteGarrison::connect("garrison.db", GarrisonConfig::default(), "paladin-001")
            .await?
    );

    let paladin = PaladinBuilder::new(llm_adapter)
        .with_garrison(garrison)
        .build()?;

    // All interactions are automatically persisted
    paladin.execute("Remember this important fact!").await?;

    Ok(())
}

Scoping by Paladin

SqliteGarrison has no separate "session" concept or .with_session_id(...) method — the scoping key is the paladin_id argument passed directly to connect():

let paladin_id = "paladin-001";

let garrison = Arc::new(
    SqliteGarrison::connect("garrison.db", GarrisonConfig::default(), paladin_id).await?
);

// Later, reconnect scoped to the same Paladin
let garrison_restored = Arc::new(
    SqliteGarrison::connect("garrison.db", GarrisonConfig::default(), paladin_id).await?
);

// History is preserved
let history = garrison_restored.recall_recent(100).await?;
println!("Restored {} memories", history.len());

Multiple Users

pub struct UserGarrison {
    db: SqliteGarrison,
    user_id: String,
}

impl UserGarrison {
    pub async fn new(db_path: &str, user_id: String) -> Result<Self> {
        let db = SqliteGarrison::connect(db_path, GarrisonConfig::default(), &user_id).await?;
        Ok(Self { db, user_id })
    }
}

#[async_trait]
impl GarrisonPort for UserGarrison {
    async fn remember(&self, mut entry: GarrisonEntry) -> Result<()> {
        // Tag entries with user_id
        entry.metadata.insert("user_id".to_string(), self.user_id.clone());
        self.db.remember(entry).await
    }

    async fn recall_recent(&self, limit: usize) -> Result<Vec<GarrisonEntry>> {
        // Filter by user_id
        let all_entries = self.db.recall_recent(limit * 2).await?;
        Ok(all_entries.into_iter()
            .filter(|e| e.metadata.get("user_id") == Some(&self.user_id))
            .take(limit)
            .collect())
    }

    // Implement other methods...
}

// Usage
let alice_garrison = Arc::new(UserGarrison::new("garrison.db", "alice".to_string()).await?);
let bob_garrison = Arc::new(UserGarrison::new("garrison.db", "bob".to_string()).await?);

let alice_paladin = PaladinBuilder::new(llm_adapter.clone())
    .with_garrison(alice_garrison)
    .build()?;

let bob_paladin = PaladinBuilder::new(llm_adapter)
    .with_garrison(bob_garrison)
    .build()?;

Database Schema

The scoping column is paladin_id, not session_id (there is no session concept — see Scoping by Paladin above), embeddings live in a separate table, and SQLite indexes are declared with standalone CREATE INDEX statements, not inline in CREATE TABLE (SQLite does not support the inline INDEX (...) syntax at all — this excerpt is the real migrations/001_create_garrison_tables.sql, trimmed to the entries/metadata tables):

-- migrations/001_create_garrison_tables.sql
CREATE TABLE IF NOT EXISTS garrison_entries (
    id TEXT PRIMARY KEY NOT NULL,
    paladin_id TEXT NOT NULL,
    role TEXT NOT NULL CHECK(role IN ('system', 'user', 'assistant', 'tool')),
    content TEXT NOT NULL,
    timestamp TEXT NOT NULL,
    token_count INTEGER,
    metadata TEXT, -- JSON blob for flexible metadata
    created_at TEXT NOT NULL DEFAULT (datetime('now')),
    updated_at TEXT NOT NULL DEFAULT (datetime('now'))
);

CREATE INDEX IF NOT EXISTS idx_paladin_timestamp
ON garrison_entries(paladin_id, timestamp DESC);

CREATE INDEX IF NOT EXISTS idx_paladin_role
ON garrison_entries(paladin_id, role);

-- Vector embeddings live in a separate table, not inline in garrison_entries
CREATE TABLE IF NOT EXISTS garrison_embeddings (
    entry_id TEXT PRIMARY KEY NOT NULL,
    embedding BLOB,
    embedding_model TEXT NOT NULL,
    dimension INTEGER NOT NULL,
    created_at TEXT NOT NULL DEFAULT (datetime('now')),
    FOREIGN KEY (entry_id) REFERENCES garrison_entries(id) ON DELETE CASCADE
);

-- Per-Paladin config and running totals — there is no garrison_sessions table
CREATE TABLE IF NOT EXISTS garrison_metadata (
    paladin_id TEXT PRIMARY KEY NOT NULL,
    max_entries INTEGER NOT NULL DEFAULT 100,
    max_tokens INTEGER,
    eviction_strategy TEXT NOT NULL DEFAULT 'importance_based',
    preserve_recent_count INTEGER NOT NULL DEFAULT 10,
    total_entries INTEGER NOT NULL DEFAULT 0,
    total_tokens INTEGER NOT NULL DEFAULT 0
);

Memory Windowing

Intelligently manage context size to respect LLM token limits.

Token-Based Windowing

// Get most recent entries that fit within token limit
let window = garrison.recall_recent(4000).await?;

println!("Window contains {} entries", window.len());
println!("Total tokens: {}",
    window.iter().map(|e| e.token_count.unwrap_or(0)).sum::<u32>());

Sliding Window

pub struct SlidingWindowGarrison {
    garrison: Arc<dyn GarrisonPort>,
    window_size: u32,
}

impl SlidingWindowGarrison {
    pub fn new(garrison: Arc<dyn GarrisonPort>, window_size: u32) -> Self {
        Self { garrison, window_size }
    }
}

#[async_trait]
impl GarrisonPort for SlidingWindowGarrison {
    async fn recall_recent(&self, _limit: usize) -> Result<Vec<GarrisonEntry>> {
        // Always return windowed history
        self.garrison.recall_recent(self.window_size).await
    }

    // Forward other methods to inner garrison
    async fn remember(&self, entry: GarrisonEntry) -> Result<()> {
        self.garrison.remember(entry).await
    }

    // ... other methods
}

// Usage - Paladin always sees only recent context
let windowed = Arc::new(SlidingWindowGarrison::new(garrison, 4000));

let paladin = PaladinBuilder::new(llm_adapter)
    .with_garrison(windowed)
    .build()?;

Smart Windowing with Priorities

pub struct PriorityWindowGarrison {
    garrison: Arc<dyn GarrisonPort>,
    window_size: u32,
}

impl PriorityWindowGarrison {
    async fn get_prioritized_window(&self) -> Result<Vec<GarrisonEntry>> {
        let all_entries = self.garrison.recall_recent(1000).await?;

        // Always include system prompts
        let system_entries: Vec<_> = all_entries.iter()
            .filter(|e| e.role == ConversationRole::System)
            .cloned()
            .collect();

        // Calculate remaining token budget
        let system_tokens: u32 = system_entries.iter()
            .map(|e| e.token_count.unwrap_or(0))
            .sum();

        let remaining_budget = self.window_size.saturating_sub(system_tokens);

        // Fill with most recent non-system entries
        let mut recent_entries: Vec<_> = all_entries.iter()
            .filter(|e| e.role != ConversationRole::System)
            .rev()
            .cloned()
            .collect();

        let mut token_sum = 0u32;
        let mut windowed_recent = Vec::new();

        for entry in recent_entries {
            let entry_tokens = entry.token_count.unwrap_or(0);
            if token_sum + entry_tokens <= remaining_budget {
                token_sum += entry_tokens;
                windowed_recent.push(entry);
            } else {
                break;
            }
        }

        // Combine: system + recent (chronological order)
        windowed_recent.reverse();
        let mut result = system_entries;
        result.extend(windowed_recent);

        Ok(result)
    }
}

Summarization for Compression

pub struct SummarizingGarrison {
    garrison: Arc<dyn GarrisonPort>,
    summarizer: Arc<dyn LlmPort>,
    window_size: u32,
    summary_threshold: usize,
}

impl SummarizingGarrison {
    async fn maybe_summarize(&self) -> Result<()> {
        let entries = self.garrison.recall_recent(self.summary_threshold).await?;

        if entries.len() >= self.summary_threshold {
            // Create summary of old entries
            let old_entries: Vec<_> = entries.iter()
                .take(self.summary_threshold / 2)
                .collect();

            let conversation_text = old_entries.iter()
                .map(|e| format!("{:?}: {}", e.role, e.content))
                .collect::<Vec<_>>()
                .join("\n");

            let prompt = format!(
                "Summarize this conversation in 2-3 paragraphs, preserving key facts:\n\n{}",
                conversation_text
            );

            let summary = self.summarizer.generate(&prompt).await?;

            // GarrisonPort has no remove_entry()/selective-delete method — old entries
            // are not explicitly deleted here; they age out via the configured
            // EvictionStrategy once the summary entry below pushes past the limit.
            self.garrison.remember(GarrisonEntry {
                id: Uuid::new_v4(),
                role: ConversationRole::System,
                content: format!("Previous conversation summary: {}", summary),
                timestamp: Utc::now(),
                metadata: HashMap::from([
                    ("type".to_string(), "summary".to_string()),
                ]),
                token_count: None,
            }).await?;
        }

        Ok(())
    }
}

Semantic Search

Retrieve relevant memories by meaning using embeddings.

LongTermGarrisonPort::search_similar (above) is a real trait method, but the tree's own examples/garrison_semantic_search.rs documents it as "planned for future implementation" with no concrete Garrison-side adapter — the vector-search path that is actually implemented today is the separate Sanctum subsystem used below, not a Garrison extension.

Setup with Embeddings

There is no VectorGarrison type and no paladin_memory::embedding module — semantic search is a separate subsystem (Sanctum, crates/paladin-memory/src/sanctum/) built on the real EmbeddingPort trait and OpenAIEmbeddingAdapter (paladin_llm::openai::embedding), not a Garrison variant, and it is not wired into PaladinBuilder::with_garrison automatically:

use paladin_llm::openai::embedding::{OpenAIEmbeddingAdapter, OpenAIEmbeddingConfig};
use paladin_memory::sanctum::qdrant_adapter::QdrantSanctumAdapter;
use paladin_ports::output::embedding_port::EmbeddingPort;
use paladin_ports::output::sanctum_port::{SanctumPort, SanctumQuery};
use paladin_core::platform::container::sanctum::{MemoryBuilder, SanctumEntry};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Embedding generation and vector storage are two separate ports
    let embedder = OpenAIEmbeddingAdapter::new(OpenAIEmbeddingConfig {
        api_key,
        ..Default::default()
    });
    let sanctum = QdrantSanctumAdapter::new("http://localhost:6334", "paladin_memories", 1536).await?;

    // Store entries with embeddings generated explicitly (not automatic on paladin.execute())
    for text in [
        "I love hiking in the mountains",
        "My favorite color is blue",
        "I work as a software engineer",
    ] {
        let embedding = embedder.embed_text(text).await?;
        let memory = MemoryBuilder::new("paladin-001".to_string(), text.to_string()).build()?;
        sanctum.store(SanctumEntry::new(memory, embedding.vector)?).await?;
    }

    // Semantic search — SanctumSearchResult carries the similarity score, unlike
    // LongTermGarrisonPort::search_similar, which returns plain Vec<GarrisonEntry>
    let query_embedding = embedder.embed_text("outdoor activities").await?;
    let query = SanctumQuery::new(query_embedding.vector, 5);
    let results = sanctum.search(query).await?;

    for result in results {
        println!("Similarity: {:.2} - {}", result.score, result.entry.memory.content);
    }
    // Output: High similarity for "hiking in the mountains"

    Ok(())
}

Hybrid Search (Keyword + Semantic)

pub struct HybridGarrison {
    garrison: Arc<dyn LongTermGarrisonPort>,
}

impl HybridGarrison {
    pub async fn hybrid_search(
        &self,
        query: &str,
        limit: usize,
    ) -> Result<Vec<GarrisonEntry>> {
        // Get keyword matches
        let keyword_results = self.garrison.search(query, limit * 2).await?;

        // Get semantic matches
        let embedding = self.embedder.embed_text(query).await?;
        let semantic_results = self.garrison
            .semantic_search(embedding, limit * 2)
            .await?;

        // Merge and deduplicate
        let mut combined: HashMap<Uuid, (GarrisonEntry, f32)> = HashMap::new();

        // Add keyword results with base score
        for entry in keyword_results {
            combined.insert(entry.id, (entry, 0.5));
        }

        // Add semantic results, boosting score if already present
        for (entry, similarity) in semantic_results {
            combined.entry(entry.id)
                .and_modify(|(_, score)| *score += similarity * 0.5)
                .or_insert((entry, similarity * 0.5));
        }

        // Sort by combined score
        let mut sorted: Vec<_> = combined.into_values().collect();
        sorted.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap());

        Ok(sorted.into_iter()
            .take(limit)
            .map(|(entry, _)| entry)
            .collect())
    }
}

RAG (Retrieval-Augmented Generation)

pub struct RAGPaladin {
    paladin: Paladin,
    garrison: Arc<dyn LongTermGarrisonPort>,
}

impl RAGPaladin {
    pub async fn execute_with_rag(&self, query: &str) -> Result<PaladinResult> {
        // Retrieve relevant context from long-term memory
        let embedding = self.embedder.embed_text(query).await?;
        let relevant_memories = self.garrison
            .semantic_search(embedding, 5)
            .await?;

        // Build augmented prompt
        let context = relevant_memories.iter()
            .map(|(entry, _)| entry.content.as_str())
            .collect::<Vec<_>>()
            .join("\n\n");

        let augmented_query = format!(
            "Context from previous conversations:\n{}\n\n\
             Current question: {}",
            context, query
        );

        // Execute with retrieved context
        self.paladin.execute(&augmented_query).await
    }
}

// Usage
let rag_paladin = RAGPaladin {
    paladin,
    garrison: vector_garrison,
};

let response = rag_paladin.execute_with_rag(
    "What programming languages do I know?"
).await?;

Memory Types

Episodic Memory

Memory of specific events and experiences.

// Add episodic memory
garrison.remember(GarrisonEntry {
    id: Uuid::new_v4(),
    role: ConversationRole::User,
    content: "I visited Paris last summer".to_string(),
    timestamp: Utc::now(),
    metadata: HashMap::from([
        ("memory_type".to_string(), "episodic".to_string()),
        ("event_type".to_string(), "travel".to_string()),
        ("location".to_string(), "Paris, France".to_string()),
        ("timeframe".to_string(), "summer 2023".to_string()),
    ]),
    token_count: Some(10),
}).await?;

Semantic Memory

General knowledge and facts.

// Add semantic memory (facts)
garrison.remember(GarrisonEntry {
    id: Uuid::new_v4(),
    role: ConversationRole::System,
    content: "User prefers Python over JavaScript for backend development".to_string(),
    timestamp: Utc::now(),
    metadata: HashMap::from([
        ("memory_type".to_string(), "semantic".to_string()),
        ("category".to_string(), "preferences".to_string()),
        ("topic".to_string(), "programming".to_string()),
    ]),
    token_count: Some(15),
}).await?;

Procedural Memory

Knowledge about how to do things.

// Add procedural memory
garrison.remember(GarrisonEntry {
    id: Uuid::new_v4(),
    role: ConversationRole::System,
    content: "To deploy this project: cargo build --release && docker build -t app .".to_string(),
    timestamp: Utc::now(),
    metadata: HashMap::from([
        ("memory_type".to_string(), "procedural".to_string()),
        ("task".to_string(), "deployment".to_string()),
    ]),
    token_count: Some(20),
}).await?;

Best Practices

1. Choose the Right Garrison Type

// ✅ Use InMemoryGarrison for:
// - Temporary chatbots
// - Stateless services
// - Testing and development

let garrison = Arc::new(InMemoryGarrison::new(
    GarrisonConfig::new(100, Some(4000))
));

// ✅ Use SqliteGarrison for:
// - Multi-Paladin applications
// - Per-Paladin contexts
// - Production services needing persistence

let garrison = Arc::new(
    SqliteGarrison::connect("garrison.db", GarrisonConfig::default(), "paladin-001").await?
);

// ✅ Use Sanctum for:
// - Long-term knowledge bases
// - RAG applications
// - Semantic retrieval needs
//
// Sanctum (crates/paladin-memory/src/sanctum/) is a separate long-term-memory subsystem
// from Garrison, not a Garrison variant — there is no "VectorGarrison" type.

let sanctum = Arc::new(
    QdrantSanctumAdapter::new("http://localhost:6334", "paladin_memories", 1536).await?
);

2. Set Appropriate Token Limits

// Model context windows
const GPT_4_TURBO: u32 = 128_000;
const GPT_4: u32 = 8_192;
const GPT_3_5: u32 = 16_385;
const CLAUDE_3: u32 = 200_000;

// Reserve tokens for: system prompt + response + buffer
let response_tokens = 1000;
let system_prompt_tokens = 500;
let buffer = 500;

let available_for_history = GPT_4 - response_tokens - system_prompt_tokens - buffer;

let garrison = InMemoryGarrison::new(
    GarrisonConfig::new(100, Some(available_for_history))  // ~6000 tokens
);

3. Add Metadata for Better Organization

garrison.remember(GarrisonEntry {
    id: Uuid::new_v4(),
    role: ConversationRole::User,
    content: message.clone(),
    timestamp: Utc::now(),
    metadata: HashMap::from([
        ("user_id".to_string(), user_id.clone()),
        ("session_id".to_string(), session_id.to_string()),
        ("channel".to_string(), "web".to_string()),
        ("language".to_string(), "en".to_string()),
        ("importance".to_string(), "high".to_string()),
    ]),
    token_count: Some(estimate_tokens(&message)),
}).await?;

4. Clean Up Old Memories

GarrisonPort has no age-based or per-entry removal method — forget_all() is the only removal operation the trait exposes, and it clears everything. Age-based cleanup is handled automatically by GarrisonConfig's eviction strategy (with_eviction_strategy, with_preserve_recent) rather than an application-level remove_before(cutoff) call:

// Automatic: entries beyond max_entries/max_tokens are evicted per the configured
// EvictionStrategy on every `remember()` call — no scheduled task is required.
let garrison = InMemoryGarrison::new(
    GarrisonConfig::new(1000, Some(50_000))
        .with_eviction_strategy(EvictionStrategy::ImportanceBased)
);

// Manual, full reset (the only removal GarrisonPort exposes):
garrison.forget_all().await?;

5. Implement Conversation Branching

pub struct BranchingGarrison {
    garrison: Arc<dyn GarrisonPort>,
    current_branch: RwLock<Uuid>,
}

impl BranchingGarrison {
    pub async fn create_branch(&self, from_entry: Uuid) -> Result<Uuid> {
        let branch_id = Uuid::new_v4();

        // Copy history up to branch point
        let history = self.garrison.recall_recent(1000).await?;
        let branch_history: Vec<_> = history.into_iter()
            .take_while(|e| e.id != from_entry)
            .collect();

        // Store branch metadata
        self.garrison.remember(GarrisonEntry {
            id: Uuid::new_v4(),
            role: ConversationRole::System,
            content: format!("Branch created from entry {}", from_entry),
            timestamp: Utc::now(),
            metadata: HashMap::from([
                ("type".to_string(), "branch".to_string()),
                ("branch_id".to_string(), branch_id.to_string()),
                ("parent_entry".to_string(), from_entry.to_string()),
            ]),
            token_count: None,
        }).await?;

        *self.current_branch.write().await = branch_id;
        Ok(branch_id)
    }
}

Advanced Patterns

Memory Consolidation

pub struct ConsolidatingGarrison {
    garrison: Arc<dyn GarrisonPort>,
    llm: Arc<dyn LlmPort>,
}

impl ConsolidatingGarrison {
    pub async fn consolidate_memories(&self) -> Result<()> {
        let entries = self.garrison.recall_recent(100).await?;

        // Group by topic using LLM
        let topics = self.extract_topics(&entries).await?;

        // Create consolidated memory for each topic
        for (topic, topic_entries) in topics {
            let facts = self.extract_facts(&topic_entries).await?;

            self.garrison.remember(GarrisonEntry {
                id: Uuid::new_v4(),
                role: ConversationRole::System,
                content: format!("Consolidated facts about {}: {}", topic, facts),
                timestamp: Utc::now(),
                metadata: HashMap::from([
                    ("type".to_string(), "consolidated".to_string()),
                    ("topic".to_string(), topic),
                    ("source_count".to_string(), topic_entries.len().to_string()),
                ]),
                token_count: None,
            }).await?;
        }

        Ok(())
    }

    async fn extract_topics(&self, entries: &[GarrisonEntry]) -> Result<HashMap<String, Vec<GarrisonEntry>>> {
        // Use LLM to categorize entries by topic
        // Implementation details...
        Ok(HashMap::new())
    }

    async fn extract_facts(&self, entries: &[GarrisonEntry]) -> Result<String> {
        let conversation = entries.iter()
            .map(|e| &e.content)
            .cloned()
            .collect::<Vec<_>>()
            .join("\n");

        let prompt = format!(
            "Extract key facts from this conversation:\n\n{}",
            conversation
        );

        self.llm.generate(&prompt).await
    }
}

Attention Mechanism

pub struct AttentionGarrison {
    garrison: Arc<dyn LongTermGarrisonPort>,
}

impl AttentionGarrison {
    pub async fn get_attended_context(
        &self,
        query: &str,
        context_size: u32,
    ) -> Result<Vec<GarrisonEntry>> {
        // Get semantic matches
        let query_embedding = self.embedder.embed_text(query).await?.vector;
        let candidates = self.garrison
            .semantic_search(query_embedding, 50)
            .await?;

        // Score each candidate using attention mechanism
        let mut scored: Vec<_> = candidates.into_iter()
            .map(|(entry, similarity)| {
                let recency_score = self.recency_score(&entry);
                let importance_score = self.importance_score(&entry);

                // Weighted combination
                let attention = similarity * 0.5 + recency_score * 0.3 + importance_score * 0.2;

                (entry, attention)
            })
            .collect();

        // Sort by attention score
        scored.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap());

        // Select top entries within token budget
        let mut selected = Vec::new();
        let mut token_sum = 0u32;

        for (entry, _) in scored {
            let entry_tokens = entry.token_count.unwrap_or(0);
            if token_sum + entry_tokens <= context_size {
                token_sum += entry_tokens;
                selected.push(entry);
            }
        }

        Ok(selected)
    }

    fn recency_score(&self, entry: &GarrisonEntry) -> f32 {
        let age = (Utc::now() - entry.timestamp).num_seconds() as f32;
        let decay_rate = 0.0001;  // Adjust for desired decay speed
        (-decay_rate * age).exp()
    }

    fn importance_score(&self, entry: &GarrisonEntry) -> f32 {
        // Extract importance from metadata or content
        entry.metadata.get("importance")
            .and_then(|s| s.parse::<f32>().ok())
            .unwrap_or(0.5)
    }
}

Memory Reflection

pub struct ReflectiveGarrison {
    garrison: Arc<dyn GarrisonPort>,
    llm: Arc<dyn LlmPort>,
}

impl ReflectiveGarrison {
    pub async fn generate_reflections(&self) -> Result<()> {
        let recent_entries = self.garrison.recall_recent(50).await?;

        // Prompt LLM to reflect on conversation
        let conversation = recent_entries.iter()
            .map(|e| format!("{:?}: {}", e.role, e.content))
            .collect::<Vec<_>>()
            .join("\n");

        let prompt = format!(
            "Reflect on this conversation and extract:\n\
             1. Key insights about the user\n\
             2. Patterns in the discussion\n\
             3. Important facts to remember\n\n\
             Conversation:\n{}",
            conversation
        );

        let reflection = self.llm.generate(&prompt).await?;

        // Store reflection as high-importance memory
        self.garrison.remember(GarrisonEntry {
            id: Uuid::new_v4(),
            role: ConversationRole::System,
            content: format!("Reflection: {}", reflection),
            timestamp: Utc::now(),
            metadata: HashMap::from([
                ("type".to_string(), "reflection".to_string()),
                ("importance".to_string(), "high".to_string()),
            ]),
            token_count: None,
        }).await?;

        Ok(())
    }
}

Troubleshooting

Memory Not Persisting

Problem: Garrison entries disappear after restart.

Solutions:

1. Verify using SqliteGarrison, not InMemoryGarrison
2. Check database file path is correct and writable
3. Ensure proper async handling (.await on all operations)

// ❌ Won't persist
let garrison = Arc::new(InMemoryGarrison::new(config));

// ✅ Will persist
let garrison = Arc::new(SqliteGarrison::new("garrison.db").await?);

Context Window Overflow

Problem: Errors about exceeding maximum context length.

Solutions:

1. Reduce max_tokens in GarrisonConfig
2. Use get_window() instead of get_history()
3. Implement summarization for old memories

// Calculate safe token limit
let model_limit = 8192;  // GPT-4
let response_budget = 1000;
let system_prompt_tokens = 500;
let safety_buffer = 500;

let garrison_limit = model_limit - response_budget - system_prompt_tokens - safety_buffer;

let garrison = InMemoryGarrison::new(
    GarrisonConfig::new(100, Some(garrison_limit))
);

Slow Semantic Search

Problem: Embedding-based search is taking too long.

Solutions:

1. Add database indexes on embedding columns
2. Use approximate nearest neighbor (ANN) algorithms
3. Cache embeddings for frequent queries
4. Limit search scope with filters

-- Embeddings live in garrison_embeddings, not inline on garrison_entries
-- (see Database Schema above); index by model for faster lookups:
CREATE INDEX IF NOT EXISTS idx_embedding_model ON garrison_embeddings(embedding_model);

-- The workspace already ships a Qdrant adapter for vector search at scale
-- (crates/paladin-memory/src/sanctum/qdrant_adapter.rs, behind the `qdrant` feature)

Memory Leaks in Long Sessions

Problem: Memory usage grows unbounded.

Solutions:

1. Set max_entries in config
2. Implement periodic cleanup
3. Use eviction policies
4. Monitor with garrison.stats()

// Periodic monitoring — GarrisonPort has no compact()/partial-cleanup method;
// eviction already runs automatically per the configured EvictionStrategy on
// every remember() call, so this loop only needs to watch for problems.
tokio::spawn(async move {
    let mut interval = tokio::time::interval(Duration::from_secs(3600));
    loop {
        interval.tick().await;

        let stats = garrison.stats().await.unwrap();

        if stats.entry_count > 1000 {
            log::warn!("Garrison entry_count exceeds expected bound: {}", stats.entry_count);
        }
    }
});

Testing

Unit Testing

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn test_garrison_add_and_retrieve() {
        let garrison = InMemoryGarrison::new(GarrisonConfig::default());

        let entry = GarrisonEntry {
            id: Uuid::new_v4(),
            role: ConversationRole::User,
            content: "Test message".to_string(),
            timestamp: Utc::now(),
            metadata: HashMap::new(),
            token_count: Some(2),
        };

        garrison.remember(entry.clone()).await.unwrap();

        let history = garrison.recall_recent(10).await.unwrap();
        assert_eq!(history.len(), 1);
        assert_eq!(history[0].content, "Test message");
    }

    #[tokio::test]
    async fn test_token_window() {
        let garrison = InMemoryGarrison::new(
            GarrisonConfig::new(100, Some(100))
        );

        // Add entries totaling 150 tokens
        for i in 0..15 {
            garrison.remember(GarrisonEntry {
                id: Uuid::new_v4(),
                role: ConversationRole::User,
                content: format!("Message {}", i),
                timestamp: Utc::now(),
                metadata: HashMap::new(),
                token_count: Some(10),
            }).await.unwrap();
        }

        // Window should respect token limit
        let window = garrison.recall_recent(100).await.unwrap();
        let total_tokens: u32 = window.iter()
            .map(|e| e.token_count.unwrap_or(0))
            .sum();

        assert!(total_tokens <= 100);
    }
}

Examples

See working examples:

• examples/garrison_in_memory.rs - Basic in-memory usage
• examples/garrison_persistent.rs - SQLite persistence
• examples/garrison_semantic_search.rs - Placeholder text search; semantic search is not yet implemented for Garrison (see Semantic Search above)

Next Steps

• Tool Integration - Combine memory with tools
• Battalion Patterns - Shared memory in multi-agent systems
• API Reference - Garrison API documentation

Related Resources

• Token Counting Strategies
• Vector Database Integration
• Production Deployment

Tool Integration Guide

This guide covers how to integrate external tools and capabilities into your Paladins using the Arsenal system and Model Context Protocol (MCP).

Table of Contents

• Overview
• Arsenal Architecture
• MCP Protocol
• STDIO Tool Servers
• Streamable-HTTP Tool Servers
• Custom Tool Development
• Tool Result Handling
• Best Practices
• Troubleshooting

Overview

The Arsenal system enables Paladins to:

• Execute external tools and capabilities
• Search the web, access databases, run calculations
• Interact with APIs and services
• Extend functionality without modifying core code

Key Concepts:

• Arsenal: The registry of available tools
• Armament: A single tool or capability
• MCP (Model Context Protocol): Standard protocol for tool servers
• Tool Call: Request from Paladin to execute a tool
• Tool Result: Response from tool execution

Reachability note: Arsenal and MCP tool execution ship and work today — everything below this note is real and invocable. No shipped LlmPort adapter (OpenAI, Anthropic, DeepSeek, or the bundled mock) ever returns a populated function call from generate() — that narrower, wire-level path still requires a consumer-supplied LlmPort implementation that parses tool calls itself. See ADR-0042 for the tracked status of native, wire-level tool calling.

There is a shipped, opt-in alternative that does not require a custom LlmPort: the prompt-level tool-call protocol middleware pair, ToolCallProtocolMiddleware and FinishOnPlainAnswerMiddleware, lets a Paladin's own reasoning loop parse tool calls out of plain-text output and trigger an Armament without any consumer-written port implementation. Both are installed by the reasoning_agent preset. See Agent Runtime: The Tool-Call Protocol for the full mechanism.

Arsenal Architecture

Core Components

// Armament - Tool definition
pub struct Armament {
    pub name: String,
    pub description: String,
    pub parameters: Value,           // JSON Schema describing accepted parameters
    pub required_params: Vec<String>,
}

// Arsenal Port - Tool execution interface
#[async_trait]
pub trait ArsenalPort: Send + Sync {
    async fn list_armaments(&self) -> Vec<Armament>;
    async fn invoke(&self, call: ArmamentCall) -> Result<ArmamentResult, ArsenalError>;
    fn validate_call(&self, call: &ArmamentCall) -> Result<(), ArsenalError>;
}

// Armament Call - Tool invocation request
pub struct ArmamentCall {
    pub tool_name: String,
    pub arguments: HashMap<String, Value>,
    pub call_id: Uuid,
}

// Armament Result - Tool execution response
pub struct ArmamentResult {
    pub call_id: Uuid,
    pub success: bool,
    pub output: Option<Value>,
    pub error: Option<String>,
    pub execution_time_ms: u64,
}

Tool Flow

Paladin → LLM decides to use tool → ArmamentCall
    ↓
ArsenalPort validates call → Routes to correct Armament
    ↓
Tool executes (MCP server, API, local function)
    ↓
ArmamentResult → Injected into Paladin context
    ↓
Paladin continues reasoning with tool result

The first arrow — "LLM decides to use tool" — is the step no shipped adapter can take today; see the reachability note above and ADR-0042.

MCP Protocol

The Model Context Protocol (MCP) is an open standard for connecting LLM applications to external tools and data sources.

MCP Server Types

1. STDIO Servers: Command-line tools communicating via stdin/stdout
2. Streamable-HTTP Servers: Remote, optionally authenticated web services (the current, real transport; supersedes the legacy standalone SSE transport per the MCP spec — Paladin's own retired MCPSseAdapter was never actually SSE, just a mislabeled plain-HTTP-POST adapter)

MCP Message Format

// Tool Discovery Request
{
  "jsonrpc": "2.0",
  "method": "tools/list",
  "id": 1
}

// Tool Discovery Response
{
  "jsonrpc": "2.0",
  "result": {
    "tools": [
      {
        "name": "web_search",
        "description": "Search the web for information",
        "inputSchema": {
          "type": "object",
          "properties": {
            "query": {
              "type": "string",
              "description": "Search query"
            }
          },
          "required": ["query"]
        }
      }
    ]
  },
  "id": 1
}

// Tool Invocation Request
{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "name": "web_search",
    "arguments": {
      "query": "Rust async programming"
    }
  },
  "id": 2
}

// Tool Invocation Response
{
  "jsonrpc": "2.0",
  "result": {
    "content": [
      {
        "type": "text",
        "text": "Search results: ..."
      }
    ]
  },
  "id": 2
}

STDIO Tool Servers

STDIO servers are command-line programs that communicate via standard input/output.

Connecting a STDIO Server

use paladin::infrastructure::adapters::arsenal::mcp_stdio_adapter::MCPStdioAdapter;
use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin::prelude::*;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let llm_adapter = Arc::new(OpenAIAdapter::new().build()?);

    // Connect to an MCP STDIO server: MCPStdioAdapter::new(command, args) is
    // a thin builder; connect() spawns the subprocess and performs the full
    // MCP initialize -> notifications/initialized handshake, returning an
    // MCPClient with discover_tools()/invoke_tool().
    let mcp_client = MCPStdioAdapter::new("uvx", vec!["mcp-server-fetch"])
        .connect()
        .await?;
    let tools = mcp_client.discover_tools().await?;

    // Build Paladin with tool access (register discovered tools, or attach
    // an ArsenalRegistry backed by the connected client)
    let paladin = PaladinBuilder::new(llm_adapter)
        .name("ResearchAssistant")
        .system_prompt("You are a research assistant with web search capabilities. \
                        Use the web_search tool to find current information. \
                        Always cite your sources.")
        .build()
        .await?;

    // Paladin will automatically use tools when needed
    let response = paladin.execute("What are the latest Rust features in 2024?").await?;
    println!("{}", response.content);

    Ok(())
}

Popular STDIO MCP Servers

# Web search
uvx mcp-server-fetch

# File system access
uvx mcp-server-filesystem --allowed-directory ~/Documents

# Git operations
uvx mcp-server-git --repository /path/to/repo

# Database queries
uvx mcp-server-sqlite --db-path database.db

# Calculator
uvx mcp-server-calculator

Configuration Example

The YAML key is server_type (the MCPServerConfig field name, src/config/arsenal.rs:13), not type -- type deserializes as an unknown field and server_type is required, so a type:-keyed entry fails to load. There is no enabled field on MCPServerConfig; omit a server entirely to disable it.

arsenal:
  mcp_servers:
    - name: "web_search"
      server_type: "stdio"
      command: "uvx"
      args: ["mcp-server-fetch"]

    - name: "filesystem"
      server_type: "stdio"
      command: "uvx"
      args:
        - "mcp-server-filesystem"
        - "--allowed-directory"
        - "/home/user/workspace"

    - name: "calculator"
      server_type: "stdio"
      command: "uvx"
      args: ["mcp-server-calculator"]

Advanced STDIO Configuration

MCPStdioAdapter is currently a minimal builder — it only accepts a command and its arguments; working-directory, per-server env-var injection, custom timeouts, and retry policies are not yet exposed on the adapter itself (pass any required env vars via the spawned command's own args/environment, or via the process that launches Paladin):

let client = MCPStdioAdapter::new("uvx", vec!["mcp-server-fetch"])
    .connect()
    .await?;

Streamable-HTTP Tool Servers

Streamable-HTTP servers are remote, optionally authenticated MCP servers reached over HTTP(S) — Paladin's real remote transport (Phase 12.1 D-02/D-03). This supersedes the legacy standalone SSE transport in the MCP spec; Paladin's own retired MCPSseAdapter was never actually SSE, just a mislabeled, unauthenticated plain-HTTP-POST adapter, and has been removed entirely.

Connecting a Streamable-HTTP Server

use paladin::infrastructure::adapters::arsenal::mcp_streamable_http_adapter::MCPStreamableHttpAdapter;
use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin::prelude::*;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let llm_adapter = Arc::new(OpenAIAdapter::new().build()?);

    // Connect to a remote MCP server over Streamable-HTTP. The bearer token
    // is read from an env var here in application code -- never hardcode a
    // literal secret, and never accept one as a CLI argument.
    let bearer_token = std::env::var("API_KEY").ok();
    let mut adapter = MCPStreamableHttpAdapter::new("https://api.example.com/mcp");
    if let Some(token) = bearer_token {
        adapter = adapter.with_bearer_token(token);
    }
    let mcp_client = adapter.connect().await?;
    let tools = mcp_client.discover_tools().await?;

    let paladin = PaladinBuilder::new(llm_adapter)
        .name("APIAssistant")
        .system_prompt("You have access to company APIs. Use them to retrieve data.")
        .build()
        .await?;

    let response = paladin.execute("Get user statistics for last month").await?;
    println!("{}", response.content);

    Ok(())
}

Streamable-HTTP Configuration

The config-driven flow (recommended for most use cases) declares the server in config.yml and lets the CLI/loader connect it for you. (.mcp.json is Claude Code's own project-scoped MCP configuration for this repository's editor tooling; the Paladin application's config loader does not read it -- only config.yml's arsenal.mcp_servers list, src/config/arsenal.rs:37.)

arsenal:
  mcp_servers:
    - name: "company_api"
      server_type: "streamable_http"
      endpoint: "https://api.example.com/mcp"
      # NAMES the env var holding the bearer token -- never a literal secret
      # in this file. Omit entirely for an unauthenticated server.
      auth_token_env: "API_KEY"

export API_KEY="your-token"
paladin arsenal test --mcp-streamable-http "https://api.example.com/mcp" \
    --mcp-auth-token-env API_KEY

The equivalent programmatic flow — MCPStreamableHttpAdapter is a thin builder over MCPClient::connect_streamable_http, mirroring MCPStdioAdapter's shape for the remote transport:

use paladin::infrastructure::adapters::arsenal::mcp_streamable_http_adapter::MCPStreamableHttpAdapter;

let adapter = MCPStreamableHttpAdapter::new("https://api.example.com/mcp")
    .with_bearer_token(std::env::var("API_KEY")?); // never hardcode the token
    // .with_custom_headers(headers) is also available for non-bearer auth schemes

let client = adapter.connect().await?; // full initialize -> notifications/initialized handshake

Or call the underlying MCPClient constructor directly:

use paladin::infrastructure::adapters::arsenal::mcp_protocol::MCPClient;

let client = MCPClient::connect_streamable_http(
    "https://api.example.com/mcp",
    Some(&std::env::var("API_KEY")?),
    None, // optional custom headers
)
.await?;

Verifying Connectivity and Listing Tools

There is no separate health_check() — a successful connect()/connect_streamable_http() already means the full MCP handshake succeeded, so discover_tools() doubles as the liveness check:

// A successful discover_tools() call confirms the server is reachable and
// the handshake succeeded.
let tools = client.discover_tools().await?;
println!("Streamable-HTTP server is healthy — {} tool(s) available", tools.len());
for tool in tools {
    println!("Tool: {} - {}", tool.name, tool.description);
}

Custom Tool Development

Create your own tools by implementing the ArsenalPort trait.

Wiring note: PaladinBuilder has no add_armament() method (it does not appear anywhere in the tree). The builder's only Arsenal-related method is with_arsenal_registry(registry: Arc<dyn ArsenalRegistry>) (src/application/services/paladin/paladin_builder.rs:686), which attaches an ArsenalRegistry — a metadata catalog of Armaments, populated via registry.register(armament).await (crates/paladin-ports/src/output/arsenal_port.rs:809) — not a live ArsenalPort invoker. The examples below build Arc::new(<YourTool>) and call .add_armament(tool) on the builder for brevity; treat that call as shorthand for registering the tool's Armament metadata into a registry via .with_arsenal_registry(registry) and wiring the tool's actual invoke() logic through the execution path that consumes Option<Arc<dyn ArsenalPort>> (src/application/services/paladin/paladin_execution_service.rs:116), not a single builder call.

Simple Custom Tool

use paladin_ports::output::arsenal_port::{ArsenalPort, ArsenalRegistry};
use paladin_core::platform::container::arsenal::{Armament, ArmamentCall, ArmamentResult};
use async_trait::async_trait;

pub struct CalculatorTool;

#[async_trait]
impl ArsenalPort for CalculatorTool {
    async fn list_armaments(&self) -> Vec<Armament> {
        vec![
            Armament {
                name: "add".to_string(),
                description: "Add two numbers".to_string(),
                parameters: serde_json::json!({
                    "type": "object",
                    "properties": {
                        "a": { "type": "number", "description": "First number" },
                        "b": { "type": "number", "description": "Second number" }
                    },
                    "required": ["a", "b"]
                }),
                required_params: vec!["a".to_string(), "b".to_string()],
            },
            Armament {
                name: "multiply".to_string(),
                description: "Multiply two numbers".to_string(),
                parameters: serde_json::json!({
                    "type": "object",
                    "properties": {
                        "a": { "type": "number", "description": "First number" },
                        "b": { "type": "number", "description": "Second number" }
                    },
                    "required": ["a", "b"]
                }),
                required_params: vec!["a".to_string(), "b".to_string()],
            },
        ]
    }

    async fn invoke(&self, call: ArmamentCall) -> Result<ArmamentResult, ArsenalError> {
        let a = call.arguments.get("a")
            .and_then(|v| v.as_f64())
            .ok_or_else(|| ArsenalError::InvalidArguments("a".to_string()))?;

        let b = call.arguments.get("b")
            .and_then(|v| v.as_f64())
            .ok_or_else(|| ArsenalError::InvalidArguments("b".to_string()))?;

        let result = match call.tool_name.as_str() {
            "add" => a + b,
            "multiply" => a * b,
            _ => return Err(ArsenalError::ToolNotFound(call.tool_name.clone())),
        };

        Ok(ArmamentResult {
            call_id: call.call_id,
            success: true,
            output: Some(serde_json::json!(result)),
            error: None,
            execution_time_ms: 1,
        })
    }

    fn validate_call(&self, call: &ArmamentCall) -> Result<(), ArsenalError> {
        // validate_call is synchronous, so it checks against a known, static
        // required-parameter list rather than awaiting the async list_armaments().
        let required: &[&str] = match call.tool_name.as_str() {
            "add" | "multiply" => &["a", "b"],
            _ => return Err(ArsenalError::ToolNotFound(call.tool_name.clone())),
        };

        for param in required {
            if !call.arguments.contains_key(*param) {
                return Err(ArsenalError::InvalidArguments(format!("missing required parameter: {param}")));
            }
        }

        Ok(())
    }
}

// Use the custom tool
let calculator = Arc::new(CalculatorTool);

let paladin = PaladinBuilder::new(llm_adapter)
    .add_armament(calculator)
    .build()?;

API Integration Tool

use reqwest::Client;

pub struct WeatherTool {
    client: Client,
    api_key: String,
}

impl WeatherTool {
    pub fn new(api_key: String) -> Self {
        Self {
            client: Client::new(),
            api_key,
        }
    }
}

#[async_trait]
impl ArsenalPort for WeatherTool {
    async fn list_armaments(&self) -> Vec<Armament> {
        vec![
            Armament {
                name: "get_weather".to_string(),
                description: "Get current weather for a location".to_string(),
                parameters: serde_json::json!({
                    "type": "object",
                    "properties": {
                        "location": { "type": "string", "description": "City name or coordinates" },
                        "units": { "type": "string", "description": "Temperature units (celsius/fahrenheit)" }
                    },
                    "required": ["location"]
                }),
                required_params: vec!["location".to_string()],
            },
        ]
    }

    async fn invoke(&self, call: ArmamentCall) -> Result<ArmamentResult, ArsenalError> {
        let location = call.arguments.get("location")
            .and_then(|v| v.as_str())
            .ok_or_else(|| ArsenalError::InvalidArguments("location".to_string()))?;

        let units = call.arguments.get("units")
            .and_then(|v| v.as_str())
            .unwrap_or("celsius");

        // Call weather API
        let url = format!(
            "https://api.openweathermap.org/data/2.5/weather?q={}&appid={}&units={}",
            location, self.api_key, units
        );

        let response = self.client.get(&url)
            .send()
            .await
            .map_err(|e| ArsenalError::TransportError(e.to_string()))?;

        let weather_data = response.json::<serde_json::Value>()
            .await
            .map_err(|e| ArsenalError::TransportError(e.to_string()))?;

        let temp = weather_data["main"]["temp"].as_f64().unwrap_or(0.0);
        let description = weather_data["weather"][0]["description"]
            .as_str()
            .unwrap_or("unknown");

        let output = format!(
            "Weather in {}: {} with temperature of {}°",
            location, description, temp
        );

        Ok(ArmamentResult {
            call_id: call.call_id,
            success: true,
            output: Some(serde_json::json!(output)),
            error: None,
            execution_time_ms: 200,
        })
    }

    fn validate_call(&self, call: &ArmamentCall) -> Result<(), ArsenalError> {
        if call.tool_name != "get_weather" {
            return Err(ArsenalError::ToolNotFound(call.tool_name.clone()));
        }

        if !call.arguments.contains_key("location") {
            return Err(ArsenalError::InvalidArguments("missing required parameter: location".to_string()));
        }

        Ok(())
    }
}

// Usage
let weather = Arc::new(WeatherTool::new(api_key));

let paladin = PaladinBuilder::new(llm_adapter)
    .system_prompt("You can check weather. Use get_weather tool.")
    .add_armament(weather)
    .build()?;

Database Query Tool

use sqlx::SqlitePool;

pub struct DatabaseTool {
    pool: SqlitePool,
}

impl DatabaseTool {
    pub async fn new(database_url: &str) -> Result<Self, sqlx::Error> {
        let pool = SqlitePool::connect(database_url).await?;
        Ok(Self { pool })
    }
}

#[async_trait]
impl ArsenalPort for DatabaseTool {
    async fn list_armaments(&self) -> Vec<Armament> {
        vec![
            Armament {
                name: "query_database".to_string(),
                description: "Execute a read-only SQL query".to_string(),
                parameters: serde_json::json!({
                    "type": "object",
                    "properties": {
                        "query": { "type": "string", "description": "SQL SELECT query" }
                    },
                    "required": ["query"]
                }),
                required_params: vec!["query".to_string()],
            },
        ]
    }

    async fn invoke(&self, call: ArmamentCall) -> Result<ArmamentResult, ArsenalError> {
        let query = call.arguments.get("query")
            .and_then(|v| v.as_str())
            .ok_or_else(|| ArsenalError::InvalidArguments("query".to_string()))?;

        // Security: Only allow SELECT queries
        if !query.trim().to_lowercase().starts_with("select") {
            return Ok(ArmamentResult {
                call_id: call.call_id,
                success: false,
                output: None,
                error: Some("Only SELECT queries are allowed".to_string()),
                execution_time_ms: 0,
            });
        }

        let start = std::time::Instant::now();

        let rows = sqlx::query(query)
            .fetch_all(&self.pool)
            .await
            .map_err(|e| ArsenalError::TransportError(e.to_string()))?;

        // Convert rows to JSON
        let result_json = serde_json::to_value(&rows)
            .unwrap_or_else(|_| serde_json::json!([]));

        Ok(ArmamentResult {
            call_id: call.call_id,
            success: true,
            output: Some(result_json),
            error: None,
            execution_time_ms: start.elapsed().as_millis() as u64,
        })
    }

    fn validate_call(&self, call: &ArmamentCall) -> Result<(), ArsenalError> {
        if !call.arguments.contains_key("query") {
            return Err(ArsenalError::InvalidArguments("missing required parameter: query".to_string()));
        }
        Ok(())
    }
}

Tool Result Handling

Automatic Context Injection

When a Paladin invokes a tool, the result is automatically added to the conversation context:

// Paladin execution loop
loop {
    let response = llm.generate(context).await?;

    if let Some(tool_call) = response.tool_calls.first() {
        // Execute tool
        let result = arsenal.invoke(tool_call).await?;

        // Add result to context
        context.add_tool_result(result);

        // Continue reasoning with tool output
        continue;
    }

    // No more tool calls, return final response
    break Ok(response);
}

Custom Result Processing

pub struct LoggingArsenalPort<T: ArsenalPort> {
    inner: T,
}

#[async_trait]
impl<T: ArsenalPort> ArsenalPort for LoggingArsenalPort<T> {
    async fn invoke(&self, call: ArmamentCall) -> Result<ArmamentResult, ArsenalError> {
        println!("Invoking tool: {}", call.tool_name);
        println!("Parameters: {:?}", call.arguments);

        let start = std::time::Instant::now();
        let result = self.inner.invoke(call).await?;
        let duration = start.elapsed();

        println!("Tool completed in {:?}", duration);
        println!("Success: {}", result.success);

        if let Some(error) = &result.error {
            eprintln!("Tool error: {}", error);
        }

        Ok(result)
    }

    // Forward other methods
    async fn list_armaments(&self) -> Vec<Armament> {
        self.inner.list_armaments().await
    }

    fn validate_call(&self, call: &ArmamentCall) -> Result<(), ArsenalError> {
        self.inner.validate_call(call)
    }
}

// Usage
let weather_tool = Arc::new(WeatherTool::new(api_key));
let logged_tool = Arc::new(LoggingArsenalPort { inner: weather_tool });

paladin.add_armament(logged_tool);

Error Handling

match arsenal.invoke(call).await {
    Ok(result) if result.success => {
        // Tool succeeded
        process_result(&result.output);
    }
    Ok(result) => {
        // Tool failed but returned error message
        eprintln!("Tool failed: {}", result.error.unwrap_or_default());
        // Decide: retry, use fallback, or fail
    }
    Err(ArsenalError::ToolNotFound(name)) => {
        eprintln!("Tool not found: {}", name);
        // Handle missing tool
    }
    Err(ArsenalError::Timeout(secs)) => {
        eprintln!("Tool execution timed out after {} seconds", secs);
        // Retry with longer timeout
    }
    Err(e) => {
        eprintln!("Arsenal error: {}", e);
        // Handle other errors
    }
}

Best Practices

1. Clear Tool Descriptions

// ❌ Bad: Vague description
Armament {
    name: "search",
    description: "Search for stuff",
    // ...
}

// ✅ Good: Clear, specific description
Armament {
    name: "web_search",
    description: "Search the web using Google. Returns top 10 results with titles, \
                  URLs, and snippets. Use this when you need current information \
                  not in your training data.",
    // ...
}

2. Validate Inputs

fn validate_call(&self, call: &ArmamentCall) -> Result<(), ArsenalError> {
    // Check required parameters
    for param in &self.required_params {
        if !call.arguments.contains_key(param) {
            return Err(ArsenalError::InvalidArguments(format!("missing required parameter: {param}")));
        }
    }

    // Validate parameter types and values
    if let Some(url) = call.arguments.get("url") {
        if !url.as_str().unwrap_or("").starts_with("http") {
            return Err(ArsenalError::InvalidArguments("url must start with http".into()));
        }
    }

    Ok(())
}

3. Set Timeouts

let tool = CustomTool::new()
    .timeout(Duration::from_secs(30))  // Prevent hanging
    .build()?;

4. Implement Retries for Flaky Operations

async fn invoke_with_retry(&self, call: ArmamentCall) -> Result<ArmamentResult, ArsenalError> {
    let mut attempts = 0;
    let max_attempts = 3;

    loop {
        attempts += 1;

        // ArsenalError has no is_retryable() helper (unlike PaladinError) --
        // match the variants worth retrying explicitly instead.
        let is_retryable = |e: &ArsenalError| {
            matches!(e, ArsenalError::Timeout(_) | ArsenalError::TransportError(_))
        };

        match self.invoke(call.clone()).await {
            Ok(result) => return Ok(result),
            Err(e) if attempts < max_attempts && is_retryable(&e) => {
                tokio::time::sleep(Duration::from_secs(2_u64.pow(attempts))).await;
                continue;
            }
            Err(e) => return Err(e),
        }
    }
}

5. Sanitize Inputs

fn sanitize_sql(query: &str) -> Result<String, ArsenalError> {
    // Remove dangerous keywords
    let dangerous = ["DROP", "DELETE", "UPDATE", "INSERT", "CREATE", "ALTER"];
    let query_upper = query.to_uppercase();

    for keyword in dangerous {
        if query_upper.contains(keyword) {
            return Err(ArsenalError::InvalidArguments(
                format!("Query contains forbidden keyword: {}", keyword)
            ));
        }
    }

    Ok(query.to_string())
}

6. Rate Limiting

use std::sync::Arc;
use tokio::sync::Semaphore;

pub struct RateLimitedTool<T: ArsenalPort> {
    inner: T,
    semaphore: Arc<Semaphore>,
}

impl<T: ArsenalPort> RateLimitedTool<T> {
    pub fn new(inner: T, max_concurrent: usize) -> Self {
        Self {
            inner,
            semaphore: Arc::new(Semaphore::new(max_concurrent)),
        }
    }
}

#[async_trait]
impl<T: ArsenalPort> ArsenalPort for RateLimitedTool<T> {
    async fn invoke(&self, call: ArmamentCall) -> Result<ArmamentResult, ArsenalError> {
        let _permit = self.semaphore.acquire().await
            .map_err(|e| ArsenalError::TransportError(e.to_string()))?;

        self.inner.invoke(call).await
    }

    // Forward other methods...
}

7. Structured Output

// Return structured data that's easy to parse
let output = serde_json::json!({
    "status": "success",
    "data": {
        "temperature": 72.5,
        "conditions": "partly cloudy",
        "humidity": 65
    },
    "timestamp": chrono::Utc::now().to_rfc3339()
});

Ok(ArmamentResult {
    call_id: call.call_id,
    success: true,
    output: Some(output),
    error: None,
    execution_time_ms: 150,
})

8. Model-Level Structured Output (response_format)

The structured ArmamentResult output above is about the shape a tool returns, not the shape the model itself is asked to produce. To constrain a model's own completion to JSON (with an optional JSON Schema), attach a ResponseFormat to the request through LlmRequest::with_response_format — this is a request-level hint, not an Arsenal/tool concept, and it is independent of everything else in this guide.

Native support for response_format varies by provider: some adapters put it on the wire as a real constrained-decoding mode, and at least one ignores it harmlessly because it has no native JSON mode. This guide does not duplicate that per-provider breakdown — see the per-provider table in the agent-runtime user guide for the authoritative list of which adapters honor response_format natively and which fall back to prompt-only instructions.

The Tool-Call Protocol and InProcessArsenal

The reachability note above is about LLM-initiated tool calls; giving an agent a tool it can actually invoke through the reasoning loop is what the Agent Runtime guide's prompt-level tool-call protocol and InProcessArsenal add. InProcessArsenal is a closure-backed ArsenalPort — no MCP server or subprocess required — for registering a Rust closure directly as an Armament, and ToolCallProtocolMiddleware is what makes a shipped provider (which never populates LlmResponse.function_call, per ADR-0042) actually reach that tool: it renders the arsenal's catalogue into the prompt and decodes the model's JSON reply back into a synthesized tool call. See the Agent Runtime guide's Tool-Call Protocol section for the full envelope and the reasoning_agent preset, which wires both together in one call.

Troubleshooting

Tool Not Being Called

Problem: Paladin doesn't use the tool even though it should.

Solutions:

1. Check tool description is clear and relevant
2. Update system prompt to mention tool availability
3. Verify tool appears in list_armaments() output
4. Ensure LLM supports function calling (GPT-4, Claude 3+)

// Make tool usage explicit in system prompt
.system_prompt("You have access to a web_search tool. USE IT to find current information. \
                Always search before answering questions about recent events.")

MCP Server Connection Failed

Problem: Cannot connect to MCP STDIO server.

Solutions:

1. Verify command is in PATH: which uvx
2. Test command manually: uvx mcp-server-fetch
3. Check server logs for errors
4. Verify environment variables are set

// MCPStdioAdapter::new(command, args) takes both up front; there is no
// separate .command()/.args() builder step and no .debug_mode() flag --
// enable verbose logging via the process's own RUST_LOG env var instead.
let tool = MCPStdioAdapter::new("uvx", vec!["mcp-server-fetch"])
    .connect()
    .await?;

Tool Execution Timeout

Problem: Tools timing out frequently.

Solutions:

1. Increase timeout duration
2. Optimize tool implementation
3. Add caching for expensive operations
4. Use async/parallel execution where possible

let tool = CustomTool::new()
    .timeout(Duration::from_secs(120))  // Longer timeout
    .build()?;

Invalid Parameters

Problem: Tool receives wrong parameter types.

Solutions:

1. Strengthen parameter validation
2. Add type coercion in invoke()
3. Improve tool schema definitions
4. Add examples to tool descriptions

// Robust parameter extraction
let count = call.arguments.get("count")
    .and_then(|v| {
        // Try as number, then as string
        v.as_i64()
            .or_else(|| v.as_str().and_then(|s| s.parse::<i64>().ok()))
    })
    .unwrap_or(10);  // Default value

Streamable-HTTP Server Authentication

Problem: Streamable-HTTP server rejects the connection with ArsenalError::AuthFailed (a 401/403-shaped rejection during the initialize handshake).

Solutions:

1. Verify the bearer token is correct and hasn't expired
2. Confirm the token is NOT prefixed with "Bearer " yourself — MCPClient::connect_streamable_http / MCPStreamableHttpAdapter add the prefix internally; a manually-prefixed token double-prefixes and breaks auth
3. If using auth_token_env/--mcp-auth-token-env, confirm the NAMED env var is actually set in the process environment (a missing env var fails loud with an actionable error, not a silent 401)
4. Check the server's own auth requirements (some accept custom headers instead of a bearer token — use MCPStreamableHttpAdapter::with_custom_headers)

use paladin::infrastructure::adapters::arsenal::mcp_streamable_http_adapter::MCPStreamableHttpAdapter;

let adapter = MCPStreamableHttpAdapter::new("https://api.example.com/mcp")
    .with_bearer_token(std::env::var("API_KEY")?); // never a literal token, never "Bearer "-prefixed

let client = adapter.connect().await?;

Testing Tools

Unit Testing Custom Tools

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn test_calculator_add() {
        let calc = CalculatorTool;

        let call = ArmamentCall {
            tool_name: "add".to_string(),
            arguments: HashMap::from([
                ("a".to_string(), json!(5.0)),
                ("b".to_string(), json!(3.0)),
            ]),
            call_id: Uuid::new_v4(),
        };

        let result = calc.invoke(call).await.unwrap();

        assert!(result.success);
        assert_eq!(result.output, Some(json!(8.0)));
    }

    #[tokio::test]
    async fn test_invalid_parameter() {
        let calc = CalculatorTool;

        let call = ArmamentCall {
            tool_name: "add".to_string(),
            arguments: HashMap::from([
                ("a".to_string(), json!(5.0)),
                // Missing 'b' parameter
            ]),
            call_id: Uuid::new_v4(),
        };

        assert!(calc.invoke(call).await.is_err());
    }
}

Integration Testing with Paladin

#[tokio::test]
async fn test_paladin_uses_tool() {
    let llm_adapter = Arc::new(MockLlmAdapter::new());
    let calc = Arc::new(CalculatorTool);

    let paladin = PaladinBuilder::new(llm_adapter)
        .system_prompt("You have a calculator. Use it for math.")
        .add_armament(calc)
        .build()
        .unwrap();

    let response = paladin.execute("What is 15 + 27?").await.unwrap();

    assert!(response.content.contains("42"));
}

Examples

See working examples:

• examples/arsenal_stdio_tools.rs - MCP STDIO integration
• examples/arsenal_streamable_http_tools.rs - MCP Streamable-HTTP integration (authenticated remote transport)

(examples/custom_tools.rs and examples/tool_error_handling.rs, previously cited here, do not exist in the tree -- ls examples/ carries no custom-tool-implementation or error-handling-pattern example beyond what the two files above already cover.)

Next Steps

• Memory Management - Use Garrison with tools
• Battalion Patterns - Tools in multi-agent systems
• API Reference - Arsenal API documentation

Related Resources

• MCP Specification
• MCP Server Examples
• Tool Development Best Practices

Output Formatting Guide

This guide covers the Herald system for formatting and controlling Paladin output in various formats and styles.

Table of Contents

• Overview
• Herald Architecture
• Built-in Formatters
• Custom Formatters
• Streaming Output
• Multi-Format Output
• Post-Processing
• Best Practices
• Advanced Patterns
• Troubleshooting

Overview

The Herald system controls how Paladin output is formatted and presented to users.

Key Capabilities:

• Format Transformation: Convert LLM output to JSON, Markdown, HTML, etc.
• Streaming: Real-time output delivery for better UX
• Validation: Ensure output meets schema requirements
• Post-Processing: Clean, enhance, or transform responses
• Multi-Channel: Different formats for different output destinations

Key Concepts:

• Herald: Output formatting system
• Formatter: Converts raw LLM output to specific format
• OutputFormat: Target format specification (JSON, Markdown, Plain, etc.)
• StreamHandler: Processes output chunks in real-time

Herald Architecture

Core Components

The Herald trait (crates/paladin-core/src/platform/container/herald.rs:49) has seven methods, not three, and format_stream_chunk returns Result<Option<String>, HeraldError> -- None means "buffering, not ready to emit yet", not an error:

// Output format types (paladin_core::platform::container::paladin_config)
pub enum OutputFormat {
    Text,           // Raw text output
    Json,           // Structured JSON
    Structured,     // Structured data output
}

// Herald interface (paladin_core::platform::container::herald)
pub trait Herald: Send + Sync {
    /// Format a complete Paladin execution result
    fn format_paladin_result(&self, result: &PaladinResult) -> Result<String, HeraldError>;

    /// Format a complete Battalion execution result
    fn format_battalion_result(&self, result: &BattalionResult) -> Result<String, HeraldError>;

    /// Format a streaming chunk. `Ok(None)` means "buffering -- not ready to emit yet".
    fn format_stream_chunk(&self, chunk: &StreamChunk) -> Result<Option<String>, HeraldError>;

    /// Finalize streaming output with metadata (tokens, timing) once the stream completes.
    fn finalize_stream(&self, metadata: &ExecutionMetadata) -> Result<String, HeraldError>;

    /// Format an error for display. Infallible -- never returns Err.
    fn format_error(&self, error: &PaladinError) -> String;

    /// Formatter identifier, e.g. "json", "markdown", "table".
    fn name(&self) -> &str;

    /// MIME type of the formatted output, e.g. "application/json".
    fn mime_type(&self) -> &str;
}

Integration with Paladin

let paladin = PaladinBuilder::new(llm_adapter)
    .name("Assistant")
    .system_prompt("You are a helpful assistant.")
    .output_format(OutputFormat::Text)
    .with_herald(Arc::new(MarkdownHerald::default()))
    .build()
    .await?;

let response = paladin.execute("Explain async/await").await?;
// response.content is formatted as Markdown

Built-in Formatters

crates/paladin-herald/src/lib.rs:8-15 ships exactly three formatters: JsonHerald, MarkdownHerald (both always available), and TableHerald (gated behind the table feature, comfy-table; the color feature separately gates ANSI colour in MarkdownHerald, Cargo.toml:19-20). There is no HtmlHerald or CodeHerald -- the "HTML Herald" and "Code Herald" sections previously here described formatters that do not exist anywhere in the tree and have been removed. Each formatter takes a *Config struct via ::new()/::with_config() (or, for TableHerald, ::new(config) directly) -- there is no fluent per-option builder chain (.with_schema(), .with_code_highlighting(), .with_css_framework(), etc. do not exist).

Markdown Herald

Formats output as Markdown, with optional ANSI colour for terminals (color feature) and a configurable heading level.

use paladin_core::platform::container::herald::{Herald, HeraldError};
use paladin::infrastructure::adapters::herald::{MarkdownHerald, markdown_herald::MarkdownHeraldConfig};

let herald = Arc::new(MarkdownHerald::with_config(MarkdownHeraldConfig {
    include_colors: false,
    heading_level: 2,
}));

let paladin = PaladinBuilder::new(llm_adapter)
    .system_prompt("Format all responses as Markdown with proper headers and code blocks.")
    .with_herald(herald)
    .build()
    .await?;

let response = paladin.execute("Explain Rust ownership").await?;
println!("{}", response.content);

Output example:

## Rust Ownership

Ownership is a core concept in Rust that ensures memory safety.

1. Each value has a single owner
2. When the owner goes out of scope, the value is dropped
3. Values can be borrowed immutably or mutably

JSON Herald

Formats output as structured JSON, with pretty-printing and metadata inclusion toggles -- there is no schema-validation option (with_schema()/validate_output() do not exist on JsonHerald; schema conformance is a system-prompt concern, not a Herald one).

use paladin_core::platform::container::herald::{Herald, HeraldError};
use paladin::infrastructure::adapters::herald::{JsonHerald, json_herald::JsonHeraldConfig};

let herald = Arc::new(JsonHerald::with_config(JsonHeraldConfig {
    pretty: true,
    include_metadata: true,
}));

let paladin = PaladinBuilder::new(llm_adapter)
    .system_prompt("Always respond in JSON format matching this schema: \
                    {summary: string, key_points: string[], confidence: number}")
    .with_herald(herald)
    .build()
    .await?;

let response = paladin.execute("Analyze sentiment of: 'This product is amazing!'").await?;

// Parse structured output
let json: serde_json::Value = serde_json::from_str(&response.content)?;
println!("Summary: {}", json["summary"]);
println!("Key points: {:?}", json["key_points"]);

Output example (the Herald's own JSON envelope around a Paladin result, per crates/paladin-herald/src/json_herald.rs -- output/usage/execution_time_ms/ loop_count/stop_reason, plus a metadata.timestamp when include_metadata is true). usage is the full TokenUsage split: a stable six-key object where the three cache/reasoning sub-counts are null when the provider did not report them (ACCT-04, D-21):

{
  "output": "Highly positive sentiment expressing enthusiasm...",
  "usage": {
    "prompt_tokens": 30,
    "completion_tokens": 12,
    "total_tokens": 42,
    "cache_read_tokens": null,
    "cache_write_tokens": null,
    "reasoning_tokens": null
  },
  "execution_time_ms": 850,
  "loop_count": 1,
  "stop_reason": "Completed",
  "metadata": {
    "timestamp": "2026-08-24T00:00:00Z"
  }
}

Table Herald

Formats a BattalionResult's per-Paladin results as an ASCII table -- suitable for terminal output. Requires the table feature (comfy-table).

use paladin::infrastructure::adapters::herald::{TableHerald, table_herald::TableHeraldConfig};

let herald = Arc::new(TableHerald::new(TableHeraldConfig {
    max_column_width: 60,
    border_style: "rounded".to_string(),
}));

Custom Formatters

Create custom heralds for specialized output formats by implementing the Herald trait directly (crates/paladin-core/src/platform/container/herald.rs:49) -- no async_trait needed, every method is synchronous. Herald has seven required methods, not the two or three shown in older sketches below: format_paladin_result, format_battalion_result, format_stream_chunk (returns Result<Option<String>, HeraldError> -- Ok(None) means "still buffering"), finalize_stream, format_error (infallible), name, and mime_type. The rest of this section's custom-Herald sketches (XmlHerald, CsvHerald, and the streaming/post-processing/advanced-pattern examples further down) illustrate formatting logic against this real trait shape but, per D-12's no-restructure scope, were not individually rewritten to implement every required method -- treat their partial impl Herald for ... blocks as illustrating one method's body, not a complete, compiling implementation.