Paladin Documentation

Welcome to the Paladin documentation! Paladin is a Rust-based enterprise multi-agent orchestration framework built with Hexagonal Architecture and Domain-Driven Design principles.

🚀 Getting Started

New to Paladin? Start here:

1. Quickstart Guide - Get your first Paladin agent running in 15 minutes
2. Installation - Detailed setup instructions for all platforms
3. Examples Gallery - Working code examples for common use cases

📚 User Guides

Learn how to build with Paladin:

• Autonomous Agent Features - Auto-planning, prompt generation, dynamic temperature, and agent handoffs
• Battalion Orchestration - Multi-agent coordination with orchestration patterns
• Maneuver Flow DSL - Declarative workflows with Flow DSL syntax
• Tool Integration (Arsenal) - Integrate external tools via MCP protocol
• Memory Management (Garrison) - Conversation context and persistence
• Output Formatting (Herald) - Format and stream agent responses
• CLI Usage Guide - Complete command-line interface reference

🏗️ Architecture

Understand Paladin's design:

• Architecture Overview - Three-layer hexagonal architecture
• Hexagonal Design - Port/adapter pattern implementation
• Domain Model - DDD entities and relationships
• Design Patterns - Patterns used throughout Paladin

🚢 Deployment

Deploy Paladin to production:

• Docker - Containerized deployment
• Kubernetes - Cloud-native orchestration
• CI/CD - Automated pipelines with GitHub Actions
• Production Best Practices - Security, scaling, and reliability
• Versioning Policy - Lockstep versioning rules and transition criteria
• Release Checklist - Dependency-aware release and publish workflow

🔧 Operations

Monitor and maintain Paladin:

• Logging - Structured logging configuration
• Monitoring - Metrics and dashboards
• Troubleshooting - Common issues and solutions
• Performance Tuning - Optimize for throughput and latency

🤝 Contributing

Extend and improve Paladin:

• Contribution Guide - How to contribute
• Adapter Development - Create custom adapters
• Testing Guide - Testing requirements and patterns

📖 API Reference

Comprehensive API documentation is available via rustdoc:

cargo doc --open

Or browse online at: https://docs.rs/paladin (when published)

🎯 Key Concepts

Medieval Military Theme

Paladin uses a consistent Medieval Military naming convention:

Architecture Layers

Paladin follows hexagonal (ports and adapters) architecture:

• Core Layer - Pure domain logic, no external dependencies
• Application Layer - Use cases and port definitions (interfaces)
• Infrastructure Layer - Adapter implementations for external systems

Dependencies flow inward only: Infrastructure → Application → Core

💡 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: You're reading it!

📄 License

See LICENSE for details.

Installation

This guide covers adding Paladin to an existing Rust project or setting up the Paladin workspace for development.

Prerequisites

Required

Why Rust >= 1.85? Paladin uses edition 2024 features. Verify your toolchain:

rustc --version   # should print >= 1.85.0

Update with rustup update stable.

Optional (for Docker-based services)

• Docker + Docker Compose v2 -- required for the built-in Redis, MinIO, and MySQL services (see Docker Guide)

Installing Rust

# Install rustup and the stable toolchain
curl --proto =https --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env

# Build tools -- Linux (Ubuntu/Debian)
sudo apt-get install -y build-essential pkg-config libssl-dev

# Build tools -- macOS (via Homebrew)
brew install openssl pkg-config

Windows users should use rustup-init.exe or WSL 2.

Adding Paladin to a Rust Project

Cargo.toml -- choose your crates

Paladin v0.5.0 is published as a workspace of focused crates. Add only what you need:

[dependencies]
# Core framework -- always required
paladin-ai-core   = "0.5.0"
paladin-ports     = "0.5.0"

# LLM providers (pick one or more)
paladin-llm       = { version = "0.5.0", features = ["llm-openai"] }

# Multi-agent orchestration (optional)
paladin-battalion = "0.5.0"

# Memory / Garrison (optional)
paladin-memory    = "0.5.0"

# Storage adapters (optional)
paladin-storage   = "0.5.0"

# Async runtime (required)
tokio = { version = "1", features = ["full"] }

Umbrella crate

The paladin-ai umbrella crate (v0.5.0) re-exports everything and accepts workspace feature flags:

[dependencies]
paladin-ai = { version = "0.5.0", features = ["redis-queue", "s3-storage"] }
tokio      = { version = "1", features = ["full"] }

Feature Flag Profiles

Verification

cargo check

No errors means all selected features resolved correctly.

Cloning the Source for Development

# 1. Clone
git clone https://github.com/DF3NDR/paladin-dev-env.git
cd paladin-dev-env

# 2. Build the workspace
cargo build

# 3. Run unit tests
cargo test --workspace --lib

# 4. (Optional) Start backing services
make services-up   # Redis, MinIO, MySQL via Docker Compose

See Development Setup for the full contributor workflow.

Environment Variables for LLM Keys

Paladin reads API keys exclusively from environment variables -- never put keys in config files.

# Set at least one provider key before running
export OPENAI_API_KEY="sk-..."       # OpenAI
export DEEPSEEK_API_KEY="sk-..."     # DeepSeek
export ANTHROPIC_API_KEY="sk-..."    # Anthropic

Copy .env.example to .env for local development (.env is git-ignored).

Next Steps

• Quickstart -- write your first Paladin agent in minutes
• Configuration -- full config.yml schema reference
• User Guides -- in-depth agent patterns

Quickstart

Get a Paladin agent running in under five minutes.

Prerequisites

Complete Installation first and set your LLM API key:

export OPENAI_API_KEY="sk-..."

Create a New Project

cargo new my-paladin-agent
cd my-paladin-agent

Add Paladin to Cargo.toml:

[dependencies]
paladin-ai-core   = "0.5.0"
paladin-ports     = "0.5.0"
paladin-llm       = { version = "0.5.0", features = ["llm-openai"] }
tokio             = { version = "1", features = ["full"] }

Your First Paladin Agent

Replace src/main.rs with the following:

// src/main.rs -- Hello, Paladin!
use paladin_ai_core::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin_ai_core::application::services::paladin::paladin_execution_service::PaladinExecutionService;
use paladin_ports::output::llm_port::LlmPort;
use paladin_llm::openai::OpenAIAdapter;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // 1. Create an LLM adapter (reads OPENAI_API_KEY from env)
    let llm_port: Arc<dyn LlmPort> = Arc::new(OpenAIAdapter::from_env()?);

    // 2. Build the Paladin using the fluent builder
    let paladin = PaladinBuilder::new(llm_port.clone())
        .system_prompt("You are a concise and helpful assistant.")
        .name("HelloPaladin")
        .model("gpt-4")
        .temperature(0.7)
        .max_loops(1)
        .build()
        .await?;

    // 3. Create an execution service
    let service = PaladinExecutionService::new(llm_port, Default::default(), None, None);

    // 4. Execute with a prompt
    let result = service.execute(&paladin, "Say hello in one sentence.").await?;

    println!("Output : {}", result.output);
    println!("Tokens : {}", result.token_count);
    println!("Time   : {}ms", result.execution_time_ms);

    Ok(())
}

Run it:

cargo run

Expected output (exact wording varies):

Output : Hello! I am your AI assistant, ready to help.
Tokens : 18
Time   : 342ms

Running the Built-in Examples

The Paladin workspace ships with ready-to-run examples:

# Clone the workspace if you haven't already
git clone https://github.com/DF3NDR/paladin-dev-env.git
cd paladin-dev-env

# Start backing services (Redis, MinIO) -- optional for basic examples
make services-up

# Run the basic Paladin example
cargo run --example basic_paladin

# Sequential multi-agent pipeline
cargo run --example formation_sequential

# Concurrent multi-agent execution
# cargo run --example phalanx_concurrent

Understanding the Output

PaladinExecutionService::execute returns a PaladinResult with these fields:

What's Next?

Configuration

Paladin is configured via a YAML file (config.yml by default) and environment variables. Environment variables take precedence over file values and use the APP_ prefix format shown throughout this guide.

Loading Configuration

// Load from the default config.yml in the current directory
let settings = paladin_ai_core::config::ApplicationSettings::load()?;

// Or specify a path
let settings = paladin_ai_core::config::ApplicationSettings::from_file("config.yml")?;

LLM Provider

llm:
  default_provider: "openai"   # openai | deepseek | anthropic

  openai:
    base_url: "https://api.openai.com/v1"
    default_model: "gpt-4"
    default_temperature: 0.7
    timeout_seconds: 300
    max_retries: 3

  deepseek:
    base_url: "https://api.deepseek.com/v1"
    default_model: "deepseek-chat"
    default_temperature: 0.7
    timeout_seconds: 300
    max_retries: 3

  anthropic:
    base_url: "https://api.anthropic.com/v1"
    default_model: "claude-3-5-sonnet-20241022"
    default_temperature: 0.7
    timeout_seconds: 300
    max_retries: 3

API keys are read exclusively from environment variables:

Security: Never put API keys in config.yml. Use environment variables or a secrets manager (AWS Secrets Manager, HashiCorp Vault, Kubernetes Secrets).

Garrison (Short-term Memory)

The Garrison stores conversation context between Paladin turns.

garrison:
  garrison_type: "in_memory"       # in_memory | sqlite
  # path: "./garrison.db"          # Required when garrison_type = "sqlite"
  max_entries: 100                  # Max conversation turns to retain
  max_tokens: 4000                  # Context-window token budget
  tokenizer: "gpt-4"               # Model name for token counting
  eviction_strategy: "importance_based"  # importance_based | fifo | sliding_window
  preserve_recent_count: 10        # Always keep at least N recent entries

Env vars: APP_GARRISON_TYPE, APP_GARRISON_PATH, APP_GARRISON_MAX_ENTRIES, APP_GARRISON_MAX_TOKENS, APP_GARRISON_EVICTION_STRATEGY, APP_GARRISON_PRESERVE_RECENT_COUNT

Sanctum (Long-term Vector Memory)

Sanctum stores semantic memories in a vector database for RAG.

sanctum:
  enabled: false
  adapter_type: "in_memory"        # in_memory | qdrant

  qdrant:                          # Required when adapter_type = "qdrant"
    url: "http://localhost:6334"
    collection_name: "paladin_memories"
    vector_dimension: 1536         # Must match your embedding model

rag:
  top_k: 5                         # Results to retrieve
  min_similarity: 0.7              # Score threshold (0.0-1.0)
  max_tokens: 2000                 # Max tokens to inject from RAG
  timeout_seconds: 5

memory_extraction:
  enabled: true
  strategy: "on_completion"        # every_turn | on_completion | manual

Env vars: APP_SANCTUM_ENABLED, APP_SANCTUM_ADAPTER_TYPE, APP_SANCTUM_QDRANT_URL, APP_SANCTUM_QDRANT_COLLECTION_NAME, APP_SANCTUM_QDRANT_VECTOR_DIMENSION

See Sanctum Vector Memory for detail.

Arsenal (Tool System / MCP)

The Arsenal connects Paladins to external tools via the Model Context Protocol.

arsenal:
  default_timeout_seconds: 30
  max_concurrent_tools: 5
  mcp_servers:
    # STDIO server (command-line process)
    - name: "web_search"
      server_type: "stdio"
      command: "uvx"
      args: ["mcp-web-search"]

    # SSE server (HTTP-based)
    - name: "code_analyzer"
      server_type: "sse"
      endpoint: "http://localhost:8080/mcp"

Env vars: APP_ARSENAL_DEFAULT_TIMEOUT_SECONDS, APP_ARSENAL_MAX_CONCURRENT_TOOLS

See Arsenal & Tools for full integration guide.

Citadel (State Persistence)

Citadel saves Paladin state to disk for crash recovery and resumption.

citadel:
  enabled: false
  state_dir: "./paladin-states"
  autosave_enabled: false          # Save state after each execution
  cleanup_enabled: false           # Delete old state files automatically
  max_state_age_days: 30

Env vars: APP_CITADEL_ENABLED, APP_CITADEL_STATE_DIR, APP_CITADEL_AUTOSAVE_ENABLED, APP_CITADEL_CLEANUP_ENABLED, APP_CITADEL_MAX_STATE_AGE_DAYS

Battalion (Multi-agent Orchestration)

battalion:
  default_timeout_seconds: 300     # Per-battalion execution timeout
  error_strategy: "fail_fast"      # fail_fast | continue_on_error | retry_then_continue
  max_concurrent_paladins: 10      # Phalanx concurrency limit
  metadata_output_enabled: false   # Write execution metadata to files

  retry:                           # Used when error_strategy = retry_then_continue
    max_attempts: 3
    exponential_backoff: true
    jitter: true
    base_delay_ms: 100
    max_delay_seconds: 10

  maneuver:                        # Flow DSL (Maneuver pattern)
    error_strategy: "fail_fast"    # fail_fast | continue_parallel | ignore_errors
    output_format: "combined_text" # combined_text | structured_json
    pass_output_as_input: true
    timeout_seconds: 300
    collect_timing_metrics: true
    max_agents: 30
    max_depth: 5

Env vars: APP_BATTALION_DEFAULT_TIMEOUT_SECONDS, APP_BATTALION_ERROR_STRATEGY, APP_BATTALION_MAX_CONCURRENT_PALADINS, APP_BATTALION_RETRY_MAX_ATTEMPTS, etc.

See Battalion Patterns for Formation, Phalanx, Campaign, and Chain of Command details.

Herald (Output Formatting)

herald:
  default_formatter: "json"        # json | markdown | table

  json:
    pretty: true
    include_metadata: true

  markdown:
    include_colors: true
    heading_level: 2

  table:
    max_column_width: 60
    border_style: "rounded"        # ascii | rounded | modern | sharp | none

Env vars: APP_HERALD_DEFAULT_FORMATTER, APP_HERALD_JSON_PRETTY, APP_HERALD_MARKDOWN_INCLUDE_COLORS, APP_HERALD_TABLE_BORDER_STYLE

Autonomous Features

All autonomous features are opt-in (disabled by default). Uncomment sections in config.yml to enable:

autonomous:
  planning:
    enabled: false          # Decompose complex tasks into subtasks
    max_subtasks: 10

  prompt_generation:
    enabled: false          # Auto-generate system prompts from description
    description: null       # e.g. "Expert data analyst"

  dynamic_temperature:
    enabled: false          # Adjust temperature per task type
    min: 0.1
    max: 0.9

  handoffs:
    enabled: false          # Delegate to specialist Paladins
    strategy: "automatic"   # automatic | explicit | {threshold: 0.8}
    max_depth: 5

Env vars: APP_AUTONOMOUS_PLANNING_ENABLED, APP_AUTONOMOUS_PLANNING_MAX_SUBTASKS, APP_AUTONOMOUS_PROMPT_GENERATION_ENABLED, APP_AUTONOMOUS_DYNAMIC_TEMPERATURE_ENABLED, APP_AUTONOMOUS_HANDOFFS_ENABLED, APP_AUTONOMOUS_HANDOFFS_STRATEGY

Multi-Environment Pattern

Keep a config.yml for defaults and override per environment:

# Development
export APP_LLM_DEFAULT_PROVIDER=openai
export APP_GARRISON_TYPE=in_memory

# Staging
export APP_GARRISON_TYPE=sqlite
export APP_GARRISON_PATH=/data/garrison.db
export APP_SANCTUM_ENABLED=true

# Production
export APP_GARRISON_TYPE=sqlite
export APP_SANCTUM_ENABLED=true
export APP_SANCTUM_ADAPTER_TYPE=qdrant
export APP_CITADEL_ENABLED=true
export APP_CITADEL_AUTOSAVE_ENABLED=true

Complete Example (config.yml)

llm:
  default_provider: "openai"
  openai:
    default_model: "gpt-4"
    default_temperature: 0.7

garrison:
  garrison_type: "sqlite"
  path: "./garrison.db"
  max_entries: 200
  max_tokens: 8000

arsenal:
  default_timeout_seconds: 30
  max_concurrent_tools: 5
  mcp_servers:
    - name: "web_search"
      server_type: "stdio"
      command: "uvx"
      args: ["mcp-web-search"]

battalion:
  error_strategy: "retry_then_continue"
  max_concurrent_paladins: 10
  retry:
    max_attempts: 3
    exponential_backoff: true

herald:
  default_formatter: "markdown"

See Also

• Installation
• Quickstart
• Garrison Memory
• Arsenal & Tools
• Sanctum Vector Memory
• Battalion Patterns

Paladin Agents

A Paladin is Paladin AI's core autonomous agent entity — an LLM-powered reasoner that operates a configurable reasoning loop, maintains conversation memory via Garrison, executes external tools via Arsenal, and optionally leverages autonomous features like task planning, auto-generated prompts, and dynamic temperature.

Ready to run a number of agents? See Deployment Topologies for how to choose between embedding, hosting, queue/worker, and sidecar models.

Table of Contents

1. Quick Start
2. PaladinBuilder API
3. Execution Model
4. PaladinResult Fields
5. StopReason Variants
6. Autonomous Features
7. Memory — Garrison
8. Tools — Arsenal
9. Output Formatting — Herald
10. Configuration Reference
11. Error Handling
12. Best Practices

Quick Start

Add the paladin-ai crate and enable any desired feature flags:

[dependencies]
paladin-ai = { version = "0.5.0", features = ["llm-openai"] }
tokio = { version = "1", features = ["full"] }

Build and execute a Paladin:

use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin_ports::output::llm_port::LlmPort;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Construct an LLM adapter (e.g., OpenAI)
    let llm_port: Arc<dyn LlmPort> = Arc::new(openai_adapter());

    // Build the Paladin
    let paladin = PaladinBuilder::new(llm_port)
        .system_prompt("You are a helpful assistant.")
        .name("Assistant")
        .model("gpt-4o")
        .temperature(0.7)
        .max_loops(3)
        .timeout_seconds(120)
        .build()
        .await?;

    // Execute
    let result = paladin
        .execute("Explain the Rust ownership model in one paragraph.")
        .await?;

    println!("{}", result.output);
    println!("Tokens used: {}", result.token_count);
    println!("Stop reason: {:?}", result.stop_reason);
    Ok(())
}

PaladinBuilder API

PaladinBuilder is located at src/application/services/paladin/paladin_builder.rs. All methods are fluent (return Self). Call .build().await? at the end.

Core Configuration

Integrations

Autonomous Features

Execution Model

A Paladin's inner reasoning loop:

┌─────────────────────────────────────────────────────────────────┐
│  1. Build Prompt                                                 │
│     System prompt + Garrison history + User input               │
├─────────────────────────────────────────────────────────────────┤
│  2. LLM Call (via LlmPort)                                       │
│     Generate response from the configured model                  │
├─────────────────────────────────────────────────────────────────┤
│  3. Check Stop Conditions                                        │
│     • Stop word detected in output?  → StopWord(word)           │
│     • loop_count ≥ max_loops?        → MaxLoops                 │
│     • Elapsed > timeout?             → Timeout                  │
├─────────────────────────────────────────────────────────────────┤
│  4. Tool Execution (if Arsenal attached)                        │
│     Parse tool-call JSON in response → invoke via ArsenalPort   │
│     Append tool result to context                                │
├─────────────────────────────────────────────────────────────────┤
│  5. Update Garrison                                              │
│     Store assistant turn and any tool results                    │
├─────────────────────────────────────────────────────────────────┤
│  6. Loop or Complete                                             │
│     If no stop condition: loop_count++ → back to step 1         │
│     Otherwise: build PaladinResult and return                   │
└─────────────────────────────────────────────────────────────────┘

PaladinResult Fields

Returned by execute() and streamed by execute_stream().

Check completeness:

if result.stop_reason.is_successful() {
    println!("Complete output: {}", result.output);
} else {
    println!("Partial output ({}): {}", result.stop_reason, result.output);
}

StopReason Variants

Autonomous Features

All autonomous features are opt-in (disabled by default) to maintain backward compatibility.

Autonomous Planning (MaxLoops::Auto)

When enabled, the Paladin uses an LLM call to decompose the user's task into subtasks before executing them sequentially.

use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin::core::platform::container::paladin::MaxLoops;

let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are a research assistant.")
    .enable_autonomous_planning(true)
    // max_loops controls the subtask cap when using auto planning:
    .max_loops(10)
    .build()
    .await?;

The PaladinResult.plan field contains the TaskPlan with each subtask's description and result.

Auto-Generated System Prompts

Instead of writing a system prompt manually, provide an agent description and let the LLM generate an optimized prompt:

let paladin = PaladinBuilder::new(llm_port)
    .agent_description("Expert in Rust async programming and tokio runtime")
    .enable_autonomous_prompts(true)
    .build()
    .await?;

Tip: Calling .system_prompt(...) on the same builder disables auto-generation for that instance — the manual prompt always takes precedence.

Dynamic Temperature

Temperature increases linearly from the configured base value toward 1.0 over the reasoning loops. This encourages broader exploration in later iterations when the agent may be stuck:

let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are a problem solver.")
    .temperature(0.3)          // Start temperature
    .max_loops(5)
    .enable_dynamic_temperature(true)  // Reaches ~1.0 by loop 5
    .build()
    .await?;

Agent Handoffs

A Paladin can delegate sub-tasks to specialist agents at runtime using the Arsenal handoff tool. Register specialist agents on the builder:

let paladin = PaladinBuilder::new(llm_port.clone())
    .system_prompt("Routing coordinator. Delegate to specialists.")
    .with_specialist(Arc::new(code_reviewer_paladin))
    .with_specialist(Arc::new(security_auditor_paladin))
    .build()
    .await?;

Delegation records appear in PaladinResult.handoff_history.

Memory — Garrison

Attach a Garrison adapter to give the Paladin persistent conversation memory.

use paladin_memory::garrison::in_memory_garrison::InMemoryGarrison;
use paladin_ports::output::garrison_port::GarrisonPort;
use std::sync::Arc;

let garrison: Arc<dyn GarrisonPort> = Arc::new(InMemoryGarrison::new());

let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are a memory-enabled assistant.")
    .with_garrison(garrison)
    .build()
    .await?;

Available Garrison adapters (in crates/paladin-memory/):

See Garrison Memory for full documentation.

Tools — Arsenal

Attach an Arsenal registry backed by MCP (Model Context Protocol) servers:

use paladin_ports::output::arsenal_port::ArsenalRegistry;

// Registry pre-loaded from config.yml arsenal.mcp_servers section
let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are a web researcher with tool access.")
    .with_arsenal_registry(arsenal_registry)
    .build()
    .await?;

See Arsenal Tools for MCP server configuration and custom tool implementation.

Output Formatting — Herald

Format execution results using a Herald adapter:

use paladin::infrastructure::adapters::herald::JsonHerald;
use paladin_core::platform::container::herald::Herald;

let herald: Arc<dyn Herald> = Arc::new(JsonHerald::default());

let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are an API assistant.")
    .with_herald(herald)
    .build()
    .await?;

See Herald Output for available formatters.

Configuration Reference

All builder values can also be set through config.yml:

paladin:
  default_model: "gpt-4o"
  default_temperature: 0.7
  default_max_loops: 3
  timeout_seconds: 300
  retry_attempts: 3

autonomous:
  planning:
    enabled: false
    max_subtasks: 10
  prompt_generation:
    enabled: false
  dynamic_temperature:
    enabled: false
  handoffs:
    enabled: false
    max_depth: 3

See Configuration for the full schema.

Error Handling

PaladinError variants from paladin_core::platform::container::paladin_error:

Best Practices

• Always set a system prompt that clearly defines the agent's role and constraints.
• Set timeout_seconds appropriate for your task; defaults to 300s.
• Use add_stop_word for structured output tasks so the agent knows when it is done.
• Enable Garrison for any multi-turn conversation to maintain context.
• Check stop_reason.is_successful() before consuming result.output in production.
• Prefer execute_stream() for tasks > 30s so the caller can render output incrementally.
• Use autonomous features sparingly — they add LLM overhead; profile before enabling in loops.

Battalion Orchestration Patterns

The Battalion system in crates/paladin-battalion/ coordinates multiple Paladin agents through eight distinct execution patterns, plus the Commander strategy router that can select a pattern automatically.

Table of Contents

1. Overview
2. Quick Start
3. The Eight Patterns
   • Formation — Sequential
   • Phalanx — Concurrent
   • Campaign — Graph/DAG
   • Chain of Command — Hierarchical
   • Conclave — Mixture of Experts
   • Council — Collaborative Discussion
   • Grove — Semantic Routing
   • Maneuver — Flow DSL
4. Commander — Strategy Router
5. Error Handling
6. Performance Notes
7. Best Practices

Overview

All services require only Arc<dyn PaladinPort> (from paladin-ports) — they never import LLM provider libraries directly.

Quick Start

[dependencies]
paladin-ai = { version = "0.5.0", features = ["llm-openai"] }
tokio = { version = "1", features = ["full"] }

use paladin_battalion::formation_service::FormationExecutionService;
use paladin_core::platform::container::battalion::formation::Formation;
use paladin_core::platform::container::battalion::BattalionConfig;
use std::sync::Arc;

// Each Paladin is built with PaladinBuilder (see paladin-agents.md)
let paladins = vec![analyzer, processor, summarizer];
let config = BattalionConfig::default();
let formation = Formation::new(paladins, config)?;

let service = FormationExecutionService::new(paladin_port);
let result = service.execute(&formation, "Analyze the Q3 earnings report").await?;
println!("{}", result.output);

The Eight Patterns

Formation — Sequential

Source: crates/paladin-battalion/src/formation_service.rs

Output from each Paladin feeds the input of the next. Ideal for multi-step data transformation pipelines.

use paladin_battalion::formation_service::FormationExecutionService;
use paladin_core::platform::container::battalion::formation::Formation;

let formation = Formation::new(vec![extractor, analyzer, writer], config)?;
let service = FormationExecutionService::new(paladin_port);
let result = service.execute(&formation, "Raw data...").await?;

Configuration keys: sequential timeout, error strategy.

Phalanx — Concurrent

Source: crates/paladin-battalion/src/phalanx_service.rs

All Paladins receive the same input and execute concurrently via tokio tasks. Results are aggregated according to the AggregationStrategy.

use paladin_battalion::phalanx_service::PhalanxExecutionService;
use paladin_core::platform::container::battalion::phalanx::{AggregationStrategy, Phalanx};

let phalanx = Phalanx::new(
    vec![security_auditor, performance_analyst, style_checker],
    AggregationStrategy::Concatenate,
    config,
)?;

let service = PhalanxExecutionService::new(paladin_port);
let result = service.execute(&phalanx, "Review this Rust code...").await?;

AggregationStrategy variants: Concatenate, FirstSuccess, Majority, Custom.

Concurrency is bounded by a tokio::sync::Semaphore (configurable via max_concurrency in BattalionConfig).

Campaign — Graph/DAG

Source: crates/paladin-battalion/src/campaign_service.rs

Paladins are arranged in a directed acyclic graph. Execution is topologically sorted so upstream agents complete before downstream agents begin.

use paladin_battalion::campaign_service::CampaignExecutionService;
use paladin_core::platform::container::battalion::campaign::Campaign;

let campaign = Campaign::builder()
    .add_node("ingest", ingest_paladin)
    .add_node("analyze", analyze_paladin)
    .add_node("report", report_paladin)
    .add_edge("ingest", "analyze")
    .add_edge("analyze", "report")
    .config(config)
    .build()?;

let service = CampaignExecutionService::new(paladin_port);
let result = service.execute(&campaign, "Start").await?;

Independent branches execute concurrently; the service enforces dependency order.

Chain of Command — Hierarchical

Source: crates/paladin-battalion/src/chain_of_command_service.rs

A commander Paladin decomposes the task and routes sub-tasks to specialist Paladins, then synthesizes their outputs.

use paladin_battalion::chain_of_command_service::ChainOfCommandExecutionService;
use paladin_core::platform::container::battalion::chain_of_command::ChainOfCommand;

let chain = ChainOfCommand::new(
    commander_paladin,
    vec![backend_dev, frontend_dev, qa_engineer],
    config,
)?;

let service = ChainOfCommandExecutionService::new(paladin_port);
let result = service.execute(&chain, "Build a login feature").await?;

Conclave — Mixture of Experts

Source: crates/paladin-battalion/src/conclave_execution_service.rs

Multiple expert Paladins process the same task in parallel; an aggregator Paladin synthesizes their outputs into a final response.

use paladin_battalion::conclave_execution_service::ConclaveExecutionService;
use paladin_core::platform::container::battalion::conclave::Conclave;

let conclave = Conclave::new(
    vec![legal_expert, technical_expert, business_expert],
    synthesis_paladin,
    config,
)?;

let service = ConclaveExecutionService::new(paladin_port);
let result = service.execute(&conclave, "Should we adopt microservices?").await?;
// result.aggregated_output contains the synthesized response
// result.successful_expert_count() shows how many experts contributed

Council — Collaborative Discussion

Source: crates/paladin-battalion/src/council_service.rs

Paladins take turns responding to each other in a structured discussion, building toward a shared conclusion or consensus.

use paladin_battalion::council_service::CouncilService;
use paladin_core::platform::container::battalion::council::Council;

let council = Council::new(
    vec![optimist_paladin, skeptic_paladin, moderator_paladin],
    config,  // includes discussion_rounds
)?;

let service = CouncilService::new(paladin_port);
let result = service.execute(&council, "Evaluate adopting async Rust").await?;

Grove — Semantic Routing

Source: crates/paladin-battalion/src/grove_service.rs

The Grove routes the input to the most semantically appropriate Paladin from the registered specialists, using LLM-based capability matching.

use paladin_battalion::grove_service::GroveExecutionService;
use paladin_core::platform::container::battalion::grove::Grove;

let grove = Grove::new(
    vec![python_expert, rust_expert, go_expert],
    config,
)?;

let service = GroveExecutionService::new(paladin_port);
let result = service.execute(&grove, "Help me with Rust lifetimes").await?;
// Routes to rust_expert automatically

Maneuver — Flow DSL

Source: crates/paladin-battalion/src/maneuver/

Maneuver is a declarative flow DSL that lets you compose multiple Battalion patterns in a single workflow definition. See Maneuver Flow DSL for full syntax and examples.

Commander — Strategy Router

Source: crates/paladin-battalion/src/commander.rs

The Commander provides a single entry-point that automatically selects the optimal pattern based on input analysis and the number/capabilities of Paladins provided.

use paladin_battalion::commander::Commander;
use paladin_core::platform::container::battalion::{BattalionConfig, BattalionStrategy};

let commander = Commander::new(paladin_port, paladin_registry);

// Auto-select strategy
let result = commander
    .execute(paladins, "Analyze and summarize this report", BattalionStrategy::Auto, config)
    .await?;

// Or force a specific strategy
let result = commander
    .execute(paladins, "Run in parallel", BattalionStrategy::Phalanx, config)
    .await?;

Auto Mode Heuristics

Error Handling

All services use ErrorStrategy from paladin_core::platform::container::battalion:

use paladin_core::platform::container::battalion::{BattalionConfig, ErrorStrategy};

let config = BattalionConfig {
    error_strategy: ErrorStrategy::ContinueOnError,
    max_concurrency: Some(4),
    timeout_seconds: 120,
    ..Default::default()
};

BattalionResult fields: output: String, paladin_results: Vec<PaladinResult>, status: BattalionStatus, execution_time_ms: u64, token_usage: TokenUsage.

Performance Notes

• Phalanx concurrency is capped by BattalionConfig::max_concurrency (default: unbounded). Set this to avoid overloading upstream LLM rate limits.
• Formation adds one LLM call per Paladin sequentially — keep chains short (<6) for latency-sensitive workloads.
• Campaign parallelises independent branches automatically; no manual coordination needed.
• The Commander auto-router adds a small analysis overhead (~50ms); negligible for most tasks.

Best Practices

• Formation: Keep the chain ≤5 agents and design each stage to produce clean hand-off text.
• Phalanx: Set max_concurrency to stay within LLM provider rate limits.
• Campaign: Validate your DAG has no cycles before deployment (Campaign::build() checks this).
• Conclave: Ensure expert agents have distinct, non-overlapping system prompts for better synthesis.
• Council: Include a moderator Paladin to keep discussions on track.
• Grove: Write precise capability descriptions in each Paladin's agent_description field.
• Commander Auto: Test your routing decisions with representative inputs before production.

Orchestration

The Battalion runtime in crates/paladin-battalion/ coordinates multiple Paladin agents through a family of orchestration patterns, a strategy router (Commander), a cron-style job scheduler, and an event/trigger system. This guide is the comprehensive reference for choosing a pattern and wiring it up.

For a quick pattern-by-pattern cheat sheet see Battalion Patterns; for the declarative flow language see Maneuver Flow DSL; for how agents and workflows call each other see the Agent ↔ Orchestrator Bridge. For how a Battalion fits among the ways to run agents, see Deployment Topologies.

Every code example targets the current v0.5.0 workspace. The substantive examples are real, compiled code pulled from the paladin-doc-examples crate via mdBook {{#include}}, so they are checked against the live API; a few illustrative fragments are marked rust,ignore. The API forms are verified against crates/paladin-battalion/ and crates/paladin-ports/.

Table of Contents

1. Workflow Patterns Overview
2. Formation — Sequential
3. Phalanx — Parallel
4. Campaign — Graph / DAG
5. Chain of Command — Hierarchical
6. Commander — Dynamic Strategy Routing
7. Job Scheduling
8. Event and Trigger System
9. Configuration Reference
10. See Also

Workflow Patterns Overview

All orchestration services depend only on Arc<dyn PaladinPort> (from paladin-ports) — they never import an LLM provider crate directly. Pick a pattern by the shape of the work:

Conclave (mixture-of-experts), Council (turn-taking discussion), and Grove (semantic routing) are additional patterns documented in Battalion Patterns. The declarative Maneuver flow DSL has its own guide: Maneuver Flow DSL.

Decision Flowchart

flowchart TD
    start([Have a task + several Paladins]) --> q1{One fixed order of steps?}
    q1 -->|Yes| formation[Formation — sequential]
    q1 -->|No| q2{Steps independent, run together?}
    q2 -->|Yes| phalanx[Phalanx — parallel]
    q2 -->|No| q3{Explicit dependencies / branches?}
    q3 -->|Yes| campaign[Campaign — DAG]
    q3 -->|No| q4{A lead agent should delegate?}
    q4 -->|Yes| chain[Chain of Command]
    q4 -->|No| q5{Pattern varies per request?}
    q5 -->|Yes| commander[Commander — auto-route]
    q5 -->|No| formation

Formation — Sequential

Source: crates/paladin-battalion/src/formation_service.rs

Each Paladin's output becomes the next Paladin's input. Ideal for refinement pipelines (extract → analyze → write). If a stage fails, the configured ErrorStrategy decides whether the chain short-circuits (FailFast, the default) or continues.

#![allow(unused)]
fn main() {
use paladin_battalion::formation_service::FormationExecutionService;
use paladin_core::platform::container::battalion::formation::Formation;
use paladin_core::platform::container::battalion::{BattalionConfig, ErrorStrategy};

/// Run three Paladins in sequence; each one's output feeds the next.
pub async fn run_formation() -> Result<(), Box<dyn std::error::Error>> {
    let paladin_port = mock_paladin_port();
    let extractor = create_paladin("Extractor");
    let analyzer = create_paladin("Analyzer");
    let writer = create_paladin("Writer");

    let config = BattalionConfig {
        error_strategy: ErrorStrategy::FailFast, // first failure aborts the chain
        ..Default::default()
    };
    let formation = Formation::new(vec![extractor, analyzer, writer], config)?;

    let service = FormationExecutionService::new(paladin_port);
    let result = service
        .execute(&formation, "Raw Q3 earnings data...")
        .await?;

    println!("Final output: {}", result.final_output);
    Ok(())
}
}

Error handling / short-circuit: with ErrorStrategy::FailFast the first failing stage stops the Formation and returns the error. With ContinueOnError, a failed stage is skipped and its input is passed through to the next stage. Keep chains short (≤5) for latency-sensitive paths — each stage is one sequential LLM round-trip.

Phalanx — Parallel

Source: crates/paladin-battalion/src/phalanx_service.rs

Every Paladin receives the same input and runs concurrently on tokio tasks. Results are combined according to an AggregationStrategy, and concurrency is bounded by Phalanx::with_max_concurrency so you don't exceed LLM rate limits.

#![allow(unused)]
fn main() {
use paladin_battalion::phalanx_service::PhalanxExecutionService;
use paladin_core::platform::container::battalion::phalanx::{AggregationStrategy, Phalanx};

/// Fan the same input out to several Paladins concurrently, then aggregate.
pub async fn run_phalanx() -> Result<(), Box<dyn std::error::Error>> {
    let paladin_port = mock_paladin_port();
    let security = create_paladin("SecurityAuditor");
    let perf = create_paladin("PerformanceAnalyst");
    let style = create_paladin("StyleChecker");

    let phalanx = Phalanx::new(vec![security, perf, style], BattalionConfig::default())?
        .with_aggregation(AggregationStrategy::CollectAll)
        .with_max_concurrency(4); // cap concurrent Paladins

    let service = PhalanxExecutionService::new(paladin_port);
    let result = service
        .execute(&phalanx, "Review this Rust module...")
        .await?;

    println!("Aggregated: {}", result.final_output);
    Ok(())
}
}

AggregationStrategy variants: CollectAll (gather all outputs), FirstSuccess (first to finish wins), Majority (consensus), and Custom(String).

Campaign — Graph / DAG

Source: crates/paladin-battalion/src/campaign_service.rs

Paladins are arranged in a directed acyclic graph. The service topologically sorts the graph so every upstream node completes before its downstream nodes start; independent branches run concurrently. Campaign::build() rejects cycles.

#![allow(unused)]
fn main() {
use paladin_battalion::campaign_service::CampaignExecutionService;
use paladin_core::platform::container::battalion::campaign::{
    Campaign, CampaignEdge, EdgeCondition,
};

/// Arrange Paladins as a DAG: `ingest → analyze → report`.
pub async fn run_campaign() -> Result<(), Box<dyn std::error::Error>> {
    let paladin_port = mock_paladin_port();

    let mut campaign = Campaign::new(BattalionConfig::default());
    let ingest = campaign.add_paladin(create_paladin("Ingest"));
    let analyze = campaign.add_paladin(create_paladin("Analyze"));
    let report = campaign.add_paladin(create_paladin("Report"));

    // Edges define dependencies; `EdgeCondition::Always` is unconditional.
    campaign.add_edge(CampaignEdge::new(ingest, analyze, EdgeCondition::Always))?;
    // A conditional edge only traverses when the upstream output matches:
    campaign.add_edge(CampaignEdge::new(
        analyze,
        report,
        EdgeCondition::Contains("ready".to_string()),
    ))?;
    campaign.set_entry_point(ingest)?;

    let service = CampaignExecutionService::new(paladin_port);
    let result = service.execute(&campaign, "Start").await?;

    println!("Campaign output: {}", result.final_output);
    Ok(())
}
}

Paladins are added with add_paladin (returning a Uuid), wired with add_edge using CampaignEdge::new(source, target, condition), and the graph's start is set with set_entry_point. Use EdgeCondition::Contains/Regex for conditional branching; validate() (called by execute) rejects cycles.

Chain of Command — Hierarchical

Source: crates/paladin-battalion/src/chain_of_command_service.rs

A commander Paladin decomposes the task, routes sub-tasks to specialist (subordinate) Paladins, and synthesizes their outputs into a final answer.

#![allow(unused)]
fn main() {
use paladin_battalion::chain_of_command_service::ChainOfCommandExecutionService;
use paladin_core::platform::container::battalion::chain_of_command::ChainOfCommand;

/// A commander Paladin delegates to specialists and synthesizes their work.
pub async fn run_chain_of_command() -> Result<(), Box<dyn std::error::Error>> {
    let paladin_port = mock_paladin_port();
    let commander = create_paladin("Commander");
    let specialists = vec![
        create_paladin("BackendDev"),
        create_paladin("FrontendDev"),
        create_paladin("QaEngineer"),
    ];

    let chain = ChainOfCommand::new(commander, specialists, BattalionConfig::default())?;

    let service = ChainOfCommandExecutionService::new(paladin_port);
    let result = service.execute(&chain, "Build a login feature").await?;

    println!("Selected specialists: {:?}", result.selected_specialists);
    println!("Reasoning: {}", result.reasoning);
    for output in &result.outputs {
        println!("- {output}");
    }
    Ok(())
}
}

The service returns a DelegationResult with selected_specialists, reasoning, and the specialists' outputs. Give each subordinate a distinct agent_description so the commander can route accurately.

Commander — Dynamic Strategy Routing

Source: crates/paladin-battalion/src/commander.rs

The Commander is a single entry-point that selects a pattern automatically (Auto mode) based on the input text and the number/capabilities of the Paladins, or runs an explicit strategy you name. It also collects rich telemetry and can export execution metadata to JSON.

Auto mode

#![allow(unused)]
fn main() {
use paladin_battalion::commander::CommanderBuilder;
use paladin_core::platform::container::battalion::BattalionStrategy;

/// Let the Commander auto-select the best pattern for the input.
pub async fn run_commander_auto() -> Result<(), Box<dyn std::error::Error>> {
    let paladin_port = mock_paladin_port();

    let commander = CommanderBuilder::new(paladin_port)
        .strategy(BattalionStrategy::Auto)
        .paladins(vec![
            create_paladin("Analyzer"),
            create_paladin("Processor"),
            create_paladin("Synthesizer"),
        ])
        .build()?;

    let result = commander
        .execute("Analyze and summarize this report")
        .await?;

    println!("Strategy selected: {:?}", result.strategy_used);
    if let Some(reason) = &result.strategy_selection_reasoning {
        println!("Reasoning: {reason}");
    }
    println!("Output: {}", result.final_output);
    Ok(())
}
}

Explicit strategy

let commander = CommanderBuilder::new(paladin_port)
    .strategy(BattalionStrategy::Formation) // force a specific pattern
    .paladins(pipeline_paladins)
    .build()?;
let result = commander.execute(input).await?;

Auto-mode heuristics (first match wins)

Maneuver is explicit-only and is never chosen by Auto mode. Strategy selection typically adds ~0–5 ms of overhead; the decision is reported in result.strategy_selection_reasoning.

Metadata export

Point the Commander at a directory and it writes one JSON file per execution ({strategy}_{timestamp}_{uuid}.json) for audit, cost, and performance analysis.

use paladin_core::platform::container::battalion::BattalionConfig;
use std::path::PathBuf;

let config = BattalionConfig::new("audited_battalion")
    .with_metadata_dir(PathBuf::from("./battalion_metadata"));

let commander = CommanderBuilder::new(paladin_port)
    .strategy(BattalionStrategy::Auto)
    .paladins(paladins)
    .config(config)
    .build()?;

let result = commander.execute(input).await?;
// Metadata written to ./battalion_metadata/{strategy}_{timestamp}_{uuid}.json

Each file records battalion_id, strategy_used, duration_ms, total_tokens, per-Paladin paladin_results (output, execution_time_ms, token_count, stop_reason), per_paladin_times, per_paladin_tokens, and strategy_selection_reasoning.

Job Scheduling

Source: crates/paladin-ports/src/output/scheduler_port.rs and queue_port.rs

The scheduler runs jobs on a 6-field cron schedule; the queue ports manage asynchronous work items. A Redis-backed implementation is gated behind the root redis-queue feature.

Prerequisites: the Redis-backed queue requires the redis-queue feature and a running Redis instance. Run make dev to start it (alongside MinIO, MySQL, Qdrant).

Scheduling a recurring job

JobSpec carries a human label, a cron expression, and arbitrary metadata. SchedulerPort returns a JobId you can use to query status or cancel.

#![allow(unused)]
fn main() {
use paladin_ports::output::scheduler_port::{JobSpec, JobStatus, SchedulerPort};

/// Schedule a recurring job with a 6-field cron expression.
pub async fn run_scheduling() -> Result<(), Box<dyn std::error::Error>> {
    let scheduler: Arc<dyn SchedulerPort> = mock_scheduler();
    scheduler.start().await?;

    // 6-field cron: sec min hour day month weekday
    let spec = JobSpec::new("daily-digest", "0 0 9 * * *") // every day at 09:00:00
        .with_metadata("workflow", "news-digest");

    let job_id = scheduler.schedule_job(spec).await?;

    let status: JobStatus = scheduler.get_job_status(&job_id).await?;
    println!("job {job_id:?} is {status:?}");

    // Later: scheduler.cancel_job(&job_id).await?;
    Ok(())
}
}

JobStatus lifecycle: Scheduled → Running → Completed (or Failed { .. } / Cancelled). JobInfo (from get_job_info) adds created_at, last_run, next_run, run_count, and failure_count.

Queue management, retry, and timeouts

The FullQueuePort trait composes enqueue/dequeue, batch, priority, and management operations (pause_queue, resume_queue, retry_item, purge_failed, get_queue_stats). Retry and timeout behavior for battalion execution is controlled by the battalion.retry and battalion.default_timeout_seconds configuration (see Configuration Reference).

use paladin_ports::output::queue_port::{FullQueuePort, QueueStats};

let stats: QueueStats = queue.get_queue_stats("news-digest").await?;
println!("pending: {}, processing: {}", stats.pending_items, stats.processing_items);

// Retry a failed item or purge the dead-letter set
queue.retry_item("news-digest", item_id).await?;
let purged = queue.purge_failed("news-digest").await?;

Event and Trigger System

Source: crates/paladin-core/src/platform/container/trigger.rs

A Trigger binds an incoming event to an action when a TriggerCondition matches. Events are matched by event_type_pattern, optional source_pattern, payload conditions, minimum priority, and optional TimeCondition windows (active hours/days and a cooldown).

Defining a condition and firing an event

Build a TriggerCondition (and TriggerConfig), then fire a matching event through the orchestrator bridge. fire_event returns an EventDispatchResult reporting how many triggers matched and their IDs.

#![allow(unused)]
fn main() {
use paladin_core::base::entity::message::MessagePriority;
use paladin_core::platform::container::trigger::{TimeCondition, TriggerCondition, TriggerConfig};
use paladin_ports::output::orchestrator_port::{FireEventRequest, OrchestratorPort};

/// Build a trigger condition and fire a matching event.
pub async fn run_events() -> Result<(), Box<dyn std::error::Error>> {
    let condition = TriggerCondition {
        event_type_pattern: "critical_finding".to_string(),
        source_pattern: Some("security-*".to_string()),
        payload_conditions: vec![],
        min_priority: Some(MessagePriority::High),
        time_conditions: Some(TimeCondition {
            active_hours: Some((9, 17)),            // only 09:00–17:00
            active_days: Some(vec![1, 2, 3, 4, 5]), // Mon–Fri
            cooldown_seconds: Some(300),            // at most once per 5 min
        }),
    };

    let config = TriggerConfig {
        max_retries: 3,
        timeout_seconds: 60,
        preserve_after_completion: false,
        ttl_seconds: 3600,
        processing_priority: MessagePriority::High,
    };

    // Fire an event through the orchestrator bridge.
    let orchestrator = mock_orchestrator();
    let result = orchestrator
        .fire_event(FireEventRequest {
            event_type: "critical_finding".to_string(),
            payload: serde_json::json!({ "severity": "high", "cve": "CVE-2025-0001" }),
            source: "security-scanner".to_string(),
        })
        .await?;

    println!(
        "{} trigger(s) fired: {:?}",
        result.triggered_count, result.trigger_ids
    );
    Ok(())
}
}

A matched trigger initiates the bound workflow (e.g. scheduling a job or queuing a Paladin run). See the Agent ↔ Orchestrator Bridge for end-to-end recipes that combine events, triggers, and agent execution.

Configuration Reference

All battalion behavior is configurable through the battalion: section of config.yml:

battalion:
  default_timeout_seconds: 300     # Per-battalion execution timeout
  error_strategy: "fail_fast"      # fail_fast | continue_on_error | retry_then_continue
  max_concurrent_paladins: 10      # Phalanx concurrency limit
  metadata_output_enabled: false   # Write execution metadata to files

  retry:                           # Used when error_strategy = retry_then_continue
    max_attempts: 3
    exponential_backoff: true
    jitter: true
    base_delay_ms: 100
    max_delay_seconds: 10

Environment overrides follow the APP_BATTALION_* convention (e.g. APP_BATTALION_ERROR_STRATEGY, APP_BATTALION_MAX_CONCURRENT_PALADINS). See Configuration for the full schema.

BattalionResult (returned by the Formation/Phalanx/Campaign/Commander services) exposes: final_output: String, paladin_results: Vec<PaladinResult>, status: BattalionStatus, strategy_used: BattalionStrategy, total_tokens: u64, per_paladin_times, and per_paladin_tokens. (Chain of Command returns a DelegationResult instead.)

See Also

• Agent ↔ Orchestrator Bridge — agents triggering workflows and workflows invoking agents, with use-case recipes.
• Battalion Patterns — concise cheat sheet for all eight patterns including Conclave, Council, and Grove.
• Maneuver Flow DSL — declarative composition of multiple patterns.
• Content Processing — feeding a content pipeline into agent analysis.
• Crate Map — where paladin-battalion and paladin-ports sit in the workspace.

Content Processing

The paladin-content crate (crates/paladin-content/) ingests content from external sources, runs it through aggregation/analysis use cases, hands it to a Paladin agent for AI enrichment, and delivers the result. This guide covers the ingestion adapters, the processing use cases, the content → agent bridge, and delivery — documenting only what is wired into the compiled crate today.

Every code example targets the current v0.5.0 workspace. The substantive examples are real, compiled code pulled from the paladin-doc-examples crate via mdBook {{#include}} (a few illustrative fragments are rust,ignore). The API forms are verified against crates/paladin-content/src/.

Feature flags. Content processing lives behind the root content-processing feature, which enables paladin-content. Within the crate, news-api enables the News API fetcher and llm enables LLM-powered analysis. See the Crate Map for the full flag table.

Table of Contents

1. Content Ingestion Sources
2. Aggregation and the Processing Pipeline
3. Content → Agent Bridge
4. Content Delivery
5. Capabilities and Limitations
6. See Also

Content Ingestion Sources

Every fetcher produces a ContentItem (paladin_core::platform::container::content::ContentItem), the common currency of the pipeline. Sources are constructed and configured programmatically (there is no dedicated content: section in config.yml yet — see Limitations).

PDF / documents — PdfExtractor

PdfExtractor parses a PDF (from a path or raw bytes) into a Document. DocumentAdapter wraps document parsing for the pipeline.

#![allow(unused)]
fn main() {
use paladin_content::adapters::document::pdf_extractor::PdfExtractor;
use std::path::Path;

/// Extract a PDF (from a path or raw bytes) into a `Document`.
pub fn ingest_pdf() -> Result<(), Box<dyn std::error::Error>> {
    let extractor = PdfExtractor::new();
    let document = extractor.extract(Path::new("./reports/q3-earnings.pdf"))?;
    // Or from bytes already in memory:
    // let document = extractor.extract_bytes(&pdf_bytes)?;
    Ok(())
}
}

HTTP endpoints — HttpContentFetcher

HttpContentFetcher fetches a URL and returns a ContentItem. It implements the ContentFetchingService trait, so it can be driven directly or through the FetchContent use case.

#![allow(unused)]
fn main() {
use paladin_content::adapters::input::http_content_fetcher::HttpContentFetcher;
use paladin_content::services::content_fetching_service::{ContentFetchingService, FetchContent};

/// Fetch a URL into a `ContentItem`, directly and via the `FetchContent` use case.
pub fn ingest_http() -> Result<(), Box<dyn std::error::Error>> {
    let fetcher = HttpContentFetcher::new();
    // Direct use:
    let item = fetcher.fetch_content("https://example.com/article")?;

    // Or wrapped in the use case (same trait, swappable adapter):
    let fetch = FetchContent::new(HttpContentFetcher::new());
    let item = fetch.execute("https://example.com/article")?;
    Ok(())
}
}

News / feeds — NewsApiFetcher (feature news-api)

NewsApiFetcher polls a News API endpoint. It takes an API key and reuses an HttpContentFetcher for transport.

#![allow(unused)]
fn main() {
use paladin_content::adapters::input::news_api_fetcher::NewsApiFetcher;

/// Construct a News API fetcher (feature `news-api`).
pub fn ingest_news() {
    let fetcher = NewsApiFetcher::new("YOUR_NEWS_API_KEY".to_string())
        .with_content_fetcher(HttpContentFetcher::new());
}
}

Files — FileContentFetcher

For local ingestion and testing, FileContentFetcher reads a file from disk and infers its content type from the extension. Unlike the HTTP fetcher, it implements ContentIngestionPort (paladin_ports::input): its fetch_content takes a ContentItem describing the source path and returns a populated ContentItem. (It is an internal #[doc(hidden)] adapter; the primary documented ingestion paths are HTTP, PDF, and the News API above.)

Aggregation and the Processing Pipeline

Once items are fetched, the use cases combine and analyze them. Each use case is generic over a trait, so adapters are swappable.

flowchart LR
    src[(Sources: PDF / HTTP / News / File)] --> fetch[FetchContent]
    fetch --> agg[AggregateContent]
    agg --> sum[ContentSummarizer]
    sum --> ai[LlmContentAnalyzer]
    ai --> deliver[DeliverContentUseCase]
    deliver --> out[(Destinations)]

Aggregation

AggregateContent wraps a ContentListService and merges a vector of JSON values into a single aggregated value — useful for collapsing multiple fetched sources before analysis.

#![allow(unused)]
fn main() {
use paladin_content::services::content_aggregator_service::AggregateContent;

/// Merge JSON from several sources into one aggregated value.
pub fn aggregate() {
    // `MockListService` implements the `ContentListService` trait.
    let aggregator = AggregateContent::new(MockListService);
    let source_a = serde_json::json!({ "title": "A" });
    let source_b = serde_json::json!({ "title": "B" });
    let aggregated = aggregator.execute(vec![source_a, source_b]);
}
}

Summarization

ContentSummarizer produces summaries and keywords without an LLM call (deterministic text processing), returning a ContentSummary plus ContentMetadata.

#![allow(unused)]
fn main() {
use paladin_content::services::content_summarizer_service::ContentSummarizer;

/// Summarize a `ContentItem` and extract keywords (no LLM call).
pub fn summarize() {
    let item = text_content_item("A long article body about quarterly earnings...");
    let summarizer = ContentSummarizer::new();
    let summary = summarizer.summarize_content(&item, 500); // max 500 chars
    let keywords = summarizer.extract_keywords(&item);
}
}

Content → Agent Bridge

The llm feature enables LlmContentAnalyzer, which passes a ContentItem plus a prompt to a Paladin LLM analysis service for AI enrichment. This is the seam where the content pipeline meets the agent layer.

LlmContentAnalyzer::analyze_with_prompt_async takes an LlmContentAnalysisInput (prompt: PromptItem, content: ContentItem) and an LlmContentAnalysisConfig (model, retries, timeout, max_content_length), and returns the analysis as JSON.

#![allow(unused)]
fn main() {
use paladin_content::services::content_llm_analysis_service::{
    LlmContentAnalysisConfig, LlmContentAnalysisInput, LlmContentAnalyzer,
};
use paladin_llm::llm_analysis_service::LlmAnalysisService;
use paladin_llm::mock::MockLlmAdapter;
use paladin_ports::output::llm_port::LlmPort;

/// Pass content + a prompt to a Paladin LLM service for AI enrichment.
pub async fn content_to_agent() -> Result<(), Box<dyn std::error::Error>> {
    // In production this is a real provider (e.g. OpenAIAdapter); here a mock.
    let llm: Arc<dyn LlmPort> =
        Arc::new(MockLlmAdapter::new().with_response("{\"summary\":\"...\"}"));
    let llm_service = Arc::new(LlmAnalysisService::new(llm));

    let analyzer = LlmContentAnalyzer::new(llm_service);
    let input = LlmContentAnalysisInput {
        prompt: text_prompt_item("Summarize the key risks in this article."),
        content: text_content_item("Latest article body..."),
    };
    let config = LlmContentAnalysisConfig::default(); // gpt-3.5-turbo, 3 retries, 30s timeout

    let analysis = analyzer
        .analyze_with_prompt_async(&input, &config)
        .await
        .map_err(|e| -> Box<dyn std::error::Error> { e.into() })?;
    println!("{}", serde_json::to_string_pretty(&analysis)?);
    Ok(())
}
}

Use the async method (analyze_with_prompt_async). The sync analyze_with_prompt is a compatibility stub that returns an error directing callers to the async path.

For richer agent interactions — an agent that triggers a workflow, or a workflow step that invokes a full Paladin agent loop — see the Agent ↔ Orchestrator Bridge.

Content Delivery

DeliverContentUseCase sends processed content to a destination through the ContentDeliveryService port (paladin_ports::output::content_delivery_port). It takes a DeliveryRequest and returns a DeliveryResponse (with a DeliveryStatus).

#![allow(unused)]
fn main() {
use paladin_content::services::content_delivery_service::DeliverContentUseCase;
use paladin_ports::output::content_delivery_port::{
    ContentPayload, DeliveryMethod, DeliveryPriority, DeliveryRequest,
};

/// Deliver processed content through a `ContentDeliveryService`.
pub fn deliver() -> Result<(), Box<dyn std::error::Error>> {
    let delivery = DeliverContentUseCase::new(MockDeliveryAdapter);

    let request = DeliveryRequest {
        recipient_id: "ops-team".to_string(),
        delivery_method: DeliveryMethod::Email {
            to: "ops@example.com".to_string(),
            subject: "Daily digest".to_string(),
        },
        content_payload: ContentPayload::SingleItem(text_content_item("Digest body...")),
        priority: DeliveryPriority::Normal,
        scheduled_time: None,
        metadata: None,
    };

    let response = delivery.execute(request)?;
    println!("delivery status: {:?}", response.status);
    Ok(())
}
}

For push/email/system notification of delivered content, wire the delivery adapter to the notification adapters (paladin-notifications) or fire a notification through the orchestrator bridge — see the bridge recipes.

Capabilities and Limitations

The crate's manifest declares some features whose adapters are not yet implemented in v0.5.0. To keep this guide honest:

For web-scraping and RSS today, fetch the raw resource with HttpContentFetcher and parse it in your own adapter. Filtering/dedup must likewise be done in caller code until the content_filtering_service module is completed and re-enabled.

See Also

• Agent ↔ Orchestrator Bridge — end-to-end recipes combining content ingestion with agent analysis and notification.
• Orchestration — running the analysis Paladin inside a Battalion workflow.
• Paladin Agents — building the Paladin that performs the AI enrichment.
• Crate Map — paladin-content exports and feature flags.

Agent ↔ Orchestrator Bridge

Paladin agents and Battalion workflows interact bidirectionally:

• An agent can trigger orchestration — schedule a job, enqueue an item, fire an event, or send a notification — through a narrow, policy-guarded port.
• A workflow can invoke an agent — run a single Paladin or a whole Battalion as a step and feed its output back into the workflow.

This guide covers both directions, how to configure the bridge safely, and four end-to-end recipes. It builds on the Orchestration and Content Processing guides.

Every example targets the current v0.5.0 workspace. The substantive examples are real, compiled code pulled from the paladin-doc-examples crate via mdBook {{#include}} (one illustrative fragment is rust,ignore). API forms are verified against crates/paladin-ports/src/output/orchestrator_port.rs, paladin_executor_port.rs, battalion_port.rs, and the concrete OrchestratorBridgeAdapter in src/application/services/orchestration/.

Table of Contents

1. Agents Triggering Orchestration
2. Orchestration Invoking Agents
3. Configuring the Bridge
4. Use-Case Recipes
5. See Also

Agents Triggering Orchestration

The seam is OrchestratorPort (crates/paladin-ports/src/output/orchestrator_port.rs). It exposes exactly four actions, mirrored by the BridgeAction enum:

The concrete adapter, OrchestratorBridgeAdapter, wraps an Arc<Orchestrator> and a BridgePolicy. It enforces the policy before performing any underlying call, so an agent can never exceed the actions or per-execution caps it was granted.

sequenceDiagram
    participant Agent as Paladin agent (tool call)
    participant Bridge as OrchestratorBridgeAdapter
    participant Policy as BridgePolicy
    participant Orch as Orchestrator

    Agent->>Bridge: fire_event(FireEventRequest)
    Bridge->>Policy: is_allowed(FireEvent)?
    Policy-->>Bridge: true
    Bridge->>Policy: cap_for(FireEvent)
    Policy-->>Bridge: 3
    Bridge->>Orch: dispatch event (within cap)
    Orch-->>Bridge: EventDispatchResult
    Bridge-->>Agent: Ok(EventDispatchResult)

Tool-based invocation from an agent loop

Expose the bridge to a Paladin as a tool. When the agent decides to act, the tool implementation calls the relevant OrchestratorPort method. The agent never touches the Orchestrator directly — only the policy-guarded port.

#![allow(unused)]
fn main() {
use paladin_ports::output::orchestrator_port::{
    BridgeAction, BridgePolicy, FireEventRequest, OrchestratorBridgeError, OrchestratorPort,
};

/// An agent fires a domain event through the policy-guarded bridge.
pub async fn agent_triggers_orchestration() -> Result<(), Box<dyn std::error::Error>> {
    // Grant ONLY the actions this agent should perform, with explicit caps.
    let mut allowed = HashSet::new();
    allowed.insert(BridgeAction::FireEvent);
    let policy = BridgePolicy::new(allowed, 0, 0, 5, 0); // up to 5 events, nothing else

    // In production this is an `OrchestratorBridgeAdapter`; here a mock stands in.
    let bridge: Arc<dyn OrchestratorPort> = mock_orchestrator();
    let _ = &policy; // the real adapter is constructed as `::new(orchestrator, policy)`

    match bridge
        .fire_event(FireEventRequest {
            event_type: "critical_finding".to_string(),
            payload: serde_json::json!({ "severity": "high" }),
            source: "security-agent".to_string(),
        })
        .await
    {
        Ok(result) => println!("fired; {} trigger(s) matched", result.triggered_count),
        Err(OrchestratorBridgeError::ActionNotAllowed(_)) => {
            eprintln!("policy forbids this action")
        }
        Err(OrchestratorBridgeError::QuotaExceeded { .. }) => {
            eprintln!("per-execution cap reached")
        }
        Err(e) => return Err(e.into()),
    }
    Ok(())
}
}

OrchestratorBridgeError distinguishes ActionNotAllowed (the policy doesn't grant the action) from QuotaExceeded (the per-execution cap is reached), so an agent can react sensibly instead of failing opaquely.

Orchestration Invoking Agents

The reverse direction uses the executor ports:

• PaladinExecutorPort (paladin_executor_port.rs) — run a single Paladin: async fn execute(&self, paladin: &Paladin, input: &str) -> Result<PaladinResult, PaladinError>.
• BattalionPort (battalion_port.rs) — run/monitor a whole Battalion by id: execute(battalion_id) -> BattalionResult, plus status and cancel.

A workflow step builds the input string (passing context from earlier steps), calls the executor, and reads the result back out.

sequenceDiagram
    participant WF as Workflow step
    participant Exec as PaladinExecutorPort
    participant Paladin as Paladin agent

    WF->>Exec: execute(&paladin, input_with_context)
    Exec->>Paladin: run agent loop
    Paladin-->>Exec: PaladinResult { output, token_count, ... }
    Exec-->>WF: Ok(PaladinResult)
    Note over WF: feed result.output into the next step

#![allow(unused)]
fn main() {
use paladin_core::platform::container::paladin::Paladin;
use paladin_ports::output::paladin_executor_port::PaladinExecutorPort;

/// A workflow step runs a single Paladin, passing context via the input string.
pub async fn orchestration_invokes_agent(
    analyst: &Paladin,
) -> Result<(), Box<dyn std::error::Error>> {
    let executor: Arc<dyn PaladinExecutorPort> = mock_executor();

    let upstream = "Q3 revenue rose 12% QoQ; churn fell to 2.1%.";
    let input = format!("Summarize the key risks given this context:\n{upstream}");

    let result = executor.execute(analyst, &input).await?;

    println!("agent said: {}", result.output);
    println!(
        "tokens: {}, stop reason: {:?}",
        result.token_count, result.stop_reason
    );
    Ok(())
}
}

PaladinResult carries output, token_count, execution_time_ms, loop_count, and stop_reason — everything the workflow needs to decide what to do next. To invoke a whole Battalion instead of a single agent, use BattalionPort::execute(battalion_id) and read the BattalionResult (see Orchestration → Configuration Reference).

Configuring the Bridge

Bridge behavior is configured programmatically through BridgePolicy — there is no dedicated config.yml bridge section in v0.5.0. A policy is two things: the set of allowed actions, and a per-execution cap for each action.

#![allow(unused)]
fn main() {
/// Build least-privilege and default bridge policies.
pub fn configure_bridge() {
    use paladin_ports::output::orchestrator_port::{BridgeAction, BridgePolicy};

    // Explicit, least-privilege: allow scheduling + notifications only,
    // with caps of (jobs=2, queue=0, events=0, notifications=5).
    let mut allowed = HashSet::new();
    allowed.insert(BridgeAction::ScheduleJob);
    allowed.insert(BridgeAction::SendNotification);
    let policy = BridgePolicy::new(allowed, 2, 0, 0, 5);

    // Builder-style: start from caps and add actions.
    let policy = BridgePolicy::new(HashSet::new(), 1, 1, 1, 1)
        .allow(BridgeAction::FireEvent)
        .allow(BridgeAction::QueueItem);

    // Conservative-but-usable default: all four actions, cap 3 each.
    let policy = BridgePolicy::default();
}
}

The three forms shown are: an explicit least-privilege policy, the builder-style .allow(..), and the conservative-but-usable Default (all four actions, cap 3 each). Prefer an explicit least-privilege policy for agents you don't fully trust.

Tip: because the adapter enforces the policy before every call, tightening a policy is a safe, local change — you don't have to audit the agent's prompt to constrain what it can do.

Use-Case Recipes

1. News monitoring pipeline with AI analysis

NewsApiFetcher → AI summarization (LlmContentAnalyzer) → notification via the bridge.

#![allow(unused)]
fn main() {
use paladin_ports::output::orchestrator_port::SendNotificationRequest;

/// Recipe: notify the result of an AI summary through the bridge.
pub async fn recipe_news_notification(
    bridge: &Arc<dyn OrchestratorPort>,
    summary: &str,
) -> Result<(), Box<dyn std::error::Error>> {
    bridge
        .send_notification(SendNotificationRequest {
            channel: "email".to_string(),
            recipient: "ops@example.com".to_string(),
            subject: "Daily news digest".to_string(),
            body: summary.to_string(),
        })
        .await?;
    Ok(())
}
}

See Content Processing for the ingestion/analysis half and Orchestration → Job Scheduling to run this on a cron.

2. Research workflow

A web/HTTP tool gathers sources, a Paladin synthesizes them, and a Formation assembles the final report.

// 1. Agent gathers sources via an HTTP tool (Arsenal), producing notes.
// 2. Synthesis Paladin run as a workflow step:
let synthesis = executor.execute(&synthesizer, &collected_notes).await?;
// 3. Formation assembles intro → body → conclusion from the synthesis.
let report = formation_service.execute(&report_formation, &synthesis.output).await?;

3. Scheduled batch enrichment (job queue)

A recurring job enqueues items; a worker drains the queue and runs each through a Paladin.

#![allow(unused)]
fn main() {
use paladin_core::platform::container::schedule::Schedule;
use paladin_ports::output::orchestrator_port::{QueueItemRequest, ScheduleJobRequest};

/// Recipe: schedule a recurring batch job and enqueue an item.
pub async fn recipe_scheduled_batch(
    bridge: &Arc<dyn OrchestratorPort>,
    content_id: &str,
) -> Result<(), Box<dyn std::error::Error>> {
    bridge
        .schedule_job(ScheduleJobRequest {
            name: "nightly-enrichment".to_string(),
            description: "Enrich the day's content with AI tags".to_string(),
            schedule: Schedule::Daily(2, 0), // 02:00 daily
        })
        .await?;

    bridge
        .queue_item(QueueItemRequest {
            queue_name: "enrichment".to_string(),
            payload: serde_json::json!({ "content_id": content_id }),
        })
        .await?;
    Ok(())
}
}

4. Trigger-initiated agent run

An agent fires a domain event; a registered Trigger matches it and initiates a Paladin run — fully event-driven, no polling.

#![allow(unused)]
fn main() {
/// Recipe: an agent fires an event that a Trigger turns into a Paladin run.
pub async fn recipe_trigger_initiated(
    bridge: &Arc<dyn OrchestratorPort>,
) -> Result<(), Box<dyn std::error::Error>> {
    let dispatch = bridge
        .fire_event(FireEventRequest {
            event_type: "anomaly_detected".to_string(),
            payload: serde_json::json!({ "metric": "latency_p99", "value": 920 }),
            source: "monitor-agent".to_string(),
        })
        .await?;

    println!("{} trigger(s) initiated", dispatch.triggered_count);
    Ok(())
}
}

See Also

• Orchestration — the Battalion patterns, job scheduler, and trigger system the bridge drives.
• Content Processing — the ingestion/analysis pipeline used in recipes 1 and 3.
• Paladin Agents — building the agents on both sides of the bridge.
• Crate Map — where OrchestratorPort and the executor ports live.

Arsenal Tools

The Arsenal system (crates/paladin-ports/src/output/arsenal_port.rs) gives Paladins access to external tools and services through the Model Context Protocol (MCP). Tools are called Armaments; the registry that holds them is the Arsenal.

Table of Contents

1. Concepts
2. Quick Start — STDIO Server
3. SSE Server Configuration
4. config.yml Reference
5. ArsenalPort Trait
6. ArsenalRegistry Trait
7. Attaching Arsenal to a Paladin
8. Custom Armaments (Direct Rust Tools)
9. Handoff Tool
10. Error Handling
11. Best Practices

Concepts

Quick Start — STDIO Server

STDIO servers are the most common MCP transport. The process is spawned and communicated with via newline-delimited JSON on stdin/stdout.

1. Configure in config.yml

arsenal:
  mcp_servers:
    - name: web_search
      type: stdio
      command: uvx
      args: ["mcp-server-brave-search"]
      env:
        BRAVE_API_KEY: "${BRAVE_API_KEY}"
    - name: filesystem
      type: stdio
      command: npx
      args: ["-y", "@modelcontextprotocol/server-filesystem", "/workspace"]

2. Build a Paladin with the Arsenal

use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin_ports::output::llm_port::LlmPort;
use paladin_ports::output::arsenal_port::ArsenalRegistry;
use std::sync::Arc;

// Arsenal registry is built from config.yml automatically when using
// PaladinBuilder::from_config() or can be constructed manually.
let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are a research assistant with web search access.")
    .with_arsenal_registry(arsenal_registry)
    .build()
    .await?;

let result = paladin.execute("Find the latest Rust release notes").await?;
println!("{}", result.output);

The Paladin will automatically detect tool-call JSON in LLM responses, invoke the tool via the Arsenal, and feed results back into the reasoning loop.

SSE Server Configuration

HTTP/SSE servers expose a REST endpoint:

arsenal:
  mcp_servers:
    - name: my_api_server
      type: sse
      endpoint: "http://localhost:8080/mcp"
      timeout_seconds: 30
      max_retries: 3

The MCPSseAdapter sends requests and reads responses over the SSE stream:

use paladin::infrastructure::adapters::arsenal::mcp_sse_adapter::MCPSseAdapter;

let mut adapter = MCPSseAdapter::new("http://localhost:8080/mcp");
adapter.connect().await?;

config.yml Reference

arsenal:
  mcp_servers:
    - name: <identifier>          # Unique name used in logs and errors
      type: stdio | sse           # Transport type
      # STDIO fields:
      command: <executable>       # e.g. python3, npx, uvx
      args: [<arg>, ...]          # Command-line arguments
      env:                        # Optional environment variables
        KEY: value
      # SSE fields:
      endpoint: <url>             # Full URL of the SSE endpoint
      timeout_seconds: 30         # Request timeout
      max_retries: 3              # Retry attempts on failure

ArsenalPort Trait

Defined in crates/paladin-ports/src/output/arsenal_port.rs:

#[async_trait]
pub trait ArsenalPort: Send + Sync {
    /// List all available armaments from this MCP server
    async fn list_armaments(&self) -> Vec<Armament>;

    /// Invoke an armament with the given arguments
    async fn invoke(&self, call: ArmamentCall) -> Result<ArmamentResult, ArsenalError>;

    /// Validate call arguments against the armament's JSON schema
    fn validate_call(&self, call: &ArmamentCall) -> Result<(), ArsenalError>;
}

Direct usage:

use paladin_core::platform::container::arsenal::ArmamentCall;
use serde_json::json;
use std::collections::HashMap;

let mut args = HashMap::new();
args.insert("query".to_string(), json!("Rust 2024 edition features"));

let call = ArmamentCall::new("web_search", args);
arsenal_port.validate_call(&call)?;

let result = arsenal_port.invoke(call).await?;
if result.success {
    println!("{}", result.output.unwrap());
}

ArsenalRegistry Trait

Defined alongside ArsenalPort:

#[async_trait]
pub trait ArsenalRegistry: Send + Sync {
    /// Register a new armament in the registry
    async fn register(&self, armament: Armament);

    /// Remove an armament by name
    async fn remove(&self, name: &str);

    /// Get all registered armament descriptors
    async fn list(&self) -> Vec<Armament>;

    /// Look up a specific armament by name
    async fn get(&self, name: &str) -> Option<Armament>;
}

Attaching Arsenal to a Paladin

use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin_ports::output::arsenal_port::ArsenalRegistry;
use std::sync::Arc;

let paladin = PaladinBuilder::new(llm_port)
    .system_prompt(
        "You are a coding assistant. Use the filesystem tool to read files when needed."
    )
    .with_arsenal_registry(Arc::new(my_registry))
    .build()
    .await?;

Custom Armaments (Direct Rust Tools)

Implement ArsenalPort to expose any Rust function as a tool:

use async_trait::async_trait;
use paladin_core::platform::container::arsenal::{
    Armament, ArmamentCall, ArmamentResult, ArsenalError,
};
use paladin_ports::output::arsenal_port::ArsenalPort;

pub struct CalculatorTool;

#[async_trait]
impl ArsenalPort for CalculatorTool {
    async fn list_armaments(&self) -> Vec<Armament> {
        vec![Armament {
            name: "calculate".to_string(),
            description: "Evaluate a mathematical expression".to_string(),
            input_schema: serde_json::json!({
                "type": "object",
                "properties": {
                    "expression": { "type": "string" }
                },
                "required": ["expression"]
            }),
        }]
    }

    async fn invoke(&self, call: ArmamentCall) -> Result<ArmamentResult, ArsenalError> {
        let expr = call.args["expression"].as_str().unwrap_or_default();
        // ... evaluate ...
        Ok(ArmamentResult { success: true, output: Some(serde_json::json!(42)), error: None })
    }

    fn validate_call(&self, call: &ArmamentCall) -> Result<(), ArsenalError> {
        if call.args.contains_key("expression") {
            Ok(())
        } else {
            Err(ArsenalError::InvalidArguments("expression is required".into()))
        }
    }
}

Handoff Tool

The handoff_tool in crates/paladin-core/src/platform/container/arsenal/handoff_tool.rs is a built-in Armament that allows a Paladin to delegate sub-tasks to specialist agents at runtime. Register specialist agents on the builder:

let coordinator = PaladinBuilder::new(llm_port)
    .system_prompt("You are a coordinator. Delegate to specialists when needed.")
    .with_specialist(Arc::new(code_paladin))
    .with_specialist(Arc::new(test_paladin))
    .build()
    .await?;

The LLM will emit a tool-call for handoff when it determines a specialist is more appropriate. Delegation records appear in PaladinResult.handoff_history.

Error Handling

ArsenalError variants (from paladin_core::platform::container::arsenal):

Best Practices

• Validate before invoking — call validate_call() to catch argument errors early.
• Set timeouts — all MCP servers should have timeout_seconds to avoid blocking the reasoning loop indefinitely.
• Describe tools well — the Armament description is what the LLM reads to decide whether to call the tool; make it precise.
• Namespace tool names — use server_name.tool_name convention to avoid collisions when registering multiple servers.
• Test with mock — implement a MockArsenalPort in tests to avoid spawning real subprocesses.

Garrison Memory

The Garrison is Paladin AI's conversation memory system. When attached to a Paladin it stores and retrieves conversation history, giving the agent context across multiple reasoning loops and between invocations.

Garrison is defined in crates/paladin-ports/src/output/garrison_port.rs (the GarrisonPort trait) with adapter implementations in crates/paladin-memory/src/garrison/.

Table of Contents

1. Concepts
2. Quick Start
3. Garrison Adapters
4. GarrisonPort Trait
5. GarrisonConfig
6. Conversation Roles
7. Attaching to a Paladin
8. Long-Term Memory with Embeddings
9. config.yml Reference
10. Error Handling
11. Best Practices

Concepts

Quick Start

use paladin_memory::garrison::InMemoryGarrison;
use paladin_core::platform::container::garrison::{GarrisonConfig, GarrisonEntry, ConversationRole};
use paladin_ports::output::garrison_port::GarrisonPort;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let garrison = Arc::new(InMemoryGarrison::new(GarrisonConfig::default()));

    // Store a user message
    garrison.remember(GarrisonEntry::new(
        ConversationRole::User,
        "What is Rust's ownership model?".to_string(),
    )).await?;

    // Store the assistant reply
    garrison.remember(GarrisonEntry::new(
        ConversationRole::Assistant,
        "Rust's ownership model ensures memory safety without a GC...".to_string(),
    )).await?;

    // Recall last 10 entries
    let history = garrison.recall_recent(10).await?;
    for entry in &history {
        println!("{:?}: {}", entry.role, entry.content);
    }

    // Search by keyword
    let results = garrison.search("ownership", 5).await?;
    println!("Found {} messages about ownership", results.len());

    Ok(())
}

Garrison Adapters

Both adapters are in crates/paladin-memory/src/garrison/.

InMemoryGarrison

use paladin_memory::garrison::InMemoryGarrison;
use paladin_core::platform::container::garrison::GarrisonConfig;

let garrison = InMemoryGarrison::new(GarrisonConfig::default());

SqliteGarrison

use paladin_memory::garrison::SqliteGarrison;
use paladin_core::platform::container::garrison::GarrisonConfig;

let garrison = SqliteGarrison::connect(
    "./garrison.db",
    GarrisonConfig::default(),
    "my-paladin-id",
).await?;

SqliteGarrison::connect() creates the file and runs migrations automatically.

GarrisonPort Trait

Full interface defined in crates/paladin-ports/src/output/garrison_port.rs:

#[async_trait]
pub trait GarrisonPort: Send + Sync {
    /// Store a new conversation entry
    async fn remember(&self, entry: GarrisonEntry) -> Result<(), GarrisonError>;

    /// Retrieve the N most recent entries (newest last)
    async fn recall_recent(&self, limit: usize) -> Result<Vec<GarrisonEntry>, GarrisonError>;

    /// Full-text search across stored entries
    async fn search(&self, query: &str, limit: usize) -> Result<Vec<GarrisonEntry>, GarrisonError>;

    /// Clear all entries from this garrison
    async fn forget_all(&self) -> Result<(), GarrisonError>;

    /// Get storage statistics
    async fn stats(&self) -> Result<GarrisonStats, GarrisonError>;
}

GarrisonStats

pub struct GarrisonStats {
    pub entry_count: usize,   // Total stored entries
    pub total_tokens: u32,    // Cumulative token count
    pub size_bytes: Option<u64>, // Storage size (adapters may not support this)
}

GarrisonConfig

use paladin_core::platform::container::garrison::GarrisonConfig;

// max_entries: window size; max_tokens: token budget per context window
let config = GarrisonConfig::new(200, Some(8000));

Conversation Roles

use paladin_core::platform::container::garrison::ConversationRole;

ConversationRole::User       // Human turn
ConversationRole::Assistant  // Paladin / LLM turn
ConversationRole::System     // System instruction
ConversationRole::Tool       // Tool call result

Attaching to a Paladin

use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin_memory::garrison::SqliteGarrison;
use paladin_core::platform::container::garrison::GarrisonConfig;
use paladin_ports::output::garrison_port::GarrisonPort;
use std::sync::Arc;

let garrison: Arc<dyn GarrisonPort> = Arc::new(
    SqliteGarrison::connect("./memory.db", GarrisonConfig::default(), "agent-1").await?
);

let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are a persistent memory assistant.")
    .with_garrison(garrison)
    .build()
    .await?;

Once attached, the Paladin automatically:

1. Retrieves recent history before each LLM call.
2. Appends the user turn and assistant response after each loop.

Long-Term Memory with Embeddings

The LongTermGarrisonPort trait (also in garrison_port.rs) extends GarrisonPort with semantic similarity search using vector embeddings:

pub trait LongTermGarrisonPort: GarrisonPort {
    async fn remember_with_embedding(
        &self,
        entry: GarrisonEntry,
        embedding: Vec<f32>,
    ) -> Result<(), GarrisonError>;

    async fn search_similar(
        &self,
        query_embedding: Vec<f32>,
        limit: usize,
    ) -> Result<Vec<GarrisonEntry>, GarrisonError>;
}

For full vector-based semantic memory, consider Sanctum — see Sanctum Vector Memory.

config.yml Reference

garrison:
  type: sqlite        # "in_memory" or "sqlite"
  path: ./garrison.db # SQLite only
  max_entries: 100
  max_tokens: 8000    # Optional token budget
  eviction_strategy: oldest  # "oldest" (default)

Error Handling

GarrisonError variants:

Best Practices

• Always use SqliteGarrison in production — InMemoryGarrison loses all history when the process restarts.
• Set max_tokens to stay within the LLM's context window; large histories degrade performance.
• Use one Garrison per Paladin — shared garrisons across multiple agents mix conversation contexts and confuse the LLM.
• Call forget_all() between sessions if context carry-over is undesirable (e.g., fresh chat sessions).
• Use search() to retrieve relevant past entries rather than dumping the full history into the prompt.

Sanctum Vector Memory

Sanctum is Paladin AI's long-term semantic memory system. It stores memories as vector embeddings, enabling similarity-based retrieval across sessions — unlike Garrison which stores sequential conversation history, Sanctum finds conceptually similar past experiences.

Sanctum is defined in crates/paladin-ports/src/output/sanctum_port.rs (the SanctumPort trait) with adapter implementations in crates/paladin-memory/src/sanctum/.

Table of Contents

1. Sanctum vs. Garrison
2. Quick Start
3. Sanctum Adapters
4. SanctumPort Trait
5. SanctumEntry and Memory Types
6. Searching with SanctumQuery
7. RAG — Retrieval-Augmented Generation
8. Attaching to a Paladin
9. Docker Setup (Qdrant)
10. config.yml Reference
11. Error Handling
12. Best Practices

Sanctum vs. Garrison

Quick Start

Prerequisite: A running Qdrant instance. Use make dev to start the Docker Compose stack, or docker run -p 6334:6334 qdrant/qdrant.

use paladin_memory::sanctum::QdrantSanctumAdapter;
use paladin_memory::services::rag_retrieval_service::RAGRetrievalService;
use paladin_core::platform::container::sanctum::{Memory, MemoryType, SanctumEntry};
use paladin_ports::output::sanctum_port::{SanctumPort, SanctumQuery};
use paladin_ports::output::embedding_port::EmbeddingPort;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let sanctum = Arc::new(
        QdrantSanctumAdapter::new("http://localhost:6334", "memories", 1536).await?
    );
    let embedder: Arc<dyn EmbeddingPort> = Arc::new(openai_embedder());

    // Store a memory
    let content = "Rust's borrow checker prevents data races at compile time.";
    let embedding = embedder.embed_text(content).await?;

    let memory = Memory::builder("agent-1".to_string(), content.to_string())
        .memory_type(MemoryType::Semantic)
        .importance(0.9)
        .build()?;

    let entry = SanctumEntry {
        memory,
        embedding: embedding.vector.clone(),
        dimension: embedding.vector.len(),
    };

    sanctum.store(entry).await?;

    // Semantic search
    let query_vec = embedder.embed_text("memory safety in Rust").await?.vector;
    let results = sanctum.search(SanctumQuery {
        embedding: query_vec,
        limit: 5,
        filter: None,
    }).await?;

    for r in results {
        println!("[score: {:.3}] {}", r.score, r.entry.memory.content);
    }
    Ok(())
}

Sanctum Adapters

Both in crates/paladin-memory/src/sanctum/.

QdrantSanctumAdapter

Production-grade vector store with HNSW indexing.

use paladin_memory::sanctum::QdrantSanctumAdapter;

let sanctum = QdrantSanctumAdapter::new(
    "http://localhost:6334",  // Qdrant URL
    "paladin_memories",       // Collection name
    1536,                     // Vector dimension (match your embedding model)
).await?;

The collection is auto-created if it does not exist.

InMemorySanctum

Fast, ephemeral vector store for development and testing.

use paladin_memory::sanctum::InMemorySanctumAdapter;

let sanctum = InMemorySanctumAdapter::new(1536);

SanctumPort Trait

#[async_trait]
pub trait SanctumPort: Send + Sync {
    /// Store a single memory with its embedding
    async fn store(&self, entry: SanctumEntry) -> Result<(), SanctumError>;

    /// Store multiple memories in a single batch operation
    async fn store_batch(&self, entries: Vec<SanctumEntry>) -> Result<(), SanctumError>;

    /// Search for semantically similar memories
    async fn search(&self, query: SanctumQuery) -> Result<Vec<SanctumSearchResult>, SanctumError>;

    /// Delete a memory by its ID
    async fn delete(&self, id: &str) -> Result<bool, SanctumError>;
}

SanctumEntry and Memory Types

use paladin_core::platform::container::sanctum::{Memory, MemoryType, SanctumEntry};

let memory = Memory::builder("paladin-id".to_string(), "content here".to_string())
    .memory_type(MemoryType::Semantic)   // Semantic | Episodic | Procedural
    .importance(0.8)                      // 0.0–1.0
    .add_metadata("topic".to_string(), serde_json::json!("rust"))
    .build()?;

let entry = SanctumEntry {
    memory,
    embedding: vec![0.1_f32; 1536],  // Your embedding vector
    dimension: 1536,
};

MemoryType variants:

Searching with SanctumQuery

use paladin_ports::output::sanctum_port::{SanctumQuery, SanctumFilter};

// Basic similarity search
let results = sanctum.search(SanctumQuery {
    embedding: query_vec,
    limit: 10,
    filter: None,
}).await?;

// With metadata filter
let results = sanctum.search(SanctumQuery {
    embedding: query_vec,
    limit: 5,
    filter: Some(SanctumFilter {
        paladin_id: Some("agent-1".to_string()),
        memory_type: Some(MemoryType::Semantic),
        min_importance: Some(0.7),
        ..Default::default()
    }),
}).await?;

// Each result contains:
// result.entry   → the SanctumEntry
// result.score   → cosine similarity (0.0–1.0, higher = more similar)

RAG — Retrieval-Augmented Generation

The RAGRetrievalService in crates/paladin-memory/src/services/rag_retrieval_service.rs automates memory retrieval and injection into the Paladin's prompt context:

use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin_ports::output::sanctum_port::SanctumPort;
use paladin_ports::output::embedding_port::EmbeddingPort;
use std::sync::Arc;

let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are a knowledgeable assistant.")
    .with_sanctum(sanctum_port)       // Vector store
    .with_embedding_port(embedder)    // Embedding provider
    .build()
    .await?;

When Sanctum and an embedding port are both attached, the Paladin will automatically:

1. Embed the user's input query.
2. Retrieve the top-K most similar memories from Sanctum.
3. Prepend retrieved context to the prompt before the LLM call.
4. Extract and store important information from the response.

The RAG retrieval config is controlled via config.yml:

rag:
  enabled: true
  top_k: 5
  min_score: 0.7
  inject_into_prompt: true

Attaching to a Paladin

use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin_memory::sanctum::QdrantSanctumAdapter;
use std::sync::Arc;

let sanctum = Arc::new(
    QdrantSanctumAdapter::new("http://localhost:6334", "memories", 1536).await?
);
let embedder = Arc::new(openai_embedder());

let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are a knowledge-augmented assistant.")
    .with_sanctum(sanctum)
    .with_embedding_port(embedder)
    .build()
    .await?;

Docker Setup (Qdrant)

The development Docker Compose stack includes Qdrant:

make dev           # Starts Redis, MinIO, MySQL, and Qdrant
# or individually:
docker run -p 6334:6334 -p 6333:6333 qdrant/qdrant

Default connection: http://localhost:6334 (gRPC) / http://localhost:6333 (REST dashboard).

config.yml Reference

sanctum:
  type: qdrant           # "qdrant" or "in_memory"
  url: "http://localhost:6334"
  collection: paladin_memories
  vector_dimension: 1536 # Must match embedding model output dimension

rag:
  enabled: true
  top_k: 5               # Number of similar memories to retrieve
  min_score: 0.7         # Minimum cosine similarity threshold
  inject_into_prompt: true

memory_extraction:
  enabled: true
  strategy: selective    # "all" or "selective"

Error Handling

SanctumError variants:

Best Practices

• Match dimensions — set vector_dimension to exactly the output size of your embedding model (OpenAI text-embedding-3-small = 1536, text-embedding-3-large = 3072).
• Use store_batch() when loading a knowledge base — it is significantly faster than individual store() calls.
• Set min_score in SanctumQuery to filter out low-quality matches; 0.7 is a good starting point.
• Separate collections per agent or per use-case to avoid cross-contamination in multi-agent systems.
• Use InMemorySanctum in tests to avoid requiring a running Qdrant instance.

Herald Output Formatting

The Herald system provides pluggable output formatters for Paladin and Battalion execution results. A Herald transforms a PaladinResult or BattalionResult into a human-readable or machine-readable string — JSON, Markdown, or ASCII table.

The Herald trait is defined in crates/paladin-core/src/platform/container/herald.rs. Adapters are in src/infrastructure/adapters/herald/.

Table of Contents

1. Overview
2. Available Heralds
3. Herald Trait
4. Attaching to a Paladin
5. Attaching to a Battalion Service
6. Custom Herald Implementation
7. Streaming Output
8. Error Handling

Overview

Available Heralds

JsonHerald

Serialises PaladinResult and BattalionResult to JSON.

use paladin::infrastructure::adapters::herald::JsonHerald;
use paladin::infrastructure::adapters::herald::json_herald::JsonHeraldConfig;

// Default: pretty = true, include_metadata = true
let herald = JsonHerald::new();

// Compact JSON without metadata
let herald = JsonHerald::with_config(JsonHeraldConfig {
    pretty: false,
    include_metadata: false,
});

let json_str = herald.format_paladin_result(&result)?;
// {"output": "...", "token_count": 150, "execution_time_ms": 1230, ...}

MarkdownHerald

Formats results with Markdown headings, status badges, and code blocks. Supports ANSI colour codes for terminal output.

use paladin::infrastructure::adapters::herald::MarkdownHerald;
use paladin::infrastructure::adapters::herald::markdown_herald::MarkdownHeraldConfig;

// Default: auto-detects terminal colour support
let herald = MarkdownHerald::new();

// Custom: force no colours, H1 headings
let herald = MarkdownHerald::with_config(MarkdownHeraldConfig {
    include_colors: false,
    heading_level: 1,
});

TableHerald

Renders results as ASCII tables using comfy-table.

use paladin::infrastructure::adapters::herald::TableHerald;
use paladin::infrastructure::adapters::herald::table_herald::TableHeraldConfig;

// Default configuration
let herald = TableHerald::default();

// Custom: 80-char column width, rounded borders
let herald = TableHerald::new(TableHeraldConfig {
    max_column_width: 80,
    border_style: "rounded".to_string(),
});

Herald Trait

pub trait Herald: Send + Sync {
    /// Format a completed Paladin result
    fn format_paladin_result(&self, result: &PaladinResult) -> Result<String, HeraldError>;

    /// Format a completed Battalion result
    fn format_battalion_result(&self, result: &BattalionResult) -> Result<String, HeraldError>;

    /// Format a streaming chunk — returns Some(String) when output is ready to emit,
    /// or None if the Herald is buffering
    fn format_stream_chunk(&self, chunk: &StreamChunk) -> Result<Option<String>, HeraldError>;
}

Attaching to a Paladin

use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin::infrastructure::adapters::herald::JsonHerald;
use paladin_core::platform::container::herald::Herald;
use std::sync::Arc;

let herald: Arc<dyn Herald> = Arc::new(JsonHerald::new());

let paladin = PaladinBuilder::new(llm_port)
    .system_prompt("You are an API assistant.")
    .with_herald(herald)
    .build()
    .await?;

let result = paladin.execute("List all Rust 2024 edition features").await?;
// result.output is already formatted as JSON
println!("{}", result.output);

Attaching to a Battalion Service

Formation, Phalanx, and other services accept a Herald via .with_herald():

use paladin_battalion::phalanx_service::PhalanxExecutionService;
use paladin::infrastructure::adapters::herald::MarkdownHerald;
use std::sync::Arc;

let service = PhalanxExecutionService::new(paladin_port)
    .with_herald(Arc::new(MarkdownHerald::new()));

let result = service.execute(&phalanx, "Analyse this dataset").await?;
println!("{}", result.output);

Custom Herald Implementation

Implement the Herald trait to create a bespoke formatter:

use paladin_core::platform::container::herald::{
    Herald, HeraldError, PaladinResult, BattalionResult, StreamChunk,
};

pub struct CsvHerald;

impl Herald for CsvHerald {
    fn format_paladin_result(&self, result: &PaladinResult) -> Result<String, HeraldError> {
        Ok(format!(
            "{},{},{},{:?}\n",
            result.output.replace(',', ";"),
            result.token_count,
            result.execution_time_ms,
            result.stop_reason,
        ))
    }

    fn format_battalion_result(&self, result: &BattalionResult) -> Result<String, HeraldError> {
        Ok(result.output.clone())
    }

    fn format_stream_chunk(&self, chunk: &StreamChunk) -> Result<Option<String>, HeraldError> {
        Ok(Some(chunk.text.clone()))
    }
}

Streaming Output

Use format_stream_chunk() during execute_stream():

use paladin_ports::output::paladin_port::PaladinStreamChunk;
use paladin_core::platform::container::herald::Herald;

let mut stream = paladin.execute_stream("Generate a long report").await?;
let herald = MarkdownHerald::new();

while let Some(chunk_result) = stream.recv().await {
    match chunk_result {
        Ok(chunk) => {
            // chunk.text is raw text; wrap in a StreamChunk for the Herald
            if let Some(formatted) = herald.format_stream_chunk(&chunk.into())? {
                print!("{}", formatted);
            }
            if chunk.is_final { break; }
        }
        Err(e) => eprintln!("Stream error: {}", e),
    }
}

Error Handling

HeraldError variants from paladin_core::platform::container::herald_error:

Maneuver: Flow DSL Orchestration

Declarative multi-agent workflows with dynamic execution patterns

Table of Contents

1. Overview
2. Quick Start
3. Flow DSL Syntax
4. Execution Patterns
5. Configuration
6. CLI Commands
7. Visualization
8. Error Handling
9. Performance
10. Best Practices
11. API Reference
12. Troubleshooting

Overview

Maneuver is a declarative Battalion orchestration pattern that uses a Flow DSL (Domain-Specific Language) to define complex agent execution patterns. Unlike other Battalion patterns that require explicit code, Maneuver allows you to express workflows as simple text expressions.

Key Features

• Declarative Syntax: Define workflows as text expressions (agent1 -> agent2)
• Mixed Patterns: Combine sequential and parallel execution in a single flow
• Visual Feedback: ASCII and Mermaid.js visualization of flow graphs
• Type-Safe Parsing: Compile-time validation of flow expressions
• Commander Integration: Automatic pattern detection for "flow" keywords

Comparison with Other Patterns

Quick Start

Installation

Maneuver is included in paladin-battalion. Add it to your workspace:

[dependencies]
paladin-battalion = { version = "0.5.0", path = "crates/paladin-battalion" }
tokio = { version = "1.0", features = ["full"] }

Basic Example

use paladin_battalion::maneuver::service::ManeuverExecutionService;
use paladin_battalion::maneuver::Maneuver;
use paladin_battalion::maneuver::parser::FlowParser;
use std::collections::HashMap;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Define flow using DSL
    let flow = FlowParser::parse("analyzer -> summarizer -> reviewer")?;

    // Create Paladins
    let mut agents = HashMap::new();
    agents.insert("analyzer".to_string(), create_paladin("analyzer", "Analyze input"));
    agents.insert("summarizer".to_string(), create_paladin("summarizer", "Summarize"));
    agents.insert("reviewer".to_string(), create_paladin("reviewer", "Final review"));

    // Create Maneuver
    let maneuver = Maneuver::new("doc-workflow", agents, flow, Default::default())?;

    // Execute
    let service = ManeuverExecutionService::new(Arc::new(paladin_port));
    let result = service.execute(&maneuver, "Document to process").await?;

    println!("Final output: {}", result.final_output);
    Ok(())
}

CLI Quick Start

# Create a Maneuver configuration
paladin battalion new my-workflow --type maneuver -o workflow.yaml

# Visualize the flow
paladin maneuver visualize -c workflow.yaml --format ascii

# Validate configuration
paladin maneuver validate -c workflow.yaml --verbose

# Execute the workflow
paladin battalion run -c workflow.yaml -t maneuver -i "Process this input"

Flow DSL Syntax

The Flow DSL uses a simple, intuitive syntax for defining agent execution patterns.

Basic Syntax

Sequential Execution

agent1 -> agent2 -> agent3

Output from agent1 flows as input to agent2, then to agent3.

Parallel Execution

(agent1, agent2)

Both agent1 and agent2 execute concurrently with the same input.

Note: Use commas (,) for parallel, not pipes (|).

Nested Patterns

agent1 -> (agent2, agent3) -> agent4

1. agent1 executes first
2. Output flows to both agent2 and agent3 (parallel)
3. Combined output flows to agent4

Syntax Rules

Valid Examples

# Simple sequential
agent1 -> agent2

# Simple parallel
(agent1, agent2)

# Mixed nested
start -> (analyzer, reviewer) -> end

# Complex workflow
intake -> (technical, business, security) -> synthesis -> review

# Deep nesting
a -> (b -> (c, d), e) -> f

Invalid Syntax

# ❌ Pipe operator (use comma instead)
(agent1 | agent2)

# ❌ Missing parentheses for parallel
agent1 -> agent2, agent3

# ❌ Spaces in agent names
my agent -> another agent

# ❌ Empty groups
() -> agent1

# ❌ Trailing operators
agent1 ->

Execution Patterns

Sequential Pattern

Flow: agent1 -> agent2 -> agent3

Behavior:

1. Execute agent1 with initial input
2. Pass agent1 output to agent2 as input
3. Pass agent2 output to agent3 as input
4. Return agent3 output as final result

Use Cases:

• Data transformation pipelines
• Multi-stage analysis
• Progressive refinement

Example:

// Flow: "extractor -> translator -> formatter"
let flow = FlowParser::parse("extractor -> translator -> formatter")?;

// Input: "Extract data from: <raw_text>"
// extractor output: "Data: {...}"
// translator output: "Translated: {...}"
// formatter output: "Formatted report: {...}" (final)

Parallel Pattern

Flow: (agent1, agent2, agent3)

Behavior:

1. Execute all agents concurrently with same input
2. Wait for all to complete
3. Combine outputs (concatenation or custom logic)
4. Return combined result

Use Cases:

• Multi-perspective analysis
• Expert panel reviews
• Parallel processing

Example:

// Flow: "(tech_reviewer, business_reviewer, security_reviewer)"
let flow = FlowParser::parse("(tech_reviewer, business_reviewer, security_reviewer)")?;

// All receive: "Review this proposal: {...}"
// Output combines all three perspectives

Nested Pattern

Flow: agent1 -> (agent2, agent3) -> agent4

Behavior:

1. Execute agent1 with initial input
2. Pass output to both agent2 and agent3 (parallel)
3. Wait for both to complete
4. Combine their outputs
5. Pass combined result to agent4
6. Return agent4 output as final result

Use Cases:

• Divide-and-conquer workflows
• Multi-faceted analysis with synthesis
• Complex decision trees

Example:

// Flow: "analyzer -> (summarizer, translator) -> reviewer"
let flow = FlowParser::parse("analyzer -> (summarizer, translator) -> reviewer")?;

// 1. analyzer processes input
// 2. summarizer + translator work in parallel on analysis
// 3. reviewer synthesizes both outputs into final result

Execution Order Visualization

Sequential: agent1 → agent2 → agent3
           t₀      t₁      t₂

Parallel:   agent1
           ↙      ↘
       agent2    agent3
           ↘      ↙
         (combine)

Nested:     agent1
              ↓
          ┌───┴───┐
      agent2   agent3
          └───┬───┘
           agent4

Configuration

Maneuver Configuration

use paladin_battalion::maneuver::{ManeuverConfig, ErrorStrategy, OutputFormat};
use std::time::Duration;

let config = ManeuverConfig::new()
    .with_error_strategy(ErrorStrategy::ContinueParallel)
    .with_output_format(OutputFormat::Concatenate)
    .with_pass_output_as_input(true)
    .with_timeout(Duration::from_secs(300))
    .with_timing_metrics(true);

let maneuver = Maneuver::new("workflow", agents, flow, config)?;

Error Strategies

pub enum ErrorStrategy {
    /// Stop immediately on first error
    FailFast,

    /// Continue parallel branches but fail sequential chains on error
    ContinueParallel,

    /// Log errors but continue execution regardless
    IgnoreErrors,
}

When to Use:

• FailFast: Critical workflows where any failure invalidates the result
• ContinueParallel: Parallel sections can fail independently
• IgnoreErrors: Best-effort workflows, collect whatever partial results are available

Output Formats

pub enum OutputFormat {
    /// Concatenate all outputs with newlines (default)
    Concatenate,

    /// JSON array with each agent's output as an element
    JsonArray,
}

Example Outputs:

// Concatenate (default)
"Output from agent1\n---\nOutput from agent2\n---\nOutput from agent3"

// JsonArray
r#"["output from agent1", "output from agent2", "output from agent3"]"#

YAML Configuration

type: maneuver
name: "document-workflow"

# Flow expression using DSL
flow: "analyzer -> (summarizer, translator) -> reviewer"

# Available Paladins (must match names in flow)
paladins:
  - inline:
      name: "analyzer"
      system_prompt: "Analyze the input document"
      model: "gpt-4"
      temperature: 0.7
      provider:
        type: openai

  - inline:
      name: "summarizer"
      system_prompt: "Create a concise summary"
      model: "gpt-4"
      temperature: 0.5
      provider:
        type: openai

  - inline:
      name: "translator"
      system_prompt: "Translate to simple language"
      model: "gpt-4"
      temperature: 0.5
      provider:
        type: openai

  - inline:
      name: "reviewer"
      system_prompt: "Final review and synthesis"
      model: "gpt-4"
      temperature: 0.6
      provider:
        type: openai

# Optional: visualize before execution
visualize: "ascii"

CLI Commands

Create Maneuver Configuration

paladin battalion new my-workflow --type maneuver --output workflow.yaml

Creates a template YAML file with example flow and agents.

Visualize Flow

# ASCII tree visualization
paladin maneuver visualize -c workflow.yaml --format ascii

# Mermaid flowchart (for documentation)
paladin maneuver visualize -c workflow.yaml --format mermaid

# Save to file
paladin maneuver visualize -c workflow.yaml --format ascii -o flow.txt

Output Example (ASCII):

└─> analyzer
    ├─> [PARALLEL]
    │   ├─> summarizer
    │   └─> translator
    └─> reviewer

Output Example (Mermaid):

flowchart LR
    agent_analyzer
    agent_analyzer --> parallel_1[Parallel]
    parallel_1 --> agent_summarizer
    parallel_1 --> agent_translator
    parallel_1 --> agent_reviewer

Validate Configuration

# Basic validation
paladin maneuver validate -c workflow.yaml

# Verbose validation with detailed output
paladin maneuver validate -c workflow.yaml --verbose

Validates:

• Flow expression syntax
• All agents referenced in flow exist in config
• Paladin configuration structure
• Provider settings

Execute Maneuver

# Interactive execution
paladin battalion run -c workflow.yaml -t maneuver

# With input provided
paladin battalion run -c workflow.yaml -t maneuver -i "Process this text"

# Save output to file
paladin battalion run -c workflow.yaml -t maneuver -i "Input" -o result.json

# Verbose execution
paladin battalion run -c workflow.yaml -t maneuver -v

Visualization

ASCII Tree Format

Perfect for terminal output and debugging:

└─> intake
    ├─> [PARALLEL]
    │   ├─> technical
    │   ├─> business
    │   └─> security
    └─> synthesis
        └─> review

Features:

• Box-drawing characters (├─>, └─>, │)
• Clear hierarchy visualization
• Sequential and parallel markers
• Nested structure representation

Mermaid Flowchart Format

Ideal for documentation and presentations:

flowchart LR
    agent_intake
    agent_intake --> parallel_1[Parallel]
    parallel_1 --> agent_technical
    parallel_1 --> agent_business
    parallel_1 --> agent_security
    parallel_1 --> agent_synthesis
    agent_synthesis --> agent_review

Features:

• Web-ready visualization
• Integrates with GitHub/GitLab/documentation tools
• Professional diagram quality
• Exportable to SVG/PNG

Programmatic Visualization

use paladin_battalion::maneuver::visualizer::{FlowVisualizer, VisualizationFormat};

let flow = FlowParser::parse("a -> (b, c) -> d")?;

// ASCII visualization
let ascii = FlowVisualizer::to_ascii(&flow);
println!("{}", ascii);

// Mermaid visualization
let mermaid = FlowVisualizer::to_mermaid(&flow);
println!("{}", mermaid);

// Using format parameter
let viz = FlowVisualizer::visualize(&flow, VisualizationFormat::Ascii);

Error Handling

Validation Errors

use paladin_battalion::maneuver::parser::FlowParseError;

match FlowParser::parse("agent1 -> (agent2 | agent3)") {
    Ok(flow) => { /* Success */ },
    Err(FlowParseError::InvalidCharacter { position, character }) => {
        eprintln!("Invalid character '{}' at position {}", character, position);
        // Error: Invalid character '|' at position 17
    },
    Err(e) => eprintln!("Parse error: {}", e),
}

Execution Errors

use paladin_battalion::maneuver::ManeuverError;

match service.execute(&maneuver, input).await {
    Ok(result) => println!("Success: {}", result.final_output),
    Err(ManeuverError::AgentNotFound(name)) => {
        eprintln!("Agent '{}' not found in configuration", name);
    },
    Err(ManeuverError::ExecutionError(msg)) => {
        eprintln!("Execution failed: {}", msg);
    },
    Err(e) => eprintln!("Error: {}", e),
}

Error Recovery

// Configure error handling strategy
let config = ManeuverConfig::new()
    .with_error_strategy(ErrorStrategy::IgnoreErrors);

// Execution continues despite failures
let result = service.execute(&maneuver, input).await?;

// Check status
match result.status {
    ExecutionStatus::Success => println!("All agents succeeded"),
    ExecutionStatus::PartialSuccess => println!("Some agents failed"),
    ExecutionStatus::Failed => println!("Execution failed"),
}

// Inspect individual outputs
for (agent, output) in result.step_outputs {
    if output.is_empty() {
        println!("Agent {} failed", agent);
    }
}

Performance

Benchmarks

Based on battalion_benchmarks.rs:

Optimization Tips

1. Minimize Sequential Chains

❌ Slow: a -> b -> c -> d -> e -> f (6 sequential calls)

✅ Fast: a -> (b, c, d) -> e (3 stages total)

2. Use Parallel Where Possible

// Slow: Sequential when order doesn't matter
"tech_review -> security_review -> legal_review"

// Fast: Parallel independent reviews
"(tech_review, security_review, legal_review)"

3. Configure Timeouts

let config = ManeuverConfig::new()
    .with_timeout(Duration::from_secs(120))  // Per-agent timeout
    .with_error_strategy(ErrorStrategy::ContinueParallel);  // Don't wait for failures

4. Optimize Agent Prompts

• Keep system prompts concise
• Use lower max_loops values when possible
• Set appropriate temperature values

5. Monitor Timing Metrics

let config = ManeuverConfig::new()
    .with_collect_timing_metrics(true);

let result = service.execute(&maneuver, input).await?;

if let Some(metrics) = result.timing_metrics {
    for (agent, duration) in metrics {
        println!("{}: {}ms", agent, duration.as_millis());
    }
}

Best Practices

1. Flow Design

Keep Flows Simple

// ✅ Good: Clear, easy to understand
"intake -> analyze -> decide"

// ❌ Bad: Too complex, hard to debug
"a -> (b -> (c, d -> (e, f)), g -> (h, i)) -> j"

Use Descriptive Names

// ✅ Good: Clear purpose
"document_analyzer -> sentiment_classifier -> report_generator"

// ❌ Bad: Cryptic names
"agent1 -> agent2 -> agent3"

2. Agent Configuration

Specialize Agents

Each agent should have a clear, focused responsibility:

- name: "analyzer"
  system_prompt: "Analyze technical feasibility only. Focus on implementation challenges."

- name: "risk_assessor"
  system_prompt: "Assess security and privacy risks only."

- name: "synthesizer"
  system_prompt: "Combine technical analysis and risk assessment into recommendation."

Use Consistent Naming

Match agent names in flow expression exactly:

// Flow uses: analyzer, summarizer, reviewer
flow: "analyzer -> summarizer -> reviewer"

// Paladins must use same names:
agents.insert("analyzer", ...);
agents.insert("summarizer", ...);
agents.insert("reviewer", ...);

3. Error Handling

Always Handle Errors

// ✅ Good: Explicit error handling
match service.execute(&maneuver, input).await {
    Ok(result) => process_result(result),
    Err(ManeuverError::AgentNotFound(name)) => {
        log_error!("Missing agent: {}", name);
        return default_result();
    },
    Err(e) => {
        log_error!("Execution failed: {}", e);
        retry_with_fallback();
    },
}

// ❌ Bad: Unwrapping
let result = service.execute(&maneuver, input).await.unwrap();

Choose Appropriate Strategy

// Critical workflows: fail fast
let config = ManeuverConfig::new()
    .with_error_strategy(ErrorStrategy::FailFast);

// Best-effort workflows: collect partial results
let config = ManeuverConfig::new()
    .with_error_strategy(ErrorStrategy::IgnoreErrors);

4. Testing

Validate Flows Early

#[test]
fn test_workflow_validation() {
    let flow = FlowParser::parse("analyzer -> summarizer").unwrap();

    let mut agents = HashMap::new();
    agents.insert("analyzer".to_string(), create_test_agent("analyzer"));
    agents.insert("summarizer".to_string(), create_test_agent("summarizer"));

    let result = Maneuver::new("test", agents, flow, Default::default());
    assert!(result.is_ok());
}

Test Visualizations

#[test]
fn test_flow_visualization() {
    let flow = FlowParser::parse("a -> (b, c)").unwrap();
    let ascii = FlowVisualizer::to_ascii(&flow);

    assert!(ascii.contains("PARALLEL"));
    assert!(ascii.contains("a"));
    assert!(ascii.contains("b"));
    assert!(ascii.contains("c"));
}

5. Documentation

Document Complex Flows

# Flow explanation:
# 1. Intake agent validates and normalizes input
# 2. Three specialists analyze in parallel:
#    - Technical feasibility
#    - Business value
#    - Security implications
# 3. Synthesis agent combines all perspectives
# 4. Final review for quality assurance
flow: "intake -> (technical, business, security) -> synthesis -> review"

API Reference

Core Types

FlowParser

pub struct FlowParser;

impl FlowParser {
    /// Parse a flow expression from text
    pub fn parse(input: &str) -> Result<FlowExpression, FlowParseError>
}

FlowExpression

#[derive(Debug, Clone, PartialEq, Eq)]
pub enum FlowExpression {
    /// Single agent execution
    Agent(String),

    /// Sequential execution (agent₁ → agent₂ → ...)
    Sequential(Vec<FlowExpression>),

    /// Parallel execution (agent₁, agent₂, ...)
    Parallel(Vec<FlowExpression>),
}

impl FlowExpression {
    /// Get all agent names referenced in this expression
    pub fn agent_names(&self) -> Vec<String>
}

Maneuver

pub struct Maneuver {
    pub name: String,
    pub agents: HashMap<String, Paladin>,
    pub flow: FlowExpression,
    pub config: ManeuverConfig,
}

impl Maneuver {
    /// Create a new Maneuver with validation
    pub fn new(
        name: impl Into<String>,
        agents: HashMap<String, Paladin>,
        flow: FlowExpression,
        config: ManeuverConfig,
    ) -> Result<Self, ManeuverError>

    /// Validate that all flow agents exist
    pub fn validate(&self) -> Result<(), ManeuverError>
}

ManeuverConfig

pub struct ManeuverConfig {
    pub error_strategy: ErrorStrategy,
    pub output_format: OutputFormat,
    pub pass_output_as_input: bool,
    pub timeout: Option<Duration>,
    pub collect_timing_metrics: bool,
    pub detailed_observability: bool,
}

impl ManeuverConfig {
    pub fn new() -> Self
    pub fn with_error_strategy(self, strategy: ErrorStrategy) -> Self
    pub fn with_output_format(self, format: OutputFormat) -> Self
    pub fn with_timeout(self, timeout: Duration) -> Self
}

ManeuverResult

pub struct ManeuverResult {
    /// Final aggregated output
    pub final_output: String,

    /// Individual agent outputs
    pub step_outputs: HashMap<String, String>,

    /// Execution order
    pub execution_order: Vec<String>,

    /// Per-agent timing (if enabled)
    pub timing_metrics: Option<HashMap<String, Duration>>,

    /// Execution status
    pub status: ExecutionStatus,
}

ManeuverExecutionService

pub struct ManeuverExecutionService {
    paladin_port: Arc<dyn PaladinPort>,
}

impl ManeuverExecutionService {
    pub fn new(paladin_port: Arc<dyn PaladinPort>) -> Self

    pub async fn execute(
        &self,
        maneuver: &Maneuver,
        input: &str,
    ) -> Result<ManeuverResult, ManeuverError>
}

Visualization

FlowVisualizer

pub struct FlowVisualizer;

impl FlowVisualizer {
    /// Generate ASCII tree visualization
    pub fn to_ascii(flow: &FlowExpression) -> String

    /// Generate Mermaid flowchart
    pub fn to_mermaid(flow: &FlowExpression) -> String

    /// Generate visualization in specified format
    pub fn visualize(flow: &FlowExpression, format: VisualizationFormat) -> String
}

pub enum VisualizationFormat {
    Ascii,
    Mermaid,
}

Troubleshooting

Common Issues

1. Parse Error: Invalid Character '|'

Problem: Using pipe operator for parallel execution

// ❌ Wrong
let flow = FlowParser::parse("(agent1 | agent2)")?;

Solution: Use comma instead

// ✅ Correct
let flow = FlowParser::parse("(agent1, agent2)")?;

2. AgentNotFound Error

Problem: Agent name in flow doesn't match configured agents

// Flow references "analyzer"
let flow = FlowParser::parse("analyzer -> summarizer")?;

// But agent is named "Analyzer" (different case)
agents.insert("Analyzer".to_string(), paladin);

Solution: Use exact same names

// ✅ Correct - exact match
agents.insert("analyzer".to_string(), paladin);

3. Missing Parentheses for Parallel

Problem: Forgetting parentheses around parallel agents

// ❌ Wrong - will be parsed as "agent1 -> agent2", "agent3"
let flow = FlowParser::parse("agent1 -> agent2, agent3")?;

Solution: Always use parentheses for parallel

// ✅ Correct
let flow = FlowParser::parse("agent1 -> (agent2, agent3)")?;

4. Timeout Errors

Problem: Agents taking too long to execute

// Default timeout may be too short
let config = ManeuverConfig::default();  // 300s default

Solution: Increase timeout for slow workflows

// ✅ Longer timeout
let config = ManeuverConfig::new()
    .with_timeout(Duration::from_secs(600));  // 10 minutes

5. Partial Results from Parallel Execution

Problem: Some agents fail in parallel execution

Solution: Use appropriate error strategy

// Continue despite failures
let config = ManeuverConfig::new()
    .with_error_strategy(ErrorStrategy::ContinueParallel);

let result = service.execute(&maneuver, input).await?;

// Check which agents succeeded
for (agent, output) in result.step_outputs {
    if !output.is_empty() {
        println!("{} succeeded: {}", agent, output);
    }
}

Debugging Tips

1. Enable Verbose Logging

env_logger::init();  // In main()

// Set RUST_LOG=debug
// Will show detailed execution trace

2. Visualize Before Executing

paladin maneuver visualize -c config.yaml --format ascii

Visual inspection often reveals flow logic issues.

3. Validate Configuration

paladin maneuver validate -c config.yaml --verbose

Catches configuration mismatches before execution.

4. Check Timing Metrics

let config = ManeuverConfig::new()
    .with_collect_timing_metrics(true);

let result = service.execute(&maneuver, input).await?;

if let Some(metrics) = result.timing_metrics {
    for (agent, duration) in metrics {
        if duration > Duration::from_secs(60) {
            println!("⚠️  {} took {}s", agent, duration.as_secs());
        }
    }
}

5. Inspect Individual Outputs

let result = service.execute(&maneuver, input).await?;

// Check each agent's output
for agent in result.execution_order {
    if let Some(output) = result.step_outputs.get(&agent) {
        println!("\n=== {} ===", agent);
        println!("{}", output);
    }
}

Getting Help

• Documentation: https://github.com/DF3NDR/paladin-dev-env/docs
• Issues: https://github.com/DF3NDR/paladin-dev-env/issues
• Discussions: https://github.com/DF3NDR/paladin-dev-env/discussions
• Examples: examples/ directory in repository

Advanced Topics

Custom Output Formatting

use paladin_battalion::maneuver::OutputFormat;

// Implement custom aggregation logic
let config = ManeuverConfig::new()
    .with_output_format(OutputFormat::JsonArray);

// Result will be JSON:
// {"agent1": "output1", "agent2": "output2"}

Integration with Commander

Commander automatically detects Maneuver patterns:

use paladin_battalion::commander::CommanderBuilder;
use paladin_core::platform::container::battalion::BattalionStrategy;

// Maneuver is explicit-only — it is NOT selected by Auto mode.
// You must explicitly set BattalionStrategy::Maneuver and provide a flow expression.
let commander = CommanderBuilder::new(paladin_port)
    .strategy(BattalionStrategy::Maneuver)
    .paladins(paladins)
    .flow("agent1 -> agent2 -> agent3".to_string())
    .build()?;

let result = commander.execute("Process this document").await?;

Performance Tuning

For high-throughput systems:

// Minimize overhead
let config = ManeuverConfig::new()
    .with_collect_timing_metrics(false)  // Disable if not needed
    .with_detailed_observability(false)  // Reduce logging
    .with_error_strategy(ErrorStrategy::FailFast);  // Fast failure

// Use connection pooling for LLM providers
// Pre-validate flows at startup
// Cache parsed flow expressions

Last Updated: February 2026 Version: 0.1.0 Status: Production Ready

Paladin Configuration Guide

This guide covers how to configure Paladin agents for optimal performance, from basic setup to advanced tuning.

Table of Contents

• Basic Configuration
• System Prompt Best Practices
• Model Selection
• Temperature and Sampling
• Stop Words and Termination
• Timeout and Retry Settings
• Advanced Configuration

Basic Configuration

Minimal Setup

use paladin::prelude::*;

let paladin = PaladinBuilder::new(llm_adapter)
    .name("Assistant")
    .system_prompt("You are a helpful assistant.")
    .build()?;

Common Configuration

let paladin = PaladinBuilder::new(llm_adapter)
    .name("DataAnalyst")
    .system_prompt("You are an expert data analyst. Provide clear, data-driven insights.")
    .model("gpt-4")
    .temperature(0.7)
    .max_loops(5)
    .timeout_seconds(120)
    .build()?;

Full Configuration

let paladin = PaladinBuilder::new(llm_adapter)
    .name("ResearchAssistant")
    .system_prompt("You are a research assistant specializing in academic papers.")
    .user_name("Researcher")
    .model("gpt-4-turbo")
    .temperature(0.8)
    .max_loops(10)
    .add_stop_word("END").add_stop_word("STOP").add_stop_word("FINAL_ANSWER")
    .timeout_seconds(300)
    .retry_attempts(3)
    .with_garrison(garrison)
    .add_armament(search_tool)
    .add_armament(calculator_tool)
    .build()?;

System Prompt Best Practices

The system prompt defines your Paladin's behavior and capabilities. Follow these best practices:

1. Be Specific About Role

❌ Vague:

.system_prompt("You are helpful.")

✅ Specific:

.system_prompt("You are a senior software engineer specializing in Rust. \
                You provide code reviews focused on safety, performance, and idiomatic patterns.")

2. Define Output Format

.system_prompt("You are a JSON API. Always respond with valid JSON. \
                Structure: {\"status\": \"success|error\", \"data\": {...}, \"message\": \"...\"}  \
                Never include markdown code blocks or explanations outside the JSON.")

3. Set Boundaries

.system_prompt("You are a customer support agent for TechCorp. \
                - Only answer questions about our products and services \
                - Escalate billing questions to the finance team \
                - Do not provide medical, legal, or financial advice \
                - Be polite and professional at all times")

4. Include Examples (Few-Shot)

.system_prompt("You categorize customer feedback as: FEATURE_REQUEST, BUG_REPORT, or PRAISE. \
                \
                Examples: \
                Input: 'The app crashes when I upload large files' \
                Output: BUG_REPORT \
                \
                Input: 'It would be great to have dark mode' \
                Output: FEATURE_REQUEST \
                \
                Input: 'Love the new design!' \
                Output: PRAISE")

5. Specify Tone and Style

.system_prompt("You are a technical writer creating documentation for developers. \
                - Use clear, concise language \
                - Prefer active voice \
                - Include code examples \
                - Target audience: junior to mid-level developers \
                - Avoid jargon unless necessary")

Model Selection

Choose the right model for your use case:

OpenAI Models

// GPT-4 Turbo - Best for complex reasoning
.model("gpt-4-turbo")  // Latest turbo model
.model("gpt-4")        // Standard GPT-4

// GPT-3.5 - Fast and cost-effective
.model("gpt-3.5-turbo")  // Recommended for most tasks

When to use:

• GPT-4: Complex reasoning, code generation, detailed analysis
• GPT-3.5: Simple queries, classification, summarization

DeepSeek Models

// DeepSeek Chat - Strong coding capabilities
.model("deepseek-chat")

// DeepSeek Coder - Specialized for code
.model("deepseek-coder")

When to use:

• deepseek-chat: General purpose, good for multi-turn conversations
• deepseek-coder: Code generation, technical documentation

Anthropic Models

// Claude 3 Family
.model("claude-3-opus")    // Most capable
.model("claude-3-sonnet")  // Balanced
.model("claude-3-haiku")   // Fastest

When to use:

• Opus: Complex analysis, long documents, creative writing
• Sonnet: General purpose, good balance of speed and quality
• Haiku: Fast responses, simple queries, high throughput

Model Comparison

Temperature and Sampling

Temperature controls randomness in responses:

Temperature Scale

// 0.0 - Deterministic, focused (best for factual tasks)
.temperature(0.0)

// 0.3-0.5 - Slightly varied (good for classification)
.temperature(0.4)

// 0.7 - Balanced (general purpose)
.temperature(0.7)

// 0.9-1.0 - Creative, diverse (brainstorming, creative writing)
.temperature(0.9)

// >1.0 - Very random (experimental, not recommended)
.temperature(1.2)

Use Cases by Temperature

Example: Task-Specific Configuration

// Code Review - Deterministic
let code_reviewer = PaladinBuilder::new(llm_adapter)
    .system_prompt("Review Rust code for safety and best practices.")
    .temperature(0.2)
    .build()?;

// Content Writer - Creative
let writer = PaladinBuilder::new(llm_adapter)
    .system_prompt("Write engaging blog posts about technology.")
    .temperature(0.9)
    .build()?;

// Customer Support - Balanced
let support = PaladinBuilder::new(llm_adapter)
    .system_prompt("Help customers with product questions.")
    .temperature(0.7)
    .build()?;

Stop Words and Termination

Control when a Paladin stops generating:

Basic Stop Words

let paladin = PaladinBuilder::new(llm_adapter)
    .add_stop_word("END").add_stop_word("STOP").add_stop_word("###")
    .build()?;

Use Cases

1. Structured Output

// Stop at delimiter for parsing
.system_prompt("Generate a list of items. End with '---'")
.add_stop_word("---")

2. Multi-Step Reasoning

// Stop when final answer is reached
.system_prompt("Think step by step. When done, output FINAL_ANSWER: <answer>")
.add_stop_word("FINAL_ANSWER:")

3. Dialog Systems

// Stop at turn boundaries
.system_prompt("You are user A in a conversation. End each turn with [END_TURN]")
.add_stop_word("[END_TURN]")

Max Loops

Prevent infinite reasoning loops:

// Default: 3 loops
.max_loops(3)

// For simple tasks: 1 loop
.max_loops(1)

// For complex reasoning: 10+ loops
.max_loops(15)

What is a loop? A loop is one reasoning cycle: prompt → LLM → response → (optional tool calls) → repeat.

Timeout and Retry Settings

Timeout Configuration

use std::time::Duration;

let paladin = PaladinBuilder::new(llm_adapter)
    .timeout_seconds(60)  // 60 second timeout
    .build()?;

Recommended Timeouts:

• Simple queries: 30 seconds
• Complex reasoning: 120 seconds
• With tool calls: 300 seconds

Retry Configuration

let paladin = PaladinBuilder::new(llm_adapter)
    .retry_attempts(3)                        // Retry up to 3 times
    .build()?;

Error Handling

match paladin.execute(input).await {
    Ok(response) => println!("Success: {}", response.content),
    Err(PaladinError::Timeout(secs)) => {
        eprintln!("Request timed out after {} seconds", secs);
        // Increase timeout or simplify prompt
    }
    Err(PaladinError::LlmError(msg)) => {
        eprintln!("LLM error: {}", msg);
        // Check API key, rate limits, model availability
    }
    Err(PaladinError::MaxLoopsExceeded) => {
        eprintln!("Max reasoning loops exceeded");
        // Increase max_loops or refine system prompt
    }
    Err(e) => eprintln!("Other error: {}", e),
}

Advanced Configuration

Configuration from File

use paladin::config::ApplicationSettings;

let config = ApplicationSettings::load_from("config.yml")?;
let paladin = PaladinBuilder::from_config(&config.paladin)?;

config.yml:

paladin:
  name: "Assistant"
  system_prompt: "You are a helpful assistant."
  model: "gpt-4"
  temperature: 0.7
  max_loops: 5
  timeout_seconds: 120
  retry_attempts: 3
  stop_words:
    - "END"
    - "STOP"

Environment-Based Configuration

let model = std::env::var("PALADIN_MODEL").unwrap_or("gpt-3.5-turbo".to_string());
let temperature = std::env::var("PALADIN_TEMPERATURE")
    .ok()
    .and_then(|s| s.parse::<f32>().ok())
    .unwrap_or(0.7);

let paladin = PaladinBuilder::new(llm_adapter)
    .model(&model)
    .temperature(temperature)
    .build()?;

Dynamic Configuration

struct PaladinFactory;

impl PaladinFactory {
    fn create_for_task(task_type: &str, llm_adapter: Arc<dyn LlmPort>) -> Result<Paladin> {
        match task_type {
            "code_review" => Self::create_code_reviewer(llm_adapter),
            "creative_writing" => Self::create_writer(llm_adapter),
            "data_analysis" => Self::create_analyst(llm_adapter),
            _ => Self::create_default(llm_adapter),
        }
    }

    fn create_code_reviewer(llm_adapter: Arc<dyn LlmPort>) -> Result<Paladin> {
        PaladinBuilder::new(llm_adapter)
            .system_prompt("Expert Rust code reviewer")
            .temperature(0.2)
            .model("gpt-4")
            .build()
    }

    // ... other factory methods
}

Configuration Validation

let paladin = PaladinBuilder::new(llm_adapter)
    .temperature(0.7)
    .build()?;  // Validates configuration

// Manual validation
if let Err(e) = paladin.validate() {
    eprintln!("Invalid configuration: {}", e);
}

Configuration Checklist

Before deploying a Paladin, verify:

• System prompt is clear and specific
• Appropriate model selected for task
• Temperature suitable for use case (0.2 for factual, 0.9 for creative)
• Max loops set appropriately (1-3 for simple, 10+ for complex)
• Timeout configured (30-300 seconds)
• Retry logic in place for production
• Stop words defined if needed
• Error handling implemented
• Configuration tested with sample inputs

Performance Tuning

For Throughput

// Fast model, simple prompts
let paladin = PaladinBuilder::new(llm_adapter)
    .model("gpt-3.5-turbo")
    .temperature(0.7)
    .max_loops(1)
    .timeout_seconds(30)
    .build()?;

For Quality

// Best model, detailed prompts
let paladin = PaladinBuilder::new(llm_adapter)
    .model("gpt-4")
    .temperature(0.5)
    .max_loops(10)
    .timeout_seconds(300)
    .build()?;

For Cost Efficiency

// Cheaper model, efficient prompts
let paladin = PaladinBuilder::new(llm_adapter)
    .model("deepseek-chat")
    .temperature(0.7)
    .max_loops(3)
    .build()?;

Next Steps

• Battalion Patterns - Multi-agent orchestration
• Tool Integration - Add capabilities with Arsenal
• Memory Management - Use Garrison for context
• Examples - See configuration in action

Related Documentation

• Quickstart Guide
• API Reference
• Performance Tuning

Memory Management Guide

This guide covers how to use the Garrison memory system to give your Paladins conversation context, long-term knowledge, and semantic search capabilities.

Table of Contents

• Overview
• Garrison Architecture
• In-Memory Garrison
• Persistent Garrison
• Memory Windowing
• Semantic Search
• Memory Types
• Best Practices
• Advanced Patterns
• Troubleshooting

Overview

The Garrison system provides Paladins with:

• Conversation Context: Maintain multi-turn dialogue history
• Memory Windowing: Manage token limits intelligently
• Persistence: Save and restore sessions across restarts
• Semantic Search: Retrieve relevant memories by meaning, not just keywords
• Embeddings: Vector-based similarity for long-term memory

Key Concepts:

• Garrison: Memory storage system for a Paladin
• GarrisonEntry: Single memory record (message, observation, fact)
• ConversationHistory: Ordered sequence of interactions
• Memory Window: Limited context size respecting token limits
• Long-Term Memory: Persistent storage with semantic retrieval

Garrison Architecture

Core Components

// Single memory entry
pub struct GarrisonEntry {
    pub id: Uuid,
    pub role: ConversationRole,
    pub content: String,
    pub timestamp: DateTime<Utc>,
    pub metadata: HashMap<String, String>,
    pub token_count: Option<u32>,
}

// Conversation roles
pub enum ConversationRole {
    System,    // System prompts
    User,      // User messages
    Assistant, // Paladin responses
    Tool,      // Tool execution results
}

// Memory interface
#[async_trait]
pub trait GarrisonPort: Send + Sync {
    async fn remember(&self, entry: GarrisonEntry) -> Result<(), GarrisonError>;
    async fn recall_recent(&self, limit: usize) -> Result<Vec<GarrisonEntry>, GarrisonError>;
    async fn search(&self, query: &str, limit: usize) -> Result<Vec<GarrisonEntry>, GarrisonError>;
    async fn forget_all(&self) -> Result<(), GarrisonError>;
    async fn stats(&self) -> Result<GarrisonStats, GarrisonError>;
}

// Extended port for long-term memory
#[async_trait]
pub trait LongTermGarrisonPort: GarrisonPort {
    async fn remember_with_embedding(
        &self,
        entry: GarrisonEntry,
        embedding: Vec<f32>
    ) -> Result<(), GarrisonError>;

    async fn search_similar(
        &self,
        query_embedding: Vec<f32>,
        limit: usize
    ) -> Result<Vec<(GarrisonEntry, f32)>, GarrisonError>;
}

Memory Flow

User Input → Garrison adds User entry
    ↓
Paladin retrieves relevant history (window or search)
    ↓
LLM generates response with full context
    ↓
Garrison adds Assistant entry
    ↓
(Optional) Tool calls → Garrison adds Tool entries
    ↓
Repeat for next interaction

In-Memory Garrison

Fastest option for short-lived sessions where persistence isn't needed.

Basic Usage

use paladin_memory::garrison::InMemoryGarrison;
use paladin_core::platform::container::garrison::{GarrisonEntry, ConversationRole, GarrisonConfig};
use paladin::prelude::*;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let llm_adapter = Arc::new(OpenAiAdapter::new().build()?);

    // Create in-memory garrison
    let garrison = Arc::new(InMemoryGarrison::new(
        GarrisonConfig::default()
            .with_max_entries(100)
            .with_max_tokens(4000)
    ));

    // Build Paladin with memory
    let paladin = PaladinBuilder::new(llm_adapter)
        .name("ChatBot")
        .system_prompt("You are a helpful assistant with memory of our conversation.")
        .with_garrison(garrison.clone())
        .build()?;

    // First interaction
    let response1 = paladin.execute("My name is Alice").await?;
    println!("Bot: {}", response1.content);

    // Second interaction - Paladin remembers
    let response2 = paladin.execute("What's my name?").await?;
    println!("Bot: {}", response2.content);  // Should say "Alice"

    // Check garrison statistics
    let stats = garrison.stats().await?;
    println!("Total memories: {}", stats.total_entries);
    println!("Total tokens: {}", stats.total_tokens);

    Ok(())
}

Configuration Options

let garrison = InMemoryGarrison::new(
    GarrisonConfig::default()
        // Maximum number of entries to retain
        .with_max_entries(100)

        // Maximum total tokens across all entries
        .with_max_tokens(4000)

        // Token estimation strategy
        .with_token_counter(TokenCounter::Gpt4)

        // Eviction policy when limits reached
        .with_eviction_policy(EvictionPolicy::Fifo)  // First-in-first-out
);

Eviction Policies

pub enum EvictionPolicy {
    // Remove oldest entries first
    Fifo,

    // Remove least recently accessed entries
    Lru,

    // Remove entries based on importance score
    ImportanceBased,

    // Custom eviction logic
    Custom(Arc<dyn Fn(&[GarrisonEntry]) -> Vec<Uuid> + Send + Sync>),
}

// Example: Custom eviction keeping system prompts
let garrison = InMemoryGarrison::new(
    GarrisonConfig::default()
        .with_eviction_policy(EvictionPolicy::Custom(Arc::new(|entries| {
            // Never evict system prompts, evict oldest user messages
            entries.iter()
                .filter(|e| e.role == ConversationRole::User)
                .take(10)
                .map(|e| e.id)
                .collect()
        })))
);

Persistent Garrison

SQLite-backed storage for sessions that need to survive restarts.

Setup

use paladin_memory::garrison::InMemoryGarrison;
use paladin_core::platform::container::garrison::{GarrisonEntry, ConversationRole, GarrisonConfig};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create persistent garrison
    let garrison = Arc::new(
        SqliteGarrison::new("garrison.db")
            .await?
            .with_config(GarrisonConfig::default())
    );

    let paladin = PaladinBuilder::new(llm_adapter)
        .with_garrison(garrison)
        .build()?;

    // All interactions are automatically persisted
    paladin.execute("Remember this important fact!").await?;

    Ok(())
}

Session Management

// Create session-based garrison
let session_id = Uuid::new_v4();

let garrison = Arc::new(
    SqliteGarrison::new("garrison.db")
        .await?
        .with_session_id(session_id)
);

// Later, restore the same session
let garrison_restored = Arc::new(
    SqliteGarrison::new("garrison.db")
        .await?
        .with_session_id(session_id)  // Same session ID
);

// History is preserved
let history = garrison_restored.recall_recent(100).await?;
println!("Restored {} memories", history.len());

Multiple Users

pub struct UserGarrison {
    db: SqliteGarrison,
    user_id: String,
}

impl UserGarrison {
    pub async fn new(db_path: &str, user_id: String) -> Result<Self> {
        let db = SqliteGarrison::new(db_path).await?;
        Ok(Self { db, user_id })
    }
}

#[async_trait]
impl GarrisonPort for UserGarrison {
    async fn remember(&self, mut entry: GarrisonEntry) -> Result<()> {
        // Tag entries with user_id
        entry.metadata.insert("user_id".to_string(), self.user_id.clone());
        self.db.remember(entry).await
    }

    async fn recall_recent(&self, limit: usize) -> Result<Vec<GarrisonEntry>> {
        // Filter by user_id
        let all_entries = self.db.recall_recent(limit * 2).await?;
        Ok(all_entries.into_iter()
            .filter(|e| e.metadata.get("user_id") == Some(&self.user_id))
            .take(limit)
            .collect())
    }

    // Implement other methods...
}

// Usage
let alice_garrison = Arc::new(UserGarrison::new("garrison.db", "alice".to_string()).await?);
let bob_garrison = Arc::new(UserGarrison::new("garrison.db", "bob".to_string()).await?);

let alice_paladin = PaladinBuilder::new(llm_adapter.clone())
    .with_garrison(alice_garrison)
    .build()?;

let bob_paladin = PaladinBuilder::new(llm_adapter)
    .with_garrison(bob_garrison)
    .build()?;

Database Schema

-- migrations/001_create_garrison_tables.sql
CREATE TABLE IF NOT EXISTS garrison_entries (
    id TEXT PRIMARY KEY,
    session_id TEXT NOT NULL,
    role TEXT NOT NULL,
    content TEXT NOT NULL,
    timestamp INTEGER NOT NULL,
    metadata TEXT,
    token_count INTEGER,
    embedding BLOB,

    INDEX idx_session_timestamp (session_id, timestamp),
    INDEX idx_session_role (session_id, role)
);

CREATE TABLE IF NOT EXISTS garrison_sessions (
    session_id TEXT PRIMARY KEY,
    user_id TEXT,
    created_at INTEGER NOT NULL,
    updated_at INTEGER NOT NULL,
    metadata TEXT
);

Memory Windowing

Intelligently manage context size to respect LLM token limits.

Token-Based Windowing

// Get most recent entries that fit within token limit
let window = garrison.recall_recent(4000).await?;

println!("Window contains {} entries", window.len());
println!("Total tokens: {}",
    window.iter().map(|e| e.token_count.unwrap_or(0)).sum::<u32>());

Sliding Window

pub struct SlidingWindowGarrison {
    garrison: Arc<dyn GarrisonPort>,
    window_size: u32,
}

impl SlidingWindowGarrison {
    pub fn new(garrison: Arc<dyn GarrisonPort>, window_size: u32) -> Self {
        Self { garrison, window_size }
    }
}

#[async_trait]
impl GarrisonPort for SlidingWindowGarrison {
    async fn recall_recent(&self, _limit: usize) -> Result<Vec<GarrisonEntry>> {
        // Always return windowed history
        self.garrison.recall_recent(self.window_size).await
    }

    // Forward other methods to inner garrison
    async fn remember(&self, entry: GarrisonEntry) -> Result<()> {
        self.garrison.remember(entry).await
    }

    // ... other methods
}

// Usage - Paladin always sees only recent context
let windowed = Arc::new(SlidingWindowGarrison::new(garrison, 4000));

let paladin = PaladinBuilder::new(llm_adapter)
    .with_garrison(windowed)
    .build()?;

Smart Windowing with Priorities

pub struct PriorityWindowGarrison {
    garrison: Arc<dyn GarrisonPort>,
    window_size: u32,
}

impl PriorityWindowGarrison {
    async fn get_prioritized_window(&self) -> Result<Vec<GarrisonEntry>> {
        let all_entries = self.garrison.recall_recent(1000).await?;

        // Always include system prompts
        let system_entries: Vec<_> = all_entries.iter()
            .filter(|e| e.role == ConversationRole::System)
            .cloned()
            .collect();

        // Calculate remaining token budget
        let system_tokens: u32 = system_entries.iter()
            .map(|e| e.token_count.unwrap_or(0))
            .sum();

        let remaining_budget = self.window_size.saturating_sub(system_tokens);

        // Fill with most recent non-system entries
        let mut recent_entries: Vec<_> = all_entries.iter()
            .filter(|e| e.role != ConversationRole::System)
            .rev()
            .cloned()
            .collect();

        let mut token_sum = 0u32;
        let mut windowed_recent = Vec::new();

        for entry in recent_entries {
            let entry_tokens = entry.token_count.unwrap_or(0);
            if token_sum + entry_tokens <= remaining_budget {
                token_sum += entry_tokens;
                windowed_recent.push(entry);
            } else {
                break;
            }
        }

        // Combine: system + recent (chronological order)
        windowed_recent.reverse();
        let mut result = system_entries;
        result.extend(windowed_recent);

        Ok(result)
    }
}

Summarization for Compression

pub struct SummarizingGarrison {
    garrison: Arc<dyn GarrisonPort>,
    summarizer: Arc<dyn LlmPort>,
    window_size: u32,
    summary_threshold: usize,
}

impl SummarizingGarrison {
    async fn maybe_summarize(&self) -> Result<()> {
        let entries = self.garrison.recall_recent(self.summary_threshold).await?;

        if entries.len() >= self.summary_threshold {
            // Create summary of old entries
            let old_entries: Vec<_> = entries.iter()
                .take(self.summary_threshold / 2)
                .collect();

            let conversation_text = old_entries.iter()
                .map(|e| format!("{:?}: {}", e.role, e.content))
                .collect::<Vec<_>>()
                .join("\n");

            let prompt = format!(
                "Summarize this conversation in 2-3 paragraphs, preserving key facts:\n\n{}",
                conversation_text
            );

            let summary = self.summarizer.generate(&prompt).await?;

            // Replace old entries with summary
            for entry in old_entries {
                self.garrison.remove_entry(entry.id).await?;
            }

            self.garrison.remember(GarrisonEntry {
                id: Uuid::new_v4(),
                role: ConversationRole::System,
                content: format!("Previous conversation summary: {}", summary),
                timestamp: Utc::now(),
                metadata: HashMap::from([
                    ("type".to_string(), "summary".to_string()),
                ]),
                token_count: None,
            }).await?;
        }

        Ok(())
    }
}

Semantic Search

Retrieve relevant memories by meaning using embeddings.

Setup with Embeddings

use paladin_memory::garrison::InMemoryGarrison;
use paladin_core::platform::container::garrison::{GarrisonEntry, ConversationRole, GarrisonConfig};
use paladin_memory::embedding::OpenAIEmbeddingPort;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create garrison with embedding support
    let embedding_service = Arc::new(OpenAIEmbeddingService::new(api_key)?);

    let garrison = Arc::new(
        VectorGarrison::new("garrison.db")
            .await?
            .with_embedding_service(embedding_service)
    );

    let paladin = PaladinBuilder::new(llm_adapter)
        .with_garrison(garrison.clone())
        .build()?;

    // Add entries - embeddings generated automatically
    paladin.execute("I love hiking in the mountains").await?;
    paladin.execute("My favorite color is blue").await?;
    paladin.execute("I work as a software engineer").await?;

    // Semantic search
    let results = garrison.semantic_search("outdoor activities", 5).await?;

    for (entry, similarity) in results {
        println!("Similarity: {:.2} - {}", similarity, entry.content);
    }
    // Output: High similarity for "hiking in the mountains"

    Ok(())
}

Hybrid Search (Keyword + Semantic)

pub struct HybridGarrison {
    garrison: Arc<dyn LongTermGarrisonPort>,
}

impl HybridGarrison {
    pub async fn hybrid_search(
        &self,
        query: &str,
        limit: usize,
    ) -> Result<Vec<GarrisonEntry>> {
        // Get keyword matches
        let keyword_results = self.garrison.search(query, limit * 2).await?;

        // Get semantic matches
        let embedding = self.embedding_service.embed(query).await?;
        let semantic_results = self.garrison
            .semantic_search(embedding, limit * 2)
            .await?;

        // Merge and deduplicate
        let mut combined: HashMap<Uuid, (GarrisonEntry, f32)> = HashMap::new();

        // Add keyword results with base score
        for entry in keyword_results {
            combined.insert(entry.id, (entry, 0.5));
        }

        // Add semantic results, boosting score if already present
        for (entry, similarity) in semantic_results {
            combined.entry(entry.id)
                .and_modify(|(_, score)| *score += similarity * 0.5)
                .or_insert((entry, similarity * 0.5));
        }

        // Sort by combined score
        let mut sorted: Vec<_> = combined.into_values().collect();
        sorted.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap());

        Ok(sorted.into_iter()
            .take(limit)
            .map(|(entry, _)| entry)
            .collect())
    }
}

RAG (Retrieval-Augmented Generation)

pub struct RAGPaladin {
    paladin: Paladin,
    garrison: Arc<dyn LongTermGarrisonPort>,
}

impl RAGPaladin {
    pub async fn execute_with_rag(&self, query: &str) -> Result<PaladinResult> {
        // Retrieve relevant context from long-term memory
        let embedding = self.embedding_service.embed(query).await?;
        let relevant_memories = self.garrison
            .semantic_search(embedding, 5)
            .await?;

        // Build augmented prompt
        let context = relevant_memories.iter()
            .map(|(entry, _)| entry.content.as_str())
            .collect::<Vec<_>>()
            .join("\n\n");

        let augmented_query = format!(
            "Context from previous conversations:\n{}\n\n\
             Current question: {}",
            context, query
        );

        // Execute with retrieved context
        self.paladin.execute(&augmented_query).await
    }
}

// Usage
let rag_paladin = RAGPaladin {
    paladin,
    garrison: vector_garrison,
};

let response = rag_paladin.execute_with_rag(
    "What programming languages do I know?"
).await?;

Memory Types

Episodic Memory

Memory of specific events and experiences.

// Add episodic memory
garrison.remember(GarrisonEntry {
    id: Uuid::new_v4(),
    role: ConversationRole::User,
    content: "I visited Paris last summer".to_string(),
    timestamp: Utc::now(),
    metadata: HashMap::from([
        ("memory_type".to_string(), "episodic".to_string()),
        ("event_type".to_string(), "travel".to_string()),
        ("location".to_string(), "Paris, France".to_string()),
        ("timeframe".to_string(), "summer 2023".to_string()),
    ]),
    token_count: Some(10),
}).await?;

Semantic Memory

General knowledge and facts.

// Add semantic memory (facts)
garrison.remember(GarrisonEntry {
    id: Uuid::new_v4(),
    role: ConversationRole::System,
    content: "User prefers Python over JavaScript for backend development".to_string(),
    timestamp: Utc::now(),
    metadata: HashMap::from([
        ("memory_type".to_string(), "semantic".to_string()),
        ("category".to_string(), "preferences".to_string()),
        ("topic".to_string(), "programming".to_string()),
    ]),
    token_count: Some(15),
}).await?;

Procedural Memory

Knowledge about how to do things.

// Add procedural memory
garrison.remember(GarrisonEntry {
    id: Uuid::new_v4(),
    role: ConversationRole::System,
    content: "To deploy this project: cargo build --release && docker build -t app .".to_string(),
    timestamp: Utc::now(),
    metadata: HashMap::from([
        ("memory_type".to_string(), "procedural".to_string()),
        ("task".to_string(), "deployment".to_string()),
    ]),
    token_count: Some(20),
}).await?;

Best Practices

1. Choose the Right Garrison Type

// ✅ Use InMemoryGarrison for:
// - Temporary chatbots
// - Stateless services
// - Testing and development

let garrison = Arc::new(InMemoryGarrison::new(
    GarrisonConfig::default().with_max_tokens(4000)
));

// ✅ Use SqliteGarrison for:
// - Multi-session applications
// - User-specific contexts
// - Production services needing persistence

let garrison = Arc::new(
    SqliteGarrison::new("garrison.db").await?
        .with_session_id(session_id)
);

// ✅ Use VectorGarrison for:
// - Long-term knowledge bases
// - RAG applications
// - Semantic retrieval needs

let garrison = Arc::new(
    VectorGarrison::new("garrison.db").await?
        .with_embedding_service(embedding_service)
);

2. Set Appropriate Token Limits

// Model context windows
const GPT_4_TURBO: u32 = 128_000;
const GPT_4: u32 = 8_192;
const GPT_3_5: u32 = 16_385;
const CLAUDE_3: u32 = 200_000;

// Reserve tokens for: system prompt + response + buffer
let response_tokens = 1000;
let system_prompt_tokens = 500;
let buffer = 500;

let available_for_history = GPT_4 - response_tokens - system_prompt_tokens - buffer;

let garrison = InMemoryGarrison::new(
    GarrisonConfig::default()
        .with_max_tokens(available_for_history)  // ~6000 tokens
);

3. Add Metadata for Better Organization

garrison.remember(GarrisonEntry {
    id: Uuid::new_v4(),
    role: ConversationRole::User,
    content: message.clone(),
    timestamp: Utc::now(),
    metadata: HashMap::from([
        ("user_id".to_string(), user_id.clone()),
        ("session_id".to_string(), session_id.to_string()),
        ("channel".to_string(), "web".to_string()),
        ("language".to_string(), "en".to_string()),
        ("importance".to_string(), "high".to_string()),
    ]),
    token_count: Some(estimate_tokens(&message)),
}).await?;

4. Clean Up Old Memories

// Periodic cleanup
pub async fn cleanup_old_memories(
    garrison: &SqliteGarrison,
    days_to_keep: i64,
) -> Result<usize> {
    let cutoff = Utc::now() - Duration::days(days_to_keep);

    let removed = garrison
        .remove_before(cutoff)
        .await?;

    println!("Removed {} old memories", removed);
    Ok(removed)
}

// Scheduled cleanup
tokio::spawn(async move {
    let mut interval = tokio::time::interval(Duration::from_secs(86400)); // Daily
    loop {
        interval.tick().await;
        if let Err(e) = cleanup_old_memories(&garrison, 30).await {
            eprintln!("Cleanup failed: {}", e);
        }
    }
});

5. Implement Conversation Branching

pub struct BranchingGarrison {
    garrison: Arc<dyn GarrisonPort>,
    current_branch: RwLock<Uuid>,
}

impl BranchingGarrison {
    pub async fn create_branch(&self, from_entry: Uuid) -> Result<Uuid> {
        let branch_id = Uuid::new_v4();

        // Copy history up to branch point
        let history = self.garrison.recall_recent(1000).await?;
        let branch_history: Vec<_> = history.into_iter()
            .take_while(|e| e.id != from_entry)
            .collect();

        // Store branch metadata
        self.garrison.remember(GarrisonEntry {
            id: Uuid::new_v4(),
            role: ConversationRole::System,
            content: format!("Branch created from entry {}", from_entry),
            timestamp: Utc::now(),
            metadata: HashMap::from([
                ("type".to_string(), "branch".to_string()),
                ("branch_id".to_string(), branch_id.to_string()),
                ("parent_entry".to_string(), from_entry.to_string()),
            ]),
            token_count: None,
        }).await?;

        *self.current_branch.write().await = branch_id;
        Ok(branch_id)
    }
}

Advanced Patterns

Memory Consolidation

pub struct ConsolidatingGarrison {
    garrison: Arc<dyn GarrisonPort>,
    llm: Arc<dyn LlmPort>,
}

impl ConsolidatingGarrison {
    pub async fn consolidate_memories(&self) -> Result<()> {
        let entries = self.garrison.recall_recent(100).await?;

        // Group by topic using LLM
        let topics = self.extract_topics(&entries).await?;

        // Create consolidated memory for each topic
        for (topic, topic_entries) in topics {
            let facts = self.extract_facts(&topic_entries).await?;

            self.garrison.remember(GarrisonEntry {
                id: Uuid::new_v4(),
                role: ConversationRole::System,
                content: format!("Consolidated facts about {}: {}", topic, facts),
                timestamp: Utc::now(),
                metadata: HashMap::from([
                    ("type".to_string(), "consolidated".to_string()),
                    ("topic".to_string(), topic),
                    ("source_count".to_string(), topic_entries.len().to_string()),
                ]),
                token_count: None,
            }).await?;
        }

        Ok(())
    }

    async fn extract_topics(&self, entries: &[GarrisonEntry]) -> Result<HashMap<String, Vec<GarrisonEntry>>> {
        // Use LLM to categorize entries by topic
        // Implementation details...
        Ok(HashMap::new())
    }

    async fn extract_facts(&self, entries: &[GarrisonEntry]) -> Result<String> {
        let conversation = entries.iter()
            .map(|e| &e.content)
            .cloned()
            .collect::<Vec<_>>()
            .join("\n");

        let prompt = format!(
            "Extract key facts from this conversation:\n\n{}",
            conversation
        );

        self.llm.generate(&prompt).await
    }
}

Attention Mechanism

pub struct AttentionGarrison {
    garrison: Arc<dyn LongTermGarrisonPort>,
}

impl AttentionGarrison {
    pub async fn get_attended_context(
        &self,
        query: &str,
        context_size: u32,
    ) -> Result<Vec<GarrisonEntry>> {
        // Get semantic matches
        let query_embedding = self.embed(query).await?;
        let candidates = self.garrison
            .semantic_search(query_embedding, 50)
            .await?;

        // Score each candidate using attention mechanism
        let mut scored: Vec<_> = candidates.into_iter()
            .map(|(entry, similarity)| {
                let recency_score = self.recency_score(&entry);
                let importance_score = self.importance_score(&entry);

                // Weighted combination
                let attention = similarity * 0.5 + recency_score * 0.3 + importance_score * 0.2;

                (entry, attention)
            })
            .collect();

        // Sort by attention score
        scored.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap());

        // Select top entries within token budget
        let mut selected = Vec::new();
        let mut token_sum = 0u32;

        for (entry, _) in scored {
            let entry_tokens = entry.token_count.unwrap_or(0);
            if token_sum + entry_tokens <= context_size {
                token_sum += entry_tokens;
                selected.push(entry);
            }
        }

        Ok(selected)
    }

    fn recency_score(&self, entry: &GarrisonEntry) -> f32 {
        let age = (Utc::now() - entry.timestamp).num_seconds() as f32;
        let decay_rate = 0.0001;  // Adjust for desired decay speed
        (-decay_rate * age).exp()
    }

    fn importance_score(&self, entry: &GarrisonEntry) -> f32 {
        // Extract importance from metadata or content
        entry.metadata.get("importance")
            .and_then(|s| s.parse::<f32>().ok())
            .unwrap_or(0.5)
    }
}

Memory Reflection

pub struct ReflectiveGarrison {
    garrison: Arc<dyn GarrisonPort>,
    llm: Arc<dyn LlmPort>,
}

impl ReflectiveGarrison {
    pub async fn generate_reflections(&self) -> Result<()> {
        let recent_entries = self.garrison.recall_recent(50).await?;

        // Prompt LLM to reflect on conversation
        let conversation = recent_entries.iter()
            .map(|e| format!("{:?}: {}", e.role, e.content))
            .collect::<Vec<_>>()
            .join("\n");

        let prompt = format!(
            "Reflect on this conversation and extract:\n\
             1. Key insights about the user\n\
             2. Patterns in the discussion\n\
             3. Important facts to remember\n\n\
             Conversation:\n{}",
            conversation
        );

        let reflection = self.llm.generate(&prompt).await?;

        // Store reflection as high-importance memory
        self.garrison.remember(GarrisonEntry {
            id: Uuid::new_v4(),
            role: ConversationRole::System,
            content: format!("Reflection: {}", reflection),
            timestamp: Utc::now(),
            metadata: HashMap::from([
                ("type".to_string(), "reflection".to_string()),
                ("importance".to_string(), "high".to_string()),
            ]),
            token_count: None,
        }).await?;

        Ok(())
    }
}

Troubleshooting

Memory Not Persisting

Problem: Garrison entries disappear after restart.

Solutions:

1. Verify using SqliteGarrison, not InMemoryGarrison
2. Check database file path is correct and writable
3. Ensure proper async handling (.await on all operations)

// ❌ Won't persist
let garrison = Arc::new(InMemoryGarrison::new(config));

// ✅ Will persist
let garrison = Arc::new(SqliteGarrison::new("garrison.db").await?);

Context Window Overflow

Problem: Errors about exceeding maximum context length.

Solutions:

1. Reduce max_tokens in GarrisonConfig
2. Use get_window() instead of get_history()
3. Implement summarization for old memories

// Calculate safe token limit
let model_limit = 8192;  // GPT-4
let response_budget = 1000;
let system_prompt_tokens = 500;
let safety_buffer = 500;

let garrison_limit = model_limit - response_budget - system_prompt_tokens - safety_buffer;

let garrison = InMemoryGarrison::new(
    GarrisonConfig::default().with_max_tokens(garrison_limit)
);

Slow Semantic Search

Problem: Embedding-based search is taking too long.

Solutions:

1. Add database indexes on embedding columns
2. Use approximate nearest neighbor (ANN) algorithms
3. Cache embeddings for frequent queries
4. Limit search scope with filters

-- Add index for faster vector search
CREATE INDEX idx_embeddings ON garrison_entries(embedding);

-- Consider using specialized vector databases
-- PostgreSQL with pgvector extension
-- Qdrant, Milvus, or Weaviate for production

Memory Leaks in Long Sessions

Problem: Memory usage grows unbounded.

Solutions:

1. Set max_entries in config
2. Implement periodic cleanup
3. Use eviction policies
4. Monitor with garrison.stats()

// Periodic memory management
tokio::spawn(async move {
    let mut interval = tokio::time::interval(Duration::from_secs(3600));
    loop {
        interval.tick().await;

        let stats = garrison.stats().await.unwrap();

        if stats.total_entries > 1000 {
            // Trigger cleanup
            garrison.compact().await.unwrap();
        }
    }
});

Testing

Unit Testing

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn test_garrison_add_and_retrieve() {
        let garrison = InMemoryGarrison::new(GarrisonConfig::default());

        let entry = GarrisonEntry {
            id: Uuid::new_v4(),
            role: ConversationRole::User,
            content: "Test message".to_string(),
            timestamp: Utc::now(),
            metadata: HashMap::new(),
            token_count: Some(2),
        };

        garrison.remember(entry.clone()).await.unwrap();

        let history = garrison.recall_recent(10).await.unwrap();
        assert_eq!(history.len(), 1);
        assert_eq!(history[0].content, "Test message");
    }

    #[tokio::test]
    async fn test_token_window() {
        let garrison = InMemoryGarrison::new(
            GarrisonConfig::default().with_max_tokens(100)
        );

        // Add entries totaling 150 tokens
        for i in 0..15 {
            garrison.remember(GarrisonEntry {
                id: Uuid::new_v4(),
                role: ConversationRole::User,
                content: format!("Message {}", i),
                timestamp: Utc::now(),
                metadata: HashMap::new(),
                token_count: Some(10),
            }).await.unwrap();
        }

        // Window should respect token limit
        let window = garrison.recall_recent(100).await.unwrap();
        let total_tokens: u32 = window.iter()
            .map(|e| e.token_count.unwrap_or(0))
            .sum();

        assert!(total_tokens <= 100);
    }
}

Examples

See working examples:

• examples/garrison_in_memory.rs - Basic in-memory usage
• examples/garrison_persistent.rs - SQLite persistence
• examples/garrison_semantic_search.rs - Embedding-based retrieval
• examples/memory_windowing.rs - Token management strategies

Next Steps

• Tool Integration - Combine memory with tools
• Battalion Patterns - Shared memory in multi-agent systems
• API Reference - Garrison API documentation

Related Resources

• Token Counting Strategies
• Vector Database Integration
• Production Deployment

Tool Integration Guide

This guide covers how to integrate external tools and capabilities into your Paladins using the Arsenal system and Model Context Protocol (MCP).

Table of Contents

• Overview
• Arsenal Architecture
• MCP Protocol
• STDIO Tool Servers
• SSE Tool Servers
• Custom Tool Development
• Tool Result Handling
• Best Practices
• Troubleshooting

Overview

The Arsenal system enables Paladins to:

• Execute external tools and capabilities
• Search the web, access databases, run calculations
• Interact with APIs and services
• Extend functionality without modifying core code

Key Concepts:

• Arsenal: The registry of available tools
• Armament: A single tool or capability
• MCP (Model Context Protocol): Standard protocol for tool servers
• Tool Call: Request from Paladin to execute a tool
• Tool Result: Response from tool execution

Arsenal Architecture

Core Components

// Armament - Tool definition
pub struct Armament {
    pub name: String,
    pub description: String,
    pub schema: ToolSchema,
    pub required_params: Vec<String>,
}

// Arsenal Port - Tool execution interface
#[async_trait]
pub trait ArsenalPort: Send + Sync {
    async fn list_armaments(&self) -> Result<Vec<Armament>>;
    async fn invoke(&self, call: &ArmamentCall) -> Result<ArmamentResult>;
}

// Armament Call - Tool invocation request
pub struct ArmamentCall {
    pub tool_name: String,
    pub parameters: HashMap<String, Value>,
    pub call_id: Uuid,
}

// Armament Result - Tool execution response
pub struct ArmamentResult {
    pub call_id: Uuid,
    pub success: bool,
    pub output: String,
    pub error: Option<String>,
}

Tool Flow

Paladin → LLM decides to use tool → ArmamentCall
    ↓
ArsenalPort validates call → Routes to correct Armament
    ↓
Tool executes (MCP server, API, local function)
    ↓
ArmamentResult → Injected into Paladin context
    ↓
Paladin continues reasoning with tool result

MCP Protocol

The Model Context Protocol (MCP) is an open standard for connecting LLM applications to external tools and data sources.

MCP Server Types

1. STDIO Servers: Command-line tools communicating via stdin/stdout
2. SSE Servers: Web services using Server-Sent Events

MCP Message Format

// Tool Discovery Request
{
  "jsonrpc": "2.0",
  "method": "tools/list",
  "id": 1
}

// Tool Discovery Response
{
  "jsonrpc": "2.0",
  "result": {
    "tools": [
      {
        "name": "web_search",
        "description": "Search the web for information",
        "inputSchema": {
          "type": "object",
          "properties": {
            "query": {
              "type": "string",
              "description": "Search query"
            }
          },
          "required": ["query"]
        }
      }
    ]
  },
  "id": 1
}

// Tool Invocation Request
{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "name": "web_search",
    "arguments": {
      "query": "Rust async programming"
    }
  },
  "id": 2
}

// Tool Invocation Response
{
  "jsonrpc": "2.0",
  "result": {
    "content": [
      {
        "type": "text",
        "text": "Search results: ..."
      }
    ]
  },
  "id": 2
}

STDIO Tool Servers

STDIO servers are command-line programs that communicate via standard input/output.

Connecting a STDIO Server

use paladin_ports::output::arsenal_port::{ArsenalPort, ArsenalRegistry};
use paladin_core::platform::container::arsenal::{Armament, ArmamentCall, ArmamentResult};
use paladin::prelude::*;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let llm_adapter = Arc::new(OpenAiAdapter::new().build()?);

    // Connect to an MCP STDIO server
    let web_search = MCPStdioAdapter::new()
        .command("uvx")
        .args(vec!["mcp-server-fetch"])
        .build()
        .await?;

    // Build Paladin with tool access
    let paladin = PaladinBuilder::new(llm_adapter)
        .name("ResearchAssistant")
        .system_prompt("You are a research assistant with web search capabilities. \
                        Use the web_search tool to find current information. \
                        Always cite your sources.")
        .add_armament(Arc::new(web_search))
        .build()?;

    // Paladin will automatically use tools when needed
    let response = paladin.execute("What are the latest Rust features in 2024?").await?;
    println!("{}", response.content);

    Ok(())
}

Popular STDIO MCP Servers

# Web search
uvx mcp-server-fetch

# File system access
uvx mcp-server-filesystem --allowed-directory ~/Documents

# Git operations
uvx mcp-server-git --repository /path/to/repo

# Database queries
uvx mcp-server-sqlite --db-path database.db

# Calculator
uvx mcp-server-calculator

Configuration Example

arsenal:
  mcp_servers:
    - name: "web_search"
      type: "stdio"
      command: "uvx"
      args: ["mcp-server-fetch"]
      enabled: true

    - name: "filesystem"
      type: "stdio"
      command: "uvx"
      args:
        - "mcp-server-filesystem"
        - "--allowed-directory"
        - "/home/user/workspace"
      enabled: true

    - name: "calculator"
      type: "stdio"
      command: "uvx"
      args: ["mcp-server-calculator"]
      enabled: true

Advanced STDIO Configuration

let web_search = MCPStdioAdapter::new()
    .command("uvx")
    .args(vec!["mcp-server-fetch"])
    .working_directory("/tmp")
    .env("API_KEY", api_key)
    .timeout(Duration::from_secs(30))
    .max_retries(3)
    .build()
    .await?;

SSE Tool Servers

SSE (Server-Sent Events) servers are web services that provide MCP tools over HTTP.

Connecting an SSE Server

use paladin_ports::output::arsenal_port::{ArsenalPort, ArsenalRegistry};
use paladin_core::platform::container::arsenal::{Armament, ArmamentCall, ArmamentResult};
use paladin::prelude::*;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let llm_adapter = Arc::new(OpenAiAdapter::new().build()?);

    // Connect to an MCP SSE server
    let api_tools = MCPSseAdapter::new()
        .endpoint("https://api.example.com/mcp")
        .api_key(std::env::var("API_KEY")?)
        .build()
        .await?;

    let paladin = PaladinBuilder::new(llm_adapter)
        .name("APIAssistant")
        .system_prompt("You have access to company APIs. Use them to retrieve data.")
        .add_armament(Arc::new(api_tools))
        .build()?;

    let response = paladin.execute("Get user statistics for last month").await?;
    println!("{}", response.content);

    Ok(())
}

SSE Configuration

let api_server = MCPSseAdapter::new()
    .endpoint("https://api.example.com/mcp")
    .api_key("your-api-key")
    .bearer_token("bearer-token")  // Alternative auth
    .headers(HashMap::from([
        ("X-Custom-Header", "value"),
    ]))
    .timeout(Duration::from_secs(60))
    .retry_config(RetryConfig {
        max_attempts: 3,
        initial_delay: Duration::from_secs(1),
        max_delay: Duration::from_secs(10),
        exponential_backoff: true,
    })
    .build()
    .await?;

SSE Health Checks

// Verify server is reachable
if api_server.health_check().await? {
    println!("SSE server is healthy");
}

// List available tools
let tools = api_server.list_armaments().await?;
for tool in tools {
    println!("Tool: {} - {}", tool.name, tool.description);
}

Custom Tool Development

Create your own tools by implementing the ArsenalPort trait.

Simple Custom Tool

use paladin_ports::output::arsenal_port::{ArsenalPort, ArsenalRegistry};
use paladin_core::platform::container::arsenal::{Armament, ArmamentCall, ArmamentResult};
use async_trait::async_trait;

pub struct CalculatorTool;

#[async_trait]
impl ArsenalPort for CalculatorTool {
    async fn list_armaments(&self) -> Result<Vec<Armament>, ArsenalError> {
        Ok(vec![
            Armament {
                name: "add".to_string(),
                description: "Add two numbers".to_string(),
                schema: ToolSchema::new()
                    .add_param("a", ParamType::Number, "First number", true)
                    .add_param("b", ParamType::Number, "Second number", true),
                required_params: vec!["a".to_string(), "b".to_string()],
            },
            Armament {
                name: "multiply".to_string(),
                description: "Multiply two numbers".to_string(),
                schema: ToolSchema::new()
                    .add_param("a", ParamType::Number, "First number", true)
                    .add_param("b", ParamType::Number, "Second number", true),
                required_params: vec!["a".to_string(), "b".to_string()],
            },
        ])
    }

    async fn invoke(&self, call: &ArmamentCall) -> Result<ArmamentResult, ArsenalError> {
        let a = call.parameters.get("a")
            .and_then(|v| v.as_f64())
            .ok_or_else(|| ArsenalError::InvalidParameter("a".to_string()))?;

        let b = call.parameters.get("b")
            .and_then(|v| v.as_f64())
            .ok_or_else(|| ArsenalError::InvalidParameter("b".to_string()))?;

        let result = match call.tool_name.as_str() {
            "add" => a + b,
            "multiply" => a * b,
            _ => return Err(ArsenalError::ToolNotFound(call.tool_name.clone())),
        };

        Ok(ArmamentResult {
            call_id: call.call_id,
            success: true,
            output: result.to_string(),
            error: None,
            execution_time_ms: 1,
        })
    }

    fn validate_call(&self, call: &ArmamentCall) -> Result<(), ArsenalError> {
        // Validate tool exists
        let tools = self.list_armaments().await?;
        if !tools.iter().any(|t| t.name == call.tool_name) {
            return Err(ArsenalError::ToolNotFound(call.tool_name.clone()));
        }

        // Validate required parameters
        let tool = tools.iter().find(|t| t.name == call.tool_name).unwrap();
        for param in &tool.required_params {
            if !call.parameters.contains_key(param) {
                return Err(ArsenalError::MissingParameter(param.clone()));
            }
        }

        Ok(())
    }
}

// Use the custom tool
let calculator = Arc::new(CalculatorTool);

let paladin = PaladinBuilder::new(llm_adapter)
    .add_armament(calculator)
    .build()?;

API Integration Tool

use reqwest::Client;

pub struct WeatherTool {
    client: Client,
    api_key: String,
}

impl WeatherTool {
    pub fn new(api_key: String) -> Self {
        Self {
            client: Client::new(),
            api_key,
        }
    }
}

#[async_trait]
impl ArsenalPort for WeatherTool {
    async fn list_armaments(&self) -> Result<Vec<Armament>, ArsenalError> {
        Ok(vec![
            Armament {
                name: "get_weather".to_string(),
                description: "Get current weather for a location".to_string(),
                schema: ToolSchema::new()
                    .add_param("location", ParamType::String, "City name or coordinates", true)
                    .add_param("units", ParamType::String, "Temperature units (celsius/fahrenheit)", false),
                required_params: vec!["location".to_string()],
            },
        ])
    }

    async fn invoke(&self, call: &ArmamentCall) -> Result<ArmamentResult, ArsenalError> {
        let location = call.parameters.get("location")
            .and_then(|v| v.as_str())
            .ok_or_else(|| ArsenalError::InvalidParameter("location".to_string()))?;

        let units = call.parameters.get("units")
            .and_then(|v| v.as_str())
            .unwrap_or("celsius");

        // Call weather API
        let url = format!(
            "https://api.openweathermap.org/data/2.5/weather?q={}&appid={}&units={}",
            location, self.api_key, units
        );

        let response = self.client.get(&url)
            .send()
            .await
            .map_err(|e| ArsenalError::ExecutionError(e.to_string()))?;

        let weather_data = response.json::<serde_json::Value>()
            .await
            .map_err(|e| ArsenalError::ExecutionError(e.to_string()))?;

        let temp = weather_data["main"]["temp"].as_f64().unwrap_or(0.0);
        let description = weather_data["weather"][0]["description"]
            .as_str()
            .unwrap_or("unknown");

        let output = format!(
            "Weather in {}: {} with temperature of {}°",
            location, description, temp
        );

        Ok(ArmamentResult {
            call_id: call.call_id,
            success: true,
            output,
            error: None,
            execution_time_ms: 200,
        })
    }

    fn validate_call(&self, call: &ArmamentCall) -> Result<(), ArsenalError> {
        if call.tool_name != "get_weather" {
            return Err(ArsenalError::ToolNotFound(call.tool_name.clone()));
        }

        if !call.parameters.contains_key("location") {
            return Err(ArsenalError::MissingParameter("location".to_string()));
        }

        Ok(())
    }
}

// Usage
let weather = Arc::new(WeatherTool::new(api_key));

let paladin = PaladinBuilder::new(llm_adapter)
    .system_prompt("You can check weather. Use get_weather tool.")
    .add_armament(weather)
    .build()?;

Database Query Tool

use sqlx::SqlitePool;

pub struct DatabaseTool {
    pool: SqlitePool,
}

impl DatabaseTool {
    pub async fn new(database_url: &str) -> Result<Self, sqlx::Error> {
        let pool = SqlitePool::connect(database_url).await?;
        Ok(Self { pool })
    }
}

#[async_trait]
impl ArsenalPort for DatabaseTool {
    async fn list_armaments(&self) -> Result<Vec<Armament>, ArsenalError> {
        Ok(vec![
            Armament {
                name: "query_database".to_string(),
                description: "Execute a read-only SQL query".to_string(),
                schema: ToolSchema::new()
                    .add_param("query", ParamType::String, "SQL SELECT query", true),
                required_params: vec!["query".to_string()],
            },
        ])
    }

    async fn invoke(&self, call: &ArmamentCall) -> Result<ArmamentResult, ArsenalError> {
        let query = call.parameters.get("query")
            .and_then(|v| v.as_str())
            .ok_or_else(|| ArsenalError::InvalidParameter("query".to_string()))?;

        // Security: Only allow SELECT queries
        if !query.trim().to_lowercase().starts_with("select") {
            return Ok(ArmamentResult {
                call_id: call.call_id,
                success: false,
                output: String::new(),
                error: Some("Only SELECT queries are allowed".to_string()),
                execution_time_ms: 0,
            });
        }

        let start = std::time::Instant::now();

        let rows = sqlx::query(query)
            .fetch_all(&self.pool)
            .await
            .map_err(|e| ArsenalError::ExecutionError(e.to_string()))?;

        // Convert rows to JSON
        let result_json = serde_json::to_string_pretty(&rows)
            .unwrap_or_else(|_| "[]".to_string());

        Ok(ArmamentResult {
            call_id: call.call_id,
            success: true,
            output: result_json,
            error: None,
            execution_time_ms: start.elapsed().as_millis() as u64,
        })
    }

    fn validate_call(&self, call: &ArmamentCall) -> Result<(), ArsenalError> {
        if !call.parameters.contains_key("query") {
            return Err(ArsenalError::MissingParameter("query".to_string()));
        }
        Ok(())
    }
}

Tool Result Handling

Automatic Context Injection

When a Paladin invokes a tool, the result is automatically added to the conversation context:

// Paladin execution loop
loop {
    let response = llm.generate(context).await?;

    if let Some(tool_call) = response.tool_calls.first() {
        // Execute tool
        let result = arsenal.invoke(tool_call).await?;

        // Add result to context
        context.add_tool_result(result);

        // Continue reasoning with tool output
        continue;
    }

    // No more tool calls, return final response
    break Ok(response);
}

Custom Result Processing

pub struct LoggingArsenalPort<T: ArsenalPort> {
    inner: T,
}

#[async_trait]
impl<T: ArsenalPort> ArsenalPort for LoggingArsenalPort<T> {
    async fn invoke(&self, call: &ArmamentCall) -> Result<ArmamentResult, ArsenalError> {
        println!("Invoking tool: {}", call.tool_name);
        println!("Parameters: {:?}", call.parameters);

        let start = std::time::Instant::now();
        let result = self.inner.invoke(call).await?;
        let duration = start.elapsed();

        println!("Tool completed in {:?}", duration);
        println!("Success: {}", result.success);

        if let Some(error) = &result.error {
            eprintln!("Tool error: {}", error);
        }

        Ok(result)
    }

    // Forward other methods
    async fn list_armaments(&self) -> Result<Vec<Armament>, ArsenalError> {
        self.inner.list_armaments().await
    }

    fn validate_call(&self, call: &ArmamentCall) -> Result<(), ArsenalError> {
        self.inner.validate_call(call)
    }
}

// Usage
let weather_tool = Arc::new(WeatherTool::new(api_key));
let logged_tool = Arc::new(LoggingArsenalPort { inner: weather_tool });

paladin.add_armament(logged_tool);

Error Handling

match arsenal.invoke(&call).await {
    Ok(result) if result.success => {
        // Tool succeeded
        process_result(&result.output);
    }
    Ok(result) => {
        // Tool failed but returned error message
        eprintln!("Tool failed: {}", result.error.unwrap_or_default());
        // Decide: retry, use fallback, or fail
    }
    Err(ArsenalError::ToolNotFound(name)) => {
        eprintln!("Tool not found: {}", name);
        // Handle missing tool
    }
    Err(ArsenalError::Timeout) => {
        eprintln!("Tool execution timed out");
        // Retry with longer timeout
    }
    Err(e) => {
        eprintln!("Arsenal error: {}", e);
        // Handle other errors
    }
}

Best Practices

1. Clear Tool Descriptions

// ❌ Bad: Vague description
Armament {
    name: "search",
    description: "Search for stuff",
    // ...
}

// ✅ Good: Clear, specific description
Armament {
    name: "web_search",
    description: "Search the web using Google. Returns top 10 results with titles, \
                  URLs, and snippets. Use this when you need current information \
                  not in your training data.",
    // ...
}

2. Validate Inputs

fn validate_call(&self, call: &ArmamentCall) -> Result<(), ArsenalError> {
    // Check required parameters
    for param in &self.required_params {
        if !call.parameters.contains_key(param) {
            return Err(ArsenalError::MissingParameter(param.clone()));
        }
    }

    // Validate parameter types and values
    if let Some(url) = call.parameters.get("url") {
        if !url.as_str().unwrap_or("").starts_with("http") {
            return Err(ArsenalError::InvalidParameter("url must start with http".into()));
        }
    }

    Ok(())
}

3. Set Timeouts

let tool = CustomTool::new()
    .timeout(Duration::from_secs(30))  // Prevent hanging
    .build()?;

4. Implement Retries for Flaky Operations

async fn invoke_with_retry(&self, call: &ArmamentCall) -> Result<ArmamentResult, ArsenalError> {
    let mut attempts = 0;
    let max_attempts = 3;

    loop {
        attempts += 1;

        match self.invoke(call).await {
            Ok(result) => return Ok(result),
            Err(e) if attempts < max_attempts && e.is_retryable() => {
                tokio::time::sleep(Duration::from_secs(2_u64.pow(attempts))).await;
                continue;
            }
            Err(e) => return Err(e),
        }
    }
}

5. Sanitize Inputs

fn sanitize_sql(query: &str) -> Result<String, ArsenalError> {
    // Remove dangerous keywords
    let dangerous = ["DROP", "DELETE", "UPDATE", "INSERT", "CREATE", "ALTER"];
    let query_upper = query.to_uppercase();

    for keyword in dangerous {
        if query_upper.contains(keyword) {
            return Err(ArsenalError::SecurityViolation(
                format!("Query contains forbidden keyword: {}", keyword)
            ));
        }
    }

    Ok(query.to_string())
}

6. Rate Limiting

use std::sync::Arc;
use tokio::sync::Semaphore;

pub struct RateLimitedTool<T: ArsenalPort> {
    inner: T,
    semaphore: Arc<Semaphore>,
}

impl<T: ArsenalPort> RateLimitedTool<T> {
    pub fn new(inner: T, max_concurrent: usize) -> Self {
        Self {
            inner,
            semaphore: Arc::new(Semaphore::new(max_concurrent)),
        }
    }
}

#[async_trait]
impl<T: ArsenalPort> ArsenalPort for RateLimitedTool<T> {
    async fn invoke(&self, call: &ArmamentCall) -> Result<ArmamentResult, ArsenalError> {
        let _permit = self.semaphore.acquire().await
            .map_err(|e| ArsenalError::ExecutionError(e.to_string()))?;

        self.inner.invoke(call).await
    }

    // Forward other methods...
}

7. Structured Output

// Return structured data that's easy to parse
let output = serde_json::json!({
    "status": "success",
    "data": {
        "temperature": 72.5,
        "conditions": "partly cloudy",
        "humidity": 65
    },
    "timestamp": chrono::Utc::now().to_rfc3339()
});

Ok(ArmamentResult {
    call_id: call.call_id,
    success: true,
    output: output.to_string(),
    error: None,
    execution_time_ms: 150,
})

Troubleshooting

Tool Not Being Called

Problem: Paladin doesn't use the tool even though it should.

Solutions:

1. Check tool description is clear and relevant
2. Update system prompt to mention tool availability
3. Verify tool appears in list_armaments() output
4. Ensure LLM supports function calling (GPT-4, Claude 3+)

// Make tool usage explicit in system prompt
.system_prompt("You have access to a web_search tool. USE IT to find current information. \
                Always search before answering questions about recent events.")

MCP Server Connection Failed

Problem: Cannot connect to MCP STDIO server.

Solutions:

1. Verify command is in PATH: which uvx
2. Test command manually: uvx mcp-server-fetch
3. Check server logs for errors
4. Verify environment variables are set

let tool = MCPStdioAdapter::new()
    .command("uvx")
    .args(vec!["mcp-server-fetch"])
    .debug_mode(true)  // Enable verbose logging
    .build()
    .await?;

Tool Execution Timeout

Problem: Tools timing out frequently.

Solutions:

1. Increase timeout duration
2. Optimize tool implementation
3. Add caching for expensive operations
4. Use async/parallel execution where possible

let tool = CustomTool::new()
    .timeout(Duration::from_secs(120))  // Longer timeout
    .build()?;

Invalid Parameters

Problem: Tool receives wrong parameter types.

Solutions:

1. Strengthen parameter validation
2. Add type coercion in invoke()
3. Improve tool schema definitions
4. Add examples to tool descriptions

// Robust parameter extraction
let count = call.parameters.get("count")
    .and_then(|v| {
        // Try as number, then as string
        v.as_i64()
            .or_else(|| v.as_str().and_then(|s| s.parse::<i64>().ok()))
    })
    .unwrap_or(10);  // Default value

SSE Server Authentication

Problem: SSE server returns 401 Unauthorized.

Solutions:

1. Verify API key is correct
2. Check token hasn't expired
3. Ensure correct authentication method (bearer vs api-key)
4. Check server CORS settings

let tool = MCPSseAdapter::new()
    .endpoint("https://api.example.com/mcp")
    .bearer_token("your-token")  // Use bearer auth instead of api_key
    .build()
    .await?;

Testing Tools

Unit Testing Custom Tools

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn test_calculator_add() {
        let calc = CalculatorTool;

        let call = ArmamentCall {
            tool_name: "add".to_string(),
            parameters: HashMap::from([
                ("a".to_string(), json!(5.0)),
                ("b".to_string(), json!(3.0)),
            ]),
            call_id: Uuid::new_v4(),
        };

        let result = calc.invoke(&call).await.unwrap();

        assert!(result.success);
        assert_eq!(result.output, "8");
    }

    #[tokio::test]
    async fn test_invalid_parameter() {
        let calc = CalculatorTool;

        let call = ArmamentCall {
            tool_name: "add".to_string(),
            parameters: HashMap::from([
                ("a".to_string(), json!(5.0)),
                // Missing 'b' parameter
            ]),
            call_id: Uuid::new_v4(),
        };

        assert!(calc.invoke(&call).await.is_err());
    }
}

Integration Testing with Paladin

#[tokio::test]
async fn test_paladin_uses_tool() {
    let llm_adapter = Arc::new(MockLlmAdapter::new());
    let calc = Arc::new(CalculatorTool);

    let paladin = PaladinBuilder::new(llm_adapter)
        .system_prompt("You have a calculator. Use it for math.")
        .add_armament(calc)
        .build()
        .unwrap();

    let response = paladin.execute("What is 15 + 27?").await.unwrap();

    assert!(response.content.contains("42"));
}

Examples

See working examples:

• examples/arsenal_stdio_tools.rs - MCP STDIO integration
• examples/arsenal_sse_tools.rs - MCP SSE integration
• examples/custom_tools.rs - Custom tool implementation
• examples/tool_error_handling.rs - Error handling patterns

Next Steps

• Memory Management - Use Garrison with tools
• Battalion Patterns - Tools in multi-agent systems
• API Reference - Arsenal API documentation

Related Resources

• MCP Specification
• MCP Server Examples
• Tool Development Best Practices

Output Formatting Guide

This guide covers the Herald system for formatting and controlling Paladin output in various formats and styles.

Table of Contents

• Overview
• Herald Architecture
• Built-in Formatters
• Custom Formatters
• Streaming Output
• Multi-Format Output
• Post-Processing
• Best Practices
• Advanced Patterns
• Troubleshooting

Overview

The Herald system controls how Paladin output is formatted and presented to users.

Key Capabilities:

• Format Transformation: Convert LLM output to JSON, Markdown, HTML, etc.
• Streaming: Real-time output delivery for better UX
• Validation: Ensure output meets schema requirements
• Post-Processing: Clean, enhance, or transform responses
• Multi-Channel: Different formats for different output destinations

Key Concepts:

• Herald: Output formatting system
• Formatter: Converts raw LLM output to specific format
• OutputFormat: Target format specification (JSON, Markdown, Plain, etc.)
• StreamHandler: Processes output chunks in real-time

Herald Architecture

Core Components

// Output format types (paladin_core::platform::container::paladin_config)
pub enum OutputFormat {
    Text,           // Raw text output
    Json,           // Structured JSON
    Structured,     // Structured data output
}

// Herald interface (paladin_core::platform::container::herald)
pub trait Herald: Send + Sync {
    /// Format a complete Paladin execution result
    fn format_paladin_result(&self, result: &PaladinResult) -> Result<String, HeraldError>;

    /// Format a complete Battalion execution result
    fn format_battalion_result(&self, result: &BattalionResult) -> Result<String, HeraldError>;

    /// Format a streaming chunk
    fn format_stream_chunk(&self, chunk: &StreamChunk) -> Result<String, HeraldError>;
}

Integration with Paladin

let paladin = PaladinBuilder::new(llm_adapter)
    .name("Assistant")
    .system_prompt("You are a helpful assistant.")
    .output_format(OutputFormat::Text)
    .with_herald(Arc::new(MarkdownHerald::default()))
    .build()?;

let response = paladin.execute("Explain async/await").await?;
// response.content is formatted as Markdown

Built-in Formatters

Plain Text Herald

No formatting, returns raw LLM output.

use paladin_core::platform::container::herald::{Herald, HeraldError};
use paladin::infrastructure::adapters::herald::{JsonHerald, MarkdownHerald, TableHerald};

let herald = Arc::new(MarkdownHerald::new());

let paladin = PaladinBuilder::new(llm_adapter)
    .with_herald(herald)
    .build()?;

let response = paladin.execute("Hello").await?;
println!("{}", response.content);  // Raw output

Markdown Herald

Formats output as Markdown with proper structure.

use paladin_core::platform::container::herald::{Herald, HeraldError};
use paladin::infrastructure::adapters::herald::{JsonHerald, MarkdownHerald, TableHerald};

let herald = Arc::new(MarkdownHerald::new()
    .with_code_highlighting(true)
    .with_header_ids(true)
    .with_table_of_contents(true)
);

let paladin = PaladinBuilder::new(llm_adapter)
    .system_prompt("Format all responses as Markdown with proper headers and code blocks.")
    .with_herald(herald)
    .build()?;

let response = paladin.execute("Explain Rust ownership").await?;
println!("{}", response.content);

Output example:

# Rust Ownership

Ownership is a core concept in Rust that ensures memory safety.

## Key Rules

1. Each value has a single owner
2. When the owner goes out of scope, the value is dropped
3. Values can be borrowed immutably or mutably

## Example

```rust,ignore
fn main() {
    let s1 = String::from("hello");
    let s2 = s1;  // s1 is moved
    // println!("{}", s1);  // Error: s1 is no longer valid
}

Benefits

• Memory safety without garbage collection
• No data races at compile time
• Zero-cost abstractions

### JSON Herald

Formats output as structured JSON.

```rust,ignore
use paladin_core::platform::container::herald::{Herald, HeraldError};
use paladin::infrastructure::adapters::herald::{JsonHerald, MarkdownHerald, TableHerald};
use serde_json::json;

let herald = Arc::new(JsonHerald::new()
    .with_schema(json!({
        "type": "object",
        "properties": {
            "summary": {"type": "string"},
            "key_points": {
                "type": "array",
                "items": {"type": "string"}
            },
            "confidence": {"type": "number"}
        },
        "required": ["summary", "key_points"]
    }))
    .validate_output(true)
);

let paladin = PaladinBuilder::new(llm_adapter)
    .system_prompt("Always respond in JSON format matching this schema: \
                    {summary: string, key_points: string[], confidence: number}")
    .with_herald(herald)
    .build()?;

let response = paladin.execute("Analyze sentiment of: 'This product is amazing!'").await?;

// Parse structured output
let json: serde_json::Value = serde_json::from_str(&response.content)?;
println!("Summary: {}", json["summary"]);
println!("Key points: {:?}", json["key_points"]);

Output example:

{
  "summary": "Highly positive sentiment expressing enthusiasm",
  "key_points": [
    "Strong positive emotion indicated by 'amazing'",
    "Exclamation mark reinforces enthusiasm",
    "No negative indicators present"
  ],
  "confidence": 0.95
}

HTML Herald

Formats output as styled HTML.

use paladin_core::platform::container::herald::{Herald, HeraldError};
use paladin::infrastructure::adapters::herald::{JsonHerald, MarkdownHerald, TableHerald};

let herald = Arc::new(JsonHerald::new()
    .with_css_framework(CssFramework::Tailwind)
    .with_syntax_highlighting(true)
    .with_responsive_design(true)
);

let paladin = PaladinBuilder::new(llm_adapter)
    .with_herald(herald)
    .build()?;

let response = paladin.execute("Create a todo list").await?;

// Serve as web page
let html = format!(r#"
<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8">
    <title>Paladin Response</title>
    <link href="https://cdn.jsdelivr.net/npm/tailwindcss@2/dist/tailwind.min.css" rel="stylesheet">
</head>
<body class="bg-gray-100 p-8">
    {}
</body>
</html>
"#, response.content);

Code Herald

Specialized for code generation with syntax validation.

use paladin_core::platform::container::herald::{Herald, HeraldError};
use paladin::infrastructure::adapters::herald::{JsonHerald, MarkdownHerald, TableHerald};

let herald = Arc::new(CodeHerald::new()
    .language("rust")
    .with_syntax_check(true)
    .with_formatting(true)
    .with_linting(true)
);

let paladin = PaladinBuilder::new(llm_adapter)
    .system_prompt("You are a Rust code generator. Return ONLY valid Rust code.")
    .with_herald(herald)
    .build()?;

let response = paladin.execute("Write a function to reverse a string").await?;

// Output is validated, formatted Rust code
println!("{}", response.content);

Output:

pub fn reverse_string(s: &str) -> String {
    s.chars().rev().collect()
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_reverse_string() {
        assert_eq!(reverse_string("hello"), "olleh");
        assert_eq!(reverse_string(""), "");
    }
}

Custom Formatters

Create custom heralds for specialized output formats.

Simple Custom Herald

use paladin_core::platform::container::herald::{Herald, HeraldError};
use paladin::infrastructure::adapters::herald::{JsonHerald, MarkdownHerald, TableHerald};
use async_trait::async_trait;

pub struct UppercaseHerald;

impl Herald for UppercaseHerald {
    fn format_paladin_result(&self, result: &PaladinResult) -> Result<String, HeraldError> {
        Ok(content.to_uppercase())
    }

    fn format_stream_chunk(&self, chunk: &StreamChunk) -> Result<String, HeraldError> {
        Ok(chunk.to_uppercase())
    }
}

// Usage
let herald = Arc::new(UppercaseHerald);
let paladin = PaladinBuilder::new(llm_adapter)
    .with_herald(herald)
    .build()?;

XML Herald

use paladin_core::platform::container::herald::{Herald, HeraldError};
use paladin::infrastructure::adapters::herald::{JsonHerald, MarkdownHerald, TableHerald};
use quick_xml::Writer;
use std::io::Cursor;

pub struct XmlHerald {
    root_element: String,
}

impl XmlHerald {
    pub fn new(root_element: &str) -> Self {
        Self {
            root_element: root_element.to_string(),
        }
    }
}

impl Herald for XmlHerald {
    fn format_paladin_result(&self, result: &PaladinResult) -> Result<String, HeraldError> {
        let mut writer = Writer::new(Cursor::new(Vec::new()));

        // Write XML declaration
        writer.write_event(quick_xml::events::Event::Decl(
            quick_xml::events::BytesDecl::new("1.0", Some("UTF-8"), None)
        ))?;

        // Parse content as structured data
        let data: serde_json::Value = serde_json::from_str(content)
            .map_err(|e| HeraldError::FormatError(e.to_string()))?;

        // Convert to XML
        self.json_to_xml(&mut writer, &self.root_element, &data)?;

        let xml_bytes = writer.into_inner().into_inner();
        Ok(String::from_utf8(xml_bytes)?)
    }
}

// Usage
let herald = Arc::new(XmlHerald::new("response"));

let paladin = PaladinBuilder::new(llm_adapter)
    .system_prompt("Return JSON that will be converted to XML")
    .with_herald(herald)
    .build()?;

CSV Herald

use paladin_core::platform::container::herald::{Herald, HeraldError};
use paladin::infrastructure::adapters::herald::{JsonHerald, MarkdownHerald, TableHerald};
use csv::Writer;

pub struct CsvHerald {
    headers: Vec<String>,
    delimiter: u8,
}

impl CsvHerald {
    pub fn new(headers: Vec<String>) -> Self {
        Self {
            headers,
            delimiter: b',',
        }
    }

    pub fn with_delimiter(mut self, delimiter: u8) -> Self {
        self.delimiter = delimiter;
        self
    }
}

impl Herald for CsvHerald {
    fn format_paladin_result(&self, result: &PaladinResult) -> Result<String, HeraldError> {
        // Parse JSON array
        let rows: Vec<serde_json::Value> = serde_json::from_str(content)
            .map_err(|e| HeraldError::FormatError(e.to_string()))?;

        let mut wtr = Writer::from_writer(vec![]);

        // Write headers
        wtr.write_record(&self.headers)?;

        // Write data rows
        for row in rows {
            let record: Vec<String> = self.headers.iter()
                .map(|h| {
                    row.get(h)
                        .map(|v| v.to_string())
                        .unwrap_or_default()
                })
                .collect();

            wtr.write_record(&record)?;
        }

        wtr.flush()?;
        let csv_bytes = wtr.into_inner()?;
        Ok(String::from_utf8(csv_bytes)?)
    }
}

// Usage
let herald = Arc::new(CsvHerald::new(vec![
    "name".to_string(),
    "age".to_string(),
    "city".to_string(),
]));

let paladin = PaladinBuilder::new(llm_adapter)
    .system_prompt("Return data as JSON array of objects with name, age, city fields")
    .with_herald(herald)
    .build()?;

let response = paladin.execute("Generate 5 sample user records").await?;
// Output is formatted CSV

Streaming Output

Process and format output in real-time for better user experience.

Basic Streaming

use paladin_core::platform::container::herald::{Herald, HeraldError};
use paladin::infrastructure::adapters::herald::{JsonHerald, MarkdownHerald, TableHerald};
use futures::StreamExt;

let herald = Arc::new(MarkdownHerald::default());

let paladin = PaladinBuilder::new(llm_adapter)
    .with_herald(herald.clone())
    .build()?;

// Execute with streaming
let mut stream = paladin.execute_stream("Write a story").await?;

while let Some(chunk) = stream.next().await {
    let chunk = chunk?;

    // Format chunk
    let formatted = herald.format_chunk(&chunk.content).await?;

    // Print in real-time
    print!("{}", formatted);
    std::io::stdout().flush()?;
}
println!();

Streaming with Accumulation

pub struct StreamAccumulator {
    herald: Arc<dyn Herald>,
    buffer: String,
}

impl StreamAccumulator {
    pub fn new(herald: Arc<dyn Herald>) -> Self {
        Self {
            herald,
            buffer: String::new(),
        }
    }

    pub async fn process_chunk(&mut self, chunk: &str) -> Result<String, HeraldError> {
        self.buffer.push_str(chunk);

        // Format accumulated content
        self.herald.format(&self.buffer).await
    }

    pub fn buffer(&self) -> &str {
        &self.buffer
    }
}

// Usage
let mut accumulator = StreamAccumulator::new(herald);
let mut stream = paladin.execute_stream("Explain quantum computing").await?;

while let Some(chunk) = stream.next().await {
    let chunk = chunk?;
    let formatted_so_far = accumulator.process_chunk(&chunk.content).await?;

    // Update UI with fully formatted content
    update_ui(&formatted_so_far);
}

Progress Indicators

pub struct ProgressHerald {
    inner: Arc<dyn Herald>,
    show_progress: bool,
}

impl Herald for ProgressHerald {
    fn format_stream_chunk(&self, chunk: &StreamChunk) -> Result<String, HeraldError> {
        let formatted = self.inner.format_chunk(chunk).await?;

        if self.show_progress {
            // Add visual progress indicator
            Ok(format!("{} .", formatted))
        } else {
            Ok(formatted)
        }
    }

    fn format_paladin_result(&self, result: &PaladinResult) -> Result<String, HeraldError> {
        self.inner.format_paladin_result(result)
    }
}

Multi-Format Output

Generate output in multiple formats simultaneously.

Multi-Format Herald

pub struct MultiFormatHerald {
    heralds: HashMap<String, Arc<dyn Herald>>,
}

impl MultiFormatHerald {
    pub fn new() -> Self {
        Self {
            heralds: HashMap::new(),
        }
    }

    pub fn add_format(mut self, name: &str, herald: Arc<dyn Herald>) -> Self {
        self.heralds.insert(name.to_string(), herald);
        self
    }

    pub async fn format_all(&self, content: &str) -> Result<HashMap<String, String>, HeraldError> {
        let mut results = HashMap::new();

        for (name, herald) in &self.heralds {
            let formatted = herald.format(content).await?;
            results.insert(name.clone(), formatted);
        }

        Ok(results)
    }
}

// Usage
let multi_herald = MultiFormatHerald::new()
    .add_format("json", Arc::new(JsonHerald::default()))
    .add_format("markdown", Arc::new(MarkdownHerald::default()))
    .add_format("html", Arc::new(JsonHerald::new()));

let paladin = PaladinBuilder::new(llm_adapter).build()?;
let response = paladin.execute("Summarize Rust features").await?;

// Generate all formats
let all_formats = multi_herald.format_all(&response.content).await?;

// Save or serve each format
std::fs::write("output.json", &all_formats["json"])?;
std::fs::write("output.md", &all_formats["markdown"])?;
std::fs::write("output.html", &all_formats["html"])?;

Adaptive Format Selection

pub struct AdaptiveHerald {
    formats: HashMap<String, Arc<dyn Herald>>,
    default: Arc<dyn Herald>,
}

impl AdaptiveHerald {
    pub async fn format_for_context(
        &self,
        content: &str,
        context: &OutputContext,
    ) -> Result<String, HeraldError> {
        let herald = self.select_herald(context);
        herald.format(content).await
    }

    fn select_herald(&self, context: &OutputContext) -> &Arc<dyn Herald> {
        match context.channel {
            OutputChannel::Web => self.formats.get("html").unwrap_or(&self.default),
            OutputChannel::Api => self.formats.get("json").unwrap_or(&self.default),
            OutputChannel::Terminal => self.formats.get("markdown").unwrap_or(&self.default),
            OutputChannel::File(ref ext) => {
                self.formats.get(ext.as_str()).unwrap_or(&self.default)
            }
        }
    }
}

pub struct OutputContext {
    pub channel: OutputChannel,
    pub user_preferences: HashMap<String, String>,
}

pub enum OutputChannel {
    Web,
    Api,
    Terminal,
    File(String),
}

// Usage
let adaptive = AdaptiveHerald::new()
    .with_format("html", Arc::new(JsonHerald::new()))
    .with_format("json", Arc::new(JsonHerald::default()))
    .with_format("markdown", Arc::new(MarkdownHerald::default()))
    .with_default(Arc::new(MarkdownHerald::new()));

// Format based on context
let web_output = adaptive.format_for_context(
    &content,
    &OutputContext {
        channel: OutputChannel::Web,
        user_preferences: HashMap::new(),
    }
).await?;

let api_output = adaptive.format_for_context(
    &content,
    &OutputContext {
        channel: OutputChannel::Api,
        user_preferences: HashMap::new(),
    }
).await?;

Post-Processing

Transform or enhance output after formatting.

Sanitization Herald

pub struct SanitizingHerald {
    inner: Arc<dyn Herald>,
    remove_patterns: Vec<regex::Regex>,
}

impl SanitizingHerald {
    pub fn new(inner: Arc<dyn Herald>) -> Self {
        Self {
            inner,
            remove_patterns: vec![
                // Remove potential PII
                regex::Regex::new(r"\b\d{3}-\d{2}-\d{4}\b").unwrap(),  // SSN
                regex::Regex::new(r"\b[\w\.-]+@[\w\.-]+\.\w+\b").unwrap(),  // Email
                regex::Regex::new(r"\b\d{3}-\d{3}-\d{4}\b").unwrap(),  // Phone
            ],
        }
    }
}

impl Herald for SanitizingHerald {
    fn format_paladin_result(&self, result: &PaladinResult) -> Result<String, HeraldError> {
        let formatted = self.inner.format(content).await?;

        // Remove sensitive patterns
        let mut sanitized = formatted;
        for pattern in &self.remove_patterns {
            sanitized = pattern.replace_all(&sanitized, "[REDACTED]").to_string();
        }

        Ok(sanitized)
    }

    // Implement other methods...
}

Enhancement Herald

pub struct EnhancingHerald {
    inner: Arc<dyn Herald>,
}

impl Herald for EnhancingHerald {
    fn format_paladin_result(&self, result: &PaladinResult) -> Result<String, HeraldError> {
        let formatted = self.inner.format(content).await?;

        // Add enhancements
        let enhanced = self.add_table_of_contents(&formatted);
        let enhanced = self.add_footnotes(&enhanced);
        let enhanced = self.add_timestamps(&enhanced);

        Ok(enhanced)
    }

    fn add_table_of_contents(&self, content: &str) -> String {
        // Extract headers and generate TOC
        let headers = self.extract_headers(content);

        if headers.is_empty() {
            return content.to_string();
        }

        let toc = headers.iter()
            .map(|(level, text, id)| {
                let indent = "  ".repeat(*level - 1);
                format!("{}* [{}](#{})", indent, text, id)
            })
            .collect::<Vec<_>>()
            .join("\n");

        format!("## Table of Contents\n\n{}\n\n{}", toc, content)
    }

    fn add_footnotes(&self, content: &str) -> String {
        // Process [^1] style footnote references
        // Implementation...
        content.to_string()
    }

    fn add_timestamps(&self, content: &str) -> String {
        format!("Generated at: {}\n\n{}", chrono::Utc::now().to_rfc3339(), content)
    }
}

Caching Herald

use std::collections::HashMap;
use std::sync::RwLock;

pub struct CachingHerald {
    inner: Arc<dyn Herald>,
    cache: RwLock<HashMap<String, String>>,
    max_cache_size: usize,
}

impl Herald for CachingHerald {
    fn format_paladin_result(&self, result: &PaladinResult) -> Result<String, HeraldError> {
        // Check cache
        {
            let cache = self.cache.read().unwrap();
            if let Some(cached) = cache.get(content) {
                return Ok(cached.clone());
            }
        }

        // Format
        let formatted = self.inner.format(content).await?;

        // Store in cache
        {
            let mut cache = self.cache.write().unwrap();

            // Evict oldest if at capacity
            if cache.len() >= self.max_cache_size {
                if let Some(key) = cache.keys().next().cloned() {
                    cache.remove(&key);
                }
            }

            cache.insert(content.to_string(), formatted.clone());
        }

        Ok(formatted)
    }

    // Implement other methods...
}

Best Practices

1. Match Format to Use Case

// ✅ API endpoints - use JSON
let api_herald = Arc::new(JsonHerald::new()
    .with_schema(api_schema)
    .validate_output(true)
);

// ✅ Documentation - use Markdown
let docs_herald = Arc::new(MarkdownHerald::new()
    .with_table_of_contents(true)
    .with_code_highlighting(true)
);

// ✅ Web display - use HTML
let web_herald = Arc::new(JsonHerald::new()
    .with_css_framework(CssFramework::Bootstrap)
    .with_responsive_design(true)
);

// ✅ Data export - use CSV
let export_herald = Arc::new(CsvHerald::new(headers));

2. Validate Structured Output

let herald = Arc::new(JsonHerald::new()
    .with_schema(schema)
    .validate_output(true)  // Validate against schema
);

// Paladin will retry if output doesn't match schema
let paladin = PaladinBuilder::new(llm_adapter)
    .system_prompt("CRITICAL: Output MUST be valid JSON matching the schema")
    .with_herald(herald)
    .max_retries(3)  // Retry on validation failures
    .build()?;

3. Use Streaming for Long Responses

// ❌ Bad: Wait for complete response
let response = paladin.execute(long_prompt).await?;
println!("{}", response.content);  // User waits 30 seconds

// ✅ Good: Stream for immediate feedback
let mut stream = paladin.execute_stream(long_prompt).await?;
while let Some(chunk) = stream.next().await {
    let chunk = chunk?;
    print!("{}", chunk.content);  // Immediate output
    std::io::stdout().flush()?;
}

4. Layer Heralds for Composability

// Layer: Base -> Enhancement -> Sanitization -> Caching
let herald = Arc::new(
    CachingHerald::new(
        Arc::new(SanitizingHerald::new(
            Arc::new(EnhancingHerald::new(
                Arc::new(MarkdownHerald::default())
            ))
        )),
        100,  // cache size
    )
);

5. Provide Format Guidance in System Prompt

// ✅ Explicit format instructions
let paladin = PaladinBuilder::new(llm_adapter)
    .system_prompt(
        "You MUST respond in valid JSON format:\n\
         {\n\
           \"answer\": \"your response\",\n\
           \"confidence\": 0.0 to 1.0,\n\
           \"sources\": [\"source1\", \"source2\"]\n\
         }\n\
         Do NOT include any text outside this JSON structure."
    )
    .with_herald(Arc::new(JsonHerald::default()))
    .build()?;

Advanced Patterns

Template-Based Herald

use handlebars::Handlebars;

pub struct TemplateHerald {
    handlebars: Handlebars<'static>,
    template_name: String,
}

impl TemplateHerald {
    pub fn new(template: &str, template_name: &str) -> Result<Self, HeraldError> {
        let mut handlebars = Handlebars::new();
        handlebars.register_template_string(template_name, template)?;

        Ok(Self {
            handlebars,
            template_name: template_name.to_string(),
        })
    }
}

impl Herald for TemplateHerald {
    fn format_paladin_result(&self, result: &PaladinResult) -> Result<String, HeraldError> {
        // Parse content as JSON
        let data: serde_json::Value = serde_json::from_str(content)?;

        // Render template
        let rendered = self.handlebars.render(&self.template_name, &data)?;

        Ok(rendered)
    }

    // Implement other methods...
}

// Usage
let template = r#"
{{title}}

**Summary:** {{summary}}

# Details

{{#each items}}
- {{this}}
{{/each}}

*Generated: {{timestamp}}*
"#;

let herald = Arc::new(TemplateHerald::new(template, "report")?);

let paladin = PaladinBuilder::new(llm_adapter)
    .system_prompt("Return JSON: {title, summary, items: [], timestamp}")
    .with_herald(herald)
    .build()?;

Diff Herald

pub struct DiffHerald {
    previous_content: RwLock<Option<String>>,
}

impl Herald for DiffHerald {
    fn format_paladin_result(&self, result: &PaladinResult) -> Result<String, HeraldError> {
        let previous = self.previous_content.read().unwrap().clone();

        let formatted = if let Some(prev) = previous {
            // Generate diff
            self.generate_diff(&prev, content)
        } else {
            // First time, show all
            content.to_string()
        };

        // Update previous content
        *self.previous_content.write().unwrap() = Some(content.to_string());

        Ok(formatted)
    }

    fn generate_diff(&self, old: &str, new: &str) -> String {
        // Use diff algorithm
        // Implementation...
        format!("--- Old\n+++ New\n{}", new)
    }
}

Troubleshooting

Invalid JSON Output

Problem: JSON Herald fails to parse LLM output.

Solutions:

1. Strengthen system prompt with explicit JSON instructions
2. Add JSON schema to prompt
3. Enable output validation with retries
4. Use JSON mode in LLM if supported

let paladin = PaladinBuilder::new(llm_adapter)
    .system_prompt(
        "CRITICAL INSTRUCTION: You MUST respond with ONLY valid JSON. \
         No additional text before or after. No markdown code blocks. \
         Just pure JSON.\n\n\
         Schema: {\"result\": string, \"confidence\": number}"
    )
    .output_format(OutputFormat::Json)  // Some LLMs support JSON mode
    .max_retries(3)
    .build()?;

Streaming Format Inconsistency

Problem: Streamed chunks don't format correctly.

Solutions:

1. Use accumulation pattern
2. Implement chunk boundary detection
3. Buffer until complete format units

pub struct BufferedStreamHerald {
    buffer: RwLock<String>,
    delimiter: String,
}

impl BufferedStreamHerald {
    fn format_stream_chunk(&self, chunk: &StreamChunk) -> Result<String, HeraldError> {
        let mut buffer = self.buffer.write().unwrap();
        buffer.push_str(chunk);

        // Check for complete units (e.g., sentences, paragraphs)
        if buffer.ends_with(&self.delimiter) {
            let complete = buffer.clone();
            buffer.clear();
            Ok(complete)
        } else {
            Ok(String::new())  // Not ready yet
        }
    }
}

Performance Issues with Complex Formatting

Problem: Formatting is slow for large outputs.

Solutions:

1. Implement caching
2. Use lazy formatting (format on demand)
3. Optimize regex patterns
4. Consider parallel processing

// Lazy formatting
pub struct LazyHerald {
    inner: Arc<dyn Herald>,
    cached_result: RwLock<Option<String>>,
}

impl LazyHerald {
    pub async fn get_formatted(&self, content: &str) -> Result<String, HeraldError> {
        // Check cache
        if let Some(cached) = self.cached_result.read().unwrap().as_ref() {
            return Ok(cached.clone());
        }

        // Format and cache
        let formatted = self.inner.format(content).await?;
        *self.cached_result.write().unwrap() = Some(formatted.clone());

        Ok(formatted)
    }
}

Testing

Unit Testing Heralds

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn test_json_herald_formats_correctly() {
        let herald = JsonHerald::default();

        let input = r#"{"name": "Alice", "age": 30}"#;
        let formatted = herald.format(input).await.unwrap();

        // Verify valid JSON
        let parsed: serde_json::Value = serde_json::from_str(&formatted).unwrap();
        assert_eq!(parsed["name"], "Alice");
        assert_eq!(parsed["age"], 30);
    }

    #[tokio::test]
    async fn test_json_herald_validates_schema() {
        let schema = json!({
            "type": "object",
            "properties": {
                "name": {"type": "string"}
            },
            "required": ["name"]
        });

        let herald = JsonHerald::new().with_schema(schema);

        // Valid
        assert!(herald.validate(r#"{"name": "Bob"}"#).is_ok());

        // Invalid - missing required field
        assert!(herald.validate(r#"{"age": 25}"#).is_err());
    }
}

Examples

See working examples:

• examples/herald_markdown_output.rs - Markdown formatting
• examples/herald_json_output.rs - Structured JSON output
• examples/herald_streaming.rs - Real-time streaming
• examples/herald_custom_formatter.rs - Custom herald implementation

Next Steps

• Tool Integration - Format tool results
• Battalion Patterns - Format multi-agent outputs
• API Reference - Herald API documentation

Related Resources

• Handlebars Templates
• JSON Schema
• Markdown Specification

Architecture Overview

Paladin is a Rust workspace of nine focused crates organised around Hexagonal Architecture (Ports & Adapters) and Domain-Driven Design. Each workspace crate maps to a distinct architectural layer, keeping the core domain free of all external dependencies.

For how to run agents built on this architecture — embedded, hosted, queue/worker, or sidecar — see Deployment Topologies.

Workspace Crates at a Glance

Three-Layer Hexagonal Architecture

┌─────────────────────────────────────────────────────────────────┐
│                      External World                              │
│   LLMs · Databases · Redis · MinIO · MCP tools · HTTP clients   │
└──────────────┬──────────────────────────────────┬───────────────┘
               │                                  │
               │  Infrastructure adapters         │
               │  paladin-llm                     │
               │  paladin-memory                  │
               │  paladin-storage                 │
               │  paladin-notifications            │
               │  paladin-content                 │
               │  paladin-web                     │
               │                                  │
┌──────────────▼──────────────────────────────────▼───────────────┐
│               Application Boundary  (paladin-ports)              │
│   LlmPort · GarrisonPort · SanctumPort · ArsenalPort            │
│   CitadelPort · FileStoragePort · NotificationPort · …          │
│                                                                  │
│               Application Services  (paladin-battalion)          │
│   FormationService · PhalanxService · CampaignService           │
│   ChainOfCommandService · Commander · ConclaveService           │
│   CouncilService · GroveService · ManeuverService               │
└──────────────┬──────────────────────────────────┬───────────────┘
               │  depends on (inward only)         │
┌──────────────▼──────────────────────────────────▼───────────────┐
│                   Core Domain  (paladin-ai-core)                  │
│   Paladin · Battalion · Garrison · Arsenal · Citadel             │
│   Herald · Sanctum · Node<T> · PaladinError · …                 │
│   No I/O · No external SDK imports · Pure domain logic           │
└─────────────────────────────────────────────────────────────────┘

Dependency Flow Rule

Dependencies flow inward only:

• paladin-ai-core imports nothing from the workspace.
• paladin-ports imports only paladin-ai-core.
• paladin-battalion imports paladin-ai-core + paladin-ports.
• Infrastructure crates (paladin-llm, paladin-memory, etc.) import paladin-ai-core + paladin-ports. They never import each other.
• The root paladin-ai umbrella crate imports everything.

This rule is enforced by Cargo's dependency graph — paladin-ai-core cannot accidentally pull in reqwest or sqlx.

Layer 1: Core Domain (crates/paladin-core)

Package name: paladin-ai-core

Pure business logic with zero external dependencies.

crates/paladin-core/src/
├── base/                      # Framework primitives
│   ├── node.rs                # Node<T> entity pattern
│   ├── collection.rs
│   ├── field.rs
│   └── message.rs
└── platform/
    ├── container/
    │   ├── paladin.rs             # Paladin aggregate root
    │   ├── paladin_config.rs
    │   ├── paladin_error.rs
    │   ├── garrison.rs            # Garrison memory domain
    │   ├── arsenal/               # Tool system domain
    │   ├── citadel.rs             # State persistence domain
    │   ├── herald.rs              # Output formatting domain
    │   ├── sanctum.rs             # Vector memory domain
    │   └── battalion/             # Battalion domain types
    └── manager/
        ├── scheduler.rs
        └── event_manager.rs

Constraints:

• No imports from paladin-ports or any infrastructure crate
• No I/O operations
• No HTTP clients, database drivers, or LLM SDKs

Layer 2: Application Boundary (crates/paladin-ports + crates/paladin-battalion)

Port Contracts (paladin-ports)

Defines abstract trait interfaces for every external integration point:

crates/paladin-ports/src/
├── output/
│   ├── llm_port.rs              # LLM provider abstraction
│   ├── garrison_port.rs         # Memory CRUD operations
│   ├── sanctum_port.rs          # Vector memory search
│   ├── arsenal_port.rs          # Tool invocation
│   ├── citadel_port.rs          # State persistence
│   ├── file_storage_port.rs     # File upload/download
│   ├── notification_port.rs     # Alert delivery
│   ├── queue_port.rs            # Async task queue
│   └── …
└── input/
    ├── content_delivery_port.rs
    └── …

Orchestration Services (paladin-battalion)

crates/paladin-battalion/src/
├── formation_service.rs         # Sequential pipeline (N→N+1)
├── phalanx_service.rs           # Concurrent (parallel) execution
├── campaign_service.rs          # DAG / graph-based execution
├── chain_of_command_service.rs  # Hierarchical delegation
├── conclave_execution_service.rs # Mixture-of-experts synthesis
├── council_service.rs           # Multi-agent discussion
├── grove_service.rs             # Semantic routing
├── maneuver/                    # Flow DSL (parser + runtime)
└── commander.rs                 # Auto-detect strategy router

Layer 3: Infrastructure Adapters

Each adapter implements the corresponding port trait from paladin-ports.

System Components

Paladin (Agent)

Create via PaladinBuilder
        │
        ▼
   ┌─────────┐
   │  Idle   │ ← waiting for input
   └────┬────┘
        │  execute()
        ▼
   ┌─────────┐
   │ Running │ ← LLM reasoning loop (1..max_loops)
   └────┬────┘
        ├── tool call? → Arsenal.invoke() → inject result → continue
        ├── stop word? → StopWordDetected
        └── max loops? → MaxLoops

Battalion (Orchestration)

Eight patterns routed by the Commander auto-detector:

Garrison (Short-term Memory)

Conversation history stored in paladin-memory:

• InMemoryGarrison — always available, zero deps
• SqliteGarrison — persistent (feature sqlite)

Configured via garrison: section in config.yml.

Sanctum (Long-term Vector Memory)

Semantic memory in paladin-memory:

• InMemorySanctum — in-process (testing / dev)
• QdrantSanctumAdapter — production (feature qdrant)

Arsenal (Tool System)

MCP-compatible tool registry. Connects to external tools via:

• STDIO process servers (command-line tools)
• SSE HTTP servers (web services)

Configured via arsenal.mcp_servers in config.yml.

Technology Stack

See Also

• Hexagonal Design — port and adapter patterns in detail
• Domain Model — all domain entities and the Medieval Military naming convention
• Crate Map — workspace crate dependency graph
• Design Patterns — PaladinBuilder, error types, port traits

Hexagonal Architecture (Ports & Adapters)

Paladin uses Hexagonal Architecture to keep the core domain testable, swappable, and free of infrastructure lock-in.

Core Concepts

The rule: dependencies point inward only.

External world
   ↓ (adapters in paladin-llm, paladin-memory, …)
paladin-ports   (trait contracts)
   ↓
paladin-ai-core (pure domain)

Port Traits

All port traits live in crates/paladin-ports/src/output/ (infrastructure-facing ports) or src/input/ (ingestion-facing ports).

LLM Port

// crates/paladin-ports/src/output/llm_port.rs
#[async_trait]
pub trait LlmPort: Send + Sync {
    async fn generate(
        &self,
        messages: &[Message],
        config: &LlmConfig,
    ) -> Result<LlmResponse, LlmError>;

    async fn generate_stream(
        &self,
        messages: &[Message],
        config: &LlmConfig,
    ) -> Result<Pin<Box<dyn Stream<Item = Result<String, LlmError>> + Send>>, LlmError>;
}

Garrison Port

// crates/paladin-ports/src/output/garrison_port.rs
#[async_trait]
pub trait GarrisonPort: Send + Sync {
    async fn add_entry(&self, entry: GarrisonEntry) -> Result<(), GarrisonError>;
    async fn get_window(&self, max_tokens: usize) -> Result<Vec<GarrisonEntry>, GarrisonError>;
    async fn clear(&self) -> Result<(), GarrisonError>;
}

Sanctum Port

// crates/paladin-ports/src/output/sanctum_port.rs
#[async_trait]
pub trait SanctumPort: Send + Sync {
    async fn store(&self, memory: Memory) -> Result<MemoryId, SanctumError>;
    async fn search(&self, query: &str, top_k: usize) -> Result<Vec<Memory>, SanctumError>;
}

Arsenal Port

// crates/paladin-ports/src/output/arsenal_port.rs
#[async_trait]
pub trait ArsenalPort: Send + Sync {
    async fn list_tools(&self) -> Result<Vec<ToolDefinition>, ArsenalError>;
    async fn invoke(&self, call: &ToolCall) -> Result<ToolResult, ArsenalError>;
}

File Storage Port

// crates/paladin-ports/src/output/file_storage_port.rs
#[async_trait]
pub trait FileStoragePort: Send + Sync {
    async fn upload(&self, key: &str, data: Vec<u8>) -> Result<(), StorageError>;
    async fn download(&self, key: &str) -> Result<Vec<u8>, StorageError>;
}

Adapter Implementations

Each port trait is implemented by one or more adapters in an infrastructure crate.

LLM Adapters (crates/paladin-llm)

// crates/paladin-llm/src/openai/mod.rs
pub struct OpenAIAdapter { /* ... */ }

#[async_trait]
impl LlmPort for OpenAIAdapter {
    async fn generate(
        &self,
        messages: &[Message],
        config: &LlmConfig,
    ) -> Result<LlmResponse, LlmError> {
        // calls https://api.openai.com/v1/chat/completions
    }
}

Memory Adapters (crates/paladin-memory)

Storage Adapters (crates/paladin-storage)

Adding a New Adapter

Follow these steps to add, say, a PostgreSQL Garrison adapter:

1. Create the adapter file in the appropriate infrastructure crate:

crates/paladin-memory/src/garrison/postgres_garrison.rs

2. Implement the port trait:

// crates/paladin-memory/src/garrison/postgres_garrison.rs
use paladin_ports::output::garrison_port::GarrisonPort;

pub struct PostgresGarrison {
    pool: sqlx::PgPool,
}

#[async_trait]
impl GarrisonPort for PostgresGarrison {
    async fn add_entry(&self, entry: GarrisonEntry) -> Result<(), GarrisonError> {
        // INSERT INTO garrison ...
    }

    async fn get_window(&self, max_tokens: usize) -> Result<Vec<GarrisonEntry>, GarrisonError> {
        // SELECT ... ORDER BY created_at DESC LIMIT ...
    }

    async fn clear(&self) -> Result<(), GarrisonError> {
        // DELETE FROM garrison
    }
}

3. Gate behind a feature flag in crates/paladin-memory/Cargo.toml:

[features]
postgres = ["sqlx/postgres"]

[dependencies]
sqlx = { version = "0.7", optional = true }

4. Export from lib.rs under the feature gate:

#[cfg(feature = "postgres")]
pub mod postgres_garrison;

5. Write tests using the existing garrison integration test pattern.

6. Document the adapter in Garrison Memory.

Dependency Injection with Arc<dyn Port>

Services receive port implementations via Arc<dyn Trait>. This is how the application layer stays decoupled from concrete adapters:

use std::sync::Arc;
use paladin_ports::output::llm_port::LlmPort;
use paladin_ports::output::garrison_port::GarrisonPort;

pub struct PaladinExecutionService {
    llm:      Arc<dyn LlmPort>,
    garrison: Option<Arc<dyn GarrisonPort>>,
}

impl PaladinExecutionService {
    pub fn new(llm: Arc<dyn LlmPort>) -> Self {
        Self { llm, garrison: None }
    }

    pub fn with_garrison(mut self, g: Arc<dyn GarrisonPort>) -> Self {
        self.garrison = Some(g);
        self
    }
}

Swap implementations at construction time — no code changes needed in the service.

Testing with Mock Adapters

Use MockLlmAdapter (from paladin-llm with the mock feature) in unit tests:

use paladin_llm::mock::MockLlmAdapter;
use std::sync::Arc;
use paladin_ports::output::llm_port::LlmPort;

let mock: Arc<dyn LlmPort> = Arc::new(
    MockLlmAdapter::new().with_response("Test response".to_string())
);
let service = PaladinExecutionService::new(mock);

See Also

• Architecture Overview
• Crate Map — workspace dependency graph
• Design Patterns — PaladinBuilder, error enums, service composition
• Contributing Providers — adding LLM providers

Domain Model

This document describes all domain entities in paladin-ai-core and the Medieval Military naming convention that provides Paladin's ubiquitous language.

Medieval Military Naming Convention

Paladin applies Domain-Driven Design's ubiquitous language principle through a consistent Medieval Military theme. Use these terms in all code, documentation, and discussions:

Node Pattern

All domain entities use the Node<T> wrapper which adds identifier, timestamps, and metadata to any data payload:

// crates/paladin-core/src/base/node.rs
pub struct Node<T> {
    pub id:         Uuid,
    pub created_at: DateTime<Utc>,
    pub updated_at: DateTime<Utc>,
    pub node:       T,         // The domain payload
}

Domain types are type aliases:

// crates/paladin-core/src/platform/container/paladin.rs
pub type Paladin = Node<PaladinData>;

Core Domain Entities

Paladin (Aggregate Root)

// PaladinData — the domain payload inside Node<PaladinData>
pub struct PaladinData {
    pub name:          String,
    pub system_prompt: String,
    pub model:         String,
    pub temperature:   f32,
    pub max_loops:     u32,
    pub stop_words:    Vec<String>,
    pub status:        PaladinStatus,
}

pub enum PaladinStatus {
    Idle,
    Running,
    Completed,
    Failed,
    Timeout,
}

pub type Paladin = Node<PaladinData>;

Invariants:

• system_prompt must not be empty
• temperature must be in 0.0..=2.0
• max_loops must be >= 1

Garrison (Memory Domain)

// crates/paladin-core/src/platform/container/garrison.rs
pub struct GarrisonEntry {
    pub role:       MessageRole,   // User | Assistant | System | Tool
    pub content:    String,
    pub token_count: usize,
    pub metadata:   HashMap<String, Value>,
}

Adapters: InMemoryGarrison, SqliteGarrison (feature sqlite) — see Garrison Memory.

Arsenal (Tool Domain)

// crates/paladin-core/src/platform/container/arsenal/
pub struct ToolDefinition {
    pub name:        String,
    pub description: String,
    pub parameters:  serde_json::Value,  // JSON Schema
}

pub struct ToolCall {
    pub id:        String,
    pub tool_name: String,
    pub arguments: serde_json::Value,
}

pub struct ToolResult {
    pub tool_call_id: String,
    pub content:      String,
    pub is_error:     bool,
}

Citadel (State Persistence)

// crates/paladin-core/src/platform/container/citadel.rs
pub struct CitadelEntry {
    pub paladin_name: String,
    pub state:        PaladinState,
    pub saved_at:     DateTime<Utc>,
}

Herald (Output Formatting)

// crates/paladin-core/src/platform/container/herald.rs
pub trait Herald: Send + Sync {
    fn format(&self, result: &PaladinResult, paladin: &Paladin) -> Result<String, HeraldError>;
}

Implementations: JsonHerald, MarkdownHerald, TableHerald — see Herald Output.

Base Primitives (crates/paladin-core/src/base/)

Error Types

Each domain has a dedicated error enum using thiserror:

See Design Patterns for the error handling convention.

Bounded Contexts

See Also

• Architecture Overview
• Hexagonal Design
• Crate Map
• Design Patterns — PaladinBuilder and error conventions

Design Patterns

Reference for the key Rust design patterns used consistently across the Paladin codebase.

1. Node<T> Entity Pattern

All persistent domain entities are wrapped in Node<T>, which adds identity and timestamps without polluting the domain data struct:

pub struct Node<T> {
    pub id:         Uuid,
    pub created_at: DateTime<Utc>,
    pub updated_at: DateTime<Utc>,
    pub node:       T,
}

// Usage — domain type alias
pub type Paladin = Node<PaladinData>;

// Access payload fields through .node
let name = &paladin.node.name;
let model = &paladin.node.model;

2. Builder Pattern (PaladinBuilder)

Complex objects are constructed using a fluent builder that validates configuration before returning the entity.

// crates/paladin-ai-core (application services layer within the crate)
use paladin_ai_core::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin_ports::output::llm_port::LlmPort;
use std::sync::Arc;

let llm: Arc<dyn LlmPort> = Arc::new(my_adapter);

let paladin = PaladinBuilder::new(llm.clone())
    .system_prompt("You are a concise assistant.")
    .name("MyAgent")
    .model("gpt-4")
    .temperature(0.7)       // 0.0–2.0
    .max_loops(3)            // default: 1
    .stop_word("DONE")       // optional stop trigger
    .build()
    .await?;

build() validates all required fields and returns Err(PaladinError) if any invariant is violated, so the caller never receives an invalid Paladin.

Builder Conventions

• Required fields are set in new() (e.g., llm_port)
• Optional fields use fn field(mut self, …) -> Self (consuming builder)
• build() or build_async() calls the internal validate() method
• Always return Result<T, Error> from build() — never panic

3. Port Trait Pattern

All external integrations are expressed as async_trait port traits:

use async_trait::async_trait;

#[async_trait]
pub trait LlmPort: Send + Sync {
    async fn generate(
        &self,
        messages: &[Message],
        config: &LlmConfig,
    ) -> Result<LlmResponse, LlmError>;
}

Rules for port traits:

• Must be Send + Sync — adapters are shared across async tasks via Arc
• Return Result<T, SpecificError> — never anyhow::Error in port signatures
• Use #[async_trait] for all async fn in traits (until RPITIT stabilises)
• Define in crates/paladin-ports/src/output/ or src/input/

4. Error Handling with thiserror

Each domain module has its own error enum:

#[derive(Debug, thiserror::Error)]
pub enum PaladinError {
    #[error("Configuration error: {0}")]
    ConfigurationError(String),

    #[error("Execution error: {0}")]
    ExecutionError(String),

    #[error("LLM error: {0}")]
    LlmError(#[from] LlmError),

    #[error("Timeout after {0} seconds")]
    Timeout(u64),

    #[error("Stop word detected: {0}")]
    StopWordDetected(String),
}

#[derive(Debug, thiserror::Error)]
pub enum BattalionError {
    #[error("Paladin error: {0}")]
    PaladinError(#[from] PaladinError),

    #[error("Formation error: {0}")]
    FormationError(String),

    #[error("Invalid graph: {0}")]
    InvalidGraph(String),
}

Conventions:

• Layer-specific error types (core, ports, infrastructure)
• Use #[from] for automatic From impl when converting inner errors
• Never use .unwrap() or .expect() in production paths
• Use ? to propagate errors with context

5. Dependency Injection via Arc<dyn Trait>

Services receive dependencies at construction time via Arc<dyn Port>:

pub struct PaladinExecutionService {
    llm:             Arc<dyn LlmPort>,
    garrison:        Option<Arc<dyn GarrisonPort>>,
    circuit_breaker: Arc<CircuitBreaker>,
}

impl PaladinExecutionService {
    pub fn new(
        llm: Arc<dyn LlmPort>,
        circuit_breaker: Arc<CircuitBreaker>,
        garrison: Option<Arc<dyn GarrisonPort>>,
        herald: Option<Arc<dyn Herald>>,
    ) -> Self { /* … */ }
}

This makes swapping implementations (e.g., MockLlmAdapter in tests, OpenAIAdapter in production) a construction-time concern only.

6. Feature-Gated Modules

Optional infrastructure dependencies are gated behind Cargo feature flags to keep compilation lean:

// crates/paladin-memory/src/lib.rs

/// SQLite-backed garrison (requires feature `sqlite`)
#[cfg(feature = "sqlite")]
pub mod sqlite_garrison;

/// Qdrant-backed sanctum (requires feature `qdrant`)
#[cfg(feature = "qdrant")]
pub mod qdrant_sanctum_adapter;

And in Cargo.toml:

[features]
sqlite = ["sqlx/sqlite"]
qdrant = ["qdrant-client"]

7. Service Composition

Application services compose port dependencies to implement use cases:

pub struct FormationService {
    paladins: Vec<Paladin>,
    llm:      Arc<dyn LlmPort>,
    garrison: Option<Arc<dyn GarrisonPort>>,
}

impl FormationService {
    /// Execute the formation: each Paladin's output feeds the next
    pub async fn execute(&self, input: &str) -> Result<FormationResult, BattalionError> {
        let mut current_input = input.to_string();
        let mut results = Vec::new();

        for paladin in &self.paladins {
            let result = self.execute_paladin(paladin, &current_input).await?;
            current_input = result.output.clone();
            results.push(result);
        }

        Ok(FormationResult { results })
    }
}

8. Circuit Breaker

PaladinExecutionService accepts a CircuitBreaker to prevent cascading failures when an LLM provider is unavailable:

use paladin_ai_core::infrastructure::resilience::circuit_breaker::CircuitBreaker;
use std::time::Duration;

let circuit_breaker = Arc::new(CircuitBreaker::new(
    3,                        // open after 3 consecutive failures
    2,                        // close after 2 consecutive successes
    Duration::from_secs(30),  // half-open probe interval
));

9. Testing Conventions

Unit tests — in the same file

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_paladin_data_validates_empty_prompt() {
        let data = PaladinData { system_prompt: "".into(), /* … */ };
        assert!(data.validate().is_err());
    }
}

Integration tests — in tests/

// tests/formation_integration_test.rs
use paladin_ai_core::…;
use paladin_llm::mock::MockLlmAdapter;

#[tokio::test]
async fn test_formation_passes_output_to_next() {
    let mock = Arc::new(MockLlmAdapter::new().with_response("step result".into()));
    // …
}

Doc tests — in rustdoc comments

/// Validates the Paladin configuration.
///
/// ```rust
/// # use paladin_ai_core::platform::container::paladin::PaladinData;
/// let data = PaladinData { system_prompt: "Hello".into(), … };
/// assert!(data.validate().is_ok());
/// ```
pub fn validate(&self) -> Result<(), PaladinError> { /* … */ }

See Also

• Architecture Overview
• Hexagonal Design
• Domain Model
• Contributing — Development Setup

Crate Map

This page documents all nine workspace crates, their roles, feature flags, and dependency relationships.

Workspace Overview

paladin-ai  (root umbrella, v0.5.0)
├── paladin-ai-core          # Core domain
├── paladin-ports            # Port trait contracts
├── paladin-battalion        # Orchestration services
├── paladin-llm              # LLM adapters
├── paladin-memory           # Memory adapters
├── paladin-storage          # SQL adapters
├── paladin-notifications    # Notification adapters
├── paladin-content          # Content adapters
└── paladin-web              # HTTP server

Dependency Graph

graph TD
    root["paladin-ai (root)"]
    core["paladin-ai-core"]
    ports["paladin-ports"]
    batt["paladin-battalion"]
    llm["paladin-llm"]
    mem["paladin-memory"]
    stor["paladin-storage"]
    notif["paladin-notifications"]
    cont["paladin-content"]
    web["paladin-web"]

    root --> core
    root --> ports
    root --> batt
    root --> llm
    root --> mem
    root --> stor
    root --> notif
    root --> cont
    root --> web

    ports --> core
    batt --> core
    batt --> ports
    llm --> core
    llm --> ports
    mem --> core
    mem --> ports
    stor --> core
    stor --> ports
    notif --> core
    notif --> ports
    cont --> core
    cont --> ports
    web --> core
    web --> ports

Crate Details

paladin-ai-core

Directory: crates/paladin-core/ Layer: Core domain External deps: serde, uuid, chrono, tokio (runtime only), thiserror

Pure domain types — zero infrastructure dependencies.

Key modules:

src/
├── base/
│   ├── node.rs          # Node<T> entity wrapper
│   ├── collection.rs    # Paginated collections
│   ├── field.rs         # Dynamic field definitions
│   └── message.rs       # Inter-agent messages
└── platform/container/
    ├── paladin.rs             # Paladin aggregate
    ├── paladin_config.rs
    ├── paladin_error.rs
    ├── garrison.rs            # Garrison domain types
    ├── garrison_error.rs
    ├── arsenal/               # Arsenal + ToolDefinition
    ├── citadel.rs
    ├── herald.rs
    ├── sanctum.rs
    └── battalion/             # Battalion domain types

paladin-ports

Directory: crates/paladin-ports/ Layer: Application boundary External deps: paladin-ai-core, async-trait, tokio

Port trait contracts. No infrastructure SDKs.

Key modules:

src/output/
├── llm_port.rs              # LlmPort
├── garrison_port.rs         # GarrisonPort
├── sanctum_port.rs          # SanctumPort
├── arsenal_port.rs          # ArsenalPort
├── citadel_port.rs          # CitadelPort
├── file_storage_port.rs     # FileStoragePort
├── notification_port.rs     # NotificationPort
├── queue_port.rs            # QueuePort
├── embedding_port.rs        # EmbeddingPort
└── …

paladin-battalion

Directory: crates/paladin-battalion/ Layer: Application services External deps: paladin-ai-core, paladin-ports, tokio, serde

All eight orchestration patterns + Commander router.

Key modules:

src/
├── formation_service.rs          # Sequential N→N+1
├── phalanx_service.rs            # Concurrent parallel
├── campaign_service.rs           # DAG / graph
├── chain_of_command_service.rs   # Hierarchical delegation
├── conclave_execution_service.rs # Expert synthesis
├── council_service.rs            # Multi-agent discussion
├── grove_service.rs              # Semantic routing
├── maneuver/                     # Flow DSL
└── commander.rs                  # Strategy auto-router

paladin-llm

Directory: crates/paladin-llm/ Layer: Infrastructure (LLM adapters) External deps: paladin-ai-core, paladin-ports, reqwest, serde_json, tokio

Feature flags:

Key modules: src/openai/, src/anthropic/, src/deepseek/, src/mock.rs

paladin-memory

Directory: crates/paladin-memory/ Layer: Infrastructure (memory adapters) External deps: paladin-ai-core, paladin-ports, optionally sqlx, qdrant-client, tiktoken-rs

Feature flags:

Key modules:

src/
├── garrison/
│   ├── in_memory.rs     # InMemoryGarrison (always)
│   └── sqlite.rs        # SqliteGarrison (feature: sqlite)
├── sanctum/
│   ├── in_memory.rs     # InMemorySanctum (always)
│   └── qdrant.rs        # QdrantSanctumAdapter (feature: qdrant)
└── services/
    ├── memory_extraction_service.rs
    └── rag_retrieval_service.rs

paladin-storage

Directory: crates/paladin-storage/ Layer: Infrastructure (SQL repositories) External deps: paladin-ai-core, paladin-ports, optionally sqlx, mysql

Feature flags: sqlite, mysql

paladin-notifications

Directory: crates/paladin-notifications/ Layer: Infrastructure (notification adapters) Feature flags: email (lettre + handlebars), push (stub), system

paladin-content

Directory: crates/paladin-content/ Layer: Infrastructure (content processing)

Provides HTTP/file content fetcher, RSS/news ingestion, document parsing, and LLM-powered content analysis pipelines.

paladin-web

Directory: crates/paladin-web/ Layer: Infrastructure (HTTP server) External deps: actix-web, axum, tokio, serde

User management REST API, RBAC middleware, content delivery endpoints.

paladin-ai (root umbrella)

Directory: / (workspace root) Feature flags:

Adding a New Crate

1. Create crates/my-new-crate/ with Cargo.toml + src/lib.rs
2. Add to [workspace.members] in the root Cargo.toml
3. Set the layer: depend only on crates at the same or inner layers
4. Add to the root paladin-ai umbrella via an optional dependency
5. Document in this crate map

See Also

• Crate Map & Feature Flags (API Reference) — consumer/dependency view with copy-paste Cargo.toml profiles.
• Architecture Overview
• Hexagonal Design
• Domain Model

Deployment Topologies — Choosing How to Run Your Agents

A deployment topology is how you run agents — the process and concurrency model — independent of how you package and ship them (Docker, Kubernetes, CI/CD, which live in the Deployment section). The two are complementary: you pick a topology here, then package it there.

If you want to build a number of different agents on top of Paladin, the first decision is which of these five topologies fits. Paladin is designed to be embedded as a library — paladin-ai (library name paladin) is your composition root, not a framework that owns your process — so every topology below is something you assemble in your own binary.

The five topologies at a glance

Two of these are documented in depth elsewhere. The embedded-library and Battalion pages here are short topology overviews — they link to the full Paladin Agents and Orchestration Patterns guides for the complete API.

Choosing a topology

flowchart TD
    start([I want to run a number of agents]) --> q1{Do the agents collaborate on one task?}
    q1 -->|Yes| battalion[Battalion orchestration]
    q1 -->|No| q2{Does an external caller need to invoke them?}
    q2 -->|No| embedded[Embedded library]
    q2 -->|Yes| q3{Need scale-out, retries, or backpressure?}
    q3 -->|Yes| queue[Queue / worker]
    q3 -->|No| q4{Need hard process / deploy isolation per agent?}
    q4 -->|No| http[HTTP service host]
    q4 -->|Yes| sidecar[Sidecar]

Recommendation

Start with the embedded library topology (and Battalion when agents collaborate). When you need an external caller, wrap it in an HTTP service host. Reach for the queue / worker topology only when load or fault-isolation demands it, and the sidecar topology only when you need process isolation that in-process hosting cannot give — it carries the most operational overhead.

These topologies also compose: a worker process is an embedded host; a sidecar is an HTTP host called from another process.

Embedded Library (Single Process)

The simplest topology: depend on paladin-ai (library name paladin) and build your agents directly in your own binary. Paladin is designed for this — the root crate is a composition root, not a framework that owns your process — so "embed it as a library and build each agent's behaviour in your app" is the grain of the design, not a workaround.

The code blocks below are compiled examples pulled from the paladin-doc-examples crate via mdBook {{#include}}, so they are guaranteed to match the current API.

When to choose it

• Choose it when you control invocation in-code, all agents share one process, and you want the least moving parts. It is the right starting point for almost every project.
• Look elsewhere when an external client needs to call your agents (HTTP service host), you need scale-out or backpressure (queue / worker), or the agents collaborate on a single task (Battalion orchestration).

One agent

Build an agent with the fluent PaladinBuilder, then run it through a PaladinExecutionService. The mock LLM keeps the example offline; swap in OpenAIAdapter::from_env()? (or another adapter) for real use.

use std::sync::Arc;
use std::time::Duration;

use paladin::MockLlmAdapter;
use paladin::application::services::paladin::paladin_execution_service::PaladinExecutionService;
use paladin::infrastructure::resilience::circuit_breaker::CircuitBreaker;
use paladin::prelude::*; // PaladinBuilder, LlmPort, Paladin, ...

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // An offline mock LLM so this runs without an API key.
    // For real use: `Arc::new(OpenAIAdapter::from_env()?)`.
    let llm: Arc<dyn LlmPort> =
        Arc::new(MockLlmAdapter::new().with_response("Hello from Paladin!"));

    // Build an agent with the fluent builder.
    let agent = PaladinBuilder::new(llm.clone())
        .name("Greeter")
        .system_prompt("You are a friendly assistant.")
        .build()
        .await?;

    // Execute it and print the result.
    let breaker = Arc::new(CircuitBreaker::new(5, 2, Duration::from_secs(30)));
    let service = PaladinExecutionService::new(llm, breaker, None, None);
    let result = service
        .execute(&agent, "Say hello in one sentence.")
        .await?;

    println!("{}", result.output);
    Ok(())
}

See the Paladin Agents guide for the full builder API — system prompt, model, temperature, loops, stop words, vision, memory (Garrison), and tools (Arsenal).

Multiple distinct agents in one process

Because Paladins are Send + Sync and everything runs on tokio, you can keep many different agents resident in one process and route to them. A small agent registry — a map from a name to an agent plus its execution service — is all you need:

#![allow(unused)]
fn main() {
use std::collections::HashMap;
use std::sync::Arc;
use std::time::Duration;

use paladin::MockLlmAdapter;
use paladin::application::services::paladin::paladin_execution_service::PaladinExecutionService;
use paladin::infrastructure::resilience::circuit_breaker::CircuitBreaker;
use paladin::prelude::*; // PaladinBuilder, LlmPort, Paladin, PaladinResult

/// Several *distinct* agents, each with its own execution service, all resident
/// in one process. Build the registry once, then route each request to an agent
/// by name. This is the in-process foundation the HTTP-host topology serves.
pub struct AgentRegistry {
    agents: HashMap<String, (Paladin, Arc<PaladinExecutionService>)>,
}

impl AgentRegistry {
    /// Construct a registry of agents that differ by system prompt (and could
    /// differ by model, tools, or memory). One shared LLM port and circuit
    /// breaker are reused across them here; give each its own if they diverge.
    pub async fn new() -> Result<Self, Box<dyn std::error::Error>> {
        let llm: Arc<dyn LlmPort> = Arc::new(MockLlmAdapter::new());
        let breaker = Arc::new(CircuitBreaker::new(5, 2, Duration::from_secs(30)));

        let mut agents = HashMap::new();
        for (name, prompt) in [
            (
                "researcher",
                "You research topics thoroughly and cite sources.",
            ),
            ("summarizer", "You write concise, faithful summaries."),
        ] {
            let agent = PaladinBuilder::new(llm.clone())
                .name(name)
                .system_prompt(prompt)
                .build()
                .await?;
            let service = Arc::new(PaladinExecutionService::new(
                llm.clone(),
                breaker.clone(),
                None, // garrison (memory) — none in this minimal example
                None, // arsenal (tools)  — none in this minimal example
            ));
            agents.insert(name.to_string(), (agent, service));
        }
        Ok(Self { agents })
    }

    /// Route an input to a named agent and run it in-process.
    pub async fn run(
        &self,
        agent: &str,
        input: &str,
    ) -> Result<String, Box<dyn std::error::Error>> {
        let (paladin, service) = self
            .agents
            .get(agent)
            .ok_or_else(|| format!("no agent named '{agent}'"))?;
        let result: PaladinResult = service.execute(paladin, input).await?;
        Ok(result.output)
    }
}
}

Each entry can differ by system prompt, model, tools, or memory — that is what makes them "different agents." Calls are independent and run concurrently on the runtime, so several run(..) futures can be in flight at once.

This registry is also the foundation of the next topology: the HTTP service host wraps exactly this map behind an HTTP handler so an external client can invoke each agent. When the agents instead collaborate on one task, reach for Battalion orchestration.

← Back to Choosing a topology

Battalion Orchestration (Many Agents, One Runtime)

When several agents should collaborate on one task — rather than serve independent requests — use a Battalion. It runs many Paladins in a single tokio runtime with a coordination pattern built in, so you express the relationship between agents instead of hand-rolling the concurrency.

The example below is compiled code pulled from the paladin-doc-examples crate via mdBook {{#include}}, so it matches the current API.

When to choose it

• Choose it when the agents form a workflow: a pipeline, a fan-out/fan-in, a DAG, or a lead delegating to specialists. The Battalion owns ordering, concurrency limits, and error strategy for you.
• Look elsewhere when the agents are independent request handlers — a plain agent registry (optionally behind an HTTP host) fits better than an orchestration pattern.

This is still a single-process topology — it composes naturally with the others: a worker or an HTTP host can run a Battalion as the unit of work it executes.

Example: parallel agents (Phalanx)

A Phalanx fans the same input out to several Paladins concurrently and aggregates the results — the most direct "many agents, one runtime" pattern. Note the with_max_concurrency cap and the BattalionConfig:

#![allow(unused)]
fn main() {
use paladin_battalion::phalanx_service::PhalanxExecutionService;
use paladin_core::platform::container::battalion::phalanx::{AggregationStrategy, Phalanx};

/// Fan the same input out to several Paladins concurrently, then aggregate.
pub async fn run_phalanx() -> Result<(), Box<dyn std::error::Error>> {
    let paladin_port = mock_paladin_port();
    let security = create_paladin("SecurityAuditor");
    let perf = create_paladin("PerformanceAnalyst");
    let style = create_paladin("StyleChecker");

    let phalanx = Phalanx::new(vec![security, perf, style], BattalionConfig::default())?
        .with_aggregation(AggregationStrategy::CollectAll)
        .with_max_concurrency(4); // cap concurrent Paladins

    let service = PhalanxExecutionService::new(paladin_port);
    let result = service
        .execute(&phalanx, "Review this Rust module...")
        .await?;

    println!("Aggregated: {}", result.final_output);
    Ok(())
}
}

Picking a pattern

The full guides cover every pattern with a worked, compiled example, plus Conclave, Council, Grove, and the Maneuver flow DSL:

• Orchestration Patterns — the comprehensive reference.
• Battalion Orchestration Patterns — pattern-by-pattern walkthrough.

← Back to Choosing a topology

HTTP Service Host

Run one long-lived process that keeps several distinct agents resident behind an HTTP API, so external clients can invoke them and many requests run concurrently. This is the closest topology to "a running instance you hit."

Paladin ships this out of the box. The paladin-server binary (the web-server feature) serves a complete agent API — execution, streaming, async jobs, discovery, runtime registration, health/readiness, authentication, and an OpenAPI-documented /v1 surface. You configure it; you don't have to compose the endpoint yourself. (You can still embed the same routes in your own axum app — see Embedding.)

When to choose it

• Choose it when an external client needs request/response access to your agents, and a single in-process call won't do.
• Look elsewhere when you only call agents from your own code (embedded library), or you need scale-out / backpressure (queue / worker), or hard per-agent process isolation (sidecar).

The shipped server

The agent API is served under a /v1 version prefix; operational and docs endpoints are unversioned.

Every error is a structured envelope { "error": { "code", "message", "details" } }; every response carries an x-request-id. Each run is bounded by a timeout (server default, per-agent, or per-request), and on expiry the work is cancelled (504, or a terminal error SSE event).

Request flow

sequenceDiagram
    participant Client
    participant Server as paladin-server
    participant Service as PaladinExecutionService
    participant Agent as Paladin
    Client->>Server: POST /v1/agents/{id}/execute  (X-API-Key / Bearer)
    Server->>Server: authenticate + authorize (allowed_roles)
    Server->>Service: execute(agent, input)
    Service->>Agent: run (LLM + tools + memory)
    Agent-->>Service: PaladinResult
    Service-->>Server: output
    Server-->>Client: 200 JSON { output, … }

Configuring the host

Agents and host settings come from config.yml (see config.example.yml). A minimal shape:

server:
  host: "0.0.0.0"
  port: 8080

http:
  auth:
    enabled: true                  # fail-closed: the server refuses to start with no credentials
    api_keys:
      - { key: "${PALADIN_API_KEY_CI}", name: "ci", role: "admin" }
  docs:
    enabled: true                  # GET /openapi.json + Swagger UI at /docs

agents:
  - id: "researcher"
    model: "gpt-4"
    system_prompt: "You research topics thoroughly."
    allowed_roles: ["admin", "user"]   # empty ⇒ any authenticated caller

Authentication & authorization

Auth is enabled by default and fail-closed — with no credentials configured the server refuses to start (set http.auth.enabled: false for trusted/dev use). Callers present an API key (X-API-Key) or a JWT (Authorization: Bearer); a key/token maps to a role. Per-agent allowed_roles gate invocation, and runtime register/deregister require an admin role. /health, /ready, /openapi.json, and /docs are always reachable without a credential.

Running it

Binary:

PALADIN_CONFIG=./config.yml \
OPENAI_API_KEY=sk-... PALADIN_API_KEY_CI=sk-... \
cargo run --bin paladin-server --features web-server

Docker (Dockerfile.server):

make docker-build-server
docker run --rm -p 8080:8080 \
  -e OPENAI_API_KEY=sk-... -e PALADIN_API_KEY_CI=sk-... paladin-server:latest
# or: docker compose -f docker/docker-compose.server.yml up --build

Kubernetes (k8s/server/) — Deployment + Service + ConfigMap with liveness /health and readiness /ready probes:

kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/server/secret.yaml -f k8s/server/

Versioning

The agent API is versioned under /v1: only additive, backward-compatible changes are made within it; breaking changes ship under a new prefix (/v2). The /openapi.json contract is generated from the handlers and guarded against drift.

Embedding in your own app

You can also mount the agent registry and your own handler inside an existing axum app instead of running the binary. cargo check compiles this in full, so it can't drift from the API:

#![allow(unused)]
fn main() {
use std::sync::Arc;
use std::time::Duration;

use paladin::MockLlmAdapter;
use paladin::application::services::paladin::paladin_builder::PaladinBuilder;
use paladin::application::services::paladin::paladin_execution_service::PaladinExecutionService;
use paladin::infrastructure::resilience::circuit_breaker::CircuitBreaker;
use paladin::infrastructure::web::{
    AgentApiState, AgentRegistry, HttpLayersConfig, agent_router, with_http_layers,
};
use paladin_ports::output::llm_port::LlmPort;
use paladin_ports::output::paladin_executor_port::PaladinExecutorPort;
use paladin_ports::output::streaming_executor_port::StreamingExecutorPort;

/// Build a resident agent registry and serve Paladin's shipped agent API — `/v1/agents/…`
/// (buffered, streaming, async jobs, discovery, registration) plus `/health` and `/ready` —
/// inside your own `axum` process. This is the same router the `paladin-server` binary uses,
/// so the endpoints are provided for you rather than hand-written.
pub async fn serve_agents() -> Result<(), Box<dyn std::error::Error>> {
    let llm: Arc<dyn LlmPort> = Arc::new(MockLlmAdapter::new());
    let breaker = Arc::new(CircuitBreaker::new(5, 2, Duration::from_secs(30)));

    // One execution service backs both the buffered and streaming handles.
    let service = Arc::new(PaladinExecutionService::new(
        llm.clone(),
        breaker,
        None,
        None,
    ));
    let executor: Arc<dyn PaladinExecutorPort> = service.clone();
    let streamer: Arc<dyn StreamingExecutorPort> = service;

    let paladin = PaladinBuilder::new(llm)
        .name("researcher")
        .system_prompt("You research topics thoroughly.")
        .build()
        .await?;

    // Resident agents, keyed by id, shared across concurrent requests.
    let registry = AgentRegistry::new();
    registry.insert_with_streaming("researcher", Arc::new(paladin), executor, Some(streamer));

    // `agent_router` mounts the agent API under `/v1` plus the unversioned health probes;
    // `with_http_layers` adds the cross-cutting layers (request-id, CORS, body limit, timeout,
    // rate limit). Auth is open here (the library default); `paladin-server` enables it from
    // config. To also serve the OpenAPI spec + Swagger UI, merge `openapi::docs_router`.
    let state = AgentApiState::new(Arc::new(registry));
    let app = with_http_layers(agent_router(state), &HttpLayersConfig::default());

    let listener = tokio::net::TcpListener::bind("0.0.0.0:8080").await?;
    axum::serve(listener, app).await?;
    Ok(())
}
}

See also

• The bundled user/auth routes (paladin-web) a real service often also needs — Crate Map & Feature Flags.
• Running the same agent host in a separate process, called over the network — Sidecar.

← Back to Choosing a topology

Queue / Worker (Distributed)

Decouple requesting an agent run from executing it: producers enqueue jobs onto a Redis-backed queue, and a pool of workers dequeue and run them. This gives you horizontal scale, backpressure under load, retries, and fault isolation — a slow or failing worker doesn't block producers.

The example below is compiled code pulled from the paladin-doc-examples crate via mdBook {{#include}}. The Redis calls compile but are not executed by the check gate, so it stays in sync with the RedisQueueAdapter API without needing a live Redis.

Prerequisites: Run make dev (starts Redis) first, and enable the redis-queue feature on paladin-storage.

When to choose it

• Choose it when you need scale-out across workers/hosts, backpressure for bursty load, automatic retries, or isolation between job execution and your request path.
• Look elsewhere when load is low and in-process execution suffices (embedded library), or you only need synchronous request/response (HTTP service host).

Producer and worker

The producer enqueues a typed AgentJob; the worker dequeues it (as generic JSON), runs the agent through a PaladinExecutionService, and marks the item complete:

#![allow(unused)]
fn main() {
use std::sync::Arc;
use std::time::Duration;

use serde::{Deserialize, Serialize};

use paladin_core::base::entity::message::{Location, Message};
use paladin_core::platform::container::queue_item::QueueItem;
use paladin_ports::output::queue_port::QueuePort;
use paladin_storage::redis::{RedisQueueAdapter, RedisQueueConfig};

use paladin::application::services::paladin::paladin_execution_service::PaladinExecutionService;
use paladin::prelude::*; // Paladin, PaladinResult, ...

const QUEUE: &str = "agent-jobs";

/// The unit of work a producer enqueues and a worker executes.
#[derive(Clone, Serialize, Deserialize)]
struct AgentJob {
    agent: String,
    input: String,
}

/// **Producer** — connect to Redis and enqueue an agent job. Many producers can
/// enqueue concurrently; the queue absorbs bursts and applies backpressure.
pub async fn enqueue_job() -> Result<(), Box<dyn std::error::Error>> {
    let queue = RedisQueueAdapter::new(RedisQueueConfig::default(), None).await?;
    queue.create_queue(QUEUE.to_string(), None).await?;

    let job = AgentJob {
        agent: "summarizer".to_string(),
        input: "Summarise the Q3 earnings call.".to_string(),
    };
    let message = Message::new(
        Location::service("producer"),
        Location::service("worker"),
        job,
    );
    let item = QueueItem::new(QUEUE.to_string(), message, None);

    let id = queue.enqueue(QUEUE, item).await?;
    println!("enqueued job {id}");
    Ok(())
}

/// **Worker** — pull jobs off the queue and run them through a
/// `PaladinExecutionService`. Run many of these (in this process or across hosts)
/// to scale out. Each item is marked in-progress, then completed with its result.
pub async fn run_worker(
    queue: &RedisQueueAdapter,
    service: &PaladinExecutionService,
    agent: &Paladin,
) -> Result<(), Box<dyn std::error::Error>> {
    while let Some(item) = queue.dequeue(QUEUE).await? {
        let item_id = item.action.id;
        queue
            .start_processing(QUEUE, item_id, "worker-1".to_string())
            .await?;

        // The dequeued payload is generic JSON; read the agent input from it.
        let input = item.message.payload()["input"].as_str().unwrap_or_default();
        let result: PaladinResult = service.execute(agent, input).await?;

        queue
            .complete_processing(
                QUEUE,
                item_id,
                Some(serde_json::json!({ "output": result.output })),
            )
            .await?;
    }
    Ok(())
}
}

Run many workers — in this process via several tokio tasks, or as separate processes across hosts — all pulling from the same queue. start_processing / complete_processing / fail_processing track each item's lifecycle, and failures can retry up to the configured limit.

Configuring the queue

RedisQueueConfig is typically populated from config.yml:

queue:
  redis_host: "localhost"
  redis_port: 6379
  redis_db: 0
  connection_timeout: 30
  key_prefix: "paladin:queue"
  max_retries: 3

See also

• Standing up Redis and the adapter in detail — Redis Queue Adapter Setup.
• Each worker is itself an embedded agent host; a worker can also run a Battalion as its unit of work.

← Back to Choosing a topology

Sidecar (Separate Process)

Run an agent in its own process, deployed and scaled independently, and have your main application call it over the network. Use this when you need hard process, security, or deploy isolation per agent — at the cost of more operational moving parts.

The caller-side example below is compiled code pulled from the paladin-doc-examples crate via mdBook {{#include}}, so it matches the current reqwest API.

Paladin ships no IPC / gRPC / RPC / sidecar transport. There is no first-class "call an agent in another process" mechanism in the workspace. The sidecar pattern is composed: the agent runs behind an HTTP service host (the server side), and your app calls it with a plain HTTP client (the caller side, below). You own the wire contract — the URL shape and the request/response types.

When to choose it

• Choose it when an agent needs its own process boundary: independent deployment or scaling, a different security context, language/runtime isolation, or blast-radius containment.
• Look elsewhere when in-process hosting gives the same benefit more cheaply — a single HTTP service host keeps many agents resident in one process and avoids the network hop and extra deployment entirely. Prefer it unless isolation is a hard requirement.

The two sides

• Server side — the agent runs behind the HTTP service host exactly as documented there (POST /agents/{id}/execute), deployed as its own process or container.
• Caller side — your app makes an HTTP request to that endpoint:

#![allow(unused)]
fn main() {
use serde::{Deserialize, Serialize};

#[derive(Serialize)]
struct ExecuteRequest {
    input: String,
}

#[derive(Deserialize)]
struct ExecuteResponse {
    output: String,
}

/// Call an agent that runs in a *separate process* (a sidecar) over HTTP. The
/// wire contract is the one the [HTTP service host] defines —
/// `POST /agents/{id}/execute` — because Paladin provides no first-class sidecar
/// transport. The contract (URL shape, request/response types) is yours to own.
pub async fn call_sidecar_agent(
    base_url: &str,
    agent: &str,
    input: &str,
) -> Result<String, Box<dyn std::error::Error>> {
    let client = reqwest::Client::new();
    let resp: ExecuteResponse = client
        .post(format!("{base_url}/agents/{agent}/execute"))
        .json(&ExecuteRequest {
            input: input.to_string(),
        })
        .send()
        .await?
        .error_for_status()?
        .json()
        .await?;
    Ok(resp.output)
}
}

The request/response structs here must mirror the host's contract. Because that contract is consumer-owned, keep the two sides in sync yourself (e.g. a shared crate of DTOs).

What a first-class sidecar would need

Paladin does not provide this today; documented here as a limitation and a possible future direction. A first-class sidecar transport would add:

• A transport port trait (e.g. a RemoteAgentPort) abstracting "execute this agent in another process," with HTTP and/or gRPC adapters.
• A serialization contract for agent requests/results across the process boundary, so the wire types are defined and versioned by the framework rather than hand-rolled per consumer.

Until then, the composed HTTP host + client above is the supported approach.

See also

• The server side of this pattern — HTTP service host.
• Keeping agents in one process instead — Embedded library.

← Back to Choosing a topology

Docker Deployment Guide

Complete guide for deploying Paladin using Docker, including multi-architecture support, versioning strategies, and production best practices.

Table of Contents

• Overview
• Prerequisites
• Quick Start
• Docker Images
• Configuration
• Environment Variables
• Volumes and Persistence
• Networking
• Multi-Container Setup
• Multi-Architecture Support
• Image Versioning
• Health Checks
• Resource Limits
• Production Deployment
• Troubleshooting

Overview

Paladin provides official Docker images for easy deployment across environments. Images are:

• Multi-architecture: Support for AMD64 and ARM64
• Versioned: Semantic versioning with immutable tags
• Optimized: Multi-stage builds for minimal image size
• Secure: Non-root user, minimal attack surface

Prerequisites

# Docker 20.10+
docker --version

# Docker Compose 2.0+ (optional)
docker-compose --version

# For building from source
make --version
cargo --version

Quick Start

Development shortcut: For local development use make dev (starts all services via docker/docker-compose.dev.yml) or make services-up (starts Redis + MinIO only). See make help for all targets.

Run Prebuilt Image

# Pull and run latest Paladin image
docker run -d \
  --name paladin \
  -p 8080:8080 \
  -e OPENAI_API_KEY=your_api_key_here \
  -v paladin-data:/app/data \
  ghcr.io/your-org/paladin:latest

Build and Run Locally

# Clone repository
git clone https://github.com/your-org/paladin.git
cd paladin

# Build Docker image
docker build -t paladin:local .

# Run container
docker run -d \
  --name paladin \
  -p 8080:8080 \
  -v ./config.yml:/app/config.yml \
  -v paladin-data:/app/data \
  paladin:local

Docker Images

Official Images

Paladin images are available from GitHub Container Registry:

# Latest stable release
ghcr.io/your-org/paladin:latest

# Specific version
ghcr.io/your-org/paladin:v0.4.3

# Latest commit on main branch
ghcr.io/your-org/paladin:main

# Development builds (feature branches)
ghcr.io/your-org/paladin:dev-<branch-name>

Image Variants

Dockerfile

Paladin's multi-stage Dockerfile optimizes for size and security. There are two Dockerfiles in the repository:

• Dockerfile — Standard two-stage build (builder → runtime)
• Dockerfile.chef — Cargo-chef optimized build for faster CI (caches Rust dependencies as a separate layer)

# Standard Dockerfile — two stages

# Stage 1: Builder (rust:1.93-slim-bookworm)
FROM rust:1.93-slim-bookworm AS builder
WORKDIR /app

RUN apt-get update && apt-get install -y \
    pkg-config libssl-dev g++ \
    && rm -rf /var/lib/apt/lists/*

COPY Cargo.toml Cargo.lock ./
COPY src ./src
COPY crates ./crates
COPY benches ./benches
COPY migrations ./migrations

RUN cargo build --release --workspace --bin paladin
RUN strip target/release/paladin

# Stage 2: Runtime (debian:12-slim)
FROM debian:12-slim
WORKDIR /app

RUN apt-get update && apt-get install -y \
    ca-certificates libssl3 \
    && rm -rf /var/lib/apt/lists/*

COPY --from=builder /app/target/release/paladin /usr/local/bin/paladin
COPY --from=builder /app/migrations /app/migrations

# Non-root user (uid/gid 65532)
RUN groupadd -g 65532 paladin && \
    useradd -u 65532 -g paladin -s /bin/false -M paladin && \
    chown -R paladin:paladin /app

USER paladin:paladin
EXPOSE 8080
CMD ["/usr/local/bin/paladin"]

Tip: Use Dockerfile.chef in CI for faster builds — cargo-chef caches the dependency compilation layer separately from application code, so only changed crates are rebuilt.

Configuration

Configuration Files

Mount configuration files as volumes:

docker run -d \
  --name paladin \
  -v ./config.yml:/app/config.yml:ro \
  -v ./secrets.yml:/app/secrets.yml:ro \
  ghcr.io/your-org/paladin:latest

Example config.yml

# config.yml
server:
  host: "0.0.0.0"
  port: 8080
  log_level: "info"

paladin:
  default_model: "gpt-4"
  default_temperature: 0.7
  default_max_loops: 3
  timeout_seconds: 300

garrison:
  type: "sqlite"
  path: "/app/data/garrison.db"
  max_entries: 1000
  max_tokens: 8000

arsenal:
  mcp_servers:
    - name: "web_search"
      type: "stdio"
      command: "uvx"
      args: ["mcp-web-search"]

llm:
  openai:
    base_url: "https://api.openai.com/v1"
    # API key from environment variable
  deepseek:
    base_url: "https://api.deepseek.com/v1"
  anthropic:
    base_url: "https://api.anthropic.com/v1"

storage:
  type: "minio"
  endpoint: "minio:9000"
  bucket: "paladin"
  use_ssl: false

queue:
  type: "redis"
  url: "redis://redis:6379"

Environment Variables

Required Variables

# LLM Provider API Keys
OPENAI_API_KEY=sk-...
DEEPSEEK_API_KEY=your_key_here
ANTHROPIC_API_KEY=your_key_here

# Redis (queue)
REDIS_PASSWORD=changeme

# MinIO (object storage)
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin

Optional Variables

# Server configuration
SERVER_HOST=0.0.0.0
SERVER_PORT=8080
LOG_LEVEL=info

# Garrison configuration
GARRISON_TYPE=sqlite
GARRISON_PATH=/app/data/garrison.db
GARRISON_MAX_ENTRIES=1000

# Paladin defaults
DEFAULT_MODEL=gpt-4
DEFAULT_TEMPERATURE=0.7
DEFAULT_MAX_LOOPS=3

Passing Environment Variables

# From command line
docker run -d \
  -e OPENAI_API_KEY=sk-... \
  -e LOG_LEVEL=debug \
  ghcr.io/your-org/paladin:latest

# From .env file
docker run -d \
  --env-file .env \
  ghcr.io/your-org/paladin:latest

# In docker-compose.yml
services:
  paladin:
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - LOG_LEVEL=info

Volumes and Persistence

Data Volumes

Paladin requires persistent storage for:

• Garrison database: Conversation history
• Citadel checkpoints: State snapshots
• Logs: Application logs
• Configuration: Custom configs

# Named volumes
docker volume create paladin-data
docker volume create paladin-logs

docker run -d \
  --name paladin \
  -v paladin-data:/app/data \
  -v paladin-logs:/app/logs \
  ghcr.io/your-org/paladin:latest

# Bind mounts (host paths)
docker run -d \
  --name paladin \
  -v $(pwd)/data:/app/data \
  -v $(pwd)/logs:/app/logs \
  ghcr.io/your-org/paladin:latest

Volume Permissions

Paladin runs as non-root user (UID 1000). Ensure host directories have correct permissions:

# Set ownership for bind mounts
sudo chown -R 1000:1000 ./data ./logs

# Or use Docker volume (recommended)
docker volume create paladin-data

Backup and Restore

# Backup volume
docker run --rm \
  -v paladin-data:/data \
  -v $(pwd)/backups:/backup \
  ubuntu tar czf /backup/paladin-data-$(date +%Y%m%d).tar.gz -C /data .

# Restore volume
docker run --rm \
  -v paladin-data:/data \
  -v $(pwd)/backups:/backup \
  ubuntu tar xzf /backup/paladin-data-20240101.tar.gz -C /data

Networking

Port Mapping

# Map container port to host
docker run -d \
  -p 8080:8080 \           # HTTP API
  -p 8081:8081 \           # Metrics endpoint
  ghcr.io/your-org/paladin:latest

Custom Networks

# Create network
docker network create paladin-net

# Run container on custom network
docker run -d \
  --name paladin \
  --network paladin-net \
  ghcr.io/your-org/paladin:latest

# Connect other services
docker run -d \
  --name redis \
  --network paladin-net \
  redis:7-alpine

Multi-Container Setup

Docker Compose

Complete setup with Redis, MinIO, and Paladin:

# docker-compose.yml
version: '3.8'

services:
  redis:
    image: redis:7-alpine
    container_name: paladin-redis
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
    command: redis-server --appendonly yes
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

  minio:
    image: minio/minio:latest
    container_name: paladin-minio
    ports:
      - "9000:9000"  # API
      - "9001:9001"  # Console
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    volumes:
      - minio-data:/data
    command: server /data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 5s
      timeout: 3s
      retries: 5

  paladin:
    image: ghcr.io/your-org/paladin:latest
    container_name: paladin
    ports:
      - "8080:8080"
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - DEEPSEEK_API_KEY=${DEEPSEEK_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - LOG_LEVEL=info
      - GARRISON_TYPE=sqlite
      - GARRISON_PATH=/app/data/garrison.db
    volumes:
      - ./config.yml:/app/config.yml:ro
      - paladin-data:/app/data
      - paladin-logs:/app/logs
    depends_on:
      redis:
        condition: service_healthy
      minio:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 3s
      retries: 3

volumes:
  redis-data:
  minio-data:
  paladin-data:
  paladin-logs:

Running with Compose

# Start all services
docker-compose up -d

# View logs
docker-compose logs -f paladin

# Stop services
docker-compose down

# Stop and remove volumes
docker-compose down -v

Multi-Architecture Support

Paladin supports AMD64 and ARM64 architectures (Apple Silicon, ARM servers):

Building Multi-Arch Images

# Create buildx builder (one-time setup)
docker buildx create --name multiarch --use
docker buildx inspect --bootstrap

# Build for multiple platforms
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t ghcr.io/your-org/paladin:v0.4.3 \
  --push \
  .

Automated Multi-Arch Builds

GitHub Actions workflow (see .github/workflows/docker-publish.yml):

- name: Build and push Docker image
  uses: docker/build-push-action@v5
  with:
    context: .
    platforms: linux/amd64,linux/arm64
    push: true
    tags: |
      ghcr.io/${{ github.repository }}:latest
      ghcr.io/${{ github.repository }}:${{ github.ref_name }}
    cache-from: type=gha
    cache-to: type=gha,mode=max

Image Versioning

Tagging Strategy

Paladin follows semantic versioning with Docker tags:

# Release v0.4.3
ghcr.io/your-org/paladin:latest       # Always points to latest release
ghcr.io/your-org/paladin:v0.4.3       # Immutable version tag
ghcr.io/your-org/paladin:v0.4         # Minor version (updates with patches)
ghcr.io/your-org/paladin:v0           # Major version

# Development
ghcr.io/your-org/paladin:main         # Latest main branch
ghcr.io/your-org/paladin:dev-feature  # Feature branch

Version Pinning

Production: Always pin to specific versions:

# ✅ Good: Immutable version
docker run ghcr.io/your-org/paladin:v0.4.3

# ❌ Avoid: Latest can change
docker run ghcr.io/your-org/paladin:latest

Development: Use latest or branch tags:

docker run ghcr.io/your-org/paladin:main

Health Checks

Built-in Health Check

Paladin includes health check endpoint:

# HTTP health check
curl http://localhost:8080/health

# Response
{
  "status": "healthy",
  "version": "0.1.0",
  "uptime": 3600,
  "components": {
    "llm": "healthy",
    "garrison": "healthy",
    "arsenal": "healthy",
    "queue": "healthy"
  }
}

Docker Health Check

# Check container health
docker inspect --format='{{.State.Health.Status}}' paladin

# View health check logs
docker inspect --format='{{range .State.Health.Log}}{{.Output}}{{end}}' paladin

Resource Limits

CPU and Memory Limits

# Set resource limits
docker run -d \
  --name paladin \
  --cpus="2.0" \
  --memory="4g" \
  --memory-swap="4g" \
  ghcr.io/your-org/paladin:latest

Docker Compose Limits

services:
  paladin:
    image: ghcr.io/your-org/paladin:latest
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 4G
        reservations:
          cpus: '1.0'
          memory: 2G

Recommended Limits

Production Deployment

Production-Ready Configuration

# docker-compose.prod.yml
version: '3.8'

services:
  paladin:
    image: ghcr.io/your-org/paladin:v0.4.3  # Pinned version
    restart: unless-stopped
    environment:
      - LOG_LEVEL=warn  # Reduce log verbosity
      - RUST_BACKTRACE=0  # Disable backtraces
    volumes:
      - paladin-data:/app/data
      - paladin-logs:/app/logs
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 4G
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s

Security Hardening

# Run as read-only filesystem
docker run -d \
  --read-only \
  --tmpfs /tmp \
  -v paladin-data:/app/data \
  ghcr.io/your-org/paladin:latest

# Drop capabilities
docker run -d \
  --cap-drop=ALL \
  --cap-add=NET_BIND_SERVICE \
  --security-opt=no-new-privileges \
  ghcr.io/your-org/paladin:latest

Secrets Management

# Use Docker secrets (Swarm mode)
echo "$OPENAI_API_KEY" | docker secret create openai_key -

docker service create \
  --name paladin \
  --secret openai_key \
  -e OPENAI_API_KEY_FILE=/run/secrets/openai_key \
  ghcr.io/your-org/paladin:latest

# Use external secrets manager
docker run -d \
  --name paladin \
  -e AWS_REGION=us-east-1 \
  -e SECRET_NAME=paladin/openai \
  --env-file <(aws secretsmanager get-secret-value --secret-id paladin/openai --query SecretString --output text | jq -r 'to_entries|map("\(.key)=\(.value|tostring)")|.[]') \
  ghcr.io/your-org/paladin:latest

Troubleshooting

Container Won't Start

# Check logs
docker logs paladin

# Common issues:
# 1. Missing environment variables
docker logs paladin 2>&1 | grep "environment variable"

# 2. Port already in use
docker run -d -p 8081:8080 paladin  # Use different host port

# 3. Volume permission issues
docker run --user $(id -u):$(id -g) paladin

Health Check Failing

# Test health endpoint manually
docker exec paladin curl -f http://localhost:8080/health

# Check service dependencies
docker-compose ps  # Are Redis/MinIO healthy?

# Increase health check timeout
docker run -d \
  --health-cmd "curl -f http://localhost:8080/health" \
  --health-interval=30s \
  --health-timeout=10s \
  --health-retries=5 \
  --health-start-period=60s \
  paladin

High Memory Usage

# Check memory stats
docker stats paladin

# Set memory limits
docker update --memory="4g" --memory-swap="4g" paladin

# Check Garrison limits in config.yml
garrison:
  max_entries: 500  # Reduce if needed
  max_tokens: 4000

Connectivity Issues

# Test network connectivity
docker exec paladin ping redis
docker exec paladin curl -v http://minio:9000

# Check DNS resolution
docker exec paladin nslookup redis

# Verify network
docker network inspect paladin-net

Image Pull Failures

# Authenticate with GitHub Container Registry
echo $GITHUB_TOKEN | docker login ghcr.io -u USERNAME --password-stdin

# Pull with explicit platform
docker pull --platform linux/amd64 ghcr.io/your-org/paladin:latest

# Use mirror/proxy (if behind firewall)
docker pull ghcr.io/your-org/paladin:latest --registry-mirror=https://mirror.example.com

Next Steps

• Kubernetes Deployment - Deploy to Kubernetes
• CI/CD Guide - Automated deployments
• Production Best Practices - Production checklist
• Monitoring - Observability setup

Kubernetes Deployment Guide

Complete guide for deploying Paladin on Kubernetes with high availability, scalability, and production best practices.

Table of Contents

• Overview
• Prerequisites
• Quick Start
• Architecture
• Kubernetes Manifests
• ConfigMaps and Secrets
• Helm Chart
• Resource Management
• High Availability
• Horizontal Scaling
• Storage
• Networking
• Monitoring
• Security
• Troubleshooting

Overview

Paladin on Kubernetes provides:

• High Availability: Multi-replica deployments with health checks
• Auto-scaling: HPA based on CPU/memory/custom metrics
• Rolling Updates: Zero-downtime deployments
• Resource Management: CPU/memory limits and requests
• Service Discovery: Internal DNS for service communication

Prerequisites

# Kubernetes 1.25+
kubectl version

# Helm 3.0+ (optional but recommended)
helm version

# kubectl-ctx and kubectl-ns (optional, for context switching)
kubectl ctx
kubectl ns

Quick Start

Using Kubectl

# Create namespace
kubectl create namespace paladin

# Apply manifests
kubectl apply -f k8s/ -n paladin

# Check status
kubectl get pods -n paladin
kubectl get svc -n paladin

# View logs
kubectl logs -f deployment/paladin -n paladin

Using Helm

# Add Paladin Helm repository
helm repo add paladin https://charts.paladin.dev
helm repo update

# Install with default values
helm install paladin paladin/paladin -n paladin --create-namespace

# Install with custom values
helm install paladin paladin/paladin \
  -n paladin \
  --create-namespace \
  --values values.yaml

# Upgrade
helm upgrade paladin paladin/paladin -n paladin

# Uninstall
helm uninstall paladin -n paladin

Architecture

┌──────────────────────────────────────────────────────┐
│              Kubernetes Cluster                       │
│                                                       │
│  ┌────────────────────────────────────────────────┐ │
│  │           Namespace: paladin                    │ │
│  │                                                  │ │
│  │  ┌──────────────┐      ┌──────────────┐       │ │
│  │  │   Ingress    │      │   Service    │       │ │
│  │  │  (External)  │─────▶│ (ClusterIP)  │       │ │
│  │  └──────────────┘      └───────┬──────┘       │ │
│  │                                 │               │ │
│  │                        ┌────────▼────────┐     │ │
│  │                        │   Deployment    │     │ │
│  │                        │  (Paladin x3)   │     │ │
│  │                        └────┬───┬───┬────┘     │ │
│  │                             │   │   │          │ │
│  │                 ┌───────────┼───┼───┼───────┐ │ │
│  │                 │           │   │   │       │ │ │
│  │            ┌────▼───┐  ┌───▼───▼───▼────┐  │ │ │
│  │            │ Redis  │  │ MinIO/S3        │  │ │ │
│  │            │StatefulSet│ │ StatefulSet    │  │ │ │
│  │            └────────┘  └────────────────┘  │ │ │
│  │                                              │ │ │
│  │  ┌──────────────┐      ┌──────────────┐   │ │ │
│  │  │  ConfigMap   │      │   Secret     │   │ │ │
│  │  │  (config.yml)│      │  (API keys)  │   │ │ │
│  │  └──────────────┘      └──────────────┘   │ │ │
│  └─────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────┘

Kubernetes Manifests

Namespace

# k8s/00-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: paladin
  labels:
    app: paladin
    environment: production

Deployment

# k8s/10-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: paladin
  namespace: paladin
  labels:
    app: paladin
    component: server
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: paladin
      component: server
  template:
    metadata:
      labels:
        app: paladin
        component: server
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8081"
        prometheus.io/path: "/metrics"
    spec:
      serviceAccountName: paladin
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000

      initContainers:
      - name: wait-for-redis
        image: busybox:1.35
        command: ['sh', '-c', 'until nc -zv redis 6379; do echo waiting for redis; sleep 2; done;']

      containers:
      - name: paladin
        image: ghcr.io/your-org/paladin:v0.4.3
        imagePullPolicy: IfNotPresent

        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        - name: metrics
          containerPort: 8081
          protocol: TCP

        env:
        - name: SERVER_HOST
          value: "0.0.0.0"
        - name: SERVER_PORT
          value: "8080"
        - name: LOG_LEVEL
          value: "info"
        - name: RUST_LOG
          value: "info,paladin=debug"

        # Secrets from Secret resource
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: paladin-secrets
              key: openai-api-key
        - name: DEEPSEEK_API_KEY
          valueFrom:
            secretKeyRef:
              name: paladin-secrets
              key: deepseek-api-key
              optional: true
        - name: ANTHROPIC_API_KEY
          valueFrom:
            secretKeyRef:
              name: paladin-secrets
              key: anthropic-api-key
              optional: true

        # Mount configuration
        volumeMounts:
        - name: config
          mountPath: /app/config.yml
          subPath: config.yml
          readOnly: true
        - name: data
          mountPath: /app/data
        - name: tmp
          mountPath: /tmp

        # Resource limits
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
          limits:
            cpu: 2000m
            memory: 4Gi

        # Health checks
        livenessProbe:
          httpGet:
            path: /health
            port: http
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3

        readinessProbe:
          httpGet:
            path: /health/ready
            port: http
          initialDelaySeconds: 10
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 3

        # Graceful shutdown
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 10"]

      volumes:
      - name: config
        configMap:
          name: paladin-config
      - name: data
        persistentVolumeClaim:
          claimName: paladin-data
      - name: tmp
        emptyDir: {}

      # Affinity for spreading pods across nodes
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - paladin
              topologyKey: kubernetes.io/hostname

Service

# k8s/20-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: paladin
  namespace: paladin
  labels:
    app: paladin
spec:
  type: ClusterIP
  selector:
    app: paladin
    component: server
  ports:
  - name: http
    port: 80
    targetPort: http
    protocol: TCP
  - name: metrics
    port: 8081
    targetPort: metrics
    protocol: TCP
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800

Ingress

# k8s/21-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: paladin
  namespace: paladin
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/rate-limit: "100"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - paladin.example.com
    secretName: paladin-tls
  rules:
  - host: paladin.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: paladin
            port:
              number: 80

ConfigMaps and Secrets

ConfigMap

# k8s/30-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: paladin-config
  namespace: paladin
data:
  config.yml: |
    server:
      host: "0.0.0.0"
      port: 8080
      log_level: "info"

    paladin:
      default_model: "gpt-4"
      default_temperature: 0.7
      default_max_loops: 3
      timeout_seconds: 300

    garrison:
      type: "sqlite"
      path: "/app/data/garrison.db"
      max_entries: 1000
      max_tokens: 8000

    arsenal:
      mcp_servers:
        - name: "web_search"
          type: "stdio"
          command: "uvx"
          args: ["mcp-web-search"]

    llm:
      openai:
        base_url: "https://api.openai.com/v1"
      deepseek:
        base_url: "https://api.deepseek.com/v1"
      anthropic:
        base_url: "https://api.anthropic.com/v1"

    storage:
      type: "minio"
      endpoint: "minio.paladin.svc.cluster.local:9000"
      bucket: "paladin"
      use_ssl: false

    queue:
      type: "redis"
      url: "redis://redis.paladin.svc.cluster.local:6379"

Secret

# Create secret from literals
kubectl create secret generic paladin-secrets \
  --from-literal=openai-api-key="sk-..." \
  --from-literal=deepseek-api-key="..." \
  --from-literal=anthropic-api-key="..." \
  -n paladin

# Or from env file
kubectl create secret generic paladin-secrets \
  --from-env-file=secrets.env \
  -n paladin

# Or from YAML (base64 encoded)
# k8s/31-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: paladin-secrets
  namespace: paladin
type: Opaque
data:
  openai-api-key: <base64-encoded-key>
  deepseek-api-key: <base64-encoded-key>
  anthropic-api-key: <base64-encoded-key>

Helm Chart

Chart Structure

paladin-chart/
├── Chart.yaml
├── values.yaml
├── templates/
│   ├── _helpers.tpl
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── ingress.yaml
│   ├── configmap.yaml
│   ├── secret.yaml
│   ├── serviceaccount.yaml
│   ├── hpa.yaml
│   ├── pdb.yaml
│   └── NOTES.txt
└── crds/

values.yaml

# Default values for paladin
replicaCount: 3

image:
  repository: ghcr.io/your-org/paladin
  tag: "v0.4.3"
  pullPolicy: IfNotPresent

serviceAccount:
  create: true
  name: paladin

service:
  type: ClusterIP
  port: 80
  targetPort: 8080

ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  hosts:
    - host: paladin.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: paladin-tls
      hosts:
        - paladin.example.com

resources:
  requests:
    cpu: 500m
    memory: 1Gi
  limits:
    cpu: 2000m
    memory: 4Gi

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70
  targetMemoryUtilizationPercentage: 80

persistence:
  enabled: true
  storageClass: "fast-ssd"
  accessMode: ReadWriteOnce
  size: 10Gi

# Paladin configuration
config:
  paladin:
    defaultModel: "gpt-4"
    defaultTemperature: 0.7
    defaultMaxLoops: 3

  garrison:
    type: "sqlite"
    maxEntries: 1000
    maxTokens: 8000

  redis:
    url: "redis://redis:6379"

  minio:
    endpoint: "minio:9000"
    bucket: "paladin"

# Secrets (should be overridden)
secrets:
  openaiApiKey: ""
  deepseekApiKey: ""
  anthropicApiKey: ""

Install with Helm

# Create values-prod.yaml
cat > values-prod.yaml <<EOF
replicaCount: 5

ingress:
  hosts:
    - host: paladin.prod.example.com
      paths:
        - path: /
          pathType: Prefix

resources:
  requests:
    cpu: 1000m
    memory: 2Gi
  limits:
    cpu: 4000m
    memory: 8Gi

autoscaling:
  enabled: true
  minReplicas: 5
  maxReplicas: 20

secrets:
  openaiApiKey: ${OPENAI_API_KEY}
EOF

# Install
helm install paladin ./paladin-chart \
  -n paladin \
  --create-namespace \
  -f values-prod.yaml

Resource Management

Resource Requests and Limits

resources:
  requests:
    cpu: 500m       # Guaranteed CPU
    memory: 1Gi     # Guaranteed memory
  limits:
    cpu: 2000m      # Max CPU (burst)
    memory: 4Gi     # Max memory (OOM if exceeded)

QoS Classes

Recommendation: Use Burstable for production (requests < limits).

Resource Quotas

# k8s/40-resourcequota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: paladin-quota
  namespace: paladin
spec:
  hard:
    requests.cpu: "10"
    requests.memory: "20Gi"
    limits.cpu: "20"
    limits.memory: "40Gi"
    pods: "50"
    services: "10"
    persistentvolumeclaims: "10"

High Availability

Pod Disruption Budget

# k8s/41-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: paladin
  namespace: paladin
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: paladin

Multi-Zone Deployment

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - paladin
        topologyKey: topology.kubernetes.io/zone

Horizontal Scaling

Horizontal Pod Autoscaler

# k8s/42-hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: paladin
  namespace: paladin
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: paladin
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 30
      - type: Pods
        value: 2
        periodSeconds: 30
      selectPolicy: Max

Storage

PersistentVolumeClaim

# k8s/50-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: paladin-data
  namespace: paladin
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 10Gi

StatefulSet for Redis

# k8s/51-redis-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  namespace: paladin
spec:
  serviceName: redis
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:7-alpine
        ports:
        - containerPort: 6379
          name: redis
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: fast-ssd
      resources:
        requests:
          storage: 5Gi

Networking

Network Policies

# k8s/60-networkpolicy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: paladin
  namespace: paladin
spec:
  podSelector:
    matchLabels:
      app: paladin
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-nginx
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: redis
    ports:
    - protocol: TCP
      port: 6379
  - to:
    - podSelector:
        matchLabels:
          app: minio
    ports:
    - protocol: TCP
      port: 9000
  - to: []  # Allow all external (LLM APIs)

Monitoring

ServiceMonitor (Prometheus Operator)

# k8s/70-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: paladin
  namespace: paladin
  labels:
    app: paladin
spec:
  selector:
    matchLabels:
      app: paladin
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics

Security

ServiceAccount and RBAC

# k8s/80-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: paladin
  namespace: paladin

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: paladin
  namespace: paladin
rules:
- apiGroups: [""]
  resources: ["configmaps", "secrets"]
  verbs: ["get", "list"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: paladin
  namespace: paladin
subjects:
- kind: ServiceAccount
  name: paladin
  namespace: paladin
roleRef:
  kind: Role
  name: paladin
  apiGroup: rbac.authorization.k8s.io

Troubleshooting

Common Issues

# Pods not starting
kubectl describe pod <pod-name> -n paladin
kubectl logs <pod-name> -n paladin

# Service not accessible
kubectl get svc -n paladin
kubectl get endpoints -n paladin

# Config issues
kubectl get configmap paladin-config -o yaml -n paladin
kubectl get secret paladin-secrets -o yaml -n paladin

# Resource constraints
kubectl top pods -n paladin
kubectl describe node <node-name>

# Network issues
kubectl exec -it <pod-name> -n paladin -- curl http://redis:6379
kubectl get networkpolicy -n paladin

Next Steps

• CI/CD - Automated deployments
• Monitoring - Observability
• Production Best Practices - Production checklist

Production Best Practices

Comprehensive checklist and guidelines for deploying Paladin in production environments.

Table of Contents

• Pre-Deployment Checklist
• Security
• Performance
• Reliability
• Monitoring
• Disaster Recovery
• Cost Optimization
• Maintenance

Pre-Deployment Checklist

Infrastructure

• Compute resources sized appropriately (CPU, memory)
• High availability configured (multiple replicas/zones)
• Auto-scaling enabled with appropriate thresholds
• Load balancing configured with health checks
• Network policies restrict unnecessary traffic
• TLS/SSL certificates configured and valid
• DNS properly configured with failover

Configuration

• Environment variables properly set (no hardcoded secrets)
• Configuration files validated and tested
• API keys rotated and secured
• Log levels set appropriately (warn/error in prod)
• Resource limits configured (CPU, memory, connections)
• Timeouts set for all external calls
• Rate limits configured to prevent abuse

Data

• Database backups automated and tested
• Volume backups scheduled and verified
• Backup retention policy defined (7d/30d/365d)
• Disaster recovery plan documented and tested
• Data encryption at rest and in transit
• Access controls properly configured

Monitoring

• Health checks configured and responding
• Metrics collection enabled (Prometheus/Grafana)
• Log aggregation configured (ELK/Loki)
• Alerting rules defined for critical metrics
• On-call rotation established
• Incident response procedures documented
• SLO/SLA defined and monitored

Testing

• Load testing performed at expected scale
• Integration tests passing in staging
• Rollback procedure tested
• Canary deployment strategy defined
• Blue-green deployment capability verified
• Smoke tests automated post-deployment

Security

Authentication & Authorization

# Use strong authentication
auth:
  type: "oauth2"
  provider: "auth0"
  scopes: ["paladin:read", "paladin:write"]

# Implement role-based access control
rbac:
  roles:
    - admin: ["*"]
    - user: ["paladin:execute", "garrison:read"]
    - viewer: ["paladin:read"]

API Key Management

# Rotate API keys regularly
OPENAI_API_KEY=$(vault kv get -field=api_key secret/openai)
DEEPSEEK_API_KEY=$(vault kv get -field=api_key secret/deepseek)

# Use separate keys for different environments
staging_key="sk-proj-staging-..."
production_key="sk-proj-prod-..."

Network Security

# Kubernetes NetworkPolicy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: paladin-network-policy
spec:
  podSelector:
    matchLabels:
      app: paladin
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-nginx
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - namespaceSelector: {}
    ports:
    - protocol: TCP
      port: 443  # HTTPS only

Container Security

# Use specific versions (not latest)
FROM rust:1.70-slim-bullseye AS builder

# Run as non-root user
USER paladin:paladin

# Read-only filesystem
docker run --read-only --tmpfs /tmp paladin

# Drop capabilities
docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE paladin

# Use security scanning
docker scan paladin:latest
snyk container test paladin:latest

Secrets Management

# Use external secrets managers
# Kubernetes External Secrets
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: paladin-secrets
spec:
  secretStoreRef:
    name: aws-secrets-manager
  target:
    name: paladin-secrets
  data:
  - secretKey: openai-api-key
    remoteRef:
      key: paladin/prod/openai-api-key

# HashiCorp Vault
vault kv put secret/paladin/prod \
  openai_api_key=sk-... \
  deepseek_api_key=...

Performance

Resource Allocation

# Production resource configuration
resources:
  requests:
    cpu: 1000m      # 1 CPU guaranteed
    memory: 2Gi     # 2GB guaranteed
  limits:
    cpu: 4000m      # 4 CPU max
    memory: 8Gi     # 8GB max (OOM if exceeded)

# Horizontal Pod Autoscaler
autoscaling:
  enabled: true
  minReplicas: 5
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

Connection Pooling

// Configure connection pools
let redis_config = RedisConfig {
    url: "redis://redis:6379".into(),
    pool_size: 20,
    connection_timeout: Duration::from_secs(5),
    idle_timeout: Some(Duration::from_secs(60)),
};

let minio_config = MinioConfig {
    endpoint: "minio:9000".into(),
    max_connections: 100,
    connection_timeout: Duration::from_secs(10),
};

Caching Strategy

# Redis caching configuration
cache:
  enabled: true
  ttl: 3600  # 1 hour
  max_size: 10000
  eviction_policy: "lru"

# Application-level caching
garrison:
  cache_embeddings: true
  cache_ttl: 86400  # 24 hours

LLM Optimization

# Optimize LLM calls
llm:
  timeout: 30s
  max_retries: 3
  retry_delay: 1s
  connection_pooling: true

  # Use faster models for simple tasks
  model_routing:
    simple_tasks: "gpt-3.5-turbo"
    complex_tasks: "gpt-4"

  # Batch similar requests
  batching:
    enabled: true
    max_batch_size: 10
    max_wait_time: 100ms

Reliability

Health Checks

# Liveness probe (restart if fails)
livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3

# Readiness probe (remove from load balancer if fails)
readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 3
  successThreshold: 1

Graceful Shutdown

// Implement graceful shutdown
use tokio::signal;

async fn shutdown_signal() {
    let ctrl_c = async {
        signal::ctrl_c()
            .await
            .expect("failed to install Ctrl+C handler");
    };

    #[cfg(unix)]
    let terminate = async {
        signal::unix::signal(signal::unix::SignalKind::terminate())
            .expect("failed to install signal handler")
            .recv()
            .await;
    };

    tokio::select! {
        _ = ctrl_c => {},
        _ = terminate => {},
    }

    tracing::info!("Shutdown signal received, starting graceful shutdown");
}

// In main
let server = axum::Server::bind(&addr)
    .serve(app.into_make_service())
    .with_graceful_shutdown(shutdown_signal());

# Kubernetes graceful termination
spec:
  terminationGracePeriodSeconds: 30
  containers:
  - lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 15"]

Circuit Breakers

// Implement circuit breakers for external services
use circuit_breaker::{CircuitBreaker, Config};

let llm_breaker = CircuitBreaker::new(Config {
    failure_threshold: 5,
    success_threshold: 2,
    timeout: Duration::from_secs(60),
});

async fn call_llm_with_breaker(prompt: &str) -> Result<Response> {
    llm_breaker.call(async {
        llm_client.generate(prompt).await
    }).await
}

Retry Logic

// Implement exponential backoff
use backoff::{ExponentialBackoff, Error as BackoffError};
use backoff::future::retry;

async fn call_with_retry<F, T>(f: F) -> Result<T>
where
    F: Fn() -> Result<T>,
{
    let backoff = ExponentialBackoff {
        max_elapsed_time: Some(Duration::from_secs(60)),
        max_interval: Duration::from_secs(30),
        ..Default::default()
    };

    retry(backoff, || async {
        f().map_err(|e| {
            if e.is_retryable() {
                BackoffError::Transient(e)
            } else {
                BackoffError::Permanent(e)
            }
        })
    }).await
}

Monitoring

Key Metrics

# Application metrics
metrics:
  - paladin_requests_total          # Total requests
  - paladin_request_duration_seconds  # Request latency
  - paladin_errors_total            # Error count
  - paladin_active_paladins         # Active Paladins
  - garrison_entries_total          # Memory entries
  - arsenal_tool_calls_total        # Tool invocations

# System metrics
  - process_cpu_seconds_total       # CPU usage
  - process_resident_memory_bytes   # Memory usage
  - go_goroutines                   # Goroutines (if applicable)

# External dependencies
  - llm_api_calls_total             # LLM API calls
  - llm_api_duration_seconds        # LLM latency
  - redis_operations_total          # Redis ops
  - minio_operations_total          # MinIO ops

Alerting Rules

# Prometheus alerting rules
groups:
- name: paladin
  interval: 30s
  rules:
  - alert: HighErrorRate
    expr: rate(paladin_errors_total[5m]) > 0.05
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "High error rate detected"

  - alert: HighLatency
    expr: histogram_quantile(0.95, paladin_request_duration_seconds) > 2
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "High P95 latency (>2s)"

  - alert: PodCrashLooping
    expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
    for: 15m
    labels:
      severity: critical
    annotations:
      summary: "Pod is crash looping"

Logging Best Practices

// Structured logging with tracing
use tracing::{info, warn, error, instrument};

#[instrument(skip(paladin), fields(paladin_id = %paladin.id))]
async fn execute_paladin(paladin: &Paladin, input: &str) -> Result<PaladinResult> {
    info!("Starting paladin execution");

    match paladin.execute(input).await {
        Ok(result) => {
            info!(
                loops_used = result.loops_used,
                output_length = result.content.len(),
                "Paladin execution completed successfully"
            );
            Ok(result)
        }
        Err(e) => {
            error!(error = %e, "Paladin execution failed");
            Err(e)
        }
    }
}

# Log aggregation configuration
logging:
  level: warn  # info in staging, warn in production
  format: json
  outputs:
    - type: stdout
    - type: file
      path: /app/logs/paladin.log
      rotation:
        max_size: 100MB
        max_age: 7d
        max_backups: 10

Disaster Recovery

Backup Strategy

# Automated backups
# 1. Database backups
0 2 * * * /scripts/backup-garrison-db.sh

# 2. Volume snapshots
kubectl exec -n paladin deployment/backup -- \
  /scripts/snapshot-volumes.sh

# 3. Configuration backups
kubectl get all,cm,secrets -n paladin -o yaml > backup-$(date +%Y%m%d).yaml

Recovery Testing

# Quarterly disaster recovery drill
1. Simulate complete cluster failure
2. Restore from backups
3. Verify data integrity
4. Measure RTO (Recovery Time Objective)
5. Measure RPO (Recovery Point Objective)
6. Document lessons learned

Multi-Region Deployment

# Deploy to multiple regions
regions:
  - name: us-east-1
    primary: true
    replicas: 5
  - name: eu-west-1
    primary: false
    replicas: 3
  - name: ap-southeast-1
    primary: false
    replicas: 3

# Cross-region replication
replication:
  garrison: async  # Eventual consistency
  citadel: sync    # Strong consistency for checkpoints

Cost Optimization

Resource Right-Sizing

# Analyze actual usage
kubectl top pods -n paladin
kubectl describe hpa paladin -n paladin

# Adjust based on metrics
resources:
  requests:
    cpu: 800m    # Reduced from 1000m
    memory: 1.5Gi  # Reduced from 2Gi

Auto-Scaling Policies

# Aggressive scale-down for cost savings
autoscaling:
  scaleDown:
    stabilizationWindowSeconds: 600  # 10 minutes
    policies:
    - type: Percent
      value: 50
      periodSeconds: 300

Spot Instances

# Use spot instances for non-critical workloads
nodeSelector:
  kubernetes.io/lifecycle: spot

tolerations:
- key: spot
  operator: Equal
  value: "true"
  effect: NoSchedule

Maintenance

Update Strategy

# Rolling update configuration
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1        # One extra pod during update
    maxUnavailable: 0  # Zero downtime

Maintenance Windows

# Schedule maintenance during low-traffic periods
# Example: Sundays 2-4 AM UTC
0 2 * * 0 /scripts/maintenance.sh

Dependency Updates

# Regular dependency updates
dependabot.yml:
  version: 2
  updates:
    - package-ecosystem: "cargo"
      directory: "/"
      schedule:
        interval: "weekly"
      open-pull-requests-limit: 10

Checklist Summary

Use this checklist before each production deployment:

## Pre-Deployment
- [ ] All tests passing (unit, integration, e2e)
- [ ] Code review completed and approved
- [ ] Security scan passed (no high/critical vulnerabilities)
- [ ] Performance benchmarks within acceptable range
- [ ] Documentation updated
- [ ] Changelog updated

## Deployment
- [ ] Backup current state
- [ ] Deploy to staging first
- [ ] Run smoke tests in staging
- [ ] Deploy to production using rolling update
- [ ] Monitor metrics during rollout
- [ ] Verify health checks passing

## Post-Deployment
- [ ] Run smoke tests in production
- [ ] Check error rates and latency
- [ ] Verify auto-scaling working
- [ ] Confirm backups running
- [ ] Update runbook if needed
- [ ] Notify stakeholders of successful deployment

Next Steps

• Monitoring - Detailed monitoring setup
• Troubleshooting - Common issues and solutions
• Performance Tuning - Optimization guide

CI/CD Guide

Complete guide for setting up continuous integration and deployment pipelines for Paladin using GitHub Actions.

Table of Contents

• Overview
• GitHub Actions Workflows
• CI Pipeline
• Docker Build Pipeline
• Release Pipeline
• Integration Testing
• Security Scanning
• Deployment Automation
• Best Practices

Overview

Paladin uses GitHub Actions for CI/CD with the following pipelines:

• CI: Build, test, lint on every PR
• Docker: Build and publish multi-arch images
• Release: Automated releases with semantic versioning
• Integration: Integration tests with Docker services
• Security: Dependency scanning and vulnerability checks

GitHub Actions Workflows

Workflow Structure

.github/
├── workflows/
│   ├── ci.yml                    # Main CI pipeline (lint, test, audit)
│   ├── docs.yml                  # MDBook build + GitHub Pages deploy
│   ├── release.yml               # Release automation
│   ├── integration-tests.yml     # Integration testing
│   ├── feature-flags.yml         # Feature-flag matrix tests
│   └── pre-commit.yml            # Pre-commit checks
└── dependabot.yml                # Dependency updates

docs.yml builds MDBook, runs ./scripts/check-doc-examples.sh (validates all fenced Rust code blocks), and deploys to GitHub Pages on merge to main.

CI Pipeline

ci.yml

name: CI

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main, develop ]

env:
  CARGO_TERM_COLOR: always
  RUST_BACKTRACE: 1

jobs:
  check:
    name: Check
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Rust
        uses: dtolnay/rust-toolchain@stable

      - name: Cache cargo registry
        uses: actions/cache@v3
        with:
          path: ~/.cargo/registry
          key: ${{ runner.os }}-cargo-registry-${{ hashFiles('**/Cargo.lock') }}

      - name: Cache cargo index
        uses: actions/cache@v3
        with:
          path: ~/.cargo/git
          key: ${{ runner.os }}-cargo-index-${{ hashFiles('**/Cargo.lock') }}

      - name: Cache cargo build
        uses: actions/cache@v3
        with:
          path: target
          key: ${{ runner.os }}-cargo-build-target-${{ hashFiles('**/Cargo.lock') }}

      - name: Check formatting
        run: cargo fmt --all -- --check

      - name: Clippy
        run: cargo clippy --all-targets --all-features -- -D warnings

      - name: Check
        run: cargo check --all-features

  test:
    name: Test
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        rust: [stable, beta]
    steps:
      - uses: actions/checkout@v4

      - name: Install Rust ${{ matrix.rust }}
        uses: dtolnay/rust-toolchain@master
        with:
          toolchain: ${{ matrix.rust }}

      - name: Run tests
        run: cargo test --all-features

      - name: Run doc tests
        run: cargo test --doc --all-features

  coverage:
    name: Code Coverage
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Rust
        uses: dtolnay/rust-toolchain@stable
        with:
          components: llvm-tools-preview

      - name: Install cargo-llvm-cov
        uses: taiki-e/install-action@cargo-llvm-cov

      - name: Generate coverage
        run: cargo llvm-cov --all-features --workspace --lcov --output-path lcov.info

      - name: Upload to Codecov
        uses: codecov/codecov-action@v3
        with:
          files: lcov.info
          fail_ci_if_error: true

Docker Build Pipeline

docker-publish.yml

name: Docker

on:
  push:
    branches: [ main ]
    tags: [ 'v*.*.*' ]
  pull_request:
    branches: [ main ]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to Container Registry
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=ref,event=branch
            type=ref,event=pr
            type=semver,pattern={{version}}
            type=semver,pattern={{major}}.{{minor}}
            type=semver,pattern={{major}}
            type=sha
            type=raw,value=latest,enable={{is_default_branch}}

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

Release Pipeline

release.yml

name: Release

on:
  push:
    tags:
      - 'v*.*.*'

permissions:
  contents: write
  packages: write

jobs:
  build-release:
    name: Build Release
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        include:
          - os: ubuntu-latest
            target: x86_64-unknown-linux-gnu
          - os: ubuntu-latest
            target: aarch64-unknown-linux-gnu
          - os: macos-latest
            target: x86_64-apple-darwin
          - os: macos-latest
            target: aarch64-apple-darwin
          - os: windows-latest
            target: x86_64-pc-windows-msvc

    steps:
      - uses: actions/checkout@v4

      - name: Install Rust
        uses: dtolnay/rust-toolchain@stable
        with:
          targets: ${{ matrix.target }}

      - name: Install cross-compilation tools (Linux ARM64)
        if: matrix.target == 'aarch64-unknown-linux-gnu'
        run: |
          sudo apt-get update
          sudo apt-get install -y gcc-aarch64-linux-gnu

      - name: Build
        run: cargo build --release --target ${{ matrix.target }}

      - name: Package (Unix)
        if: matrix.os != 'windows-latest'
        run: |
          cd target/${{ matrix.target }}/release
          tar czf paladin-${{ github.ref_name }}-${{ matrix.target }}.tar.gz paladin
          mv paladin-${{ github.ref_name }}-${{ matrix.target }}.tar.gz ${{ github.workspace }}/

      - name: Package (Windows)
        if: matrix.os == 'windows-latest'
        run: |
          cd target/${{ matrix.target }}/release
          7z a paladin-${{ github.ref_name }}-${{ matrix.target }}.zip paladin.exe
          move paladin-${{ github.ref_name }}-${{ matrix.target }}.zip ${{ github.workspace }}/

      - name: Upload artifacts
        uses: actions/upload-artifact@v3
        with:
          name: release-${{ matrix.target }}
          path: |
            paladin-*.tar.gz
            paladin-*.zip

  create-release:
    name: Create Release
    needs: build-release
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Download artifacts
        uses: actions/download-artifact@v3

      - name: Generate changelog
        id: changelog
        run: |
          # Extract changelog for this version
          VERSION="${{ github.ref_name }}"
          awk "/^## \[$VERSION\]/,/^## \[/" CHANGELOG.md | head -n -1 > release_notes.md

      - name: Create GitHub Release
        uses: softprops/action-gh-release@v2
        with:
          files: |
            release-*/paladin-*.tar.gz
            release-*/paladin-*.zip
          body_path: release_notes.md
          draft: false
          prerelease: ${{ contains(github.ref_name, '-') }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Integration Testing

integration-tests.yml

name: Integration Tests

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main, develop ]
  schedule:
    - cron: '0 0 * * 0'  # Weekly on Sunday

jobs:
  integration-tests:
    name: Integration Tests
    runs-on: ubuntu-latest

    services:
      redis:
        image: redis:7-alpine
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 6379:6379

      minio:
        image: minio/minio:latest
        env:
          MINIO_ROOT_USER: minioadmin
          MINIO_ROOT_PASSWORD: minioadmin
        options: >-
          --health-cmd "curl -f http://localhost:9000/minio/health/live"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 9000:9000

    steps:
      - uses: actions/checkout@v4

      - name: Install Rust
        uses: dtolnay/rust-toolchain@stable

      - name: Wait for services
        run: |
          timeout 60 bash -c 'until curl -f http://localhost:9000/minio/health/live; do sleep 2; done'
          timeout 60 bash -c 'until redis-cli -h localhost ping; do sleep 2; done'

      - name: Run integration tests
        run: cargo test --features integration-tests --test '*_integration_test'
        env:
          REDIS_URL: redis://localhost:6379
          MINIO_ENDPOINT: localhost:9000
          MINIO_ACCESS_KEY: minioadmin
          MINIO_SECRET_KEY: minioadmin
          RUST_LOG: debug

      - name: Integration test coverage
        run: |
          cargo install cargo-llvm-cov
          cargo llvm-cov --features integration-tests --test '*_integration_test' --lcov --output-path integration-lcov.info

      - name: Upload coverage
        uses: codecov/codecov-action@v3
        with:
          files: integration-lcov.info
          flags: integration

Security Scanning

security.yml

name: Security

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]
  schedule:
    - cron: '0 0 * * 1'  # Weekly on Monday

jobs:
  audit:
    name: Cargo Audit
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install cargo-audit
        run: cargo install cargo-audit

      - name: Run cargo audit
        run: cargo audit

  deny:
    name: Cargo Deny
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install cargo-deny
        run: cargo install cargo-deny

      - name: Run cargo deny
        run: cargo deny check

  snyk:
    name: Snyk Security Scan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Snyk
        uses: snyk/actions/rust@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        with:
          args: --severity-threshold=high

Deployment Automation

Deploy to Kubernetes

name: Deploy

on:
  push:
    tags:
      - 'v*.*.*'
  workflow_dispatch:
    inputs:
      environment:
        description: 'Environment to deploy to'
        required: true
        type: choice
        options:
          - staging
          - production

jobs:
  deploy:
    name: Deploy to ${{ github.event.inputs.environment || 'production' }}
    runs-on: ubuntu-latest
    environment:
      name: ${{ github.event.inputs.environment || 'production' }}
      url: https://paladin.${{ github.event.inputs.environment || 'prod' }}.example.com

    steps:
      - uses: actions/checkout@v4

      - name: Configure kubectl
        uses: azure/k8s-set-context@v3
        with:
          method: kubeconfig
          kubeconfig: ${{ secrets.KUBE_CONFIG }}

      - name: Deploy with Helm
        run: |
          helm upgrade --install paladin ./paladin-chart \
            --namespace paladin \
            --create-namespace \
            --set image.tag=${{ github.ref_name }} \
            --set secrets.openaiApiKey=${{ secrets.OPENAI_API_KEY }} \
            --values values-${{ github.event.inputs.environment || 'production' }}.yaml \
            --wait

      - name: Verify deployment
        run: |
          kubectl rollout status deployment/paladin -n paladin
          kubectl get pods -n paladin

Best Practices

1. Branch Protection

Configure branch protection rules in GitHub:

# Required status checks
- CI / check
- CI / test (ubuntu-latest, stable)
- CI / test (macos-latest, stable)
- CI / coverage
- Integration Tests

# Required reviews: 1
# Dismiss stale reviews: true
# Require linear history: true

2. Secrets Management

Store secrets in GitHub repository settings:

# Required secrets
GITHUB_TOKEN          # Auto-provided
OPENAI_API_KEY        # For integration tests
SNYK_TOKEN            # For security scanning
KUBE_CONFIG           # For K8s deployment

3. Caching Strategy

# Cache Cargo dependencies
- uses: actions/cache@v3
  with:
    path: |
      ~/.cargo/registry
      ~/.cargo/git
      target
    key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
    restore-keys: |
      ${{ runner.os }}-cargo-

4. Concurrency Control

# Cancel in-progress runs for same PR
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

5. Conditional Workflows

# Skip CI for docs-only changes
on:
  push:
    paths-ignore:
      - '**.md'
      - 'docs/**'

6. Matrix Testing

strategy:
  matrix:
    os: [ubuntu-latest, macos-latest, windows-latest]
    rust: [stable, beta, nightly]
  fail-fast: false  # Continue other jobs on failure

7. Artifact Retention

- uses: actions/upload-artifact@v3
  with:
    name: test-results
    path: target/test-results/
    retention-days: 30

8. Notifications

- name: Slack Notification
  if: failure()
  uses: 8398a7/action-slack@v3
  with:
    status: ${{ job.status }}
    webhook_url: ${{ secrets.SLACK_WEBHOOK }}

Next Steps

• Production Best Practices - Production checklist
• Monitoring - Observability setup
• Docker Deployment - Docker deployment guide

Logging Configuration

Complete guide for configuring and managing logs in Paladin using the tracing ecosystem.

Table of Contents

• Overview
• Configuration
• Log Levels
• Structured Logging
• Log Aggregation
• Log Analysis
• Best Practices

Overview

Paladin uses the Rust tracing crate for structured, async-aware logging with:

• Structured fields: JSON-formatted logs
• Async tracing: Spans across async boundaries
• Multiple outputs: Console, file, and external systems
• Dynamic filtering: Runtime log level adjustment

Configuration

Environment Variables

# Set log level
export RUST_LOG=info,paladin=debug

# Detailed format
export RUST_LOG_FORMAT=json

# Enable specific modules
export RUST_LOG=paladin::core=debug,paladin::infrastructure=info

config.yml

logging:
  # Global log level
  level: "info"

  # Format: json, pretty, compact
  format: "json"

  # Outputs
  outputs:
    - type: "stdout"
      level: "info"

    - type: "file"
      path: "/app/logs/paladin.log"
      level: "debug"
      rotation:
        max_size: "100MB"
        max_age: "7d"
        max_backups: 10

    - type: "loki"
      url: "http://loki:3100"
      labels:
        app: "paladin"
        environment: "production"

  # Module-specific levels
  modules:
    paladin::core: "debug"
    paladin::infrastructure::adapters: "info"
    paladin::application: "debug"

  # Sampling (for high-volume logs)
  sampling:
    enabled: true
    rate: 0.1  # Log 10% of debug messages

Log Levels

Level Hierarchy