Harness

What turns a model into an agent.

Start with how harnesses power the two InferenceNet challenges and what a harness is, then see how runnable Agent Harness systems evolved in 2025–2026 and how Harbor integrates diverse harnesses into a unified evaluation system. Next, inspect DeepAgents in detail, compare our four evaluated configurations side by side, and connect execution to self-evolution.

Challenge harnesses

Designing harnesses for the two challenges

InferenceNet Challenge invites participants to build agents for empirical research and business analysis. A harness organizes context, tool execution, state and verification around a model. Our current multi-harness comparisons and self-evolution experiments were completed in Phase 1 on econometric replication; extending harness adaptation to the finance and HR workflow on Harbor and Claude Code is the next stage.

Subtrack 01 · Phase 1 completed

Econometric replication: four harnesses on Dev100

Evaluated in InferenceNet’s own runner with an isolated run_python sandbox tool. An agent inspects research data, implements sample and estimator specifications, and submits a final Python program for private scoring. In Phase 1 we evaluated 14 models across four harness configurations—DeepAgents, DeepSeek Harness, Claude Agent SDK and an OpenAI Agents SDK runtime—on the 100-task Dev100 set, and studied modular harness self-evolution on DeepAgents.

Subtrack 02 · Pilot on Harbor · Next stage

Business Analysis & Deliverables: Harbor + Claude Code

The finance and HR tasks contributed by Prof. Liwen Zhang’s team at SUFE currently use the Harbor (0.20.0) evaluation framework driving Claude Code (2.1.114) to reconcile multi-file evidence and produce Word (.docx) and Excel (.xlsx) deliverables for independent rubric grading. Multi-harness comparisons and harness evolution have not yet been run on this subtrack; adapting custom verification hooks and harness profiles to Harbor is our next stage.

Anatomy

Model, harness, environment

The harness is the layer between a model and the task environment. It decides what the model sees, what it can do, how errors come back and when the run stops.

  1. L1

    Model

    Reads the context it is given and proposes the next step or tool call.

    • Model
    • Next step
  2. L2

    Harness

    Assembles context, exposes tools, turns errors into feedback and enforces limits and output checks.

    • Context
    • Tools
    • Feedback
    • Run limits
  3. L3

    Environment

    The sandbox, data and files the agent acts on, and the evaluator that scores the final output.

    • Sandbox
    • Data and files
    • Evaluator

01 / Definition

What a harness is

A harness is the software that runs a language model as an agent: it decides what the model sees at each step, carries out the tool calls the model makes, returns the results, keeps the state of the run and decides when the run stops.

Agent = Model + Harness every leaderboard row with a harness scores this pair Agent Model weights behind an inference API reads one context, writes one reply: text and tool calls no memory between calls of its own context reply Harness everything else in the agent Runs on Loop Tools Context Delegation Stopping Permissions Extend builds the context for every model call, runs the reply’s tool calls and feeds results back, keeps memory, hands work to other agents, enforces limits and decides when to stop tool calls results Environment files, shell, sandbox, web pages, APIs, data what the tools act on; it changes as the agent works task and limits final answer Evaluator sets the task and its limits, then grades the final answer. It stays outside the agent so it can score any Model + Harness pair the same way.
What a harness covers. The model only maps one context to one reply; the harness does everything else that makes it an agent. The environment the tools act on and the evaluator that grades the answer are outside the agent, which is why a benchmark such as InferenceNet can swap either the model or the harness and score the pair.

Next: harness systems

Definition details: terms, responsibilities and boundaries

A model takes text in and returns text. Anthropic defines an agent harness as “the system that enables a model to act as an agent” and adds that evaluating an agent means “evaluating the harness and the model working together”. LangChain puts the relationship as “Agent = Model + Harness”. The same model can score differently in a different harness, so a result belongs to the pair. Evaluation papers often call the same layer a scaffold.

The 1,000-task Selected_1000 board follows this split. A Model row is a single call with no agent loop: one code-generation call, after which InferenceNet executes the returned program. A Model + Harness row runs the model inside a named harness: the harness passes the model’s code to the run_python tool that the evaluation supplies and returns the tool’s feedback to the model before it submits a final program. The tasks, data, sandbox, budget and scorer belong to the evaluation, not to the harness. The Dev100 v10 development board has no Model rows: it runs 14 models in four harness configurations (section 06).

Six terms and where each one ends

The words this site uses, with the boundary each one draws
TermWhat it isWhere it endsIn InferenceNet
ModelA language model behind an inference API: it reads one context and writes one reply, text and tool callsIt keeps nothing between calls and runs no tools. Request settings such as temperature or reasoning effort are set by whoever builds the request, often the harnessThe model named in each leaderboard row
HarnessThe software that runs the model as an agent: it builds the context, runs the loop, executes tool calls, keeps state, hands work to other agents, enforces limits and decides when to stopIt excludes the model’s weights, the environment its tools act on and the evaluator. An extension such as a skills plugin changes a host harness but is not a harness itselfDeepAgents, DeepSeek Harness or Claude Agent SDK; in the fourth Dev100 v10 configuration, a runtime InferenceNet assembles with the OpenAI Agents SDK
AgentA model running inside a harness on a task: Agent = Model + HarnessThe same model in another harness is another agent and can score differentlyA Model + Harness row on Selected_1000, or any Dev100 v10 row
SDKA programming interface. A model SDK sends requests to a model API; an agent framework SDK supplies building blocks; a harness SDK starts or embeds a ready harness so code can drive itAn SDK is a way in, not a layer of its own: one harness can be reached through a CLI, an SDK or a protocolThe DeepSeek Harness Python SDK starts the dsh process and the Claude Agent SDK runs its bundled Claude Code CLI; DeepAgents is imported as a library, and InferenceNet assembles its own runtime from the OpenAI Agents SDK
EnvironmentWhat the tools act on: files, a shell or container, data and outside servicesThe harness gives the model an interface to it; the environment itself is not part of the harnessThe task’s data and a fresh container for each run_python call
EvaluatorThe tasks, the limits, the code that runs trials and the grader; benchmarks often call it an evaluation harnessIt sits outside the agent so that any model and harness can be scored the same wayInferenceNet’s budget, final replay and private scorer

Eight things every harness decides

  • Runs on

    Where the harness runs and how it is driven: a library in the caller’s process, a separate CLI or server process, or a hosted service, and how it reaches models.

    • library
    • CLI or server
    • hosted service
    • model adapters
  • Execution loop

    The control flow of a run: call the model, turn its reply into actions, run them, add the results and repeat, with hooks or events between the phases.

    • step
    • hooks or events
    • parallel calls
  • Tools

    The actions the model can request and how each is carried out: the tool list and descriptions, the executor behind each tool and the result returned to the model.

    • built-in tools
    • MCP
    • executor
    • result format
  • Context and memory

    What the model sees on each call and what is kept: system prompt, instruction files, the running history and its compaction, large results moved aside, and anything carried across runs.

    • system prompt
    • AGENTS.md
    • compaction
    • session log
  • Delegation

    Whether and how work is handed to other agents: subagents with their own context, parallel workers, and how their results come back to the main run.

    • subagents
    • fresh context
    • depth limit
  • Stopping and recovery

    When a run ends and how it survives failures: a reply without tool calls, a limit or a rejected step; retries, summarizing on overflow, resuming an interrupted run.

    • stop condition
    • step limit
    • retries
    • resume
  • Permissions and limits

    Controls that hold whether or not the model cooperates: which tools may run and which need approval, sandbox and file permissions, and turn or spend limits inside the harness.

    • permissions
    • approval
    • sandbox
    • budgets
  • Extension points

    How behavior is added or replaced without changing the harness’s own code: middleware or hooks, plugins, skills, MCP servers, custom tools and configuration profiles.

    • middleware
    • hooks
    • plugins
    • skills
    • profiles

Where a harness sits

Layers are roles, not fixed boxes: the same package is sometimes placed in two layers, and many harnesses use no agent framework at all.
LayerWhat it providesExamples
RuntimeDurable execution, persistence, streaming and human-in-the-loop pausesLangGraph, Temporal
FrameworkBuilding blocks: model and tool interfaces, an agent loop, middlewareLangChain, OpenAI Agents SDK, Google ADK, CrewAI; since April 2026 the OpenAI Agents SDK also ships sandbox agents, a harness layer introduced as a beta
HarnessA ready agent: default prompts, built-in tools, context management, state and controlsDeep Agents, the harness inside Claude Code, the Codex harness, DeepSeek Harness, mini-SWE-agent
ProductSurfaces people use; one harness can sit behind severalClaude Code, the Codex app and CLI, Cursor, Deep Agents Code
PlatformHosts or routes a chosen harness for a team, an API or a chat gatewayQM, OpenAgentCore, OpenClaw’s Gateway
ExtensionPackages loaded into a harness that add instructions, workflows, tools or hooksAgent Skills, MCP servers, Claude Code plugins, Superpowers
OptimizerA process that proposes, tests and keeps harness versions using feedback: offline, as an outer loop between runs, or online, editing the harness across a stream of tasks or within one runModularRSI, Self-Harness, Meta-Harness (offline, section 05); other regimes on the Self-evolving map

02 / Harness systems

Agent Harness systems, 2025–2026

First the structure: where a runnable harness sits among the systems that host it, the extensions it loads and the roles around it. Then a curated timeline of design changes, a common comparison of seven systems and a catalogue.

A benchmark supplies tasks and scores; a protocol connects tools; a framework supplies building blocks. They belong to the surrounding ecosystem, but are different comparison units from a configured, runnable harness.

The counts measure different things: 28 timeline milestones trace runnable harnesses, platforms, extensions and self-evolution methods across 2025–2026, 7 representative systems are compared mechanism by mechanism, and the Harbor section shows how diverse harnesses plug into one unified evaluation pipeline.

What belongs to the harness — and what surrounds it

Harness systems and their surrounding ecosystem The runnable harness is the comparison unit. Above it, systems host, route or delegate to a harness runtime: OpenClaw’s gateway, Qwen-Live-Harness’s realtime loop, platforms such as QM and OpenAgentCore, and delivery workflows. Inside it, design branches with representative systems. Extensions load into a host harness; frameworks help build one; tools and sandboxes support it; harness optimization changes it offline between runs or online across tasks or within a run; external evaluators judge its outputs. Systems that host, route or delegate to a harness runtime arrow: hosts, routes or delegates to Gateway + selected runtime OpenClaw: built-in runtime, Codex app-server or a harness over ACP Realtime + delegation Qwen-Live-Harness: voice and video loop hands tasks to a harness Platforms QM · OpenAgentCore: run a chosen harness per turn or session Delivery workflows Symphony · GitHub Agentic Workflows issue → harness → review → deliver Runnable Agent Harness — the comparison unit Loop / stop Tools / actions Context / memory State / recovery Delegation Verify / permissions Interactive coding Claude Code · Codex · OpenCode · Gemini CLI · ZCode · jcode Minimal loops Pi · mini-SWE-agent Programmable runtimes DeepAgents · OpenHands SDK · OpenAI Agents SDK · DeepSeek Harness Durable and hosted Pi Durable (experimental) · Claude Managed Agents · OpenAI Agents API General-purpose agents Hermes Agent · OpenHarness · Hive · OpenClaw’s built-in runtime Internal checks guide execution; external graders measure it. Model reasoning / generation context in, reply out Extensions loaded into a host: Superpowers · Spec Kit Trellis · SoL-Pi Execution support Tools · MCP / ACP Sandbox · services Traces / telemetry Construction frameworks LangGraph · agent framework SDKs configure a concrete harness Harness optimization offline: search versions between runs online: edit across tasks or within a run External evaluation Terminal-Bench: tasks / tests Harbor: run agents and score outputs Arrows show roles, not product ancestry.
The runnable harness is the unit compared on this page. Above it, OpenClaw’s gateway chooses a runtime per model, Qwen-Live-Harness runs a realtime voice and video loop that hands tasks to a background harness, and platforms such as QM and OpenAgentCore run the harness they select. Extensions load into a host; optimizers change a harness offline between runs or online across tasks or within a run. Names are representative; primary repository links are collected in section 06. Survey: taxonomy (§2.3) and systems (§6) · Claude Code architecture · OpenClaw runtimes · Qwen-Live-Harness · Self-evolving: optimization regimes

How the systems changed, 2025–2026

Agent Harness systems: selected developments, 2025–2026 Twenty-eight milestones across interactive harnesses, programmable and plugin runtimes, managed and durable platforms, and skills extensions or automated harness self-evolution in 2025 and 2026. A dot marks the date; lines connect labels to dates and do not imply product ancestry. Agent Harness systems, 2025–2026 How the running system and its optimization evolved — verified releases, open-source repositories and papers 2025 2026 Apr Jul Oct Apr Jul Oct Interactive & CLI harnesses terminal, TUI, realtime Claude Code 2025-02 · research preview terminal edit, execute, verify Codex CLI 2025-04 · open-source launch sandboxed repository loop OpenCode 2025-05 · open-source (216k★) local server + 75+ providers Gemini CLI · Qwen Code 2025-06 · open-source CLIs terminal loop, MCP, subagents Hermes Agent 2025-07 · NousResearch (255k★) general-purpose MCP runtime jcode 2026-01 · Rust TUI (21k★) heavy multi-agent workflows mini-SWE-agent 2.0 2026-02 · v2.0.0 release native tool calls, minimal loop Grok Build 2026-07 · xAI open-source (27k★) fullscreen Rust TUI harness ZCode · Qwen-Live-Harness 2026-09 · Z.ai & Qwen releases coding TUI & realtime A/V loop Programmable & plugin runtimes libraries, SDKs, modular loops Strands · Ruflo 2025-05/06 · SDK & swarm (76k★) model-driven loop & federation DeepAgents 2025-07 · LangChain (30k★) middleware, virtual files, task OpenHands SDK 2025-11 · architecture paper engine, event log, agent server Claude Agent SDK 2025-11 · SDK + long-running progress artifacts across runs OpenHuman · Hive 2026-01/02 · open-source (42k★) high-density & production agents OpenHarness · Agents SDK 2026-04 · HKUDS & OpenAI v0.14 personal agent & sandbox beta DeepSeek Harness (DSH) 2026-08/09 · preview (253k★) every part is a Cordis plugin DeerFlow 2.1 · DeepAgents 0.7 2026-09/10 · runtime updates durable memory & on-demand skills Managed, durable & platforms hosted loops, sandboxes, workspaces LangGraph 1.0 · Sandboxing 2025-10 · runtime milestones durable state & OS-bounded bash Codex harness & app-server 2026-01 · engineering account JSONL thread protocol & sandbox Claude Managed · Browser 2026-04 · Anthropic & browser-use decoupled session & Chrome loop QM 2026-07 · yc-software (15.4k★) multiplayer workspace harness OpenAI Agents API 2026-09 · public beta hosted Codex loop & compaction Pi 1.0 · Pi Durable 2026-10 · release (114k★) checkpointed replay Extensions & self-evolution skills packages & automated search Superpowers 2025-10 → v7.0 (301k★) plan, TDD & review skills for 16 hosts ECC · Trellis 2026-01 · open-source (283k★) skills, instincts, memory, security Meta-Harness · AHE 2026-03/04 · arXiv papers trace-guided code & module search Self-Harness · RHO 2026-06 · arXiv papers non-regressive & retro harness edits Harness-R1 · WHALE 2026-08 · arXiv papers RL hook engineer & co-evolution ModularRSI · RRSI 2026-09 · arXiv papers contrastive 5-module evolution
Twenty-eight milestones across four lanes: interactive and CLI harnesses, programmable and plugin runtimes, managed or durable platforms, and skills extensions or self-evolution methods, based on verified releases, repositories and papers in 2025–2026.
Milestone dates and primary sources
28 developments across runnable harnesses, platforms, extensions and self-evolution
DateHarness or methodMilestoneMechanism
Claude Coderesearch previewterminal edit, execute and verify loop
Codex CLIopen-source launchrepository tool loop with local sandbox
OpenCodeopen-source repository (216k★)client–server TUI and multi-provider agent runtime
Strands Harness SDK · Rufloopen-source launches (8.9k★ / 76k★)model-driven SDK loop and multi-agent swarm orchestration
Gemini CLI · Qwen Codeopen-source launches (28.6k★)terminal tool loops, MCP integration and subagents
Hermes Agentopen-source launch (255k★)general-purpose agent runtime with MCP extensions
DeepAgentsintroduction (30.4k★)LangGraph middleware stack, virtual file system and subagent delegation
Superpowers · Kimi Code CLIopen-source launches (301k★ / 11.4k★)composable brainstorm, plan, TDD and review skills for 16 hosts; Kimi terminal harness
OpenHands Software Agent SDKarchitecture paper (v1; v2 on 22 Apr 2026)agent, conversation, workspace and agent server as separate parts
Claude Agent SDKlong-running harness designprogress artifacts and state handoff across sessions
jcode · Hiveopen-source launches (21k★ / 11.1k★)Rust multi-agent coding harness and production multi-agent runtime
ECC · Trellisopen-source launches (283k★ / 15k★)cross-harness skills, instincts, memory and security optimization
Codex harnessengineering accountprompt → model → tools → repeat, exposed over an app-server JSONL protocol
mini-SWE-agent · OpenHumanv2.0.0 release & launch (42k★)native tool calls in a minimal bash loop; high-density lightweight agent harness
Meta-Harness · AHEarXiv preprintsouter-loop harness code search over raw traces and file-level component attribution
OpenHarnessopen-source launch (16k★)open agent harness with built-in personal agent
Claude Managed Agentsarchitecture accountdecouple harness, session log and sandbox for recoverable runs
OpenAI Agents SDK · Browser Harnessv0.14.0 sandbox beta & launch (18.5k★)workspace manifests and snapshots; real logged-in Chrome harness
Self-Harness · RHOarXiv preprintsself-diagnosed non-regressive edits on DeepAgents and retrospective optimization on Codex and Claude Code
Grok Build · QMopen-source launches (27.4k★ / 15.4k★)xAI fullscreen TUI harness and multiplayer work harness
Harness-R1 · WHALEarXiv preprintsRL-trained 9B engineer for four lifecycle hooks; alternating weight and harness co-evolution
DeepSeek Harness (DSH)open-source launch & v0.1.5-rc.1 (253k★)every part—including the agent loop—is a hot-swappable Cordis plugin
OpenAI Agents APIpublic betahosted Codex harness with compaction, tool search and parallel subagents
ModularRSI · RRSIarXiv preprintscontrastive 5-module harness evolution and critic-pruned recursive improvement
ZCode · Qwen-Live-Harnessreleases (7.9k★)extensible coding harness and realtime audio/video loop delegating over ACP
DeerFlow 2.1release (84k★)durable subagent delegation, sandbox and memory workspaces
Pi 1.0 · Pi Durable · PiHarnessPi 1.0 & Pi Durable (114k★)each step checkpointed; an interrupted tool reruns only if marked replay-safe
DeepAgents · Superpowers v7.0skills updateson-demand skill-bound tool loading and unified implementer/reviewer subagent gates

Core mechanisms compared

How seven representative harnesses organize their execution loop, context, state, tools, delegation and extension surfaces.

Core runtime mechanisms across seven representative harnesses
SystemLoop and roleContextState and recoveryTools and environmentDelegationPermissions and extensions
Claude CodeDocs read 13 Oct 2026How Claude Code worksGather context, act, verify; repeats until done and can be interruptedCLAUDE.md or AGENTS.md, auto memory, on-demand skills; clears old tool output, then summarizesLocal JSONL sessions to resume or fork; file checkpoints for rewindFiles, search, shell, web; local, cloud VMs or Remote ControlSubagents in a fresh or forked context return a summaryAuto, Manual, Accept-edits and Plan modes; skills, MCP, hooks, plugins
CodexDocs read 13 Oct 2026Agent loop · Approvals and security · Codex SDKPrompt → model → tool calls, repeated; an app-server protocol embeds the same agentAGENTS.md instructions; skillsThreads persist and resumeShell and edits in an OS sandbox (Seatbelt; bwrap and seccomp); cloud environmentsSubagents and custom agentsSandbox modes × approval policies; network off by default; optional auto-review; MCP, skills, rules
OpenCodeDocs checked 13 Oct 2026Repository · Compaction · CLI · PermissionsThe agent runs in a local server; the TUI and other clients connect to itAutomatic compaction; optional pruning of old tool outputsContinue or fork a saved sessionLocal tools with LSPBuild and plan agents; subagent sessionsPer-tool allow/ask/deny; agent overrides; plugins
PiPi 1.0, 1 Oct 2026README · Pi 1.0 · Pi Durable · PiHarnessMinimal coding agent, also run in print or JSON mode, over RPC or from the SDKSkills and prompt templates; 1.0 adds deferred tool loading and mid-conversation system messagesPi Durable (experimental) checkpoints each step and resumes after a crash; a cut-off tool reruns only if replay-saferead, bash, edit, write; Cloudflare’s PiHarness (beta) runs Pi Durable in a Durable ObjectNone built in; extensions or packages add itNo built-in permission system: use a container or micro-VM; extensions and Pi packages
OpenHands SDKMain branch, 13 Oct 2026README · Architecture · Security · PaperAn Agent runs the reasoning–action loop and a Conversation owns state; the engine behind the OpenHands CLI and CloudA condenser compresses history; skills with triggersAppend-only event log (event sourcing); persistence and resume; stuck detectorTerminal, file editor, task tracker, browser, MCP; local, Docker or remote workspace through the Agent ServerSub-agents through a task tool setA security analyzer rates action risk; a confirmation policy (risky actions by default); custom tools, skills, hooks
DeepAgents 0.7.13Pinned releaseBreakdownLangChain’s create_agent graph on LangGraph; middleware wraps each model and tool callCaller’s prompt; summarizes at 85% of the window; large results moved to filesGraph state; summarize and retry on overflow; resume needs a checkpointerFile tools; execute on execution backends; caller’s tools and MCPtask: a subagent in a fresh context returns its final messageFile rules and human-in-the-loop interrupts; middleware and harness profiles
DeepSeek Harness 0.1.5-rc.1Pinned releaseRelease 0.1.5-rc.1A separate dsh process; the loop itself is a Cordis pluginPrompt sections each step; history rebuilt from an append-only JSONL log; compaction at 80%Up to 5 retries per step; resume closes interrupted turnsAbout 25 tools in the sdk profile; OS file sandboxContinuable or forked child agents, depth 3Approval policy and timeouts; Cordis plugins from bundles and patch files

02.4 / Unified evaluation

Integrating diverse harnesses into a unified evaluation system via Harbor

Harnesses differ widely in how they are packaged: some are Python libraries (DeepAgents, OpenAI Agents SDK), some are standalone CLI binaries or JSON-RPC servers (Claude Code, DeepSeek Harness, Codex CLI, OpenCode), and others are minimal reference loops (Terminus-2, mini-SWE-agent). The Harbor evaluation framework unifies them behind a single task-and-container contract so every Model + Harness pair faces the same environment, budgets and independent verifier.

Unified harness evaluation architecture on Harbor A four-stage pipeline showing how Harbor integrates diverse harnesses into one evaluation system: standardized task packages enter a unified agent adapter layer that mounts CLI harnesses, SDK/library harnesses or minimal reference loops against an isolated container workspace, followed by an out-of-loop verifier that scores statistical outputs or business deliverables. Unified Evaluation Pipeline (Harbor Architecture) any runnable harness · identical task container · independent verifier 1 · Task Package standardized input bundle Subtrack 1 · Econometrics research prompt + .dta/.csv Subtrack 2 · Business 55/17 source files + brief Shared caps & metadata: wall time · call/token budgets reference answers withheld 2 · Unified Harness Adapter mounts any harness behind one driver CLI & ACP / JSON-RPC Harnesses Claude Code · DeepSeek Harness · Codex · OpenCode Library & SDK Harnesses DeepAgents · OpenAI Agents SDK · OpenHands SDK Minimal Reference Scaffolds Terminus-2 · mini-SWE-agent (neutral baselines) 3 · Isolated Sandbox Docker / bwrap container Tool Execution run_python · bash · file edits Pre-Submission Checks diagnostics · formula & readback Trajectory & provenance log: every model call, tool output, version hash and final artifact 4 · Private Verifier outside the agent loop Subtrack 1 · Replay Scorer fresh replay → coef, SE, p-value Subtrack 2 · Rubric Judge .docx + .xlsx vs hidden rubric Audited metrics & failure tags: scored · failed · verifier_error Why unify on Harbor: Subtrack 2 runs Harbor 0.20.0 with Claude Code; mounting DeepAgents, DeepSeek Harness, OpenAI Agents SDK and evolved hooks as Harbor agent adapters gives both subtracks one container lifecycle, trace format and independent grading boundary.
How Harbor turns heterogeneous harnesses into one comparable evaluation system. The task package, container workspace and final verifier stay identical across runs; only the mounted (Model, Harness) pair inside stage 2 changes.
  • 01 · Standardized task bundle

    One task format across domains

    Packages the prompt, input datasets or business documents, environment image and resource limits while keeping reference values and grading rubrics outside the container.

  • 02 · Pluggable agent adapter

    Mount CLI, SDK or library harnesses

    Wraps standalone CLI agents (Claude Code, Codex, OpenCode), JSON-RPC processes (DeepSeek Harness) and Python graph libraries (DeepAgents) behind a uniform run interface and model-routing proxy.

  • 03 · Isolated execution & hooks

    Reproducible sandboxes and pre-submit checks

    Executes every tool call in an isolated container and records full JSONL trajectories, making it straightforward to inject statistical diagnostics or Word/Excel readback hooks before submission.

  • 04 · Out-of-loop verification

    Independent replay and rubric scoring

    Grades the submitted Python program or business deliverables in a clean verification pass after the agent terminates, separating execution errors and verifier errors from task scores.

03 / Breakdown

Anatomy of a harness: DeepAgents 0.7.13

To see how a harness works on the inside, we break down DeepAgents (LangChain’s library harness, pinned at 0.7.13 in InferenceNet Phase 1) across the eight core responsibilities: its three-layer LangGraph architecture, its middleware stack, one complete run from start to finish, and how InferenceNet wires and evolves it.

DeepAgents 0.7.13 Library harness

Middleware around a LangGraph loop

DeepAgents is a Python library built on LangChain. Its main function, create_deep_agent(), takes LangChain’s standard agent loop (create_agent) and adds middleware, code that runs around every model and tool call, plus file tools, helper agents called subagents and a place to keep files. The result is a LangGraph graph that runs inside the caller’s own Python program. Most of what it adds keeps the model’s input within its context window.

Maker
LangChain
License
MIT
Form
A Python library whose main function, create_deep_agent(), returns a compiled graph
Runs on
The caller’s Python process (3.11 or later), on LangChain’s create_agent and LangGraph. 0.7.13 accepts langchain 1.3.18 or later, below 2.0; this page reads it with langchain 1.4.0 and langgraph 1.2.11, the versions InferenceNet pins
Models
Any LangChain chat model that supports tool calling
Version
deepagents 0.7.13 (1aae374), 2 Sep 2026, the version InferenceNet pins; upstream reached 0.7.23 on 7 Oct 2026
Use
pip install deepagents==0.7.13, then create_deep_agent(model=…, tools=[…]) and invoke it with a list of messages

The template: eight slots

DeepAgents 0.7.13 deepagents==0.7.13 · langchain 1.4.0 · langgraph 1.2.11 Runs on Python library in the caller’s process · LangChain’s create_agent graph · executed by LangGraph in supersteps Execution loop who runs a step create_agent graph: model ⇄ tools before_agent and wrap_model_call hooks wrap_tool_call and after_model hooks One reply’s tool calls run concurrently Tools what it can call ls · read_file · write_file · edit_file delete · glob · grep execute, only on execution backends task: hand work to a subagent Caller’s tools, MCP included Context and memory what it sees, keeps Caller’s system prompt + profile text No built-in base prompt since 0.7.0 Summarizes at 85% of the input window Tool results over 20k tokens → files Graph state: messages and files Skills and AGENTS.md when configured Delegation other agents task tool: synchronous subagent General-purpose subagent by default Fresh context; final message returned Async subagents on remote servers Stopping and recovery end, retry, resume Ends on a reply with no tool calls recursion_limit 9,999 supersteps Context overflow: summarize, retry Unanswered tool calls patched on entry Resume only with a checkpointer Permissions and limits what it may do permissions: allow, deny, interrupt interrupt_on: human approval No default sandbox or run budget Per-call timeouts on execute and glob Extend it by AgentMiddleware with six hook kinds · caller middleware after the core stack · a same-named entry replaces a default · HarnessProfiles per provider or model
DeepAgents 0.7.13 with its defaults, in the template every breakdown uses. Blue marks context and memory, the slot that most of its own code serves: a virtual file system for notes and offloaded results, summarization that moves old history into a file, subagents that work in a fresh context, and skills and AGENTS.md memory when configured. InferenceNet configures it differently (section 06).

One run, module by module

One run, module by module DeepAgents 0.7.13 on LangChain create_agent and LangGraph 01 create_deep_agent() → create_agent() builds middleware, tools and the general-purpose subagent; compiles a LangGraph graph 02 START → before_agent hooks PatchToolCallsMiddleware answers tool calls left without a result 03 model node · wrap_model_call chain Skills → Filesystem → SubAgent → Summarization → … → prompt caching → Memory 04 chat model call system message and history in, one AI message out, with the request’s tools bound 05 after_model · HumanInTheLoopMiddleware only with interrupt_on: pause; a person approves, edits, rejects or responds 06 route on the reply no tool calls: after_agent hooks, then END; otherwise one task per tool call 07 tools node · wrap_tool_call chain file tools check permissions=; results over 20,000 tokens move to files 08 tool calls in one superstep run concurrently; task runs a subagent; results go back to step 03 Stops when 06 the reply has no tool calls 08 every tool in the reply had return_direct recursion_limit 9,999 supersteps is passed Recovers by 02 answering tool calls left without a result 03 summarizing and retrying on context overflow resuming from a checkpointer, if one was passed Permission checks 07 permissions= rules: allow, deny or interrupt 05 interrupt_on pauses for a person no sandbox unless the backend provides one
One run of a DeepAgents agent with the default stack, read from the pinned source. Blue boxes are where permissions are checked; the arrow on the left is the loop back to the model node; numbers in the right column refer to the steps.

How DeepAgents is assembled

Your application create_deep_agent(model, tools, subagents, backend, middleware, …) assembled once at build time DeepAgents harness · deepagents 0.7.13, the version InferenceNet pins Middleware stack (outer → inner) core SkillsMiddlewareskills on demand FilesystemMiddlewarefile tools · offload SubAgentMiddlewaretask tool SummarizationMiddlewarecompress history PatchToolCallsMiddlewarefix dangling calls AsyncSubAgentMiddlewareremote subagents your middleware=[…] ModularRSI attaches here tail profile extra_middlewareper model prompt caching ×1–3provider caches MemoryMiddlewareinject AGENTS.md HumanInTheLoopMiddlewaretool approval _ToolExclusionMiddlewarehide excluded tools adds Tools the model sees built in, from middleware ls · read_file · write_file edit_file · delete · glob · grep executeshell backends only taskdelegate to a subagent passed by the caller tools=[…] · MCP tools Subagents (isolated context) general-purposeon by default SubAgentdeclarative CompiledSubAgentany graph AsyncSubAgentremote deployment Each subagent has its own middleware stack; sync results return via task uses Backends: files and execution StateBackendin-thread state (default) StoreBackendcross-thread store FilesystemBackendlocal disk LocalShellBackend · BaseSandbox+ execute CompositeBackendroute by path prefix Profiles: tuning per model provider profilemodel init settings harness profilecan change: · system prompt text · tool descriptions · excluded_tools · extra_middleware · excluded_middleware · the general-purpose subagent switch create_agent(model, tools, middleware, system_prompt, …) LangChain create_agent · langchain 1.4.0 compiles this StateGraph The agent loop START before_agent hooks · once per run before_model hooks · every turn model node calls the LLM after_model hooks · reverse order route last AIMessage no calls after_agent hooks → END pending tool calls tools node one Send per call ToolMessages · next turn jump_to from a hook can override routing Inside the model node ModelRequest: model, tools, system_message, messages, tool_choice, response_format wrap_model_call chain, first one outermost bind_tools, then invoke([system, *messages]) → AIMessage, plus any structured output Inside the tools node each pending call runs as its own task wrap_tool_call chain around every call returns a ToolMessage, or a Command back to the model, or END after return_direct or a structured-output tool InferenceNet’s agent, as compiled FilesystemMiddlewaremodel + tool wrap SummarizationMiddlewaremodel wrap PatchToolCallsMiddlewarebefore_agent node AnthropicPromptCachingMiddlewaremodel wrap graph: START → before_agent → model ⇄ tools LangGraph runtime · langgraph 1.2.11 runs the compiled graph in Pregel supersteps One superstep Plan nodes triggered by writes or Send Execute tasks run in parallel; writes are buffered Update reducers merge the writes into state Checkpoint saved per step when a saver is attached next superstep, until no node is triggered recursion_limit caps supersteps: 9,999 by default in DeepAgents; InferenceNet invokes with 64. A model turn plus its tools takes two steps. DeltaChannel (beta) checkpoints store only new writes, with a full snapshot every 50 updates in DeepAgents; InferenceNet has no checkpointer, so no checkpoint is written. State channels of this agent messagesDeltaChannel filesDeltaChannel jump_toEphemeralValue · cleared each step structured_responseLastValue _summarization_eventLastValue _summarization_session_idLastValue Persistence and control in InferenceNet checkpointer: none · each task is one fresh run store: none · interrupts: unused invoke(): returns only the final state
The three layers assembled by create_deep_agent(). The left column is the middleware order; for wrap_model_call, earlier entries wrap later ones. New middleware from the caller lands between core and tail; middleware whose name matches an existing entry, core or tail, replaces it in place. Filesystem, Summarization, PatchToolCalls and Anthropic prompt caching are always added, and SubAgent too while any synchronous subagent exists, which by default is the general-purpose one. Skills, AsyncSubAgent, Memory and HumanInTheLoop depend on their arguments; profile extra_middleware and _ToolExclusionMiddleware on the model’s harness profile; Bedrock and Fireworks caching on langchain-aws or langchain-fireworks being installed. The LangChain and LangGraph layers describe langchain 1.4.0 and langgraph 1.2.11; the compiled middleware, nodes and channels come from building InferenceNet’s agent (dev-gyc-eval, 92e56c8) offline with those versions. The blue slot marks where caller middleware lands: InferenceNet’s own FilesystemMiddleware, limited to six file tools, replaces the core entry of the same name instead, so the slot stays empty until ModularRSI attaches there on the Self-evolving page.

Three layers: LangGraph, LangChain, DeepAgents

What each package contributes to a DeepAgents agent at the versions InferenceNet pins. The Self-evolving page builds on this stack and links back here.
LayerPackageWhat it contributes
Runtimelanggraph 1.2.11Executes the compiled graph in supersteps, keeps state in channels, saves a checkpoint after each superstep when a checkpointer is passed, pauses for interrupts and raises an error past the recursion limit
Frameworklangchain 1.4.0, create_agentBuilds the model node, the tools node and the hook nodes; defines the six middleware hook kinds; sends each tool call to the tools node as its own task; sets the recursion limit to 9,999
Harnessdeepagents 0.7.13, create_deep_agentChooses the default middleware stack, the file tools and storage backends, the general-purpose subagent, summarization and the harness profiles

The default middleware stack

What create_deep_agent() passes to LangChain’s create_agent in 0.7.13, in order. For wrap_model_call and wrap_tool_call the first entry is outermost.
OrderMiddlewareHooksWhat it does
1SkillsMiddleware, with skills=before_agent, wrap_model_callLists each skill’s name and description in the system prompt; the model reads a skill’s full file when it needs it
2FilesystemMiddleware, alwaystools, wrap_model_call, wrap_tool_callAdds ls, read_file, write_file, edit_file, delete, glob, grep and execute over the backend; hides what the backend cannot serve; moves tool results over 20,000 tokens into files; applies permissions= rules
3SubAgentMiddleware, while a subagent existstools, wrap_model_callAdds the task tool, which runs a subagent on a fresh message list and returns its final message
4SummarizationMiddleware, alwayswrap_model_callSummarizes older messages once the history reaches 85% of the model’s input window and keeps the most recent messages that fit in 10% of that window; when the window is unknown, it summarizes at 170,000 tokens and keeps the last six messages. Saves the evicted history to a file and retries after a context overflow; summarized messages stay in state
5PatchToolCallsMiddleware, alwaysbefore_agentAdds a result for any earlier tool call that never got one
6AsyncSubAgentMiddleware, with async subagentstools, wrap_model_callStarts, checks, updates, cancels and lists background tasks on remote Agent Protocol servers
–The caller’s middleware=anyInserted after the entries above; an entry with a default’s name replaces that default in place
7Profile extra_middlewarevariesModel-specific additions; the built-in Codex profile adds TodoListMiddleware and its write_todos tool
8Prompt caching, alwayswrap_model_callAdds Anthropic cache markers, and Bedrock or Fireworks ones when those packages are installed; other models ignore it
9MemoryMiddleware, with memory=before_agent, wrap_model_callLoads the listed AGENTS.md files once and adds them to the system prompt
10HumanInTheLoopMiddleware, with interrupt_on=after_modelPauses before matching tool calls run; a person approves, edits, rejects or responds
11Tool exclusion, if a profile excludes toolswrap_model_call, wrap_tool_callRemoves excluded tools from each request and rejects calls to them

In 0.7.23, SkillsMiddleware moves to just before prompt caching and UnsupportedContentMiddleware is added after HumanInTheLoopMiddleware.

One task, step by step
  1. 01

    Assemble the agent once

    create_deep_agent() resolves the model and its harness profile, builds the middleware stack, the general-purpose subagent and the tool list, and hands them to LangChain’s create_agent, which compiles a LangGraph graph.

    Default backend
    StateBackend, files in graph state
    Recursion limit
    9,999
  2. 02

    Enter the graph

    invoke() starts the graph and runs the before_agent hooks once. In the default stack only PatchToolCallsMiddleware has one: it answers any earlier tool call that never got a result. Skills and memory, when configured, load their files here.

    Default before_agent
    PatchToolCallsMiddleware
  3. 03

    Shape the request

    The model node builds a request from the model, the tools, the system message and the history, and passes it through the wrap_model_call chain, outermost first. File tools the backend cannot serve are dropped, history past the threshold is summarized and cache markers are added.

    Summary trigger
    85% of the input window, else 170,000 tokens
  4. 04

    Call the model

    The innermost handler binds the tools and calls the chat model with the system message followed by the history; one AI message comes back. If the provider rejects the request as too long, the history is summarized and the call is made again.

    Model input
    system message, then the history
  5. 05

    Route on the reply

    after_model hooks run; the default stack has none, while human-in-the-loop adds one that can pause for approval. A reply without tool calls leaves the loop; otherwise each tool call goes to the tools node as its own task.

    No tool calls
    leave the loop
  6. 06

    Run the tool calls

    Each call passes the wrap_tool_call chain and runs as a task in the same superstep, so the calls of one reply run concurrently. Oversized results move into backend files; task runs a subagent and returns its last message. The results go back to the model node.

    Result offload
    over 20,000 tokens
  7. 07

    End the run

    When the model replies without tool calls, any after_agent hooks run and the graph ends; invoke() returns the final state, whose last AI message is the answer. Passing the recursion limit raises an error. Within a run, state passes from one superstep to the next as usual; a checkpointer, if one is passed, saves a checkpoint after each superstep so the run can be resumed later.

    Answer
    the last AI message
    Checkpointer
    none unless passed

In InferenceNet. InferenceNet’s DeepAgents arm runs this version with run_python and six file tools for notes, no subagents, and its own budget outside DeepAgents. It is the harness of the Selected_1000 Model + Harness rows and one of the four Dev100 v10 configurations (section 06); the ModularRSI study on the Self-evolving page starts from it with its own settings.

How InferenceNet wires DeepAgents for evaluation

Controller · inferencenet.harness Per task attempt, in the controller’s process safe_task keeps the id, model row and data files build_agent() → create_deep_agent( tools = [run_python] subagents = [] backend = StateBackend() middleware = [FilesystemMiddleware] system_prompt = fixed instructions ) a harness profile turns off general-purpose invoke(messages), recursion_limit = 64 messages: the evaluator’s task request Pre-run checks deepagents == 0.7.13, else refuse compiled tools = six file tools + run_python Docker and provider identities preflighted Per-task budget, same as the DSH arm ≤6 model calls · ≤4 run_python calls 32,768 output tokens · 600 s · 90 s per run Unchanged deepagents 0.7.13 · langchain 1.4.0 langgraph 1.2.11 · StateBackend Summarization · PatchToolCalls · caching builds DeepAgents graph · same process PatchToolCallsMiddlewarebefore_agent model node · wrap_model_call, outer → inner FilesystemMiddlewaresix file tools SummarizationMiddlewareunchanged AnthropicPromptCachingMiddlewareno-op BudgetedChatOpenAIaccounted model tools node ls · read_file · write_filenotes only edit_file · glob · grepin graph state run_python(code)InferenceNet tool START → before_agent → model ⇄ tools → END recursion_limit 64 · no checkpointer Model provider upstream model API Responses API · HTTPS non-streaming request Model boundary · in process BudgetedChatOpenAI._generate() store off · max_retries 0 · no redirects tools ⊆ the seven approved names max_output_tokens = remaining allowance records the request, raw response and usage summaries count against the same budget a cut-off reply fails the attempt 401, 403, 429 or unknown outcome halts the study run_python · Python function tool counts the call · deadline = min(time left, 90 s) syntax check with ast.parse and compile over budget: returns JSON with an error and “Submit your final complete program now.” runs code feedback fresh Docker container no network · read-only · 6 GB · 2 CPUs feedback: stdout · stderr · exit_code · timed_out valid_prediction · seconds · never references or scores Final answer the last AI message has no tool calls → exactly one Python fence → a fresh replay within the wall budget → private scoring with local-paper-v1 Trust boundary The agent runs inside the controller’s Python process. The model reaches outside only through run_python, which runs code in an isolated container. The file tools edit notes kept in graph state. The task carries no reference values, every model call is checked and recorded, and scoring starts after the agent returns.
InferenceNet’s DeepAgents evaluation wiring. Blue marks what InferenceNet configures around DeepAgents 0.7.13: a budgeted model client, six virtual file tools for notes, and the isolated run_python container tool.

04 / Compare

Our four evaluated harnesses side by side

In Phase 1 (Dev100 v10), InferenceNet evaluated 14 models across four harness configurations on the same 100 econometric replication tasks: DeepAgents, DeepSeek Harness, Claude Agent SDK and an OpenAI Agents SDK runtime. Below are the four configurations and a side-by-side comparison of their architecture, loop control, context management and evaluation wiring.

The four configurations in Dev100 v10

The four evaluated harness configurations compared across the eight core dimensions
Dimension DeepAgents 0.7.13 DeepSeek Harness 0.1.5-rc.1 Claude Agent SDK 0.2.165 OpenAI Agents SDK 0.23.1
Category Library harness on LangChain & LangGraph Plugin-based product harness driven via Python SDK Harness SDK driving bundled Claude Code CLI (2.1.294) Agent framework Runner assembled by InferenceNet
Runs on Caller’s Python process; compiled StateGraph in Pregel supersteps Separate Node.js / binary dsh process over stdio JSON-RPC 2.0 Subprocess running the bundled Claude Code CLI binary Caller’s Python process via Runner.run loop
Execution loop model ⇄ tools graph with wrap_model_call and wrap_tool_call middleware dsh-agent-loop Cordis plugin with waterfall events around each phase Claude Code native agent loop (gather context → act → verify) Turn-based model-to-function-tool loop up to the configured turn cap
Tools in Dev100 run_python plus six virtual file tools (ls, read_file, write_file, edit_file, glob, grep) for notes Restricted 14-row profile: run_python is the only tool Built-in tools disabled; run_python exposed as the single MCP tool No built-in tools; run_python bound as the single function tool
Context & memory Virtual files in graph state; offloads tool results >20k tokens; summarizes at 85% window Append-only JSONL session log rebuilt each step; stock compaction disabled in restricted profile Session transcript managed by Claude Code CLI; local settings files disabled for isolation Linear message history passed across turns within the run
Delegation Supports task subagents by default; turned off in InferenceNet for controlled comparison Supports depth-3 child agents in stock profile; turned off in InferenceNet profile Supports subagents in stock Claude Code; disabled in restricted Dev100 configuration Handoffs supported by SDK; single-agent loop used in Dev100
Stopping & limits Stops on a reply without tool calls; capped at ≤6 model calls, ≤4 run_python calls, 1,800 s Stops at turn/end; same 6-model / 4-tool / 1,800 s caps enforced by bridge Stops on final response or max_turns=6; same 4-tool and 1,800 s caps Stops on final response or max_turns=6; same 4-tool and 1,800 s caps
Extension surface Python AgentMiddleware hooks and HarnessProfile (used for ModularRSI evolution) Cordis plugin bundles and declarative YAML patch files (cordis.patch.yml) MCP tool servers, lifecycle hooks and CLAUDE.md / skill packages (used on Harbor in Subtrack 2) Python function tools, guardrails and run hooks

All four configurations share the same per-task caps in Dev100 v10 (≤6 model calls, ≤4 run_python executions, 32,768 output tokens, 1,800 s wall time, 90 s per tool call) and the same private replay scorer (local-paper-v1). Full per-model results and provenance are on the Leaderboard page.

05 / Self-evolution

From running a harness to evolving it

An execution loop runs a fixed harness configuration on a task; an optimization loop sits outside execution, inspects trajectories and failure modes across tasks, proposes targeted edits to the harness (prompts, tool descriptions, observation formatting, middleware hooks or code), and retains only non-regressive improvements under a fixed evaluator.

The Self-evolving page presents the two-layer execution-and-optimization diagram, compares recent offline and online self-evolution methods (ModularRSI, Self-Harness, Meta-Harness, AHE, RHO, Harness-R1, RRSI and WHALE), and documents InferenceNet’s Phase 1 ModularRSI evolution study alongside our Phase 2 roadmap for Harbor and Claude Code.

06 / Sources

Primary sources and verified links

Official engineering accounts, benchmark frameworks, high-star open-source Agent Harness repositories and self-evolution papers referenced on this page, verified via GitHub API and arXiv in October 2026.

Definitions & evaluation frameworks

Open-source harnesses & extensions

Harness self-evolution papers