Leaderboard · October 2026

Dev100 Leaderboard

How far today’s models get on Dev100, the 100 econometric replication tasks planned for Subtrack 1: 14 models, each run with four harnesses under the same limits and the same scorer. Use it as the baseline your agent has to beat. These are development results, not an official Challenge ranking.

01Dev100 · v10

Dev100 / v10 development release

The 100 Dev100 tasks will be released as open source for Subtrack 1: Econometric replication; the task package, official rules and dates are still to be announced. Below are development results for 14 models × 4 harnesses—a baseline to beat, not an official Challenge ranking.

  • 100 tasks per combination
  • 56 combinations
  • v10 final report
  • Report generated

Scroll the table to see all metrics →

Scores (%) on 100 development tasks
#ModelHarnessFull replicationCoefficient and SE within 5%Coefficient within 5%Correct signCorrect significanceUnknown or missing (ranked metric)ScoredFailedUnknownSource mix
1GPT-5.6 SolClaude Agent SDK67.0%70.0%75.0%94.0%82.0%49622Inherited SDK 100
2GPT-5.6 SolDeepAgents65.0%68.0%76.0%96.0%83.0%29820New 4 · Offline-derived 96
3GPT-6.1 SolOpenAI Agents SDK61.0%67.0%72.0%88.0%77.0%109073New 100
4GPT-5.6 SolDeepSeek Harness60.0%66.0%73.0%93.0%78.0%49622Inherited SDK 100
4GPT-5.6 SolOpenAI Agents SDK60.0%65.0%73.0%93.0%79.0%59550New 100
4GPT-6.1 SolClaude Agent SDK60.0%65.0%72.0%90.0%78.0%69442New 100
7GPT-6 AstraDeepSeek Harness59.0%60.0%70.0%85.0%76.0%118992New 100
7GPT-6 AstraClaude Agent SDK59.0%67.0%74.0%88.0%79.0%99154New 100
9GPT-6 AstraDeepAgents58.0%63.0%71.0%90.0%76.0%79334New 100
9GPT-6 AstraOpenAI Agents SDK58.0%62.0%69.0%83.0%72.0%1486113New 100
11Claude Opus 5.5OpenAI Agents SDK57.0%65.0%74.0%90.0%78.0%89253New 100
12Claude Opus 5.5DeepSeek Harness55.0%65.0%72.0%86.0%73.0%1090100New 100
12Claude Fable 5.1DeepSeek Harness55.0%62.0%70.0%83.0%68.0%1486104New 100
12GPT-6.1 SolDeepAgents55.0%60.0%72.0%88.0%74.0%99172New 100
12GPT-6.1 SolDeepSeek Harness55.0%59.0%69.0%86.0%74.0%118965New 100
16Claude Opus 5.5Claude Agent SDK54.0%62.0%69.0%87.0%75.0%109064New 100
16Claude Fable 5.1DeepAgents54.0%65.0%74.0%90.0%72.0%89253New 100
16Claude Fable 5.1Claude Agent SDK54.0%63.0%74.0%89.0%73.0%99172New 100
19Kimi K3Claude Agent SDK51.0%57.0%66.0%78.0%65.0%1882135Inherited SDK 100
19Claude Opus 4.8Claude Agent SDK51.0%61.0%72.0%89.0%72.0%79352Inherited SDK 100
19Claude Fable 5.1OpenAI Agents SDK51.0%60.0%69.0%83.0%68.0%148695New 100
22Claude Opus 5.5DeepAgents50.0%58.0%66.0%81.0%66.0%1684106New 100
23Gemini 3.8 FlashOpenAI Agents SDK49.0%51.0%55.0%66.0%60.0%3268284New 100
24Kimi K3DeepSeek Harness48.0%54.0%62.0%77.0%67.0%2080146Inherited SDK 100
25Claude Opus 4.8DeepAgents45.0%59.0%71.0%92.0%71.0%69460New 14 · Offline-derived 86
26Gemini 3.1 ProDeepAgents42.0%4.0%7.0%10.0%7.0%59550New 15 · Historical 85
26Kimi K3DeepAgents42.0%10.0%14.0%16.0%13.0%69442New 22 · Historical 78
28Gemini 3.1 ProOpenAI Agents SDK41.0%55.0%66.0%78.0%63.0%2179210New 100
28Claude Opus 4.8DeepSeek Harness41.0%55.0%67.0%87.0%66.0%109091Inherited SDK 100
30Kimi K3OpenAI Agents SDK40.0%46.0%53.0%64.0%51.0%32681913New 100
30Claude Opus 4.8OpenAI Agents SDK40.0%51.0%67.0%87.0%67.0%109091New 100
32Gemini 3.1 ProClaude Agent SDK39.0%47.0%56.0%72.0%59.0%2674206Inherited SDK 100
32Gemini 3.8 FlashDeepSeek Harness39.0%40.0%43.0%54.0%47.0%44563311New 100
34Qwen3.7-MaxDeepSeek Harness35.0%41.0%46.0%60.0%51.0%3862353Inherited SDK 100
35DeepSeek V4.1 FlashClaude Agent SDK34.0%39.0%50.0%61.0%48.0%3565269New 100
36Qwen3.7-MaxOpenAI Agents SDK32.0%36.0%40.0%52.0%41.0%46543313New 100
37Qwen3.7-MaxDeepAgents30.0%10.0%14.0%22.0%15.0%3169310New 54 · Historical 46
38Gemini 3.8 FlashDeepAgents29.0%33.0%36.0%42.0%35.0%5842562New 100
39Qwen3.8-MaxDeepSeek Harness28.0%30.0%41.0%47.0%42.0%49512920New 100
40DeepSeek V4.1 FlashDeepAgents26.0%31.0%41.0%49.0%39.0%49513910New 100
40DeepSeek V4.1 FlashDeepSeek Harness26.0%30.0%31.0%35.0%32.0%64364816New 100
40GLM 5.3Claude Agent SDK26.0%33.0%37.0%44.0%37.0%55454510New 100
40Qwen3.8-MaxDeepAgents26.0%36.0%46.0%54.0%45.0%46542620New 100
40Qwen3.8-MaxOpenAI Agents SDK26.0%31.0%40.0%48.0%39.0%47533611New 100
45DeepSeek V4 ProClaude Agent SDK24.0%29.0%31.0%49.0%36.0%4951445Inherited SDK 100
46DeepSeek V4.1 FlashOpenAI Agents SDK22.0%26.0%30.0%36.0%30.0%63375112New 100
47GLM 5.3DeepAgents19.0%22.0%24.0%31.0%26.0%67335611New 100
48Gemini 3.1 ProDeepSeek Harness18.0%23.0%24.0%30.0%26.0%6931672Inherited SDK 100
48DeepSeek V4 ProDeepSeek Harness18.0%24.0%29.0%36.0%31.0%6238566Inherited SDK 100
50DeepSeek V4 ProDeepAgents17.0%4.0%4.0%5.0%5.0%6733598New 73 · Historical 27
51GLM 5.3DeepSeek Harness13.0%14.0%15.0%19.0%18.0%8119756New 100
51Qwen3.8-MaxClaude Agent SDK13.0%15.0%20.0%23.0%20.0%75256411New 100
53GLM 5.3OpenAI Agents SDK9.0%10.0%11.0%14.0%12.0%8614779New 100
54DeepSeek V4 ProOpenAI Agents SDK8.0%10.0%13.0%22.0%17.0%78226612New 100
55Qwen3.7-MaxClaude Agent SDK5.0%7.0%8.0%11.0%7.0%8713825Inherited SDK 100
55Gemini 3.8 FlashClaude Agent SDK5.0%5.0%6.0%6.0%6.0%946904New 100
Every score uses all 100 tasks; failed and unknown runs stay in the denominator.Download development results CSV ↓

† Includes scored tasks with no recorded value for this metric: historical DeepAgents results recorded Full replication only.

v10 report summary

Report
four-harness-dev100-20261011-v10-recovered01 · sha256:8b110f06092a
Report generated
Dataset
Dev100 development set (100 tasks) · 100 tasks · Development split
Dataset relation
The development split of a pinned 100-task selection drawn from Selected_1000 (dataset revision 59f9512); tasks are not published.
Task slots
5,600
Scored
3,786
Full replication
2,204
Failed
1,502 (Infrastructure 128 · Billing, within infrastructure 105 · Model-attributed 0 · Unattributed 1,374)
Unknown
312 (Infrastructure 163 · Provider rejection 1 · Provider access 3 · Other provider error 38 · Deadline 3 · Unattributed 104)
Pending
0
Provenance mix
  • New native run · 3982 tasks
  • Offline-derived DeepAgents metric · 182 tasks
  • Historical DeepAgents metric · 236 tasks
  • Inherited native SDK run · 1200 tasks
New-run stages
10
Attempt ledger
logical_selected_new_slots 3,982 · physical_scheduled_attempts 4,138 · superseded_attempts 156 · superseded_zero_api 28 · superseded_infrastructure 28 · superseded_queue 36 · superseded_operator_interruption 64
Unknown reasons
collateral_study_interruption 158 · unclassified_unknown 104 · upstream_transport 37 · upstream_rate_limit 5 · provider_access 3 · task_deadline_exhausted 3 · upstream_content_policy 1 · upstream_provider_error 1
Failure reasons
harness_runtime_error 381 · model_call_limit_exhausted 318 · no_single_fenced_python_program 246 · final_execution_failed 229 · output_limit_exhausted 150 · provider_billing_http_402 105 · wall_time_exhausted 32 · cancelled_before_dispatch 27 · provider_request_error 14
Acceptance
controller_stage finished · controller_finished_at 2026-10-11T16:35:50.157877+00:00 · report_exit_code 0 · independent_delivery_audit_passed true · evidence Stated in the server visualization README and data.json and in the study's daily report and handoff; the underlying receipts were not accessible to this audit.

Development data JSONDevelopment provenanceDevelopment results CSV

Evaluation protocol, metrics and ModularRSI update

Tasks and four baseline harnesses

Dev100 comprises 100 econometric replication tasks from Selected_1000 (dataset revision 59f9512). Every combination evaluates 14 models across four harness configurations: DeepAgents 0.7.13, DeepSeek Harness 0.1.5rc1, Claude Agent SDK 0.2.165 (Claude Code CLI 2.1.294) and an OpenAI Agents SDK 0.23.1 runtime, under shared per-task caps of 6 model calls, 4 run_python tool calls, 32,768 output tokens and 1,800 s wall time.

ModularRSI-evolved DeepAgents (67.0%)

Applying ModularRSI self-evolution to DeepAgents (evolving loop, tool-use, completion and LangGraph ToolNode observation-audit feedback) raises GPT-5.6 Sol from 64.0% to 67.0% strict replication on a fresh paired Dev100 run (98% compilation, 78% partial replication, 95% correct sign, 81% significance), matching the top 67.0% score on the board. Explore the ModularRSI results

  • Full replication: coefficient and standard error within 1% relative error; p-value within 0.01 absolute error (local-paper-v1).
  • Coefficient and SE within 5%: coefficient and standard error both within 5% relative error.
  • Coefficient within 5%: coefficient relative error at most 5%.
  • Correct sign: estimated coefficient and reference share the same sign.
  • Correct significance: correct sign and the same p-value threshold band (p < 0.01, 0.01–0.05, 0.05–0.1, ≥ 0.1).

Report summary · Data JSON · Release provenance · Download results CSV

02Research

Research directions

We study how models and harnesses work together across econometric replication and business analysis, turning execution trajectories into better harnesses and model training signals.

  1. 01

    Model–Harness Coupling & Harbor Integration

    Compare how different models interact with library, SDK and CLI harnesses, and unify econometric and business evaluations on the Harbor container framework.

    Metacase: one run from task to graded record

  2. 02

    Self-Evolving Harnesses (ModularRSI · 64% → 67%)

    Evolve modular harness hooks and observation-audit feedback from execution traces. On Dev100, ModularRSI-evolved DeepAgents reaches 67.0% strict replication with GPT-5.6 Sol.

    See our ModularRSI implementation and results

  3. 03

    Trajectory-Based Post-Training & Co-Evolution

    Use verified agent trajectories, tool feedback and rubric evaluations to drive supervised fine-tuning, reinforcement learning and model–harness co-evolution.

    Explore training and co-evolution directions