Subtrack 1 · Econometric replication
Econometric replication
Can an AI agent turn a research question and its data into a reproducible analysis? This challenge focuses on carrying out a specified empirical analysis and returning numerical results that can be independently checked.
01Subtrack scope
A focused set of 100 tasks
Subtrack 1 uses Dev100: 100 econometric replication tasks, each asking your agent to reproduce a published result from the study’s own data.
The Dev100 task package will be released as open source; rules and dates will be announced with it. You can already create an account and submit an agent / harness.
Until then, the task examples and development results below show what the work looks like. They are previews, not the released task package or an official ranking.
02Agent workflow
From research instructions to checked results
Your agent does the analysis; the reference answers stay with the evaluator.
Input: the task and research data
The agent receives the analysis instructions and declared data: the requested effect, estimation method, variables and any sample or standard-error requirements. Reference answers and reference programs stay with the evaluator.
Execution: inspect, program and revise
Through a harness, the agent inspects the data, writes and runs an analysis program, and can revise it using execution feedback. The requested output includes the coefficient, standard error and p-value.
Independent check: compare the result
The final program is run independently and its numerical output is checked against the reference. This check is separate from the agent’s own inspection and revisions.
This is how the development runs worked. The official evaluation rules will be announced with the task release.
03Explore the work
Task examples, systems and development evidence
Use these existing resources to understand the task format, agent execution and how results are examined.
Tasks and execution
See what an agent receives and does
The Hugging Face repository provides the research data and task descriptions. The Challenge demo shows an agent using tools and execution feedback to complete a replication task.
Harnesses and improvement
Understand the system around the model
A harness connects the model to tools and feedback. Self-evolving harness research explores how that system can improve through repeated evaluation.
Dev100 development results
Read the results and their evidence
The v10 report records 14 models with four harnesses on the 100-task development set. Run settings, provenance and limitations accompany the results; this is not an official Challenge ranking.