Subtrack 1 · Econometric replication
Econometric replication
Can an AI agent turn a research question and its data into a reproducible analysis? This challenge focuses on carrying out a specified empirical analysis and returning numerical results that can be independently checked.
01 / Subtrack scope
A focused set of 100 tasks
Subtrack 1 will use the existing Dev100 set of 100 econometric replication tasks. This task count belongs to this challenge, not to the whole Inference Challenge series.
The Dev100 task package is planned for open-source release and is not yet public. Formal rules, dates and the submission channel have not been announced.
Existing task examples and development results explain the work so far. They are research resources, not the released Challenge task package or official competition rankings.
02 / Agent workflow
From research instructions to checked results
The existing research workflow separates the agent’s analysis from the evaluator’s reference answers.
Input: the task and research data
The agent receives the analysis instructions and declared data: the requested effect, estimation method, variables and any sample or standard-error requirements. Reference answers and reference programs stay with the evaluator.
Execution: inspect, program and revise
Through a harness, the agent inspects the data, writes and runs an analysis program, and can revise it using execution feedback. The requested output includes the coefficient, standard error and p-value.
Independent check: compare the result
The final program is run independently and its numerical output is checked against the reference. This check is separate from the agent’s own inspection and revisions.
This describes the recorded research workflow. The formal Challenge evaluation rules will be announced with the release.
03 / Explore the work
Task examples, systems and development evidence
Use these existing resources to understand the task format, agent execution and how results are examined.
Tasks and execution
See what an agent receives and does
The Hugging Face repository provides the research data and task descriptions. The Challenge demo shows an agent using tools and execution feedback to complete a replication task.
Harnesses and improvement
Understand the system around the model
A harness connects the model to tools and feedback. Self-evolving harness research explores how that system can improve through repeated evaluation.
Dev100 development results
Read the results and their evidence
The v10 report records 14 models with four harnesses on the 100-task development set. Run settings, provenance and limitations accompany the results; this is not an official Challenge ranking.