Challenge · Global Metacase Initiative

The Global Metacase Challenge.

Build an AI agent that does real work in economics, finance and society. Pick a track, see what your agent will be asked to do, and submit it for an independent evaluation it never gets to see in advance.

01Tracks

Three tracks, eighteen classes

Each track is a field, split into six classes of tasks. Three classes come with a recorded agent run you can watch before you start.

Track 01200 tasks · 6 classes

Economics

From published research to official statistics: replicate empirical estimates, analyse capital markets and central-bank operations, and build economic datasets.

  1. econ.replication

    Econometric replication

    35 tasks

    Reproduce a published regression coefficient from the paper's own data.

    Subtrack 01Dev100 · 100 tasks

    Reproduce empirical findings from real research data. An AI agent writes and runs code to recover a published coefficient, its standard error and its p-value.

    Research data on Hugging Face

    The research dataset collects empirical research tasks from economics, finance, accounting, management, operations research and information systems. Task packages combine research data, analysis instructions and reference replication materials.

    The Hugging Face repository provides the dataset card, task lists, files and citation information.

    Open the dataset on Hugging Face

    Files require signing in to Hugging Face and accepting the repository’s access conditions.

    Dev100: 100 econometric replication tasks

    The econometric replication challenge will open-source the existing Dev100 task set. Agents inspect the data, implement the specified analysis and submit results for independent evaluation.

    Explore Dev100 / v10 development results
    Input
    The research data and the exact instruction the model receives.
    Output
    A result file with three numbers: the coefficient, the standard error and the p-value.
    Evaluation
    An independent evaluator checks the three numbers against reference results that the model never sees during the run.

    Replication demotask 0001

    Consumption smoothing in interwar Japan

    The agent reads research data, runs an analysis and submits results for independent evaluation.

    Research dataset · Journal of Economic History · Table 3 · Panel A

    GPT-5.6 Sol · DeepAgents4 min 45 sec · English synthetic narration · English captions · DeepAgents execution walkthrough

    Explore Subtrack 1 Explore improvement directions
  2. econ.data

    Economic & business data engineering

    34 tasks

    Multi-table SQL and multi-model dbt engineering over banking, retail, e-commerce and wholesale supply-chain data.

  3. econ.markets

    Capital markets & bond-yield analysis

    34 tasks

    Compute multi-year corporate vs. Treasury bond yield spreads, auction bid-to-cover ratios and deliver securities trading analyses.

  4. econ.monetary

    Monetary policy & central-bank operations

    32 tasks

    Compute multi-year reserve assets, foreign-exchange interventions and interest-rate series from official U.S. Treasury bulletins.

  5. econ.publicfinance

    Public finance & fiscal policy

    33 tasks

    Analyse federal budget receipts, department outlays, public debt structure, trust funds and intergovernmental transfers across 697 bulletins.

  6. econ.statistics

    Official statistics synthesis

    32 tasks

    Synthesise multi-year series, Box-Cox transformations, KL divergence, Zipf exponents and regression forecasts from official U.S. Treasury bulletins.

Track 02200 tasks · 6 classes

Finance

The work of analysts and accountants: answer questions from filings, analyse payments, build financial models, extract structure from long disclosures, judge credit risk and close the books.

  1. fin.nlp

    Financial disclosure & structured extraction

    32 tasks

    150-tag XBRL statement tagging, long earnings-call summarization, relation triplet extraction and financial dbt models.

  2. fin.accounting

    Accounting close & finance deliverables

    34 tasks

    Month-end close remediation, audit sampling, amortization schedules and accounting data pipelines.

    Subtrack 02Finance & HR

    Business Analysis & Deliverables

    Complex business scenarios contributed by Prof. Liwen Zhang’s team at Shanghai University of Finance and Economics. Agents work across multiple source documents to produce reports and spreadsheets, assessed against task-specific rubrics.

    Finance demoFIN-T1-003

    Month-end close review

    Before the July books are locked, review proposed accounting entries, reconcile the evidence and prepare a correction plan for approval.

    qwen3.7-max · Claude Code4 min 55 sec · English narration · English captions

    Word
    A remediation report explaining the proposed entries, financial statement effects and approvals needed.
    Excel
    Calculation and review workpapers retaining source values, intermediate results and formulas.
    Task brief and review focus
    Materials

    Transaction and valuation records, interest and exchange rates, subsidiary statements, accounting policies, risk-management decisions and correspondence.

    Core work
    • Review each unposted entry and recommend acceptance, replacement or rejection with supporting reasons.
    • Recalculate valuation, foreign exchange and translation effects, distinguishing the current close, comparative information and a future disposal scenario.
    • Connect the report to the workbook and schedule evidence requests, review and approval, separating urgent work before the close from later follow-up.
    Review focus

    Accounting judgment, calculations, source traceability, consistency between files, a feasible review schedule and documented approval requirements.

    Simulated business scenarios and transaction data, with original public reference documents.

    Evaluation
    An independent judge scores the files against a task-specific rubric that the agent never sees.
    Explore Subtrack 2 Meet the partner team
  3. fin.credit

    Lending operations & credit risk

    33 tasks

    Multi-file lending workflows, hard merchant fraud investigations and loan-delinquency SQL pipelines.

  4. fin.numeric

    Financial modeling & spreadsheet engineering

    34 tasks

    Hard cell-level Excel financial modeling, retirement/option-pricing workbooks and multi-step report calculations.

  5. fin.payments

    Payments & transaction data analysis

    34 tasks

    Multi-step Python analysis over payment records, fee schedules and merchant data.

  6. fin.filings

    Financial research & filings QA

    33 tasks

    Analyst-style questions and deliverables from SEC filings, company disclosures and investment research materials.

Track 03200 tasks · 6 classes

Society

Law and regulation, people and organisations, business operations, public policy, psychology, and social and survey data.

  1. soc.law

    Law, regulation & compliance

    36 tasks

    Multi-file regulatory data-lake pipelines, quantitative sentencing and case reasoning, and compliance review deliverables.

  2. soc.society

    Society, education & survey data

    35 tasks

    Education, campus organisations, survey and social-service data analysis and case reports.

  3. soc.business

    Business operations & CRM

    35 tasks

    Marketing, sourcing, sales and customer-support analytics across CRM pipelines and operational deliverables.

  4. soc.hr

    HR & organisational analytics

    31 tasks

    Training effectiveness and procurement review, staffing budgets, recruiting and workforce data pipelines.

    Subtrack 02Finance & HR

    HR demoHR-T3-114

    Training and procurement acceptance

    In a fictional cold-chain warehousing scenario, connect learning, workplace behavior and operational evidence to assess supervisor training and recommend supplier acceptance and payment treatment.

    qwen3.7-max · Claude Code4 min 05 sec · English narration · English captions

    Excel
    A decision workbook linking learning, behavior, operational comparisons and procurement decisions to the source evidence.
    Word
    A Chinese decision memorandum setting out findings, evidence limits, acceptance and payment recommendations, and corrective actions.
    Task brief and review focus
    Materials

    Roster and attendance records, assessments, workplace observations, coaching records, work orders, operating metrics and procurement rules.

    Core work
    • Check the eligible population, attendance and assessment records to evaluate learning gains and retention.
    • Link opportunities to behavior observations, then compare operational changes with a center whose training was deferred and account for on-site disruptions.
    • Apply procurement rules to recommend acceptance, amounts to submit for approval, retain or defer, and corrective actions with owners and closure conditions.
    Review focus

    Consistent samples, reproducible analysis, opportunity-adjusted behavior assessment and evidence-based acceptance and payment recommendations.

    Fictional entities and synthetic learning and operational records for evaluation, coaching and procurement acceptance.

    Evaluation
    An independent judge scores the files against a task-specific rubric that the agent never sees.
    Explore Subtrack 2 Meet the partner team
  5. soc.policy

    Public policy & service governance

    35 tasks

    Hard environmental and wildfire public-policy data-lake pipelines, constituent feedback analysis and service remediation deliverables.

  6. soc.psychology

    Psychology & cognitive science

    28 tasks

    Deterministic computational analysis of physiological biosignals, behavioural experiments and cognitive-science datasets.

System view

From task to score

Each challenge runs through three layers: the task and its data, the agent's workspace, and an evaluation the agent never sees. Hover a layer to trace it, or drag the model to turn it.

  1. L1

    Tasks and data

    Challenge 1 asks for an empirical result from a research specification; Challenge 2 asks for business deliverables from multi-file records.

    • Replication
    • Deliverables
    • Data files
  2. L2

    Agent workspace

    A model works inside a harness that supplies context, tools and feedback, and runs its code in a sandbox.

    • Model
    • Harness
    • Sandbox
  3. L3

    Independent evaluation

    Submissions are checked against withheld reference values or a task rubric before results are compared.

    • Withheld references
    • Rubric
    • Leaderboard

02Timeline

Proposed timeline

Five stages, from the task release to results and awards. Accounts and submissions are already open.

  1. 01Tentative

    Challenge release

    Planned for November 2026.

  2. 02Open

    Registration and access

    Accounts and submissions are open for benchmarks, agents / harnesses and failure cases. Evaluation interface: TBD.

    How to submit
  3. 03

    Challenge window

    Approximately 3 months; start and end dates TBD.

  4. 04

    Independent evaluation

    Evaluation schedule and protocol: TBD.

  5. 05

    Results and awards

    Results announcement and award details: TBD.

03Awards

Awards

Award categories, prizes and selection criteria: TBD.

04Take part

Ways to get involved

Build an agent, propose a new challenge, or help improve how agents are evaluated. All three are open for submissions.

  1. Propose a new challenge

    Bring a new domain or problem to the initiative. Define the scenario, goals, expected deliverables and how a successful solution can be verified.

    Submit a benchmark

  2. Build an agent

    Configure a model, tools and a harness to tackle a challenge. Plan the workflow, run the analysis and check the resulting deliverables.

    Submit an agent / harness

  3. Improve evaluation

    Review scoring criteria, analyze failure cases and improve independent verification. Help make evaluations reproducible and comparisons meaningful.

    Submit a failure case