Global Metacase Initiative

Global Metacase Challenge

We present the Global Metacase Challenge（Global Metacase Challenge）, an open benchmark where AI agents take on long-horizon work from research and professional practice in Economics, Finance, and Society, each with a clear goal and an independently checked deliverable. Where a conventional trajectory logs each step, a Metacase builds on the full trajectory to record, end to end, what the agent knew, what it weighed, whether the result passed verification, and what can be reused.

The goal: AI systems and researchers retrieve and reason over similar problem-solving structures across fields, a foundation for experience-based AI.

Join the Challenge Explore the fields Watch an agent run
Nanyang Technological University, Singapore The University of Hong Kong Shanghai University of Finance and Economics PBC School of Finance, Tsinghua University

Hosted by NTU GIFTS
with HKU, SUFE and Tsinghua PBCSF

01The Metacase

A Metacase is one episode, recorded end to end.

The agent's execution trajectory sits at the centre. Before it come the goal, evidence and constraints it started from and the options it weighed; after it come the independent check and what the next attempt can borrow. Each Challenge task starts at the point of decision: the agent sees only what was known then, and the references used for scoring stay hidden from it.

  1. 01Context & EvidenceGoal · Evidence · ConstraintsWe collect highly representative, challenging long-horizon complex problems from research and professional practice — preserving the goal, raw evidence, and constraints as they stood at the decision cutoff, with zero hindsight leakage.
  2. 02Prior Practice & StrategyPractice · Options · Trade-offsRelevant knowledge and the earlier practice of people or institutions, the candidate approaches, the trade-offs between them, and why one was chosen and others set aside.
  3. 03Actions & ExecutionSteps · Tools · Intermediate workWhat was actually done: the steps, the tools used and the intermediate work produced. For an agent this is a Model + Harness run: the model decides, the harness carries out the tool calls and keeps the log.
  4. 04Outcomes & VerificationDeliverable · Result · Checks · LimitsThe final deliverable, what came of it, how it was checked and where the check falls short. An outcome may be a success, a partial success or a failure; all three are kept.
  5. 05Reflection & ReuseWhat held · What failed · When it appliesWhich judgements held and which did not, the conditions under which the experience applies, and what the next attempt should borrow or change.
  6. SignatureSolution signatureCross-case structure · retrievalClosed again, each stage leaves a tab on the fore-edge; together they trace the problem-solving structure of this case so similar solution paths can be retrieved and compared across fields.
  1. Context
  2. Strategy
  3. Actions
  4. Outcomes
  5. Reflection

02Retrieval

Retrieve experience by how a problem was solved.

Every Metacase carries a solution signature: how the evidence was framed, which options were weighed, the order of actions, and how the outcome was checked. Searching by that structure finds cases worked the same way across fields — providing a better way to aggregate and organize agent trajectories into reusable experience.

  1. Topic matchA topic search for this credit decision returns more credit cases. Laid over its signature, their paths split at most stages: they weighed evidence, acted and checked in other ways. A shared topic does not mean the experience applies.
  2. Solution matchA signature search returns cases from economics and society whose stages correspond: evidence organised alike, options compared, the same order of action, a similar check. Each ringed stage is a step that lines up; the slips are a schematic of that comparison.
  3. ReuseA person or an agent reads the matched cases for their evidence, trade-offs, outcomes and limits, decides what is worth carrying over, and tests it on the new task. A match informs the next attempt; it does not supply the answer.
  1. Topic match
  2. Solution match
  3. Reuse

03The network

A reusable network of Metacases.

Metacases are linked wherever their solution structures correspond, within a field and across fields. Each reviewed case adds a record and its links, so a new problem can start from related experience instead of from scratch.

  1. Field by fieldEach field grows from its own cases: economics, finance and society, each with its own data, institutions and practice.
  2. Linked by solution pathWhere cases in different fields were worked through the same way, their signatures correspond and the network links them.
  3. TransferPeople and agents follow those links to see how a comparable problem was handled elsewhere, then test whether the approach carries over.
  4. The foundationA foundation for experience-based AI.That is the aim: as reviewed cases accumulate, new problems start from more recorded experience. The first fields are economics, finance and society; healthcare, policy and law belong to the longer-term vision.
  1. Fields
  2. Links
  3. Transfer
  4. Foundation

04The first fields

Three fields to start, eighteen classes of cases.

Economics, finance and society come first: each offers public evidence, documented decisions and outputs that can be checked. Open a field's book to see its classes.

600tasks
3fields
18classes

Field 01 Economics

  1. econ.replication

    Econometric replication

    Reproduce a published regression coefficient from the paper's own data.

  2. econ.markets

    Capital markets & bond-yield analysis

    Compute multi-year corporate vs. Treasury bond yield spreads, auction bid-to-cover ratios and deliver securities trading analyses.

  3. econ.data

    Economic & business data engineering

    Multi-table SQL and multi-model dbt engineering over banking, retail, e-commerce and wholesale supply-chain data.

  4. econ.publicfinance

    Public finance & fiscal policy

    Analyse federal budget receipts, department outlays, public debt structure, trust funds and intergovernmental transfers across 697 bulletins.

  5. econ.monetary

    Monetary policy & central-bank operations

    Compute multi-year reserve assets, foreign-exchange interventions and interest-rate series from official U.S. Treasury bulletins.

  6. econ.statistics

    Official statistics synthesis

    Synthesise multi-year series, Box-Cox transformations, KL divergence, Zipf exponents and regression forecasts from official U.S. Treasury bulletins.

Field 02 Finance

  1. fin.accounting

    Accounting close & finance deliverables

    Month-end close remediation, audit sampling, amortization schedules and accounting data pipelines.

  2. fin.numeric

    Financial modeling & spreadsheet engineering

    Hard cell-level Excel financial modeling, retirement/option-pricing workbooks and multi-step report calculations.

  3. fin.payments

    Payments & transaction data analysis

    Multi-step Python analysis over payment records, fee schedules and merchant data.

  4. fin.filings

    Financial research & filings QA

    Analyst-style questions and deliverables from SEC filings, company disclosures and investment research materials.

  5. fin.credit

    Lending operations & credit risk

    Multi-file lending workflows, hard merchant fraud investigations and loan-delinquency SQL pipelines.

  6. fin.nlp

    Financial disclosure & structured extraction

    150-tag XBRL statement tagging, long earnings-call summarization, relation triplet extraction and financial dbt models.

Field 03 Society

  1. soc.law

    Law, regulation & compliance

    Multi-file regulatory data-lake pipelines, quantitative sentencing and case reasoning, and compliance review deliverables.

  2. soc.business

    Business operations & CRM

    Marketing, sourcing, sales and customer-support analytics across CRM pipelines and operational deliverables.

  3. soc.policy

    Public policy & service governance

    Hard environmental and wildfire public-policy data-lake pipelines, constituent feedback analysis and service remediation deliverables.

  4. soc.society

    Society, education & survey data

    Education, campus organisations, survey and social-service data analysis and case reports.

  5. soc.hr

    HR & organisational analytics

    Training effectiveness and procurement review, staffing budgets, recruiting and workforce data pipelines.

  6. soc.psychology

    Psychology & cognitive science

    Deterministic computational analysis of physiological biosignals, behavioural experiments and cognitive-science datasets.

05THE GLOBAL METACASE CHALLENGE
SCROLL TO EXPLORE
01 / THE ENTRANCE

Three portals.
Three tracks.

Economics, Finance and Society. Step through a portal to reach its track: the tasks, the data and how runs are scored.

THREE TRACKS · SIXTEEN CASE CATEGORIES
econ/ 5fin/ 6soc/ 5
  1. 1 · PROVIDER
  2. 2 · MODEL
  3. 3 · EFFORT
  4. 4 · HARNESS Awaiting a loadout Choose a harness at the equipment wheel

OpenAI · GPT-6 Astra · XHigh × DeepSeek Harness

01/ 06
METACASE / TRACE

Every case begins as a question, its raw evidence and the practice that came before.

  1. 01ENTER
  2. 02EQUIP
  3. 03ECONOMICS
  4. 04FINANCE
  5. 05SOCIETY
  6. 06CHOOSE

Economics · Finance · Society / 3 domains, 18 case categories.

∴ GLOBAL METACASE CHALLENGE

One agent. Many harnesses. Real cases, from evidence to experience.

06Participation workflow

How to participate in the challenge.

What a participant does: take a case from one of the three tracks, run a model and its harness on it end to end, have the deliverable checked against references the agent never sees, and turn what happened into something others can inspect and reuse.

  1. 1 · Choose a caseThe participant takes a case from a track and opens it: the task goal, the evidence available at the decision point and its constraints. Every entry in a track works from the same materials and is scored the same way.
  2. 2 · Configure Model + HarnessThey pick a model and the harness that runs it. The model reasons and decides; the harness supplies the tools, the workflow and the run settings, so the same model can behave differently in a different harness.
  3. 3 · Execute and recordThe agent reads the materials, plans, calls tools and writes files until it delivers end to end. Every step goes into the trajectory, so the whole run can be traced afterwards.
  4. 4 · Verify and diagnoseThe deliverable goes to an independent check against reference results and rubrics the agent never saw during the run. A result can be a pass, a partial pass or a failure, and the trajectory shows where it went wrong.
  5. 5 · ContributeWhat the run yields can be contributed in three forms: a Metacase, the structured record of the case; the Model & Harness, packaged so the run can be repeated; or a Failure Case, a reproducible failure with its diagnosis (a low score alone is not one). Contribute one or more, separately from the choice of track.
  1. Choose
  2. Configure
  3. Execute
  4. Verify
  5. Contribute

07Results so far

See how runs have been scored.

Harbor scores every run against a frozen catalog of hard tasks. Dev100 is its 100-task econometric replication split; the 14-model × 4-harness v10 development board sits beside these figures.

See the leaderboard Watch an agent run

  • 600Candidate tasks

    3 fields × 200 tasks · 18 classes

    • 35Dev100 causal replications
    • 484Hard Harbor registry tasks, pinned at commit
    • 81Workflow deliverable tasks

  • 8 × 5Models × harnesses on Dev100

    3,999 / 4,000 trials scored

    • DeepAgents
    • Terminus-2
    • Claude Agent SDK
    • OpenAI Agents SDK
    • DeepSeek Harness

  • 9Harnesses on GPT-5.6 Sol

    900 / 900 Dev100 trials scored

Submit to the Challenge

08Team & collaboration

Meet our community and collaborators.

The Global Metacase Initiative is founded by the Global Institute of Finance, Technology, and Society (GIFTS) at Nanyang Technological University and carried out with collaborating organizations in Asia and the United States. The organizations and the people behind it are listed separately below.

Meet the team Contact us

Founding organization

NTU GIFTS

Global Institute of Finance, Technology, and Society · Nanyang Technological University

Founded the Global Metacase Initiative and hosts its first pipeline, the Global Metacase Challenge.

Collaborating organizations

NTU CCDS

College of Computing and Data Science

SLAI

Shenzhen Loop Area Institute

People

  • Lin William CongFounding Director, GIFTSPresident's Chair Professor of Finance, Computing and Data ScienceNanyang Business School & CCDS, NTU
  • Ye LuoLab Director, HKU AI-Agents LabAssociate ProfessorHKU Business School, The University of Hong Kong
  • Liwen ZhangLab Director, SUFE AIFLM LabProfessorSchool of Statistics and Data Science, Shanghai University of Finance and Economics

Student leaders

Other team members and contributors