Subtrack 2 · Business Analysis & Deliverables
Business Analysis & Deliverables
Reason through a business problem. Deliver work that holds up.
Research replication and business analysis both involve agent workflows. This subtrack is organized by its task goal: analyze a business problem, support decisions with evidence and deliver a reviewable report and workbook.
Agent in action · Finance
From a finance task to independent review
01 / Agent workflow
From the task goal to a checked submission
The agent needs to plan, use tools, inspect evidence and revise its work. The outline below describes how these tasks can be approached; the pilot results show what the first runs actually delivered.
Task goal
The brief defines the decisions to support, the scope and the expected work product.
Finance: close corrections and review planning. HR: training impact and procurement recommendations.
Inside the agent workflow
The agent has the brief and materials; the rubric and reference outputs are withheld.
Understand & plan
Break the goal into questions, find the relevant files and choose the calculations and tools.
Read, reconcile & calculate
Use tools to inspect multiple files, connect claims to evidence, resolve inconsistencies and compute results.
Build the work product
Write the report and workbook with supporting evidence, calculations and explicit conclusions.
Check & revise
Reopen the files, verify formulas and totals, and check that both deliverables support the same conclusions.
If checks reveal a gap, revisit the evidence, calculations or plan and update the deliverables.
After submission · Independent evaluation
Judge against the rubric and source evidence
A separate judge evaluates the submitted files using the task rubric and supplied materials. Agent self-checks do not replace this final evaluation.
Observed in the HR pilot
Read rules and evidence → write and run a Python script → inspect results → edit the script and rerun → check the Word and Excel files.
These tool checks and revisions did not prevent the attendance errors and behavioral misclassification documented below.
The finance package combines public references with synthetic business records. The HR package uses synthetic materials. Both task briefs require using the supplied materials.
02 / Task cases
Two briefs. Two kinds of judgment.
Open each case to see the input, the required work product and what a review can catch. Descriptions summarize the task packages; source records are not included here.
FinanceFIN-T1-003
July Month-End Close: Corrections and Review PlanningResolve accounting issues and prepare a group close review supported by a calculation workbook.
55input files
Materials
Ledgers, internal policies and correspondence, accounting standards and annual reports.
18 CSV · 28 DOCX · 4 PDF · 4 HTML · 1 LOG
Deliverables
A Word review document and an Excel workbook with five non-empty worksheets, retaining the formulas needed to trace the calculations.
DOCX + XLSX
What review checks
Accounting treatment, hedge calculations, partial disposal analysis, traceability and consistency between the narrative and workbook.
25 scored criteria, including 5 penalties · 2 red lines
What the pilot exposed
The agent produced all five worksheets without formulas, failing the traceability check. It also failed to distinguish a 70% economic hedging target from the accounting hedge ratio and omitted parts of the disposal analysis.
Human resourcesHR-T3-114
Cold-Chain Supervisor Training: Impact Review and Procurement AcceptanceAssess training impact and recommend acceptance and payment for three procurement packages.
17input files
Materials
Training and attendance records, operational results, procurement evidence and supporting documents.
8 CSV · 5 XLSX · 4 DOCX
Deliverables
An Excel analysis workbook and a Word memorandum covering training outcomes, package acceptance and the allocation of payments.
XLSX + DOCX
What review checks
Population scope, attendance, center-level impact estimates, evidence-unit acceptance and payment categories. Limited opportunity to demonstrate a skill must be distinguished from poor performance.
34 scored criteria, including 6 penalties · No red lines
What the pilot exposed
The agent omitted the source index, reported 80/81 attendance instead of 82/83 and used an average across centers instead of center-level net changes. A penalty was triggered when it marked everyone with insufficient opportunity as failing the behavioral standard.
03 / How review works
Independent review after submission
The judge evaluates the final submission separately from the agent’s own checks. It examines the output files against the rubric and uses the supplied materials to verify their claims.
Rubric reward · 0–1
Weighted positive credit − weighted violations
divided by the total positive weight
The result is clipped to [0, 1]. Penalties do not increase the denominator. A confirmed red-line violation makes the finance reward zero.
- Multiple dimensions
- Professional standards, correct conclusions, calculations, instruction following, analysis, factual fidelity and safety.
- Reference outputs
- Used to calibrate structure and scale, not as an answer to copy. A different valid solution should not lose credit for differing from the reference.
- Scoring failures
verifier_errormarks an unavailable score that needs reevaluation, not a valid zero. It must be reported separately from scored outcomes.
Finance uses binary and five-level criteria; five-level ratings are normalized to [0, 1]. HR uses binary criteria. Total positive weight: finance 172; HR 106.
04 / Pilot & calibration
A first run, with limits in view
On 18 Oct 2026, each task had one reference-output scoring run and one agent run. All four received a score with no verifier error. This is a pipeline pilot, not evidence of broad model performance.
| Task | Run | Rubric reward | Review outcome |
|---|---|---|---|
FIN-T1-003 | Reference output | 1.000 | No penalty or red-line violation |
FIN-T1-003 | Agent · qwen3.7-max | 0.333 | No penalty or red-line violation |
HR-T3-114 | Reference output | 0.783 | 3 positive criteria unmet; no penalty |
HR-T3-114 | Agent · qwen3.7-max | 0.208 | 1 penalty triggered |
Scores are rounded to three decimals and are not accuracy rates. The two task rubrics differ; these four runs are not a leaderboard or a controlled comparison across domains.
Download the pilot summaryThe downloadable summary records task versions and hashes of the result and reward files. Original task materials and submissions are not included on this public site.
- Runner & agent
- Harbor 0.20.0 · Claude Code 2.1.114 · qwen3.7-max
- Finance judge
deepseek-v4.1-flashoverrides the package defaultdeepseek-flash.- HR judge
qwen3.7-plususes the package default.
Before formal evaluation
The reference also needs calibration
The HR reference scored 0.783. Its three unmet criteria reveal two issues to resolve before this becomes a formal subtrack.
Clarify cross-file requirements
Two criteria were read as requiring accepted evidence-unit counts in both the workbook and memorandum. The reference gave the counts only in the workbook. The rubric wording and reference need to agree on what each file must state.
Define formula recalculation
Thirteen formulas in the reference payment sheet had no cached values. Reading the cached results returned blanks and caused one criterion to fail. Recalculation and formula-reading rules need to be standardized.
Subtrack status
Task set, evaluation rules and submission entry point: to be announced. This page presents the task format and the initial pilot while rubric calibration is still in progress.