Live benchmark · August 2026

DiligenceBench

An equity-research agent evaluation measuring factual accuracy, analytical reasoning, and risk awareness across 150 tasks.

July 17, 2026

Malthe Have Musaeus¹ · Faisal Sayed² · Mersad Abbasi² · Daanish Khazi¹ · Karina Nguyen²

¹ Paper Instruments ² Thoughtful Lab

150tasks

143companies

5,428rubric criteria

10sectors

Benchmark design

DiligenceBench is a joint release from Paper Instruments and Thoughtful Lab: 150 open-ended equity-research tasks spanning energy, banking, biotech, insurance, technology, REITs, restaurants, industrials, and utilities.

Tasks ask agents to assess cash-flow quality, capital adequacy, off-balance-sheet exposure, reserve sufficiency, profitability inflections, accounting asymmetries, liquidity stress, segment divergence, and concentration risk. Each task requires external research rather than recall.

The benchmark has no fixed harness. It evaluates agents end-to-end—model configuration, retrieval, execution, tools, skills, and instructions. A strong response must find the correct filing, extract the relevant disclosure, calculate against it, distinguish evidence from inference, and return to the investment question.

Leaderboard

Rubric score · higher is better

#	Model	Provider	Harness	Rubric score ↓	Coverage

H3 Finance results. Every score is evaluated over all 150 tasks using GPT-5.5 as rubric judge.

New result

Grok 4.6 enters at #2

Grok 4.6 scores 53.2 in the Finance harness, 1.1 points above Claude Opus 5. The paired 95% confidence interval is −1.0 to +3.2 points, so the two should be read as statistically tied—not as a decisive model win.

Grok 4.653.2

Claude Opus 552.1

How to read it

Model and harness are one system

DiligenceBench evaluates the complete research agent. Retrieval, execution, finance tools, skills, instructions, and the underlying model all contribute to the score.

Evaluation

Dense, task-specific rubrics

Each submission is graded against roughly 30–40 binary criteria spanning factual accuracy, reasoning, and risk. Negative criteria penalize active errors.

Reference harnesses

A controlled ladder

The task, model interface, grader, and memo format remain fixed. Each rung adds domain support and tooling, isolating open-web retrieval, general execution, and finance-specific research infrastructure.

Loop

Prompt → Web → Answer

Layer added: Web search and page fetch
Working state: Conversation context only

The model searches the open web and reads fragments into context. It cannot write files, run code, or build a persistent corpus.

Sandbox

Prompt → Web → Workspace → answer.md

Layer added: Shell, Python, and writable files
Working state: Persistent sandbox

The model can download and parse filings, calculate locally, and revise a durable artifact. Discovery still relies on generic web retrieval.

Finance

Issuer → Filing → Evidence → answer.md

Layer added: SEC tools and finance skills
Working state: Persistent sandbox

The model resolves the issuer, selects the filing, and searches inside the authoritative source with instructions for source hierarchy and risk framing.

About these results

The original 16 models were evaluated across all three reference harnesses. The newly added Grok 4.6, Claude Opus 5, and Claude Sonnet 5 runs are complete H3 Finance evaluations (150/150 tasks, zero run errors); H1 and H2 values are intentionally left unavailable.

Grok 4.6 and all other entries use the published DiligenceBench tasks and GPT-5.5 judge. Scores are means of normalized weighted rubric rewards.

Research note · 4 min read

Why Grok 4.6 edged ahead

The score difference is small. The behavioral difference is not.

Grok was much more search-intensive: about 41 tool calls per task versus Opus’s 22. It made 3,293 SEC filing-search calls against 486 for Opus and uniquely satisfied 529 rubric criteria, compared with 470 uniquely satisfied by Opus.

Section scores

SectionGrok 4.6Opus 5

Factual accuracy55.054.2

Analytical reasoning53.251.3

Risk awareness50.849.3

Behavioral differences

Grok 4.6

Broader SEC search, more evidence gathering, more tool failures, and shorter final answers. Source failures appeared on 24 tasks.

Claude Opus 5

More shell-based document processing, longer answers, and fewer failed fetches. Source failures appeared on only four tasks.

Opus wrote longer answers—13,445 versus 10,966 characters—and used more turns, but it more often omitted the rubric’s exact requested formulation. Its cleaner execution did not consistently become the precise assertions rewarded by the judge.

Conclusion. Treat Grok 4.6 and Opus 5 as a statistical tie around 52–53%. Grok’s nominal advantage appears driven by aggressive search and slightly better rubric targeting; Opus was more efficient and sometimes more nuanced. Sonnet 5’s 46.2% is meaningfully below Grok, with a paired difference of +7.0 points and a confidence interval of roughly +4.5 to +9.4.

Economics

Price versus performance

Estimated mean inference cost per task at published serving rates. Each path follows one open-weight model from H1 to H3.

GLM 5.2GLM 5.1MiniMax M3Kimi K2.6Inkling

Token counts are reconstructed from complete traces with published tokenizers; cache reuse is inferred from the conversation prefix shared across turns. Hover points for exact values.

Trace example

Does Bank of America’s growth in commercial committed exposure overstate or understate the portfolio’s underlying credit risk?

H1 · Loop189 tool calls0.9

Repeats open-web search

web_search × 162
web_fetch × 26
final_answer × 1

H2 · Sandbox50 tool calls32.8

Builds a persistent local corpus

exec_command × 35
web_search × 9
apply_patch × 6

H3 · Finance44 tool calls65.7

Resolves the issuer, then stays in the filing

issuer + filings × 2
sec_filing_search × 34
other × 8

H1 repeats open-web search. H2 builds a local corpus. H3 resolves the issuer and filing, then stays inside the primary source.

Release

Research, openly.

The dataset and repository are released under the Apache 2.0 license. The public task set covers 143 large-accelerated US companies and includes 5,428 criteria, with 672 negatively weighted criteria for active errors.

All published results use GPT-5.5 as the LLM judge. Each complete response is graded once against every weighted positive and negative criterion; the final reward is the normalized weighted sum.

Reference

Citations

Citation

Musaeus, Malthe Have; Sayed, Faisal; Abbasi, Mersad; Khazi, Daanish; and Nguyen, Karina, “DiligenceBench: An Equity-Research Agent Evaluation”, Paper Instruments Blog, Jul 2026.

BibTeX

@misc{musaeusetal2026diligencebench,
  author = {Malthe Have Musaeus and Faisal Sayed and Mersad Abbasi and Daanish Khazi and Karina Nguyen},
  title = {DiligenceBench: An Equity-Research Agent Evaluation},
  howpublished = {Paper Instruments Blog},
  year = {2026},
  url = {https://www.paperinstruments.com/blog/diligence-bench}
}

Open benchmark

Run DiligenceBench
on your agent.

View repository ↗Get the dataset ↗