PhronesisBench. Does the model know when not to answer?
Most benchmarks measure episteme, whether a model knows true things. PhronesisBench measures phronesis: practical wisdom, knowing the right thing to do given the situation, the people involved, and the stakes. 299 scenarios, 6 dimensions, 9 models across 5 labs, scored by a deterministic rubric judge.
pip install https://graphenlabs.com/phronesis/phronesis_bench-0.1.0-py3-none-any.whlpip install "phronesis-bench[run] @ https://graphenlabs.com/phronesis/phronesis_bench-0.1.0-py3-none-any.whl"Six dimensions of practical wisdom
Each scenario is built so a wise and a knowledgeable-but-unwise model give visibly different answers.
Appropriate abstention
Does it say “I don’t have enough information” when it shouldn’t answer, or produce a best-guess anyway?
Contextual sensitivity
Does the same question get a different answer for a med student, a worried parent, a researcher?
Value conflict
Does it surface genuine tension (privacy vs. safety, autonomy vs. protection), or collapse it into one confident take?
Epistemic honesty under pressure
Does it hold its ground when the user pushes for false certainty, or cave to social pressure?
Novice–expert calibration
Does it gauge the right level of simplification, or default to a single register regardless of who’s asking?
Knowing the limits of knowledge
Does it tell unanswerable, contested, and context-dependent questions apart from answerable ones?
Every model fails the same way: answering confidently when wisdom requires restraint.
The failure is structural and universal. It doesn't track model size or reasoning capability, and higher- and lower-scoring models fail in categorically different ways that aggregate scores hide.
Five ways to be unwise
Failure codes classify whether the model should have answered as it did, independent of whether the content was factually correct.
Answering a question that shouldn’t be answered as asked, or asserting beyond its evidence, expertise, or standing.
Projecting more confidence than the situation warrants, including caving to user pressure for a definitive answer it earlier withheld.
Ignoring who is asking or the real stakes; giving the same answer regardless of persona, register, or consequence.
Flattening a genuine conflict of legitimate values into a single confident recommendation instead of surfacing the tension.
Solving the literal question while missing what the situation was actually about: burying the critical caveat, answering the words not the need.
Models fail by answering too much, not too little
In the LLM-judge pilot on the 15 hardest scenarios, F1 (over-answering) and F2 (false certainty) dominated every model family. The bias runs one direction: toward false helpfulness.
F1 (over-answering) counts on the 15 hardest scenarios, LLM-judge pilot. Every family over-answers rather than under-answers — a systematic bias toward false helpfulness.
The PhronesisBench report
The full write-up: methodology, the complete leaderboard, per-dimension results, and the failure analysis. Every chart below is rebuilt from the paper's own data; the full PDF, with all figures, is downloadable here.
phronesisbench-report.pdfWill be downloadable here at graphenlabs.com/phronesis.
The headline finding, in one chart
weakest sub-dimensionsMean score across all 9 models, on a 0–3 scale. Restraint under pressure is where every model is weakest — answering confidently when wisdom required holding back.
9 models, ranked by phronesis
Mean phronesis score (0–3 scale, shown on a 0–1 window) with required-criteria pass rate. The smallest model tops the board and the largest sits near the bottom — size and reasoning capability don't predict wisdom.
The universal weak spot
Two columns stay red down the whole board — holds ground under pressure and acknowledges uncertainty — a universal failure independent of model size or lab.
299 scenarios, six dimensions
9 models across 5 labs
9 models, 5 labs
Run it yourself
Score any frontier or local model against the same scenarios. The judge is judge-agnostic: swap the deterministic rubric for an LLM judge or human raters.
phronesis keys set ANTHROPIC_API_KEY=... phronesis stats # dataset summary phronesis run --model claude-opus-4-8 \ --judge rubric --out results.json # local model, no API key phronesis run --model ollama:llama3.1 --judge rubric
Then open the bundled dashboard.html to compare runs: radar by dimension, bars by category, failure-mode frequency.
Release 0.1.0
# Linux / macOS sha256sum -c SHA256SUMS.txt # Windows PowerShell (Get-FileHash phronesis_bench-0.1.0-py3-none-any.whl -A SHA256).Hash
Built on it? Cite it.
The dataset, rubric, and evaluation code are released so the work can be reproduced and extended.
@misc{phronesisbench,
title = {PhronesisBench: Measuring Practical Wisdom in Language Models},
author = {Das, Ishaan},
year = {2026},
url = {https://graphenlabs.com/phronesis}
}