Metadata-Version: 2.4
Name: phronesis-bench
Version: 0.1.0
Summary: A benchmark measuring practical wisdom (phronesis) in language models, distinct from raw knowledge.
Project-URL: Homepage, https://github.com/siddhantdas333/phronesis-bench
Project-URL: Dataset, https://github.com/siddhantdas333/phronesis-bench/tree/main/data
Author-email: Ishaan Das <ishaandas40@gmail.com>
License: MIT
License-File: LICENSE
Keywords: ai-safety,benchmark,evaluation,llm,phronesis,practical-wisdom
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Requires-Dist: pydantic>=2.6
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: gguf
Requires-Dist: llama-cpp-python>=0.2.80; extra == 'gguf'
Provides-Extra: run
Requires-Dist: anthropic>=0.40; extra == 'run'
Requires-Dist: google-genai>=0.3; extra == 'run'
Requires-Dist: httpx>=0.27; extra == 'run'
Requires-Dist: openai>=1.40; extra == 'run'
Description-Content-Type: text/markdown

# PhronesisBench

**A benchmark for practical wisdom in language models.**

Most LLM benchmarks measure *episteme* — whether the model knows true things. PhronesisBench
measures *phronesis* — whether the model knows the **right thing to do or say given the
situation, the people involved, and the stakes**. A model can be factually flawless and still
unwise: dosing a medication without asking the patient's weight, answering a question that
wasn't really askable, collapsing a genuine value conflict into a confident one-liner, or
caving to social pressure for certainty it shouldn't have.

The benchmark is three things:

1. **A dataset** — **299 hand-validated scenarios** (~50 per category) across six dimensions
   of practical wisdom, each annotated with what a *wise* response looks like, what a
   *knowledgeable-but-unwise* response looks like, and which [failure
   codes](docs/FAILURE_TAXONOMY.md) the unwise response exhibits. Reusable on its own.
2. **An evaluation harness** — run any frontier model through the scenarios and score it,
   with pluggable judges (LLM judge, human raters, or a heuristic baseline).
3. **A paper** — connecting the Aristotelian framework to empirical measurement, reporting
   where current frontier models fail at phronesis and how those failures cluster.

## The six dimensions

| Dimension | The question it asks |
|-----------|----------------------|
| **Appropriate abstention** | Does it say "I don't have enough information" when it shouldn't answer? |
| **Contextual sensitivity** | Does the same question get different answers for a med student, a worried parent, a researcher? |
| **Value conflict** | Does it acknowledge genuine tension (privacy vs. safety, autonomy vs. protection) or collapse it? |
| **Epistemic honesty under pressure** | Does it hold its ground when pushed for false certainty? |
| **Novice–expert calibration** | Does it gauge the right level of simplification, or default to one register? |
| **Knowing the limits of knowledge** | Does it tell unanswerable / contested / context-dependent questions apart from answerable ones? |

## Install

```bash
pip install phronesis-bench            # dataset + schema, no API keys needed
pip install "phronesis-bench[run]"     # + model clients for running evals
```

Or from source:

```bash
git clone https://github.com/siddhantdas333/phronesis-bench
cd phronesis-bench
pip install -e ".[run,dev]"
```

## Use the dataset

```python
from phronesis import load_scenarios

for s in load_scenarios("value_conflict"):
    print(s.id, s.prompt)
    print("  wise:", s.phronetic_response.summary)
    print("  unwise:", s.epistemic_response.summary)
```

## API keys

Set provider keys once and they're picked up automatically. Easiest is a local `.env`
file (auto-loaded, git-ignored):

```bash
phronesis keys                                    # show which keys are set
phronesis keys set GEMINI_API_KEY=...             # saves to ./.env
phronesis keys set ANTHROPIC_API_KEY=... OPENAI_API_KEY=...
```

Plain environment variables (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`)
still work and take precedence over `.env`. Get keys:
[Anthropic](https://console.anthropic.com/settings/keys),
[OpenAI](https://platform.openai.com/api-keys),
[Google AI Studio](https://aistudio.google.com/apikey). A missing key produces a message
telling you exactly how to set it.

## Run an evaluation

```bash
phronesis keys set ANTHROPIC_API_KEY=...          # or export ANTHROPIC_API_KEY=...
phronesis stats                                   # dataset summary
phronesis run --model claude-opus-4-8 --judge claude-opus-4-8 --out results.json
phronesis run --model gpt-4o --judge rubric --category abstention --limit 20

# local models, no API key:
phronesis run --model ollama:llama3.1 --judge rubric --out llama.json
phronesis run --model "local:Qwen2.5-7B@http://localhost:8000/v1" --judge rubric
```

Model specs: `claude-*`, `gpt-*`, `gemini-*` (hosted); `ollama:<m>`, `local:<m>@<url>`,
`gguf:<path>` (local — see [docs/LOCAL_MODELS.md](docs/LOCAL_MODELS.md)).

The scoring layer is **judge-agnostic**: every scenario carries checkable criteria and 0–3
anchors, so the same data drives an LLM judge, human raters, or a heuristic baseline. Each
response is scored three ways — **dimensions** (0–3), **per-criterion** (pass/fail, so you
see exactly which wise behavior was missed), and **failure codes** (which entries of the
[failure taxonomy](docs/FAILURE_TAXONOMY.md) the response exhibited). Long runs are
resilient: failures are classified into stable error codes and retried with backoff.

Long runs are **checkpointed**: results are scored as they arrive and flushed to `--out`
every N scenarios (`--checkpoint-every`, default 10), so a cancelled or crashed run still
leaves a valid partial file. Writes are atomic (temp file + rename), so a reader never sees
a truncated document. Partial files carry `"partial": true` and `"scored_so_far"`.

To compare several models, write each to its own file under `results/`:

```bash
phronesis run --model gemini-2.5-flash      --judge rubric --out results/gemini-2.5-flash.json
phronesis run --model gemini-2.5-flash-lite --judge rubric --out results/gemini-2.5-flash-lite.json
```

## Visualize results

Open [web/dashboard.html](web/dashboard.html) in any browser (no server) and either **Open
results.json** (one or more files) or **Load results/ folder** to merge every `*.json` in a
directory at once — radar by dimension, bars by category, failure-mode frequency, and a
run-error breakdown. Add more model files to `results/` and reload to grow the comparison.
Click **Load demo data** to preview, **Clear** to reset. Roadmap for a live web app:
[docs/WEB_UI_DESIGN.md](docs/WEB_UI_DESIGN.md).

## Repository layout

```
data/scenarios/*.jsonl     the dataset, one file per category (the reusable artifact)
src/phronesis/             models (schema), dataset loader, runner, scoring, CLI
tests/                     dataset integrity + scoring pipeline tests
docs/                      schema and scoring-dimension specifications
paper/                     the write-up
```

## Citing

```bibtex
@misc{phronesisbench,
  title  = {PhronesisBench: Measuring Practical Wisdom in Language Models},
  author = {Das, Ishaan},
  year   = {2026},
  url    = {https://github.com/siddhantdas333/phronesis-bench}
}
```

## License

MIT. The dataset is released for research use; please cite if you build on it.
