deciding-with-confidence

Answers yes/no, multiple-choice and rubric-score questions about a piece of text with a probability for every option and a calibrated confidence, in the request and response shapes of OpenAI's Decisions API. Claude Haiku 5.5 does the judging; the skill estimates the probabilities that a purpose-built decision model would read off its logits.

echo '{"input": "I was charged twice.", "questions": [{"type": "choice", "name": "dept",
  "instructions": "Which department?", "choices": [{"value": "billing"}, {"value": "shipping"}]}]}' \
  | python3 scripts/decide.py run -

It runs through the Anthropic API (pip install anthropic and a key), the claude CLI, Amazon Bedrock, or parallel subagents inside Claude Code: the deciding-with-confidence plugin installs a decider agent pinned to Haiku 5.5 alongside the skill. On a 104-item eval it reached 0.846 accuracy against 0.894 for a purpose-built decision model, with matching calibration error. Its wrong answers come at low confidence, so thresholding on confidence and on agreement between samples catches most of them. See SKILL.md for the procedure and references/method.md for the measurements.

assets/eval.jsonl includes 64 items from the banking77 test split (PolyAI, CC-BY-4.0); each item names its source.

Skill folder: deciding-with-confidence

Release of deciding-with-confidence version 0.2.0

📥 Download & Install

⬇️ Download deciding-with-confidence.zip

To install:

1. Click the download link above (ignore the "Source code" archives below - they're auto-generated by GitHub)
2. Go to Claude.ai Customize › Skills
3. Upload the downloaded ZIP file
4. Requires paid Claude Pro or Team account

See official documentation for more details.

Recent Changes

ff42701 deciding-with-confidence 0.2.0: measured numbers for the new default and the stack
3a14f4a deciding-with-confidence: the conversation itself as evidence (input_transcript)
02e9146 deciding-with-confidence: one low-effort sample by default, angle stack, file inputs, batch
e44b0c7 docs: Update CHANGELOG.md for released skills