Checks whether the AI that grades your AI can be trusted.

How it works

Five steps, from labels to a report

The idea is simple. A person grades a small set of answers once. judgekeeper then measures the judge against those grades, and keeps measuring.

1. 1 Examples graded by a person

   Pick about 100 answers from your product. A person marks each one pass or fail. That mark is called a human label. You can label in a spreadsheet, or in a small page judgekeeper opens on your own computer:

   judgekeeper label items.jsonl --out labels.csv

2. 2 Freeze them

   judgekeeper saves the labeled examples as an anchor set: the fixed answer key every later check uses. It records a hash, a short fingerprint of the file such as sha256 2e75fe64157b…. If anyone edits a label later, the fingerprint no longer matches and judgekeeper stops.

   judgekeeper import-labels labels.csv -o anchors.jsonl

3. 3 The judge grades them, three times

   Your judge grades every example. Each grade is a verdict. Each full pass over the examples is a run. Running three times shows how often the judge changes its mind.

   judgekeeper judge anchors.jsonl --callable my_judges:judge --runs 3 --out runs/mine

   Here my_judges:judge is your own Python function. judgekeeper can also call Anthropic or OpenAI models directly, or any program. See Works with what you already use.

4. 4 Compare with the people

   judgekeeper lines up each verdict with the human label for the same answer. It counts where they agree and where they do not, run by run.

   judgekeeper validate anchors.jsonl runs/mine --out reports/mine

5. 5 Read the report

   You get one page. It says how often the judge is right, how noisy it is, where it is weak, and lists every disagreement with the judge's own reason. It ends with a verdict: usable as a gate, usable with care or not trustworthy as a gate. Open reports/mine/report.html in your browser, or see a real one.

judgekeeper check results.csv --judge verdict --human label --out reports/my-judge

Two more things it measures

Noise and position bias

In the real result below, 3.6% of items changed verdict in at least one of three runs, and 9.9% of verdicts changed when the two answers swapped places.

Integrations

Works with what you already use

judgekeeper does not replace your tools. It reads what they already save. Open a tile for the exact command.

judgekeeper check results.csv --judge verdict --human label --out reports/my-judge/

# mypkg/judges.py
def my_judge(item):
    return call_my_model(item)   # "pass", "fail", True, False, a score, or {"verdict": ..., "reason": ...}

judgekeeper judge anchors.jsonl --callable mypkg.judges:my_judge --runs 3 --out runs/mine/

judgekeeper judge anchors.jsonl --exec "node judge.js" --runs 3 --out runs/mine/

promptfoo eval -o results.json --repeat 3
judgekeeper import promptfoo results.json --metric helpfulness --labels labels.csv --out reports/helpfulness/

export DEEPEVAL_RESULTS_FOLDER=deepeval-results   # then run your DeepEval tests 3 times
judgekeeper import deepeval deepeval-results/ --metric "Correctness [GEval]" --labels labels.csv --out reports/correctness/

inspect eval task.py --epochs 3 --log-format json
judgekeeper import inspect logs/ --metric model_graded_qa --labels labels.csv --out reports/qa/

pip install "judgekeeper[mlflow]"
judgekeeper import mlflow --experiment my-app-eval --metric correctness --out reports/correctness/

judgekeeper import langfuse --judge-score helpfulness --human-score helpfulness_human --from 2026-09-01 --pass-if "score>=0.5" --out reports/helpfulness/

judgekeeper judge anchors.jsonl --runner anthropic --model claude-haiku-4-5-20251001 --prompt prompts/single.md --runs 3 --out runs/haiku/

judgekeeper judge anchors.jsonl --runner openai --model "$JUDGE_MODEL" --prompt prompts/single.md --runs 3 --out runs/openai/

judgekeeper judge anchors.jsonl --runner openai --model "$JUDGE_MODEL" --base-url http://localhost:11434/v1 --prompt prompts/single.md --runs 3 --out runs/local/

- uses: judgekeeper/judgekeeper@main   # pin to a release tag
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  with:
    anchors: evals/anchors.jsonl
    runner: anthropic
    model: claude-haiku-4-5-20251001
    prompt: prompts/judge.md
    runs: 3

pytest --judgekeeper-report reports/my-judge/report.json

def test_judge_still_agrees_with_humans(judgekeeper_gate):
    judgekeeper_gate("reports/my-judge/report.json")

Privacy

Your data and keys stay with you

An API key is the password your code uses to call a model provider. With judgekeeper, your key goes from your machine straight to your provider, and nowhere else.

• No server. judgekeeper runs where you run it. Nothing is sent to its makers.
• No key flag. No option takes a key. --api-key-env takes the name of a variable, never the key.
• Keys come only from environment variables, settings your shell or CI passes to programs, such as ANTHROPIC_API_KEY.
• A test checks it. judgekeeper's own test suite checks that no key appears in any file or message it writes.
• Error messages are scrubbed. Anything that looks like a key is replaced with [REDACTED] before it is printed.

Step-by-step key setup for your laptop, GitHub Actions and cloud sessions