Check your
LLM-as-a-judge
in 5 steps.
Question
Can I return a lamp after 60 days?
0 of 4 marked
Answer
Of course! We accept returns at any time, no questions asked.
What each step looks like.
$ judgekeeper judgekeeper 1.1.2 Step 1 of 5: Find Eval tool: promptfoo File: results.json Answers: 40 answers LLM-as-a-judge: my-judge-v1 Rule: "The answer solves the customer's problem and is polite." judgekeeper has read from your file: question each test's vars answer each test's output grade your LLM-as-a-judge's Pass or Fail Your LLM-as-a-judge passed 23 answers and failed 17. Checked: your LLM-as-a-judge gave every answer a clear Pass or Fail. Step 2 of 5: Mark A result needs at least 15 Pass and 15 Fail marks from you. It also needs 20 marks on answers your LLM-as-a-judge passed and 20 on answers it failed (or all of them, if there are fewer). Opening the marking page in your browser… The marking page is open in your browser. It shows the answers one at a time. Mark each one Pass or Fail, the way you want your LLM-as-a-judge to decide. Your LLM-as-a-judge's grade stays hidden. Next: mark answers on the page until a "See the numbers so far" link appears, then click it.
Example with made-up answerssee its report
$ judgekeeper
judgekeeper 1.1.2
Step 1 of 5: Find
Eval tool: promptfoo
File: results.json
Answers: 40 answers
LLM-as-a-judge: my-judge-v1
Rule: "The answer solves the customer's problem and is polite."
judgekeeper has read from your file:
question each test's vars
answer each test's output
grade your LLM-as-a-judge's Pass or Fail
Your LLM-as-a-judge passed 23 answers and failed 17.
Checked: your LLM-as-a-judge gave every answer a clear Pass or
Fail.
Step 2 of 5: Mark
A result needs at least 15 Pass and 15 Fail marks from you. It
also needs 20 marks on answers your LLM-as-a-judge passed and 20
on answers it failed (or all of them, if there are fewer).
Opening the marking page in your browser…
The marking page is open in your browser. It shows the answers
one at a time. Mark each one Pass or Fail, the way you want your
LLM-as-a-judge to decide. Your LLM-as-a-judge's grade stays
hidden.
Next: mark answers on the page until a "See the numbers so far"
link appears, then click it.
$ judgekeeper
judgekeeper 1.1.2
Step 2 of 5: Mark
You marked 12 answers so far: 8 Pass, 4 Fail.
That's not enough for a result yet.
A result needs at least 15 Pass and 15 Fail marks from you. It
also needs 20 marks on answers your LLM-as-a-judge passed and 20
on answers it failed (or all of them, if there are fewer).
The marking page is still open in your browser. The bar at the
top fills as you mark.
Next: mark answers on the page until a "See the numbers so far"
link appears, then click it.
$ judgekeeper
judgekeeper 1.1.2
Step 3 of 5: Measure
Result: your LLM-as-a-judge needs work.
You and your LLM-as-a-judge disagreed on 16 of 37 answers.
You passed 20 answers. Of those, your LLM-as-a-judge passed 12
and failed 8.
You failed 17 answers. Of those, your LLM-as-a-judge passed 8
and failed 9.
To tell judgekeeper why you disagree on 8 of them, run
`judgekeeper mark`.
`judgekeeper fix` uses your reasons to find where your
LLM-as-a-judge goes wrong.
8 more disagreements are kept aside on purpose, so a new rule
can be tested on answers you haven't looked at again.
How often your LLM-as-a-judge agrees with you (estimates):
When you say Pass, it also says Pass: about 63% (between 52% and 70%)
When you say Fail, it also says Fail: about 49% (between 38% and 66%)
The result line uses wider ranges than these, so a lucky streak
can't decide it. You can look after every mark.
Kappa (agreement beyond coincidence): about 0.13 (between -0.09
and 0.33). It never decides the result.
Full report: .judgekeeper/report.html (open it in your browser)
The marking page is still open. Mark more answers to make these
ranges narrower, so the numbers are more certain.
Next: to improve your LLM-as-a-judge's rule, run
`judgekeeper fix`.
$ judgekeeper fix
judgekeeper 1.1.2
Step 4 of 5: Fix
Where your LLM-as-a-judge goes wrong:
Your LLM-as-a-judge made 8 mistakes on the 18 marks used here.
The other 19 marks are kept aside to test a new rule. In 4, it
said Pass where you said Fail. In the other 4, it said Fail
where you said Pass.
When it said Pass and you said Fail, the answers are long
(usually about 12 words, while answers you both failed have
about 8).
All 8 mistakes your LLM-as-a-judge made are saved in
.judgekeeper/fix/mistakes.md. Use it to write a better rule, or
hand it to a coding agent to help you write one.
Your LLM-as-a-judge's rule is in promptfooconfig.yaml, line 14:
"The answer solves the customer's problem and is polite."
.judgekeeper/fix/marked-answers.csv holds all 37 marked answers,
questions and answers only, so you can't tell which are kept
aside.
promptfoo: grade marked-answers.csv again with your new rule,
with the `echo` provider and no new answers.
https://github.com/judgekeeper/judgekeeper/blob/main/docs/eval-tools.md
shows the whole config.
Next: change your LLM-as-a-judge's rule in your eval tool, grade
the answers in .judgekeeper/fix/marked-answers.csv again, save
the results, then run `judgekeeper fix <new results file>`.
$ judgekeeper fix new-rule.json
judgekeeper 1.1.2
Step 4 of 5: Fix (testing your new rule)
New results: new-rule.json
Your LLM-as-a-judge, tested on the 19 marks kept aside:
Old rule:
When you say Pass: agrees on 6 of 10
When you say Fail: agrees on 5 of 9
New rule:
When you say Pass: agrees on 10 of 10
When you say Fail: agrees on 9 of 9
With the new rule, 8 answers went from wrong to right, and
none from right to wrong.
Result: your new rule is better. A difference this large is
unlikely to be a coincidence.
Next: copy your new rule into promptfooconfig.yaml (line 14),
then run `judgekeeper check new-rule.json` to re-check it on all
your marks.
$ judgekeeper check
judgekeeper 1.1.2
Step 5 of 5: Re-check
Results: results.json
Your LLM-as-a-judge's model changed:
my-judge-v1 -> my-judge-v2
Compared with before: worse. Your new LLM-as-a-judge agrees with
you less on the answers you passed: 6 of 20, down from 12.
Agreed with you: Old LLM-as-a-judge New LLM-as-a-judge
When you said Pass 12 of 20 6 of 20 Worse
When you said Fail 9 of 17 9 of 17 No big drop
On answers you pass, agreement drops by about 30 percentage
points (somewhere between 15 and 49).
On answers you fail, no drop, though these marks can't rule out
a drop of up to 19 points.
One run, so a small change may be a coincidence.
Next: go back to your old LLM-as-a-judge, or try another change
and run `judgekeeper check` again.
Have eval results?
Check your LLM-as-a-judge. Your file stays on your computer.