LLM-as-a-judge:my-judge-v1Eval tool:promptfoo
judgekeeper™

Your LLM-as-a-judge needs work.

You and your LLM-as-a-judge disagreed on 16 of 37 answers.

How often your LLM-as-a-judge agrees with you

When you say Pass, your LLM-as-a-judge also says Passabout 63%between 52% and 70%
When you say Fail, your LLM-as-a-judge also says Failabout 49%between 38% and 66%
KappaKappa is agreement beyond coincidence. 0 means no better than coincidence, 1 means perfect agreement.0.13between −0.09 and 0.33

You passed 20 answers. Of these, your LLM-as-a-judge passed 12 and failed 8.

You failed 17 answers. Of these, your LLM-as-a-judge passed 8 and failed 9.

You said Fail, your LLM-as-a-judge said Pass

4 more are kept aside to test your fix.

QuestionAnswerIts reason
Is the oak table solid wood?Yes, every table we sell is made of solid oak.Answers the question politely.

Is the rug washable?Yes, all our rugs can go in the washing machine on a hot wash.Answers the question, and is friendly.

My mug arrived chipped. What now?Sorry about that! Just throw it away and order a new one. It should arrive fine this time.Says sorry and gives a clear next step.

You said Pass, your LLM-as-a-judge said Fail

4 more are kept aside to test your fix.

QuestionAnswerIts reason
Can I call you?We don't have a phone line, but the chat on our site replies within an hour.Doesn't answer the question asked.

Are your mugs microwave safe?Yes.One word: not polite enough.

Can I return a sale item?Yes, within 30 days, the same as full-price items.Doesn't thank the customer.

What next

Your marks and reasons are saved in this project's .judgekeeper folder. You can close this page.

Next, in your project folder:

judgekeeper fix

To tell judgekeeper why you disagree on 8 of them, run judgekeeper mark.