Setup and API keys

On your laptop

1. Create a key just for judgekeeper

   In your provider's console, make a new key and give it a low spending limit. If it ever leaks, little can be spent and you can delete it without breaking anything else.

2. Put it in your shell profile

   Your shell profile is a file your terminal reads when it starts: ~/.zshrc on a Mac, usually ~/.bashrc on Linux. Open it in a text editor and add one line, with your key in place of the dots:

   export ANTHROPIC_API_KEY=...

   For OpenAI the name is OPENAI_API_KEY. This file lives in your home folder, outside any project, so it never ends up in a repository.

3. Open a new terminal and check

   The profile is read when a terminal starts. This prints set if the key is there, without showing the key:

   echo ${ANTHROPIC_API_KEY:+set}

4. Run a judge

   judgekeeper judge anchors.jsonl --runner anthropic --model claude-haiku-4-5-20251001 --prompt prompts/single.md --runs 3 --out runs/haiku/

A key under another name

Any variable name works. Tell judgekeeper the name with --api-key-env. It takes a name, never the key itself:

export OPENROUTER_API_KEY=...
judgekeeper judge anchors.jsonl --runner openai --model "$JUDGE_MODEL" --base-url https://openrouter.ai/api/v1 --api-key-env OPENROUTER_API_KEY --prompt prompts/single.md --runs 3 --out runs/openrouter/

Other providers and local models

--base-url sends the requests to another address instead of the provider's default. Use --runner openai for any server that speaks the OpenAI format:

judgekeeper judge anchors.jsonl --runner openai --model "$JUDGE_MODEL" --base-url http://localhost:11434/v1 --prompt prompts/single.md --runs 3 --out runs/local/

The address goes into the judge's fingerprint. The same model behind a different address counts as a different judge.

In GitHub Actions

1. Store the key as a repository secret

   On GitHub, open your repository, then Settings, Secrets and variables, Actions, New repository secret. Name it ANTHROPIC_API_KEY and paste the key there. GitHub hides it in logs.

2. Pass it as env, never as an input

   The workflow hands the secret to the step as an environment variable. The judgekeeper action has no input for a key, on purpose.

3. Copy the example workflow

   Save this as .github/workflows/judge-gate.yml in your repository and change the paths. It runs every Monday, to catch a silent model update, and on pull requests that change the judge. It is the same file as docs/examples/workflows/judge-gate.yml.

   # Copy to .github/workflows/judge-gate.yml in your repo and adjust the paths.
   # Before the first run: `judgekeeper freeze evals/anchors.jsonl`, judge and validate once,
   # then `judgekeeper baseline set reports/report.json` and commit .judgekeeper/baseline.json.
   name: judge gate
   
   on:
     # Weekly: the provider can change what a model id serves without any change in your repo.
     # A scheduled run catches judge drift (a new snapshot, a silent update) between PRs.
     schedule:
       - cron: "17 6 * * 1"  # Mondays 06:17 UTC
     # Pull requests that change the judge itself: its prompt, the anchor set it is measured
     # against, the committed baseline or the gate thresholds. Other PRs cannot move the judge,
     # so they do not pay for judge calls.
     pull_request:
       paths:
         - "prompts/judge.md"
         - "evals/anchors.jsonl"
         - "evals/anchors.manifest.json"
         - ".judgekeeper/baseline.json"
         - "judgekeeper.toml"
   
   permissions:
     contents: read
   
   jobs:
     gate:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v4
         - uses: judgekeeper/judgekeeper@main  # pin to a release tag or commit sha
           env:
             # Keys come from repository secrets, never from action inputs.
             ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
           with:
             anchors: evals/anchors.jsonl
             runner: anthropic
             model: claude-haiku-4-5-20251001
             prompt: prompts/judge.md
             runs: 3
             flaky-as: pass  # FLAKY is reported in the summary but does not fail the build
   
     # Weekly only: the gate job above has just re-judged the frozen anchor set. Compare those
     # verdicts item by item with the baseline's to tell judge drift from a change in your system,
     # even when the declared fingerprint is identical (a silent provider-side update).
     # The baseline must carry per-item verdicts (`items` in report.json, schema_version 2): if it
     # predates them, re-run `validate` on the baseline runs and `baseline set` again.
     attribute:
       needs: gate
       if: always() && github.event_name == 'schedule'
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v4
         - uses: actions/setup-python@v5
           with:
             python-version: "3.12"
         # pin @main to a release tag or commit sha, as for the action above
         - run: pip install "judgekeeper @ git+https://github.com/judgekeeper/judgekeeper@main"
         - uses: actions/download-artifact@v4
           with:
             name: judgekeeper-gate  # the gate action's artifact-name
             path: judgekeeper-out
         - name: Attribute
           run: |
             set +e
             judgekeeper attribute judgekeeper-out/report.json --baseline .judgekeeper/baseline.json
             code=$?
             cat judgekeeper-out/attribution.md >> "$GITHUB_STEP_SUMMARY"
             exit $code  # 0 STABLE, 6 JUDGE_DRIFT, 7 SYSTEM_CHANGE, 2 usage, 3 anchors changed

In cloud coding sessions

Claude Code on the web runs your session on a machine in the cloud. Setting a key there takes three steps:

1. Use the Environment variables box

   Open the environment's settings and add the key in the Environment variables box, under a name of your own:

   JUDGEKEEPER_ANTHROPIC_KEY=...

   Do not use ANTHROPIC_API_KEY there: the session itself uses Anthropic's variables.

2. Point judgekeeper at that name and at Anthropic's address

   Those machines set ANTHROPIC_BASE_URL to an internal proxy, so pass Anthropic's own address too:

   judgekeeper judge anchors.jsonl --runner anthropic --model claude-haiku-4-5-20251001 --prompt prompts/single.md --api-key-env JUDGEKEEPER_ANTHROPIC_KEY --base-url https://api.anthropic.com --runs 3 --out runs/haiku/

3. Start a new session

   Changes to the environment apply to new sessions only. A session that was already running does not see the key.

Where a key must never go

If a key leaks, delete it in your provider's console straight away and make a new one.

Good habits

• Use a dedicated key for judgekeeper, with a low spending limit.
• Count the calls before a run: items times runs, and twice that for two-answer comparisons. judgekeeper asks you to confirm with --yes above 1,000 calls for your own judge functions.
• judgekeeper replaces anything that looks like a key with [REDACTED] in everything it prints or writes. That is a safety net, not a reason to paste keys anywhere.