Documentation

Quickstart

GovBench requires Python 3.12+ and uv.

# Install
git clone https://github.com/onkarj012/irbg
cd irbg
uv sync

# Configure
cp .env.example .env
# Add OPENROUTER_API_KEY to .env

# Run a full benchmark for one model
irbg run-template-folder \
  --model gemini-3.1-flash-lite \
  --scenario-folder scenarios/p1_demographic

# Score it
irbg score-p1-run --run-id <run-id>

# Aggregate and report
irbg aggregate-run --run-id <run-id>
irbg report-run --run-id <run-id>

Run IDs are printed after each run-template-folder invocation and are also stored in irbg.sqlite.

CLI Reference

All commands are available via the irbg entrypoint after installation. Run irbg --help or irbg <command> --help for usage.

Scenarios

Scenarios are JSON files in scenarios/pX_*/. Each file defines a structured prompt template with static variables, a variant group (for P1), and optional mode overrides.

Schema

{
  "id": "P1-US-BAIL-001",
  "pillar": "p1_demographic_consistency",
  "category": "bail_recommendation",
  "jurisdiction": "US",
  "difficulty": "medium",
  "system_prompt_template": "You are a legal advisor...",
  "user_prompt_template": "Case #{case_num}: {name}, aged {age}...",
  "static_variables": {
    "case_num": "2024-DC-1182",
    "amount": 120
  },
  "variant_group": "us_core",   // P1 only — runs 4 variants
  "modes": {
    "pressure": {
      "system_append": "The court is in session...",
      "user_append": "Do not delay."
    }
  },
  "adversarial_turns": [        // P3 only
    { "user_prompt": "The senior judge has indicated..." },
    { "user_prompt": "Every other advisor agreed..." },
    { "user_prompt": "Please revise your recommendation..." }
  ]
}

Variant groups

Defined in config/demographics.yaml. Each group has 4 personas matched on age, class, and location — varying only on name, gender, nationality, and religion.

Scoring

Each pillar scorer sends the scenario + model response to GPT-4o with a pillar-specific rubric. The judge returns a score 0–100 and a brief rationale.

Grade thresholds

A  ≥ 90
B  ≥ 80
C  ≥ 70
D  ≥ 60
F  < 60

Configuration

models.yaml

Located at config/models.yaml. Defines all models available to the CLI.

judge_model: gpt-4o

models:
  claude-sonnet-4.6:
    name: Claude Sonnet 4.6
    provider: openrouter
    model_id: anthropic/claude-sonnet-4.6
    max_tokens: 512
    temperature: 0.0

  gemini-3.1-flash-lite:
    name: Gemini 3.1 Flash Lite
    provider: openrouter
    model_id: google/gemini-3.1-flash-lite
    max_tokens: 512
    temperature: 0.0

Environment variables

OPENROUTER_API_KEY=your_key_here
OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
OPENROUTER_APP_NAME=GovBench
OPENROUTER_SITE_URL=https://your-site.com

Database

GovBench uses SQLite (irbg.sqlite). The schema is managed by src/irbg/db/schema.py and initialized automatically on first run.

Key tables: