Skip to content

Add reusable LLM judge provider support for GPT-5.5 and DeepSeek V4 Pro - #9

Draft
taivu1998 wants to merge 1 commit into
marin-community:mainfrom
taivu1998:tdv/llm-judge-provider
Draft

Add reusable LLM judge provider support for GPT-5.5 and DeepSeek V4 Pro#9
taivu1998 wants to merge 1 commit into
marin-community:mainfrom
taivu1998:tdv/llm-judge-provider

Conversation

@taivu1998

Copy link
Copy Markdown

Summary

Adds a small reusable LLM judge provider layer for Evalchemy so new proof/rubric benchmarks can use GPT-5.5 and DeepSeek V4 Pro without copying provider-specific API code into each benchmark.

This PR adds:

  • eval.judges.JudgeConfig with provider/model presets, redacted config metadata, env-var validation, CLI/YAML helpers, and stable config hashing.
  • eval.judges.JudgeClient with OpenAI Responses API support for gpt-5.5 and OpenAI-compatible Chat Completions support for deepseek-v4-pro.
  • Shared JudgeRequest, JudgeResult, and ProofJudgeResult schemas for structured rubric/proof grading.
  • Evalchemy runner plumbing for judge CLI flags, nested YAML annotator: config, task constructor kwargs, output metadata, and resume fingerprints.
  • Focused offline tests for config resolution, provider request payloads, schema validation, resume fingerprint behavior, YAML/argparse wiring, and task-manager constructor filtering.

Design notes

  • Existing legacy annotator flows remain backward compatible. A flat legacy annotator_model such as gpt-4o-mini-2024-07-18 does not create a shared JudgeConfig or change resume metadata.
  • The new judge layer is opt-in via GPT-5.5, DeepSeek model names, or explicit reusable judge provider settings.
  • API keys are never stored in config metadata, output metadata, resume fingerprints, or tests; only env-var names are retained.
  • OpenAI Responses calls pass store=False and use strict structured-output JSON schema.
  • DeepSeek calls use response_format={"type":"json_object"} plus documented thinking-mode knobs through the OpenAI-compatible client.
  • No existing benchmark internals are migrated in this PR; new proof/rubric benchmarks can opt in by accepting judge_config.

Validation

  • python -m compileall -q eval/judges eval/eval.py eval/task.py eval/resume/wiring.py tests/judges
  • python -m pytest tests/judges -q -rs -> 23 passed
  • python -m pytest tests/resume/test_fingerprint_compute.py tests/resume/test_cli_wiring.py tests/test_passk.py -q -> 67 passed, 1 skipped
  • python -m pytest tests -q -> 126 passed, 6 skipped
  • git diff --check

No live provider calls were made; provider behavior is validated with mocked SDK clients and real request/response parsing code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant