Agent Evaluation Infrastructure
Summary
End-to-end agent evaluation: containerized execution & judging engine, black-box CLI-agent evaluation service, and automated task/rubric generation with trustworthy calibration.
Problem
Tool-using, multi-turn agents can't be measured by static QA benchmarks; they need isolated real execution, reproducible judging, and solvable/discriminative/trustworthy tasks.
Solution
Built a containerized execution engine, Agent-as-Judge scoring, a two-container anti-cheating evaluation service, and automated task/rubric generation with triple calibration.
Impact
- Supported large-scale async concurrent evaluation across public and self-built agent benchmarks.
- Upgraded bulk task creation into a trustworthy evaluation-source production line.
Execution & Judging Engine
Isolated Docker sandbox runs CLI agents and collects artifacts/traces; Agent-as-Judge does multi-dim rubric scoring; asyncio Pipeline+Stage for concurrency.
Key Techniques
Trustworthy Calibration
Reverse-validate self-authored tasks: solvability (strong model passes) + discrimination (weak model fails) + rubric consistency (Cohen's kappa / bootstrap).