What changed

GitHub Copilot Evals can create repeatable tests for skills, custom agents and MCP tools, require approval before each run, show passed and failed checks and compare behavior before and after a customization change. It uses the extensible local Vally engine.

Why it matters

Teams gain a productized regression loop for agent customizations rather than relying on one successful prompt, but local execution still needs trustworthy tasks, graders and environment isolation.

What to do

Create one eval around a denied MCP action and one around an expected repository change, run each repeatedly before and after a customization, and inspect both final state and trajectory rather than accepting aggregate pass rates alone.

Caveat

The product is in public preview and GitHub does not claim that generated checks or local Vally scores predict production outcomes. AI variability means small run counts can mislead.