[WIP] Test repair agent: Part1#6375
Conversation
| "mcp__firefox__evaluate_testcase", | ||
| "mcp__firefox__evaluate_js_shell", |
There was a problem hiding this comment.
These two likely won't be useful for this agent, they are not running the same test cases we run in CI.
If we want the agent to be able to run Firefox tests, we either need to add other tools, or we just let it discover by itself how to run them using bash and the AGENTS.md/CLAUDE.md in the Firefox repo.
Take into account many test failures are also platform specific, so for example we can't reproduce a win or mac failure if we are only running on linux (unless we run the tests on taskcluster)
There was a problem hiding this comment.
I see, I wasn't sure about these either
| Treat this as a genuine regression: a commit that landed since the test was last | ||
| green introduced the failure. | ||
|
|
||
| Failing test groups (manifests) and a representative failing test in each: |
There was a problem hiding this comment.
Why not passing all failing tests?
| - analysis.md: the detailed reasoning, with evidence from the logs and diffs. | ||
| - verdict.json: an object with keys "culprit_commit" (a full sha from the | ||
| candidates above, or null if none is convincing), "culprit_bug" (integer or | ||
| null), "recommendation" ("backout" or "land_fix") and "confidence" |
There was a problem hiding this comment.
Should we also have a recommendation to "rerun" in case the agent is not sure it is actually a real failure? Or just rely on confidence being low?
There was a problem hiding this comment.
That's an interesting idea
| return None | ||
|
|
||
|
|
||
| def _last_green(branch: str, rev: str, group: str) -> str | None: |
There was a problem hiding this comment.
We might want to do this at the test level instead of the group level, also we need to take into account the platform where the test run (e.g. if the test is only failing on linux, we risk last_green to find a commit where the test run and passed on win but didn't run on linux)
2 participants
Tested locally https://wandb.ai/moz-bugbug/hackbot-test/weave/agents/conversations?filters[agent_name]=test-repair