Job 03 · Not yet tested
Which tool reviews my PRs and catches real bugs?
Reviewers that comment on a pull request. The pack is one diff with five seeded bugs. We count catches and false positives.
People who want a reviewer that finds defects, not a style bot.
The standard test
Pack 0.1.0 · specified · Pull request with five seeded bugs
PR with 5 seeded bugs; caught vs false positives
Setup
The same diff for every tool. The five bugs are behaviour defects planted on purpose. Style nits do not count as catches. A comment that flags a non-bug is a false positive. The reviewer does not get the answer key.
Inputs
The pull request diff and the repository it belongs to. The bug list stays with the human reviewer.
Time limit
One wall-clock cap for every tool in the job. The number is published with the pack before the first run and does not move between tools.
What we record
- Bugs caught
- Bugs missed
- False positives
- Time
- Cost
- Tool version
- Plan
How the FitScore is built
FitScore = weighted average × 20, rounded to the nearest integer. The weighted average is the sum of (criterion score × weight) divided by 100.
- Output quality30
- Reliability15
- Cost and free tier15
- Speed10
- Setup friction10
- Agent, API, and MCP support10
- Privacy and data10
5 excellent · 4 good · 3 usable · 2 poor · 1 fails. Full method.
Candidates
Candidates for AI code review. This order is the launch list, not a ranking. No FitScores are published.
Version and plan recorded when a test is published
Version and plan recorded when a test is published
Version and plan recorded when a test is published
Version and plan recorded when a test is published
Version and plan recorded when a test is published
Version and plan recorded when a test is published
Questions this page answers
- Which tool reviews my PRs and catches real bugs?
- FitForTool has not published a FitScore for this job. The launch candidates are CodeRabbit, GitHub Copilot code review, Greptile, Qodo, Cursor Bugbot, and Graphite Diamond. Each one is listed as not yet tested. A ranking is published only after every tool in the comparison has had the same test pack run, and a person has reviewed the result. Until that page shows a FitScore, there is no winner.
- How will FitForTool test ai code review?
- PR with 5 seeded bugs; caught vs false positives. The same diff for every tool. The five bugs are behaviour defects planted on purpose. Style nits do not count as catches. A comment that flags a non-bug is a false positive. The reviewer does not get the answer key. We record bugs caught, bugs missed, false positives, time, cost, tool version, and plan. Those notes are then scored from 1 to 5 on seven shared criteria. The pack has been specified and has not been run.
- Can a vendor pay to rank higher on this job?
- No. FitForTool does not sell position or score. A sponsorship, if we run one later, is labelled and sits outside the ranked list. An affiliate link is disclosed and does not change order. A vendor may ask for a retest, and we rerun the same pack. Sponsors do not see the result before it is public.
- What does "Not yet tested" mean?
- The tool is a candidate for this job. There is no published hands-on run, so there is no FitScore, no criterion score, and no rank. Research-only is a different label, used when we have read the docs and still have not run the pack. Neither label is a score.