With the increased use of AI coding tools, codebases have become more prone to vulnerabilities, which increased the need for effective code reviews. To address this, we introduce RevEval (AI Code Review Eval), which benchmarks the top four AI code review tools across 309 pull requests from repositories of varying sizes and evaluates their performance using input from 10 developers and an LLM-as-a-judge.
Benchmark Results
CodeRabbit ranked as the most successful code review tool across 51% of 309 PRs:
To measure the ranking we used the LLM-as-a-judge scores. We examined which AI code review tool achieved the highest score in each PR (scored using our LLM-as-a-judge), and then calculated the percentage of all PRs in which each tool ranked first.
CodeRabbit scored highest in both manual human evaluations and LLM-as-a-judge evaluations followed by Greptile and GitHub Copilot:
When calculating the average score, all three evaluation categories were weighted equally. Large repository scores and small repository scores were evaluated by LLM-as-a-judge, and developer evaluations were manually completed to double-check LLM-as-a-judge scores.
Human evaluations
We asked the developers who participated in the evaluations which AI code review tool they would prefer to integrate into their workflows. Since CTOs play a key decision-making role in software development, we highlighted their responses in a separate chart:
Detailed comparison
We calculated the average number of bugs per PR by counting all bugs/issues reported by each code review tool and dividing by the total number of PRs (309). Not all PRs in our codebase contain bugs or issues. GitHub Copilot does not explicitly report when it detects a bug in a PR; therefore, it was excluded from this comparison.
You can see our methodology below.
Features
* It is provided by CodeRabbit’s “agentic pre-merge checks” feature. It automatically validates pull requests against quality standards and custom organizational requirements before merging, and returns pass/fail results with explanations directly in the PR walkthrough. Each check can be configured to either warn developers or block merges entirely. While GitHub Copilot, Cursor BugBot, and Greptile provide PR review features, they function as advisory systems that offer feedback and suggestions rather than systematic validation frameworks.
** Cursor and GitHub Copilot may offer more capabilities beyond their code-review components; only the features of Cursor Bugbot and GitHub Copilot Code Review are included in our comparison.
Features vary depending on subscription plans, so some features that are marked as available above may not be available in your subscription.
In automated code reviews, CodeRabbit, GitHub Copilot and Cursor Bugbot were easier to configure than Greptile because automated code reviews cannot be enabled for an empty repository in Greptile.
Feature deep-dive
CodeRabbit
- 40+ built-in linters and security scanners.
- AST pattern-based custom instructions.
- Adapts from developer feedback over time.
- Developers can tag @coderabbitai to ask follow-ups, request fixes, question recommendations.
- Supports custom MCP servers for additional context.
GitHub Copilot Code Review
- “Implement suggestion” button hands off to Copilot coding agent.
- Tight integration with GitHub ecosystem.
- Custom instructions via copilot-instructions.md.
Greptile
- Learns team’s coding standards from PR comment history.
- With pattern repos developers can reference related repos in greptile.json so they may provide with additional context.
- Developers can reply with @greptileai for follow-up questions or fix suggestions.
- Greptile learns from thumbs up/down feedbacks.
- Sequence diagrams auto-generated for all PRs.
Cursor BugBot
- After a bug is identified by BugBot, developers can use “Fix in Cursor” button to quickly open the Cursor to fix the Bug.
- Developers can customize their code review rules in BUGBOT.md files.
We also intended to benchmark Graphite; however, due to a bug in their dashboard, we were unable to enable automated code reviews for new repositories. We contacted their support team on October 25, 2025, but the response did not resolve the issue. Despite follow-up emails and a message in their Slack channel, the problem remained unresolved.
Components and integrations
* All of these solutions support GitHub.
Methodology
We created separate benchmark repositories for each tool within our dedicated GitHub organization.
After enabling automatic code reviews for each tool in its assigned repository, we opened pull requests in sequence, waited for the tool to complete its review, and then closed the PRs to record the results. We did not modify or tune any tool settings. Each tool was evaluated using its default configuration, exactly as installed.
Our workflow begins by cloning the source repository as it existed on a selected baseline date, then replaying the pull requests submitted after that date one by one, preserving the original repository structure.
We used the November 2025 versions of all products. Our benchmark consisted of 2 different ranges of source repositories:
1. Well-known, medium-large size repositories
We aimed to see how well AI code review tools understand repositories with large and complex structures. We have 289 PRs reviewed overall across 7 repositories.
2. Small and new repositories
We are aware that we cannot feed our LLM-as-a-judge with the
whole repository in the large repositories, since their context windows are not sufficient for that. Therefore, to overcome this, we also evaluated the first 3-5 PRs of new and small repositories. MCP servers fit our needs perfectly. Consequently, we chose 8 official MCP servers and had 20 PRs reviewed on them.
Our dataset contains code written by experienced developers. We did not evaluate performance on fully AI-generated codebases.
Developer Evaluations
We randomly selected 35 PRs and assigned them to 10 developers, with each PR being evaluated 5 times by developers. Our aim in repeating the evaluation was to minimise developers’ bias. Developers assessed the results in a vendor-agnostic way.
Most of them reached the same high-level insights:
- CodeRabbit’s detailed reviews are helpful, and it is successful in bug detection.
- Greptile provided successful summaries, but the sequence diagrams it generated are not necessary for some PRs.
Figure 1: Example sequence diagram provided by Greptile. Greptile generates the diagrams for every PR.1
- GitHub Copilot is very successful at finding typos in code and makes spot-on suggestions; its analysis is shorter than those of CodeRabbit and Greptile.
- Cursor Bugbot provides less detailed and less accurate analysis.
After the evaluations, they also stated that they will start using them in their own repositories as a support tool for developers.
LLM-as-a-Judge
We used GPT-5 to evaluate the reviews. After the evaluation, we used GPT-4o to structure the output in JSON format.
Our evaluation workflow includes:
- For large repositories: The original PR body, diff, and comments/reviews from the tools.
- For small repositories: Whole codebase, the original PR body, diff, and comments/reviews from the tools.
Here is the full prompt we used:
Evaluate each tool on these dimensions (scale 1-5):
1. Correctness
Are identified issues actually real problems/bugs/fixes in the code?
– 5 (Excellent): All identified issues are real problems
– 4 (Good): Most issues are real, minor misidentifications
– 3 (Acceptable): Mix of real and questionable issues
– 2 (Poor): Most identified issues are not actual problems
– 1 (Failed): Cannot identify real issues, all findings are incorrect
2. Completeness
Did it catch important issues? How comprehensive is the review?
– 5 (Excellent): Catches all critical issues and most important ones.
– 4 (Good): Catches major issues, misses some minor ones
– 3 (Acceptable): Catches some important issues but has notable gaps
– 2 (Poor): Misses several critical issues
– 1 (Failed): Misses all or nearly all critical issues
3. Actionability
Are suggestions clear and implementable? Does it include patches/fixes? If no bugs in the code, write “null” to actionability to all the tools, do not give any scores to any tools for that PR.
– 5 (Excellent): All suggestions include clear patches/fixes and are directly implementable
– 4 (Good): Most suggestions have clear guidance, some include patches
– 3 (Acceptable): Suggestions are somewhat clear but lack patches for some issues
– 2 (Poor): Suggestions are mostly unclear or not implementable
– 1 (Failed): No clear suggestions or guidance provided
4. Depth
Does it show understanding of the code’s logic and purpose?
– 5 (Excellent): Demonstrates deep understanding of code logic, architecture, and purpose
– 4 (Good): Shows good understanding with minor gaps
– 3 (Acceptable): Surface-level understanding, misses some context
– 2 (Poor): Shallow or incorrect explanations of code behavior
– 1 (Failed): No understanding of the code’s logic and purpose
Output Format
For each tool, provide:
1. Detailed reasoning: What did it find? Did it miss important issues? Patches included? Deep understanding of the codebase? Specific examples.
2. Individual scores (1-5 for each dimension, using the scaling above)
Example Output
Tool A:
Reasoning: Tool A demonstrated excellent correctness by identifying a real memory leak in the connection pooling logic at line 145, providing a specific patch using a context manager. It also caught the missing error handling in the API endpoint with actionable code. The completeness score reflects that while it found major issues, it missed the race condition in the async handler that could cause production problems. All 4 comments were substantive and directly implementable. The depth was strong, showing an understanding of the resource management patterns and error propagation in the codebase.
Correctness: 5
Completeness: 4
Actionability: 5
Depth: 4
Tool B:
Reasoning: Tool B correctly identified the input validation vulnerability on line 89 and provided a clear fix using parameter sanitization. However, completeness suffered significantly as it missed the critical security vulnerability in the authentication flow that allows token reuse. The actionability was mostly good – suggestions included code snippets. The depth was acceptable but superficial, focusing on surface-level checks rather than understanding the security model or data flow implications.
Correctness: 4
Completeness: 1
Actionability: 4
Depth: 2
Tools to evaluate: CodeRabbit, Cursor Bugbot, Github Copilot, Greptile
Be objective and thorough. Use specific examples from the reviews to support your scores.
What is AI code review?
AI code review is the automated analysis of source code using machine learning models, primarily large language models (LLMs), to identify bugs, inefficiencies, and potential vulnerabilities. In addition to detecting issues, these systems can provide context-aware explanations, suggest concrete fixes, and generate patches that help developers improve both code quality and maintainability. Many AI review tools also assist with documentation by summarizing changes and producing descriptive comments or explanations for newly added code.
Because AI models can evaluate code rapidly and at scale, they significantly accelerate the review process and make it easier to catch issues early while maintaining consistent coding standards across large or fast-moving projects.
In modern AI-assisted development environments such as Cursor or Claude Code, developers may unintentionally lose track of how their codebase evolves when “vibe coding” or relying heavily on auto-generated suggestions. This can introduce hidden vulnerabilities or logical inconsistencies. AI code review tools help mitigate these risks by providing an additional layer of structured and systematic analysis to validate and improve AI-generated code.
Benefits of AI code review
Efficiency and speed
AI code review tools can analyze code in real time, providing immediate feedback and flagging potential issues as developers work. They are capable of detecting errors and security vulnerabilities that human reviewers may overlook, particularly in large or rapidly evolving codebases. By automating routine checks, these tools allow developers to concentrate on higher-level reasoning, complex problem solving, and architectural decisions.
Improved code quality
AI code review tools help maintain consistent coding standards across teams by identifying stylistic inconsistencies and deviations from best practices. They also offer detailed feedback and recommendations on a wide range of coding issues, from minor improvements to significant bugs. Over time, developers can learn from this feedback, refine their coding habits, and adopt new techniques that strengthen the overall quality of their work.
Limitations and challenges
Over-reliance on AI tools
A common concern with AI code review is excessive dependence on automated feedback. Although AI can be a valuable source of insight, it should not be treated as a complete substitute for human expertise. Automated reviews can accelerate workflows, but human reviewers remain essential for ensuring correctness, context awareness, and alignment with project goals. In our benchmark, developers consistently stated that they would not rely on these tools blindly. They viewed them as assistants that supplement human judgment rather than replace it.
Managing false positives and false negatives
False positives occur when the tool incorrectly identifies working code as problematic, while false negatives occur when genuine issues are missed. In our evaluation, the most significant concern was false negatives. The tools were more likely to overlook important issues than to raise incorrect warnings. This highlights the need for continuous improvement in the underlying models and algorithms.
To address these challenges, AI code review tools must evolve through better training, enhanced context handling, and more accurate reasoning capabilities.
Best practices for using AI code reviews
Tips from experts
Pair AI reviews with human insights: Use AI code reviews alongside human reviews to ensure that the code is both technically sound and aligns with project goals.
Customize rules to fit your project: Adjust the AI tool’s rules to match your project’s coding standards to reduce unnecessary alerts.
Use AI feedback as a learning tool: Treat AI suggestions as a way to learn and improve, discussing them with your team to understand why and how to avoid similar issues in the future.
Acknowledgements
We extend our sincere gratitude to the developers who contributed their time and expertise to perform the manual evaluations:
Aziz Durmaz (CTO at a transportation and logistics company)
Berk Kalelioğlu (co-founder at a game development studio)
Elif Ece Örnek (software engineer at a travel website)
Haydar Külekçi (consultant at search technologies & AI company)
Mehmet Şirin Can (head of development at AIMultiple)
Mehmet Korkmaz (CTO at a media company in e-sports and video games industry)
Murat Orno (former CTO at a regional payment platform with 500+ employees)
Orçun Candan (full-stack developer at AIMultiple)
Yalçın Börlü (senior software engineer at a health and wellness company)
Yiğit Dinç (co-founder of a legal tech company)
We also thank the developers and maintainers of the open-source repositories included in our benchmark for their work and valuable contributions to the community.
Anonymization of original developer identities
To conduct the benchmark responsibly, we anonymized all original developer names and email addresses when replaying pull requests from upstream repositories. Because the benchmark repositories are public, preserving original author information could unintentionally expose personal data and create the risk of notifying developers each time a recreated pull request is opened or updated. Although GitHub does not typically notify authors when their commits are replayed in a separate repository, we considered it best practice to avoid any possibility of unwanted notifications, attribution issues, or privacy concerns.
Anonymization ensures that:
- Developers are not disturbed by thousands of automated PR events.
- Personal information is not republished in a different public repository.
- Benchmarks remain unbiased, preventing tools or LLM judges from being influenced by recognizable author names.
- Ethical and privacy standards are maintained when working with open-source contributions.
Only identity metadata was altered; all code, diffs, commit ordering, and file structures were preserved exactly to maintain the authenticity and reproducibility of the benchmark.
Reference Links
Cem's work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE and NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and resources that referenced AIMultiple.
Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.
He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem's work in Hypatos was covered by leading technology publications like TechCrunch and Business Insider.
Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.
Be the first to comment
Your email address will not be published. All fields are required.