AI Code Review Tool — CodeRabbit Replaces Me And I Like It

In this particular case I have to say I’m really glad AI is replacing me, because I can focus on moving things 1px to the left and right. The real coding work!

It’s been a while since I wrote about CodeRabbit. I found that tool in March 2024, and using it still for my projects. You know that I like productivity tools, and CodeRabbit is one of the ones that helps me with that.

An interesting problem with AI in general is how AI can learn from what we do. It is the major problem faced by AI developers, because even if they process all the available data, still AI has to understand what you’re doing.

One option is to train whole AI on everything including your codebase. But that’s time consuming. Another option is to train separate AI model on your codebase and make two AI’s communicate. Other option is to create context prompts based on things that AI learns. It means that when you AI has to learn something, it is processed into a form AI understands, and than that prompt is executed on start of the system.

There are many ways to tackle that issue, and everyone tries to figure out how to do it. If it comes to open source, local tools, it’s still a missing piece. It’s not trivial to make local model to learn from your code and changes.

Therefore I was quite glad CodeRabbit offers that feature for AI code reviews. It learns about the code base but also all code reviews, interactions and so on.

Learning Changes To The Coding Guidelines

Look for example at this code review:

CodeRabbit suggests to use asterisk import because it found there’s a pattern in how imports are used. If there’s more than four imports asterisk import was used instead.

But let’s say the rules changed, and developers decided that maybe it’s not a good idea to use asterisk imports:

So the developer tells CodeRabbit about it. What happens is it answers with a comment summarizing how it understands it, you can also unfold new learning and learning removed.

AI is able to instantly learn the new rule.

I like that the details are displayed as well, because if AI will understand it incorrectly it can be improved further. For example we could tell CodeRabbit that the rule applies to all projects, not only “The Artemis”.

At first it looked to me like more work has to be done. But I’ve observed that once the model is fine-tuned it really starts to replace me in code reviews.

Well, I have mixed feelings about where it’s going :) But repeating the same thing again and again to all developers all the time is something I can live without :)

Context Of The Whole Project

I love also that CodeRabbit is able to process whole project. It doesn’t work only on that one specific file that's changed. For example:

CodeRabbit noticed that the simulation.loadbalancing should be tested against possible values. If it’s set to a value outside of the allowed ones, the code should throw an error.

But the developer notices that the validation is done already in simulation.py file:

So locally CodeRabbit is right, but globally is wrong. After receiving the comment CodeRabbit actually recognizes that multi-file scheme going through the simulation.py file. It’s not like it believes everything Philip says, it has to check it:

Based on it it learns something new:

User: PhilipDeegan
PR: PhilipDeegan/PHARE#87
File: pyphare/pyphare/pharein/__init__.py:177-177
Timestamp: 2024-01-16T16:42:31.598Z

Learning: The user has clarified that the
validation for the `loadbalancing` parameter
is handled within the `simulation.py` file
through the `check_loadbalancing` function,
which checks for supported types.

So it combined the project knowledge with Philip’s comment into a specific and precise learning it will use immediately for future code.

I imagine if Philip was wrong, CodeRabbit would say something like:

No Philip, you are not right. Simulation.py doesn’t contain validation. I’m creating a ticket for you right now

:D

But I didn’t face such situation, I’ll share if I will.

So that’s it. If you find it interesting check out CodeRabbit. There’s a free version and also free trial for higher tiers. It’s not that expensive anyways, $12 per month. What I like the most is that the Pro version is free for open source projects.