AI Driven Code Reviews - Optimisation

We’ve been using AI for our code reviews for a couple of months now and we are seeing a definite improvement in the quality of code.

Average number of comments per review

Since the agent is run as a gate before a pull request is created, the number of comments on PRs has steadily decreased.

Note: There is another contributor to this trend. These days, I incorporate more changes myself rather than ask the developer to change. The flip side of this is that by doing the changes myself, I’m depriving developers of the opportunity to learn.

Average review time

You would think that the average review time will come down but that is NOT the case. If anything, I spend the same amount of time reviewing as before. It is only that I look for deeper patterns now.

AI reviewer optimisation

While the quality of the reviews has steadily increased, we have also made changes to optimise the process started trying to conserve token usage. We’ve done this using the following principles

• Divide the review into several different phases
  • Each phase has its own context & output
• Give clear instructions to the AI on what commands to run (to reduce the need for it to think or interpret instructions)
• The final outcome is intentionally terse & created by concatenating (again using commands) the output of the previous phases.

Next Steps

I plan to add metrics around token usage. Claude, for instance, makes it easy to track token usage per-conversation.

Earlier posts on this topic can be found here. The charts were generated using the here