Evaluating AI Performance — Charlene Gilbert

When I became Lead Software Engineer at Site Impact, I inherited the codebase one of our early successful uses of AI: a classification service that eliminated a manual process bottlenecking the workflow of a majority of our orders. At the time it was still in pilot mode and not actually fully integrated in our process. My team finally completed the integration into the order workflow and that success can be attributed to convincing stakeholders that wiring up the classifier service, confirming it gracefully handled errors, and ensuring that there was a way to recover from a botched classification attempt was a different problem than the actual AI classifications being produced.

For a couple of sprints I had to field questions from QAs and the Product Owner like:

Given this input, the AI returned the following classification but I disagree with it.
Given the same input multiple times, the AI returned different results.

The user story would be pushed back to devs and I would have to step in and remind the team that we were not in the business of testing whether the classification is correct, we were in the business of testing the classification process was working. The questions that needed to be satisfactorily answered were instead:

Given this input, did our system successfully receive and persist the AI's classification?
Given something went wrong with the classification service, was our system able to successfully escalate to a human?
Given the classification trigger, is our system allowing the classifier microservice to get the first attempt at classification, saving humans' time?

Now of course, everyone's concerns about the classification were valid. From the perspective of getting the service operational in production, it was an out of scope concern, but the same input producing multiple results was problematic. Additionally, the cost of a misclassification was significant enough that AI classifier accuracy was very important.

I set out planning a separate epic to address classification accuracy. I sketched out a few potential options:

Setting the temperature to 0, to reduce different classifications on the same input (with some caveats).
Fine-tuning the general purpose LLM we were using on our considerable corpus of human classifications.
Creating or using an existing taxonomy to assist in classification.

At the same time, another application team had started to use the classification service, encountered the same issues and had a few of their own ideas:

Upgrading the model.
Optimizing the current prompt with the help of a separate, more powerful LLM.

These were easy enough to implement and I agreed that we should try their options first.

However, I was quite firm on not pushing any changes into production until we had built an evaluation harness we could run against any proposed optimizations. The eye test may have got us to where we were and it was good enough to realize the project in production. But, if I was going to have my team spend their time and the company's money on these hypothesis, I needed to be able to justify that effort with measurable results.

Before I summarize how I built the evaluation harness, I will shout out a book that has been a go-to resource in my career: The Hundred-Page Machine Learning Book by Andriy Burkov. I knew precision and recall weren't the right metrics to benchmark against as our AI classifier was powered by a general purpose model and not a model trained on our order data. I also intuitively knew from my experience that even humans produce different classification given the same input. Burkov's book, a survey of ML algorithms and how they are implemented and evaluated, was where I landed on Cohen's kappa as the metric that fit our use case. The question shifted from "Is the AI correct" to "Is the AI agreeing with humans".

So the evaluation consisted of 3 groups of classifiers:

The "original" classifications made by various humans years ago
Human classifications done by two people who were tasked with being the humans-in-the-loop when our current classifier was unavailable
AI classifications

I compared agreement across all three groups and built a confusion matrix, so that alongside measuring inter-classifier agreement, I could also flag which categories caused the most disagreement.

The results led us to the following conclusions:

AI and human classifications lined up about as often as two separate human classifiers would. Using more powerful models resulted in a slightly higher Cohen's Kappa but not enough to justify the added cost.
A few categories were overly broad, for example "General". Humans were more likely to default to it when a more specific option was actually available.
A small number of categories accounted for most of the confusion between humans and AI.

Backed by measurable proof of improvement, the other team's prompt optimizations were released. I also was able to conclude that fixing the categories would be a more fruitful direction than model fine tuning. Being able to test hypotheses and discover real optimizations is the power of evaluating AI performance.