Stop using LGTM@Few as a metric (Better RAG)¶

I work with a few seed series a startups that are ramping out their retrieval augmented generation systems. I've noticed a lot of unclear thinking around what metrics to use and when to use them. I've seen a lot of people use "LGTM@Few" as a metric, and I think it's a terrible idea. I'm going to explain why and what you should use instead.

If you want to learn about my consulting practice check out my services page. If you're interested in working together please reach out to me via email

When giving advice to developers on improving their retrieval augmented generation, I usually say two things:

1. Look at the Data
2. Don't just look at the Data

Wise men speak in paradoxes because we are afraid of half-truths. This blog post will try to capture when to look at data and when to stop looking at data in the context of retrieval augmented generation.

I'll cover the different relevancy and ranking metrics, some stories to help you understand them, their trade-offs, and some general advice on how to think.

When to look at data?¶

Look at data when the problem is very new. Do not rely on any kinds of metrics just yet. Look at the queries people are asking. Look at the documents that people are submitting. Look at the text chunks and see whether or not a single text chunk could possibly answer a question your user might have, or if you need multiple text chunks to piece together a complete answer. Look at the results from initial prototypes to understand if the retrieval task is technically feasible.

There's a lot of intuition you can gain from just looking at the data.

When to stop?¶

At some point, you're going to actually want to build a system. You're going to want to iterate and improve on it. You will likely get nowhere if all you're doing is 'looking at things'. You will spend too much time guessing as to what will improve something rather than trying to measure and improve something.

"What gets measured gets managed."

Instead, define metrics, run tests, investigate when and where the metrics are poor, and then start looking at the data again.

graph LR
    A[Look at Data];
    A --> B[Work on System]
    B --> C[Define Metrics]
    C --> D[Look at Metrics]
    D --> E[Look at Data with Poor Metrics]
    E --> B

Well, let's take a closer look at what kind of metrics we can use and how they might improve our system. And I'll give an intuitive understanding of why and how some of these metrics break down. But first I also want to talk about the importance of speed.

Importance of Velocity¶

How quickly you can get metrics and run tests determines the nature of how you iterate on your software. If you're looking at a metric that takes a long time to compute, you're going to be waiting a long time to iterate on your system. So do whatever it takes to make the test that you run and the metrics you build as fast as possible!

Example via RAG

• Slow Metric: Collocating human preferences and consulting domain experts.
• Still Slow Metric: AI-generated metrics. When using something like GPT4, things can become very slow.
• Fast Metrics: Accuracy, Precision, Recall, MRR, NDCG, are computationally cheap given the labels.

The goal is to reason about the trade-offs between fast metrics and slow data. It takes a long time to get enough data so you can move fast. But if you never do that work, we're always gonna be stuck.

Simple Metrics for Relevancy and Ranking¶

In the retrieval context, there are plenty of metrics to choose from. I'm gonna go describe a couple of them. But before we do that, we need to understand what @k means.

Understanding @K¶

The simplest idea we should think about is the idea of @k. When we do RAG, we first have to retrieve a set of K documents. Then we will do some re-ranking potentially. And then select the top end results to show to a user or to a language model. Consider the following pipeline:

1. Fetch n documents via Keyword Search
2. Fetch n documents via Semantic Search
3. Combine them and re-rank
4. Select the top 25 chunks to show to LLM
5. Top 5 documents are shown to the user.

graph LR
    X[Query] --> A[Keyword Search];
    X --> B[Semantic Search];
    B --> C[Reranker]
    A --> C[Reranker]
    C --> D[Top 25]
    C --> E[Top 5]
    D --> Y[LLM]
    E --> Z[User]

Now let's look at some interpretations of top-k results.

I strongly recommend you not focus too hard on generation from an LLM, And to stay focused on being able to provide the right context. You will be able to get a language model to be more robust and as language models improve, they will only get more resilient to irrelevant information. However, as you build out your business, this data set that you curate on relevancy will stay with you.

Thinking about the metrics @ K¶

Now let's look at some metrics that we can use to evaluate the performance of our retrieval augmented generation system. The goal isn't to give a mathematical breakdown of these metrics, but instead give you a sense of what they mean and how they might be useful. And how I like to explain and interpret them at the limits.

Mean Average Recall (MAR) @ K¶

Focuses on the system's capability to retrieve all relevant documents within the top K results, emphasizing the breadth of relevant information captured.

Formula for Standard Recall

Mean Average Precision (MAP) @ K¶

Assesses the accuracy of the top K retrieved documents, ensuring the relevance and precision of retrieved content.

Formula for Standard Precision

Here's a quick table of how I like to interpret my precision and recall trade-offs.

Mean Reciprocal Rank (MRR) @ K¶

Highlights the importance of quickly surfacing at least one relevant document, with an emphasis on the efficiency of relevance delivery. Matters a lot when there are only a few items we can show to the user at any given time.

Formula

Normalized Discounted Cumulative Gain (NDCG) @ K¶

A nuanced measure that evaluates both the presence and graded relevance of documents, rewarding systems that present the most valuable information first.

How to improve¶

Once you have a system in place and some metrics you want to improve, again, the steps are very simple.

1. Choose a metric that aligns with your goals. Distinguish between primary metrics (that must improve) and guardrail metrics (that must not regress).
2. Formulate a hypothesis and adjust the system.
3. Evaluate the impact on your chosen metric.
4. Look at poorly performing examples, and iterate.
5. Go back to step 2.

Metrics lead to Business Outcomes¶

All of these metrics must ultimately be in service of something else. By improving things like precision, recall, and relevancy, what we're really hoping to do is generate better results for the business. The question then you have to ask yourself is, "What does that actually improve?". Here are a couple of things that you might want to consider.

1. User Satisfaction: Are users happy with the answers they're getting? Could be defined by Thumb Up/Down or NPS.
2. Engagement: Are users coming back to the platform? Are they spending more time on the platform?
3. Conversion: Are users buying more things? Are they clicking on more ads?
4. Retention: Are users staying on the platform longer? Are they coming back more often? Do we want to improve time spent?
5. Revenue: Are we making more money?
6. Cost: Are we spending less money on infrastructure?
7. Efficiency: Are we able to answer more questions with the same amount of resources?

The sooner you can relate some of these short-term fast metrics with larger slow metrics, The more you can make sure that you're going down the right path. Rather than trying to optimize something that has no impact down the road.

Conclusion¶

I hope this post has provided you with better intuition on how to think about relevancy of your text chunk.

1. Analyze data manually when facing a new problem, without relying on metrics initially.
2. Distinguish between primary metrics (that must improve) and guardrail metrics (that must not regress).
3. Velocity, clock speed of your iteration, is paramount. Make sure you can measure and iterate quickly.
4. Define metrics, conduct tests, investigate areas of poor performance, and then reevaluate the system.
5. Explore simple metrics for relevance and ranking, such as MAR, MAP, MRR, and NDCG.
6. Remember that these metrics should ultimately align with desired business outcomes.

Additional Notes¶

1. Notice that for MAR and MAP, They do not depend on the rank only the presence of the relevant document in the top K. This is why they are often used in the context of retrieval.
2. Notice that for MRR and NDCG, they depend on the rank of the relevant document. This is why they are often used in the context of ranking. If you end up building a sophisticated RAG application, you'll find that a lot of the time, many of the queries are just asking for documents which great opportunity to consider a ranking metric above just a regular context retrieval mechanism. If each document is 20 pages, you'll likely really care about which document shows up first.
3. Showing your work is super important for products that need to gain the user's trust. Again, ranking becomes really relevant even though language models themselves might not care.