2024¶

The RAG Playbook

When it comes to building and improving Retrieval-Augmented Generation (RAG) systems, too many teams focus on the wrong things. They obsess over generation before nailing search, implement RAG without understanding user needs, or get lost in complex improvements without clear metrics. I've seen this pattern repeat across startups of all sizes and industries.

But it doesn't have to be this way. After years of building recommendation systems, instrumenting them, and more recently consulting on RAG applications, I've developed a systematic approach that works. It's not just about what to do, but understanding why each step matters in the broader context of your business.

Here's the flywheel I use to continually infer and improve RAG systems:

1. Initial Implementation
2. Synthetic Data Generation
3. Fast Evaluations
4. Real-World Data Collection
5. Classification and Analysis
6. System Improvements
7. Production Monitoring
8. User Feedback Integration
9. Iteration

Let's break this down step-by-step:

On Getting Recognized: The Unexpected Price of Online Success

I never thought I'd be writing about the challenges of being recognized in public. A year ago, I was just another data scientist trying to build connections and establish myself in the field. Now, I'm grappling with the unintended consequences of my growing online presence. Here's how it all unfolded, and what I've learned about the price of distribution in the digital age.

Art of Looking at RAG Data

In the past year, I've done a lot of consulting on helping companies improve their RAG applications. One of the biggest things I want to call out is the idea of topics and capabilities.

I use this distinction to train teams to identify and look at the data we have to figure out what we need to build next.

Predictions for the Future of RAG

In the next 6 to 8 months, RAG will be used primarily for report generation. We'll see a shift from using RAG agents as question-answering systems to using them more as report-generation systems. This is because the value you can get from a report is much greater than the current RAG systems in use. I'll explain this by discussing what I've learned as a consultant about understanding value and then how I think companies should describe the value they deliver through RAG.

Rag is the feature, not the benefit.

10 Ways to Be Data Illiterate (and How to Avoid Them)

Data literacy is an essential skill in today's data-driven world. As AI engineers, understanding how to properly handle, analyze, and interpret data can make the difference between success and failure in our projects. In this post, we will explore ten common pitfalls that lead to data illiteracy and provide actionable strategies to avoid them. By becoming aware of these mistakes and learning how to address them, you can enhance your data literacy and ensure your work is both accurate and impactful. Let's dive in and discover how to navigate the complexities of data with confidence and competence.

Advice to Young People, The Lies I Tell Myself

I'm really not qualified to give advice. But enough people DM'd me on Twitter, so here it is. I don't have to answer the same question over and over again. After some more editing I realised that I am actually writing this for my younger sister Katherine.

If you want to know who I am, check out blog/whoami or my Twitter. or consider subscribing to my newsletter below.

Thoughts on Marketing, Positioning, and Scaling my Consulting

As a former MLE, I had always relied on my hands-on coding skills to excel in my career. However, a hand injury forced me to adapt and pivot towards consulting. In this essay, I share my journey of becoming a successful AI consultant and the lessons I've learned about marketing, positioning, and scaling my business.

• Transitioning to consulting and building a marketing plan
• Generating leads and creating valuable content
• Closing leads using situational assessments

Losing My Hands Pt. 2: The Two Jasons

This is part of an ongoing series that discusses my experience of suffering an RSI during the COVID-19 Pandemic and the ongoing struggles and disruptions it still causes. You can read part one here. My goal here is to interrogate how the injury affected my values and my approach to life. You'll probably be disappointed if you're looking for a detailed medical report.

This comment partly inspired this installation of the series. This article isn't meant to be preachy but rather introspective of the value systems that shape me and many other men.

Losing my Hands, Advice and Tips for RSI

This is the final entry (for now) of a series I wrote about my experience with tendonitis in both my wrists and thumbs during COVID-19. The injury occurred in 2020 due to overworking at my previous job, where I would put in excessive hours for weeks on end with minimal breaks. The debilitating pain prevented me from eating, dressing myself properly, and programming, leading to a significant disruption in my work and personal life. Ultimately, this experience led me to make three major changes:

1. Consulting full-time: Sometimes, you need to take a leap of faith; other times, you need a swift, firm shove in the back to overcome your limitations. The RSI was the latter that lead me down the path of higher leverage.

2. Revaluating my values: This doesn't necessarily mean discarding the traits that led to me getting an RSI, but it certainly made me rethink what is actually valuable and what I am trying to do with my life. Sorry to disappoint, but there has been no 'epiphany'. I am not discarding worldly possessions and pursuing a spirit quest to find 'my true self'. My hands are now a limited commodity, and I fear reinjuring myself. This means if I'm going to use my hands, it must be purposeful and deliberate.

3. Writing: Thankfully, we live in an era where you no longer need hands to express yourself through text. Writing helped me realize that I have spent my first three decades learning and absorbing knowledge and information. Now, in my thirties, my goal will be to share the thoughts and stories I've collected.

You can read the previous two entries here: Losing My Hands, Losing My Hands Pt. 2: The Two Jasons.

Systematically Improving Your RAG

These are notes generated after a call I had with Hamel on a 'system' to improve a RAG system. I've also written some other work like Rag is not Embeddings and how to build a Terrible RAG System and how complexity can be broken down into smaller pieces.

By the end of this post, you'll have a clear understanding of my systematic approach to improving RAG applications for the companies I work with. We'll cover key areas such as:

• Create synthetic questions and answers to quickly evaluate your system's precision and recall
• Make sure to combine full-text search and vector search for optimal retrieval
• Implementing the right user feedback mechanisms to capture specifically what you're interested in studying
• Use clustering to find segments of queries that have issues, broken down into topics and capabilities
• Build specific systems to improve capabilities
• Continuously monitoring, evaluating as real-world data grows

Through this step-by-step runbook, you'll gain practical knowledge on how to incrementally enhance the performance and utility of your RAG applications, unlocking their full potential to deliver exceptional user experiences and drive business value. Let's dive in and explore how to systematically improve your RAG systems together!