2024¶

Stop using LGTM@Few as a metric (Better RAG)

I work with a few seed series a startups that are ramping out their retrieval augmented generation systems. I've noticed a lot of unclear thinking around what metrics to use and when to use them. I've seen a lot of people use "LGTM@Few" as a metric, and I think it's a terrible idea. I'm going to explain why and what you should use instead.

For more writing on RAG and evaluation, start with RAG series index.

When giving advice to developers on improving their retrieval augmented generation, I usually say two things:

1. Look at the Data
2. Don't just look at the Data

Wise men speak in paradoxes because we are afraid of half-truths. This blog post will try to capture when to look at data and when to stop looking at data in the context of retrieval augmented generation.

I'll cover the different relevancy and ranking metrics, some stories to help you understand them, their trade-offs, and some general advice on how to think.

My year at 1100ng/dL

I'm not a doctor, but I did manage to double my testosterone levels in a year. I'm going to talk about what I did, what I learned, and what I think about it:

1. It's just a fact that male testosterone levels have been dropping for the past couple of years.
2. I felt like I was in a rut and I wanted to feel better, and I did.
3. I was such a psycho about it that I decided to go off the protocol.
4. Despite that, I still think every man should get their levels tested and see if they can improve them. And just understand how they feel.

Indie Consulting

As I've shared insights on building a consulting practice, marketing strategies, and referral techniques, it's important to understand the unique position of indie consulting in the broader landscape. In this post, we'll explore how indie consulting differs from traditional large-scale consulting firms and why it can offer more value to clients.

Indie consulting is fundamentally distinct from the practices of well-known institutions. For a critical perspective on these large firms, I recommend watching John Oliver's insightful critique of McKinsey or this concise TikTok video that encapsulates the issues with big consulting firms.

In contrast to these large firms, indie consulting focuses on specialized expertise, direct accountability, and long-term value creation for clients. It's about leveraging personal experience and skills to solve specific problems, rather than applying generic frameworks or strategies. This approach aligns closely with the pricing strategies and tools I've discussed in previous posts, all aimed at delivering maximum value to clients.

A Critique on Couches

Here are some fragmented reasons as to why I don't like having a couch.

The couch, often positioned facing a television, symbolizes the societal imposition of a predetermined essence onto our living spaces. This arrangement, reminiscent of Sartre's concept of bad faith, dictates the room's function and restricts its potential. It mirrors the limitations we place upon ourselves when we conform to societal expectations, disregarding our authentic selves.

For real.

Tips for probabilistic software

This writing stems from my experience advising a few startups, particularly smaller ones with plenty of junior software engineers trying to transition into machine learning and related fields. From this work, I've noticed three topics that I want to address. My aim is that, by the end of this article, these younger developers will be equipped with key questions they can ask themselves to improve their ability to make decisions under uncertainty.

1. Could an experiment just answer my questions?
2. What specific improvements am I measuring?
3. How will the result help me make a decision?
4. Under what conditions will I reevaluate if results are not positive?
5. Can I use the results to update my mental model and plan future work?

Public Baths

Going to American baths is just so weird. I spent my summer in Japan visiting different onsens, and it was both a natural and spiritual experience. Before entering the water, everyone would bathe in the front, and kids would learn from their dads how to bathe. I would often sit on the edges of cliffs, gazing at the water or the sunrise, and it felt like we were monkeys, freely splashing about in nature.

In contrast, the time I spent in LA or New York City at various bathhouses was different. No one looked like an animal; instead, everyone seemed focused on optimization. People barely bathed before entering the water, wearing their dirty little speedos and swim trunks that they had definitely peed in the month before.

Gross.

Anatomy of a Tweet

The goal of this post is basically to share what I have learned about writing a tweet, how to think about writing a hook, and a few comments on how the body and the cta needs to retain and reward the user. Its not much, I've only been on twitter for about 6 month.

I used to hate rich people.

This entire piece of writing is dedicated to a recent response on Hacker News. I hope you can see, as a member of reality, that I write this sincerely.

—

Learning to Learn

After writing my post advice for young people, a couple of people asked about my learning process. I could discuss overcoming plateaus or developing mastery, learning for the joy of learning. I could also talk about how to avoid feeling overwhelmed by new topics and break them down into smaller pieces. However, I think that has been done before.

Instead, I'm going to explore a new style. I'm just going to go through a chronological telling of my life and what I learned from just trying new things. I'm going to talk about the tactics and strategies and see how this pans out.

How to build a terrible RAG system

If you've followed my work on RAG systems, you'll know I emphasize treating them as recommendation systems at their core. In this post, we'll explore the concept of inverted thinking to tackle the challenge of building an exceptional RAG system.

This approach aligns with our broader discussion on RAG systems, which you can explore further in our RAG flywheel article and our comprehensive guide on Levels of Complexity in RAG Applications.