Skip to content

2024

Everything I Learned from AI Consulting

As I've shared insights on building a consulting practice, marketing strategies, and referral techniques, I've realized there's so much more to discuss about the world of AI consulting. This post aims to consolidate everything I've learned, building upon the foundation laid in my previous articles. Whether you're just starting out or looking to refine your approach, this comprehensive guide will provide valuable insights into the consulting process, from positioning yourself as an expert to closing high-value deals. For those interested in the tools that support my consulting work, don't forget to check out my article on the consulting stack I use.

These insights are drawn from my extensive experience, including a livestream I conducted on the topic. If you're interested in the original discussion, you can find the YouTube live stream here.

Best Tools for Indie Consultants

As an indie consultant, having the right tools can make or break your business. Over the years, I've refined my "consulting stack" - a collection of software and services that keep my operations smooth and professional. This post is an extension of my thoughts on AI consulting and freelancing in the AI gold rush.

In this guide, I'll share the key components of my stack and why they matter. Whether you're just starting out or looking to optimize your existing practice, these tools can help streamline your operations and enhance your professional image. I'll cover everything from setting up your business foundation to essential software for day-to-day operations.

By the end, you'll have a comprehensive todo list for setting up your business foundation and implementing your own consulting stack. This advice stems from my personal experience and lessons learned from AI consulting, aimed at helping you avoid common pitfalls and accelerate your success in the consulting world.

Optimizing Tool Retrieval in RAG Systems: A Balanced Approach

RAG Course

This is based on a conversation that came up during office hours from my RAG course for engineering leaders. There's another cohort that's coming up soon, so if you're interested in that, you can sign up here.

When it comes to Retrieval-Augmented Generation (RAG) systems, one of the key challenges is deciding how to select and use tools effectively. As someone who's spent countless hours optimizing these systems, many people ask me whether or not they should think about using retrieval to choose which tools to put into the prompt. What this actually means is that we're interested in making precision and recall trade-offs. I've found that the key lies in balancing recall and precision. Let me break down my approach and share some insights that could help you improve your own RAG implementations.

In this article, we'll cover:

  1. The challenge of tool selection in RAG systems
  2. Understanding the recall vs. precision tradeoff
  3. The "Evergreen Tools" strategy for optimizing tool selection

The RAG Playbook

When it comes to building and improving Retrieval-Augmented Generation (RAG) systems, too many teams focus on the wrong things. They obsess over generation before nailing search, implement RAG without understanding user needs, or get lost in complex improvements without clear metrics. I've seen this pattern repeat across startups of all sizes and industries.

But it doesn't have to be this way. After years of building recommendation systems, instrumenting them, and more recently consulting on RAG applications, I've developed a systematic approach that works. It's not just about what to do, but understanding why each step matters in the broader context of your business.

Here's the flywheel I use to continually infer and improve RAG systems:

  1. Initial Implementation
  2. Synthetic Data Generation
  3. Fast Evaluations
  4. Real-World Data Collection
  5. Classification and Analysis
  6. System Improvements
  7. Production Monitoring
  8. User Feedback Integration
  9. Iteration

Let's break this down step-by-step:

How I want you to write

I'm gonna write something technical.

It's often less about the nitty-gritty details of the tech stuff and more about learning something new or getting a solution handed to me on a silver platter.

Look, when I read, I want something out of it. So when I write, I gotta remember that my readers want something too. This whole piece? It's about cluing in anyone who writes for me, or wants me to write for them, on how I see this whole writing product thing.

I'm gonna lay out a checklist of stuff I'd like to have. It'll make the whole writing gig a bit smoother, you know?

Crafting Compelling Titles

I often come across titles like "How to do X with Y,Z technology." These don't excite me because X or Y are usually unfamiliar unless they're already well-known. Its rarely the dream to use X unless X is the dream.

My dream isn’t to use instructor, its to do something valueble with the data it extracts

An effective title should:

  • Evoke an emotional response
  • Highlight someone's goal
  • Offer a dream or aspiration
  • Challenge or comment on a belief
  • Address someone's problems

I believe it's more impactful to write about specific problems. If this approach works, you can replicate it across various scenarios rather than staying too general.

  • Time management for everyone can be a 15$ ebook
  • Time management for executives is a 2000$ workshop

Aim for titles that answer questions you think everyone is asking, or address thoughts people have but can't quite articulate.

Instead of "How I do something" or "How to do something," frame it from the reader's perspective with "How you can do something." This makes the title more engaging. Just make sure the difference is advisory if the content is subjective. “How I made a million dollars” might be more reasonable than “How to make a million dollars” since you are the subject and the goal might be to share your story in hopes of helping others.

This approach ultimately trains the reader to have a stronger emotional connection to your content.

  • "How I do X"
  • "How You Can do X"

Between these two titles, it's obvious which one resonates more emotionally.

You can take it further by adding specific conditions. For instance, you could target a particular audience or set a timeframe:

  • How to set up Braintrust
  • How to set up Braintrust in 5 minutes

NO adjectiives

I want you to almost always avoid adjectives and try to use evidence instead. Instead of saying "production ready," you can write something like "scaling this to 100 servers or 1 million documents per second." Numbers like that will tell you exactly what the specificity of your product is. If you have to use adjectives rather than evidence, you are probably making something up.

There's no reason to say something like "blazingly fast" unless those things are already known phrases.

Instead, say "200 times faster" or "30% faster." A 30% improvement in recommendation system speed is insane.

There's a 200 times performance improvement because we went from one programming language to another. It's just something that's a little bit more expected and understandable.

Another test that I really like using recently is tracking whether or not the statements you make can be:

  • Visualized
  • Proven false
  • Said only by you

If you can nail all three, the claim you make will be more likely to resonate with an audience because only you can say it.

Earlier this year, I had an example where I embedded all of Wikipedia in 17 minutes with 20 bucks, and it got half a million views. All we posted was a video of me kicking off the job, and then you can see all the log lines go through. You see the number of containers go from 1 out of 50 to 50 out of 50.

It was easy to visualize and could have been proven false by being unreproducible. Lastly, Modal is the only company that could do that in such an effortless way, which made it unique.

Strong Introduction

So, if you end up doing any kind of sales, you'll realize that.

What you actually need to understand is not what you have to offer as the product, but the size of the pain that the prospect is going through.

  • There are going to be readers that are just kind of curious and bored. They're not really going to be the ones that care about the product itself unless you can contextualize the pain for them.
  • It's really important to have an introduction that contextualizes the pain and foreshadows the solution.

If we can build that trust and I can correctly describe the pain that you are going through, then you will believe me when I am predicting the pain that you may also go through in the future. Ultimately, that is how you become a leader in the space—by demonstrating your ability to be right consistently.

The next time you publish or write something, they will believe it, and they will believe that they get value from it.

Strong Hooks

In the same sense that a title should often try to change the "how I" to a "how you" by eliciting an emotional response, the introduction can also help select the reader into a group that is feeling the pain.

This is the same reason why a plumber will have an introduction that says, "Do you have a leaky faucet? Call 1-800-PLUMBERS." That's a much more selective hook than just "I'm the best plumber in town." You can say that to everybody, whereas if someone answers the question of whether they have a leaky faucet, it automatically selects them to be a part of their readership.

I truly believe if you try to build a product too soon for everybody, you're gonna end up in a bad place.

Foreshadow Content

Once you hook them, you still have to first retain them. You can do that by foreshadowing the content you'll cover and even hinting at the reward.

For example, an introduction could look like the following:

  • If you're making $10,000 a month consulting right now, my goal at the end of this blog post is to help you increase your prices by:
    • Asking the right questions so you understand the value of the solution you're offering.
    • Providing tips on writing proposals and offering different options you can let your customer pay you more.
    • Lastly, sharing some anecdotes with you on how I became more comfortable with charging two or three times more than I did when I started

Here, I've pre-qualified the reader for a certain range and told them what their goal is by the end of the post.

Two things are just the tips and the questions I'm going to suggest, and then the final reward is something a little bit more personal. Ideally, they read the first two knowing that my personal stories are coming after. That intro itself outlines the entire post.

Use Lists

Once you've hooked your audience, you have to retain them and reward them.

You'll also see that as part of the foreshadowing content, I've listed three items that I want them to take away. I can also be specific with the number of questions and the number of tips I'm providing.

By using lists and counting things, I can give them a notion of progress towards the final conclusion. If I'm on the second part, I know where I am in the story. The list itself allows us to break down the body and give the reader a sense of position. Since they know where they are, they know where they're going to be.

Demonstrate being an Oracle

One thing that's also really valuable to call out is the fact that you want to be seen as a leader or an oracle to this audience.

For example, if we go back to this charging more case, it's one thing to demonstrate that you understand that the reader's dream is to be able to charge more. It's useful that you're giving them a couple of tips and stories. But what can be even more powerful is to simultaneously:

  • Demonstrate your knowledge of the current problem.
  • Predict future problems as they come.
  • Foreshadow or reference those future issues in later content.

For example, the things you do to go from $10,000 to $30,000 a month are very different from what you would need to do to get to $200,000 a month. They require things like hiring, prioritizing your services, and improving distribution. If you can foreshadow that and call that out, when your audience gets to those levels, there's a chance they will remember what you said.

They'll think, "Wow, Jason was not only right about where I was but also where I was going." This brings a tremendous sense of trust and value.

It's not just this idea of future work or future considerations of the current work, but actually being able to predict the problems they're going to have in the future and suggesting that you are also the solution to those problems. You can foreshadow that as part of a series or whatever, but the general idea remains.

Have a Strong CTA

You also have to think about what the reward for yourself is. Ultimately, you should be writing this because you think the message is important and that you believe your audience should and deserves to get this message. The content of this post delivers the reward to those who stay and finish the article.

But at the end, you also have to get something in return. You should ask your user to do something. If it's a tweet, a simple one could be a repost, a like, or a share. It could even be a follow. It could be entering some GitHub link and giving the repost a star, etc.

What I realized to be very important was to make sure you only ask for one thing and don't split the attention. If you do that, you can have solid metrics on how you phrase CTAs and how that converts to certain content. For example:

  • How many people from the tweet go to the blog post?
  • How many go from the blog post to a subscription to a newsletter?

The more I think about it, the more I believe that most people should be capturing information into a newsletter rather than just Twitter. A direct email is so much more powerful.

No matter what it is, make sure you only ask for one thing. Sometimes it's to sign up, sometimes it's to try a one-click deploy. But if there's no action that your users can take, you've definitely made a mistake.

It's also good to call out that taking the action should have some kind of outcome as well.

  • If you want to see more of this content, follow me because I post twice a week.
  • That would be an example that I can qualify as a user and set expectations on what the outcome is.
  • If you like it, then subscribe. You'll get two posts a week.

I also think in many situations we should have tiers of qualified CTUs.

If you want to split your traffic, you should have an obvious condition as to which one someone should take. For example, on Indexify:

  • If you're dealing with terabyte-scale datasets, contact us.
  • If you want to try the Open Source library

It sets a prequalifier.

The terabyte-scale dataset is an evidence-based pre-filter. It could have said "in production," but something so specific like a terabyte or a terabyte a day really qualifies who should contact indexify

Living My Best Life: A $20 Million Daydream

I've been playing a little thought experiment lately: If I had $20 million in the bank, how would I want to live my life? It's not about the money per se, but about imagining a life where financial constraints aren't the primary driver of my decisions. Here's what I've come up with.

Work: Finding the Sweet Spot

With financial freedom, I'd aim for a work schedule that balances productivity and passion:

  • 2-3 days a week of focused work
  • Flexibility to pursue projects I'm truly excited about
  • Time for creative exploration and learning

The goal isn't to stop working, but to work on things that truly matter to me, still enjoy learning and teaching.

Living Space: A Home That Inspires

My dream home would be a space that nurtures creativity and relaxation:

  • Spacious, with plenty of room for hosting friends
  • A dedicated home office for those work days
  • A killer outdoor setup for grilling and chilling
  • A pool for daily swims and relaxation
  • High-end audio system for immersive music experiences
  • Iconic furniture pieces like an Eames chair and Noguchi lamps

It's not just about luxury, but about creating an environment that fuels my passions and supports my lifestyle.

Activities: Pursuing Passions

With more time and resources, I'd dive deeper into the activities that bring me joy:

Pottery

  • Regular sessions to hone my craft
  • Potentially pursuing an apprenticeship or hosting a show
  • Exploring hand-building techniques
  • Creating functional art, like fish-serving plates

Brazilian Jiu-Jitsu

  • Low-intensity training 2-3 times a week
  • Focus on technique and enjoyment rather than competition
  • Staying injury-free and maintaining fitness

Swimming and Spearfishing

  • Daily swims in my home pool for fitness
  • Occasional spearfishing trips to connect with nature
  • Learning to prepare the fish I catch

Travel: Expanding Horizons

With financial freedom, I'd approach travel more intentionally:

  • Quarterly cross-country trips (2 weeks each)
  • Annual international trip to Europe or Japan
  • Two freediving trips per year, possibly swimming with whales or orcas

The key is balancing adventure with my home-based passions and work commitments.

Lifestyle: Quality Over Quantity

Money can't buy happiness, but it can certainly improve quality of life:

  • High-quality, varied meals (possibly with a personal chef)
  • Regular entertainment (monthly jazz or comedy shows)
  • Hosting friends for dinners and gatherings
  • House cleaner and personal assistant for support

The goal is to create a life rich in experiences and connections, not just possessions.

Personal Growth: Never Stop Learning

Even with financial security, I'd continue investing in myself:

  • Educating myself on investing and financial management
  • Being more selective with commitments and relationships
  • Focusing on "hell yes" experiences and friendships
  • Writing more, both for personal growth and to share insights

Reflections on This Exercise

This thought experiment isn't about daydreaming of wealth. It's about understanding what truly matters to me and how I can start incorporating elements of this ideal life into my current reality.

Some key takeaways:

  1. Balance is crucial - even with unlimited resources, I'd want a mix of work, play, and personal growth.
  2. Experiences and relationships matter more than possessions.
  3. Health and wellness are non-negotiable, regardless of wealth.
  4. Creativity and learning remain central to my ideal life.

While I may not have $20 million, this exercise has clarified my priorities and given me a roadmap for the life I want to create. It's a reminder that many aspects of our "ideal" life are within reach if we're intentional about our choices.

What would your $20 million life look like? More importantly, what elements of that life can you start incorporating today?

On Getting Recognized: The Unexpected Price of Online Success

I never thought I'd be writing about the challenges of being recognized in public. A year ago, I was just another data scientist trying to build connections and establish myself in the field. Now, I'm grappling with the unintended consequences of my growing online presence. Here's how it all unfolded, and what I've learned about the price of distribution in the digital age.

Art of Looking at RAG Data

In the past year, I've done a lot of consulting on helping companies improve their RAG applications. One of the biggest things I want to call out is the idea of topics and capabilities.

I use this distinction to train teams to identify and look at the data we have to figure out what we need to build next.

My Self-Reflection on Success and Growth

In his essay "What's Going On Here, With This Human?", Graham Duncan discusses the importance of seeing people clearly, both in the context of hiring and in understanding oneself. He suggests asking the question "what's going on here with this person in front of me?" and provides a framework for expanding one's ability to see others more clearly. Inspired by this essay, I asked myself some probing questions to better understand my own strengths, weaknesses, and motivations. Here are my reflections:

Predictions for the Future of RAG

In the next 6 to 8 months, RAG will be used primarily for report generation. We'll see a shift from using RAG agents as question-answering systems to using them more as report-generation systems. This is because the value you can get from a report is much greater than the current RAG systems in use. I'll explain this by discussing what I've learned as a consultant about understanding value and then how I think companies should describe the value they deliver through RAG.

Rag is the feature, not the benefit.