I hosted a lightning lesson featuring Ben from Raindrop and Sid from Oleve to discuss AI monitoring, production testing, and data analysis frameworks. This session explored how to effectively identify issues in AI systems, implement structured monitoring, and develop frameworks for improving AI products based on real user data.
Today I spoke to an executive about SaaS products, and they told me something that shifted my perspective entirely: AI agents need to be compared to budgets that companies draw from headcount, not tooling.
This is one of those insights that seems obvious in retrospect, but completely changes how you think about positioning AI tools in today's market—especially in this era of widespread tech layoffs and economic uncertainty.
The world of RAG evaluation feels needlessly complex. Everyone's building frameworks, creating metrics, and generating dashboards that make you feel like you need a PhD just to know if your system is working.
I'll be hosting industry experts to share practical techniques for enhancing your Retrieval Augmented Generation (RAG) systems.
This is some of the notes I've taken for learnindieconsulting.com
Let me share something I wish I'd understood sooner: consistent content creation isn't just a marketing tactic—it's the foundation of a thriving consulting business.
When I started my consulting journey, I was stuck in the time-for-money trap. I'd jump on Zoom calls with prospects, explain the same concepts repeatedly, and wonder why scaling was so difficult. Then I had a realization that changed everything: what if I could have these conversations at scale?
Now I extract blog post ideas from every client call. Every Friday, I review about 17 potential topics from the week's conversations. I test them with social posts, see which ones get traction (some get 700 views, others 200,000), and develop the winners into comprehensive content.
Here's why this approach has transformed my business:
Last year, I wrote a post called "RAG is more than embeddings" that generated around 30,000 views on Hacker News. This single piece became one of the biggest catalysts for my consulting business.
The real magic happens when a prospect asks a question, and I can send them a blog post I wrote months ago. It immediately signals that I've already thought deeply about their problem. They think, "This person has already solved the exact challenge I'm facing."
Content helped me escape the limitations of hourly billing. After getting tired of repeating the same RAG implementation advice at $1,000 per hour, I created a course that's more accessible while freeing my time for higher-value work.
This ladder of offerings at different price points lets clients engage with me at whatever level makes sense for them—from free blog posts to high-end consulting. Without content, this business model wouldn't be possible.
I don't try to perfectly identify my audience before creating. Instead, I use content as a sensor to discover what resonates.
When I started on Twitter, I had about 500 followers. I noticed my follower-to-tweet ratio was around 1.7, meaning if I wanted 10,000 followers, I'd need to write roughly 7,000 tweets. So I committed to consistent creation—it's like losing weight; it depends on how often you're working out.
Volume negates luck, and these efforts compound quickly. Don't wait months—focus on creating volume and consistency.
Every piece of content I create becomes part of a growing library that continuously generates value. Unlike client projects that end, my content:
Now let me walk you through exactly how I approach content creation, from foundational frameworks to practical execution.
Everything I create revolves around a simple but powerful formula:
This equation isn't just about demonstrating value as a consultant—it's about creating content that resonates deeply with readers.
When I write about RAG implementation, I don't just explain technical details. I show readers:
This value equation shapes everything—my blog posts, social content, course materials, and sales conversations. It's not just theory; it directly improves how I communicate with clients.
Every piece of content I create follows this proven structure:
This isn't just marketing-speak. Practicing this framework has made me more articulate during sales conversations and client calls. It's like "wax on, wax off"—you're training communication muscles that serve you across contexts.
Let me share an example of how I applied this to a recent blog post:
Attention: "Most RAG systems fail because engineers focus exclusively on embeddings while ignoring three critical components."
Interest: "After implementing 20+ RAG systems across industries, I've identified patterns that consistently separate high-performing systems from failures."
Desire: "By addressing these overlooked elements, you can build systems that deliver consistently relevant results and earn genuine user trust."
Action: "Use my evaluation framework (linked here) to identify the specific weaknesses in your current implementation."
I've learned that your title isn't just important—it's about 80% of the work. A mediocre post with a compelling title will outperform a brilliant post with a boring title every time.
Here's an example from my own experience:
When I first started writing, I'd use titles like: "I spent the weekend playing around with ChatGPT"
This attracted almost no interest because:
When I shifted to titles like: "3 ChatGPT Prompts That Doubled My Content Creation Speed"
I never commit to a full article without testing the title first. Here's my process:
For example, when writing about pricing strategies, I tested:
The last one generated 4x more engagement than the others, so that became my focus for the full article.
One question I hear constantly is: "How do I find my audience?"
My answer might surprise you: I don't try to perfectly identify my audience before creating content. Instead, I create content, see what resonates, and refine my understanding through this feedback loop.
When I built my open-source tool Instructor, I started with a message about wanting a simple tool that does one thing well, without unnecessary complexity. The audience came secondary to the message and dream.
That said, I do have a mental checklist I run through before creating content:
But I hold these ideas loosely, allowing real-world feedback to refine my understanding.
Each platform attracts different audiences and serves different purposes in my content ecosystem:
I use Twitter as my primary testing ground. With a follower-to-tweet ratio between 0.5-4, I know that generating 10,000 followers requires 2,500-20,000 tweets.
Twitter has been especially valuable for connecting with founders, investors, and technical audiences. Most of my industry relationships—including collaborations with other prominent consultants—came through Twitter.
When targeting enterprise clients, VPs, Directors, and corporate decision-makers, I shift to LinkedIn. The content is more formal and business-focused, emphasizing outcomes over technical details.
I've found that LinkedIn works better for corporate/enterprise VP-level content, while Twitter is ideal for founders and investors.
I strongly believe in owning your distribution channel. While platforms like Medium or Towards Data Science might seem appealing, they diminish your personal brand.
On my own website, I can publish blog content while also offering email capture and a services page. This creates a journey where people might read my posts over several months before they're ready to explore my services.
I use a simple framework—markdown docs with MkDocs Material—making it easy to edit and maintain content without relying on third-party platforms.
The quality of subscribers matters more than quantity. "If you offered me a hundred thousand AI news newsletter subscribers versus 200 CTOs in Silicon Valley, I would probably pay $80,000 for that CTO email list."
I started my newsletter relatively late, after my Twitter following had grown significantly. If you're starting now, make email capture a priority from the beginning.
Over time, I've developed a systematic approach to creating content that consistently delivers results:
I extract content ideas from multiple sources:
I've automated much of this process. After client calls, I use AI tools to extract potential blog ideas from the transcripts. Every Friday, I review these ideas, identifying patterns and high-potential topics.
For each piece, I follow this proven structure:
The Hook (Attention) I open with something that immediately grabs attention—a provocative question, surprising statistic, or challenge to conventional wisdom.
For example, instead of starting with "RAG systems are important," I might open with: "90% of RAG implementations fail because they focus on the wrong problem entirely."
The Value Promise (Interest) I clearly state what the reader will learn and why it matters to them specifically.
Example: "In this guide, you'll discover the three overlooked components that determine RAG success—insights I've gathered from implementing systems for companies like X, Y, and Z."
The Main Content (Desire) I deliver actionable insights using specific examples and case studies. I demonstrate expertise without overwhelming with technical details.
The key is focusing on the hierarchy of value. Technical details are often the "lowest pole of value." I've learned that clients care much more about business outcomes than technical implementations.
The Call to Action (Action) I provide a clear next step that aligns with my business goals while offering genuine value.
Before publishing, I optimize for different reading styles:
For Skimmers:
For Deep Readers:
I spend 20% of my time creating content and 80% on distribution. Here's my approach:
I turn blog posts into tweet threads, convert videos into blog posts, and extract key points for LinkedIn posts. This maximizes the return on every piece of content I create.
When my hands were hurting from typing too much, I started recording videos. This created an efficient workflow:
I share blog posts across all social channels, mention content in relevant communities, and reference past pieces in new content. This creates a network effect where each piece strengthens the others.
I respond to all comments quickly, tag relevant people who might find my content valuable, and follow up with engaged readers. This turns content into conversations, which often lead to client relationships.
I track several metrics to evaluate content effectiveness:
The most important question I ask new clients: "How did you find me?" Their answers inform my content strategy.
I'm strategic about estimating the Total Addressable Market (TAM) for my content:
Specific vs. General Content Highly specific titles reach smaller audiences but convert better. General titles reach larger audiences but convert poorly.
For example, "How to implement a RAG system" reaches a broad audience of developers, while "How to create RAG systems for financial compliance" reaches a tiny audience but with much higher conversion potential.
Balance TAM with Conversion Potential I aim for the sweet spot: content specific enough to convert well but broad enough to reach sufficient potential clients.
Before publishing anything, I run through this mental checklist:
The greatest benefit of consistent content creation is the flywheel effect it creates:
This has allowed me to work with increasingly better clients on more interesting problems at higher rates. My content acts as a filter, attracting people who value my specific expertise and approach.
Remember: Great content doesn't just attract an audience—it attracts the right audience who values your expertise enough to pay for it. And that makes all the difference in building a sustainable, fulfilling consulting business.
I hope this guide helps you create content that not only resonates with your audience but also drives real business results. If you have questions or want to share your content journey, reach out—I'm always interested in hearing how these approaches work for others.
Imagine this: you open Cursor, ask it to build a feature in YOLO-mode, and let it rip. You flip back to Slack, reply to a few messages, check your emails, and return...
It's still running.
What the hell is going on? .sh files appear, there's a fresh Makefile, and a mysterious .gitignore. Anxiety creeps in. Should you interrupt it? Could you accidentally trash something critical?
Relax—you're not alone. This anxiety is common, especially among developers newer to powerful agents like Cursor's. Fortunately, Git is here to save the day.
In Part 1, you learned the basics of safely using Git with Cursor agents. Now, let's level up your workflow by diving into advanced Git practices and explicitly instructing Cursor to handle these for you.
Systematically improving RAG systems
This transcript is based off of a guest lecture given in my course, Systematically Improving RAG Applications
Retrieval-Augmented Generation (RAG) systems have become essential tools for enterprises looking to harness their vast repositories of internal knowledge. While the theoretical foundations of RAG are well-understood, implementing these systems effectively in enterprise environments presents unique challenges that aren't addressed in academic literature or consumer applications. This article delves into advanced techniques for fine-tuning embedding models in enterprise RAG systems, based on insights from Manav Rathod, a software engineer at Glean who specializes in semantic search and ML systems for search ranking and assistant quality.
The discussion focuses on a critical yet often overlooked component of RAG systems: custom-trained embedding models that understand company-specific language, terminology, and document relationships. As Jason Liu aptly noted during the session, "If you're not fine-tuning your embeddings, you're more like a Blockbuster than a Netflix." This perspective highlights how critical embedding fine-tuning has become for competitive enterprise AI systems.
I never planned to become a consultant. But somewhere between building machine learning systems and growing my Twitter following, companies started sliding into my DMs with the same message: "Help, our AI isn't working."
So I started charging to join their stand-ups. Sometimes I didn't even code. I just asked uncomfortable questions.
Here's what I've learned watching companies burn millions on AI.
The head of AI at a major company once showed me their evaluation framework. They had 23 evals. One of them tested if their AI could "talk like a pirate."
I shit you not.
Meanwhile, 75% of their users were just trying to make their AI read Excel spreadsheets. But they had three engineers focused on a fancy document integration that less than 10% of users touched.
This is the pattern everywhere I go. Companies obsess over sophisticated AI capabilities while ignoring what users actually do with their product.
The truth? Your AI problem is almost always a data problem. Or a process problem. Or a people problem. But rarely is it actually an AI problem.
"When GPT-5 comes out, all our problems will be solved."
I've heard this exact sentiment in different words from CTOs who should know better. It's magical thinking, and it's everywhere.
Here's a hard truth: capabilities are monotonically increasing. Each new model will be better than the last. But if your recall sucks now, ChatGPT 47.0 isn't going to save you.
Fix your retrieval. Fix your data pipeline. Fix your evaluation process. These are things you control today.
When I tell clients this, they often look disappointed. They wanted me to tweak some prompt magic or recommend a fancy new model. Instead, I'm telling them to do the unsexy work of:
One client went from 27% recall to 85% in four hours once they stopped praying to the model gods and started making targeted improvements to their data preprocessing. It wasn't glamorous, but it worked.
Most AI teams are measuring the wrong things. They obsess over model perplexity or ROUGE scores while their business burns.
At a sales tech company, engineers spent weeks trying to perfect their call transcript summarization accuracy. Meanwhile, sales managers didn't care about perfect summaries — they wanted to know which prospects had buying objections.
Classification first, extraction second. Always.
When we shifted from "how accurately can we summarize this call?" to "can we identify sales objections with 95% recall?", suddenly the business value was obvious. The sales team could follow up on objections, coach their reps better, and close more deals.
The best metric isn't technical. It's: "Does this solve a real problem someone will pay for?"
Ask most AI teams how long it takes to run an experiment, and you'll get embarrassed silence. Four days? A week? "Well, we have to coordinate with three teams and..."
The companies winning with AI can run 10 experiments before lunch.
One client went from spending weeks fine-tuning prompts to implementing a system where they could:
All in the same day.
Your ability to learn fast is everything. Build for learning velocity above all else.
As I tell my clients: how do you get better at jiu-jitsu? Mat time. More experiments, more data, more learning.
Nobody cares that your RAG system has 78.9% recall. They care that FancyCorp saved $2M using your product.
I was advising a construction tech company where we dramatically improved their document retrieval. But the testimonial that closed their next funding round wasn't "They improved our retrieval precision by 40%."
It was: "Their system automatically identifies contractors who haven't signed liability waivers 4 days before they're scheduled on site. This prevented $80,000 in construction delays last month alone."
When marketing your AI, translate your metrics into outcomes people actually care about.
The highest-leverage consulting I do isn't technical at all. It's identifying where the actual value is.
At a document management company, we discovered that 60% of questions weren't about document content at all. Users were asking: "Who last modified this document?" They needed contact information to verify decisions.
So we modified every text chunk to include metadata about creation date, modification date, and owner contact info. Suddenly, the most valuable feature wasn't fancy RAG—it was helping people find who to call.
At another client, we found that what users really wanted wasn't answering questions about blueprints. They wanted to count rooms. We could have spent months improving general QA, or we could build a specialized counter. We built the counter.
Your users are telling you what they value. But you have to listen.
Here's the biggest challenge facing AI companies: getting more customers doesn't make your product better. It just creates more opportunities to disappoint people.
Netflix gets better when more people use it. Spotify gets better. Doordash gets better.
But most AI apps? Adding users just adds load without improving quality.
What if having 1,000 users made your recall go up by 20%? You'd have a flywheel. Right now, most companies are just pushing a boulder.
This is why few AI companies have achieved escape velocity. The ones that do figure out how to create virtuous loops where more usage makes their systems smarter in ways users can feel.
You know you've found product-market fit when user growth starts hurting. One client went viral with their AI assistant, hitting 400,000 MRR almost overnight.
The next week? 340,000. Week after? 280,000.
Why? Their product wasn't ready for scale. They blew their first impression with thousands of users who will never come back.
AI products often focus so much on capabilities that they forget fundamentals like stability, speed, and reliability. The most elegant RAG system means nothing if it takes 10 seconds to respond or breaks under load.
I've fired clients when I realized I couldn't help them. Not because their AI challenges were too hard, but because their organizational challenges were too deep.
When a client books meetings with me at 5am while I'm about to board a flight, that's not an AI problem. When a team is rebuilding the same database integration for the third time, that's not an AI problem.
Sometimes the highest-value thing I can do is tell clients: "Your issue isn't that your AI is bad. It's that your company's processes are broken."
And sometimes the best consulting is just giving smart people permission to trust their instincts. Like when an engineer tells me: "The only way forward is to try random permutations of these parameters to see what works," and I say, "Yes, that's called hyperparameter optimization, and it's exactly what you should be doing."
The pattern I see across every client is this: companies spend too much time on capabilities and not enough on connections. They build amazing AI features without connecting them to business outcomes.
The best AI isn't the most impressive in a demo. It's the one that solves a real problem customers will pay for.
When a client asks how they're doing, I don't start with technical metrics. I ask: "Are pilots converting to paid? Are usage numbers climbing? Are customers telling their friends?"
Those are the only metrics that matter.
Remember:
And if all else fails, remember what I tell prospective clients: "If we meet at a party six months from now, I don't want you to say 'Jason helped us improve our AI answers.' I want you to say 'Jason helped us close our Series A.'"
That's the difference between solving AI problems and solving business problems.
If you found this useful, follow me on Twitter @jxnlco for more AI insights. And if you're struggling with your AI strategy, DM me. I might be able to help.
Retrieval-Augmented Generation (RAG) is a simple, powerful idea: attach a large language model (LLM) to external data, and harness better, domain-specific outputs. Yet behind that simplicity lurks a maze of hidden pitfalls: no metrics, no data instrumentation, not even clarity about what exactly we’re trying to improve.
In this mega-long post, I’ll lay out everything I know about systematically improving RAG apps—from fundamental retrieval metrics, to segmentation and classification, to structured extraction, multimodality, fine-tuned embeddings, query routing, and closing the loop with real user feedback. It’s the end-to-end blueprint for building and iterating a RAG system that actually works in production.
I’ve spent years consulting on applied AI—spanning recommendation systems, spam detection, generative search, and RAG. That includes building ML pipelines for large-scale recommendation frameworks, doing vision-based detection, curation of specialized datasets, and more. In short, I’ve seen many “AI fails” up close. Over time, I’ve realized that gluing an LLM to your data is just the first step. The real magic is how you measure, iterate, and keep your system from sliding backward.
We’ll break everything down in a systematic, user-centric way. If you’re tired of random prompt hacks and single-number “accuracy” illusions, you’re in the right place.