2025¶

10 “Foot Guns" for Fine-Tuning and Few-Shots

Let me share a story that might sound familiar.

A few months back, I was helping a Series A startup with their LLM deployment. Their CTO pulled me aside and said, "Jason, we're burning through our OpenAI credits like crazy, and our responses are still inconsistent. We thought fine-tuning would solve everything, but now we're knee-deep in training data issues."

Fast forward to today, and I’ve been diving deep into these challenges as an advisor to Zenbase, a production level version of DSPY. We’re on a mission to help companies get the most out of their AI investments. Think of them as your AI optimization guides, they've been through the trenches, made the mistakes, and now we’re here to help you avoid them.

In this post, I’ll walk you through some of the biggest pitfalls. I’ll share real stories, practical solutions, and lessons learned from working with dozens of companies.

Making Money is Negative Margin

In 2020 I had a hand injury that ended my career for 2-3 years. I've only managed to bounce back into being an indie consultant and educator. On the way back to being a productive member of society I've learned a few things:

1. I have what it takes to be successful, whether that's the feeling of never wanting to be poor again, or some internal motivation, or the 'cares a lot' or the 'chip on the shoulder' - whatever it is, I believe I will be successful
2. The gift of being enough is the greatest gift I can give myself
3. I will likely make too many sacrifices by default, not too few, and it will reflect in my regrets later in life

Prompt Template Resource System Specification

Overview

A token-efficient system for referencing external resources in LLM prompts without including their full content, designed to optimize token usage when LLMs generate template calls.

Template Definition Syntax

@template
def template_name(param1, param2, ...):
    # I recognize that these should really be chat messages
    return Template("""
    Template content with placeholders:

    <param1>
    {{param1}}
    </param1>

    <param2>
    {{param2}}
    </param2>
    """)

Resource Reference Types

Template Usage

# Basic usage with mixed resource types
response = template_name(
    param1="file://path/to/resource.txt",
    param2="This is direct string content"
)

# Using various tagged output references
response = template_name(
    param1="context://artifact#summary-12345",
    param2="context://thought#reasoning-67890",
    param3="context://candidate-profile"
)

Tagged Outputs & Memory Management

XML Tag Creation and Reference

Any XML tag can be used to create referenceable content. Examples:

<artifact id="summary-123">
Professional developer with 10 years experience...
</artifact>

<thought id="reasoning-456">
This candidate has strong technical skills but limited management experience.
</thought>

<response id="feedback-789">
Your solution correctly implements the algorithm but could be optimized further.
</response>

Reference in subsequent calls:

create_notion_page(title=str, body="context://artifact#summary-123")
follow_up_question(reasoning="context://thought#reasoning-456")
email_template(feedback="context://response#feedback-789")

Error Handling

Comments

Generally, I think here that if we can just save XML tagged data as resources and get names back out, that we can pass them around as context in a way that's more productive.