Building an AI Agent from Scratch

Most tutorials on building AI agents start with a framework — LangChain, CrewAI, AutoGen. They're useful, but they hide what's actually happening. When something breaks, you're debugging abstractions instead of understanding fundamentals.

I wanted to learn how agents really work. So I built one from scratch: a daily planning tool called planmyday. No frameworks. Just Python, the OpenAI API, and a state machine.

Here's what I learned.

Build for yourself

The best side projects solve your own problems.

I built planmyday because I was tired of planning my day in my head. Existing tools felt like forms — static fields to fill out. I wanted something that would ask me questions, push back on unrealistic plans, and remember what I'd told it before.

When you build for yourself, you skip the hardest part of product development: understanding the problem. You don't need user research. You don't guess at features. You feel the friction and fix it.

Every feature in planmyday exists because something annoyed me during actual use. Session context kept disappearing? Added persistence. Tired of re-explaining my schedule? Built a profile that learns over time. Wanted to know if I followed the plan? Added check-ins.

No roadmap. Just friction, noticed and removed.

You don't need a framework

Frameworks like LangChain are powerful, but they're also complex. For learning how agents work, they get in the way.

An agent is simpler than it looks:

1. •A loop — keep running until done
2. •State — track where you are in the process
3. •Decisions — choose what to do next based on state
4. •Memory — persist context between turns

That's it. You can build this with vanilla Python.

state = "idle"

while state != "done":
    if state == "idle":
        goal = get_user_input()
        state = "questions"
    
    elif state == "questions":
        questions = llm.ask_clarifying_questions(goal)
        answers = get_user_input(questions)
        state = "planning"
    
    elif state == "planning":
        plan = llm.generate_plan(goal, answers)
        show_plan(plan)
        state = "feedback"
    
    elif state == "feedback":
        feedback = get_user_input()
        if feedback == "looks good":
            state = "done"
        else:
            plan = llm.refine_plan(plan, feedback)

This is a state machine. The agent doesn't just respond — it decides what to do next based on where it is. That's the difference between an agent and a chatbot.

Chatbot: User says something → LLM responds.

Agent: User says something → Agent decides action → Executes → Updates state → Repeats.

The state machine is what makes it feel like a collaborator instead of a text box.

Structured outputs matter

The other piece that made planmyday reliable: structured outputs.

When you ask an LLM to "return JSON," you're hoping it complies. Sometimes it adds markdown formatting. Sometimes it hallucinates extra fields. Sometimes it just... doesn't.

OpenAI's structured output feature (with Pydantic) solves this:

class Plan:
    schedule: list[ScheduleItem]
    priorities: list[str]
    notes: str

response = openai.responses.parse(
    model="gpt-4o-mini",
    input=messages,
    text_format=Plan
)

plan = response.output  # guaranteed to match schema

No parsing. No regex. No "please format your response as..." prompts. The LLM returns a validated object or fails cleanly. This single pattern made the difference between a frustrating prototype and a tool I could rely on daily.

Publish anyway

I didn't build planmyday for an audience. I built it for my mornings. But I published it anyway.

Publishing forces a level of care that "just for me" never does:

• •You write actual documentation
• •You handle edge cases strangers might hit
• •You think about error messages and installation
• •You clean up the embarrassing parts

Even if no one downloads it, the act of making it public raised my standards. The code got better. The README got clearer. The rough edges got smoothed.

And maybe someone else hates planning their day too.

Start with less

You don't need much to build an AI agent:

• •Python
• •An LLM API (OpenAI, Anthropic, etc.)
• •A state machine (a while loop and some if statements)
• •Persistence (a JSON file works fine)

Skip the frameworks until you understand what they're abstracting. Build something small. Use it yourself. Publish it when it's good enough, not when it's perfect.

The best way to learn how agents work is to build one. planmyday is how I learned.