How to Ship Your First Generative AI Feature

For Product Managers who want to ship their first GenAI or LLM-powered feature but are unsure where to start.

Bryce York
2 min readSep 30, 2024

--

This article is a working outline for my upcoming course for Product Managers who want to ship their first GenAI or LLM-powered feature but are unsure where to start.

Join The Waitlist

I’ve been in product management for well over a decade & have been using/building with AI/ML since 2015. Here’s my high-level framework for shipping your first LLM-powered feature:

  1. Identify a problem worth solving that’s a good fit for Gen AI
  2. Start by solving the problem with prompting (i.e., have a conversation with your favorite frontier model)
  3. Define the overall UX (chatbot, under the hood, click-to-generate, etc.)
  4. Define your core evals (i.e., fast & slow automated checks for result Quality)
  5. Model selection (select the right model for your unit economics & latency needs)
  6. Create your minimum viable solution on your target model (get from conversation to single prompt, iterating until you pass your fast evals). Protip: don’t rush into fine-tuning or even RAG
  7. Finalize & implement your slow evals with metrics & monitoring (slow evals, incl. things like user feedback and user retention). Protip: leverage automation & define rituals for reviewing metrics
  8. Get it in front of users for learning & validation!
  9. Ship it! 🚢 🎉

Shipping GenAI features isn’t as hard as it might seem, so get out there and build something your customers will love.

Want to go deeper? This article is a working outline for my upcoming course for Product Managers who want to ship their first GenAI or LLM-powered feature but are unsure where to start.

Join The Waitlist

--

--

Bryce York
Bryce York

Written by Bryce York

Startup Product Leader • Product (management) geek sharing hard-learned lessons from 10+ yrs in early-stage & growth-stage startups