How to Ship Your First Generative AI Feature
For Product Managers who want to ship their first GenAI or LLM-powered feature but are unsure where to start.
This article is a working outline for my upcoming course for Product Managers who want to ship their first GenAI or LLM-powered feature but are unsure where to start.
I’ve been in product management for well over a decade & have been using/building with AI/ML since 2015. Here’s my high-level framework for shipping your first LLM-powered feature:
- Identify a problem worth solving that’s a good fit for Gen AI
- Start by solving the problem with prompting (i.e., have a conversation with your favorite frontier model)
- Define the overall UX (chatbot, under the hood, click-to-generate, etc.)
- Define your core evals (i.e., fast & slow automated checks for result Quality)
- Model selection (select the right model for your unit economics & latency needs)
- Create your minimum viable solution on your target model (get from conversation to single prompt, iterating until you pass your fast evals). Protip: don’t rush into fine-tuning or even RAG
- Finalize & implement your slow evals with metrics & monitoring (slow evals, incl. things like user feedback and user retention). Protip: leverage automation & define rituals for reviewing metrics
- Get it in front of users for learning & validation!
- Ship it! 🚢 🎉
Shipping GenAI features isn’t as hard as it might seem, so get out there and build something your customers will love.
Want to go deeper? This article is a working outline for my upcoming course for Product Managers who want to ship their first GenAI or LLM-powered feature but are unsure where to start.