Product Managers, give me 6 minutes to make you understand:
The true meaning of AI Product Sense
In product management, problems are ambiguous, information is limited, 2+2 is not always 4.
Hence, it’s hard to know the “right” answer
Yet some PMs know the "right" thing to do, despite the ambiguity.
And that is product sense:
Ability to find the right solution for the users and the business, despite limited and ambiguous information.
And fundamentally, the definition hasn’t changed (even with AI in the picture). But designing AI products adds a layer extra complexity which requires PMs to answer a few extra questions.
Create/clarify the overall goal
User Discovery
Problem discovery
Solution discovery (this is different with AI)
Alignment with larger goals
----------------------------------
Building and delivering the solution
Measure success and collect feedback
Let’s talk about the added complexity that AI brings with it.
Here are the questions you should be thinking about
1. Which layers do we need: determine what powers the heart of your solution. Is it data, model, UX, Product. Do this early on so you know exactly where to focus your energy on.
2. Define each layer’s job: for all practical reasons your product would use all of the above layers. So it’s important to define what each layer is responsible for and how you define success for each of them. This is imp because when your product breaks at scale, you’ll know exactly where to look.
3. Guardrails: most PMs are very good at defining what the product should do. But with AI, it’s important to also define what it shouldn’t do. You don’t want a customer service chat bot reveal trade secrets or offer full refunds to every customer.
4 and 5 Define good and bad quality: it’s critical to define what good quality looks like, create a golden set, run tests against an EVAL framework so you know how good or bad your product is doing. At the same time also think what bad quality looks like, so your model knows exactly what not to do.
6. Design how user experiences the product: what does the user do when she receives a good response? Does she end the session? Does she start a new session? Similarly, what does the user do when she receives a bad or inaccurate answer? Can she rerun the same query? Can she course correct? All of these are very imp questions to answer before you start building.
7. How will you collect feedback: will you use explicit signals like thumbs up? Or implicit signals like engagement, clicks, time spent? Whatever you choose, you need to know upfront so you design the solution to enable the right feedback loops.
in a few hours I will break this down with a lot more details in my masterclass. We’re going to tear down perplexity as a product and understand how the PMs there answered these questions.
session is free to attend, but seats are limited. See you there (sign up link in comments)
date: 4th april
time: 12pm BST / 430pm IST
