You’ve spent months improving checkout. Your team finally has a winner: a 4% lift in conversion.
There’s one problem.
On older Android phones, one in every 500 orders results in a duplicate charge. The system catches the error and automatically refunds the customer within 24 hours.
Your team is ready to launch. The conversion improvement is real. The bug affects a small percentage of orders, and there’s a fix on the roadmap.
You’re making the launch call.
Ship it. The conversion lift outweighs a rare error that’s automatically corrected.
Limit the rollout. Exclude the affected Android devices and launch everywhere else.
Hold it. Fix the duplicate charges before anyone gets the new checkout.
Which would you choose?
Before you settle on an answer, there’s one more detail.
The order is for $5,000.
A customer could see $10,000 charged to their account. They’ll get the extra $5,000 back within 24 hours, but what if their rent or another payment is due in the meantime?
Would you still make the same call?
I put this decision to a group of product leaders during a workshop on Product Quality. Some wanted to hold the launch. Others argued for excluding older Android devices. A few thought the conversion improvement justified shipping.
As we discussed their choices, the details started to matter. The amount of money involved. The type of customer. The consequences of waiting for a refund.
And this was a relatively contained problem. We knew which devices were affected, how often the bug occurred, and how to reverse the charge.
What happens when the product’s failures are harder to predict or contain?
That’s where AI changes the quality conversation.
I gave the group two more scenarios: an AI travel planner that sometimes recommends closed restaurants, and an AI support agent that resolves most customer issues but performs worse than a human.
Would you ship either one?
A closed restaurant might mean walking around the corner for another option. It might also mean arriving hungry after a 20-mile drive. An AI support agent might save customers time, but only if the problems it can’t solve don’t leave them stuck.
The group had to weigh the same kinds of tradeoffs without the clear boundaries of the checkout bug.
As AI makes building and shipping software faster, these decisions are becoming more important. A product can perform well on average while creating experiences that customers find unacceptable.
Keep reading for:
Why did Digg’s redesign send users toward Reddit?
How did Chrome turn reliability into an advantage over Internet Explorer?
What did Zoom understand about quality that Skype had lost sight of?
Should you ship a checkout that sometimes charges customers twice?
Is a warning enough to ship an AI travel planner with wrong recommendations?
Is an AI support agent ready if it can hand off to a human?
Why isn’t a successful AI demo enough?
How do you build a living quality system for AI?
What can you cut from scope to protect quality?
How do you define “good enough”?
How will you know when quality slips after launch?
How do failures improve your next decision?
How do you measure Product Quality?
How do you help your team get better at raising Product Quality?
How do you develop the judgment to make the call?
The full breakdown, workshop replay, and slides are included with your paid subscription. If you’re already a paid subscriber, they’re all below. Otherwise, upgrade to get access. The annual plan is available for just $180 until 10/31.
