How Amplitude ships product improvements in 1–2 days with OpenAI

Amplitude, the product analytics company, says its Wave agent can take a team from spotting a product problem to shipping a fix in one to two days, down from roughly a month. Wave pulls together analytics, session replays, customer feedback and code, proposes changes with technical plans, sends them to coding agents including Codex, then measures the result.

This is an OpenAI customer story, so it's partly marketing. But it's a clear example of AI agents doing a full loop of product work, from noticing what users actually do to testing a change against a control group.

We like this one more than most vendor case studies, because the example is small and checkable. On Amplitude's own homepage, Wave looked at 100 session replays and saw that 80 users typed their own questions while only 12 used the suggestion cards. It proposed personalizing the cards based on recent charts, dashboards or the page someone came from, while keeping free typing and generic fallbacks. Engineers reviewed it, a coding agent built it, and an A/B test showed a 160% lift in messages sent through the cards after one day.

That's useful AI. It found a modest fix nobody had prioritized and tested it properly. One of Amplitude's engineers called it the kind of improvement that just gets ignored. Most product backlogs are full of those.

Now the caveats. A one-day experiment is short, and a 160% lift on a little-used feature can be a small absolute change. The model comparisons come from Amplitude's own evaluations in an OpenAI-published story. The results are mixed, too. On the Data Science Agent test, OpenAI's model scored slightly lower than a rival, 67.5 versus 69.3, though at about half the cost per run. We'd treat these as one company's internal results, not general benchmarks.

The bigger question is where this leads. Amplitude says most teams send bug fixes straight to coding agents while new features get human review. That split seems right. If self-improving products become normal, the risk is optimizing hard for what's easy to measure, like clicks, engagement and conversion, while things that are hard to measure, like trust or user wellbeing, get less attention. An agent that ships a change the same day it spots an opportunity can also ship a manipulative change fast.

I'm encouraged by the design choices here, including model routing by task and humans approving bigger bets. I'd like to see teams set clear limits on what the agent can optimize for before scaling it.

Amplitude says Wave is roughly 80% powered by OpenAI models. It uses a heavier reasoning model for investigations, a cheaper model for high-volume checks, and an image model for mockups. Wave is in beta with customers. Among customer-launched changes, Amplitude says one in four went live the same day Wave surfaced the opportunity.