Technically an A/B test is almost indistinguishable from a feature flag: users are split deterministically into groups and each sees its own variant. The difference isn’t in the rollout mechanics but in the goal — variants carry metrics, and the result is read statistically to separate a real effect from random noise.
It all starts with a hypothesis, not a button. A good one reads like “change X will increase metric Y for segment Z, because…”. Without a success metric chosen in advance, a test turns into staring at charts hunting for anything good — and something almost always “turns up.”
Three things are needed infrastructurally. First, a distribution mechanism: the same deterministic percentage rollout as flags, only across N variants. Second, exposure logging: the fact “user U saw variant B” must be recorded, or there is nothing to tie behaviour to. Third, a “variant ↔ metric” link: events (click, purchase, retention) must map to the assigned variant.
Then statistics. You need a sample size and test duration estimated in advance: a small sample yields a “result” that falls apart the next week. The classic mistake is peeking — stopping the test the moment a number happens to look pretty. A fixed horizon or sequential testing fixes it.
Separately, you track guardrail metrics — numbers that must not drop even if the main metric rises: latency, errors, unsubscribes, revenue. A variant that lifted clicks but hurt retention is not a win.
It’s worth remembering the effects that distort the picture: the novelty effect (people react to the new simply because it’s new), seasonality, and interference between tests running at once. So experiments are isolated, and you don’t run a dozen overlapping ones on the same audience.
In practice, A/B infrastructure on Unleash gave teams a way to decide on data rather than on opinions and on who argues loudest — and lifted user satisfaction by about 15%. The discipline is simple: state the hypothesis in advance, wait for significance, and respect the guardrail metrics.