Five teams I have worked alongside shipped software five different ways, and for years each believed its way was simply “how it’s done”. It took a bad quarter and one very expensive industry story to make me sit down and compare them properly. The frame that finally made the comparison fair comes from ReleaseOps, the release-strategy lens that separates installing a build from showing its behaviour to users — because once those are distinct acts, every mechanism can be judged on the same three questions: how fast can it turn back, what does it cost, and when does it fail to warn you?
Why ReleaseOps Insists on the Comparison
The starting point is the distinction itself. Deploying puts a version in place; releasing lets users meet what it does. Fuse the two and shipping is a stunt — everything lands at once and prayer is the monitoring plan. Keep them separate and you gain two independent controls: one decides which version runs where, the other decides who experiences which behaviour. Each of the five mechanisms below is really a different bargain between those controls, which is why no single one wins in every situation.
1. Rolling update — the default that hides its risks
The workhorse. New version replaces old, machine by machine, until the fleet has moved. Nothing extra to buy, which is why it ships with nearly every platform. The bill arrives later: during the transition your tier runs two versions at once, and there is no lever for steering traffic away from a bad build. Undo happens machine by machine too — minutes, not seconds — while users keep arriving at whatever half-migrated state exists.
2. Blue-green — the fast switch with a rent bill
Stand up two complete environments and flip between them. Recovery is measured in seconds, which is why teams with strict downtime tolerance love it. The costs are equally concrete: you pay for double capacity, and your data layer has to serve whichever colour is live — the schema must work for the outgoing version and the incoming one at the same time. Think of it as a before-and-after comparison with no partial traffic: the whole world moves, or none of it does.
3. Canary — a small crowd, watched closely
Send a narrow slice of real users to the new version and watch what happens. Widening takes seconds to minutes when the signal is good, and reversing is just as quick. The catch is signal size: a canary can only detect what its traffic volume makes visible. A quiet service at one percent gives you almost nothing to read, and a ceremonial five-minute glance proves nothing either way. Size the slice and the duration to the change you fear, not to the calendar.
4. Rings and waves — exposure with a soak timer
Widen in groups and let each group bake before the next joins: start with the smallest thing you can actually deploy — a one-box — then an internal cell that behaves like a production shard, then customers in widening circles. The rhythm buys you discovery time; a defect that only shows on day three never reaches ring five. The discipline cost is real, though: bake windows slow headline features and demand honest instrumentation at every stage.
5. Feature flags — instant off, slow interest
Ship code dark and toggle exposure per user, per cohort, per region. Nothing reverses faster: flip the switch, done in seconds. The interest payments arrive as flag debt — dozens of dormant switches nobody dares remove — and as combinatorial risk, because every new flag multiplies the configurations your tests never covered. Flags also only guard the paths they wrap; a defect outside the flagged code pays them no attention.
A ReleaseOps Horror Story That Settled the Argument
When the five teams finally compared notes, the clincher was an old failure, retold in a training cohort that had been working through a structured SRE curriculum. In 2012, a trading firm pushed new code to some servers and left an old flag pointing at a machine that never received the deployment. The flag still contained live power. For forty-five minutes the stray behaviour traded against the market and burned roughly $460 million. Every mechanism on our list was present in that story in miniature: a deployment that missed a server, a flag treated as harmless, no rehearsal for the moment both failed together. Fast toggles and staged rollouts are safety devices, but devices need inspection.
Two mapping notes sharpen the picture before the choice. First, the mechanisms do not all touch the same controls: a flag turns only the exposure side, while canaries and blue-green flips reposition the deployment at the same time as the audience. Second, the decision is per change, not per company. A risky pricing-engine rewrite might travel dark behind a flag, soak through three rings and keep its kill switch for a month, while a typo fix in a footer simply rolls. Treating the menu as per-change gear selection is what experienced teams do differently from teams still searching for the one true method.
Choosing Without Pretending There Is One Answer
Once the five teams laid their habits side by side, the honest conclusion stopped sounding like a verdict and started sounding like a menu:
- Need the plainest path with no extra infrastructure? Roll, and accept mixed versions plus a minutes-scale undo.
- Need seconds-scale reversal and can afford twice the hardware? Blue-green, after checking the schema suits both colours.
- Have real traffic and a change worth measuring? Canary, sized so the feared defect is actually visible.
- Ship to a fleet that teaches you things over days? Rings, with bake time you actually respect.
- Want exposure control per user without redeploying? Flags, with expiry dates written down at birth.
Most mature setups end up combining them: flags for exposure, a canary for measurement, rings for widening. If the vocabulary is new to you, the gentlest on-ramps are a career-first guide to DevOps without overwhelm for the wider context, and — if your interests run toward hardware — this piece on deploying and managing robot fleets, where the same staged-exposure logic applies to physical machines that cannot be rolled back at all. The Ops Atlas entry on ReleaseOps lays out all five mechanisms with their caveats side by side, and it is the fairest treatment of the trade-offs I have found in one place. Revisit the menu whenever the product changes shape, because a bargain that suited ten thousand users rarely suits ten million.
