Worked at a place with a literal settings table of 5K (non-default) records, it was many hundreds of possible settings.
I also seen condensed flags where the flag record embedded which IDs were activated, with a massive overwrite risk anytime someone touched any setting.
Also flag-averse modules where there was so much magic that one one could ever understand exactly what would happen until they loaded up a real or similarly structured set of records. And they didn't trust their results for more than a month or so because changes were frequent as things needed to evolve. The case for this craziness is that too many flags means folks will overlook things, or we'll forget to make the new shiny on-by-default after the roll out period.
IME there is a balance between every feature and code path getting a flag and nothing ever does. Of course the flags themselves introduce complexity and risk. And there's the work to remove them with the vestiges and QA that change for regressions.
I never found a great way to incorporate feature flags into my workflow without inducing significant mental churn managing 12-step rollouts over a dozen independent active flags. The changes I tend to make are sweeping, non-trivial refactors of base libraries with hundreds or possibly thousands of callers. These sorts of changes are exceptionally hard to flag (especially API changes), and it's stupidly easy for another developer to fat-finger a merge conflict resolution and drop one of my flag gates.
To this day I've never found any good guidelines for flagging changes like this without resorting to widespread file duplication, abuse of OOP, or an ad-hoc versioning system. It comes as no surprise that nobody in my organization was willing to do this sort of work.
In my old org the policy was to flag every change, not just new features and behaviors. I'd even argue that refactoring is often a higher risk than adding a new feature path, because the scope of a refactor can touch nearly every user journey. So I can understand why the policy applies to refactoring. I'm just lamenting that this doesn't seem to be well-trodden ground, and being a person who cares deeply about reducing system complexity makes this whole thing kind of a bummer.
Feature flags are great. Working on a crash / memory corruption in an optional subsystem? Just disable the subsystem to unblock coworkers on the same branch while you track down the cause.
Feature flags are terrible. Working on a crash / memory corruption in an optional subsystem? You disabled it previously for your local build, and you'll lose hours failing to repro despite QA giving excellent repro steps.
(For my own gamedev background, I learned to mute audio by setting volume to 0 instead of by disabling the audio subsystem.)
I've been introducing feature flags into our component at work.
The reason why rollbacks isn't sufficient for us is that our service is semi-stateful (postgres connections are stateful, we proxy those connections). Because of this, we always keep around old pods for 5 days to let connections drain.
A deploy+rollback ends up with 3x the pods lying around, and if we deploy a fix patch that's now 4x - and if we don't deploy the fix we have 2 weeks of changes piled up for the next release.
Because of this, we instead use the feature flag. We can toggle it on and off very quickly for risky changes, and it makes no changes to pod counts
The design is postgres. It's our customers that do stateful things on those connections. Unfortunately it's tools like pgdump/pgrestore that are not tolerant to connection interruption.
Node failure is indeed a problem. Not much we can do there. Fortunately it's rare enough historically.
We are working to split the code so that the process that holds tcp connections isn't the process that handles the logic though, so it's easier to update inplace
What does it mean to do “stateful things on a connection”? Feel free to point me to literature if you’d be repeating something already written or well known.
I agree with the entire premise but I do think the cost argument is a bit overblown. Adding "unnecessary" feature flags isn't really that big of a deal imo, feature flags are cheap to add and maintain. Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.
I think the true cost is that feature flags can cause code bloat and readability issues, since engineers typically aren't great about cleaning up feature flags after things have been rolled out. I think that's an easily solvable problem that doesn't really necessitate a scarcity mindset of "just use less feature flags / only when necessary" though. LaunchDarkly makes it pretty easy to track feature flag usage and remind people to clean up old ones.
> Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.
I would argue that if flipping a feature flag isn't faster than doing a rollback, your feature flags are not functional. They should be essentially instantaneous and the first thing to reach for as soon as you've identified a problem.
> Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.
This I totally agree with. Using feature flags well requires some discipline on the team's part to stay on top of them and remove them once they've served their purpose.
> (...) feature flags are cheap to add and maintain.
Not quite. When you add a feature flag, you now have a system that has N separate code flows that you need to verify before launch,and then have to modify and retest to remove the feature flag. A feature flag ends up doubling the workload to test and verify the feature.
> Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.
This is perhaps the most relevant point that the blogger misses. Feature flags are runtime switches that can extend simultaneously to multiple systems and even clients. You can't pull features from clients with a release, where week-long release processss can barely get you a 60% uptake. Redeploying away features in the backend can easily take half an hour, specially if it's a system that does multi region rolling deployments.
If you want instant switches, your best option is switches flipped at runtime. That's what a feature flag is.
"At Google, our philosophy is that “rollbacks are normal.” When an error is found or reasonably suspected in a new release, the releasing team rolls back first and investigates the problem second"
Well this all makes sense now. My enterprise laid off most of our QA team and I see a constant stream of rollback announcements. Who needs regression testing anymore?
That's a good point about not using feature flags to mitigate risk, and how rollbacks are a better alternative. Teams need to be in the habit of performing a rollback though. Sometimes, the rollback process can be black magic if the engineer handling an incident isn't familiar with that process. Having a bunch of flags in a system is a great way to end up with nondeterministic errors.
And that brings us to another great point, which is too many flags is problematic. So often there's an excuse made in the nature of, "we'll go back and remove this later," but later never comes.
At the end of the day, it's rigor that separates good teams from bad teams. Good teams will rigorously review old code and remove it; it's all too easy to do the opposite.
> That's a good point about not using feature flags to mitigate risk, and how rollbacks are a better alternative. T
I doubt anyone making this sort of claim has any professional experience maintaining any sort of user facing software.
Features that require cross-system support can't easily be pulled out with a revert, particularly in CICD systems where cherry picking a revert can easily be incomplete/miss a bug fix and reverting the whole history will end up inadvertently pulling out some other feature.
Also, a deployment takes time, particularly in services with more than one deployment region. Moreso if n>1 separate systems are involved.
In comparison, runtime config changes are enforced in a fraction of the time, and pulling out the code just involves a roll forward. Simple.
> Both Google and Amazon do rollbacks instead of feature flag switches.
I'm sorry, but you're completely wrong. I know for a fact that Amazon uses extensively a A/B testing system for feature flags. Others in this discussion already mentioned it.
I’m not wrong. They do use feature flags, but not as a substitute for rolling back failed or buggy deployments. They (and Google too) use feature flags in the ways according to the best practices discussed in the article. Google also covers this well in their highly-regarded Site Reliability Engineering book.
> I liked the article. Every feature flag contributes to hockey stick growth of version variations of your software.
This is specious reasoning. When you add a feature flag, your goal is to purposely introduce version variations. How come is your explicit goal framed as this sneaky gotcha?
And let's be serious for a second. Do you see this "hockey stick growth" as an issue when you look at user settings?
> So cognitive overload is unavoidable after feature flags 5 as you have so many permutations.
Not really. Feature flags have a life cycle which is managed by release managers, and the goal is to always get rid of them asap. They are introduced to manage the risk of a rollout, and when the transition is over you pull the feature flags. Done.
Some things that initially seem like benefits at a small scale become less so at large scale. Owning one cat can be lots of fun. Owning 50 cats, not so much.
Everyone intends to collect the garbage. Few actually do.
Or because rollbacks (and release) are slower than flipping a flag.
I’d rather invest in the feature flag upfront so I can unbreak customers in 2min rather than telling them to wait hours for the rollback.
Why are rollback and release so slow? Because we’re deploying to 40+ availability zones world-wide, totaling many thousands of servers, and we don’t want to worsen the impact by doing an accelerated “everything-at-once” rollback.
As an SRE / DBRE, I hate FFs because it means I can be lulled into a false sense of security when something I saw didn’t initially fail, only to start doing so the next week.
Depending on the exact specifics of what you mean by "canary", they have their own issues.
If you mean "deploy the next version gradually, slowing rolling more traffic to it": this puts an upper bound on deployment velocity: whatever latency you have from getting a canary from 0% to 100%, that then determines how quickly you can deploy. The "off" state of a feature flag is usually the pre-existing code, and while nothing is guaranteed to be 100% safe, it's usually a good bet. Then we can worry (post deploy) about ramping that FF from 0% to 100%. It raises the number of steps per deployment, too. (It's not just deploy, it's deploy canary, canary to 10%, canary to 20%, etc., make prod canary. I have seen each of those steps then get bogged down by people needing to have their nerves managed.)
If you mean something like "canary that feature branch" (which I have seen, more rarely) — that style has problems in "how do you keep the feature branch up to date with `main`/`prod`?" (or whatever you call your currently deployed version.) Basically, if I canary a feature branch — and lets say I need to make modifications, and those take time — and then prod is deployed to a new version later than my branch, now that new version has code that, if the user hits my canary, my canary lacks. (And yes, I've seen this in real companies, where a engineer ends up confused as to why their request is failing, b/c it is hitting a canary that is out of date / diverged from prod.)
> If you mean "deploy the next version gradually, slowing rolling more traffic to it": this puts an upper bound on deployment velocity
Yes, that’s the point. I have never, not once, seen a place that emphasized high development velocity that also had anything resembling stability.
Note also that I mentioned I am an SRE / DBRE: it’s baffling and frustrating to me that companies will hire people whose job it is to create stable and reliable systems, then ignore them when they say “you’re moving too quickly.” Instead, we get treated as S-tier helpdesk.
Finally, to the mention of extra steps, it’s not that hard to automate. N ReplicaSets get rolled to the new release; if after M minutes all metrics are nominal, ramp up, else roll back and page.
> Risk management should indeed be a priority for all teams. But there are better ways of doing this than relying on feature flags, especially if your team has control over its own deployments. The vast majority of your bugs should be caught by your automated test suite and/or QA process. And the last few stragglers should be handled using incremental deployments, production alarms and rollbacks.
There is some confusion in this post. For risk management, feature flags are used to mitigate the risk of issues not being caught by the QA process. Wishing for a QA process to be 100% flawless doesn't make it so.
Also, guess what an incremental deployment is.
I think the blogger tried to make sweeping statements but didn't actually spent any time thinking through the arguments.
Another issue is launching behind feature flags often skips integration/e2e testing, which couldn't possibly test with every combination of every flag. So your release which is "safer" behind a flag is actually untested until it's launched.
> So your release which is "safer" behind a flag is actually untested until it's launched.
It's only untested if your intention is to not test it.
There is nothing preventing you from testing a feature flag in e2e tests. In fact, testing is perhaps the primary reason why user overrides are supported by feature flag systems.
This sort of combinatorical complexity only really exists in theory. The overwhelmingly common case is that FF A and FF B are independent, and can be successfully tested independently. You test FF A on & off, FF B on & off; your tests grow linearly with your FF set (which you should be pruning as soon as the feature is deployed).
Also, since the original comment stated,
> often skips integration/e2e testing
They're probably responding with the expectation of this being compared to doing no testing at all. Merely assuming your FFs are independent, and testing them as such, is greater than nothing at all, even if the rare correlation might exist.
And where a correlation might exist, well, that's what your judgement & expertise is for. Absolutely test that if you find yourself in that case.
> how? if you have 100 feature flags in your system each with 3 potential states, which states do you run in your e2es?
You're going way out of your way to imagine problems where there are none. Feature flags are ephemeral and work as an ad-hoc release toggled at runtime. You create a feature flag, you commit the changes you need to commit behind the feature flag. Before switching the feature flag you run tests in preparation for the feature flag release, an finally you flip the flag. After that point, you either rollback the feature flag or you remove the feature flag.
I also seen condensed flags where the flag record embedded which IDs were activated, with a massive overwrite risk anytime someone touched any setting.
Also flag-averse modules where there was so much magic that one one could ever understand exactly what would happen until they loaded up a real or similarly structured set of records. And they didn't trust their results for more than a month or so because changes were frequent as things needed to evolve. The case for this craziness is that too many flags means folks will overlook things, or we'll forget to make the new shiny on-by-default after the roll out period.
IME there is a balance between every feature and code path getting a flag and nothing ever does. Of course the flags themselves introduce complexity and risk. And there's the work to remove them with the vestiges and QA that change for regressions.
To this day I've never found any good guidelines for flagging changes like this without resorting to widespread file duplication, abuse of OOP, or an ad-hoc versioning system. It comes as no surprise that nobody in my organization was willing to do this sort of work.
Feature flags are terrible. Working on a crash / memory corruption in an optional subsystem? You disabled it previously for your local build, and you'll lose hours failing to repro despite QA giving excellent repro steps.
(For my own gamedev background, I learned to mute audio by setting volume to 0 instead of by disabling the audio subsystem.)
The reason why rollbacks isn't sufficient for us is that our service is semi-stateful (postgres connections are stateful, we proxy those connections). Because of this, we always keep around old pods for 5 days to let connections drain.
A deploy+rollback ends up with 3x the pods lying around, and if we deploy a fix patch that's now 4x - and if we don't deploy the fix we have 2 weeks of changes piled up for the next release.
Because of this, we instead use the feature flag. We can toggle it on and off very quickly for risky changes, and it makes no changes to pod counts
This design seems rather brittle. What happens if a node spontaneously fails, or you need to move to another AZ? (Assuming you’re in the cloud.)
Node failure is indeed a problem. Not much we can do there. Fortunately it's rare enough historically.
We are working to split the code so that the process that holds tcp connections isn't the process that handles the logic though, so it's easier to update inplace
I think the true cost is that feature flags can cause code bloat and readability issues, since engineers typically aren't great about cleaning up feature flags after things have been rolled out. I think that's an easily solvable problem that doesn't really necessitate a scarcity mindset of "just use less feature flags / only when necessary" though. LaunchDarkly makes it pretty easy to track feature flag usage and remind people to clean up old ones.
I would argue that if flipping a feature flag isn't faster than doing a rollback, your feature flags are not functional. They should be essentially instantaneous and the first thing to reach for as soon as you've identified a problem.
> Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.
This I totally agree with. Using feature flags well requires some discipline on the team's part to stay on top of them and remove them once they've served their purpose.
Not quite. When you add a feature flag, you now have a system that has N separate code flows that you need to verify before launch,and then have to modify and retest to remove the feature flag. A feature flag ends up doubling the workload to test and verify the feature.
> Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.
This is perhaps the most relevant point that the blogger misses. Feature flags are runtime switches that can extend simultaneously to multiple systems and even clients. You can't pull features from clients with a release, where week-long release processss can barely get you a 60% uptake. Redeploying away features in the backend can easily take half an hour, specially if it's a system that does multi region rolling deployments.
If you want instant switches, your best option is switches flipped at runtime. That's what a feature flag is.
Well this all makes sense now. My enterprise laid off most of our QA team and I see a constant stream of rollback announcements. Who needs regression testing anymore?
And that brings us to another great point, which is too many flags is problematic. So often there's an excuse made in the nature of, "we'll go back and remove this later," but later never comes.
At the end of the day, it's rigor that separates good teams from bad teams. Good teams will rigorously review old code and remove it; it's all too easy to do the opposite.
I doubt anyone making this sort of claim has any professional experience maintaining any sort of user facing software.
Features that require cross-system support can't easily be pulled out with a revert, particularly in CICD systems where cherry picking a revert can easily be incomplete/miss a bug fix and reverting the whole history will end up inadvertently pulling out some other feature.
Also, a deployment takes time, particularly in services with more than one deployment region. Moreso if n>1 separate systems are involved.
In comparison, runtime config changes are enforced in a fraction of the time, and pulling out the code just involves a roll forward. Simple.
I'm sorry, but you're completely wrong. I know for a fact that Amazon uses extensively a A/B testing system for feature flags. Others in this discussion already mentioned it.
Source: worked at AWS for 7 years.
And knock it off with the attitude.
1 feature flag = 2 behaviors 2 feature flags = 4 behaviors
So cognitive overload is unavoidable after feature flags 5 as you have so many permutations.
This is specious reasoning. When you add a feature flag, your goal is to purposely introduce version variations. How come is your explicit goal framed as this sneaky gotcha?
And let's be serious for a second. Do you see this "hockey stick growth" as an issue when you look at user settings?
> So cognitive overload is unavoidable after feature flags 5 as you have so many permutations.
Not really. Feature flags have a life cycle which is managed by release managers, and the goal is to always get rid of them asap. They are introduced to manage the risk of a rollout, and when the transition is over you pull the feature flags. Done.
Everyone intends to collect the garbage. Few actually do.
I’d rather invest in the feature flag upfront so I can unbreak customers in 2min rather than telling them to wait hours for the rollback.
Why are rollback and release so slow? Because we’re deploying to 40+ availability zones world-wide, totaling many thousands of servers, and we don’t want to worsen the impact by doing an accelerated “everything-at-once” rollback.
Just use canaries, I’m begging you.
If you mean "deploy the next version gradually, slowing rolling more traffic to it": this puts an upper bound on deployment velocity: whatever latency you have from getting a canary from 0% to 100%, that then determines how quickly you can deploy. The "off" state of a feature flag is usually the pre-existing code, and while nothing is guaranteed to be 100% safe, it's usually a good bet. Then we can worry (post deploy) about ramping that FF from 0% to 100%. It raises the number of steps per deployment, too. (It's not just deploy, it's deploy canary, canary to 10%, canary to 20%, etc., make prod canary. I have seen each of those steps then get bogged down by people needing to have their nerves managed.)
If you mean something like "canary that feature branch" (which I have seen, more rarely) — that style has problems in "how do you keep the feature branch up to date with `main`/`prod`?" (or whatever you call your currently deployed version.) Basically, if I canary a feature branch — and lets say I need to make modifications, and those take time — and then prod is deployed to a new version later than my branch, now that new version has code that, if the user hits my canary, my canary lacks. (And yes, I've seen this in real companies, where a engineer ends up confused as to why their request is failing, b/c it is hitting a canary that is out of date / diverged from prod.)
Yes, that’s the point. I have never, not once, seen a place that emphasized high development velocity that also had anything resembling stability.
Note also that I mentioned I am an SRE / DBRE: it’s baffling and frustrating to me that companies will hire people whose job it is to create stable and reliable systems, then ignore them when they say “you’re moving too quickly.” Instead, we get treated as S-tier helpdesk.
Finally, to the mention of extra steps, it’s not that hard to automate. N ReplicaSets get rolled to the new release; if after M minutes all metrics are nominal, ramp up, else roll back and page.
I imagine that's got its own risks and challenges if there is a lot of bottlenecks or dependencies, like multi-tenant data store(s).
There is some confusion in this post. For risk management, feature flags are used to mitigate the risk of issues not being caught by the QA process. Wishing for a QA process to be 100% flawless doesn't make it so.
Also, guess what an incremental deployment is.
I think the blogger tried to make sweeping statements but didn't actually spent any time thinking through the arguments.
It's only untested if your intention is to not test it.
There is nothing preventing you from testing a feature flag in e2e tests. In fact, testing is perhaps the primary reason why user overrides are supported by feature flag systems.
Also, since the original comment stated,
> often skips integration/e2e testing
They're probably responding with the expectation of this being compared to doing no testing at all. Merely assuming your FFs are independent, and testing them as such, is greater than nothing at all, even if the rare correlation might exist.
And where a correlation might exist, well, that's what your judgement & expertise is for. Absolutely test that if you find yourself in that case.
You're going way out of your way to imagine problems where there are none. Feature flags are ephemeral and work as an ad-hoc release toggled at runtime. You create a feature flag, you commit the changes you need to commit behind the feature flag. Before switching the feature flag you run tests in preparation for the feature flag release, an finally you flip the flag. After that point, you either rollback the feature flag or you remove the feature flag.
This is not rocket science.