Feature Flag Observability: Don't Wing It
November 05, 2019

Dave Karow
Split Software

The benefits of a feature flag rollout are clear to many DevOps teams. The ability to control the gradual release of a feature, and then be able to retract the feature without having to restart the entire release, has made the job of development and operations teams a lot easier.

A solid feature flag solution provides a standardized, scalable and transparent way to accomplish that across all of your projects. While that may sound like a dream come true, there's more to consider when implementing feature flags than just whether or not you need to retract a faulty feature.
 
The ability to gradually roll out a feature flag means that careful consideration is needed when determining when to ramp it up. Deciding when is the right time to release the feature to more users is a process that should rely heavily on data. To do that, you must identify what you are looking for in the data.

Teams at some of the leading tech companies that have built amazing in-house systems for control and observation with feature flags have discovered that building in fine-grained control and automated observability into their continuous delivery practices is as critical as maintaining version control or implementing continuous integration.
 
Surprisingly, even though these teams saw the benefit of built-in control and automated observability many of them do not have a structured and repeatable way to detect the impact of the rollouts. Many companies were "winging it" with ad-hoc exploration or they were waiting for some global health metric results.
 
Ad hoc exploration is too random of a process to specifically identify when it is correct to allow more users to access the new feature. This process is also very susceptible to observation bias — the inclination to see the results we expect to see or want to see. With the aim of gradual rollouts with feature flags being to gather data on new code and determine whether it accomplishes set goals (such as increased engagement and more satisfied customers), ad hoc exploration does not allow for the data to be carefully analyzed.
 
The main question that should be asked when launching a new feature is "Does my new code make any difference?"

An additional question should be, "Is the difference positive and does it impact users without impacting other factors such as response time, errors, unsubscribes and bad app reviews?"
 
A feature flag allows you to make the new feature accessible to a select group of your users, thus making it easier to determine if the new feature causes any change in the user experience, such as response time, error rates and app crashes. Being able to compare the group that has the new feature and the group that doesn't makes it a lot easier to determine the success of the new feature. But how you correctly identify and analyze the difference is important.
 
One way to go about this is to utilize auto-calculating. Auto-calculating impact metrics is the process of using the same system that is controlling the rollout to keep track of how well it is doing. It is important to do it in a way that is repeatable, consistent and scalable.

The key to this process is to watch it through the lens of the control panel, the instrument that is deciding which users get access to the new code. While the inputs can come from anywhere and can be a mix of existing and new instrumentation, watching each cohort through this lens makes it much easier to get an early warning than if you watch your global metric, which will only bring a negative symptom to your attention if the impact is catastrophic. Separating the users into one group with the new code and one group without it, and then observing them separately, allows the system to detect less massive impacts, which you then can compare.
 
Overall, it's important to determine the correct metrics to analyze your data from the feature release against, and then it is great practice to utilize a system that can examine the data in the two groups separately (some companies have dubbed this an experimentation platform) allowing you to better determine when is best to ramp up the release and when it's time to roll it back and hit the drawing board.

Dave Karow is the Continuous Delivery Evangelist at Split Software
Share this

Industry News

January 27, 2021

Indigo.Design announced the public preview of Indigo.Design App Builder.

January 27, 2021

ARMO announced its launch out of stealth having secured $4.5 million in seed funding from Pitango First.

January 27, 2021

CloudSphere announced the appointment of Jane Gilson as the company’s CEO successor to Patrick McNally.

January 26, 2021

JFrog announced an agreement with Docker.

January 26, 2021

SUSE released Longhorn 1.1.

January 26, 2021

HAProxy released HAProxy Kubernetes Ingress Controller 1.5.

January 25, 2021

Progress announced the new release of Progress Kendo UI, a complete collection of JavaScript UI components.

January 25, 2021

CloudNatix announced the close of a $4.5M Seed round financing led by DNX Ventures, with the participation from a new investor Cota Capital and existing investors: Incubate Fund, Vela Partners and 468 Capital.

January 25, 2021

Quali announced $54 million in new funding, co-led by Greenfield Partners and JVP.

January 21, 2021

Platform9 released Platform9 Release 5.0, with a number of new features to provide operational efficiencies for its freedom, growth, and enterprise managed Kubernetes products.

January 21, 2021

Infragistics announced the release of Infragistics Ultimate 20.2, a complete UX and UI solution for  design and development teams  which is fully compatible with .NET 5, Microsoft’s latest  release of .NET development platform.

January 21, 2021

Couchbase Cloud is now available on Microsoft Azure.

January 20, 2021

Hitachi Vantara announced the availability of Hitachi Kubernetes Service, enabling customers to consistently and securely deploy, manage, monitor, and govern Kubernetes clusters across major cloud providers and on premises.

January 20, 2021

Internal announced the launch of an enterprise-ready app development platform for internal tools.

January 20, 2021

StackPulse announced a $20 million Series A led by GGV Capital.