Feature Flag Observability: Don't Wing It
November 05, 2019

Dave Karow
Split Software

The benefits of a feature flag rollout are clear to many DevOps teams. The ability to control the gradual release of a feature, and then be able to retract the feature without having to restart the entire release, has made the job of development and operations teams a lot easier.

A solid feature flag solution provides a standardized, scalable and transparent way to accomplish that across all of your projects. While that may sound like a dream come true, there's more to consider when implementing feature flags than just whether or not you need to retract a faulty feature.
 
The ability to gradually roll out a feature flag means that careful consideration is needed when determining when to ramp it up. Deciding when is the right time to release the feature to more users is a process that should rely heavily on data. To do that, you must identify what you are looking for in the data.

Teams at some of the leading tech companies that have built amazing in-house systems for control and observation with feature flags have discovered that building in fine-grained control and automated observability into their continuous delivery practices is as critical as maintaining version control or implementing continuous integration.
 
Surprisingly, even though these teams saw the benefit of built-in control and automated observability many of them do not have a structured and repeatable way to detect the impact of the rollouts. Many companies were "winging it" with ad-hoc exploration or they were waiting for some global health metric results.
 
Ad hoc exploration is too random of a process to specifically identify when it is correct to allow more users to access the new feature. This process is also very susceptible to observation bias — the inclination to see the results we expect to see or want to see. With the aim of gradual rollouts with feature flags being to gather data on new code and determine whether it accomplishes set goals (such as increased engagement and more satisfied customers), ad hoc exploration does not allow for the data to be carefully analyzed.
 
The main question that should be asked when launching a new feature is "Does my new code make any difference?"

An additional question should be, "Is the difference positive and does it impact users without impacting other factors such as response time, errors, unsubscribes and bad app reviews?"
 
A feature flag allows you to make the new feature accessible to a select group of your users, thus making it easier to determine if the new feature causes any change in the user experience, such as response time, error rates and app crashes. Being able to compare the group that has the new feature and the group that doesn't makes it a lot easier to determine the success of the new feature. But how you correctly identify and analyze the difference is important.
 
One way to go about this is to utilize auto-calculating. Auto-calculating impact metrics is the process of using the same system that is controlling the rollout to keep track of how well it is doing. It is important to do it in a way that is repeatable, consistent and scalable.

The key to this process is to watch it through the lens of the control panel, the instrument that is deciding which users get access to the new code. While the inputs can come from anywhere and can be a mix of existing and new instrumentation, watching each cohort through this lens makes it much easier to get an early warning than if you watch your global metric, which will only bring a negative symptom to your attention if the impact is catastrophic. Separating the users into one group with the new code and one group without it, and then observing them separately, allows the system to detect less massive impacts, which you then can compare.
 
Overall, it's important to determine the correct metrics to analyze your data from the feature release against, and then it is great practice to utilize a system that can examine the data in the two groups separately (some companies have dubbed this an experimentation platform) allowing you to better determine when is best to ramp up the release and when it's time to roll it back and hit the drawing board.

Dave Karow is the Continuous Delivery Evangelist at Split Software
Share this

Industry News

April 02, 2020

VMware announced the general availability of VMware vSphere 7, the biggest evolution of vSphere in over a decade.

April 02, 2020

Grafana Labs announced that Cortex v1.0 is generally available for production use.

April 02, 2020

IT Revolution announced new dates, extended pricing and its first round of confirmed speakers for DevOps Enterprise Summit Las Vegas 2020. Hosted at The Cosmopolitan of Las Vegas, DevOps Enterprise Summit will now take place November 9-11, 2020.

April 01, 2020

Compuware Corporation announced new capabilities that enable application development teams to automate performance tests early in the development lifecycle, helping large enterprises speed time to market and improve application performance—while decreasing the significant and unnecessary cost of wasted time.

April 01, 2020

PlanetScale released the newest version of PlanetScaleDB, a multi-cloud database.

April 01, 2020

Datawire announced the newest release of Ambassador Edge Stack that is designed to speed up the inner development loop.

March 31, 2020

Push Technology will provide Diffusion Cloud, Push Technology’s Real-Time API Management Cloud Platform, free for all existing customers and new customers developing systems in the cloud during these challenging times.

March 31, 2020

Rancher Labs announced the general availability of Rancher 2.4.

March 31, 2020

Kasten announced the general availability of Kasten K10 v2.5.

March 30, 2020

DevOps Institute, a global member-based association for advancing the human elements of DevOps, announced a new Open Testing program that removes the prerequisite of leveraging formal courseware to achieve certifications from DevOps Institute's extensive portfolio.

March 30, 2020

Oracle announced the general availability of Java 14 (Oracle JDK 14).

March 30, 2020

Akamai announced March 2020 updates to the Akamai Intelligent Edge Platform.

March 26, 2020

Redgate’s new SQL Monitor now ensures that DevOps teams can monitor and track deployments at all times.

March 26, 2020

Split Software announced a two-way data integration with Google Analytics that can instantly detect performance issues caused by new features.

March 26, 2020

Cloudreach earned the Kubernetes on Microsoft Azure advanced specialization.