Feature Flag Observability: Don't Wing It
November 05, 2019

Dave Karow
Split Software

The benefits of a feature flag rollout are clear to many DevOps teams. The ability to control the gradual release of a feature, and then be able to retract the feature without having to restart the entire release, has made the job of development and operations teams a lot easier.

A solid feature flag solution provides a standardized, scalable and transparent way to accomplish that across all of your projects. While that may sound like a dream come true, there's more to consider when implementing feature flags than just whether or not you need to retract a faulty feature.
 
The ability to gradually roll out a feature flag means that careful consideration is needed when determining when to ramp it up. Deciding when is the right time to release the feature to more users is a process that should rely heavily on data. To do that, you must identify what you are looking for in the data.

Teams at some of the leading tech companies that have built amazing in-house systems for control and observation with feature flags have discovered that building in fine-grained control and automated observability into their continuous delivery practices is as critical as maintaining version control or implementing continuous integration.
 
Surprisingly, even though these teams saw the benefit of built-in control and automated observability many of them do not have a structured and repeatable way to detect the impact of the rollouts. Many companies were "winging it" with ad-hoc exploration or they were waiting for some global health metric results.
 
Ad hoc exploration is too random of a process to specifically identify when it is correct to allow more users to access the new feature. This process is also very susceptible to observation bias — the inclination to see the results we expect to see or want to see. With the aim of gradual rollouts with feature flags being to gather data on new code and determine whether it accomplishes set goals (such as increased engagement and more satisfied customers), ad hoc exploration does not allow for the data to be carefully analyzed.
 
The main question that should be asked when launching a new feature is "Does my new code make any difference?"

An additional question should be, "Is the difference positive and does it impact users without impacting other factors such as response time, errors, unsubscribes and bad app reviews?"
 
A feature flag allows you to make the new feature accessible to a select group of your users, thus making it easier to determine if the new feature causes any change in the user experience, such as response time, error rates and app crashes. Being able to compare the group that has the new feature and the group that doesn't makes it a lot easier to determine the success of the new feature. But how you correctly identify and analyze the difference is important.
 
One way to go about this is to utilize auto-calculating. Auto-calculating impact metrics is the process of using the same system that is controlling the rollout to keep track of how well it is doing. It is important to do it in a way that is repeatable, consistent and scalable.

The key to this process is to watch it through the lens of the control panel, the instrument that is deciding which users get access to the new code. While the inputs can come from anywhere and can be a mix of existing and new instrumentation, watching each cohort through this lens makes it much easier to get an early warning than if you watch your global metric, which will only bring a negative symptom to your attention if the impact is catastrophic. Separating the users into one group with the new code and one group without it, and then observing them separately, allows the system to detect less massive impacts, which you then can compare.
 
Overall, it's important to determine the correct metrics to analyze your data from the feature release against, and then it is great practice to utilize a system that can examine the data in the two groups separately (some companies have dubbed this an experimentation platform) allowing you to better determine when is best to ramp up the release and when it's time to roll it back and hit the drawing board.

Dave Karow is the Continuous Delivery Evangelist at Split Software
Share this

Industry News

November 14, 2019

Raytheon Company is collaborating with Red Hat to develop a new, security-focused software development solution, known as DevSecOps, for enterprise environments.

November 14, 2019

Fugue has open sourced the Fugue Rego Toolkit (Fregot) to enhance the experience working with the Rego policy language.

November 14, 2019

Sysdig announced Sysdig Secure 3.0 to provide enterprises with threat prevention at runtime using Kubernetes-native Pod Security Policies (PSP).

November 13, 2019

Testim introduced the Testim Development Kit, a new way for developers to quickly create resilient tests directly in code.

November 13, 2019

Rollbar announced an error monitoring solution for Salesforce’s Apex platform.

November 13, 2019

StackRox announced version 3.0 of the StackRox Kubernetes Security Platform.

November 12, 2019

VMware announced rapid advancement of VMware Tanzu, a new portfolio of products and services designed to transform the way enterprises build, run and manage software on Kubernetes.

November 12, 2019

SmartBear released ReadyAPI 3.0. This latest release addresses the increasing requirement for organizations to consistently deliver high-quality APIs in order to meet accelerated business demands within compressed release cycles.

November 12, 2019

Aqua Security announced its expansion into cloud security posture management (CSPM) with its acquisition of CloudSploit.

November 07, 2019

To help developers increase the speed and quality of their SQL coding, enhance efficiency, and take advantage of the latest improvements in SQL Server, Redgate has released a major upgrade for its most popular tool, SQL Prompt.

November 07, 2019

CloudBees announced a partnership with Atos and VMware surrounding a solution to help customers adopt DevOps best practices at scale on Atos’ recently announced Atos Digital Hybrid Cloud (DHC) powered by VMware Tanzu and CloudBees cloud native continuous integration/continuous delivery (CI/CD) enterprise solution.

November 07, 2019

Fugue announced the release of the Fugue Best Practices Framework to help cloud engineering and security teams identify and remediate dangerous cloud resource misconfigurations that aren’t addressed by common compliance frameworks.

November 06, 2019

Red Hat and the Quarkus community announced Quarkus 1.0.

November 06, 2019

Copado announced its Winter 20 release to provide Salesforce customers the fastest path to continuous innovation.

November 06, 2019

Applause announced its new solution for AI training and testing.