Feature Flag Observability: Don't Wing It
November 05, 2019

Dave Karow
Split Software

The benefits of a feature flag rollout are clear to many DevOps teams. The ability to control the gradual release of a feature, and then be able to retract the feature without having to restart the entire release, has made the job of development and operations teams a lot easier.

A solid feature flag solution provides a standardized, scalable and transparent way to accomplish that across all of your projects. While that may sound like a dream come true, there's more to consider when implementing feature flags than just whether or not you need to retract a faulty feature.
 
The ability to gradually roll out a feature flag means that careful consideration is needed when determining when to ramp it up. Deciding when is the right time to release the feature to more users is a process that should rely heavily on data. To do that, you must identify what you are looking for in the data.

Teams at some of the leading tech companies that have built amazing in-house systems for control and observation with feature flags have discovered that building in fine-grained control and automated observability into their continuous delivery practices is as critical as maintaining version control or implementing continuous integration.
 
Surprisingly, even though these teams saw the benefit of built-in control and automated observability many of them do not have a structured and repeatable way to detect the impact of the rollouts. Many companies were "winging it" with ad-hoc exploration or they were waiting for some global health metric results.
 
Ad hoc exploration is too random of a process to specifically identify when it is correct to allow more users to access the new feature. This process is also very susceptible to observation bias — the inclination to see the results we expect to see or want to see. With the aim of gradual rollouts with feature flags being to gather data on new code and determine whether it accomplishes set goals (such as increased engagement and more satisfied customers), ad hoc exploration does not allow for the data to be carefully analyzed.
 
The main question that should be asked when launching a new feature is "Does my new code make any difference?"

An additional question should be, "Is the difference positive and does it impact users without impacting other factors such as response time, errors, unsubscribes and bad app reviews?"
 
A feature flag allows you to make the new feature accessible to a select group of your users, thus making it easier to determine if the new feature causes any change in the user experience, such as response time, error rates and app crashes. Being able to compare the group that has the new feature and the group that doesn't makes it a lot easier to determine the success of the new feature. But how you correctly identify and analyze the difference is important.
 
One way to go about this is to utilize auto-calculating. Auto-calculating impact metrics is the process of using the same system that is controlling the rollout to keep track of how well it is doing. It is important to do it in a way that is repeatable, consistent and scalable.

The key to this process is to watch it through the lens of the control panel, the instrument that is deciding which users get access to the new code. While the inputs can come from anywhere and can be a mix of existing and new instrumentation, watching each cohort through this lens makes it much easier to get an early warning than if you watch your global metric, which will only bring a negative symptom to your attention if the impact is catastrophic. Separating the users into one group with the new code and one group without it, and then observing them separately, allows the system to detect less massive impacts, which you then can compare.
 
Overall, it's important to determine the correct metrics to analyze your data from the feature release against, and then it is great practice to utilize a system that can examine the data in the two groups separately (some companies have dubbed this an experimentation platform) allowing you to better determine when is best to ramp up the release and when it's time to roll it back and hit the drawing board.

Dave Karow is the Continuous Delivery Evangelist at Split Software
Share this

Industry News

June 23, 2022

Akana by Perforce now offers BlazeMeter to customers, previously a solution with Broadcom Layer7.

June 23, 2022

Coder announced the release of a new open source project that gives developers and data scientists a consistent, secure, yet flexible way to create cloud workspaces in minutes.

June 23, 2022

GitGuardian is announcing a series of new features to address developer experience in securing the software development lifecycle.

June 22, 2022

OctoML released a major platform expansion to accelerate the development of AI-powered applications by eliminating bottlenecks in machine learning deployment.

June 22, 2022

Snow Software announced new functionality and integrations for Snow Atlas, a purpose-built platform that provides a framework to accelerate data-driven technology decision-making.

June 22, 2022

Traefik Labs launched Traefik Hub, a new cloud service that eliminates the complexity of management and automation of Kubernetes and Docker networking at scale.

June 21, 2022

The Linux Foundation, the nonprofit organization enabling mass innovation through open source, announced the new Open Programmable Infrastructure (OPI) Project.

June 21, 2022

Docker announced the acquisition of Atomist, a company founded to improve developer productivity and keep cloud native applications safe.

June 21, 2022

SmartBear released BitBar, an all-in-one web and native mobile app testing solution.

June 16, 2022

Armory announced general availability of Armory Continuous Deployment-as-a-Service.

June 16, 2022

Infragistics announced the launch of App Builder On-Prem.

June 16, 2022

LambdaTest launched Test-at-Scale (TAS), a test intelligence and observability platform, to help development teams with shift-left testing.

June 16, 2022

NetApp announced continued innovations and solutions to provide enterprises with more simplicity, more security and more flexibility for their hybrid multicloud environments. These new capabilities include improved ransomware protection, hybrid cloud storage in a single subscription, unified management in a single user interface, and close collaboration with VMware to help transition workloads to the cloud.

June 16, 2022

Code Intelligence announces $12 million (11M€) in Series A funding led by Tola Capital.

June 15, 2022

Keysight Technologies and Sauce Labs have partnered to deliver cloud-based testing of enterprise applications on mobile devices, browsers and secure desktops.