Feature Flag Observability: Don't Wing It
November 05, 2019

Dave Karow
Split Software

The benefits of a feature flag rollout are clear to many DevOps teams. The ability to control the gradual release of a feature, and then be able to retract the feature without having to restart the entire release, has made the job of development and operations teams a lot easier.

A solid feature flag solution provides a standardized, scalable and transparent way to accomplish that across all of your projects. While that may sound like a dream come true, there's more to consider when implementing feature flags than just whether or not you need to retract a faulty feature.
 
The ability to gradually roll out a feature flag means that careful consideration is needed when determining when to ramp it up. Deciding when is the right time to release the feature to more users is a process that should rely heavily on data. To do that, you must identify what you are looking for in the data.

Teams at some of the leading tech companies that have built amazing in-house systems for control and observation with feature flags have discovered that building in fine-grained control and automated observability into their continuous delivery practices is as critical as maintaining version control or implementing continuous integration.
 
Surprisingly, even though these teams saw the benefit of built-in control and automated observability many of them do not have a structured and repeatable way to detect the impact of the rollouts. Many companies were "winging it" with ad-hoc exploration or they were waiting for some global health metric results.
 
Ad hoc exploration is too random of a process to specifically identify when it is correct to allow more users to access the new feature. This process is also very susceptible to observation bias — the inclination to see the results we expect to see or want to see. With the aim of gradual rollouts with feature flags being to gather data on new code and determine whether it accomplishes set goals (such as increased engagement and more satisfied customers), ad hoc exploration does not allow for the data to be carefully analyzed.
 
The main question that should be asked when launching a new feature is "Does my new code make any difference?"

An additional question should be, "Is the difference positive and does it impact users without impacting other factors such as response time, errors, unsubscribes and bad app reviews?"
 
A feature flag allows you to make the new feature accessible to a select group of your users, thus making it easier to determine if the new feature causes any change in the user experience, such as response time, error rates and app crashes. Being able to compare the group that has the new feature and the group that doesn't makes it a lot easier to determine the success of the new feature. But how you correctly identify and analyze the difference is important.
 
One way to go about this is to utilize auto-calculating. Auto-calculating impact metrics is the process of using the same system that is controlling the rollout to keep track of how well it is doing. It is important to do it in a way that is repeatable, consistent and scalable.

The key to this process is to watch it through the lens of the control panel, the instrument that is deciding which users get access to the new code. While the inputs can come from anywhere and can be a mix of existing and new instrumentation, watching each cohort through this lens makes it much easier to get an early warning than if you watch your global metric, which will only bring a negative symptom to your attention if the impact is catastrophic. Separating the users into one group with the new code and one group without it, and then observing them separately, allows the system to detect less massive impacts, which you then can compare.
 
Overall, it's important to determine the correct metrics to analyze your data from the feature release against, and then it is great practice to utilize a system that can examine the data in the two groups separately (some companies have dubbed this an experimentation platform) allowing you to better determine when is best to ramp up the release and when it's time to roll it back and hit the drawing board.

Dave Karow is the Continuous Delivery Evangelist at Split Software
Share this

Industry News

September 24, 2020

NetApp announced the availability of Elastigroup for Microsoft Azure Spot Virtual Machines (VMs).

September 24, 2020

CloudBees announced a robust new set of DevSecOps capabilities for CloudBees CI and CloudBees CD. The new capabilities enable customers to perform early and frequent security checks and ensure that security is an integral part of the whole software delivery pipeline workflow, without sacrificing speed or increasing risk.

September 24, 2020

Pulumi announced the release of a Pulumi-native provider for Microsoft Azure that provides 100% coverage of Azure Resource Manager (ARM), the deployment and management service for Azure that enables users to create, update and delete resources in their Azure accounts.

September 23, 2020

Puppet announced new Windows services, integrations and enhancements aimed at making it easier to automate and manage infrastructure using tools Windows admins rely on. The latest updates include services around Group Policy Migration and Chocolatey, as well as enhancements to the Puppet VS Code Extension, and a new Puppet PowerShell DSC Builder module.

September 23, 2020

Red Hat announced the release of Red Hat OpenShift Container Storage 4.5, delivering Kubernetes-based data services for modern, cloud-native applications across the open hybrid cloud.

September 23, 2020

Copado, a native DevOps platform for Salesforce, has acquired ClickDeploy.

September 22, 2020

CloudBees announced general availability of the first two modules of its Software Delivery Management solution.

September 22, 2020

Applause announced the availability of its Bring Your Own Testers (BYOT) feature that enables clients to manage their internal teams – employees, friends, family members and existing customers – and invite them to test cycles in the Applause Platform alongside Applause’s vetted and expert community of testers.

September 22, 2020

Kasten announced the integration of the K10 data management platform with VMware vSphere and Tanzu Kubernetes Grid Service.

September 21, 2020

PagerDuty entered into a definitive agreement to acquire Rundeck, a provider of DevOps automation for enterprise.

September 21, 2020

Grafana Labs announced the release of Grafana Metrics Enterprise, a modern Prometheus-as-a-Service solution designed for the scale, architecture, and security needs of enterprises as they expand their observability initiatives.

September 21, 2020

Portshift's Cloud Workload Protection platform is now available through the Red Hat Marketplace.

September 17, 2020

env0, a developer of Infrastructure-as-Code (IaC) management software, announced the availability of its new open source solution for Terraform users, Terratag.

September 17, 2020

Push Technology announced a partnership with Innova Solutions, an ACS Solutions company, specializing in global information technology services.

September 17, 2020

Alcide achieved the AWS Outposts Ready designation, part of the Amazon Web Services (AWS) Service Ready Program.