Experimentation Done Right: It's All About the Data
May 01, 2019

Dave Karow
Split Software

By now, the concept of experimentation in software development is fairly well known. Most development teams understand at a high level the benefits that can be achieved through experimentation. Perhaps the most important of those is the ability to identify positive or negative impacts of a feature — in terms of both app performance and customer experience — earlier in the development process.

But many companies simply are not getting the most out of their efforts around experimentation or are reluctant to fully embrace it. Why? For starters, there's risk. And the degree of risk a company is willing to tolerate varies depending on their business.

Earlier this year, for example, Instagram decided to make a widespread change to the swipe button feature in its interface. The company was immediately flooded with negative feedback from many of its app users. In truth, the risk was relatively inconsequential to the ubiquitous social app because, despite user backlash, the change was rolled back and barely registered a blip on the company's radar. In other words, it is unlikely that Instagram lost money over the hiccup and in media interviews spokespeople were able to simply brush it off as a "bug."

For another, less innovative company however, taking that same risk with a major feature of a product or app could prove far costlier and more detrimental. Nearly every software feature release is designed to make improvements to the software itself. The reality is that not every feature release will result in positive improvement and the consequences for this could be severe: loss of users, revenue, or worse — both.

The goal is being able to run worthwhile experiments without disrupting the things you need to do to run your business. And that's where many companies attempting experimentation run into problems.

Experimenting the Hard Way

Companies understand that protecting their user experience is important and that releasing faster is important. That's the reason they have invested in monitoring tools, alerting systems and continuous delivery pipelines. When it comes to experimentation though, they often find that it is hard to do at scale in a repeatable fashion. The lack of tooling designed specifically for the experimentation problem set means someone has to step up and do a lot of ad-hoc data science work to make sense of results. The need and interest are there, as most teams know intuitively that better data will help them learn what works and does not work, earlier in the development process.

Your business may be doing many of the things one would associate with experimentation, and even reaping some benefits. Maybe you are performing canary rollouts or a/b tests, which have allowed you to accelerate feature releases or measure the impact of features. The problem is, the operational cost of doing those things can be high. You may be able to run a few experiments, but you will not be able to run very many because it's simply too difficult a process to do ad-hoc. As a result, features are actually being released more slowly because of the operational cost or dependence on scarce data science resources for one-off study of the results. The rate of innovation has slowed, reducing the value of experimentation.

Data is the Key

If instead businesses approach experimentation in a way that controls risk and streamlines ingestion and analysis of results data, they can do it in a much more effective way. And the key to doing that is access to data: how easily can you observe changes to key metrics when you conduct experiments? When data is siloed or must be manually curated during every experiment, it is less valuable and actionable. You also run greater risk of different teams drawing entirely different conclusions from the same data, because there's no common point of reference from which to make decisions. Lots of companies collect data today, but the data relevance — its breadth and scope — is not what it needs to be in order to be able to make actionable decisions.

Companies must remove the roadblocks separating them from their data. After all, if you are going to make decisions about anything, you want to be able to do it with the strength of relevant information. By making data ubiquitous, rather than scarce, you can establish a common language for measurement, which is the first step to being able to do meaningful experiments that positively impact both the user and your business.

Purpose-built experimentation platforms marry actionable data to changes that multiple teams make, eliminating the overhead and inconsistency of ad-hoc data analysis. With reliable tooling in hand and a repeatable process for making contextual decisions, companies can more easily embrace experimentation at scale. As the cost of "turning the crank" and making sense of the data for each experiment goes down and the number of experiments goes up, these companies give themselves more opportunities to unlock innovation and course-correct quickly in the face of failing ideas.

Dave Karow is the Continuous Delivery Evangelist at Split Software
Share this

Industry News

December 10, 2019

Redgate Software launched its fourth annual State of Database DevOps Survey.

December 10, 2019

Compuware has signed a definitive agreement to acquire the assets of INNOVATION Data Processing, a provider of enterprise data protection, business continuance and storage resource management solutions serving the mainframe market.

December 10, 2019

Dynatrace announced its Autonomous Cloud Enablement (ACE) Practice to accelerate DevOps’ movement to autonomous cloud operations.

December 09, 2019

NS1, announced the expansion of its suite of integrations to include Kubernetes, Consul, Avi Networks (VMWare NSX), NGINX, and HAProxy.

December 09, 2019

CloudBees announced an extension of its partnership with Google. As a Google Cloud Run launch partner, CloudBees will offer developers more flexibility in their deployment of containerized applications.

December 09, 2019

EPAM Systems has expanded its crowdtesting software solutions to enable user story testing.

December 05, 2019

Parasoft announced the newest release of Parasoft C/C++test, the unified C and C++ development testing solution for enterprise and embedded applications.

December 05, 2019

Datadog announced Security Monitoring, a new product that enables real-time threat detection across the entire stack and deeper collaboration between security, developers, and operations teams.

December 05, 2019

Pulumi announced the availability of Pulumi Crosswalk for Kubernetes, an open source collection of frameworks, tools and user guides that help developers and operators work better together delivering production workloads using Kubernetes.

December 04, 2019

CloudBees announced a Preview Program for CloudBees CI/CD powered by Jenkins X, a Software as a Service (SaaS) continuous integration and continuous delivery solution running on Google Cloud Platform.

December 04, 2019

Rancher Labs announced the general availability of K3s, their lightweight, certified Kubernetes distribution purpose built for small footprint workloads, along with the beta release of Rio, their new application deployment engine for Kubernetes that delivers a fully integrated deployment experience from operations to pipeline.

December 04, 2019

WhiteSource announced a new integration with Codefresh, the Kubernetes-native CI/CD solution.

December 03, 2019

Styra is addressing one of the most significant enterprise blockers of Kubernetes: compliance. With Styra, enterprises can move Kubernetes clusters into production en masse while complying with traditional governance, audit, and compliance rules and regulations.

December 03, 2019

Nureva added 13 agile-themed templates to Span Workspace, Nureva’s expansive cloud-based digital canvas for visual planning and team collaboration.

December 03, 2019

Threat Stack announced support for AWS Fargate in the Threat Stack Cloud Security Platform.