Experimentation Done Right: It's All About the Data
May 01, 2019

Dave Karow
Split Software

By now, the concept of experimentation in software development is fairly well known. Most development teams understand at a high level the benefits that can be achieved through experimentation. Perhaps the most important of those is the ability to identify positive or negative impacts of a feature — in terms of both app performance and customer experience — earlier in the development process.

But many companies simply are not getting the most out of their efforts around experimentation or are reluctant to fully embrace it. Why? For starters, there's risk. And the degree of risk a company is willing to tolerate varies depending on their business.

Earlier this year, for example, Instagram decided to make a widespread change to the swipe button feature in its interface. The company was immediately flooded with negative feedback from many of its app users. In truth, the risk was relatively inconsequential to the ubiquitous social app because, despite user backlash, the change was rolled back and barely registered a blip on the company's radar. In other words, it is unlikely that Instagram lost money over the hiccup and in media interviews spokespeople were able to simply brush it off as a "bug."

For another, less innovative company however, taking that same risk with a major feature of a product or app could prove far costlier and more detrimental. Nearly every software feature release is designed to make improvements to the software itself. The reality is that not every feature release will result in positive improvement and the consequences for this could be severe: loss of users, revenue, or worse — both.

The goal is being able to run worthwhile experiments without disrupting the things you need to do to run your business. And that's where many companies attempting experimentation run into problems.

Experimenting the Hard Way

Companies understand that protecting their user experience is important and that releasing faster is important. That's the reason they have invested in monitoring tools, alerting systems and continuous delivery pipelines. When it comes to experimentation though, they often find that it is hard to do at scale in a repeatable fashion. The lack of tooling designed specifically for the experimentation problem set means someone has to step up and do a lot of ad-hoc data science work to make sense of results. The need and interest are there, as most teams know intuitively that better data will help them learn what works and does not work, earlier in the development process.

Your business may be doing many of the things one would associate with experimentation, and even reaping some benefits. Maybe you are performing canary rollouts or a/b tests, which have allowed you to accelerate feature releases or measure the impact of features. The problem is, the operational cost of doing those things can be high. You may be able to run a few experiments, but you will not be able to run very many because it's simply too difficult a process to do ad-hoc. As a result, features are actually being released more slowly because of the operational cost or dependence on scarce data science resources for one-off study of the results. The rate of innovation has slowed, reducing the value of experimentation.

Data is the Key

If instead businesses approach experimentation in a way that controls risk and streamlines ingestion and analysis of results data, they can do it in a much more effective way. And the key to doing that is access to data: how easily can you observe changes to key metrics when you conduct experiments? When data is siloed or must be manually curated during every experiment, it is less valuable and actionable. You also run greater risk of different teams drawing entirely different conclusions from the same data, because there's no common point of reference from which to make decisions. Lots of companies collect data today, but the data relevance — its breadth and scope — is not what it needs to be in order to be able to make actionable decisions.

Companies must remove the roadblocks separating them from their data. After all, if you are going to make decisions about anything, you want to be able to do it with the strength of relevant information. By making data ubiquitous, rather than scarce, you can establish a common language for measurement, which is the first step to being able to do meaningful experiments that positively impact both the user and your business.

Purpose-built experimentation platforms marry actionable data to changes that multiple teams make, eliminating the overhead and inconsistency of ad-hoc data analysis. With reliable tooling in hand and a repeatable process for making contextual decisions, companies can more easily embrace experimentation at scale. As the cost of "turning the crank" and making sense of the data for each experiment goes down and the number of experiments goes up, these companies give themselves more opportunities to unlock innovation and course-correct quickly in the face of failing ideas.

Dave Karow is the Continuous Delivery Evangelist at Split Software
Share this

Industry News

July 09, 2020

ShiftLeft released a new version of NextGen Static Analysis (NG SAST), including new workflows, purpose-built for developers that significantly improve security, while enhancing productivity.

July 09, 2020

RunSafe Security announced a partnership with JFrog that will enable RunSafe to supercharge binary protections via a simple plugin that JFrog users can deploy within their Artifactory repositories and instantly protect binaries and containers.

July 09, 2020

LeanIX closed $80 million in Series D funding led by new investor Goldman Sachs Growth.

July 08, 2020

Afi.ai introduced Afi Data Platform, a cloud-based replication and resiliency service that helps to monitor, predict downtime and recover K8s applications.

July 08, 2020

D2iQ announced the release of Conductor, a new interactive learning platform that enables enterprises to access hands-on cloud native courses and training.

July 08, 2020

SUSE entered into a definitive agreement to acquire Rancher Labs.

July 07, 2020

Micro Focus announced AI-powered enhancements to the intelligent testing capabilities of the UFT Family, a unified set of solutions designed to reduce the overall complexity of automating the functional testing processes.

July 07, 2020

Push Technology announced the launch of a new Service API capability for Diffusion Cloud, Push’s Real-Time API Management Cloud Platform.

July 07, 2020

Lightrun exited stealth and announced $4M in seed funding for the first complete continuous debugging and observability platform for production applications.

July 01, 2020

JFrog announced the launch of ChartCenter, a free, security-focused central repository of Helm charts for the community.

July 01, 2020

Kong announced a significant upgrade to open source Kuma, Kuma 0.6, available today.

July 01, 2020

Compuware Corporation, a BMC company, announced new capabilities that further automate and integrate test data and test case execution, empowering IT teams to achieve high-performance application development quality, velocity and efficiency.

June 30, 2020

Couchbase announced the general availability of Couchbase Cloud, a fully-managed Database-as-a-Service (DBaaS).

June 30, 2020

Split Software announced new capabilities designed to accelerate the adoption of feature flags in large-scale organizations.

June 30, 2020

WhiteHat Security announced a discounted Web + Mobile Application Security bundle to help organizations secure the digital future.