Experimentation Done Right: It's All About the Data
May 01, 2019

Dave Karow
Split Software

By now, the concept of experimentation in software development is fairly well known. Most development teams understand at a high level the benefits that can be achieved through experimentation. Perhaps the most important of those is the ability to identify positive or negative impacts of a feature — in terms of both app performance and customer experience — earlier in the development process.

But many companies simply are not getting the most out of their efforts around experimentation or are reluctant to fully embrace it. Why? For starters, there's risk. And the degree of risk a company is willing to tolerate varies depending on their business.

Earlier this year, for example, Instagram decided to make a widespread change to the swipe button feature in its interface. The company was immediately flooded with negative feedback from many of its app users. In truth, the risk was relatively inconsequential to the ubiquitous social app because, despite user backlash, the change was rolled back and barely registered a blip on the company's radar. In other words, it is unlikely that Instagram lost money over the hiccup and in media interviews spokespeople were able to simply brush it off as a "bug."

For another, less innovative company however, taking that same risk with a major feature of a product or app could prove far costlier and more detrimental. Nearly every software feature release is designed to make improvements to the software itself. The reality is that not every feature release will result in positive improvement and the consequences for this could be severe: loss of users, revenue, or worse — both.

The goal is being able to run worthwhile experiments without disrupting the things you need to do to run your business. And that's where many companies attempting experimentation run into problems.

Experimenting the Hard Way

Companies understand that protecting their user experience is important and that releasing faster is important. That's the reason they have invested in monitoring tools, alerting systems and continuous delivery pipelines. When it comes to experimentation though, they often find that it is hard to do at scale in a repeatable fashion. The lack of tooling designed specifically for the experimentation problem set means someone has to step up and do a lot of ad-hoc data science work to make sense of results. The need and interest are there, as most teams know intuitively that better data will help them learn what works and does not work, earlier in the development process.

Your business may be doing many of the things one would associate with experimentation, and even reaping some benefits. Maybe you are performing canary rollouts or a/b tests, which have allowed you to accelerate feature releases or measure the impact of features. The problem is, the operational cost of doing those things can be high. You may be able to run a few experiments, but you will not be able to run very many because it's simply too difficult a process to do ad-hoc. As a result, features are actually being released more slowly because of the operational cost or dependence on scarce data science resources for one-off study of the results. The rate of innovation has slowed, reducing the value of experimentation.

Data is the Key

If instead businesses approach experimentation in a way that controls risk and streamlines ingestion and analysis of results data, they can do it in a much more effective way. And the key to doing that is access to data: how easily can you observe changes to key metrics when you conduct experiments? When data is siloed or must be manually curated during every experiment, it is less valuable and actionable. You also run greater risk of different teams drawing entirely different conclusions from the same data, because there's no common point of reference from which to make decisions. Lots of companies collect data today, but the data relevance — its breadth and scope — is not what it needs to be in order to be able to make actionable decisions.

Companies must remove the roadblocks separating them from their data. After all, if you are going to make decisions about anything, you want to be able to do it with the strength of relevant information. By making data ubiquitous, rather than scarce, you can establish a common language for measurement, which is the first step to being able to do meaningful experiments that positively impact both the user and your business.

Purpose-built experimentation platforms marry actionable data to changes that multiple teams make, eliminating the overhead and inconsistency of ad-hoc data analysis. With reliable tooling in hand and a repeatable process for making contextual decisions, companies can more easily embrace experimentation at scale. As the cost of "turning the crank" and making sense of the data for each experiment goes down and the number of experiments goes up, these companies give themselves more opportunities to unlock innovation and course-correct quickly in the face of failing ideas.

Dave Karow is the Continuous Delivery Evangelist at Split Software
Share this

Industry News

September 16, 2019

Oracle announced the general availability of Java SE 13 (JDK 13).

September 16, 2019

Data Intensity launched its Automation-as-a-Service offering.

September 16, 2019

Mobile Labs launched the final addition to its mobile device cloud suite: GigaFox Red and GigaFox Silver.

September 12, 2019

Rafay Systems announced the general availability of its turnkey, SaaS-based offering designed to confront a complex set of ongoing challenges enterprises and service providers face when modernizing their applications.

September 12, 2019

StackRox announced the availability of the StackRox App for the Sumo Logic Continuous Intelligence Platform.

September 12, 2019

Lacework is receiving $42 million from Sutter Hill Ventures and Liberty Global Ventures.

September 11, 2019

Clubhouse released a fully featured Free Plan that offers the full power of its flagship product to teams up to 10 people.

September 11, 2019

Sectigo released integrations with five of the most popular DevOps configuration management and container orchestration platforms.

September 11, 2019

Kong announced the release of a new open source project called Kuma.

September 10, 2019

Parasoft is excited to announce that Parasoft SOAtest, an API and UI functional testing solution, has won a 2019 API Award in the Best in Microservices Infrastructure category.

September 10, 2019

Cohesity announced the launch of Cohesity Agile Dev and Test, a new solution that addresses a key bottleneck organizations face in building applications at speed.

September 10, 2019

Split Software announced the addition of Feature Monitoring, an automated detection capability for its feature delivery platform that reduces detection times of errors in a code release.

September 09, 2019

US Signal announced the launch of its managed Website and Application Security Solution.

September 09, 2019

Tasktop announced that Jama Software is now offering the cloud version of its Tasktop Integration Hub for Jama Connect to automate and visualize the flow of product-critical information across the software delivery value stream.

September 09, 2019

Mesosphere announced a significant expansion in strategy and product portfolio as well as a new company name - D2iQ.