DEVOPSdigest

By now, the concept of experimentation in software development is fairly well known. Most development teams understand at a high level the benefits that can be achieved through experimentation. Perhaps the most important of those is the ability to identify positive or negative impacts of a feature — in terms of both app performance and customer experience — earlier in the development process.

But many companies simply are not getting the most out of their efforts around experimentation or are reluctant to fully embrace it. Why? For starters, there's risk. And the degree of risk a company is willing to tolerate varies depending on their business.

Earlier this year, for example, Instagram decided to make a widespread change to the swipe button feature in its interface. The company was immediately flooded with negative feedback from many of its app users. In truth, the risk was relatively inconsequential to the ubiquitous social app because, despite user backlash, the change was rolled back and barely registered a blip on the company's radar. In other words, it is unlikely that Instagram lost money over the hiccup and in media interviews spokespeople were able to simply brush it off as a "bug."

For another, less innovative company however, taking that same risk with a major feature of a product or app could prove far costlier and more detrimental. Nearly every software feature release is designed to make improvements to the software itself. The reality is that not every feature release will result in positive improvement and the consequences for this could be severe: loss of users, revenue, or worse — both.

The goal is being able to run worthwhile experiments without disrupting the things you need to do to run your business. And that's where many companies attempting experimentation run into problems.

Experimenting the Hard Way

Companies understand that protecting their user experience is important and that releasing faster is important. That's the reason they have invested in monitoring tools, alerting systems and continuous delivery pipelines. When it comes to experimentation though, they often find that it is hard to do at scale in a repeatable fashion. The lack of tooling designed specifically for the experimentation problem set means someone has to step up and do a lot of ad-hoc data science work to make sense of results. The need and interest are there, as most teams know intuitively that better data will help them learn what works and does not work, earlier in the development process.

Your business may be doing many of the things one would associate with experimentation, and even reaping some benefits. Maybe you are performing canary rollouts or a/b tests, which have allowed you to accelerate feature releases or measure the impact of features. The problem is, the operational cost of doing those things can be high. You may be able to run a few experiments, but you will not be able to run very many because it's simply too difficult a process to do ad-hoc. As a result, features are actually being released more slowly because of the operational cost or dependence on scarce data science resources for one-off study of the results. The rate of innovation has slowed, reducing the value of experimentation.

Data is the Key

If instead businesses approach experimentation in a way that controls risk and streamlines ingestion and analysis of results data, they can do it in a much more effective way. And the key to doing that is access to data: how easily can you observe changes to key metrics when you conduct experiments? When data is siloed or must be manually curated during every experiment, it is less valuable and actionable. You also run greater risk of different teams drawing entirely different conclusions from the same data, because there's no common point of reference from which to make decisions. Lots of companies collect data today, but the data relevance — its breadth and scope — is not what it needs to be in order to be able to make actionable decisions.

Companies must remove the roadblocks separating them from their data. After all, if you are going to make decisions about anything, you want to be able to do it with the strength of relevant information. By making data ubiquitous, rather than scarce, you can establish a common language for measurement, which is the first step to being able to do meaningful experiments that positively impact both the user and your business.

Purpose-built experimentation platforms marry actionable data to changes that multiple teams make, eliminating the overhead and inconsistency of ad-hoc data analysis. With reliable tooling in hand and a repeatable process for making contextual decisions, companies can more easily embrace experimentation at scale. As the cost of "turning the crank" and making sense of the data for each experiment goes down and the number of experiments goes up, these companies give themselves more opportunities to unlock innovation and course-correct quickly in the face of failing ideas.