Test on Mobile Devices That Matter to Your Business
June 15, 2015

Amir Rozenberg
Perfecto

Choosing the right smartphones and tablets to test your apps against is just as important as the tests themselves. After all, if your app isn’t working at the right place (your customer’s device) at the right time (all the time), angry customer tweets and two-star app reviews will follow and the consequences for your business will be severe.
 
In some cases, your app won’t even make it into the app store. Twenty percent of apps are rejected from the Apple store because they’re buggy, crash-prone or have an inferior user interface. And when a new version of Android arrives, apps that worked fine on the previous OS version start to act like rejects themselves.
 
On the hardware side of the fence, new devices arrive every quarter, and device usage varies by geography. Windows Phones may have 3.8% market share in the US but in Europe it has an average market share of close to 9%, according to mobile market research firm, ComTech. Basing your testing of Windows Phones solely on US market data when your app also has a presence in, say, France (12%) and Italy (13.3%), would be a costly mistake.
 
The bottom line is that mobile devices and operating systems are always in flux. You need to keep a close eye on market changes so you’ll be testing your apps and responsive websites against the devices that represent your target users. To that end, we created a Mobile Test Coverage Index report, a quarterly release that aggregates data from over 4,000 device profiles and 360,000 hours of customer usage data, as well as mobile market share data and future device release timelines, to create a list of the 32 most relevant mobile devices.
 
The device selections address five key sectors: device model, screen size, GA date, hardware system and recommended operating systems. The number of devices in each test bracket may vary from quarter to quarter to reflect the restless mobile market. The report will be continuously updated to meet market changes.

While the report will not hand-deliver an answer for what devices and operating systems the company should specifically test against, it can be used as a benchmark from which it can customize their test coverage according to their own business and customer needs.
 
Some noteworthy insights from the research:


■ Almost a third (30 percent) of the market can be covered with 10-16 devices – including popular devices such as the Samsung Galaxy S5, iPhone 6 and HTC One M8 – based on market adoption, market device leaders, reference devices and device characteristics.

■ Regarding operating systems: iOS 7 still accounts for some 17 percent of total users out there, and should be included at the same level of iOS 8 test plans. Additionally, Android 5.0 adoption rates are low in both the US and Europe with Android KitKat (4.4.x) and JellyBean (4.2.x, 4.3.x) rating as the most commonly used versions.

■ Geographically speaking, iPhones hold twice as much share in the US than in Europe, with iPhone devices capturing 35% of the U.S. market, as opposed to 16% in EU5.

We can’t overstate how important device coverage is to a mobile app testing strategy, for both business and technical executives alike. A relevant device and operating system mix will get an organization much closer to the true end-user experience. However, it’s a challenge to keep your test coverage relevant given that mobile devices have a shelf life of around nine months. Benchmarks built from data like this report will help, but in an ever-changing market it’s essential to keep reviewing continually updated research.

Amir Rozenberg is Director of Product Management at Perfecto

The Latest

December 14, 2018

DEVOPSdigest invited DevOps experts for their predictions on how DevOps and related technologies will evolve and impact business in 2019. Part 2 covers DevOps culture and people ...

December 13, 2018

DEVOPSdigest's annual list of DevOps Predictions is one of the most popular series of content on DEVOPSdigest. DevOps experts offer thoughtful, insightful, and sometimes controversial predictions on how DevOps and related technologies will evolve and impact business in 2019 ...

December 11, 2018

Companies expect increased reliance on Cloud Native Applications (CNAs), however security concerns could prove to be a major obstacle, according to The State of Cloud Native Security ...

December 06, 2018

The general consensus tends to be that in the world of agile and DevOps, ITSM teams are increasingly being left behind. But the truth is, in more forward-thinking IT organizations, this isn’t the case. The fact is that ITSM is playing, or at least should play, a growing role in support of agile and DevOps initiatives. But this role still remains limited due to the fact that DevOps teams, and their management, are (more often than not) leaving them out as a tool of choice ...

December 05, 2018

The industry is revealing increasingly optimistic attitudes towards mainframes, with 93% of executives and 92% of all respondents viewing the mainframe as a strong long-term platform – the highest level in five years – according to the 2018 Mainframe Research Report from BMC ...

December 03, 2018

ActiveState surveyed developers and programmers in 92 countries to better understand their pain points and assess how businesses can better work with their organizations. The survey results establish a starting point for understanding the challenges that coders confront when working with open source runtimes ...

November 29, 2018

Organizations with established DevSecOps programs and practices greatly outperform their peers in how quickly they address flaws. The most active DevSecOps programs fix flaws more than 11.5 times faster than the typical organization, due to ongoing security checks during continuous delivery of software builds, largely the result of increased code scanning, according the latest State of Software Security (SOSS) report from CA Veracode ..

November 27, 2018

The push to make banking products digitally ready (and very quickly) has spurred the old “buy vs. build” debate in bank IT departments: Should we build our own software from scratch in-house? Or do we buy off-the-shelf solutions from third-parties? And while this dichotomy may have been a suitable mentality years ago at the start of the digital transformation revolution in banking, it simply no longer fits with the reality of today's more complex development landscape ...

November 26, 2018

With the rise of next-generation technologies, businesses have access to more data than ever, creating opportunities to develop new channels for revenue. Contributing to the increase in data is a growing reliance on the external supply chain. However, with the influx of data comes the necessity to understand the entire third-party ecosystem; its benefits and risks. Some of the most devastating breaches have been attributed to a third party ...

November 20, 2018

In today's digital economy, monitoring is a must. Your customers must be able to access your website and your apps, interact, purchase — and monitoring is one way to make sure this keeps happening. But the first question has to be: What should be monitored? With this in mind, APMdigest asked experts from across the IT industry for their opinions on what IT departments should be monitoring to ensure digital performance ...

Share this