5 Keys for Effective Mobile Performance Testing
May 06, 2013

Eran Kinsbruner
Perfecto

While mobile is quickly becoming the de-facto market platform for many of the business-critical applications deployed by banks, insurance companies and other enterprise organizations, the need to ensure an optimal end user experience mandates a robust mobile performance testing environment.

Accordingly, mobile enterprises require an environment which can provide insights into the key mobile application performance indicators - such as response time, availability and business critical transactions - on various devices (and operating systems) being used across different networks and carriers.

Let's begin by taking a look at the current situation, which clearly underscores the biggest challenge for enterprises in terms of ensuring successful mobile business apps.

According to recent surveys, it seems that end users are very conscious of application performance – in some cases even more so than functionality. The gaps in performance on different networks and locations are evident: the need to performance-test the application before launch date, on real devices, and provide sufficient insight to optimize it.

Based on our experience with mobile enterprises, building an efficient mobile performance test strategy should consist (at a minimum) of the following five pillars:

1. Defining the supported devices and operating systems

Mobile devices have a significant impact on application performance. Smartphones and tablets are, in essence, small computing devices and deliver powerful capabilities. On the other hand, however, they are highly constrained in terms of resources. The problem is that end users expect and demand the same level of performance (if not better) from their mobile apps as that which they get on their desktop computer. Therefore, selecting the right mix of mobile devices to test on prior to launch is one of the basic criteria for effective performance testing.

In our findings, different devices (and even the same device with a different OS version) may have a significantly different response to degraded network conditions or to server load.

Note that in order to stay in sync with your user community, the list of devices supported by your application should be dynamic and change in response to market trends (new devices, OS versions, etc.). This list should be updated on a quarterly basis, and your testing plan should take this into account.

2. Selecting the key business transactions

Select the functions of your application that users care about the most, and focus on testing against them using realistic and clear KPIs. In the initial testing stage, it is recommended to isolate your test environment and see how these business transactions work in a “clean room” environment. (Subsequently, these scenarios should also be tested in a production-like environment with interrupts such as incoming calls, messages, etc.).

3. Simulating various network conditions

Depending on the device and OS, mobile applications may behave differently, depending on the type of carrier network infrastructure/technology (3G, 4G, WiFi). According to Shunra, an authority in network virtualization for software testing, “Production network conditions such as inconsistent bandwidth, high jitter, increased latency and packet loss all work to degrade application performance."

Thus, it is imperative to analyze the impact of network conditions in your mobile performance testing scenarios. Your Performance team should simulate such network conditions, measure the user-facing KPIs (see above), and generate a network traffic sniffer (PCAP), which can serve as input to the optimization analysis phase as well as a baseline for the server load test script.

4. Building server load

When presented with material load, servers begin to perform in a way that may impact the end user experience. For instance, packets sent to the mobile device may be delayed, lost, de-sequenced, etc. The server load test measures the mobile end user experience on a real device while applying material load on the server farm.

The measurements on real devices are done simply by running recurring tests on the real devices repeatedly throughout the test session and measuring the KPIs users care about. You can make the synthetic traffic load “mobile-relevant” by converting the network traffic sniffer PCAP file into a load script. Make sure you are able to isolate device issues from network issues and application functional defects, which may be specific to a device or mobile OS.

5. Analyzing, debugging and optimizing

The output of the performance cycle based on the steps above would typically be a detailed report showing the key statistics (response time, availability and other) per transaction and per real mobile device under different conditions. Such a report should highlight and present the bottlenecks in terms of network traffic, detailed device vitals, etc. Your Performance team needs to analyze and triage these reports in order to pinpoint the root cause of any performance issues (which may be network-related problems or issues related to the device itself).

Bottom Line

As the mobile market continues to grow, new operating systems and a growing number of vendors are joining this rapidly expanding ecosystem. In the world of enterprise mobility, the importance of mobile performance testing continues to increase and has necessarily become a key part of the software development life cycle. By employing the right testing strategy and tools, and enabling access to the widest variety of real devices and simulated networks, your organization can meet the challenges of mobile application performance, assure end-user satisfaction and optimize business results.

Eran Kinsbruner is Director of Product Marketing at Perfecto Mobile.

The Latest

March 23, 2017

Mature development organizations ensure automated security is woven into their DevOps practice, early, everywhere, and at scale, according to Sonatype's 2017 DevSecOps Community Survey ...

March 21, 2017

When it comes to food, we all know what's considered "good" and what's "bad". We can all understand this simple rule when eating. But for many, when it comes to software development, simple rules and advice from nutritional labels aren't always there for us ...

March 20, 2017

Monitoring and understanding what software is really doing, and maintaining good levels of software quality is increasingly important to software vendors today. Even a minor bug is capable of shutting down whole systems, and there is a real risk that development cycle pressure competes with quality assurance best practices ...

March 16, 2017

More than half (54 percent) of IT professionals surveyed indicate they have no access to self-service infrastructure, according to a new DevOps survey of 2,000 IT industry executives by Quali.This means that more than half of respondents take a ticket-based approach to infrastructure delivery, impacting productivity and increasing time to market ...

March 15, 2017

Driven by the adoption of cloud and modernization of application architectures, DevOps practices are fast gaining ground in companies that are interested in moving fast – with software eating everything - between "write code and throw it across the wall" to creating more pragmatic mechanisms that induce and maintain operational rigor. The intent behind DevOps (and DevSecOps) is quite noble and excellent in theory. Where it breaks down is in practice ...

March 13, 2017

There might be many people across organizations who claim that they’re using a DevOps approach, but often times, the “best practices” they’re using don’t align with DevOps methodologies. They can say what they do is “DevOps”, but what we’ve found is that many are actually not following basic agile methodology principles, and that’s not DevOps ...

March 09, 2017

The velocity and complexity of software delivery continues to increase as businesses adapt to new economic conditions. Optimizing and automating your deployment pipelines will dramatically reduce your lead times and enable you to deliver software faster and with better quality. Here are three more most common areas that generate the longest lead times ...

March 08, 2017

Every enterprise IT organization is unique in that it will have different bottlenecks and constraints in its deployment pipelines. With that being said, there are some common problem areas that typically produce the longest lead times in your software delivery process. Here are the most common areas that generate the longest lead times ...

March 06, 2017

The findings of an independent survey of IT leaders, application developers and database administrators, conducted by IDG Research for Datical, indicate that database administrators are unable to keep up with the pace and frequency of database changes caused by the accelerated pace of application releases, thus creating a bottleneck and delaying digital transformation initiatives. An overwhelming number of databases administrators (91 percent) and application development managers (90 percent) cited database updates as the cause for application release delays ...

March 02, 2017

A "Boost Caboose" is a secondary engine pulled on a trailer for the explicit purpose of increasing the output of the primary engine. In many ways, DevOps is its own form of Boost Caboose for application of agile methodologies within the modern software factory and SDLC/ADLC processes ...

Share this