Are post go-live mobile performance issues leaving you scratching your head? After all the unit testing, regression testing, and integration testing, it still wasn’t enough. How did that bug slip by you?
Chances are you neglected to do enough real device testing.
There’s a disconnect when it comes to our approach to mobile app testing. Despite developing an application for mobile devices, we don’t test on mobile devices. Instead, we sit on our desk and test on emulators and simulators. And when you think about it, that’s kind of like having a house built only to try to virtually test the structural integrity.
Unfortunately, the results can also be very similar.
In this article, we highlight the common woes of ignoring real device testing, what keeps people from testing on real devices over imitations, and how the landscape is shifting.
Emulators and simulators versus a real device
First, let’s talk about some of the alternatives to using a real device, namely emulators and simulators. Although the terms are commonly used interchangeably, emulators and simulators are different. A simulator tries to replicate the configuration of a device but can’t mimic hardware.
You get a replication of the operating system and the device configurations, but you can’t effectively test things like CPU or memory usage. Think of it as a reflection of a device—it looks like the real deal, but doesn’t feel like it. Simulators are best for light, UI-based testing like checking screen resolution and performing simple tasks that don’t put too much of a tax on the device, like navigation.
An emulator, like a simulator, mimics the software and configuration of a device. It also attempts to replicate the underlying hardware. This makes it more suitable for deeper testing than the simulator, like performing complex operations and even monitoring usage metrics. Almost every testing tool allows you to leverage emulators. They’re super cost-effective and have their place. (For more on the differences between simulators vs. emulators, see this article.)
Real device testing involves testing on a real device. Crazy, right? Real device testing is about experiencing an application exactly as a user would. While emulators can bring us near-perfect replication, there are some things they can’t do. Real device testing is the only way to truly replicate the experience of a user.
They definitely should have used a real device
Organizations don’t do enough real device testing. It tends to be more expensive and it ends up just being a lot easier to load up the emulator. Unfortunately, there are a lot of things that could go wrong if you ignore testing on a real device altogether.
Let’s take a look at four examples of such scenarios:
Scenario #1: Camera problems
A few days after a restaurant releases a new mobile app for its rewards program, certain Android users begin complaining about their app crashing when they try to scan their receipt barcode.
They used emulators and tested to make sure the tester could input the receipt number manually, but didn’t test in enough real devices. If they tested on a variety of real devices, they could’ve found that instead of asking for permission to access the camera, the device rebooted.
Scenario #2: Memory leak
Users of a calorie-tracking app begin to notice that the app is performing much slower than when they first started using it. There is a noticeable delay between components and pages. Sometimes they don’t render at all.
And now users are beginning to explore other apps.
If they had tested on a real device, they would’ve noticed that their app wasn’t releasing memory effectively, and was increasing memory usage over time, causing poor performance and degradation of functionality.
Scenario #3: GPS
A social app begins to quickly gain popularity. Users enjoy the ability to interact with others, its beautiful aesthetics, and its welcoming community.
A newly released feature uses geolocation to allow for better interaction, inspiring users to take their interactions on the go. Some Android users are having a hard time identifying the right location. They notice the app has frequent trouble picking up their GPS location.
If they’d used a real device cloud, they could’ve tested the GPS on multiple devices around the world, finding the issue earlier on.
Scenario #4: CPU
Some users of the above-mentioned app experience extremely slow speeds during important interactions.
The development team noticed that during resource-intensive functions of the app, there’s not enough memory being allocated properly, causing usage to skyrocket.
Since the testing was performed on emulators, instead of real devices, developers never noticed this serious, yet common issue.
An app that is not optimized for real device usage can consume a significant amount of battery power, leading to shorter battery life and a poor user experience. This can happen due to background processes, resource-intensive operations, or other factors that are not properly identified and addressed during testing.
Real device testing online is easy
These are just a few scenarios where depending on an imitation of a device to run tests rather than testing on the device itself can lead to real problems. So why not just skip the uncertainty of testing on simulators and emulators? Why leave something to chance? And no, you don’t need to manage the storage and maintenance of hundreds of physical devices.
With Sofy’s Real Device Lab, you have access to hundreds of mobile devices, all hosted in the cloud. You don’t even have to search for different builds and environments. With every device acquisition, you receive a clean device with your application installed. Then, just test your application, automate the test, and run it across every device in minutes.
It doesn’t make sense to spend so much time developing a mobile app only to leave the testing to an emulator. Real device testing identifies more issues and results in smoother go-lives. Save yourself the regret.
Disclaimer: The views and opinions expressed above are those of the contributor and do not necessarily represent or reflect the official beliefs or positions of Sofy.
