The first time I drove a U-Haul moving truck, I thought, do they just let anyone drive this?
Well, OK, clearly not just anyone.
You need your driver’s license, but other than that, yes, anyone.
This was such a different feeling than driving my commuter—6,000 pounds different! The truck took corners differently. It accelerated differently. It needed to be driven differently. I would not know the way the truck felt if I hadn’t driven it, and despite having my license, I felt I was not prepared for the differences.
The same can be true with mobile testing—just because you’ve tested on one device, even of the same OS, doesn’t mean it would easily translate to another.
Luckily, it’s easy to procure virtual devices through emulators, virtual representations of a device, as part of a testing strategy. QA teams can quickly render a virtual device, deploy their application, and start end-to-end mobile testing. Testing with emulators can help developers verify that their code works on all versions of operating systems, other phones, and tablets. They’ve got confidence that their app can work anywhere!
But does it?
While there is certainly a benefit to emulator testing, there are also plenty of gaps, and when it comes to mobile device testing, nothing beats the real thing. Let’s take a look at some of our testing options and dive deeper into real device testing.
Yet before we begin and launch into the many benefits of cloud-based real device testing, let’s quickly review the terms simulator and emulator. Specifically, how do they differ?
What is a simulator?
A simulator attempts to replicate the configuration and software of a device. However, simulations do not mimic the hardware. With a simulator, you get a replication of the operating system and the device version configurations, but you wouldn’t be able to effectively test CPU performance. Simulators are best used for light, UI-based testing like checking screen resolution and performing simple tasks like navigation that don’t put too much of a tax on the device.
What is an emulator?
An emulator, like a simulator, mimics the software and configuration of a device. In addition, it also attempts to replicate the underlying hardware. This makes emulators more suitable for deeper testing than the simulator – like performing complex operations and monitoring usage metrics.
Depending on the use case, it can make sense to use one over the other in different situations. Simulators do things that unit testing can’t, emulators do things that simulators can’t.
But real device testing does it all.
What is cloud-based real device testing?
The phrase cloud-based real device testing is mostly self-explanatory: Mobile application testing takes place on the device itself, rather than an emulator or simulator. And this is done from anywhere via the cloud. Whether accessed via the cloud or in person, real device testing is about being closer to how the user will experience the app.
While the ideal emulator could conceivably provide near-perfect replication, there are some things emulators simply can’t do and real device testing is the only way to truly replicate the experience of a user. For example, consider the following:
- Network noise: The device is connected to a WiFi network and experiences traffic typical to real world, real user use. Are text messages or notifications from other applications affecting the performance of the app? Only real device testing can certainly represent a truly connected device.
- CPU: Real device testing doesn’t have to mimic the effects of usage because it contains the actual hardware and memory allocation representative of a real testing scenario. How does your app react when multiple other apps are running in the background? What are the effects on the battery during background processes and foreground processes?
- UI/UX: Emulators can conceivably provide a very close representation of an application UI, but nothing can match the user experience like a real device. How do screen resolution, swipe responsiveness, and haptics feel in the palm of your hand? Experience the application with different screen brightness and background processes.
- Battery: Does your application drain more battery than it should? Using real device testing can reveal battery usage issues in a way that emulators cannot. (Read more on how to avoid battery drain here.)
So, why wouldn’t we just always use the real thing?
Real device testing offers the most accurate representation of a device’s performance in end-to-end testing. But this benefit generally comes at a steep cost. Firstly, it requires the physical device, which needs to be purchased, updated, and managed, all with new devices coming out frequently. It also requires the tester to have access to a device lab, which in a remote environment has traditionally proven difficult.
Another potential downside is that real devices can lend themselves to false positive and negative results. Real devices are just that—real. Device quality issues may impact app performance through no fault of the application. Some devices will be defunct or work less optimally than others of the same.
While many QA teams will jump at the chance to test on actual devices over emulators as early into the software development cycle as possible, because of the traditional increased cost and logistic complexity of real device testing, it has generally been common to save real device testing to the end of the development cycle as an effective way to perform end-to-end testing.
Cloud-based real-device testing
Fortunately, due to a series of recent technical innovations, you don’t have to have the real device in front of you to perform real device testing. Today, you can leverage the cloud to host your own mobile devices or use other devices to perform testing.
Sofy’s Real Device Lab grants testers access to hundreds of devices in the cloud. Wherever your team is, they benefit from access to real device testing. With each device acquisition, you get a clean device with your application installed – no more searching for builds and environments. You can also bring your device and grant access through the cloud to the rest of the team. And there’s no need to maintain or update devices in the lab. Sofy handles that.
If you’re using emulators and simulators, keep in mind that there are real benefits to real device testing, namely more accuracy, more consistency, and deeper testing. These benefits of real device testing aren’t limited to owning each device. Today, you can take advantage of real devices in the cloud, where devices are virtualized and made available to your entire QA team.
Disclaimer: The views and opinions expressed above are those of the contributor and do not necessarily represent or reflect the official beliefs or positions of Sofy.
