With the growing reliance on mobile devices and apps in our everyday lives, ensuring your application performs flawlessly across many devices is more crucial than ever. However, the landscape of mobile app testing is fraught with challenges. With countless devices on the market, each with different operating systems, screen sizes, and resolutions, selecting the right devices for testing can be daunting.
We’ve compiled this guide to help you navigate these challenges in 2024 and choose the right devices for testing your mobile app.
Understanding Mobile App Testing Challenges
The sheer variety of mobile devices available today presents a significant challenge. There are over 40 different iOS devices and 24,000 Android devices. Additionally, around 13 new mobile devices will be released in 2024, from the latest iPhone 16 to the Pixel Pro 9, each with unique specifications and capabilities.
This challenge, known as device fragmentation, complicates the app testing process, making it difficult to ensure consistent app performance across all devices. With so many different devices and OS versions, companies must identify the best strategies for choosing the right devices to test their app.
Best Practices: Choosing Mobile Devices for Testing
Considering Your Audience and Perform Market Research
Begin by understanding your target audience. Who are they, and what devices do they use? Market research is invaluable here, providing insights into the most popular devices among your potential users. This approach ensures you’re focused on testing the devices your audience uses.
Work with your Marketing team to identify user personas and usage patterns, or use external data sources like Statista to see which devices are popular. In your search, consider the following data topics:
- Market share
- Traffic volume
- Operating system (OS) versions
- Screen sizes
- Popularity
- Region
- Industry
- Company size
If your app is already on the market, you can also use your app’s user analytics and data to decide which devices to focus your testing on. There are many helpful analytics tools out there (like Google Analytics or Glassbox) that can help you visualize your user data.
Mobile Device Market Research Trends from 2023
Here are some mobile device trends you can look at to help you make an informed decision:
- Android has a 69.88% global market share, while iOS has a 29.39% global market share (99% combined OS market share).
- This Stat Counter report breaks down the global smartphone market share by manufacturer in 2023:
- Apple: 29%
- Samsung: 25.4%
- Xiaomi: 11.6%
- Oppo: 6.2%
- Vivo: 5.2%
- Huawei: 4.1%
- Realme: 3.7%
- Other: 14.8%
- Here are the most used Android versions worldwide in 2023:
- Android 13: 24.35%
- Android 12: 20.19%
- Android 11: 19.64%
- Android 10: 16.62%
- 9.0 Pie: 7.17%
- 8.0 Oreo: 2.21%
- 7.0 Nougat: 1.81%
- 6.0 Marshmallow 1.56%
- Other: 2.87%
- Here are the most used iOS versions worldwide:
- iOS 17: 66%
- iOS 16: 23%
- Earlier versions: 11%
- This Yahoo! Finance report lists the top 10 bestselling smartphones worldwide (by units shipped) in 2023, using data from Omdia’s Smartphone Model Market Tracker:
- App iPhone 14 Pro Max: 26.5 million
- Apple iPhone 14 Pro: 21 million
- Apple iPhone 14: 16.5 million
- Apple iPhone 13: 15.5 million
- Samsung Galaxy A14: 12.4 million
- Samsung Galaxy S23 Ultra: 9.6 million
- Samsung Galaxy A14 5G: 9 million
- Samsung Galaxy A54 5G: 8.8 million
- Samsung Galaxy A34 5G: 7.1 million
- Apple iPhone 11: 6.9 million
Device Lab vs. Real-Device Cloud Farm vs. Emulators/Simulators
Choosing the right testing environment is pivotal for ensuring the quality and success of your mobile app. Between device labs, real-device cloud farms, and online emulators/simulators, each option offers distinct advantages and challenges.
Device Lab
A device lab is a collection of physical devices—smartphones, tablets, wearables—organized in a specific location where testers can manually test the mobile app across different hardware configurations, operating systems, and network conditions.
Pros:
- Real User Conditions: Testing on actual devices provides the most accurate insights into how an app will perform in the hands of users.
- Comprehensive Interaction Testing: Allows for the testing of hardware-dependent features, such as camera, GPS, accelerometer, etc.
Cons:
- Costly: Requires significant investment in purchasing, updating, and maintaining various devices.
- Scalability Issues: Expanding the device lab to include new or replace outdated devices can be logistically challenging and expensive.
Real-Device Cloud Farm
Real-device cloud farms offer access to physical devices hosted in a cloud infrastructure. Users can remotely access and test their apps on these devices through the internet, providing flexibility and a broad selection of devices without needing physical possession.
Pros:
- Accessibility and Scalability: Offers the ability to test on many devices without the need to own them, quickly scaling up or down based on testing needs.
- Cost-Effectiveness: Reduces the overhead costs associated with purchasing and maintaining an extensive collection of devices.
Cons:
- Network Dependency: Testing quality can be influenced by internet connection speeds and reliability.
- Limited Immediate Access: There may be wait times for accessing specific popular devices in high-demand periods.
Emulators & Simulators
Emulators and simulators are software applications that mimic the software and hardware of mobile devices. They allow developers and testers to execute mobile apps within a simulated environment on their computers without physical devices
Pros:
- Cost-Efficiency: They eliminate the need for a physical device, making it a budget-friendly option for initial testing phases.
- Quick Setup and Testing: Ideal for automated testing and continuous integration, allowing for rapid development cycles.
Cons:
- Limited Realism: Can’t perfectly replicate real devices’ hardware performance or characteristics, potentially overlooking issues arising on actual hardware.
- Hardware Feature Testing: Inadequate for testing app interactions with device-specific hardware features, like biometrics or cameras.
Decide on the Number of Devices to Test On
While testing on as many devices as possible is tempting, it’s essential to balance thoroughness and practicality. Start with the most popular devices among your target audience and expand as needed. The goal is to cover a broad spectrum of devices without allocating unnecessary resources to less common models.
The size of your business will also determine how many devices you want to test your app on. For smaller businesses with limited resources, cost and time are important considerations when choosing how many devices to test on.
Based on the size of your business, we’ve laid out some recommendations on how many devices you should test on:
|
|
Startups |
Small to medium businesses (SMB) |
Enterprise |
|
Number of devices |
10-15 |
15-20 |
25+ |
Choose a Diverse Range of Devices
To ensure comprehensive testing coverage, select various devices with different operating systems (iOS, Android), screen sizes, and resolutions. This variety helps identify any device-specific issues that could affect user experience. Additionally, consider testing older and newer models on different OS versions to gauge your app’s performance across various hardware and software capabilities.
Again, the size of your business will determine the amount of time and resources you can dedicate to testing your app. Based on the size of your business, we’ve recommended which devices you should consider testing based on popularity, OS version, and screen size:
|
|
Startups |
Small to medium businesses (SMB) |
Enterprise |
|
Android |
Samsung Galaxy S23 (Android 14) Google Pixel 8 / 7 (Android 13) Samsung Galaxy S20/21 (Android 12) Samsung Galaxy A50 / A51 / A52 (Android 11) Samsung Galaxy A40 / A41 / A42 (Android 10) Samsung Galaxy A11 / A12 / A13 (Android 9) Google Pixel 8 / 8 Pro Samsung Galaxy Tab A8 (latest Android version) |
Same as Startups, but add: Samsung Galaxy S23/S22 Ultra (Android 14) Samsung Note 20 / Note 20 Ultra (Android 14) Google Pixel 8/7 Pro (Android 13) Samsung Galaxy S7/8/9 (Android 8) Samsung Galaxy Tab S9 (Android 13) Samsung Galaxy Tab S6 Lite (Android 12) |
Same as Startups and SMB, but add: Samsung Galaxy Z Fold5 Samsung Galaxy Z Flip5 Xiaomi 14 Ultra / 14 Lite Huawei P60 / P60 Pro OPPO Find X5 Vivo V30 Vivo Y100t |
|
iOS |
iPhone 15 (iOS 17.x) iPhone SE 2nd or 3rd gen (iOS 16.x) iPhone X / XS (iOS 15.x) iPhone 6S / 7S / 8 (iOS 14.x) iPad 9th or 10th gen (latest iPadOS) |
Same as Startups, but add: iPhone 15/14/13 Pro Max (iOS 17.x) iPhone 13/12 mini (iOS 15.x) iPad Mini / Pro (latest iPadOS) |
Same as Startups and SMB, but mix in additional iPhones / iPads of different sizes and resolutions.
|
Be sure to choose devices that cover a wide range of screen sizes, resolutions, and software versions.
Reassess and Make Changes Based on Market and Device Usage Changes
The mobile device market is dynamic, with new devices being released and older ones becoming obsolete. Regularly reassess your device selection based on the latest market trends and device usage data. This proactive approach ensures that your testing strategy remains relevant and effective.
Testing More Devices with Sofy’s Real-Device Cloud
Navigating the complex landscape of mobile app testing in 2024 requires a strategic approach to device selection. By understanding your audience, choosing the right mix of devices, and staying adaptable to market changes, you can ensure that your app delivers a seamless and enjoyable user experience.
Sofy offers access to over 100 real devices for testing in one integrated platform. With Sofy, you can streamline your testing process, ensuring comprehensive coverage across a wide range of devices, all while saving time and resources. Embrace the future of mobile app testing with confidence and demo Sofy today, knowing you have the tools and strategies to succeed in the ever-changing mobile landscape.
