Table of Contents
EXECUTIVE SUMMARY
The unparalled growth of smartphone devices has opened up avenues for organizations to integrate them into the mainstream computing environment. Today’s mobile apps deliver complex functionality on platforms that have limited resources for computing.
Mobile app testing is complicated by the fact that it has, without any debate, the most complex array of devices/handsets, numerous OEMs, variety of OS versions, different screen resolutions, and carriers for any mobile platform available.
Despite encircling a multitude number of devices, there are few things that can and should be tested across the field. It is especially important to test these mobile apps on as many devices as possible because something that works perfectly on one device might cause a bug on another. Your end users will use a variety of smartphones and so your app needs to work consistently on all these smartphones for each of the OEMs.
Many enterprise applications (desktop based, web based) are now being made available and ported as mobile apps on all major Mobile OS. These mobile apps will empower workforces across various departments such as sales, supply chain, on-field support executives, delivery agents etc.
Although mobile apps have limited computing resources they are often built to be as agile & reliable as desktop based OR web based applications. In order to meet these requirements & challenges, mobile app testing has evolved as a seperate stream of testing where the need for unique testing techniques is vital for example, Crash Discovery Testing technique.
Users want mobile apps to be simple & fast. Just one nagging bug or a frequent app crash or crash upon app launch can spoil the entire user experience. With so much competition in this mobile app space, if the users do not get an excellent & compelling experience with your app, they will switch to a another app or your competitor app much faster before you say, There is an app update available !
Yet surprisingly, there is no previously existing comprehensive guide on how to test the particular complexities of mobile device apps.
UNIQUE CHALLENGES IN MOBILE APP TESTING
- Mobile apps can be classified under 3 categories: Standalone apps, Consumer apps & Enterprise apps.
- Standalone apps reside in the device & do not interface with external systems.
Consumer apps are used mostly by individual users for their own interest & doesn’t interact with a number of external systems - Enterprise apps are built to perform resource intensive transactions that are typical of corporate computing environments. Enterprise apps also interface with external systems through WAP or HTTP protocols.
A few of the unique challenges in testing mobile apps are listed below:
Device(s) Diversity
The major challenge is the multiplicity of mobile devices with different capabilities, features & restrictions. Devices may have different technical capabilities such as amount of available memory, screen resolution, screen orientation & size of the display, network connectivity options & support for different standards & interfaces. This cannot be ignored because the fewer number of devices we test & certify, we are taking a chance of our mobile app locking cut on other potential devices/customers.
Network related challenges
There are over 500+ mobile network operators in the world & it is quite not possible to detail the network challenges without zeroing in on the location. It is a simple fact that we must be connected to the target network, to test an app on a particular telecom network. Ofcourse, travelling to every network operator as required is not possible as it becomes very expensive and obvious cost trade offs to be considered as well.
Use of Emulators & Actual devices
With the proliferation of smart phones & multiple OEMs/OS versions, usage of emulators is ever increasing. Emulators can be beneficial for testing features of an app; however actual devices should be used for validating the test results because of the computing processor or memory factors.
- Other challenges
- Limited power and handset storage
Network bypass is not possible to do with all emulators - Different mobile carriers/manufacturers
- Different mobile platforms/OS
- The cost of setting up a test script is higher than the cost of single manual test execution
WHAT IS A CRASH 
A crash is a sudden failure in a mobile app (or mobile OS or in the smartphone hardware) causing the app functionality to cease, often exiting on its own after encountering unhandled errors. This is different from a hang/freeze where the app or OS continues to run without obvious response to input.
a. IMPACT OF APP CRASH(es)
Nothing is worse than having an app crashing on you everytime you launch it. It can be more frustrating when the app dipped into your wallet before introducing you to the world of app irritation.
- Data loss & productivity, as the data won’t be saved
- App crashing on you again & again due to the previous crashes
- Loss of time as the user might have to login again or restart from beginning
- Loss of credibility
- End user patience dies yielding to the app uninstallation and moving on competitor app
b. REASONS FOR AN APP CRASH
One of the reasons for app crashes is the proliferation of Mobile OS’s, OEM devices, multiple screen resolutions, touch and keypad input methods, and frequent OS or app updates.
- Memory leaks
- Stack smashing
- Stress testing/multi tasking
- Sloppy/error code
- Synchronization conflicts
- Spotty network connection
- Incompatibility between app & OS as the app is not updated to the latest OS
- Hardware incompatibility between hybrid apps, mobile web apps when integrated to work with native apps like GPS, Camera, Contacts, Calender
- The 3rd party APIs/web-services used by the app when not supported by the device OS due to lack of OS update
c. CRASH LOGS-HOW TO FIND 
i. iOS [iPhone, iPad] : Sync your iOS device to iTunes application to copy the logs from the device to your PC & is available at C:\Users\<PC Name>\AppData\Roaming\Apple Computer\Logs\CrashReporter
ii. Android : Few 3rd party apps are available on the Google Play store through which we can locate & diagnose the app crashes & the related logs upon download & installation.
Few of these apps include Log collector, LogCat, CrashLogPro, LogCat Recorder, Log Viewer, Get Log Data etc.
TOP#10 APP CRASHES-TESTING SCENARIOS
a. TOP 10-APP CRASHES 
- Tapping on an interactive element during heavy data loading process triggers to the app to crash
- Tapping on a field/button when the app is revisited after an idle state triggers the app to crash
- Simultaneous tapping on mobile app objects/buttons, causes the app to crash
- Fast continous navigation between multiple screens within the app, causes it to crash
- Performing functional actions on the app & suddenly the network connectivity weakens or loses or fluctuates causes the app to crash
- Heavy input (increased data length) to an app field causes the app to crash when it cannot handle the data input eg. Fast forwarding the seek bar of a volume control, input a long list of string/special characters/numbers to a form field
- Revisiting an app from background when it is trying to load data, causes the app to crash
- The app during a search of current location using GPS services & suddenly when the GPS is turned OFF or network connectivity is lost, crashes the app
- Toggling between the languages available when the app is loading the data crashes the app
- Continous zoom in and zoom out of the app screen may trigger an app crash
b. TOP 10-TEST SCENARIOS 
- Disconnect the network connectivity in between the app usage to observe the app to crash
- Toggle between the network connectivity [WiFi to 3G ; WiFi to hotspot etc] in between the app usage to observe the app to crash
- In between the app usage, attempt for an external interruption(s) such as voice call, events, reminders, alarms, whatsapp call, camera launch etc to observe the app to crash
- Fast adjust the device orientation multiple times when the keypad is open to observe the app to crash
- Revisit an active app after an idle state to observe the app to crash
- Keep the device memory running low by opening lot of other apps in the background and observe that your app will crash during usage
- Fast scroll up, scroll down the app screen for data/text selection to observe the app crash
- Fast change the device orientation multiple times when the app is subjected functionally to observe the app crash
- Push the app to background when login or data processsing is happening & revisit the app immediately and do back & forth these actions, causes the app to crash
- Toggle between 1x to 2x mode on the iPad when the app is in use, to observe the app crash
CRASHES – STATISTICS
a. DO iOS APPS CRASH MORE THAN ANDROID ?
Mobile apps monitoring company ‘Criticism’ compiled stats for app crashes & discovered that the largest proportion of app crashes from both iOS & Android platforms were on iOS 16.0.1 with 28.64% of overall crashes (in a normalized dataset). That makes senses since iOS 16 was still relatively new at that time and many apps still need to work out the kinks with the new OS.
b. WHY ARE THESE APP CRASHES MORE & MORE INCREASING ?
The data raises 2 main questions for us: why do apps on these Mobile OS crash so much and do iOS apps crash more than Android apps ? This can be due to hardware issues, location/GPS related issues, camera interference issues or probably the network connectivity type related issues i.e., how a smart phone connects to 3G/4G/5G, Wifi, hotspot or that the device is not connected to the internet at a certain time or that something happens during the switch between 4G and WiFi. There could also be issues with language support on certain devices & memory related issues if an app consumes too much memory without releasing it ontime when not needed.
CONCLUSION
A few guidelines to prevent mobile apps from crashes
- A few guidelines to prevent mobile apps from crashes
- Usage of proper error & exception handling in the source code
- By preventing memory leaks i.e., proper allocation & de-allocation of memory back to the Mobile OS
- By regularly installing the updates made available to the app and/or Mobile OS
- Avoid usage of 3rd party APIs while building iOS apps
- Conducting testing in uncontrolled real world test conditions (field based testing) is necessary, especially for multi-tier based mobile apps
- Verify the end to end functional workflows in all possible supported OS versions, platforms, OEMs, screen resolutions, device orientations atleast once
- Conduct stress testing, GUI testing, compatibility testing using real devices and not simulators/emulators due to memory & processor considerations
USA