The main goals of software testing are to evaluate the application’s quality, to validate that features work as expected, and to provide feedback on usability and functionality. As with any step of the development process, shorter feedback loops are always better.
The purpose of this article is to provide an overview of the testing landscape specifically as it relates to apps built using Apache Cordova/PhoneGap. Doing so should help you create the shortest feedback loop possible given the many factors you’ll need to consider when coming up with a test plan.
The Testing Landscape
One of the most important takeaways of this article is that there is no substitute for testing on physical devices. There is something fundamentally different about using a keyboard and mouse versus using a touch screen and an on screen keyboard.
For instance, a mouse pointer offers a very high level of precision so it’s easier to click on small UI elements or to choose between items that are placed closely together. Touch keyboards are more cumbersome, making data entry heavy applications a chore on mobile devices.
Additionally, the on screen keyboard itself can cause UI problems, as it takes up valuable screen real estate. On the flip side, swipes and gestures tend to be easier using your finger on a touch screen than they are with a mouse.
While there are many great tools out there to help you speed up the testing process, most of them use a virtualized environment that you interact with using a mouse and keyboard. Frequently, the majority of your testing can be done using these tools, but it is very important to always do some testing on physical devices for the reasons listed above and also because the different platforms and various versions of those platforms can all have slight differences.
Given these considerations, there ends up being a few different levels of mobile app testing, which are: (1) using a mobile browser ‘simulator’, (2) using a platform specific emulated mobile device, and (3) using a physical device. There is even a fourth level, which is (4) test automation using cloud testing services. The details and pros and cons of each of these levels will be the topic of part 2 of this article.
Test Plan Factors
When coming up with a test plan, there will be a number of factors you’ll need to consider, as there is no one size fits all solution.
In general, your choices here will be iOS, Android, and maybe Windows Phone. However, Cordova supports many more platforms than just these three. Each platform you choose to support will incur a significant cost in both time and money.
Target Versions of Each Platform
iOS – Apple users tend to upgrade their devices very quickly, especially when compared to the other platforms. At the time of this writing (Oct 2014), iOS 8 has just been released, so for the next few months at least, you may have to continue to support iOS 7. Beyond that, be sure to check the numbers to see if it’s worth continuing to do so or if you can move to just supporting iOS 8.
Android – The Android platform is still very fragmented, but the most common ‘breakpoints’ that you’ll need to decide on supporting and testing are 2.3, 4.0-4.3, and 4.4 (plus a new version due out Nov 2014). Part of the reason for the fragmentation is that there are many more phone makers and models, which means that some phones will end up getting left behind (either by the manufacturer or the carrier) and are not upgradeable to the latest version of the platform.
Windows Phone – This platform makes up a very small percentage of the smart phone market, and it wasn’t until WP 8.1 that it started to make some headway. Version 8.1 made some significant changes over 8.0, so you’ll really need to check the numbers to make a compelling case for supporting anything before 8.1.
Advice – The general advice is twofold: 1) research your target market, and 2) start small. Identify your target market, and try to find out what platforms they use. Once you know that, try to determine some actual numbers behind which versions they typically have. If you have a website with analytics attached to it, you can gain some insight here by seeing which devices people are using to browse your site.
Each platform and version you decide to support will add to the amount of testing you need to do. If you start by supporting only the most popular versions of each platform, this allows you to focus your development and testing efforts and keeps your velocity high. Consider adding support for an older version of a platform only if there is quantifiable demand for it.
Team Size & Budget
Many software development decisions come down to team size and budget, and mobile app testing is no different. How many developers do you have? How many testers do you have? Do you even have dedicated testers?
If your team is very small, you might consider keeping the list of supported platforms/versions very small and then do the bulk of your testing on physical devices. If your team is very large and you have many dedicated testers, it can become pricey to buy physical devices for each of your testers to use. In this case, you might make heavier use of emulators or cloud testing services. If money is tight, you will most likely make more use of emulators. If money is not tight, buying a suite of physical devices can be a great choice as it provides the best testing environment.
The actual features and functionality of your app can affect how you can test it. For instance, does your app make heave use of gestures, swipes, or other touch-centric interactions? If so, you will have a very hard time testing these features on anything but a physical device.
Does your app require the use of Cordova plug-ins? If so, you won’t be able to test the dependent features using a simulator, meaning you have to run in an emulator or on a physical device.
Does your app access web services? If so, you’ll need to host those services somewhere that the emulators or physical devices will be able to access, as they typically won’t be able to hit services running on a developer’s machine.
I hope this article gave you some insight into the overall Cordova/PhoneGap testing landscape and helped provide some guidance for developing your test plan. In the next article, we’ll go through the characteristics and pros & cons of each level of testing, including simulators, emulators, physical devices, and cloud testing services. And as always, if you ever need any help with your mobile app development and testing, Imaginet is always here to help.
Imaginet offers Mobile Application development & testing for various platforms, including Windows Phone, iPhone, Android, and Blackberry. To learn more, contact us at 1-800-989-6022 or firstname.lastname@example.org.
Imaginet is your trusted technology partner who turns your business innovation ideas into reality. 18+ years | 1100+ satisfied customers | 2500+ successful engagements. Located in Dallas (Irving), Winnipeg, and Calgary. Services offered worldwide. Contact us today at email@example.com or 1-800-989-6022.