Blog_featureimage_CordovePhoneGap-Testing-Part-2

In Part 1 of this two part series, I talked about many of the considerations and trade-offs you’ll have to make when coming up with a plan for testing your Cordova app. The ultimate goal is to create a plan that effectively uses each level of testing so that your team can deliver a high quality app in an efficient manner.

In this article, I will explain in detail each level of testing so you can understand the capabilities of each testing method as well as their strengths and weaknesses.

Mobile Browser Emulator/Simulator

Cordova mobile app testingOften referred to as a simulator, this testing environment is mostly used on developer machines to perform “first pass” testing and debugging. It is meant to approximate the environment in which your web code will run when deployed to a device and run in each platform’s “web view” control (essentially a platform specific mobile browser).

The Apache Ripple simulator comes with many popular Cordova based development tools, such as the Multi-Device Hybrid App tools for Visual Studio and Telerik AppBuilder.

 

Strengths

  • Very fast testing cycles as the code is run straight from the developer’s computer.
  • Can access back-end services running on developer’s machine.
  • Familiar environment for debugging JavaScript, HTML, and CSS because it’s just like the Google Chrome developer tools.
  • Can simulate some phone features like geo location, network availability, and speed.
  • Easily take screenshots and video.
  • Low cost (free).

Weaknesses

  • Cannot run Cordova plug-ins.
  • Not the same browser/web view that will eventually run your code on each device. There are many small differences that can lead to platform specific bugs.
  • No software (on-screen) keyboard.
    • Note: The presence of a keyboard can considerably affect the UI and screen layout on small screens. The input experience is vastly different when using an on-screen touch keyboard vs. using a full desktop keyboard.)

Platform Specific Emulator

stock_mobile94Each mobile platform has emulators available for testing apps in a virtualized environment. The emulators function very similarly to actual devices, though they are generally slower.

They also have a big drawback in that all interactions are performed with the mouse and keyboard, which is obviously not how the app will eventually be used.

Strengths

  • Test many different device versions and configurations using one physical computer.
  • Can test Cordova plug-ins.
  • Can easily deploy code directly to the emulator from many IDEs.
    • Note: If targeting iOS and developing on Windows, deploying to an iOS emulator on a Mac can be a pain. The Visual Studio tools can help (documentation).
  • Can access back-end services running on developer’s machine.
    • Note: Very easy with Windows 8.1 and Android, and with some effort, iOS emulators can work too.
  • Can debug using Chrome dev tools with Android 4.4+.
  • Easily take screenshots and video.
  • Low cost, sort of (free only if you own a Mac).

Weaknesses

  • Mac required for iOS emulation.
  • Performance is slower than physical devices.
  • Android emulation can be extremely slow if CPU does not support vt-X.
  • Has software keyboard, but you use it with your mouse, which is very different than how you use it with your hands.

Physical Devices

stock_mobile92Testing on physical devices is the only way to know for certain how your app will function out in the wild. The biggest obvious drawback is that it can cost a lot of money to own enough devices to cover the spectrum of supported platforms and versions of each platform.

In general, you might be able to get away with testing on physical versions of only the most popular phones and then leave the rest to emulators or testing services. For instance, as of this writing (Oct 2014), you might consider buying some combination of an iPhone 6 (or 5), Google Nexus 5, Samsung Galaxy S5, Samsung Galaxy Note 4 (or 3), and a Nokia Lumia 930 if you’re supporting Windows 8.1.

Strengths

  • 100% accurate representation of final product.
  • Only true software keyboard experience.
  • Fast performance.
  • Can test Cordova plug-ins.
  • Harder to debug, but still possible.
  • Screenshots are easy, especially with Dropbox photo sync/upload.

Weaknesses

  • Costly to own a bunch of different devices.
  • Cannot access back-end services running on developer’s machine.
  • Deployment is more involved and slower, but there are tools to help.
  • Logging bugs can be trickier as it’s harder to record the screen.
    • Note: Use Reflector to stream iOS screen to PC and then use screen capture software to record video. Also useful for demos.

Testing Services

stock_mobile93As mobile development in general continues to mature, automated testing of mobile apps is maturing along with it. There is now a growing number of testing and automation options available, ranging from paid cloud hosted solutions like Perfecto Mobile to free open source frameworks like Appium.

While the initial setup process can be time consuming, one of the main benefits of automated UI testing is that you can increase your confidence that code changes have not broken existing functionality on any of the platforms and devices you support.

Strengths

  • Automate tests on a wide variety of platforms and devices.
  • Many different automation options available, plus manual testing too.
  • Can be cheaper than owning a bunch of physical devices.
  • Can test Cordova plug-ins.

Weaknesses

  • Can potentially require a large time investment to setup and maintain tests.
  • Doesn’t adequately test ‘touch’ features and gestures.
  • Can potentially cost a lot of money.
  • Can be difficult to debug issues.
  • Must deploy back-end services somewhere accessible by automated tests.

Wrapping Up

As you can see, there is a wide variety of options to choose from for testing mobile apps. Which option or combination of options you choose will depend on a number of factors, many of which were covered in Part 1 of this series.

In the end, the scale of your development and testing efforts ends up being one of the determining factors. If the app your making is very large and business critical, you will likely be more willing to do whatever it takes to ensure your app is working perfectly. On the other hand, if it’s just a small app you built for fun or to solve a small problem, you can get away with much more rudimentary testing practices.

Hopefully this short two part series provided some insight into the mobile testing landscape and helped you and your team come up with a testing strategy that works for everyone. And as always, if you ever need any help with your mobile app development and testing, Imaginet is always here to help.

=====

Imaginet offers Mobile Application development & testing for various platforms, including Windows Phone, iPhone, Android, and Blackberry. To learn more, contact us at 1-800-989-6022 or info@imaginet.com.

Imaginet is your trusted technology partner who turns your business innovation ideas into reality. 18+ years | 1100+ satisfied customers | 2500+ successful engagements. Located in Dallas (Irving), Winnipeg, and Calgary. Services offered worldwide. Contact us today at info@imaginet.com or 1-800-989-6022.

Joel Poiron

About Joel Poiron

Joel is a Solution Developer with Imaginet in Winnipeg, MB, Canada, and has over 8 years of experience delivering business value using the Microsoft platform and mobile application frameworks. He is a full stack developer who has built solutions using ASP.NET MVC, WPF, WCF, SQL Server, Windows Azure, Apache Cordova, and Xamarin to name a few. He has a passion for the web and for continuously improving both himself and the teams he works with.

Leave a Reply

Let‘s Talk.
  • Let's Talk

  • This field is for validation purposes and should be left unchanged.