Every Mobile Strategy Should Include Testing In The Cloud
By year’s end, approximately 1.4 billion smartphones will be in operation. According to a new study conducted by ABI Research, 798 million will run on Android, 294 million on Apple’s iOS, and 45 million on Windows Phone. It’s clear that mobile now influences the relationship between companies and their customers, making it increasingly important for enterprises to fully develop their mobile testing strategy.
Over the next year, we’ll see a spike in mobile-app development from enterprises—and it’s extremely crucial that businesses don’t rush the testing process (see “Don’t Cut Testing Corners”). Ensuring high-quality and high-performing applications that are void of compliance and security risks will set these applications apart. And, now more than ever, a secure mobile testing strategy is needed.
The Importance Of A Mobile Cloud
Every good strategy needs the right solutions to execute on its overarching goals. That’s why it’s vital to incorporate a mobile application testing partner that can ensure the applications are developed, tested, monitored, and launched with ease.
The ability to test apps on a Web-based Software-as-a-Service (SaaS) platform enables testers and developers to test on real mobile handsets and tablets connected to live mobile networks and across different geo-locations globally. Through use of the cloud, organizations can streamline development, testing, support, and monitoring processes;perform more tests on each handset; and ensure optimal compatibility of the application, site, or service on more handsets, networks, and locations.
As a result, enterprises gain better quality testing, shorter time-to-market, and an overall reduction in costs and logistical efforts. Furthermore, users receive a full-functioning mobile app without the potential glitches that arise when launched without a proper testing strategy
CapGemini’s latest World Quality Reportindicates that ignoring the cloud while testing mobile apps leaves organizations vulnerable to huge risks and increases the cost of these activities.
Identifying The Cloud Framework
Typically, a mobile cloud would require the following components:
- The physical devices (smartphones, tablets)
- Dedicated cradles to hold these devices
- A server that manages and administrates these devices (handset server)
- A streaming server (for video-based streaming from the devices to the end user)
- A centralized management-based server that would provide all of the functionality capabilities (engine) to the end user to operate the devices
- Network architecture for a secure, private cloud environment (routers, etc.)
Using the above infrastructure, end users can start testing mobile phones in the cloud by remotely connecting to the device (either through a web-based or software-based solution). Users should expect the following capabilities from the cloud-based infrastructure to efficiently work with the devices and manage complex use cases:
The ability to work remotely with a device’s file system:Many mobile applications utilize internal resources that rely on mobile file systems—media files, images, audio, etc. Access to this file system allows developers to compare these against application requirements. For instance, when a bank is planning to develop a check deposit application, a developer may place pre-defined, pre-loaded images of checks on the device file system. Then, because testers can’t use real checks, they can compare file images with the actual check image via the device’s file system.
The ability to deploy iOS/Android or other applications on remote target devices:This key parameter requires the creation of various application versions with new builds. Ideally, developers need to easily and remotely install these new builds on target devices.
Obtain media feedback from the device tests: Media feedback, such as images or screen shots, or video for recording the entire session, will provide testers with the best results. Other suggestions for media feedback include establishing a USB connection over a specific protocol (RTMP) and audio for voice-related application testing (e.g., testing SIRI or another voice-related application).
Ability to install new customized ROM on the device:Support for customized ROM allows developers to change mobile OSs on devices remotely, as well as install a pre-launch version to test before an application goes to market. It ultimately gives developers a head start in the market before application deployment.
Obtain device logs and device vitals (e.g., memory consumption, CPU status, etc.):This provides insight into CPU usage and battery consumption. It reveals what end users are undergoing during application usage, which is imperative for customer retention post-deployment.
Ability to simulate location based services (GPS spoofing)for location-based applications: Tests should be able to play with location on the cloud to test the functionality of location-based mobile applications, ensuring optimal availability to end users.
Let’s look at two use cases and analyze them from a cloud-based perspective: voice-related testing in the cloud and GPS location operation. For such voice-related capabilities, the devices in the cloud would require special wired connectivity to allow voice injection through the handset server to the target device and from the device to the end user. This is possible and can be done across the various platforms.
GPS/locationtesting in the cloud can be covered by enabling (e.g., in Android) GPS Mock Location, and setting different locations via available market applications (i.e. “My Fake Location”) (Figs. 1 and 2). Then, simply check how the mobile device and application reacts to a change to its current location.
The end user leverages the cloud-based environment by having a governed environment with all devices controlled, fully charged, and connected at all times to the Internet (either via carrier plan or Wi-Fi). Adding/removing devices would be seamless to the user.
With regards to security in a mobile cloud, such a governed environment becomes a key advantage—and a requirement—in the mobile space for most users. Therefore, a mobile cloud should be able to clean up each device once the end user ceases activity.
A mobile cloud also should provide users with a “private cloud,” which has its entire set of dedicated devices connected via a specific wireless access point. It would not be visible to the outside “world” or accessible from the outside network, thus assuring a secure working environment.
Performance And Monitoring
Mobile continues to penetrate businesses through the rising rates of mobile commerce (mCommerce) transactions, mobile banking transactions, healthcare and retail transactions, etc. Thus it becomes crucial for such organizations to gain insight on the end-user experience during application usage on a “real phone.”
Monitoring these applications on real devices in a cloud-based environment assures device availability in a repeatable environment, as opposed to random monitoring that’s based on the device connectivity to the network, the application being in the “air” at a given time during monitoring, etc. When monitoring in a structured and scheduled way, the organization can collect key performance indicators from the device under test and gain insight into the end-user experience.
Related Articles
• CTIA 2013: Your Mobile Future Revealed
• 4G/LTE Brings Changes In Wireless Test Approaches
• Wireless Test Adapts To The Onrush Of Complexity
Testing and monitoring mobile performance is only possible on real devices. It’s useless to load the server while not testing and monitoring the real devices. Organizations ought to capture the real end-user experience on real devices when the server is loaded to determine carrier network problems or specific device and mobile OS issues. Failing to cover these testing and monitoring problems early will likely lead to serious financial losses and potential tarnishing of the brand or company reputation.
Taking a look at a typical mobile-application life cycle, various processes can be used by the organizations (Fig. 3). Each process has its own value and benefits (this article doesn’t cover all of them). The performance testing of an application would take place prior to the “release” stage, while the “monitoring” is done in production post deployment.
Continuous Quality, Delivery And Integration
In a mobile cloud-based environment, where devices are governed, assured, and always connected, it’s easy to deploy a mechanism for continuous quality, delivery, and integration. Users can create a build engine that runs at any given time, as well as test on real devices anywhere in the world to get insights on the stability of the application, devices, and service. Furthermore, users can test their device behavior on a specific carrier while using a customized mobile operating system (in ROM) without any concern, and with the full support of the mobile cloud vendor.
The cloud vendor in a mobile cloud-based environment has the technical expertise to support an organization’s needs. In addition, the vendor enables the team to focus on real needs without worrying about the infrastructure, devices, or any other aspect (e.g., security, networking, etc.).
The Goal Of Mobile Operators
The mobile operator tester’s ultimate goal is the certification of new devices across mobile operating systems. Ensuring devices work perfectly on a carrier’s network across different geographical locations under various network conditions (2G, 3G, 4G, LTE, Wi-Fi,)—with different operating systems, including past systems—is imperative for business.
A mobile operator (network carrier such as Verizon Wireless or Vodafone) leverages the mobile cloud in disparate ways to meet certification requirements, especially when it comes to pre-launched handsets and applications. A mobile operator needs a private, internal, and secured environment that ensures the services work prior to exposing them to all users. Standard requirements for device certification include:
- Pre-loaded internal app testing and validationthat passes all tests across all devices, OSs, network conditions, etc. If testing a mobile Web site, then it should be tested on different Web browsers.
- No interruptions to quality of service (QoS)for mobile services, such as SMS, browsing, or data connectivity, via incoming events (QoS with interruptions, i.e., calls, emails, text, push notifications, etc.)
- Service assurance and monitoringof mobile services on real devices to avoid revenue leakage, optimal performance, and customer retention.
- Debugging of deployedmobile applications and troubleshooting end user issues on real devices. If a ticket is received from the field regarding a specific device (i.e., the app crashes, fails to connect, etc.), it’s possible to test on the cloud and choose the specific device to check a bug. Testers should send a device log and a full report with evidence of the issue to R&D.
Mobile Cloud Benefits
A mobile cloud offers the flexibility of testing both new and old devices for different testing purposes. Furthermore, a mobile cloud moves with the ever-changing mobile market, allowing organizations to focus on mobile quality rather than on mobile device governance.
When testing mobile applications in the cloud, users can test both their iOS and Android application from a functional perspective and perform real device end-user testing. The latter may involve testing for loss of network during application running, enabling/disabling networks on the device, rotating screens throughout the running application, undertaking various complex scenarios involving the device camera, and doing power cycles on the devices (to observe device behavior or impact when it occurs on a live application).
Mobile cloud users also are able to measure and collect real device vitals from the device itself, such as real device logs, memory consumption, network throughput, and so on. With proper support from the mobile cloud vendor, customers working in a mobile cloud can safely deploy and test their products on customized or pre-launched mobile OS versions.
The mobile cloud is the right platform to hold new-to-the-market devices. Integrating new devices into the cloud may take a few hours, yet the benefit is huge. Mobile operators often put new and pre-released devices under their private cloud to perform handset certification prior to their market release. Because a cloud platform is secure and isolated, it eliminates concerns of such devices becoming lost, broken, etc.
The mobile cloud is a dynamic environment. Mobile cloud customers (operators, application vendors, enterprises, developers, etc.) can track the market globally, and assure that only the most up-to-date and relevant smartphones and tablets exist in both the public cloud and private cloud. Therefore, the testing itself will be highly focused, and will guarantee sufficient coverage of the market. Furthermore, it can keep track of any newly released mobile operating system, and ensure quick support (e.g., Perfecto Mobile’s mobile cloud supports the latest iOS and Android versions).
Summary
In conclusion, most enterprises have already moved to cloud-based environments for storage (DropBox, Google Drive, etc.). That said, it has become imperative to leverage the mobile environment, especially in such a dynamic, fragmented, and huge market. Testing real devices at any given time, in any location, and on any carrier gives users needed flexibility and reduces serious risks to brands, reputation, quality, and more.
Mobile introduces different and complex use cases that involves unique hardware (GPS, NFC cards, audio-based scenarios), which require different tools that are usually hard to maintain and continuously use within a small office. The mobile cloud offers a secure, scalable, and assured environment, and removes obstacles and irrelevant worries, leading to enhanced execution and delivery.
Eran Kinsbruner, director of product marketing, was formerly mobile testing CTO at Texas Instruments and project manager at Matrix. He is the co-inventor of a test exclusion automated mechanism for mobile J2ME testing at Sun Microsystems. His professional mobile testing blog is at ek121268.wordpress.com.
Mobile-device companies can put themselves in peril without proper testing, considering the potential incompatibilities that may arise among all existing and new versions of phones, etc. Apple’s release of the iOS 6.1 offers one example of how application testing can go wrong.
The iOS 6.1 version was rushed to users, and it did not receive the requisite testing. It forced Vodafone, a major global carrier, to issue public statements and text messages to its subscriber network, warning them not to download the update because it will impact 3G performance for its iPhone 4S users (Fig. A). This represents a worst-case scenario—not only did users experience usage issues after an application update, but it also had influential members of the industry calling out the mistake.
In this instance, Apple had to release a specific OS version of the application solely to overcome the issue. Testing pre-release versions through the cloud across all devices could have detected this bug prior to publication, allowing Apple to mitigate these risks (Fig. B)
