Results from the DeviceAnywhere Test Center Customer Survey

12 04 2013

A few months ago Keynote sent out a survey to our DeviceAnywhere customers and we wanted to share some of the results.

The survey first asked a few demographic questions “What is your role in your company?”.

Tcdsurveyblog1It’s somewhat surprising that only about half of users are in QA.   More developers and product/project managers were using our product than expected. Also interesting were the “Other” responses—mostly because they indicate that this was a poorly worded question! Many of these responses were “Manager/Director/VP of Engineering/QA.” The word “role” was interpreted as job title, when it was intended to mean functional area. That’s one of the good and bad things about surveys – when you get the question wrong you don’t get the best results, but it’s also very easy to identify sub-optimal questions so you can improve them for the next time.

The survey also asked users “What type of mobile products do you test using DeviceAnywhere?” and allowed multiple answers.

Tcdsurveyblog3
It’s interesting to see such a high % of customers testing Websites, where in the past Native and Hybrid Apps has been the most popular responses for our product.  It does highlight the priority that companies are placing on providing quality mobile optimized versions of their websites. Here’s a good reason why in this quote from a Forbes article: “Two-thirds of smartphone users say a mobile-friendly site makes them more likely to buy a company’s product or service.” You can see the full article here.

Now perhaps the most important question in the survey – we asked about 5 device dimensions, including Reliability (do the devices work as expected), Cleanliness (do the devices have evidence of past use), Speed (interaction with the devices), Availability (are the devices available when you want to use them, and not offline or busy) and Model Availability (do we offer the device models you need). Users were asked to rank the importance of improving each of those dimensions. The chart below shows how many respondents said a given dimension was the most important to improve.

Tcdsurveyblog4
The only surprise here is that so few respondents said Cleanliness was the most important item to improve. However, in thinking about this, it makes sense. Getting on a cluttered device is a big annoyance; however, it will not completely block testing, unlike just not having the model you need. So, this is not an endorsement of “dirty devices”—we are still working on improving the cleanup process! However, it does mean that Keynote needs to place a very high focus on the other items on the list, most important of which are device interaction speed and new model availability. We are in various stages of working on all of the above items, and future blog posts will have more details once we are closer to making improvements available.

Thanks to everyone who gave us their feedback; this information is invaluable to the product, engineering, and operations teams in helping us prioritize efforts that will provide the biggest benefit to our customers. We always want to hear from our customers, so please either respond to this post or send us an email anytime at products@keynote.com.





The Keys to a Successful Mobile App Strategy

4 04 2013

processThere are many phases in the testing of any mobile app or website, specifically when it comes to QA and development. Within development (or pre-launch) there is Functional Testing, Usability Testing, and Performance Testing, amongst others. And in production (post-launch) there is Benchmark Testing, Availability Monitoring, and Performance Monitoring. All of these types of testing are critical to the success of any mobile platform. While there are many factors that can affect an apps success,  the importance of having a mobile strategy that incorporates testing is crucial.

For development teams the use of emulators and real devices for testing is vital to ensure the app or site is of the highest quality to avoid pitfalls such as misplaced images, non-functioning apps, apps crashing, broken links, etc.

For Web sites or Web apps it is viewable by users around the world. Even if you’re initially targeting only users in a single country or on a single network, it helps to understand the global dynamic.

When you test mobile Web apps or sites you encounter several challenges presented by the nature of the global, mobile Web. As we understand the nature of each challenge, we can explore different technology options to manage issues and mitigate risk. Coming up with the right solutions for your requires an assessment of the advantages and disadvantages inherent in each of the testing options available to you and determining the technology that best suits your testing requirements. These mobile testing challenges include

devices, network, and scripting.
For native apps, while application testing has always been an important step in the application development process, its importance is becoming even more critical for the following reasons, as adapted from an ABI whitepaper:

  • Mobile applications drive productivity – they must work.
  • Device platforms vary – applications developed for one platform must work on other platforms. For example, device fragmentation on different Android devices.
  • Applications will evolve – as worker’s needs and responsibilities change, applications will be both upgraded as well as downgraded in functionality.
  • Cloud systems affect where data is stored. In addition, authorizations and connectivity APIs are not consistent across cloud service providers.
  • Multiple wireless networks – businesses connect to different radio networks based on network capabilities, worker needs, and contractual relationships. But all of these conditions can change which will require modifying the application.
  • Operator choice – devices commissioned for use on a network in France may be recommissioned for use on a network in Germany. Or a business may change mobile operators. Applications will need to be tested on different operator networks to ensure consistent connectivity when upgrading in-the-field devices.
  • Worker demographics will change device type and application.




HTML5 and the Fight Over Who will Win the OS War

2 04 2013

201209-Cross-Platform-AppsMobile developers continue to struggle to determine how and on which OS’ to develop their mobile projects on.  As Michelle Fredette recently explained in her article that “Developing applications for multiple platforms is expensive, primarily because it takes developers a lot of time to create two or three versions of a single app. Not only does the coding itself take time, but the developers also have to learn multiple authoring systems: Xcode for Apple apps, Visual Studio for Windows apps, and the Android SDK development tools and platform for Android apps. And when updates are needed, which is inevitable, changes must be replicated across the various operating systems. Costs escalate even further when you account for the extra time needed to develop for different-sized devices. Ryan Matzner, writing in Mashable Tech, estimates that adding iPad compatibility to an iPhone app can increase development costs by 50 percent.”

 

She continues “Another drawback associated with native apps is that companies such as Apple act as gatekeepers: Users have to visit their app stores to download the app or update it, and these same companies must approve the app before it goes public–a process that can take weeks. It’s a lopsided arrangement with which not everyone is comfortable. “There are real concerns with [these] companies deciding who can and who can’t publish what on their stores,” explains John Kennedy, developer of Pocket Universe, a successful iOS app.

 

These costs and hosting concerns don’t apply to HTML5 solutions, which generally incorporate CSS (cascading style sheets) and JavaScript, the code language often used to augment HTML apps. All three are considered among the easiest codes to write, and can be written in Notepad or any number of free editors. Plus, you don’t need a development environment to compile them, just a browser for rendering.

 

An additional advantage of HTML5 involves updating. In truth, people are terrible about updating their apps when new versions come out. Browser-based apps draw their features and content in the form of data hosted on the web. If there’s an update, it’s incorporated in that data. The user doesn’t have to download it from an app store. The user might not even know about it.

 

Another benefit is flexibility, allowing a user to access the same piece of content from multiple devices, something that’s especially critical in higher education. Harvey Singh, CEO and founder of Instancy, a company specializing in web and mobile learning solutions, says it’s increasingly important for users to be able to “open a course on a desktop, then go on the road and access it using a mobile device and continue where they left off.”

 

So why aren’t we hearing the death knell for native apps? Well, HTML5 also has real limitations. With native apps, for example, course content can be downloaded to users’ devices, so they don’t need constant access to the internet to work on a course. With a native app, furthermore, when network access is restored, files are automatically synchronized on the network.”

 

Keynote’s DeviceAnywhere platform prides itself on being agnostic when it comes to devices and even OS’. Since our technology can work with any device on any OS, that becomes a non-issue and allows our customers the opportunity to switch platforms as they see fit. That being said, we do see value in HTML5, especially as it pertains to automated testing of mobile apps and websites. While traditionally this has been done on a 1:1 ratio leveraging scripting, HTML5 gives us a framework to perform automated testing across any device (provided it is an HTML5-enabled device, which is  becoming the standard in the industry).

To read the rest of Michelles article click here.

To learn more about HTML5 Web Tesing with DeviceAnywhere’s Automation platform click here.





For Testers, Shipping Devices Is a Major Headache – Mobile Cloud Testing to the Rescue!

28 03 2013

deviceanywhere_mobile_cloud_testingTraditionally, to conduct real device mobile testing vendors shipped devices to testers all over the world. A major roadblock is the upkeep of these devices and maintaining them with the latest OS version or device model. Shipping back and forth becomes a huge pain point as time and cost rises exponentially. With cloud-based services there is no need to ship, purchase devices, or subscribe to service plans.

Cloud-based architecture enables easy access to application performance information anywhere in the world and eliminates the need to have a physical device on hand to test.

Some cloud-based tools allow mobile teams to receive real-time alerts against any measurement criteria, allowing them to address issues before they impact end-users. Real device testing, such as Keynote’s DeviceAnywhere platform through the cloud gives teams the ability to access real devices from anywhere in the world. This is perfect for organizations that either offshore their testing because of cost or have internationally-based teams. These teams can have access to the latest devices from locations with an Internet connection.

For a free THREE HOUR trial with access to the LARGEST library of real devices in the world – click here!





The Android OS – the Ultimate in Device Fragmentation?

14 03 2013

animoca-androidIn an article posted in 2012 in TechTarget, Kim-Mai Cutler had the chance to interview “Animoca, a Hong Kong mobile app developer that has seen more than 70 million downloads, says it does quality assurance testing with about 400 Android devices. Again, that’s testing with four hundred different phones and tablets for every app they ship!”

She continues .. “The photo above is just a sampling of Animoca’s fleet of Android test units. Yat Siu, who is CEO of Animoca’s parent company Outblaze, snapped and posted it from Outblaze’s headquarters today. In total, Siu says their studio has detected about 600 unique Android devices on their network.

“We haven’t managed to track down all of those devices because, in large part, they are no longer available for sale,” he says. Sad cakes!

On top of that, Siu said that the number of handsets from the lower-end Asian manufacturers is also growing rapidly. These are the phone makers that Nokia chief executive Stephen Elop was probably talking about in his famous “burning platform” memo when he said that are Chinese OEMs were “cranking out a device much faster than, as one Nokia employee said only partially in jest, ‘the time that it takes us to polish a PowerPoint presentation.’” If you take those out, the actual number of devices you need to test for is much lower.

But if you want to break into Asian markets, these phones matter and make it especially challenging for Android developers to ensure their apps work on every single Android device. Android fragmentation is a huge issue because developers have to check their work on dozens of devices. Animoca happens to be backed by Intel Capital and IDG-Accel, so it has the resources to buy all of these devices for testing and pay employees to use them.

But imagine the long-tail of developers! Imagine the people who make the roughly 500,000 apps in the Google Play store. Total nightmare.

It puts a real dent in Eric Schmidt’s prediction from six months ago that developers might start going Android first within six months. His deadline is up now and there aren’t signs of this happening. Appcelerator did a survey of 2,100 of its developer clients in March and found that, if anything, interest in Android development is stagnating.

Siu isn’t fazed though. He’s told me in the past that thorough QA testing makes Animoca’s apps retain users better because so many other Android developers do a bad job at it. Unlike iOS users who throw up their hands in frustration, write bad reviews and just leave, Android users tend to be delighted when they find apps that work even if they have a glitch or two.

He adds, “We like fragmentation as users prefer choice. We are not big believers that one size fits all.”

At Keynote, we see this all the time. Our customers tell us of the pain of having to manage libraries of real devices and the vast variety of OS versions even just within Android, let alone iOS, BlackBerry, or Win 8.  Developers who started with one or two devices a few years ago are now up to 20-30 today as seen in this image from another developer – Pocket Gems. They had two of the 10 top-grossing games on iOS last year, according to Apple’s iTunes Rewind.

android-table

And there is no end in sight now with the release of new form factors from Apple such as the iPhone 5 and the iPad Mini. One of our customers – Microsoft’s Windows Live team was able to replace their entire cabinet of devices with the DeviceAnywhere platform. No longer having to check out devices to team members or update devices with different OS’, or manage/maintain separate plans for each device locking them into a 2-year agreement with the carrier. Mark Boyce of Microsoft’s Windows Live deployment team said at the time of the release “Keynote’s DeviceAnywhere solution has saved our department a lot of time and headaches by eliminating most of the tedious, manual tasks associated with our in-house mobile testing process. We no longer have to chase down lost devices and SIMs, dig through boxes of chargers, or even track multiple phone bills and expense reports.”

To learn how you too can benefit from access to real devices from anywhere in the world, eliminating the need to maintain your own library or even having to ship devices around the globe, check out Keynote’s DeviceAnywhere platform and we’ll even get you started with THREE free hours, TODAY!

To read more of Ms. Cutler’s article click here.





No End in Sight to Device Fragmentation for Mobile Developers

5 03 2013

fragmentation_200Fragmentation has always been a difficult issue, but new complexities have been introduced with the proliferation of mobile devices. Many companies are challenged with figuring out how to monetize mobile assets for all types of devices in a cost-effective and efficient way.

Although there has been a reduction of mobile OS’, the number of device types for Android and iOS is increasing. With Android being an open platform for developers, it introduces inherent differences in the functionality and performance of each device. For iOS, while there is less platform fragmentation, the introduction of the iPhone5 presented a new difference in form factor (screen size) from its previous models, adding to the complexities of fragmentation.

This device diversity creates a steep challenge to stay ahead of the curve in the today’s mobile market. Not addressing these issues can mean the difference between a successful mobile launch and wasted resources for broken applications.

Testing is critical and choosing the right devices is essential. To maximize your device coverage in the most efficient way check out the free Device Planner by Keynote’s DeviceAnywhere platform. With it, you can target the right devices and platforms for your mobile apps, websites,  or services.





Aricent Supports a Tier 1 UK Operator Utilizing Keynote’s DeviceAnywhere Platform for Automated Mobile Testing

4 03 2013

aricent_logo-774095Aricent Group, a global innovation and technology services company, working closely with Keynote, has developed a flexible, best-of-breed testing environment for a Tier 1 mobile network operator in the UK. This operator and other companies can now have complete visibility into how their applications and services run – in this case ten different real mobile devices running over 1,350 test cases, with growth expectations to top 3,200 test cases in the near future.

The Keynote-Aricent partnership has succeeded in removing the time-consuming process of manually testing each software update, or each new device, therefore reducing human error. By creating an automated testing environment to the operator’s specifications, testing is done by simply pressing a button.  Furthermore, the automation process clearly demonstrates the impact of different software upgrades and new features in order to create the highest level of service and quality.

Keynote’s DeviceAnywhere cloud-based platform allows companies to test mobile apps, websites and services on devices remotely – regardless of location or time – enabling operators to speed up their delivery without compromising on quality or reliability and therefore mitigating risk.

“By working with Keynote, we are able to help our customers take their services and devices to market with unprecedented speed and quality. Their solution was a natural complement to our existing network testing infrastructure.” said Adrian Luker, program manager at Aricent. “Our customers need to be completely aware of the experience they are providing for the end user. Even the smallest update can have a huge impact, and we need to know what the impact is, before it happens. Having this superior insight into what consumers will be experiencing, means that changes can happen quickly, and we can be confident that end users will see improvements, rather than be inconvenienced. This is tremendously important in reducing risk for our customers when launching any update.”

“The mobile market is highly competitive, and with smartphones making mobile Internet access increasingly prominent, the service an operator provides – in terms of updates, applications and maintenance – must be a seamless experience” said Vik Chaudhary, vice president of product management and corporate development at Keynote. “Delivering top quality service for consumers should be the highest priority for all customers, including operators, and we are delighted to be working with Aricent to deliver this. Mobile performance and functional proficiency can no longer be a guessing game, and companies need to understand exactly what customers are, and will be experiencing, in order to fully tap into these revenue streams.”

For the full release, click here








Follow

Get every new post delivered to your Inbox.

Join 1,388 other followers