Results from the DeviceAnywhere Test Center Customer Survey

12 04 2013

A few months ago Keynote sent out a survey to our DeviceAnywhere customers and we wanted to share some of the results.

The survey first asked a few demographic questions “What is your role in your company?”.

Tcdsurveyblog1It’s somewhat surprising that only about half of users are in QA.   More developers and product/project managers were using our product than expected. Also interesting were the “Other” responses—mostly because they indicate that this was a poorly worded question! Many of these responses were “Manager/Director/VP of Engineering/QA.” The word “role” was interpreted as job title, when it was intended to mean functional area. That’s one of the good and bad things about surveys – when you get the question wrong you don’t get the best results, but it’s also very easy to identify sub-optimal questions so you can improve them for the next time.

The survey also asked users “What type of mobile products do you test using DeviceAnywhere?” and allowed multiple answers.

Tcdsurveyblog3
It’s interesting to see such a high % of customers testing Websites, where in the past Native and Hybrid Apps has been the most popular responses for our product.  It does highlight the priority that companies are placing on providing quality mobile optimized versions of their websites. Here’s a good reason why in this quote from a Forbes article: “Two-thirds of smartphone users say a mobile-friendly site makes them more likely to buy a company’s product or service.” You can see the full article here.

Now perhaps the most important question in the survey – we asked about 5 device dimensions, including Reliability (do the devices work as expected), Cleanliness (do the devices have evidence of past use), Speed (interaction with the devices), Availability (are the devices available when you want to use them, and not offline or busy) and Model Availability (do we offer the device models you need). Users were asked to rank the importance of improving each of those dimensions. The chart below shows how many respondents said a given dimension was the most important to improve.

Tcdsurveyblog4
The only surprise here is that so few respondents said Cleanliness was the most important item to improve. However, in thinking about this, it makes sense. Getting on a cluttered device is a big annoyance; however, it will not completely block testing, unlike just not having the model you need. So, this is not an endorsement of “dirty devices”—we are still working on improving the cleanup process! However, it does mean that Keynote needs to place a very high focus on the other items on the list, most important of which are device interaction speed and new model availability. We are in various stages of working on all of the above items, and future blog posts will have more details once we are closer to making improvements available.

Thanks to everyone who gave us their feedback; this information is invaluable to the product, engineering, and operations teams in helping us prioritize efforts that will provide the biggest benefit to our customers. We always want to hear from our customers, so please either respond to this post or send us an email anytime at products@keynote.com.

Advertisements




Cross-platform App Development Frameworks: An Aspirin for Developer Headaches

12 04 2013

mobile-appsOne of the biggest challenges facing enterprise mobile app developers today is designing and developing apps across multiple platforms. In 2011-12, having a mobile strategy was imperative as the “consumerization of IT” and “BYOD” trends pervaded. As a result, enterprises now must shift their mobile strategies to support multiple mobile devices and platforms or else, lose the power to manage all the various devices within their environment.

Some companies can focus on developing apps for just one device type or mobile OS if the devices were company-issued, but this is becoming less and less the case. Businesses and brands must support more than one device or risk backlash from select employees for not supporting their non-supported devices. Keep in mind – as prevalent as the iPhone seems to be, Android has been caught up in popularity.

Cross-platform app development frameworks are becoming critical tools for developers because they’re designed to lessen the time and resources that developers or development teams has to allocate to creating apps for iOS, Android, BlackBerry, Windows Phone and beyond. By not spending excess time and effort creating apps to apply for different devices and mobile OSs, they can focus on what matters most – end user experience.

However, since each cross-platform development tool is unique and exhibits diverse features, capabilities and behaviors, developers will face increased challenges and opportunities designing successful device- and OS-agnostic mobile apps.

I recently spoke on a panel at Mobile+Web DevCon and my fellow panelists and I got into a discussion about the pros and cons of using cross-platform app development frameworks.

The Pros:

  • Reuseable Codes: Rather than having to write the specific action or sequence for each platform, a developer can just write the code once and then reuse those bits in later projects or on other platforms.
  • Plugins: Most cross-development frameworks offer easy access to plugins and modules that can easily integrate with other services and tools.
  • Easy for Web Developers: Most cross-platform frameworks are dynamic and simple for web developers to jump in and use, because many of these frameworks support HTML5 and CSS3.
  • Reduced Development Costs: This is perhaps the biggest advantage because it allows companies and brands to get an app onto other platforms without having to invest in a separate developer or team.
  • Support for Enterprise and Cloud Services: In addition to plugins and modules for specific functions, most frameworks also have the option to directly integrate with cloud services, including Salesforce.com, AWS, Box.net and others.
  • Easy Deployment: Deploying apps is much faster in a cross-platform scenario because it’s easier to incorporate one development code onto multiple devices. This is especially true with many of the new cloud-based tools that various frameworks are starting to push out.

The Cons:

  • The Framework Might Not Support Every Feature of an OS or Device: If, for example, Google, Apple or Microsoft adds a new feature, the framework being used will need to be updated to support those new functionalities.
  • You Can’t Always Use Your Own Tools: Most frameworks want users to use their own development tools and suites, and that can mean that a developer has to forgo his or her own preferences and use something else, even something unfamiliar.
  • Code Might Not Run as Fast: The cross-compilation process can sometimes be slower because it may take longer to load than native tools.
  • High-End Graphics and 3D Support is Often Limited: Fortunately, game-centric development platforms, like Unity, are here to help fill in those gaps.

When considering the pros and cons of app development, Josh Clark, Interaction Design guru, states that app design is one of the major factors cross-platform developers need to be aware of — whether they use a framework or not. Designing an app for the iPhone is different than designing one for a tablet; the UI and UX conventions are different, and touch points and menus work in different ways.

In addition to app design, it’s also important to factor in who the app is being developed for. This pertains to anything from the mobile web apps to e-publications to native apps.

Certainly a couple of years back, developers could quite safely shoot for iOS first, think about Android later and ignore everything else. Now, there are many more options, and although its pros and cons are almost equivalent in nature, taking a cross-platform approach to mobile app design and development appears to be the wave of the future for app developers.

Regardless of which platform, if not all, you’re developing for, app testing will still be one of the most critical steps of the app development lifecycle. Just because an app works fine on iOS doesn’t mean it’ll work just as well on Android devices. Likewise, just because an app works fine on an emulator doesn’t mean it’ll work fine on a real device. So, test early, test often, and on real devices to ensure the quality of the app. A trend I’m realizing is that end-users are becoming very unforgiving about buggy apps which are, quite frankly, synonymous with “revenue- and reputation-killer.” App development is a hot industry and the market is saturated with app developers, so being blasé about app quality is not an option.





The Right Tool for the Job – Building an Automation Testing Matrix

9 04 2013
Untitled-1For Desktop-based testing it’s a no-brainer: Use object-based scripting to maximize reuse across platforms/browsers.
In today’s mobile world it really isn’t that simple. There are many different platforms, OS versions, form factors and carrier/manufacturer customizations. Multiply that by mobile web, native app, or some hybrid in-between and you’ve got yourself a healthy testing matrix. A daunting task for even the most skilled Automation Engineer.

In order to tackle this problem, an Automation Engineer cannot simply look at it from a “one size fits all” perspective to create a set of objects and re-use them across all combinations of platforms. For example, there are fundamental differences in how an app behaves on iOS and Android, even with something as basic as a “back button” has its quirks.

Although these fundamental differences can be grouped together as a step or action, they are unique enough to not be able to simply share an object between the two OS’s.In some cases with mobile testing, you may be able to get to the object-level, however this usually requires that you instrument your app, or test on an emulator. While this fulfills a piece of your testing matrix, you will probably need to seek a couple tools to get this done across all platforms. In other cases, the content you are testing might be HTML-based and you can test by WebKit profiling. Again, part of your testing matrix is fulfilled, however you aren’t quite there.This may be enough to satisfy a short-term goal, but at some point you need to be testing on real mobile devices.

In order to truly automate on mobile, your mobile testing “Utility Belt” needs to be designed in such a way that allows for testing by object when possible, element when possible, and also be able to quickly fall back on text or image verification in order to satisfy all areas of your testing matrix and assure the highest quality of your mobile product. Having the flexibility to be able to choose how to get the testing done is paramount since as an Automation Engineer,you very rarely have a say in how a particular app or mobile web site is developed. The job requires you to sometimes understand functionality without necessarily being privy to the construction, and there is always a tight timeline to achieve results. The right tool for the job is a tool that takes all of this into consideration, and provides a platform to consolidate all of these different types of testing approaches.The first step is to determine the type of app you are testing. Is it fully native, fully web, or somewhere in between?

  1. If it’s fully native, you may be able to get to some objects on a per platform basis, but you will probably be falling back on text and image based verification, especially if you are trying to go cross-platform.
  2. If it’s fully web, a lot of testing can be done up front in a WebKit profiler. When it comes to real devices, element-based testing can be done cross-platform if you want to instrument, or you can fall back on text and image verification.
  3. If it’s somewhere in between, you’ll need to mix and match.
The second step is to find which pieces or steps of your test cases are reusable between each other, and can accept parameterization to fulfill the task. For instance, automating the selection of an item or link on your main screen of your app or landing page: Maximize reuse by engineering a parameter to accept different values, and reuse it across each test case. Although you may need to individually determine what type of verification you will use to achieve this on a per platform or device level, you will save time in the long run when you write additional test cases. The third step is to then group those pieces or steps together by device screens or pages. This way, as you write the test cases you have an organizational structure that is easy to identify by where you are within the app or site and where you need to navigate to next.Following these steps will provide a structure that can be grown to accommodate new features within an app or new sections within a mobile web site. As mobile devices become easier to automate against, this structure can easily adapt to emerging technologies that allow for greater reuse across platforms.

Note: This article was recently published in the April, 2013 issue of Automate Software Quality Magazine.





For Testers, Shipping Devices Is a Major Headache – Mobile Cloud Testing to the Rescue!

28 03 2013

deviceanywhere_mobile_cloud_testingTraditionally, to conduct real device mobile testing vendors shipped devices to testers all over the world. A major roadblock is the upkeep of these devices and maintaining them with the latest OS version or device model. Shipping back and forth becomes a huge pain point as time and cost rises exponentially. With cloud-based services there is no need to ship, purchase devices, or subscribe to service plans.

Cloud-based architecture enables easy access to application performance information anywhere in the world and eliminates the need to have a physical device on hand to test.

Some cloud-based tools allow mobile teams to receive real-time alerts against any measurement criteria, allowing them to address issues before they impact end-users. Real device testing, such as Keynote’s DeviceAnywhere platform through the cloud gives teams the ability to access real devices from anywhere in the world. This is perfect for organizations that either offshore their testing because of cost or have internationally-based teams. These teams can have access to the latest devices from locations with an Internet connection.

For a free THREE HOUR trial with access to the LARGEST library of real devices in the world – click here!





Zillow Leverages Existing Web Development Team to Develop for Mobile and Succeeds!

26 03 2013

zillow-mobileTime and training of your resources for mobile has become a tremendous problem for companies going mobile. While many testers are adept at QA, they may not be experts in mobile. Because of device fragmentation the demand for output becomes increasingly challenging for teams that do not have the bandwidth to increase their existing resources. Now, a person managing web development takes on multiple roles (to include mobile) and has less time.

With these hurdles, it creates even greater demand to have the proper testing tools and methods in place. While the number of device types and OS fragmentation makes it cumbersome and expensive for organizations to acquire and test mobile apps, there are efficient automated and real device testing tools that make this transition cost-efficient for testers and their teams. Organizations are beginning to adopt testing-as-a-service (TaaS) as a way to bolster the capabilities of QA and IT departments with testing. For these organizations, finding the right TaaS will help reduce cost and errors.

Recently, while at the Mobile Enterprise conference in San Francisco, I learned about how Zillow was able to train their existing web-based development team on how to code for the mobile environment, as opposed to hiring new mobile developers, thus reducing overlap and ramp-up time. Zillow was able to overcome the challenges and as a result, through precision-like ad placement, meet the needs of its consumers and connect them to the right person at the right time. This strategy has made it a win-win for both consumers and realtors alike.

Spencer Rascoff, chief executive officer of Zillow Inc., talks about the company’s mobile technology and real estate search services. He speaks with Cory Johnson on Bloomberg Television’s “Street Smart.” http://http://bloom.bg/NK4maL

Companies such as Zillow who have been able to monetize on mobile are seeing the value and importance of testing as part of their mobile development process.





With Adoption of iOS 6 Apple gets Fragmented!

21 03 2013

iOS-6-fragmentation-detailed-which-device-gets-whatWe recently wrote about how the launch of the iPhone 5 and the iPad Mini, each having a different form factor than their predecessors provides unique challenges for a QA tester and developers who are trying to keep up with the onslaught of new devices into the market. Android who has been the king of fragmentation due to it’s open OS and overwhelming OEM development has had to deal with this for some time. This is new for Apple while rarely changing it’s form factor since it’s launch of the iPhone in 2007.

But as the devices have changed, so have the operating systems supporting them. With the launch of the iPhone 5 and iPad Mini came a new OS – OS 6. This new OS provided new challenges as it could now be leveraged by carriers and partners to be customized to provide competitiveness and flexibility for their users.

In a recent article entitled “iOS 6 “fragmentation” detailed: which device gets what” the author writes  “‘Fragmentation’ is a word heavily loaded against Android and a term a bit too broad for our liking. It’s true that software updates on Android devices often arrive too late, but this depends on the carrier, it’s also true that some features are available on certain devices and not on others, and on and on, but using the general term fragmentation blurs those issues and writes all Android devices under a common negative denominator.

So far, Apple has been particularly proud of its “non-fragmented” operating system and lineup, but it seems that this concept broke with iOS 6. Hidden in the fine print of iOS 6 new features and updates are some staggering inconsistencies across devices – certain features are reserved for the newer devices and will never allow on older models. If that’s not a perfect example of something begging to be labelled with the same vague word “fragmentation” than we don’t know what is.
The folks from India’s iGyaan.in have penned this brilliant table showcasing just how fragmented Apple’s ecosystem really is after iOS 6, and it’s a joy to behold for the Android fans who finally get a big weapon in their anti-Apple arsenal. We bet you know this, but let’s keep this civil and think about what this means for the ecosystem that Apple is building. “
ios-fragmentation-tn




The Android OS – the Ultimate in Device Fragmentation?

14 03 2013

animoca-androidIn an article posted in 2012 in TechTarget, Kim-Mai Cutler had the chance to interview “Animoca, a Hong Kong mobile app developer that has seen more than 70 million downloads, says it does quality assurance testing with about 400 Android devices. Again, that’s testing with four hundred different phones and tablets for every app they ship!”

She continues .. “The photo above is just a sampling of Animoca’s fleet of Android test units. Yat Siu, who is CEO of Animoca’s parent company Outblaze, snapped and posted it from Outblaze’s headquarters today. In total, Siu says their studio has detected about 600 unique Android devices on their network.

“We haven’t managed to track down all of those devices because, in large part, they are no longer available for sale,” he says. Sad cakes!

On top of that, Siu said that the number of handsets from the lower-end Asian manufacturers is also growing rapidly. These are the phone makers that Nokia chief executive Stephen Elop was probably talking about in his famous “burning platform” memo when he said that are Chinese OEMs were “cranking out a device much faster than, as one Nokia employee said only partially in jest, ‘the time that it takes us to polish a PowerPoint presentation.’” If you take those out, the actual number of devices you need to test for is much lower.

But if you want to break into Asian markets, these phones matter and make it especially challenging for Android developers to ensure their apps work on every single Android device. Android fragmentation is a huge issue because developers have to check their work on dozens of devices. Animoca happens to be backed by Intel Capital and IDG-Accel, so it has the resources to buy all of these devices for testing and pay employees to use them.

But imagine the long-tail of developers! Imagine the people who make the roughly 500,000 apps in the Google Play store. Total nightmare.

It puts a real dent in Eric Schmidt’s prediction from six months ago that developers might start going Android first within six months. His deadline is up now and there aren’t signs of this happening. Appcelerator did a survey of 2,100 of its developer clients in March and found that, if anything, interest in Android development is stagnating.

Siu isn’t fazed though. He’s told me in the past that thorough QA testing makes Animoca’s apps retain users better because so many other Android developers do a bad job at it. Unlike iOS users who throw up their hands in frustration, write bad reviews and just leave, Android users tend to be delighted when they find apps that work even if they have a glitch or two.

He adds, “We like fragmentation as users prefer choice. We are not big believers that one size fits all.”

At Keynote, we see this all the time. Our customers tell us of the pain of having to manage libraries of real devices and the vast variety of OS versions even just within Android, let alone iOS, BlackBerry, or Win 8.  Developers who started with one or two devices a few years ago are now up to 20-30 today as seen in this image from another developer – Pocket Gems. They had two of the 10 top-grossing games on iOS last year, according to Apple’s iTunes Rewind.

android-table

And there is no end in sight now with the release of new form factors from Apple such as the iPhone 5 and the iPad Mini. One of our customers – Microsoft’s Windows Live team was able to replace their entire cabinet of devices with the DeviceAnywhere platform. No longer having to check out devices to team members or update devices with different OS’, or manage/maintain separate plans for each device locking them into a 2-year agreement with the carrier. Mark Boyce of Microsoft’s Windows Live deployment team said at the time of the release “Keynote’s DeviceAnywhere solution has saved our department a lot of time and headaches by eliminating most of the tedious, manual tasks associated with our in-house mobile testing process. We no longer have to chase down lost devices and SIMs, dig through boxes of chargers, or even track multiple phone bills and expense reports.”

To learn how you too can benefit from access to real devices from anywhere in the world, eliminating the need to maintain your own library or even having to ship devices around the globe, check out Keynote’s DeviceAnywhere platform and we’ll even get you started with THREE free hours, TODAY!

To read more of Ms. Cutler’s article click here.