Essential Steps For Testing Technologies: Dos and Don’ts

Last Updated April 3, 2017

Summa Linguae Technologies is a language and technology company – we do localization, data collection, and testing for all kinds of emerging tech. This includes anything from mobile apps, websites, video games, in-car speech systems, VR headsets, fitness wearables and more. As such, we have quite a few years of experience working behind-the-scenes to get these products ready for people.

Naturally, we’ve also gained a lot of insight into tips and tricks of the trade; lessons on best practices and advice when testing technologies.

If you’re a company in the business of creating cool emerging tech devices, these insights may help you decide when and why to implement different kinds of testing as a vital part of your development life cycle.

Test Early, Fail Fast

The faster you fail, the less resources are wasted on dead-ends and fixing deeply integrated mistakes down the road (user flows, architecture and other design choices). It’s about being efficient with the resources you have. Failing fast may not be your typical pearl of wisdom in every industry. But, in the world of tech, entrepreneurs, product designers, UX developers and so on, it has become commonplace. If you start testing and evaluating early in the development process of a device, less investment goes into fixing things that shouldn’t be in the product anyways. Part of this is to conduct end-user trials as soon as possible. In other words, get your users and their insights involved early on; they’re the ones who will help discover issues with the intuitiveness of your product, problems with the user experience design, and bugs when using the product in a real-world environment.

Ripping Off A Band-aid

We recently worked on doing speech data collection, testing, and localization for a voice-activated fitness wearable. The wearable has a variety of features such as measuring heart rate, cadence, distance covered, and more. It also has the ability to understand its wearer’s natural language voice-commands in multiple languages and accents from around the world.

We recently tested a virtual assistant. Like any good assistant, it supported a variety of features; from posting to social media, to creating and managing calendar entries. During our trials, we found that while users utilized voice operations to add calendar entries, no one attempted to edit a calendar entry by voice. Months of development hours could have been better utilized by polishing other features, instead of tuning this one.

Of course, this isn’t the ideal situation – nobody likes to see something they spent time and effort on being sent to the scrap pile. But, this is a valuable example of how ‘failing fast’ would have been beneficial. Implement the necessary changes quickly and efficiently; just like ripping off a band-aid.

Communication is Key

Good communication between the testing team and product developers is hugely important. Generating reports and collecting data for the sake of having heaps of data on-hand is not the most efficient or productive use of time. Rather, ensure you’ve set up good lines of communication and reporting structures that you can actually link dots between. It allows for both parties to maintain a constant discussion of where to focus efforts, which data will be most actionable, and what type(s) of testing are needed to improve the product.

One Type of Testing Isn’t Enough

You can conduct as much lab or requirements testing as you like; it’s great insofar as measuring the success of the baseline functionality of a product. But, in the end, there are things that’ll come up in a real-world environment that you might not even think to look for otherwise.

Field Testing

Our rule-of-thumb is to always test the product the way your target end-user would use it. That means testing fitness wearables in the field with hardcore runners and cyclists; rain or shine, hot or cold. Field testing involves testing the product in a real-world environment, but in a more directed way. This allows us to test specific features of the device, and for us to get the most out of our time.

End User Testing

Though field testing is essential in determining whether the product works as it should in the environment that it is meant for, it still encompasses some level of control that is not reminiscent of a real-world use case. As such, End User Testing is necessary to truly see how the device handles being thrown around; undirected and naturalistic. End User Testing means that our testers literally take the device home with them for an allocated amount of time; this can range anywhere from one week to three months. It’s reminiscent to what is usually considered ‘beta testing’.

Will your device survive if a toddler with messy hands got hold of it? Does your device have an easily replaceable charger so that, in the case of the classic my-dog-chewed-my-charging-cable scenario, your end user will be able to use something they already have? What features will users actually use and where will they be when they use it?

Out-of-Box-Experience Testing

Finally, be sure to test the Out-of-Box-Experience (OOBE) of your device; that first-impression a user will have. Things that may seem intuitive to a person who has worked on said device for months and months, might be overwhelming or confusing to a new-user. OOBE testing is necessary to discover the intuitiveness of device set-up; because who even really reads the manual, anyways? It also helps determine whether there are things they do or don’t like about the design.

Know Your Target Market

Many companies face the difficult task of really honing in on who their target market is. By extension, many companies face the difficult task of designing their product or service specifically for that target market, rather than trying to meet in the middle of two or three. For example, it’s incredibly difficult to create a wearable that will appeal to both the elite athlete as well as the average person. But by attempting to do so, the product might not end up meeting either market’s needs.

So, pick your target market – be specific about it – and clearly communicate it with all the people involved in your product development. Everyone needs to be on the same page.

No Such Thing As An ‘Average User’

Knowing your target market is a piece of advice that has been repeated time and time again. Don’t get me wrong, it is vitally important to the success of your product. But here’s a spin to the classic narrative: don’t go designing your product for the ‘average user’ within that market. There is no such thing. In other words, if you’re designing features with the mindset of, “let’s cater our product or service to the average user”, you might end up catering to nobody at all.

The Origin

There’s a great story about the origin of ‘the average’. Originally, averages were a way to accommodate for the imprecise tools of astronomers in the 1930s. Eventually, averages made their way into our everyday lives. In the military for instance; uniforms, food rations, weapons, and even airplane cockpits were designed with the image of the ‘average’ soldier in mind. This worked, more or less, up until World War II, when the Army began recruiting hundreds of new pilots to expand its air forces.

But the expansion of the Air Force brought a decline in performance and a shocking increase in pilot deaths. After years of blaming the pilots and their training programs, the military finally realized that the cockpit itself was to blame; the standardized sizing meant that, in life-or-death situations, the pilots could not reach the controls they needed to in time. The Air Force quickly responded by implementing new adjustable equipment; adjustable foot pedals, helmet straps, flight suits, and seats. Pilot performance soared, and death rates decreased significantly.

In things like education, we still design for this mythical average person. This is under the assumption that if you design something that would fit the ‘average person’, it would fit most people. Thinking of equal fit as the foundation for designing products and experiences has consequences on their success rate. Thus, the concept that ‘fit’ makes opportunity is important when designing environments, equipment, and even entire systems to accommodate more and more people.

Ultimately, your product will enjoy greater success if you are designing it for a certain, targeted group of people. Build your product features around them and their needs. Then, with testing, you can determine which features require more or less customizability.

Do Your Research

It’s always helpful to know where you want to release your product before you start the data collection and development process. This particular voice-activated fitness wearable was designed with English in mind. The developers knew they were going to localize it, but didn’t start doing localization testing until the product was at a point where it was difficult to go back to the drawing board to conduct market research.

Do native American-English speakers have the same voice preference as native Spanish, French, or German speakers? What kind of evidence do you have that proves English-speakers enjoy engaging in casual niceties with their device? 

Importantly, don’t assume that you know what your target market wants in a product. Even if you fit within that target market yourself, keep in mind that personal experience is biased and, well, personal. A big part of making data driven decisions is to minimize the effects of making decisions based on personal preference.

Take The Time For Feedback

An extension of research is feedback. Feedback is vital to understand the way your product is being perceived by your end-users; what needs to be worked on, removed, or added into the product. So, make sure you have a solid system in place to extract that feedback from your testers.

Before we joined the voice-activated fitness wearable project, they relied wholly on app analytics and binary “Yes” or “No” survey answers for feedback. This is not to say that quantitative data isn’t important – it is. It’s easy to scale, easy to measure, and easy to collect. However, choosing to simply ignore the qualitative data side of things should be highly discouraged.

Qualitative data is extremely important; it helps you make sense of the hard numbers, and reveals experiences that otherwise may only be discovered after your product has already hit the market.

So, talk to your testers. More often than not, they’re excited to do so and will have a lot to say; they’re intimately involved with shaping the UX/UI of the product, after all. Sometimes you won’t find anything that didn’t reveal itself in the quantitative data; sometimes it’s not until the very end of the 45-minute interview that you discover something of importance. But, every time something does come up, it makes the digging worth it.  

One-Off Bugs

Unfortunately, there isn’t a magic, definitive best practice for one-off bugs. There will be times where you just can’t reproduce a bug that one of your field or end user testers experienced. These are the cases that may not come up in lab or sanity testing, but shouldn’t be disregarded as unimportant. On the one hand, these have the potential to be the most important; they have the potential to come up again once hundreds, thousands, or millions of consumers are using your product in the real-world.

However on the other hand, there are plenty of design philosophies that discourage designing and devoting resources to edge cases. Though investigation into bugs is important, it is equally as important to know when to cut your losses. If you’re trying to recreate a specific scenario, it may be very costly to wait for a user to naturally encounter that scenario in the real world.

In other words, use your teammates, your smarts, and your resources well. Know when it makes sense to keep pushing forward, and when to pull the plug on searching for that wily ‘ol bug.

Related Posts

Summa Linguae uses cookies to allow us to better understand how the site is used. By continuing to use this site, you consent to this policy.

Learn More