Developing an emerging tech device? These insights on testing technologies will help you decide when and why to implement different kinds of trials.
Testing technologies is a vital part of your development life cycle, but it takes a lot of resources. To do the testing, you need to use a combination of machine learning, human intelligence and practical simulation in an environment where the device is used.
You also need to set up and maintain test benchmarks to create and execute unit, performance, comparison, end-to-end, functional and stress tests. All components must be properly integrated and function together as expected.
Once the test cases are developed and executed, someone takes care of logging software bugs, providing root cause analysis of the identified defects and verifying bug fixes.
It’s a lot to handle. This is why outsourcing your product testing has become a popular option. Let someone else handle the testing so you focus on your core business.
We’ve gained a lot of insight into tips and tricks of the trade – lessons on best practices and advice when testing technologies that we’d like to share with you.
1. Test Early, Fail Fast
The faster you fail, the fewer resources you waste on dead ends. Fixing deeply integrated mistakes down the road is very costly.
Failing fast may not be typical wisdom in every industry. But in the world of tech entrepreneurs, product designers, UX developers, and so on, it’s common.
Conduct end-user trials as soon as possible. Get your users and their insights involved early on; they’re the ones who will help discover issues with your product, problems with the user experience and bugs when using the product in a real-world environment.
Ripping Off a Band-aid
We recently tested a virtual assistant that supports a variety of features, including the ability to create and manage calendar entries. During our trials, we found users used voice operations to add calendar entries, but no one tried to edit a calendar entry by voice.
Months of development were put into polishing other features, but this one was missed.
As a result, the company had to make big changes to their virtual assistant and had to scrap some work because of it.
This is a perfect example of how ‘failing fast’ would have been beneficial. Implement the necessary changes quickly and efficiently so the pain is short lived, and you can move on. Just like ripping off a band-aid.
2. Communication is Key
Meaningful interaction between the testing team and product developers is extremely important.
You can generate reports and collect data until the cows come home, but it means nothing unless you can apply it to your product. Strong communication will allow you to make meaningful connections between your reports, data, and your product.
With these connections, both parties can focus their efforts, decide which data is most actionable, and determine what type(s) of testing will to improve the product.
3. One Type of Testing Isn’t Enough
Conduct as much lab or requirement testing as you like; it’s effective in measuring the success of the baseline functionality of a product.
In the end, however, there are things that’ll come up in a real-world environment that you might not even think to look for otherwise. Expand your testing to the following areas:
Our rule-of-thumb is to always test the product the way your target user would apply it.
We recently worked on speech data collection, testing, and localization for a voice-activated fitness wearable. The wearable has a variety of features such as measuring heart rate, cadence, distance covered, and more. It also understands its wearer’s natural language voice-commands in multiple languages and accents from around the world.
Field testing involves testing the product in a real-world environment, but in a more directed way. For example, we needed hardcore runners and cyclists to get out rain or shine, hot or cold.
This allows us to test specific features of the device, and for us to get the most out of our time.
Field testing is essential in determining whether the product works as it should in the environment that it is intended. But it still encompasses some level of control, and it isn’t really a real-world use case. End-user testing is necessary to truly see how the device handles being thrown around.
- Will your device survive if a toddler with messy hands got hold of it?
- Does your device have an easily replaceable charger (think of the classic my-dog-chewed-my-charging-cable scenario) so your end user will be able to use something they already have?
- What features will users use and where will they be when they use it?
Our testers literally take the device home with them for a week to three months and use it everywhere to give it a real world ‘beta test.’
Out-of-Box Experience Testing
Finally, test the out-of-box-experience (OOBE) of your device – it’s first impression a user will have.
Things that may seem intuitive to a person who has worked on the device for months and months, but it may be overwhelming or confusing to a new user.
OOBE testing is necessary to discover the intuitiveness of device set-up because who even really reads the manual, anyways? It also helps determine whether there are things they do or don’t like about the design.
4. Know Your Target Market
Many companies face the difficult task of really homing in on their target market.
By extension, many companies face the difficult task of designing their product or service specifically for that target market, rather than trying to meet in the middle of two or three.
For example, it’s incredibly difficult to create a wearable that will appeal to both the elite athlete and the average person. And attempting to find that athlete/average balance, the product might not end up meeting either market’s needs.
So, pick your target market, be specific about it, and clearly communicate it with the people involved in your product development. Everyone needs to be on the same page.
5. No Such Thing as an ‘Average User’
Here’s a spin to the classic “know your target market” narrative: don’t go designing your product for the “average user” within that market. There is no such thing, and you might end up serving nobody at all.
The Origin of the Average
There’s a great story about the origin of ‘the average’.
Originally, averages were a way to accommodate the imprecise tools of astronomers in the 1930s. Eventually, averages made their way into our everyday lives.
In the military for instance, uniforms, food rations, weapons, and even airplane cockpits were designed with the image of the ‘average’ soldier in mind. This worked, more or less, until World War II, when the Army began recruiting hundreds of new pilots to expand its air forces.
The expansion of the Air Force brought a decline in performance and a shocking increase in pilot deaths. After years of blaming the pilots and their training programs, the military finally realized that the cockpit itself was to blame; the standardized sizing meant that, in life-or-death situations, the pilots could not reach the controls they needed to in time.
The Air Force quickly responded by implementing new adjustable equipment: foot pedals, helmet straps, flight suits, and seats. Pilot performance soared, and death rates decreased significantly.
Your product will enjoy greater success if you design it for a certain, targeted group of people. Build your product features around them and their needs. Then, with testing, you can determine which features require increased or decreased customizability.
6. Do Your Research
It’s always helpful to know where you want to release your product before you start the data collection and development process.
The aforementioned voice-activated fitness wearable was designed with English in mind. The developers knew they were going to localize it, but didn’t start localization testing until it was too late. The product was at a point where it was difficult to go back to the drawing board to conduct market research.
Do native American-English speakers have the same voice preference as native Spanish, French, or German speakers? What kind of evidence do you have that proves English speakers enjoy engaging in casual niceties with their device?
Don’t assume you know what your target market wants in a product. Even if you fit within that target market yourself, keep in mind that personal experience is biased and, well, personal.
A big part of making data driven decisions is to minimize the effects of making decisions based on personal preference.
7. Take Time for Feedback
Feedback is vital to understand the way your product is being perceived by your end-users: what needs to be worked on, removed, or added into the product. Make sure you have a solid system in place to extract that feedback from your testers.
Before we joined the voice-activated fitness wearable project, they relied wholly on app analytics and binary “Yes” or “No” survey answers for feedback. This is not to say that quantitative data isn’t important – it is. It’s easy to scale, easy to measure, and easy to collect. However, qualitative data is just as important.
Qualitative data makes sense of the hard numbers and reveals experiences that otherwise may only be discovered after your product has already hit the market.
So, talk to your testers. Often, they’re excited to do so and will have a lot to say; they’re intimately involved with shaping the UX/UI of the product, after all.
Sometimes you won’t find anything that didn’t reveal itself in the quantitative data; sometimes it’s not until the very end of the 45-minute interview that you discover something of importance. Every time something does come up, though, it makes digging worth it.
8. Swat One-Off Bugs
There will be times where you just can’t reproduce a bug that one of your field or end-user testers experienced. These are the cases that may not come up in lab or sanity testing but shouldn’t be disregarded as unimportant.
On the one hand, these have the potential to be the most important; they have the potential to come up again once hundreds, thousands, or millions of consumers are using your product in the real world. But on the other hand, there are plenty of design philosophies that discourage designing and devoting resources to edge cases.
Investigating bugs is important, and it’s equally as important to know when to cut your losses. If you’re trying to recreate a specific scenario, it may be very costly to wait for a user to naturally encounter that scenario in the real world.
In other words, use your teammates, your smarts, and your resources well. Know when it makes sense to keep pushing forward, and when to pull the plug on searching for that wily ‘ol bug.
Testing Technologies as a Service
We manage the complete product testing process so you can focus on your core business. We make sure that all components are properly integrated and function together as expected.
Contact us today to get the best testing for your product.
AI and Online Gaming Safety: Power-Up with a “Hybrid Approach” to Moderation
AI is helping build safer and more inclusive spaces in gaming. But human touchpoints still offer the best ...
What’s This? Introduction to Named Entity Recognition
Data collection + annotation tasks like Named Entity Recognition results in smart linguistic datasets with...