Last weekend we did a guest post for VentureBeat comparing Siri, Cortana and Google Now in five different languages – English, French, Italian, German and Mandarin.
We received a number of requests asking us about our methodology. So, we’re sharing our specific test cases and results here.
Our test cases consisted of eighteen questions that users would be likely to ask for from a personal voice assistant. All requests were executed through voice commands and were targeted to test the usefulness of the assistant (rather than accuracy in pure dictation or search).
Defaulting to search for a query which could have generated a direct response (i.e. “call the nearest Chinese restaurant”) was penalized.
Query Results by Language
Scroll through the slideshow below to see detailed results for each language.
We scored each question on a scale of 1.0, 0.5 and 0.0. Questions that scored a full pass (1.0) were successfully executed by the voice assistant returning a direct response.
In contrast, a 0.5 rating was given to questions that displayed the right answer in a web search query.
0.0 ratings were given to inaccurate answers or to questions which the voice assistant could not successfully execute.
See below for an accuracy comparison across all five languages.
If you’re interested in learning more about localizing speech technology, check out how we collect multilingual data and conduct speech recognition testing.
How to Get Ahead with Expert NLP Translation
Expert NLP translation services allow for major improvements in quality and therefore a wider reach.
Get to Know Our Specialized Linguistic Services
For over 15 years, Datamundi has been at the leading edge of creating and enriching language data for glob...