Some of the criticism of Apple’s Siri may be undeserved. Loup Ventures — a research-driven venture capital firm — recently asked Siri, Google Assistant, Alexa, and Cortana 800 questions each. This time, Google Assistant was able to answer 86% of them correctly vs. Siri at 79%, Alexa at 61%, and Cortana at 52%.
The digital assistants were graded on two metrics: 1. Did it understand what was being asked? 2. Did it deliver a correct response? The question set, designed to comprehensively test a digital assistant’s ability and utility, is broken into five categories:
- Local – Where is the nearest coffee shop?
- Commerce – Can you order me more paper towels?
- Navigation – How do I get to uptown on the bus?
- Information – Who do the Twins play tonight?
- Command – Remind me to call Steve at 2pm today.
Testing was conducted using Siri on iOS 11.4, Google Assistant on Pixel XL, Alexa via the iOS app, and Cortana via the iOS app. Google Assistant continued its outperformance, answering 86% correctly and understanding all 800 questions. Siri was close behind, correctly answering 79% and only misunderstanding 11 questions. Alexa correctly answered 61% and misunderstood 13. Cortana was the laggard, correctly answering just 52% and misunderstanding 19. (Note that nearly every misunderstood question involved a proper noun, often the name of a local town or restaurant.)
“Google Assistant has the edge in every category except Command. Siri’s lead over the Assistant in this category is odd, given they are both baked into the OS of the phone rather than living on a 3rd party app (as Cortana and Alexa do),” writes Loup Ventures. “We found Siri to be slightly more helpful and versatile (responding to more flexible language) in controlling your phone, smart home, music, etc. Our question set also includes a fair amount of music-related queries (the most common action for smart speakers). Apple, true to its roots, has ensured that Siri is capable with music on both mobile devices and smart speakers.”
Also, Siri and Google Assistant, with their deeply integrated maps capabilities, scored “meaningfully higher” in the Navigation and Local categories.
