Microsoft’s Andrew Shuman on the Cortana app’s death, natural language, and Alexa

Last month, news broke that Microsoft was killing off Cortana for Android and iOS and was removing Cortana from its Launcher app for Android. On January 31, 2020, Microsoft will end support for the Cortana app in Australia, Canada, China, Germany, India, Mexico, Spain, and the U.K. This was Microsoft’s statement at the time:

Cortana is an integral part of our broader vision to bring the power of conversational computing and productivity to all our platforms and devices. To make Cortana as helpful as possible, we’re integrating Cortana deeper into your Microsoft 365 productivity apps, and part of this evolution involves ending support for the Cortana mobile app on Android and iOS.

Microsoft refocusing Cortana for the enterprise, and specifically for Windows and Office, is not new. Nonetheless, killing off the Cortana app was a bold move. We sat down with Andrew Shuman, who has been leading the Cortana team since Javier Soltero’s departure last year, to find out why Cortana for Android and iOS is being killed off, what to expect with Cortana for Windows, his thoughts on natural language and typing, what’s going on with the Alexa integration, and more.

Cortana for Android and iOS in the U.S.

Shuman confirmed that the Cortana mobile app (and Cortana integration in Microsoft’s Android launcher) was going away in all countries except for the U.S. Previously, Microsoft listed only eight countries that would lose Cortana for Android and iOS. But other countries, such as Japan, will also be losing it. At this point, why not kill the Cortana app in the U.S. as well?

Shuman gave a few reasons. First, he said, the Cortana app is how you configure Cortana and update the firmware on Surface Headphones, which debuted only in the U.S. “We still have people who use their Cortana apps to manage their headphone operation,” said Shuman. “So if you have the Surface Headphones, you can use the Cortana app with those, and we’re still supporting that.”

That’s not a great reason. After all, Microsoft started selling Surface Headphones in Australia, Austria, Canada, France, Germany, Ireland, New Zealand, and Switzerland earlier this year. After January 30, only Surface Headphones users in the U.S. will be able to keep using Cortana for Android and iOS. All other Surface Headphones owners will have to use Cortana for Windows.

We did get some good news for these Surface Headphones owners. In the spring, Microsoft plans to launch the delayed Surface Earbuds. The accompanying Surface Audio app will not only be available for Android, iOS, and Windows, it will support existing Surface Headphones and the new Surface Earbuds for managing all settings and configuring Cortana.

Next, Shuman surprisingly mentioned the only Cortana smart speaker that saw the light of day — a device Microsoft executives typically avoid discussing. “Also, people who have Invokes in market can still use the Cortana app to configure their Invokes, so those markets are all still going,” said Shuman.

This explanation does have some legs — the Harmon Kardon Invoke was only ever sold in the U.S. But that’s because sales were so low — Microsoft was selling it at 50% off just a month after launch.

Shuman gave two more reasons for why the Cortana app is being killed in all countries except the U.S. First, Cortana for Android and iOS didn’t receive much traction outside the U.S. “The markets we turned off had much lower usage and engagement,” he said.

And finally, Cortana for Android and iOS is sticking around in the U.S. because Microsoft wants to keep experimenting. “We also think that there may be some roles for standalone assistants as an experimentation place for us to try out new ideas still. I think this point has been made a lot, but … the assistant landscape is rich with opportunity, and very not rich with actual results, sometimes. And so the opportunity to continue to try things quickly is important to us as well.”

Cortana for Windows, Office, and beyond

The latest Cortana for Windows improvements are slated for the next major Windows 10 update, likely to arrive in April or May of next year. If Cortana for Android and iOS didn’t do well in some markets outside the U.S., how is Cortana on Windows doing?

“Cortana on Windows does really well in those eight markets, and that’s part of the reason we had started to bring the mobile app there,” said Shuman. “Given our focus on the productivity users, we think that … it makes much more sense to be where our users are: inside of Office and inside of Windows.”

In other words, rival assistants have won on the phone. So Microsoft is focusing on the enterprise.

Where else can we expect Cortana to show up? Shuman noted that it’s coming to Microsoft Teams next year, but there’s no release date for that yet.

“Obviously, there’s strength in enterprises, but the kind of problems we have are very universal,” he continued. “It’s about helping people get time back. The previous apps that we were showing before were really great when we were in startup mode and trying to sort out what an assistant could be. But they were not directly aligning with those Microsoft 365 users who really don’t want to go to another experience to do their stuff. They want to be able to do it in the apps they’re using every day, like Outlook and like Teams. That’s really been our push now — over a year now of work that we’ve had underway.”

Windows and Office aside, you should expect Cortana in every Microsoft app Microsoft can think of.

“The analogy that I like to use is both Microsoft Search or Microsoft Account, which are really across all of our applications,” said Shuman. “We think of Cortana in a very similar way, where it really is a horizontal surface area that’s very person-centric. It very much knows a lot about me, and then it experiences itself across all of these apps and suites. So that’s the way that we characterize the investment moving forward. And partially, that’s just the reality of wanting to be in the experiences you’re using every day and not [having] to switch over to another one to get the experience.”

Natural language and typing

Currently, you can talk, but not type, to Cortana. Not everyone wants to talk to their phone or computer, so some assistants let you type instead. Shuman confirmed that’s in the works for Cortana, too.

“I’m really glad you’re asking that, because I think there is [an] equivalence that people do with voice and assistants, when I think it really should be broadened to be about natural language and assistants,” said Shuman. “There are two or three places where I think this is really powerful. In the Play My Emails experience, for example, we had to do a lot of work on natural language understanding just to help read out emails to you intelligently. You can imagine if you just flipped open an Outlook pane and started to read everything on the screen how awful that would be.”

“There are all these labels and all these things that your brain doesn’t read when you’re scanning it,” he continued. “You just scan through the subjects. And then you scan through who sent it. And we had to do a lot of natural language processing work so that we can read them intelligently and summarize the state of affairs.”

Unlike screen readers, which simply read out every single piece of text on the screen, Outlook’s voice mode and the Play My Emails feature can give you a summary of your inbox. It can say things like, “John just replied and added you to a conversation about this week’s budget.” Cortana in Outlook skips the header between email replies, as well as the signature file at the bottom, links, attachments, and so on.

Shuman confirmed we can expect to be able to type to Cortana next year.

“One of the things that we’re investing in Windows is this idea of being a quicker experience for people who are good typists. Those of us who’ve been around computers for a while don’t need to talk to them. But still, the power of natural language is really great. I’ll give you one example that I use now all the time on the builds I’m running, which is just managing my time in my calendar. It’s a lot easier to type in ‘dentist appointment next Friday’ than to alt-tab to Outlook, new file, tab, tab, tab, type in ‘dentist,’ type in ‘Friday,’ and find a free time. So that’s a great example, where just getting data into and out of my digital calendar is really simplified by natural language understanding.”

Alexa and privacy

We haven’t heard much about the Alexa-Cortana tie-up since the assistants crossed over to Echo speakers and Windows 10 PCs in August 2018, but Shuman talked about some of the companies’ future plans.

“We still have a great aspiration to do a lot more … kind of across the two companies,” he said. “It started really tops down. I don’t know if you remember that, but Satya [Nadella] and Jeff [Bezos] actually cooked this plan up. You can still invoke Cortana from Alexa, and vice versa. I think it will be a great area for us to lean into. We really believe in a multi-assistant world. Just like in the real world where I might have a doctor, and a lawyer, maybe a trainer, you’re going to have multiple assistants that are good at what they’re good at. We’re not going to become an ecommerce company anytime soon, but assistants are great at helping you buy things.”

(Alexa isn’t a great shopping assistant, but that’s another story.)

One would think that the Alexa-Cortana crossover could be in jeopardy, but Shuman said all is well. “We are still invested in it, for sure,” said Shuman. He was also positive when questioned whether anyone actually used it. “Yeah, we have a reasonable amount of usage. It’s still a very new experience, but it’s okay.”

Part of the problem is that it’s a pain to set up the two assistants. You need Alexa set up with an Amazon account, Cortana set up with a Microsoft account, and then you have to link them together. “We certainly would like to make it easier and easier to connect the two accounts, but we also want to be very clear to the user about when they’re using personalized data in an Amazon service or in a Microsoft service,” said Shuman.

Where your data goes

Logging in is necessary so each company can disclose what the assistant has collected from you. Conversely, there’s little reason you would want to launch the other assistant without being logged in. Any general question you may want to ask the second assistant, you can presumably just ask the first.

Shuman says Microsoft is becoming more conscious of users’ data “especially as we move out of the more pure consumer space into the enterprise and office productivity space.” (In August, Microsoft updated its privacy policy to say employees or contractors may listen to recordings from Cortana and Skype Translator, after a Vice report revealed the practice.)

“That definitely brings its own challenges and also its own opportunities,” said Shuman. “We do complete eyes-off AI models, which means that no one in my team or in any of the teams at Microsoft looks at your email. We do it through sophisticated sampling of public datasets and employee-donated data, and then we generalize it. Which is a huge amount of work and a huge technical breakthrough for us to really be truly eyes-off in how we build these models.”

“If you are using Office 365 services, we have a very strict compliant data storage model that ensures that no one can look at your data except you. As we want to bring that to other experiences and other platforms, we have to be very cognizant of that promise,” he added.

Good assistants have to know you

Microsoft is still in the process of pivoting Cortana to be a productivity assistant. This productivity focus is what killed the mobile app outside of the U.S., where it still has traction and can be used for experimentation. The productivity angle also explains work on natural language, typing, and knowing its users.

“For us to be a really good digital assistant, we have to know you and know you well,” Shuman said. “If you go after a user like an Office user, those are the people that we can help the most. We understand the projects they’re working on. We understand the people they communicate with the most.”

Shuman launched into an example: Say you tell Cortana to “call Jon.” There are many Jons at a company like Microsoft, but, he said, “Anyone who knows me knows there’s one Jon that I work with all the time — Jon Hamaker, he’s right down the hall. And so in my experience, I want to ‘call Jon’ to be Jon Hamaker.”

“That’s the kind of thing that’s truly important about an assistant. An assistant has to really know you. And that’s absolutely core to why we’re having this renewed focus on these users that we think we can offer the most value to.”