Imagining a copilot like experience, 11 years before generative AI

Back in 2012, I had the honor of attending the "Microsoft Strategy Conference", an event for up-and-coming leaders to focus on strategic opportunities and interact with very senior executives (including then CEO Steve Ballmer). I was part of a team that was tasked with the future of Office -- how could we continue to grow market share when binary file format compatibility was no longer the compelling reason that consumers and businesses had to use Office?

Our team had some very creative ideas, which I turned into a narrative about a young student named Billy who was doing a research project for school. While some of the scenarios are a little antiquated, I was surprised and thrilled to see just how well some of these held up, over ten years later. The integration of Bing in Office to support co-creation and research stood out in particular - we envisioned a copilot like experience nearly 11 years ago.

And that's probably one of the reasons the vision didn't come to pass - while we tried to come up with scenarios that could be achieved with 2012 technology, many of these ideas were beyond bleeding edge, given the technology stack of the time. Still, the overall vision was compelling and some of the scenarios remain interesting (and unrealized!) even today. The use of speech as a primary input modality in particular remains underutilized, despite technical advancements.

An interesting footnote - the day after this conference concluded in 2012, Steve Ballmer was on a stage with Steven Sinofsky, launching the first Surface PCs. Which is probably one of the reasons Steve Ballmer wasn't happy with me when I challenged him about whether or not Microsoft should take the "risk" of building its own line of PCs. "Reasonable people can disagree, Keith" is what he said at the time. Credit to Steve - it took a couple of generations to get it right, but now the Surface line-up is one of the best and most innovative products in Microsoft's portfolio.

I present to you "Billy and Office... A fairytale".

###

Billy is a fourth grader who’s been tasked with creating a presentation about Mount St. Helens. He and three classmates need to collaborate on a report that’s due in one week. His classmates have access to different technology – two have modern Windows PCs, one uses a Mac, and still another has an Android slate device. All have access to the internet.

After meeting in class to discuss the project, Billy begins his research. He wakes up his slate and taps the tile for Microsoft Office. Billy likes Office because it’s flexible and easy to use. It enables him to exercise his creativity and organize his ideas while being immensely productive. The app seems to anticipate his needs and provides useful contextual data when he needs it.

Billy swipes from the bottom of the screen to get started. He speaks to his computer: “Tell me about the eruption of Mount St. Helens”. Lots of resources are returned—images, videos, encyclopedic articles, research studies and more. Billy touches one of the articles, from Wikipedia, and the entry is brought to the foreground. “Read it” he says to the computer, and the PC dictates the content to him. He tells the PC to “Stop” when he hears interesting facts, and he uses his finger to circle those sections and drag them to his workspace – his “canvas”. When the article is complete, he tells his PC to close it, but also to add a reference.

He surveys the images and drags several interesting ones to his workspace. He finds a video of the eruption dating to 1980. He grabs it with his finger and flings it to the authoring canvas as well.

Billy uses a gesture to toss away the research windows, returning him to the canvas. He speaks to the computer again. “How many people died during the eruption?” – his PC provides the answer. He flings the results to the idea wall, his holding pen for information or ideas. He asks more questions: “Did Seattle get covered in ash?”, “How tall was the mountain before the eruption?”, “How far away did they find volcanic ash”. Each response is saved to his wall.

Billy works on the presentation for a few hours before bedtime. He’s getting sleepy, and types that Mount St Helens erupted in 1979 – the text is underlined and turns red. When Billy hovers on the sentence, it indicates that the mountain erupted in 1980 – Billy accepts the change and moves on to the next section of the report. He saves a copy with his schoolmates to his SkyDrive (historical footnote -- this was accurate at the time), and each of them is notified via Facebook that his draft is checked in.

The kids iterate on the report for a few days. They’re satisfied they have their facts straight, but the presentation still isn’t very engrossing. They decide they need some more graphics and animations to really make it come to life. Billy touches the tile for Office, and then says “Integrate art”, which enables him to quickly add graphics and animations. He uses his finger to draw a rudimentary volcano, and the PC quickly asks “Do you want to include an image or animation of a volcano?”. Billy says “Animation”, and his rudimentary finger painting is replaced by a detailed animation of a volcano, that shows how pressure builds in a tectonic plate and is eventually released by an eruption. Billy drags it to the right place, and then pulls it out of the narrative flow, treating it as a sidebar. He writes a short caption for the figure.

His teammate Jane sees that there’s one new comment on his paper. The team chose to share their paper with other students and teachers who are researching similar topics. The comment is from a teacher in Texas. “Your presentation is looking pretty good, but you forgot to include details about the effects on local plants and wildlife. Are they back yet? How long did it take from them to return?”. She accesses the project canvas via her Android device, and it’s already suggested some additional age appropriate reading on flora and fauna at Mount St. Helens. She reviews the articles, and uses what she learns to finish up.

Jane saves the data one last time. The default format is determined by the consumption device, so each contributor has a worthy viewing experience. Once all four friends sign-off on that version, it’s automatically routed to their teacher for grading and evaluation. Billy then takes his slate to school, connects it to a wireless projector in the classroom, and he and his teammates share their presentation with the class.