Last year, Microsoft gave a peek inside its HoloLens goggles, which overlay images on the real world. Inside, custom silicon that Microsoft built from scratch keeps the mixed reality intact, mapping the user’s surroundings and fixing virtual objects in place.
Now, Microsoft is mulling what it can extract about the physical world. On Monday, the company said that it is adding silicon to the second generation of the holographic processing unit – also called the HPU – for running machine learning tasks, translating signs in a foreign language or telling users where to buy products seen through the goggles.
The co-processor is still under development but Microsoft said that it will run deep neural networks, which are at the heart of image recognition and other artificial intelligence feats. It will work without sapping the battery too quickly and without having to exchange data with powerful computers in the cloud, which can create delays.
These deep learning algorithms “have two well-known challenges: they require large amounts of labelled data for training, and they require a type of compute that is not amenable to current general purpose processor/memory architectures,” Marc Pollefeys, director of science for HoloLens, wrote in a blog post.
Microsoft is taking the latest stab at creating custom processors that can analyze large amounts of data from inside drones, smartphones, and security cameras. Apple, for instance, has also been said to be plotting a new smartphone processor that will allow Siri to answer questions and other apps to work without an umbilical cord to the cloud.
Last week, Intel started selling what it calls a neural compute stick, equipped with Movidius chips that it acquired last year for doing computer vision and other tasks locally. Vivienne Sze, an associate professor of electrical engineering at MIT, and her colleagues tailored chips for stabilizing the flight of thimble-size drones and training convolutional neural networks inside phones.
The holographic chip is not Microsoft’s only foray into silicon. Last year, it plugged customized FPGAs into every one of its cloud servers, using them to accelerate image classification and Bing’s search ranking algorithm. It was a counterweight to Google, which had also taken to using homegrown silicon to replace chips from Intel and Nvidia.
Microsoft’s first holographic processing unit is designed to repackage information streaming into the goggles through multiple sensors, so that it can be quickly and easily digested by the main 14-nanometer chip from Intel. It fuses information from head-tracking cameras, depth sensors, an inertial measurement unit, and infrared camera.
Microsoft’s first generation silicon, rendered in 28 nanometers, runs on a maximum of 10 watts. It contains an array of 24 digital signal processor cores and can perform a trillion calculations per second using 65 million logic gates, said Nick Baker, who helped build it, at the Hot Chips microprocessors conference last year.
Other than its existence, Microsoft has said little about the second generation chip yet.