Is Apple working on an augmented reality device? Following the introduction of ARKit in iOS 11, we look at all the rumours that Apple is working on AR glasses.
The rumour that Apple is developing an AR headset aren’t new; rumours of such a device have been circulating for well over two years. But things heated up in June 2017, when the announcement of ARKit at WWDC 2017 made it clear that Apple is deeply interested in this area. And no wonder: with Google and Microsoft developing AR devices, it seems like a party Apple can’t afford to miss.
It also hasn’t escaped our notice that Apple’s latest mobiles – the iPhone 8, 8 Plus, and especially the iPhone X – are well equipped for AR, thanks to the A11 bionic chip, which enables AR processing. And, in the case of the iPhone X, the TrueDepth camera, which can detect users’ facial expressions and gather 3D data.
Apple is gearing up to be a big player in the AR field, but what exactly does it have up its sleeve? Is Apple making a pair of AR glasses? (Analysts believe the firm is developing microLED panels for this purpose.) And what would such a pair of glasses look like? In this feature, we look at all the latest Apple AR rumours, patents and possible release date rumours.
What is augmented reality?
First, a quick refresher on terms, as many get AR and VR confused. VR headsets are mounted on the head in a similar way to Ski goggles, and completely block your view of the outside world. The VR headset tracks your head movement, and the 3D image displayed inside the headset moves accordingly. This makes it appear as if you’re wholly inside a 3D ‘virtual’ world.
Augmented reality, on the other hand, is where the glasses are see-through and you can still see the world around you, but an image is displayed in front of your eyes.
AR also hit the headlines in recent years, first thanks to Google Glass (which displays 2D images) and more recently with an upcoming headset developed by Microsoft called Hololens that embeds 3D images in the world around you.
Apple is known to be a trailblazer, but it’s still part of the Silicon Valley tech industry (albeit a more secretive one). If Google and Microsoft are all working on AR solutions, you can bet your bottom dollar that Apple also has a prototype in its labs.
Microsoft’s VR glasses – the HoloLens – are already available for developers (at the astronomical price of $5,000), but they probably won’t get a public release for the foreseeable future. As for Google Glass, development has been halted. Assuming that Apple is working on an AR headset – we’ll look at the evidence that Apple is working on an AR headset below – when is such a device likely to launch?
We don’t think Apple will release the glasses until late 2018, possibly even 2019. This way developers will have enough time to experiment with ARKit in iOS 11 and develop useful AR apps. Apple will need a lot of content to draw consumers in; content that could be based on iOS apps that run ARKit. If it’s a choice between buying AR glasses with no available apps right now compared to AR glasses with a library of apps in a year’s time, we know which we’d go for.
A (paywalled) story from the Financial Times in March 2017 claimed that even though Apple was “stepping up” its augmented reality efforts and that it’s the most important development project within the company, the launch was “still at least a year away, perhaps even longer”. As of April 2018 there have been no announcements, so we’re definitely looking at the “even longer” part of the equation.
DigiTimes has claimed that Apple is working on microLED screens in small and large sizes, and that the small ones will be used in AR glasses as well as future Apple Watch updates. Analyst Luke Lin says the small panels may enter mass production in late 2018 or 2019.
Another report (from June 2017) based on a Foxconn insider, suggests that the Apple AR glasses project might have been pushed back until 2019, or scrapped altogether. The insider claimed that there’s a 65 percent chance that the project has been shelved.
Why Apple’s AR device is here already
We don’t have to wait for Apple to launch an AR headset. The company already sells an AR device – the iPhone.
For most people their first taste of AR came when they started playing the popular Pokemon Go app on their iPhone in the summer of 2016.
Now, thanks to the TrueDepth camera on the iPhone X, and the A11 bionic chip, in the iPhone X and the iPhone 8 models, and iOS 11 – which Apple describes as “the world’s largest augmented reality platform,” the iPhone is taking augmented reality to the next level.
The iPhone’s AR credentials weren’t a surprise, a Bloomberg report back in March 2017 claimed that AR would be introduced on the iPhone first, with glasses coming later down the line.
That report claimed that there are “hundreds of engineers devoted to the cause” including a number of people from Apple’s iPhone camera team working on AR-related features for the iPhone.
The Bloomberg report was supported by a claim from a Business Insider source (in November 2016) that Apple has teams from several acquired startups working on iPhone AR, and this reflects Apple’s short-term desire to put AR into the hands of consumers.
Integrating AR into a smartphone that millions of people already have will introduce the concept of AR, and may make them more likely to buy Apple-made glasses once they go on sale.
Evidence of Apple’s AR glasses
The November 2016 Business Insider source also claimed that Apple has hired an expert in head-mounted displays (HMDs), and that this showcases Apple’s long-term AR/glasses initiative. Apple adding AR capabilities to the iPhone allows consumers to point the phone at a real-world object, like a statue or a car, and be offered contextual information overlaid onto the real world.
Also from November 2016, a Bloomberg report claimed that Apple was working on a Google Glass-style pair of augmented reality glasses (not a VR headset as had previously been believed). According to the news outlet, Apple had even started discussions regarding the project with its hardware suppliers, and even order a small number of “near-eye displays” for internal testing displays.
Bloomberg’s sources claim that the glasses will wirelessly connect to an iPhone and will display not only information but images and other data directly in front of the wearer’s field of view.
Citing people with knowledge of the project, the in a March 2017 report, the FT claimed that Apple began assembling a team of augmented reality experts a year previously. Apparently, more resources have been devoted to the AR project than Apple’s rumoured driverless car.
Then in April 2017, more evidence of some sort of AR project came from leaked Apple employee injury reports obtained by Gizmodo. While they’re mostly about mundane workplace injuries like cafeteria burns, there were two that stand out.
One involved someone testing an unknown prototype device who “experienced discomfort in her eye and said she was able to see the laser flash at several points during the study.” She was later referred to an optometrist.
Another involved an employee who “reported eye pain after working with new prototype, thought it may be associated with use”.
While these could reflect either an AR or VR device, an internal source told Gizmodo that the injuries were likely linked to an AR prototype Apple is working on, speculating that it could be “something like glasses with an overhead display”.
Catcher Technology, a laptop chassis manufacturer and Apple supplier, made an interesting announcement during its financial results in Augiust 2017.
The company revealed that it’s set to produce a product line for a company that falls outside the company’s existing product categories. Of course there was no mention of Apple or an AR headset, but it’s not a huge leap to make when you consider that Catcher Technology is an Apple supplier and rumours suggest that the new product has links to Cupertino.
In a January 2018 report, Bloomberg presented ‘evidence’ that Apple has plans in the AR space, claiming that, despite not exhibiting at the show, Apple was present at CES, with execs meeting suppliers to discuss AR glasses, according to people familiar with the meetings.
All this suggests that Apple is gearing up for its next big product in the AR industry: Apple glasses.
Of course, the glasses may not be made available for a while yet, if ever, as Apple is notoriously secretive about its product development and the project could be canned at any moment.
Tim Cook: AR ‘a big idea like the smartphone’
There is yet more evidence that Apple is working on something in the AR space – this time coming from Apple itself.
Apple CEO Tim Cook said in an August 2016 interview with The Washington Post that Apple is “doing a lot of things” in the augmented reality space. He didn’t stop there though, as the Apple boss went on to claim that he thinks of it as a “core technology” for the company.
“I think AR is extremely interesting and sort of a core technology” Cook stated during the interview. “So yes, it’s something we’re doing a lot of things on behind that curtain we talked about.”
Following on from these comments, Cook took part in a rare interview session at Sen. Orrin Hatch’s Utah Tech Tour in Salt Lake City in October 2016 where he again discussed augmented reality, although this time he gave us a better idea on what Apple may be working on.
“AR I think is going to become really big,” said Cook. “VR, I think, is not gonna be that big, compared to AR … How long will it take? AR gonna take a little while, because there’s some really hard technology challenges there. But it will happen. It will happen in a big way. And we will wonder, when it does [happen], how we lived without it. Kind of how we wonder how we lived without our [smartphones] today.”
It’s interesting that Apple is focusing more on augmented reality than virtual reality, although it does make sense – especially with the booming popularity of apps that feature the technology like Pokemon Go, and the introduction of ARKit in iOS 11 which makes it much easier for developers to offer AR capabilities in their apps.
Cook sees AR as becoming something that a huge part of the population will use on a daily basis, going on to state that he thinks “that a significant part of the population, of developed companies, and eventually all countries, will have AR experiences every day”.
Speaking to The Independent during a trip to the UK in February 2017, Cook said that he prefers AR over VR because it “allows individuals to be present in the world but hopefully allows an improvement on what’s happening presently.”
Considering that VR seems to provide the more immersive and enjoyable experience of the two platforms at the moment, why does Cook believe that AR is the future? People don’t want to be cut off from the world, apparently.
“Most people don’t want to lock themselves out from the world for a long period of time and today you can’t do that because you get sick from it,” he said, discussing virtual reality. “With AR you can, not be engrossed in something, but have it be a part of your world, of your conversation. That has resonance.”
Cook even went as far as to liken it to the smartphone, and how that has changed the world. “I regard it as a big idea like the smartphone” he claimed. “The smartphone is for everyone, we don’t have to think the iPhone is about a certain demographic, or country or vertical market: it’s for everyone. I think AR is that big, it’s huge.”
Most recently, Tim Cook addressed AR and ARKit during the Q3 2017 earnings call. When asked a question about how people will be using ARKit, Cook replied “I could not be more excited about AR and what we’re seeing with ARKit in the early going.” He continued to explain the variety of ways he has already seen AR used, and that he thinks “AR is big and profound and this is one of those huge things that we’ll look back at and marvel on the start of it.”
Apple announces AR integration at WWDC 2017
Yet more evidence that Apple is interested in VR: Apple announced ARKit as part of iOS 11 at WWDC 2017 in June. ARKit allows developers to offer high-end augmented reality capabilities in their apps without having to develop the AR system, and improves existing AR solutions too – for example, Pokemon GO running ARKit looks much smoother than it did previously.
Per Apple’s ARKit page on its website, it claims that ARKit is a “cutting-edge platform for developing augmented reality apps for iPhone and iPad” and it offers “powerful capabilities for positional tracking and scene understanding”.
It makes sense for Apple to offer AR as part of the iPhone and iPad experience, as it’s a device already in the hands of millions of consumers. It also encourages developers to create AR apps that could in future be ported to an AR headset, if that’s the route that Apple decides to go down.
Plus, it’ll make iOS the biggest AR platform in the world on day one. Not too bad for emerging technology!
Analyst claims AR is a perfect partnership for Apple
KGI analyst and all-round Apple psychic Ming-Chi Kuo has told investors that he believes that Apple’s track record of delivering innovative and high-quality user experiences will help the company move into – and succeed – in the AR market. Citing how the iPod paved the way for the iPhone, Kuo suggests that the iPhone could provide the building blocks for a full-blown AR solution – although he didn’t provide details on what it might look like.
Generally, Kuo believes that Apple could integrate AR to redefine key product lines; an AR-powered interface could change the way that users interact with small-screen devices like the Apple Watch, eliminating issues with clunky controls.
A iPhone Hacks report claimed that Apple’s smart glasses have a unique design, according to their source.
Their insider provided details on the design – featuring a bone conduction earpiece, a microphone, an accelerometer and a capacitive frame for navigation and interaction, the glasses sound impressive.
The smart glass itself was said to be manufactured using a Kopin NED Acetate frame, and there’s even claims that Apple was looking to sell them with prescription lenses or even polarised lenses in partnership with Zeiss.
Some creative people have already come up with potential designs for Apple’s glasses. For example, this YouTube video (which enthusiastically claimed that the specs could launch in 2018) has some neat designs.
In January 2017, Apple was granted a pair of patents that could give us a glimpse at the company’s AR aspirations. Patent no. 9,560,273 details the hardware framework of an AR device that can understand its environment thanks to enhanced computer vision capabilities, while patent no. 9,558,581 details the method of overlaying virtual information on a physical environment.
Both patents were filed back in 2015 by German AR specialist Metaio shortly before Apple acquired the firm, and transferred to Apple in November 2016.
Patent 273 describes a “wearable information system having at least one camera” but goes further and discusses the possibilities of having more cameras, a screen, a UI and even internal components dedication to computer vision. The patent details a head-mounted display as the primary platform for AR, but it also suggests that a smartphone (aka the iPhone) could serve as a decent stand-in.
However, on the whole, the patents deal more with object recognition than it does the visualisation of the AR data. The issue of object recognition is a barrier in the AR industry on the whole, as existing solutions require a large amount of power and thus, aren’t ideal for real-world use.
In Apple’s case, the ‘invention’ maintains a low-power scanning mode for the majority of its operation, with high-power modes triggered in relatively short bursts.
Then, in May 2017, a patent surfaced depicting the head-mounted display of Apple’s upcoming augmented reality headset. The patent details a device and techniques for projecting a source image in a HMD comprised of left and right displays, suggesting that they may look more like standard glasses than Google’s failed Glass.
In the filing, Apple notes that some HMDs can be used to view a transparent image imposed upon a real world view. Sound familiar to augmented reality? We think so too.
Another Apple patent filing, discovered in July 2017, describes how a pair of AR-enabled glasses could interact with a smartphone. The user is said to be able to interact with the objects that appear on the glasses using their smartphone. Of course, the phone would need to be equipped with the necessary hardware and software to be compatible with the glasses, and we’re unsure if the tech is already featured on the iPhone 7 or not.
Another concept in the patent describes how the user can interact with objects that appear on the AR glasses by pointing at them – a similar mechanic is used by Microsoft in its HoloLens glasses, so it’s not a far-fetched idea. The patent claims that it’ll be possible via an array of cameras attached to the glasses, as the cameras will map and track the user’s finger as it moves towards the on-screen object.
With more advanced patents appearing, it suggests that Apple is taking augmented reality very seriously, and we might be seeing Apple’s AR-enabled device on the market sooner than originally thought.