Can Apple Vision Pro give developers, and so consumers, a whole new canvas?

Read the full trascript below:

At Apple’s most exciting event in years, the WWDC, it released the hot new Vision Pro headset, as we had earlier predicted. What does this mean for augmented and virtual reality?

Vivek Goyal, co-founder of PlayShifu, one of India’s best-known AR and VR startups, talks about his surprise that Apple started at the top end, with its mixed reality headset, the $3,500 Vision Pro, that was announced yesterday. He walks us through how Apple’s approach stands out, starting with how VisionOS, the headset’s software allows Apple’s existing app ecosystem to show up on the new hardware. Vivek is also looking forward to the time when Apple will release tools and software kits for developers to exploit the potential for 3D rendering within Vision Pro

Vivek:

I was surprised given the fact that it's an opposite approach of what Apple has done historically, going with Pro first and probably is in testament to the humongous amount of challenge that is there to achieve amazing experience for a headset device both hardware and software that Apple has decided to go pro first. But things that I loved from what I saw is as expected from Apple Content is not something that is gonna lag or follow after the hardware launch. They're going to launch with the content ecosystem that we already know exists on Apple to go live right from day one. So it's usable and has some utility right from day one rather than just being a fancy gimmick. So that's one thing. Making scalability or connectivity between the iOS, macOS ecosystem with Vision Pro is something that I was expecting and was done beautifully well in the execution.

Other thing is that hardware in VR headsets, and this is across all brands that exist in the affordable range, the display has been the biggest kind of bottleneck in achieving an amazing XR experience and what from what we saw, given the price point, Apple has gone for an extremely high-end kind of display which will make it an extremely realistic experience when you go inside it

So from the looks of it, obviously you are overlaying information in your real world. So it is not virtual reality for sure. It is mixed reality or XR, but they've also stayed away from rendering anything 3D in the demo so far. It's all 2D UIs and 2D things which are being rendered. Nothing that was 3D-ish in nature was rendered. So it's not true XR mixed reality yet, but it's somewhere in between laying the grounds because the 3D content would definitely need custom development, which keeps the existing ecosystem away. So they've been careful to first onboard the existing ecosystem of content, apps and ecosystems, and then expand it to the 3D reality.

Pankti:

Obviously, Disney has partnered with them, right, from the get-go. So who do you think is the target audience at this price point? And for content creators like Disney, then what is the actual application of a headset like that? We’re assuming children are not going to use it in this form, right?

Vivek:

Right, so I think the early adopters are gonna be, you know, upper middle class and above—tech families who pretty much buy any Apple product that comes out. Even a smaller portion of that, given the price point is almost three times that of iPhone. But from what I see, the Disney partnership is primarily to get, let's say, content of the whole Marvel ecosystem on board, right? One of the things that they've pitched strongly is that the movie experience is going to be like a theatre experience. And I think if you add a lot of Disney content meant for, it's not just kids, a lot of, you know, Marvel content is applicable for all the tech plus movie buffs, which is enough for Apple to see an early usage in the range of millions of users to start with. And then as they launch a 2000 and a sub-2000 version later on, it becomes mass market then.

Hari:

So, when somebody uses this headset, are they able to see through... Or is it that the front-facing cameras are projecting whatever they're seeing onto the eyes of the user?

Vivek:

So the term that they called this is ‘eyesight’. And they were careful enough to word it saying that give an illusion of eyes. Or it feels like you're seeing the eyes of somebody who is wearing the headset, which clearly indicates that there is a display outside and it's not see-through.

See-through hardware, display tech is not there yet, [we are] not seeing any consumer ready examples of see-through displays. So this is most probably an example of using cameras but effective placement and high resolution cameras and a display to display a realistic feel of human eyes outside, rather than a see-through display.

Hari:

What are you seeing in terms of big differences between Vision Pro and other existing headsets from Meta and others?

Vivek:

So I think hardware is going to be top-notch, given there is no constraints on pricing. So I think overall hardware quality is top-notch and differentiated from the other headsets in the market. Software ecosystem, we talked about this when Oculus and the initial few headsets came out, it all required a dedicated community to build for it. It is required that other platforms like Unreal and Unity support development for Oculus. What Apple has done is make sure that any app ever made for Mac or iOS is going to be live and available on this. So that's a straight-away game changer because you start with your standard apps, whether you're using Netflix or Disney—it'll start showing up as they start partnering, to display the content in this hardware.

Pankti:

Can you explain to us what you think the differences are between spatial computing and the metaverse as we know them?

Vivek:

Spatial computing is the fundamental layer of the Metaverse, I would say. Like you need to understand the space in which the headset is, to be able to use it effectively for any kind of content placement. Like for example, understanding the so far what we've seen, understanding the place that the headset is in, is a 3D-mapping of everything that exists. But then it goes beyond that, the lighting and the shadows. What's the floor, what's walls, what's windows, you know—identification of those and using them effectively inside the content that you render in them is basically the spatial awareness and being a spatial computer.

So when you see a movie screen placed in your hall where you are wearing this and there is a window on the left, which is casting light in the room. They detect that window and make sure that window, that lighting effect has a slight effect on top on the screen that is being casted. Otherwise it will look very much like a sticker to you. But when it embeds in the real world, you feel like it's actually a screen in your space rather than a digital overlay. So that's what, spatial computing goes beyond the physical hard things that exist in your environment. It goes on the soft information of the lighting and the spatial configuration of what exists where.

Hari:

And what do you make of the form factor—it needs an externally connected battery?

Vivek:

Yes, I think given that it is a pro version, I think they knew the limitation of the hardware itself. The weight of it is very, very important. The battery hardware, the processing hardware, everything is not there yet… you can[not yet] make everything embedded and still lightweight. This will shed weight as we go along, but as of now since this is the Pro version, they went with an external connectivity for powering the device.

Hari:

Would you be tempted to build something new at PlayShifu around this?

Vivek:

Absolutely, I think right now what we're looking at, for example, Orboot, which is an AR product that we have. Should be a couple of days’ work when they do release grids for us to make it work through this headset. Because we already have this tech of spatial computing working inside our products, it’ll be through the camera of a mobile device. For moving that to this, I think Apple is going to make it very, very easy, and we can just launch our content and apps on Vision Pro seamlessly. But given the price point, I don't think any kids are getting near this anytime soon. So our TG doesn't collide with the vision that Apple has for the next two years, but we will definitely still be launching on it as it gets available, because we know that mass market devices is a couple of years away.

Hari:

What kind of details would you like to know more about?

Vivek:

I'm optimistic on the hardware side knowing that every three years there's a drastic change in the hardware cycles. What I'm looking forward to is how do we power content which is custom made for this kind of setup, which allows for better utility overall, rather than just a gimmick, right? So yes, you can have entertainment games, but when you start going beyond that is when you start having these devices in the day, you know, everyday life of any user. So I think Apple is going to crack that way better than anybody else. And I would love to know more about how they're going to enable developers who know how to build for Mac or iOS to seamlessly start getting on boarded on this and start making for this through Xcode. So I think that's one big thing that I'm looking forward to reading up on as soon as they release more.

Also, I'm interested in what tools they provide to start experimenting in 3D. All that the demo was, was 2D content being overlaid in the space, but we have ton of 3D content ready to be rendered, which obviously requires a lot more computational power than 2D content. But I'm sure they will release some kind of experimentational SDK to start experimenting with 3D content being rendered through Vision Pro. And that's something I'm excited about.

Hari:

How far away do you think we are in terms of this kind of technology being available in a form factor that just looks like your ordinary glasses?

Vivek:

Ordinary glasses, I think it's still far away tech because I think display is not there yet. Battery is not the constraint, I would feel. Display tech is the biggest roadblock right now—in terms of having a display tech which is sharp enough and see through enough at the same time. For certain pixels to be sharp, high resolution and pixels to be see-through is the biggest tech breakthrough that can happen, which can take the XR industry to the next level.

Summary