Drawing Graphics on Apple Vision with the Metal Rendering API

142 pointsposted 9 months ago
by georginikolov
(github.com)

94 Comments

neomantra

9 months ago

This is a great read, thank you for sharing! I learned a lot from reading it and I'm pretty experienced in this space! I enjoyed checking out your other work too.

OPs article heavily covers Vertex Amplification. I didn't realize VisionOS used Vertex Amplification for this, but it makes sense since the hardware supports it.

For those interested in Mesh Shading (a few good videos released this week [1][2][3]), Vertex Amplification is a key tech for Mesh Shading, where one writes Object/Mesh functions (names in Metal API, called Task/Mesh/Amplification shaders in other APIs). Introduced by Nvidia in 2018 and only really available en masse for the last couple years, Vertex Amplification was the first time a GPU could create vertices (or "destroy" them by not emitting anything), versus the fixed mesh inputs. It's so cool and powerful and a different way of thinking about the pipeline.

This article shows the same concept, but in Vertex Shaders for multiple render targets. While you might not make a Vision app, it could be worthwhile read to further understand this architecture. I've spent a few months in Metal Mesh Shading and hadn't realized this application of it at all.

[1] https://news.ycombinator.com/item?id=41839190

[2] https://www.youtube.com/watch?v=3EMdMD1PsgY

[3] https://www.youtube.com/watch?v=EtX7WnFhxtQ (good explanation and demo)

dagmx

9 months ago

Nit: amplification shaders are not the same as mesh shaders (though in many cases they can be seen as abstractions over one)

Your description is of a mesh shaders, but an amplification shader is able to basically reuse vertex data from a vertex shader pass without use of a mesh shader.

cubefox

9 months ago

Nit: Mesh shading, or the mesh shader pipeline, has two unique parts: optional task/amplification shaders, and mandatory mesh shaders.

neomantra

9 months ago

Thanks both for pulling on this point. In drafting, I was both torn and fuzzy about the nomenclature. Also trying to be distinct about hardware features (dynamic vertex data?) versus language/API constructs. But you are right, it's important to be clear because they are different and hardware might not support both.

I was trying to see if the other graphics APIs (Vulkan, DirectX) had this Vertex Amplification in Vertex Shader feature, but it doesn't seem so? Maybe it was easier for Apple to inject the concept into Metal (advantage of controlling of the whole stack).

cubefox

9 months ago

I'm not sure this is relevant but apparently Apple introduced proper mesh shading only with the M3 generation chips:

> The [M3] GPU is faster and more efficient, and introduces a new technology called Dynamic Caching, while bringing new rendering features like hardware-accelerated ray tracing and mesh shading to Mac for the first time.

https://www.apple.com/ne/newsroom/2023/10/apple-unveils-m3-m...

That means the M2 (which is inside the Apple Vision) must have at most some sort of partial mesh shading support.

dagmx

9 months ago

AFAIK Apple support fallback to compute shader based implementations of both raytracing and mesh shading on older GPUs. It’ll be slower, but for something like amplification shaders the win would be worth it

neomantra

9 months ago

I actually researched this a bit right when it came out.... I can't find my source now, but Vision Pro has its own SOC with an M2 CPU with a "2.5" GPU (between M2 and M3 families, like M2 with some goodies). Oddly, Vision isn't discussed in the handy Metal Feature Set Table [1].

The GPU Family Apple7 was the first with Mesh Shader support, but as you noted it might be software fallbacks down there.

I did just check with my Vision Pro and I get MTLGPUFamily [2] up to apple8, which does correspond to other M2 devices. I probed a max amplification count of 8.

[1] https://developer.apple.com/metal/Metal-Feature-Set-Tables.p...

[2] https://developer.apple.com/documentation/metal/mtlgpufamily

dagmx

9 months ago

No it’s a standard M2. You can probe all the values that are exposed to software and it’ll line up. People have done tear downs and the SoC is the same size as a regular M2, which wouldn’t be possible otherwise.

lapcat

9 months ago

After finally releasing a Vision Pro version of my app last week, I've sold a total of 7 units. That's by far my worst software launch ever, even worse than my past failures.

Don't bother developing for this platform. The customer base is miniscule.

FumblingBear

9 months ago

For what it's worth, I'm a big fan of your app and was hoping for a Vision Pro release. I didn't know you released it, but will be buying it today.

I know that doesn't pay the bills, but I appreciate the hard work you do.

airstrike

9 months ago

In my hallucination, my customer base is myself. If I had extra time right now and one of these devices to play with, I'd build something I wish existed, not something I expect the general population to buy.

If it's good enough for me, it's probably already worth my time—though https://xkcd.com/1319/ comes to mind

Then maybe some subset of the population that is closer in interests to me would also find it useful, in which case the app might get more traction. That'd be nice, but not a necessary requirement for my definition of success

The big issue here is justifying spending thousands of dollars on the product. I'd go for it at $500 or so. Definitely not $2k or $5k.

latexr

9 months ago

The developer in question makes a living selling Safari extensions, and the Vision Pro app they’re referring to is a version of what’s probably their most well-known. So we’re talking about an established developer with an established user base selling a version of an established app.

Your advice makes sense in general but doesn’t apply to this specific case.

airstrike

9 months ago

It still applies to this specific case. He's released a version of his existing app hoping others might use, instead of solving a problem he identified would be best tackled with a Vision Pro than with a desktop app.

All I'm saying is "don't build an app for the masses if the market is small. build something that solves a real need that you have personally identified"

user

9 months ago

[deleted]

lapcat

9 months ago

> He's released a version of his existing app hoping others might use

I released a version of my existing app because customers requested it. I had no independent interest in the platform.

> instead of solving a problem he identified would be best tackled with a Vision Pro than with a desktop app.

I can't think of any problems that would be best tackled with a Vision Pro than with a desktop app.

wpm

9 months ago

>I'd build something I wish existed

Then you also better hope Apple permits you to use the APIs you need to build it.

assassinator42

9 months ago

I would assume targeting OpenXR would be much more productive? It's unfortunate that Apple insists on being so proprietary.

gpm

9 months ago

The app in question appears to be a safari extension... so... unlikely.

spookie

9 months ago

They are a latecomer to this market. Decisions like this are going to kill any hopes of success...

(Just a guy who has seen all proprietary APIs in this space being left in the dust when a standard arrived)

wpm

9 months ago

Friggin Apple didn't even bother porting all their apps over, not sure why anyone else should.

xivusr

9 months ago

What's yr app? I'll check it out if its still available

reaperducer

9 months ago

Don't bother developing for this platform. The customer base is miniscule.

A good hockey player don't skate to where the puck is. He skates to where it will be.

lapcat

9 months ago

> A good hockey player don't skate to where the puck is.

I'm not a hockey player.

> He skates to where it will be.

The time scale for hockey players is seconds, not years.

This is such a useless cliché.

Most developers have very limited resources. Apple can afford to invest in pipe dreams (reportedly they spent $billions working on the now cancelled car project), but I certainly can't.

reaperducer

9 months ago

[flagged]

lapcat

9 months ago

> If you can't grok an analogy, that's on you.

Please review and respect the HN guidelines: https://news.ycombinator.com/newsguidelines.html

> 1975: Don't bother writing MicroSoft BASIC. The customer base is miniscule.

You're cherry-picking a single success, with the benefit of hindsight. How about listing all of the nascent technologies from 1975? It's likely that many of them flopped, and investing in them was a waste.

> if your company, or its leadership, lack vision for the future and only invest in the present

I am my company, a sole proprietor with very limited resources. I do invest in the future to some extent, but I can't invest in every possible future. The Vision Pro future does not look particularly promising.

Besides, you don't need to be first to be successful. I certainly wasn't the first to any platform, not even close. The thing about hockey is, there's only one puck on the ice. And it's a zero-sum game, one winner and one loser. But that's not even remotely true of tech.

reaperducer

9 months ago

[flagged]

lapcat

9 months ago

> I recommend you do the same. https://news.ycombinator.com/newsguidelines.html

I quoted where you violated the guidelines: "If you can't grok an analogy, that's on you." That's a personal insult. Could you please quote where you think I violated the guidelines?

1123581321

9 months ago

He gave you a great fuller explanation, and understood what you were saying the first time. You are being rude.

KronisLV

9 months ago

While your point is valid, I think the opposite might be as well.

Just ask the people who were developing games for Stadia: https://www.theverge.com/2022/9/30/23381106/google-stadia-sh...

> News that Google would be closing down Stadia was a big surprise for everyone, including the developers making games for the platform.

Now, Apple can probably pull off a lot of stuff, but with how quickly products get killed nowadays, sometimes you might end up betting on a future that will never come to pass.

GeekyBear

9 months ago

> with how quickly products get killed nowadays

Apple's history is to pick a strategic product direction and then continue to iterate on their hardware and software for that product year after year.

For instance, Apple Watch.

Throwing products over the wall and killing them if they don't prove to be an immediate success is more if a Google/Microsoft thing.

Although it's fair to point out that Microsoft did at least update HoloLens hardware once before they killed it.

The issue with both HoloLens and Vision Pro is the nearly identical price point. You aren't going to see mass market sales figures until you can reach mass market price points.

MichaelZuo

9 months ago

That comes with the territory though.

The mere possibility of a 10x profit likely entails the possibility of a 10x risk somewhere, borne by someone, or groups of someones.

tsunamifury

9 months ago

Buddy a hockey player skates to where the puck of being passed not to where the puck might be in the second half.

I think it’s you who deeply deeply misunderstands the metaphor.

Or to put it another way.

“Timing is everything”

Cthulhu_

9 months ago

It's a risk / reward tradeoff; can you as an individual or a company afford to lose on the bet that Vision will become successful enough, AND that your app will become successful? There's much safer bets for anyone out there at the moment.

madeofpalk

9 months ago

I'm doubtful this is where the puck will be.

brudgers

9 months ago

50% of hockey players are below average.

coolspot

9 months ago

* median, below median

bee_rider

9 months ago

I mean… I think Apple hasn’t totally missed VR, and is still the only company that seems to have much of a chance of making it work…

But, a good hockey player doesn’t skate in the direction that absolutely nobody else is heading, somewhere that might even be outside the rink (in the sense that we haven’t really shown if a good UI for VR can actually be created, yet, so it might not even be a possible “part of the game,” so to speak).

reaperducer

9 months ago

a good hockey player doesn’t skate in the direction that absolutely nobody else is heading

That doesn't make sense in this context. Unless you consider Meta, Loft Dynamics, Valve, HTC, and dozens of other companies to be "absolutely nobody."

bee_rider

9 months ago

I think VR is not a serious business for Meta, just something Zuck is doing for fun. Why not, rich enough to goof off, right? They spent $4b to make $400M on their headsets.

A better business model would probably be to pay people a couple hundred dollars to take their headsets. At least that might result in some install base.

that_lurker

9 months ago

You should still get familiar with it in the event that the next one will be cheaper as that will increase the customer base a lot.

jsheard

9 months ago

The rumor mill is saying it will be cheaper... but still not cheap at ~$2000.

https://www.uploadvr.com/non-pro-apple-vision-headset-report...

Meta is having enough trouble attracting developers with hardware that's an order of magnitude cheaper and has two orders of magnitude more units in the wild. They got through the first hurdle of convincing people to try VR but keep falling at the second hurdle of user retention, headsets that are gathering dust in a closet don't make any money for developers.

Apples decision to go all-in on hand tracking compounds the software problem even further because the vast majority of VR software that has already been developed is designed around controllers with IMUs/buttons/sticks/triggers/haptics, and would be very difficult to port over to the Vision Pros much more limited control scheme.

hbn

9 months ago

I'm not convinced the demographic of people willing to spend $2000 on an Apple headset is much different from the demographic of people willing to spend $3500 on an Apple headset

GeekyBear

9 months ago

> The rumor mill is saying it will be cheaper... but still not cheap at ~$2000.

That same rumor mill has always said that an affordable consumer version is the long term goal.

Apple has started development on a consumer version of the tech in a glasses form factor several times over the years, only to decide the current tech still doesn't allow for what they want.

For instance, an iteration that was tethered to an iPhone that made it to test production in 2019:

> Apple could begin production of its long-rumored augmented reality glasses as early as the end of this year, noted analyst Ming-Chi Kuo has said.

https://www.theverge.com/2019/3/8/18256256/apple-ar-glasses-...

bee_rider

9 months ago

Apple does have the advantage of being able to project their 2D apps if they want, right? A $5k device seems like a dead end, and a $2k one seems pretty rough still (unless it totally replaces your laptop, which might make it worth considering).

They clearly haven’t executed it correctly yet. But at least they do have some hope of beating the bootstrapping problem that everybody else seems to have—they could try and get users first, and then I bet developers would quickly follow.

jayd16

9 months ago

Unless the experience is good, it doesn't beat the bootstrapping problem and I haven't seen much clamouring for iPad apps on the AVP. It's a nice to have, sure. It will keep people in the headset longer, sure. But it's not a feature that will get someone to buy it over an iPad.

bee_rider

9 months ago

I sort of agree. But I think it is part of the puzzle.

The Apple headset thing is $5000 and much bigger than a pair of sunglasses. I just don’t think the tech is here yet to make something that most people actually want. So nobody has the problem that Apple can solve yet: good enough hardware in search of useful apps.

No VR device has even gotten close to having to answer a question like “how is this useful” because the current janky hardware is only appealing to those of us who are happy to just play games, haha.

kridsdale3

9 months ago

Porn. Porn is how it is useful. It is 10x better than on a monitior/tv, and 100x better than on a mobile device. Some well heeled dirtbags are willing to pay for that.

troupo

9 months ago

> being able to project their 2D apps if they want, right

That's how they want to kickstart the app ecosystem by not doing anything.

Any non-ported app with any non-trivial functionality is likely to break in both obvious and subtle ways especially around UX. Who's gonna support the app on the platform and deal with customer complaints? Apple? You? Or the dev who might not even have the device to debug it on?

kridsdale3

9 months ago

This comment identifies why Netflix and Youtube and other Google apps (I presume) are pulled from VisionOS.

ladyanita22

9 months ago

Meta's Horizon OS is Android, so they could basically just do the same. Of course, they don't have a proper app store for that, but the compatibility should be there.

The main roadblock would be the OS support, and Meta's already has that. They are of course lacking another big roadblock, which is the store.

jsheard

9 months ago

> Meta's Horizon OS is Android, so they could basically just do the same.

They just announced they're doing that: https://developers.meta.com/horizon/blog/building-2d-apps-on...

AFAICT there won't be Play Store support though, so devs will have to publish their 2D Android apps specifically to the Quest store.

cubefox

9 months ago

Presumably Google wants VR headsets to run Google's own version of Android, perhaps called Android VR (or the like), though I haven't heard of a VR version of Android developed by Google. Previously they developed Android TV and Android for smartwatches. Even a version for cars I believe.

david_allison

9 months ago

Thank you for the link!

I maintain an Android app which doesn't require Play Services. Might as well get it on the store if Meta is encouraging it

dagmx

9 months ago

Meta just announced support for carrying standard 2D Android apps on their storefront at Connect a few weeks ago. I think the Vision Pro release has (pun intended) given them a vision to work towards (as seen by their rapid update process post launch to match features)

That said, visionOS can run iPad and iPhone apps unmodified. Meta will not bundle Google Play Services and a few other Android APIs, so APKs won’t be publishable on their store without some amount of work to use their alternate SDK.

gryn

9 months ago

you can already install android apps on the quest (since day one probably). I did that regularly over the past year, they work great.

it's just not officially supported, since they have been having a fight with google for years now. meta is claiming that google are the ones that don't want to see the play store on the platform.

marxisttemp

9 months ago

> Apple does have the advantage of being able to project their 2D apps if they want, right?

Developers have to manually check a box to enable running the iPad version of their app on visionOS so it’s entirely out of Apple’s hands. Not sure why they took this approach

dagmx

9 months ago

It’s the opposite of your claim. It’s opt out, not opt in.

> Your iPhone and iPad apps will be made available to users on Apple Vision Pro, unless you edit your apps’ availability in the App Store Connect

https://developer.apple.com/help/app-store-connect/manage-yo...

And it makes sense for some devs to not provide their app on the platform if they use unsupported APIs or if their experience is otherwise degraded for some unknowable reason. Though the side effect is Netflix and YouTube can just withhold their apps that presumably otherwise work.

marxisttemp

9 months ago

Oh thank you for the info, I’m not sure where I got my misinfo from! Presumably a podcast from the early days of the release.

dagmx

9 months ago

I think some folks confused it with the checkbox to build against the visionOS SDK.

MichaelZuo

9 months ago

Likely some sort of legal or PR reason.

thomastjeffery

9 months ago

It's not just hand tracking. We're literally here commenting on a tutorial for the Apple-exclusive Metal API.

Every step of development is a reason not to write software for this thing, and it doesn't exist in a vacuum, either.

I have trouble believing that even a successor to the Vision Pro would look significantly better than my $800 Pimax Crystal Light.

moolcool

9 months ago

You're still making a big assumption that this is a device anybody actually wants in the first place. How many of the people who bought one of these actually still use them with any regularity?

troupo

9 months ago

So, a developer should spend ~$5000 with no chance of ROI on the off chance that at some undefined point in the future this will pay off when they would have to spend another ~$5000 developing for the newer/cheaper/whatever device?

jayd16

9 months ago

I doubt it... The platform seems like a dud without some big changes. If big changes are coming, how much time to market do you really save working on this iteration?

jncfhnb

9 months ago

Could hit double digits next time

Cthulhu_

9 months ago

Familiarity is fine, but don't bet your livelihood on it just yet; have enough knowledge to be able to pivot to it if it does take off. But I'm not seeing it.

layer8

9 months ago

The platform has no killer mainstream use-case besides watching movies on a big virtual screen in solitude, with the drawback of having a heavy headset strapped to your face. I doubt that it would fare very well even sub-$1000. If they invested in gaming and allowed 3D controllers, the story might be different.

closewith

9 months ago

On track to having sold less than 15% of their Y1 target (400,000 vs 3 MM), production deliveries slashed, and the reported shareholder near-revolts, I think it's clear that the Vision Pro has flopped and there won't be a sequel any time soon.

lapcat

9 months ago

> On track to having sold less than 15% of their Y1 target (400,000 vs 3 MM)

This is a fictitious number. Apple never had the manufacturing capacity to reach that so-called target.

> the reported shareholder near-revolts

Citation needed.

andsoitis

9 months ago

GP is exaggerating (response below); at the same time many would argue that their overall argument that the Vision Pro is most probably a flop is in the ballpark. There's been much to say about price, the utility of VR, and most recently, the news that companies simply aren't bringing their apps to Vision Pro[1].

> On track to having sold less than 15% of their Y1 target (400,000 vs 3 MM)

All the reports online reflect actual vs. expected more around 450-500k vs. 800k - 1M, not 3 million.

> the reported shareholder near-revolts

The stock market's reaction to the Vision Pro announcement saw Apple's shares dip by 3%, indicating a mix of investor optimism and concern, not revolt.

[1] https://www.wsj.com/tech/personal-tech/apple-vision-pro-soft...

dagmx

9 months ago

I think both you and the person you’re responding to are correct. The supply chain reports said <1M displays producible. That’s <500K devices producible.

Regarding stock, Apple almost always dips by a significant amount during an event. They’re very much traded as buying on the hype and selling on the news.

CameronBanga

9 months ago

All supply chain reports before the release were in line with the fact that the in-headset screens were capped at 1million maximum production possible for the first year, and two screens are required for each headset.

3MM was never possible, not sure where you got these numbers.

corysama

9 months ago

I've always assumed the Vision Pro was an experimental platform that is available publicly solely to enable developers to figure out what the heck can be done with a high-quality MR device.

wpm

9 months ago

Nothing can really be done with it until Apple provides more software and APIs for it, at least, aside from some medicore iPad app ports.

And yes I'm talking about most of the "native apps".

tasoeur

9 months ago

For people who want to play with metal shaders on Apple Vision Pro I recommend this live coding app: https://shader.vision

dagmx

9 months ago

This is a pretty great reference repo in general for optimized mobile rendering, and most of it other than compositor services, can be generalized to other headsets in terms of approach (not API).

Thanks for putting this up!

Animats

9 months ago

Metal is the main reason many games are never ported to MacOS.

Metal is roughly equivalent to Vulkan, which is on Windows and Linux. But Apple just had to Think Different. So all the major rendering libraries either have a layer which abstracts over Vulkan and Metal, or they don't support MacOS at all.

That's what WGPU does. If it were not for Apple ignoring standards, much of that would be unnecessary. It adds so much excess baggage that the consensus is to go direct to Vulkan or DX12, and blow off MacOS support.

aseipp

9 months ago

If Apple actually wanted to bring games to MacOS, they'd give Valve and CodeWeavers a billion dollars and get the Steam Client working, put tons of effort into a d3dvk-for-Metal solution, and have it working with as little overhead as possible. They don't do this because they can't funnel them through the App Store. The graphics API they use for the operating isn't relevant (Vulkan isn't used very much for most games.)

The funny thing is that with the rise of Proton and Steam Deck driving its development -- there's literally no reason anymore for a game developer to ever target any "Desktop PC" platform except x86_64 Portable Executable files, using DirectX. Because Proton will just handle the port with nearly zero effort. Even on ARM64 devices like Snapdragon, the CPU is often not the limiting factor (and binary translation is only in the realm of 15%) so emulation for games is totally viable if the GPU can handle the load.

GeekyBear

9 months ago

There is no such thing as "the one true gaming graphics API"

Windows/Xbox uses Direct3D, Mac/iDevices use Metal, PlayStation uses GNM/GNMX, Nintendo Switch uses NVN.

Compared to DirectX, Vulkan is barely used at all in gaming, except on Linux and the most recent subset of Android devices.

If it weren't for Proton's DirectX emulation running on top on Vulkan, Vulkan would be largely irrelevant, from a "widely used in real world gaming" perspective.

aYsY4dDQ2NrcNzA

9 months ago

> Apple just had to Think Different.

Metal is 1.7 years older than Vulkan. (June 2014 versus February 2016)

spease

9 months ago

Does Metal allow for better performance with Apple hardware than if Vulcan were the “direct” interface, if you optimize with it?

nelup20

9 months ago

Looks great, thanks so much for writing this!

tomovo

9 months ago

I'm curious about the autoreleasepool. What's the reason to have it and how much of a difference does it make?

flohofwoe

9 months ago

It's a lifetime management system for short-lived objects, same idea as memory arenas but for manually refcounted objects you got out of an API. It keeps the object alive until the end of the autorelease scope without anybody having to explicitly call the release method (the autoreleasepool does this instead). Not sure if the concept still makes sense with ARC though, it's probably just a no-op there - or probably not seeing that Swift seems to have inherited autoreleasepools from ObjC).

tomovo

9 months ago

Hi! Yeah I know the concept from non-arc ObjC times but trying to understand the point in this context.

flohofwoe

9 months ago

There's an autoreleasepool around the per-frame code - which seems to be standard procedure for Metal code - so any object within a render frame that has been created with autorelease will be released at the end of a frame.

Metal has a couple of 'frame transient' objects (like MTLRenderPassDescriptor and MTLRenderCommandEncoder) which have a new instance created in each frame, and I guess the main job of the autoreleasepool is to clean up those transient objects (along with any other 'short-lived-junk' that might be returned from Metal API methods).

And my guess for why this is still needed in the age of ARC: I guess that ARC has a 'autorelease-blindness', e.g. it cannot figure out at compile time what objects are registered with autorelease pools (especially when the objects are passed across DLL boundaries) - it can only add retain/release calls on top. Just speculation on my part though.

meindnoch

9 months ago

The render thread is running an infinite loop. You need to manually drain the autorelease pool, otherwise autoreleased objects created during rendering won't be released until the thread exits (there's an implicit top-level pool in each NSThread which is drained on exit). In UIKit/AppKit, the autorelease pool is drained at the end of each NSRunLoop iteration by the framework, so you don't typically drain it yourself. Here they created their own runloop - an infinite `while` loop that calls `onRender()` - so they must drain the pool themselves.

georginikolov

9 months ago

I've put everything I've learned while developing graphics for Apple Vision with the Metal API and Compositor Services in writing. It covers foveation, vertex amplification, stereoscopic rendering, and more techniques that extend beyond this particular device.

fsloth

9 months ago

Thanks! Looks very interesting and usefull.