This is a great read, thank you for sharing! I learned a lot from reading it and I'm pretty experienced in this space! I enjoyed checking out your other work too.
OPs article heavily covers Vertex Amplification. I didn't realize VisionOS used Vertex Amplification for this, but it makes sense since the hardware supports it.
For those interested in Mesh Shading (a few good videos released this week [1][2][3]), Vertex Amplification is a key tech for Mesh Shading, where one writes Object/Mesh functions (names in Metal API, called Task/Mesh/Amplification shaders in other APIs). Introduced by Nvidia in 2018 and only really available en masse for the last couple years, Vertex Amplification was the first time a GPU could create vertices (or "destroy" them by not emitting anything), versus the fixed mesh inputs. It's so cool and powerful and a different way of thinking about the pipeline.
This article shows the same concept, but in Vertex Shaders for multiple render targets. While you might not make a Vision app, it could be worthwhile read to further understand this architecture. I've spent a few months in Metal Mesh Shading and hadn't realized this application of it at all.
[1] https://news.ycombinator.com/item?id=41839190
[2] https://www.youtube.com/watch?v=3EMdMD1PsgY
[3] https://www.youtube.com/watch?v=EtX7WnFhxtQ (good explanation and demo)
Nit: amplification shaders are not the same as mesh shaders (though in many cases they can be seen as abstractions over one)
Your description is of a mesh shaders, but an amplification shader is able to basically reuse vertex data from a vertex shader pass without use of a mesh shader.
After finally releasing a Vision Pro version of my app last week, I've sold a total of 7 units. That's by far my worst software launch ever, even worse than my past failures.
Don't bother developing for this platform. The customer base is miniscule.
For what it's worth, I'm a big fan of your app and was hoping for a Vision Pro release. I didn't know you released it, but will be buying it today.
I know that doesn't pay the bills, but I appreciate the hard work you do.
You should still get familiar with it in the event that the next one will be cheaper as that will increase the customer base a lot.
The rumor mill is saying it will be cheaper... but still not cheap at ~$2000.
https://www.uploadvr.com/non-pro-apple-vision-headset-report...
Meta is having enough trouble attracting developers with hardware that's an order of magnitude cheaper and has two orders of magnitude more units in the wild. They got through the first hurdle of convincing people to try VR but keep falling at the second hurdle of user retention, headsets that are gathering dust in a closet don't make any money for developers.
Apples decision to go all-in on hand tracking compounds the software problem even further because the vast majority of VR software that has already been developed is designed around controllers with IMUs/buttons/sticks/triggers/haptics, and would be very difficult to port over to the Vision Pros much more limited control scheme.
I'm not convinced the demographic of people willing to spend $2000 on an Apple headset is much different from the demographic of people willing to spend $3500 on an Apple headset
> The rumor mill is saying it will be cheaper... but still not cheap at ~$2000.
That same rumor mill has always said that an affordable consumer version is the long term goal.
Apple has started development on a consumer version of the tech in a glasses form factor several times over the years, only to decide the current tech still doesn't allow for what they want.
For instance, an iteration that was tethered to an iPhone that made it to test production in 2019:
> Apple could begin production of its long-rumored augmented reality glasses as early as the end of this year, noted analyst Ming-Chi Kuo has said.
https://www.theverge.com/2019/3/8/18256256/apple-ar-glasses-...
Apple does have the advantage of being able to project their 2D apps if they want, right? A $5k device seems like a dead end, and a $2k one seems pretty rough still (unless it totally replaces your laptop, which might make it worth considering).
They clearly haven’t executed it correctly yet. But at least they do have some hope of beating the bootstrapping problem that everybody else seems to have—they could try and get users first, and then I bet developers would quickly follow.
Unless the experience is good, it doesn't beat the bootstrapping problem and I haven't seen much clamouring for iPad apps on the AVP. It's a nice to have, sure. It will keep people in the headset longer, sure. But it's not a feature that will get someone to buy it over an iPad.
I sort of agree. But I think it is part of the puzzle.
The Apple headset thing is $5000 and much bigger than a pair of sunglasses. I just don’t think the tech is here yet to make something that most people actually want. So nobody has the problem that Apple can solve yet: good enough hardware in search of useful apps.
No VR device has even gotten close to having to answer a question like “how is this useful” because the current janky hardware is only appealing to those of us who are happy to just play games, haha.
Meta's Horizon OS is Android, so they could basically just do the same. Of course, they don't have a proper app store for that, but the compatibility should be there.
The main roadblock would be the OS support, and Meta's already has that. They are of course lacking another big roadblock, which is the store.
Meta just announced support for carrying standard 2D Android apps on their storefront at Connect a few weeks ago. I think the Vision Pro release has (pun intended) given them a vision to work towards (as seen by their rapid update process post launch to match features)
That said, visionOS can run iPad and iPhone apps unmodified. Meta will not bundle Google Play Services and a few other Android APIs, so APKs won’t be publishable on their store without some amount of work to use their alternate SDK.
> Apple does have the advantage of being able to project their 2D apps if they want, right?
Developers have to manually check a box to enable running the iPad version of their app on visionOS so it’s entirely out of Apple’s hands. Not sure why they took this approach
It’s the opposite of your claim. It’s opt out, not opt in.
> Your iPhone and iPad apps will be made available to users on Apple Vision Pro, unless you edit your apps’ availability in the App Store Connect
https://developer.apple.com/help/app-store-connect/manage-yo...
And it makes sense for some devs to not provide their app on the platform if they use unsupported APIs or if their experience is otherwise degraded for some unknowable reason. Though the side effect is Netflix and YouTube can just withhold their apps that presumably otherwise work.
Likely some sort of legal or PR reason.
You're still making a big assumption that this is a device anybody actually wants in the first place. How many of the people who bought one of these actually still use them with any regularity?
Could hit double digits next time
I doubt it... The platform seems like a dud without some big changes. If big changes are coming, how much time to market do you really save working on this iteration?
Familiarity is fine, but don't bet your livelihood on it just yet; have enough knowledge to be able to pivot to it if it does take off. But I'm not seeing it.
So, a developer should spend ~$5000 with no chance of ROI on the off chance that at some undefined point in the future this will pay off when they would have to spend another ~$5000 developing for the newer/cheaper/whatever device?
On track to having sold less than 15% of their Y1 target (400,000 vs 3 MM), production deliveries slashed, and the reported shareholder near-revolts, I think it's clear that the Vision Pro has flopped and there won't be a sequel any time soon.
All supply chain reports before the release were in line with the fact that the in-headset screens were capped at 1million maximum production possible for the first year, and two screens are required for each headset.
3MM was never possible, not sure where you got these numbers.
> On track to having sold less than 15% of their Y1 target (400,000 vs 3 MM)
This is a fictitious number. Apple never had the manufacturing capacity to reach that so-called target.
> the reported shareholder near-revolts
Citation needed.
GP is exaggerating (response below); at the same time many would argue that their overall argument that the Vision Pro is most probably a flop is in the ballpark. There's been much to say about price, the utility of VR, and most recently, the news that companies simply aren't bringing their apps to Vision Pro[1].
> On track to having sold less than 15% of their Y1 target (400,000 vs 3 MM)
All the reports online reflect actual vs. expected more around 450-500k vs. 800k - 1M, not 3 million.
> the reported shareholder near-revolts
The stock market's reaction to the Vision Pro announcement saw Apple's shares dip by 3%, indicating a mix of investor optimism and concern, not revolt.
[1] https://www.wsj.com/tech/personal-tech/apple-vision-pro-soft...
I think both you and the person you’re responding to are correct. The supply chain reports said <1M displays producible. That’s <500K devices producible.
Regarding stock, Apple almost always dips by a significant amount during an event. They’re very much traded as buying on the hype and selling on the news.
In my hallucination, my customer base is myself. If I had extra time right now and one of these devices to play with, I'd build something I wish existed, not something I expect the general population to buy.
If it's good enough for me, it's probably already worth my time—though https://xkcd.com/1319/ comes to mind
Then maybe some subset of the population that is closer in interests to me would also find it useful, in which case the app might get more traction. That'd be nice, but not a necessary requirement for my definition of success
The big issue here is justifying spending thousands of dollars on the product. I'd go for it at $500 or so. Definitely not $2k or $5k.
Any chance you can link to your app?
Don't bother developing for this platform. The customer base is miniscule.
A good hockey player don't skate to where the puck is. He skates to where it will be.
> A good hockey player don't skate to where the puck is.
I'm not a hockey player.
> He skates to where it will be.
The time scale for hockey players is seconds, not years.
This is such a useless cliché.
Most developers have very limited resources. Apple can afford to invest in pipe dreams (reportedly they spent $billions working on the now cancelled car project), but I certainly can't.
If you can't grok an analogy, that's on you. But let me simplify.
Don't bother developing for this platform. The customer base is miniscule.
1975: Don't bother writing MicroSoft BASIC. The customer base is miniscule.
I'm not pushing Apple's AR goggles. I'm not a customer, or even the target demographic. But if your company, or its leadership, lack vision for the future and only invest in the present, don't be surprised when the world passes you by.
> If you can't grok an analogy, that's on you.
Please review and respect the HN guidelines: https://news.ycombinator.com/newsguidelines.html
> 1975: Don't bother writing MicroSoft BASIC. The customer base is miniscule.
You're cherry-picking a single success, with the benefit of hindsight. How about listing all of the nascent technologies from 1975? It's likely that many of them flopped, and investing in them was a waste.
> if your company, or its leadership, lack vision for the future and only invest in the present
I am my company, a sole proprietor with very limited resources. I do invest in the future to some extent, but I can't invest in every possible future. The Vision Pro future does not look particularly promising.
Besides, you don't need to be first to be successful. I certainly wasn't the first to any platform, not even close. The thing about hockey is, there's only one puck on the ice. And it's a zero-sum game, one winner and one loser. But that's not even remotely true of tech.
> I recommend you do the same. https://news.ycombinator.com/newsguidelines.html
I quoted where you violated the guidelines: "If you can't grok an analogy, that's on you." That's a personal insult. Could you please quote where you think I violated the guidelines?
While your point is valid, I think the opposite might be as well.
Just ask the people who were developing games for Stadia: https://www.theverge.com/2022/9/30/23381106/google-stadia-sh...
> News that Google would be closing down Stadia was a big surprise for everyone, including the developers making games for the platform.
Now, Apple can probably pull off a lot of stuff, but with how quickly products get killed nowadays, sometimes you might end up betting on a future that will never come to pass.
That comes with the territory though.
The mere possibility of a 10x profit likely entails the possibility of a 10x risk somewhere, borne by someone, or groups of someones.
It's a risk / reward tradeoff; can you as an individual or a company afford to lose on the bet that Vision will become successful enough, AND that your app will become successful? There's much safer bets for anyone out there at the moment.
Buddy a hockey player skates to where the puck of being passed not to where the puck might be in the second half.
I think it’s you who deeply deeply misunderstands the metaphor.
Or to put it another way.
“Timing is everything”
50% of hockey players are below average.
I'm doubtful this is where the puck will be.
I mean… I think Apple hasn’t totally missed VR, and is still the only company that seems to have much of a chance of making it work…
But, a good hockey player doesn’t skate in the direction that absolutely nobody else is heading, somewhere that might even be outside the rink (in the sense that we haven’t really shown if a good UI for VR can actually be created, yet, so it might not even be a possible “part of the game,” so to speak).
a good hockey player doesn’t skate in the direction that absolutely nobody else is heading
That doesn't make sense in this context. Unless you consider Meta, Loft Dynamics, Valve, HTC, and dozens of other companies to be "absolutely nobody."
I think VR is not a serious business for Meta, just something Zuck is doing for fun. Why not, rich enough to goof off, right? They spent $4b to make $400M on their headsets.
A better business model would probably be to pay people a couple hundred dollars to take their headsets. At least that might result in some install base.
This is a pretty great reference repo in general for optimized mobile rendering, and most of it other than compositor services, can be generalized to other headsets in terms of approach (not API).
Thanks for putting this up!
I'm curious about the autoreleasepool. What's the reason to have it and how much of a difference does it make?
It's a lifetime management system for short-lived objects, same idea as memory arenas but for manually refcounted objects you got out of an API. It keeps the object alive until the end of the autorelease scope without anybody having to explicitly call the release method (the autoreleasepool does this instead). Not sure if the concept still makes sense with ARC though, it's probably just a no-op there - or probably not seeing that Swift seems to have inherited autoreleasepools from ObjC).
Hi! Yeah I know the concept from non-arc ObjC times but trying to understand the point in this context.
There's an autoreleasepool around the per-frame code - which seems to be standard procedure for Metal code - so any object within a render frame that has been created with autorelease will be released at the end of a frame.
Metal has a couple of 'frame transient' objects (like MTLRenderPassDescriptor and MTLRenderCommandEncoder) which have a new instance created in each frame, and I guess the main job of the autoreleasepool is to clean up those transient objects (along with any other 'short-lived-junk' that might be returned from Metal API methods).
And my guess for why this is still needed in the age of ARC: I guess that ARC has a 'autorelease-blindness', e.g. it cannot figure out at compile time what objects are registered with autorelease pools (especially when the objects are passed across DLL boundaries) - it can only add retain/release calls on top. Just speculation on my part though.
The render thread is running an infinite loop. You need to manually drain the autorelease pool, otherwise autoreleased objects created during rendering won't be released until the thread exits (there's an implicit top-level pool in each NSThread which is drained on exit). In UIKit/AppKit, the autorelease pool is drained at the end of each NSRunLoop iteration by the framework, so you don't typically drain it yourself. Here they created their own runloop - an infinite `while` loop that calls `onRender()` - so they must drain the pool themselves.
Looks great, thanks so much for writing this!
I've put everything I've learned while developing graphics for Apple Vision with the Metal API and Compositor Services in writing. It covers foveation, vertex amplification, stereoscopic rendering, and more techniques that extend beyond this particular device.
Thanks! Looks very interesting and usefull.