Eigenvalue are more flexible than singular values -- If you have procedure for computing eigenvalues of C, squared singular values of X are eigenvalues of C=XX'. Left and write singular vectors of X and eigenvectors of XX' and X'X. For rotation rotation matrix, singular values are all 1. Meanwhile eigenvalues tell you angles of rotation https://math.stackexchange.com/questions/4874616/obtaining-a...

Just going to sound really pedantic here, but RGB does not capture the entire colour space. In fact, it only captures about 35% of the colours the human eye can perceive.

https://www.oceanopticsbook.info/view/photometry-and-visibil...

Last statement is a bit sus... Muon computes matrix sign function which can be defined as setting singular values to 1, though you can also define it without SVD. Muon itself doesn't use SVD because it uses a faster method to compute matrix sign. Adam doesn't do anything related to SVD or singular values. Also not sure what you meant by "second order singular values"

Really? I find the part about the SVD in Axler's book extremely unhelpful, big blobs of opaque formulas and jargon with next to no explanation or context that basically require either knowing the topic fully beforehand or a huge amount of effort to parse.

e.g. Axler's definition of singular values is the extremely dry and technical:

> Suppose T is in L(V, W). The singular values of T are the nonnegative square roots of the eigenvalues of T†T, listed in decreasing order, each included as any times as the dimension of the corresponding eigenspace of T†T.

(Using a dagger instead of an asterisk for the conjugate transpose since HN interprets and asterisk to mean italics.)

If you already just proved a lot of stuff about eigenvalues, this could be a serviceable definition; at any rate it saves space. But it doesn't really explain the point.

I'd recommend anyone interested in this or related topics read Trefethen & Bau (1997) Numerical Linear Algebra.

I made a demo of the SVD part.

https://jmalicki.github.io/svd-grad/

The devil was in the details, and that escalated quickly from a simple idea to actually getting it to work sucked me into a ton of random deep corners.

I've recently passed my deep and generative learning exam and in fact before you said vae something was resonating with me, although in our course it wasn't made explicit we were approximating an svd.

Staying in the field of autoencoder, it blows my mind how you can pass from denoising autoencoder (computer science) to scores and eventually matching flows (physics) quite seamlessly

I also forgot that you have to constrain U and V to be orthonormal - otherwise you end up with the same low rank approximation D', but exactly which numbers go in U, S, or V can shift as it's underdetermined without that constraint.

Aren't you skipping over the noise/stochasticity part from the sampling?

It comes up anywhere that youre working with data that has some sort of correlation structure.

In image processing, the SVD makes it possible to talk about all the rich spatial correlations in the image, and pick out the strongest ones and discard noise.

This is also why it's so ubiquitous in compression algorithms, and of central importance in stuff like quantum information.

what work are you doing in computer vision that isn't entirely ML these days?

> Claude and Codex are now generating all my code for me now, and I see them spitting out SVD code frequently -- often for very special cases.

I find this so annoying. I had to PR some Claude-generated gaussian elimination routine last month and making sure it got the pivoting logic correct was a waste of my time.

This paper is actually not a bad one to get started with. Obviously you need the proper background and I am not sure how much linear algebra you've taken. Going through a senior undergrad numerical linear algebra text might be a good way to learn the prerequsites. In particular, you'll never be able to read a math paper without developing mathematical maturity via textbook exercises.

Assuming you have the background, the boring answer is "patience and practice." For years I had to practically rewrite math papers word for word in order to get anything to stick. These days I am better at reading "mentally," but still sloppy and prone to misreading (just yesterday I misread GPT's proof because I was lazy and on my phone). More so than the empirical sciences, mathematics demands you understand every sentence before moving on to the next. Skimming does you no good. It really does just take patience and perseverance.

The nice thing about this paper is that the math isn't especially advanced, and it's broken up with qualitative historical discussions. If you know a decent bit of linear algebra (enough to understand artificial neural networks), I think you can muddle through this.

Axler’s _Linear Algebra Done Right_ should get you at least part of the way there, content-wise, depending on what other math background you have.

As for reading math papers in general, it’s mostly a process of stepping through it incrementally and trying to verify the steps you don’t understand based on the surrounding context. Most of the concepts in this paper are accessible on Wikipedia or elsewhere, you can make small (e.g. 2 x 2) examples as you go and see what happens.

It’s not an easy skill to acquire from scratch, especially from outside the ivory tower.