Ask HN: What to learn for math for modeling?

67 pointsposted 4 days ago
by shivajikobardan

Item id: 45036708

27 Comments

m-a-t-t-i

2 days ago

Here is one excellent resource for all of the things you mentioned: Introduction to Computing with Geometry by Dr. C.-K. Shene https://pages.mtu.edu/~shene/COURSES/cs3621/NOTES/

When I was implementing these, the key was to visualise everything. It was much easier to see if things are correct by stepping through the algorithms graphically than just by looking at the math.

efavdb

2 days ago

Sounds like you’re interested in curve fitting. I would recommend intro to statistical learning for a good intro to related topics.

Free download

https://www.statlearning.com/

joshdavham

2 days ago

Oh wow! I had no idea they came out with a Python version!

Jgoauh

2 days ago

I am not very knownledgeable on Computer Graphics Maths, but i have an interest in Texture Synthesis so i might be able to help as i was not mathematically literate when i started. Here's my strategy :

1 - Find a recent research paper of low-medium complexity 2 - Read it in whole 3 - take note of any term i don't understand 4 - Read the research paper where this idea was introduced 5 - Repeat steps 3 - 5

I am currently reading a fascinating 1973 paper about how different statistics affect our ability to differenciate between different textures. I am not a pro of Texture Synthesis maths but i have a good idea of where the field started, where it is going and understand the decisions of modern papers much better.

bulubulu

7 hours ago

Georgia Institute of Technology has a course for these topics. It's part of their online master's program so I believe you can find a lot of resources online.

ISYE 8803 Topics on High-Dimensional Data Analytics

happy2behere

a day ago

For: spline, b-spline, cubic bezier curve, de casteljau algorithm

The good & old "The Nurbs Book" by Pieg and Tiller. You said you were a Computer Engineering Graduate, so I assume you know the basics of matrix multiplication, polynomials, etc? There's not much more needed than that, maybe some basic Calculus, understanding definition of limits and derivatives so things like continuity of knots make sense. Maybe a little knowledge on numerical methods is good, like some fixed-point iteration methods (Newton's).

mhh__

2 days ago

Everything is basically calculus and linear algebra under the hood (until it's not, at which point you'll know)

godelski

2 days ago

  > I have done math classes
What classes? It's a bit hard to suggest what is appropriate without understanding where you currently are.

  > to brush up.
But I think this might be the wrong approach. Especially for these topics. Instead of learning the minimum you need to just understand a specific algorithm, you should understand what the symbols mean, what they are doing, and how they are interacting with one another. Math is a language. But it is a language designed for precision.

A big part of why I'm saying this is because at their core, these are not difficult equations from a computational perspective. You really just need to understand addition, subtraction, multiplication, and division. My point is that the hard part about math often isn't the calculation part.

Let's take B-Splines for an example and look at what Wiki says

  > A B-spline is defined as a piecewise polynomial of order n, meaning a degree of n − 1.
Either this is simple or incomprehensible, so let's break it down.

What does piecewise mean? Here's an example piecewise function.

          a     if x > 0
  f(x) = {0     x = 0
          b     if x < 0
Given you're on HN and looking at computer graphics, I take it you're familiar with if statements? That's the same thing here. Let's translate. Looking at things from a different perspective can really help.

  if x > 0:
    return a
  else if x < 0:
    return b
  else: # x == 0
    return 0
We should think of x as a location and we're just describing something about that location. Here's another piecewise function, but now in everyday English: "Across the street all the houses end in even numbers, but on my side of the street all the houses are odd." There's always more ways to look at something. If you can't see what something means from this angle, try another. Be that another book teaching the same thing or another framework to describe the same thing. Get multiple perspectives. One will make more sense than another but which one that is tends to be personal.

"polynomial of order n"

A polynomial is an equation like a_0 + a_1 x + a_2 x^2 + ... + a_n x^{n}. Sorry, let me rewrite that: a_0 x^0 + a_1 x^1 + a_2 x^2 + ... + a_n x^{n}

Now in code

  for i in range(n):
      sum += a[i] * math.pow(x, i)
But we should break this down more, just like before. What is a_0? It is just a number. What does this represent? A point. What is a_1*x? That's a line. x is an arbitrary location. The a_1 is a scaling factor. So if a_1 is the number 2, then what our x is representing is all numbers (..., -4, -2, 0, 2, 4, ...} (is that all my neighbors across the street?) The a_2 x^2 is a curve, specifically a parabola. We can do the same thing. But also remember we're summing them together. So you have a point, a line, and a parabola. When you add them together you get something that is none of those things.

What is really important here is that these functions you are interested in have a really critical property. They can be used to approximate almost anything. That almost part is really important, but you're going to have to dig deeper.

My point here is that take time to slow down. Don't rush this stuff. I promise you that if you take time to slow down then the speed will come. But if you try to go too fast you'll just end up getting stuck. This is tricky stuff. When learning it doesn't always make a ton of sense and unfortunately(?) when you do understand it it is almost obvious. Think of the slowing down part like making a plan. In a short race you should just take off running without thinking. But if it is longer then you will go faster if you first plan and strategize. It is the same thing here and I promise you this isn't a short race. You don't win a marathon by winning a bunch of consecutive sprints. The only thing you win by trying that is a trip to the hospital.

So slow down. Break problems apart. Find one part you think is confusing and focus on that. If it all seems confusing then try to walk through and force yourself to say why it is confusing. Keep doing this until you have something you understand. It is an iterative process. It'll feel slow, but I promise it pays dividends. Any complex problem can be broken down into a bunch of small problems[0]

You got this!

[0] This sentence doesn't just apply to how to figure things out, it applies directly to what you're trying to do and why you want these functions. If it doesn't make sense now, revisit, it will later.

rramadass

2 days ago

Very nice explanation!

In particular; the idea of breaking down a polynomial as a sum of terms, mapping each term to a graphical view (i.e. analytic to coordinate geometry) and then realizing such a sum of terms can be a complex curve (i.e. a complex graphical view) which can be an approximation of almost any function.

The teaching of mathematics has become so abstract that students are not taught how to map it to geometry which is THE way to build intuition. You can understand a lot by just imagining 2D/3D mappings before generalizing to n-dimensional vector spaces. Every student/beginner should study I.M.Gelfand's Functions and Graphs so that they can train themselves to imagine the graphs corresponding to a string of symbols.

godelski

a day ago

  > You can understand a lot by just imagining 2D/3D mappings before generalizing to n-dimensional vector spaces.
While I'm a big fan of visualization I must also stress that it can equally be misleading. It's to make mistakes because you can only visualize anything above 3D like a blind man can describe an elephant[0]. It can be useful but it is also common to get caught up in one aspect while ignoring the rest. The trouble is you're not ignoring because you're rejecting what's there, you're ignoring because you can't see it and it is really hard to see it.

I also think there's this misnomer of being good at geometry or good at algebra. I lied about math being a language. It is a family of languages. It's good to be able to speak several of these languages, just like it is beneficial to be able to program in multiple languages. The best language is only the best one in a certain context, but technically any of them will work. That's why I suggest multiple perspectives. I'll even say that programming is one of those math languages ;)

[0] https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2F...

rramadass

a day ago

The point is that once you learn to visualize your analytic algebraic equations in 2D/3D through the lens of coordinate geometrical views (there are multiple perspectives here since you can have many coordinate systems) your intuition and understanding of the mappings has now been developed well enough that you can comfortably deal with higher dimensions in the abstract algebraic manner.

Imagination and Visualization are fundamental to learning Mathematics and Problem Solving. Any technique (both geometric and/or other graphical ones) which helps one to gain comprehension must be taught and shown how to map to algebraic notations. While mathematics has developed by continual abstraction and generalization the teaching of it should be from the concrete simple to abstract general.

godelski

a day ago

I mean I mostly agree but the part that I'm warning against is "you can comfortably deal with higher dimensions [visualizations]". You should always feel a bit uncomfortable in high dimensions least you be lulled into a false sense of security. In fact, this is frequently a pain point for people as they advance. They over leveraged the utility of visualization and hit a wall abstracting those same concepts to domains where you can't do any visualization. Just look at how many people claim to be able to visualize the 4th dimension and point at a tesseract.

I'm not disagreeing imagination and visualization aren't critical tools. They are. But read my comment again like a mathematician. If you misread you'll overfit the wrong concept.

rramadass

17 hours ago

You are just placing too much emphasis on caveats and limitations rather than focusing on the usefulness of the technique itself. Also techniques like Dimensionality Reduction exist to help tame higher dimensions.

None of your cautions are really that big of an impediment in practice and i would not want students to take away the wrong message from these comments. Mathematicians/Practitioners know when to switch from simple geometry to algebra and that is taught as a matter of course.

As an example, the farin and hansford book Practical Linear Algebra: A Geometry Toolbox that i mention at https://news.ycombinator.com/item?id=45060305 precisely takes this approach starting with visualizing vectors in 2D/3D geometry and then moving on to higher-dimensions. Its 1st edition was actually named The Geometry Toolbox: For Graphics and Modeling.

godelski

2 hours ago

I highly encourage you to talk to a mathematician about this stuff. In high dimensions everything is all about caveats and edge cases. Your {1,2,3}D intuition leads you astray rather than helps you closer to the answer. In fact, math is specifically an overly pedantic because nuances are so critical.

I'll use an example that we can reason from low dimensions but that many people make the wrong connections. What is the probability that in N dimensions two i.i.d randomly drawn vectors are orthogonal? We can easily intuit in low dimensions that this is a pretty unlikely event. But in high dimensions exactly the opposite is true. An easy mistake to make is to not think about this question from a continuous perspective, by reframing the question to ask what the probability that the angle between to i.i.d drawn vectors are within some epsilon bound. If we don't, then the answer is trivial in all cases: 0. A good student would be able to reason through this from looking at the 2D case and 3D case but this is only a sample of 2 and it would be naive to generalize such a notion without doing the proof.

So let's look at something a little harder. A classic problem is looking at the volume of a n-ball. Our 2D and 3D understanding make this problem look easy. The volume of the 3D ball is much larger than the 2D ball. We can even verify this my analytically solving the 4D case. Same with the 5D case! But that's when things go awry. For the unit n-ball, 5D is where the maximum volume is. By 10 dimensions you have less volume than our 2D case and by 20 dimensions your volume is quickly converging to 0.

Now let's look at something wild. Consider a hypersphere with radius r inscribed within a hypercube with sides of length 2r, what proportion of the hypersphere resides within the hypercube for a given dimension D? This sounds like an absolutely dumb question if we use or 2D and 3D logic. What amount of the sphere is inside the cube? 100%! It's right there in the problem description! Right? But this is entirely wrong. As dimensionality increases, very quickly, ~0% of the hypersphere resides within the hypercube and this is all due to the fact that hypercubes and hyperspheres have very different representations in high dimensions. Yes, there is still visual understanding we can draw to help reason, but again, this requires us to look at the problem from a different vantage point (the key thing I stressed in my original post btw). We have to look at the dual of the problem and instead ask "what is the ratio of the volume of a hypersphere to the volume of the hypercube it is inscribed within?" This will help but there's also a reason I put this question after the previous one because we know that there is not a clean linear relationship here and that at least with the sphere the volume increases and then decreases. Our visual intuition only helps if we can recognize that the relationship has everything to do with the corners of the cube.

It is easy to look at these 3 cases and see how visual intuition from low dimensions can help, but be careful about post hoc trivialization. It's easy when you know these things and after they have been told to you but they are not so clear when you're being presented with the problem. All 3 of these problems are deeply related to the nearest neighbor problem and the concentration of measure. Where we run into the problem that it becomes nearly impossible to distinguish the nearest point to the furthest point. Leveraging our lower dimensional visualization can provide help but it is important to remember that they are a small part of the much larger story here. Which of course should make sense because as the dimensionality grows our 2D/3D slices of those are a smaller and smaller portion.

This is why I referenced the blind man and the elephant. Not because you can't use visualization to aid you, but because to see the elephant there you have to sample that elephant at many different places. And just like the blind man, you don't know how big that elephant is and where things are changing.

But this stuff isn't going to help with even more abstract concepts. Like how there is no division algebra in 3D and how the largest is represented via the Octonion. How by moving to quaternions we lose commutativity and how moving to octonions we lose associativity. These are very fundamental features of math that we are losing while moving to higher dimensions.

So I'm not sure why you're pointing to Farin and Hansford's book. Their highly concentrated on low dimensions and only briefly discuss generalization at the very end. They miss many important concepts. Not because the book is poorly written but because they are far out of scope from what is being taught. My caution about high dimensional spaces does not mean I don't absolutely love Needham's books on Visual Differential Geometry and Forums or his book Visual Complex Analysis or even Carter's inspired Visual Group Theory. These are masterpieces and books I highly recommend. But it is about what is being communicated. We work frequently in lower dimensions and there is a wealth of information there. But this does not mean higher dimensions are not absolutely littered with pitfalls and paradoxes. The true study of high dimensional spaces is relatively new and drove people like Cantor and Boltzmann insane. Every small mistake that is easy to make in low dimensions becomes exponentially more important in high dimensions. Yes, there are dimensional reduction techniques and these help with varying degrees (utility being critically dependent upon the latent dimensionality of the data) but every single one of these comes with major concessions. There is often even very bad science performed due to this lack of understanding. Techniques like t-SNE and UMAP are routinely unknowingly abused to draw inaccurate conclusions about these spaces, as it is possible to perform nearly any clustering you wish (see Lior Pachter's Picasso). Even PCA is frequently abused in this area due to there being subtle assumptions being invalid on different problems.

My point is: you absolutely should warn people that there are many pitfalls when dealing with high dimensions. Anyone brushing these differences off as inconsequential nuances is just ignorant of this rich space. I don't blame anyone for being unaware of these things because it requires some very complex mathematics to truly understand, and we're talking about math that the average scientist is never going to be introduced to. You likely won't even see this in a given science PhD. Not because it doesn't matter, but because it is just very difficult math. And we learn math through a game of telephone that is often hyper focused (not a great way to learn it tbh)

RossBencina

a day ago

> Every student/beginner should study I.M.Gelfand's Functions and Graphs so that they can train themselves to imagine the graphs corresponding to a string of symbols.

Also I.M. Gelfand and A. Shen's "Algebra" for the strings of symbols.

rramadass

17 hours ago

Also his The Method of Coordinates and Trigonometry and Geometry to round out his set of five books aimed at high school students.

elcritch

2 days ago

Take some upper level physics courses. It's all about deriving equations and models rather than just using formulas.

coderenegade

2 days ago

Math is a language with many dialects, so you should treat it like learning a language in that you both have to start simple and build up, and use it. Take the time to learn what the symbols are saying for whatever dialect you are using.

It sounds like you're getting into geometric kernel programming for something like CAD. Go brush up on parametric equations, implicit vs explicit representations, and play around with simple spline examples. 2D is enough to get the idea. You'll need to know some basic calculus for this, as splines are usually constrained by the derivative at the node. You can get quite far with just Wikipedia, YouTube videos, some thinking, and some exploratory programming. Hell, throw AI in there as well.

hodgehog11

2 days ago

It's a bit difficult to say, since some of those textbooks (e.g. Shirley) cover preliminary math topics as well. It sounds like you need to get a stronger understanding of calculus and linear algebra, and approach maths from a healthier viewpoint. Something like Axler's Linear Algebra Done Right often forces people to treat linear algebra with the slow care it requires, but this may be a bit too extreme.

Now that you have some applications in mind, write down all of the terms and concepts that are involved that you don't understand. Go back to a first-year textbook, something like Stewart's Calculus is probably ideal since it is designed to work up to e.g. Bezier curves. However, this time, I would not suggest relying on the exercises for understanding. Instead, place strong emphasis on the definitions and theorems, drawing them together into a diagram with the terms from earlier to really understand why each concept is required for another concept. Your objective should be to see how these ideas lead to the definition of those curves, and why it has to be that way. Be skeptical and try thinking of alternatives; you might find them to have issues, or coincide with what has already been found! Make sure you understand each concept first before you apply it; this will not be a fast process, and if you need to check something, be sure to do it. But thinking in the context of what you want to learn later is often helpful, I find.

There are other, more detailed guides out there, mostly geared toward higher level math, e.g. https://www.susanrigetti.com/math and you might find the early parts of these useful too. Also, I have found chatbots to be reasonable at giving tailored explanations for a given topic, so there is rarely harm in using them to supplement your learning. Just don't use them to solve problems for you.

senderista

2 days ago

I love Axler, but it is absolutely the wrong book for someone only interested in the applications of linear algebra. (Strang is usually recommended in this case but I haven’t read it.)

mooder

2 days ago

Have you checked the curriculum of university program? Quick research shows MIT has stuff available. It might seem like a long shot but if you feel like you lack foundation knowledge, it might be worth looking over some of the material.

quag

2 days ago

It sounds like you've got something specific in mind when you say, "modeling". The term modeling is used in a lot of different situations to mean different things. For example, it could mean to make a 3d model in Blender, it could mean to pose for someone to paint you or to take a photo, with databases it's used to mean modeling the data, with statistics it's used to mean finding a way to simply represent and reason about the data (create a model of it).

The things you've listed out make me guess you want to write 2d or 3d image rendering software. Is that right?

If that's the case, there's no substitute for trying to recreate certain algorithms or curves using a language or tool that you're comfortable with. It'll help you build an intuition about how the mathematical object behaves and what problems it solves (and doesn't). All of these approaches were created to solve problems, understanding the theory of it doesn't quite get you there. If you don't have a good place to try out functions, I recommend https://thebookofshaders.com/05/ , https://www.desmos.com/calculator , or https://www.geogebra.org/calculator .

A good place to start is linear interpolation (lerp). It seems dead simple, but it's used extensively to blend two things together (say positions or colors) and the other things you listed are mostly fancier things built on top of linear interpolation.

https://en.wikipedia.org/wiki/Linear_interpolation

For bezier curves and surfaces here are some links I've collected over the years: https://ciechanow.ski/curves-and-surfaces/ https://pomax.github.io/bezierinfo/ https://blog.pkh.me/p/33-deconstructing-be%CC%81zier-curves.... http://www.joshbarczak.com/blog/?p=730 https://kynd.github.io/p5sketches/drawings.html https://raphlinus.github.io/graphics/curves/2019/12/23/flatt...

A final note: a lot of graphics math involves algebra. Algebra can be fun, but it also can be frustrating and tedious, particularly when you're working through something large and make a silly mistake and the result doesn't work. I suggest using sympy to rearrange equations or do substitutions and so on. It can seem like overkill but as soon as you save a few hours debugging it's worth it. It also does differentiation and integration for you along with simplifying equations.

https://docs.sympy.org/latest/tutorials/intro-tutorial/intro...

RossBencina

a day ago

For geometry, start with high-school algebra and coordinate geometry, and learn how to put the two together: equations representing geometry. By which I mean, being able to write down equations for geometric objects and manipulate the equations to get them into different useful forms, and to compute useful things. A first example would be studying different ways of writing down equations for the line [1], and then learn how each of the forms relate, how to use algebra to move between them, and what each for is useful for. For example finding the intersection of two lines involves simultaneously solving for (x,y) that satisfy both equations, and there's a particular form that is best suited to that. Another form is better if you want to "move along a line" say, parameterised by time. One form might be best for GPU rendering. Once you're comfortable with algebraic expressions for lines, then you can move on to the more complex curves that you listed -- the equations for these curves can be manipulated in analogous ways. EDIT: importantly you want to learn not how to "plug and chug" numbers, but instead how to use these equations in code: e.g. how would you use the equations to write code to draw a line/curve, and how to derive functions to intersect curves, connect curves at tangents, etc.

There's more to it than that of course. For example working out equations for tangents to curves will become important. But start with the above. You mention de casteljau algorithm, I am not familiar with that, but it looks like it might be heading more into algebra, in particular manipulating polynomials, which you learn to write down in high-school, study more in pre-calculus and calculus, but can get pretty complicated, for example if you want to solve a quintic equation.

Two elementary books:

McDougal, Little, "Geometry for Enjoyment and Challenge"

Joseph H. Kindle, "Theory and Problems of Plane and Solid Analytic Geometry (Shaum's outline)"

The following are approachable, but don't directly address your interest in curves:

Joseph O'Rourke, "Computational Geometry in C," second edition (basics everyone should know, starts with primitives like knowing the best way to compute the intersection of two line segments.

Christer Ericson, "Real-time collision detection" (a bit more specific, but a fantastic book, recommended to me here on HN in the past)

And of course I must reference Inigo Quilez' [2] "SDF of a ..." videos which give an illuminating perspective on using equations to describe geometry[3].

[1] https://math.libretexts.org/Bookshelves/Applied_Mathematics/...

[2] https://iquilezles.org/

[3] SDF of a Line https://www.youtube.com/watch?v=PMltMdi1Wzg

EDIT: some of the IRC servers have #math channels, and some of those are open to drop ins asking questions like yours (others seem to be full of people studying for exams).

rramadass

2 days ago

What you are trying to learn/understand falls under the rubric of Scientific Computing/Numerical Methods/Numerical Analysis/Numerical Algorithms. The mathematics underpinning them is quite wide but mostly Linear Algebra and Calculus. You might find the following useful for your study;

1) Scientific Computing by Michael Heath - Classic text covering a broad swath of domains and tries to build motivation/intuition before the mathematics (affordable Indian edition available).

2) Mathematical Principles for Scientific Computing and Visualization by Gerald Farin and Dianne Hansford - Nice overview of needed background. The authors also have a book named Practical Linear Algebra: A Geometry Toolbox which you might find a ideal companion.

3) Numerical Methods: Fundamentals and Applications by Rajesh Kumar Gupta - A relatively recent book with a really broad coverage of subjects and detailed mathematical expositions (affordable Indian edition available).

4) Numerical Algorithms: Methods for Computer Vision, Machine Learning, and Graphics by Justin Solomon - Good explanations (affordable Indian edition available). Free ebook available at https://people.csail.mit.edu/jsolomon/

5) Finally, Mathematics for Physicists: Introductory Concepts and Methods by Alexander Altland and Jan Von Delft is an excellent book to have as a reference. It has three sections viz. Linear Algebra, Calculus and Vector Calculus. The presentation is very precise, does not focus on proofs/lemmas but on concepts and covers a wide swath of important mathematics.