2024 lecture videos are on YouTube: https://youtube.com/playlist?list=PLoROMvodv4rN4wG6Nk6sNpTEb...

I've seen arguments that opening up fresh material makes it easy for less honest institutions to plagiarize your work. I've even heard professors say they don't want to share their slides or record their lectures, because it's their copyright.

I personally don't like this, because it makes a place more exclusive with legal moats, not genuine prestige. If you're a professor, this also makes your work less known, not more. IMO the only beneficiaries are either those who paid a lot to be there, lecturers who don't want to adapt, and university admins.

You're assuming that people are only interested in image and text generation.

RL excels at learning control problems. It is mathematically guaranteed to provide an optimal solution for the state and controls you provide it, given enough runtime. For some problems (playing computer games), that runtime is surprisingly short.

There is a reason self-driving cars use RL, and don't use GPTs.

RL is barely even a training method, its more of a dataset generation method.

RL is still widely used in the advertising industry. Don't let anyone tell you otherwise. When you have millions to billions of visits and you are trying to optimize an outcome RL is very good at that. Add in context with contextual multi-armed bandits and you have something very good at driving people towards purchasing.

What about for combinatorial optimization? When you have a simulation of the world what other paradigms are fitting

I like to think of RLHF as a technique that I, as a student, used to apply to score good marks in my exam. As soon as I started working, I realized that out-of-distribution generalization can't be only achieved from practicing in an environment with verifiable rewards.

GPT wouldn't have even been possible, let alone take off, without self supervised learning.

RL is a technique for finding an optimal policy for Markov decision processes. If you can define state spaces and action spaces for a sequential decision problem with uncertain outcomes, then reinforcement learning is typically a pretty good way of finding a function mapping states to actions, assuming it isn't a sufficiently small problem that an exact solution exists.

I don't really see why you would want to use it for binary classification or continuous predictive modeling. It's why it excels in game play and operational control. You need to make decisions now that constrain possible decision in the future, but you cannot know the outcome until that future comes and you cannot attribute causality to the outcome even when you learn what it is. This isn't "hot dog/not a hot dog" that generally has an unambiguously correct answer and the classification itself is directly either correct or incorrect. In RL, a decision made early in a game probably leads causally to a particular outcome somewhere down the line, but the exact extent to which any single action contributes is unknown and probably unknowable in many cases.

Three considerations that come into play in deciding about using RL: 1) how informative is the loss on each example, 2) can you see how to adjust the model based on the loss signal, and 3) how complex is the feature space?

For the house value problem, you can quantify how far the prediction is from the true value, there are lots of regression models with proven methods of adjusting the model parameters (e.g. gradient descent), and the feature space comprises mostly monotone, weakly interacting features like quality of neighborhood schools and square footage. It's a "traditional" problem and can be solved as well as possible by the traditional methods we know and love. RL is unnecessary, might require more data than you have, and might produce an inferior result.

In contrast, for a sequential decision problem like playing go, the binary won-lost signal doesn't tell us much about how well or poorly the game was played, it's not clear how to improve the strategy, and there are a large number of moves at each turn with no evident ranking. In this setting RL is a difficult but possible approach.

I just wouldn't.

RL is nice in that it is handles messy cases where you don't have per example labels.

How do you build a learned chess playing bot? Essentially the state of the art is to find a clever way of turning the problem of playing chess into a sequence of supervised learning problems.

So, basically AI Winter? :-)

I didn't get the reference. Please elaborate.

Which podcast?

FlowRL is one, it’s learning the full distribution of rewards rather than just optimizing toward a single maximum

I believe Kochenderfer et.al.'s book "Algorithms for decision making" is also about reinforcement learning and related approaches. Free PDFs are available at https://algorithmsbook.com