French is interesting because it evolved this so-called "dummy subject" pattern (which is actually called "subject-prominence") for an entirely different reason than English.

French is descended from Latin, which is "pro-drop" in the sense that you'd drop the "dummy subject" and simply write "pluit." Later, Old French, like English, lost most of its cases. Because of that, many word endings started sounding very similar (a problem known as "agreement syncretization"). The pro-drop "null" subject disappears in the record roughly as that happens, presumably to help disambiguate.

This is an entirely different path than English took, and it's also not a path any other Romance language took. Spanish isn't even subject-prominent. It fully allows the "null" construction, e.g., "llueve", which literally is just "rains."

Slovenian: Dežuje.

One word (rains) is the full sentence. Because nothing else needs to be said.

The sentence does have a very obvious subject. The environment is producing rain. The subject is the environment and its condition is being elaborated. It = environment.

The rain is not magical, the environment is producing it. The environment is raining. It's raining outside means the environment is producing rain outside.

Who could possibly confuse that as lacking a subject.

https://www.cambridge.org/core/journals/journal-of-linguisti...

Il pleut*

I guess the author means in contrast to some cases like "Pluvas" in Esperanto, see https://www.youtube.com/watch?v=fOBkKcbJUAE for a nice employ of it.

Italian, Spanish and Latin (for example) can generally avoid to express explicitly the subject in a dedicated pronoun, as the verb already encode the it through specific flection. In contrast in French and German only in imperative tense one can formally make such an ellipsis (colloquially of course one can encounter more nuances).

So nothing exceptionnal

Yes, as I mentioned to another commenter, I'm talking about the original formulation because that's what the article talks about. We have more options now, although (as you seem to be suggesting) they do all necessarily model position. They have to—there is no other way to do things like recover the subject in an English sentence without positional information.

This has been addressed several times now in the peer comments to this one. I am addressing the original formulation of the problem because the article is about the original formulation. There are other options available now, including causal masking, but in the original formulation positional encodings are THE most important single ingredient in success, because many aspects of semantics cannot be recovered from anything except position. One example is the subject in an English sentence.

They don't really have that many different characters, it's just that the way Chinese script was adapted to computers use a rigid set of that many pre-composed characters.

To clarify, I'm talking about the original formulation of the problem, where positional encodings are crucial, because the article is talking about the original formulation of the problem.

But these days, there are many more options. They all maintain this positional information in one way or another. The one you cite (as I understand it) drops the positional encodings after pretraining, and keeps (e.g.) the causal mask.

It's hard to speak for every language on Earth, but generally languages that have less flexible word orderings will have correspondingly more complicated morphology. In those cases, order does obviously still matter, but instead of in word orderings, it would be in things like prefixes and suffixes.

There isn't really a way to get around this. Natural language IS structured as a recursive tree. Our job is to extract this tree from a flat stream of tokens (or similar), and straightforwardly the only way to do that is to make the ordering of the tokens meaningful. There is no other option! This is how 100% of parsing works in any context whatsoever, both theoretically and practically.

The problem lies more in #2 than #1. It's fine to repurpose words in new domains, but there's zero guarantee that translates back into anything useful about the original domain.

As a former perceptual neuroscientist, I can assure you that ML "attention" has very little to do with biological attention.

I'm skeptical it requires embracing new meanings, it only requires being unbiased (non human centric in thinking). If an organic looking alien landed on Earth, circa 1950, with the capabilities of whatever the cutting edge version of eg Mythos is, humanity at the time would have had absolutely zero problem recognizing it as de facto intelligent, even if it only responded when prompted during interactions. People would have merely categorized it as a different type of intelligence. Instead of embracing that intelligence can come in many forms, as sci-fi has proposed about the appearance of alien creatures for many decades, when confronted with the arrival of another type of intelligence, humanity becomes cowardly as it always does when its perceived specialness is challenged.

People are doing what people do: they're desperately moving the goal post to keep pretending humanity is particularly special and that intelligence is particularly special. Unique snowflakes and all that, we're the center of the universe. It's nothing more than elaborate human weakness & fragility on display.

We're in the first inning of this, intelligence is not going to end up being special, it's going to be trivial and low cost. See: Chinese models ten years from now.

Astra, Opus, Fable, Argon - they're all radically beyond the top 1% of humanity intellectually already, and it's going to get a lot worse quickly. The sole aspect the top 1% beat them on, and increasingly rarely, is at hyper specialization, where a person is elite at what they do. For $20-$100-$200 per month I can get access to an omnimath genius intellect far beyond any human that has ever existed, and that AI will work for me like a dog. Humanity should be scared, and the reason HN hates AI so much, is because it will wipe out very large numbers of their lucrative jobs that they imagined would form long-term careers (six figures, stock options, path to wealth, elitism re competition).

It's misleading. When it's right, it's right by coincidence, since the vast majority of algorithmic terms that seem like they should provide insight into the human mind, like "attention" could just as easily be called "finger" or "tape head".

And suddenly we're looking at people who read ticker tape or by following with their finger for insights into neuroscience?

It's bad metaphysics. If you really want to do bad metaphysics, then you can, but it's also hardly an attack on you if others promote norms of not doing bad metaphysics.

Ah neat, is that related to k-nearest neighbor imputation too?

https://scikit-learn.org/stable/modules/generated/sklearn.im...

https://scikit-learn.org/stable/modules/generated/sklearn.me...

Aren't all of these (including convolution) specific instances of Graph Neural Networks?

I don't have any specific experience in ML but I always liked the formalism presented in https://www.youtube.com/watch?v=J1YCdVogd14 as a way to unify everything

The crucial question, however, is "Where do the weights come from?" :-)

btw, "kernel smoothing" etc. are still fairly complex terms that sound a bit like math magic: 1-nearest-neighbor (1-nn), k-nearest-neighbor (knn), and weighted knn would be even simpler and possibly more descriptive.

Thanks for writing that out, so I didn't have to do the same but less eloquently :)

Attention is not based on Euclidean closeness, therefore it is not kernel smoothing. Saying otherwise would be the same as saying kernel smoothing is just an average (e.g. omitting the important part that it has to be distance-based-weighted).

This was unexpectedly good.

Now if you could please explain the rest of math in similarly simple terms?

Why are they calling it 'attention' if it's just a weighted average? Human attention is something very different.

You are just too tired this morning (hit with tires)

It means generated output is based on mathematical relations of vectors that represent words or part of words. I believe pre-language cavemen are intelligent with the caveat that transfer of knowledge is severely reduced. I am even tempted to say that cats and dogs are more intelligent than LLMs.

To me it merely signals that there are certain mathematical relationships in our language that can be deduced (by neural networks). Nothing more.

The whole article is about just that : the LLM has to know what an "animal" and a "street" is to perform effectively.

But then dumbly concludes : oh is language intelligence ? ("Perhaps our form of intelligence is tantamount to language modeling, too?"). C'mon.