nostrebored
13 hours ago
No particular love for Sam Altman, but the article reads as “I was right about X, so my interpretation of Y is correct.”
I don’t see any particular contradiction. Moving fabs onshore is absolutely in the interest of both parties.
Jedd
13 hours ago
That wasn't my take - it was more broadly a 'given past statements and accompanying behaviour, we can't trust future statements to align with future behaviour'.
Can you more explicitly describe what the X and Y points you allude to are?
nostrebored
13 hours ago
Every “6 months ago I said this!” added little to the article. In a real investigative report it would lay out the facts and fairly objectively state “6 months ago, Sam Altman was accused of …”
The article as a whole just seems libelous? Almost personal?
chrisco255
12 hours ago
It's not a report, it's a blog post. Of course it's personal.
nostrebored
11 hours ago
You regularly make personal attacks against individuals on the internet?
Maybe I’m just old, but I don’t see the appeal?
If you’re trying to convince people, then you should probably have a convincing argument. Otherwise it feels like kiwifarms-posting with a megaphone.
chrisco255
8 hours ago
A blog is an opinion piece. The subject of interest, Sam Altman, is a public figure and CEO of one of the fastest growing tech companies. He's testified in front of Congress on AI regulation and has a lot of pull and influence on regulators. Some of the things and actions he's taken in the past are controversial, thus, thinkpieces get written. The AI industry is quickly en route to trillion dollar plus territory (already there if you count Nvidia as an AI company). There's a lot of money and emotions at stake for the AI gold rush. When someone is at the forefront of these types of things, like other public business figures with controversial tactics (Musk, Gates, Jobs, Kalanick, etc) it draws attention.
113
2 hours ago
> You regularly make personal attacks against individuals on the internet?
Isn't that basically the history of the internet?
dvfjsdhgfv
3 hours ago
Well, sama managed to convince a lot of people to give his company billions, is making apocalyptic predictions that some CEOs take seriously etc. Making sure people at large realize the guy has a very loose relationship with truth, for many years, seems like public service. It's only libel if you spread false statements which Marcus is careful not to make.
milowata
11 hours ago
Maybe you’re missing the context of who Gary Marcus is
samrus
13 hours ago
The crux of the article was altman trying to get a bailout, and when people called him on his bullshit, lying thay he never wanted a bailout. You cant trust the man
ineedasername
12 hours ago
It looks like the actual crux was Altman's plea that the backtops be granted... Not to OpenAI? The document linkedfrom the article that had the actual ask, rather than cleanup over deliberate misatributions by others, was to... "Extend eligibility", for the AMIC money already carved out, to companies that are producers of things such as "grid components such as transformers and specialized steel".
So: Altman did not ask for OpenAI loans to be guaranteed, nor did the CFO. It was on behalf of others drawn into the needs of the industry the AMIC grant was supposed to support. Self interested by OpenAI? Sure! And also not about to make the top 10k leaderboard for "sleezy things companies do".
samrus
16 minutes ago
You cant be this naive. You have to read between the lines. Altman has laready shown hes willing to boil us frogs with his attempts IPO a non profit. Hes slimy, he doesnt just say things and stand by them, he tests the waters and lies about ever wanting to swim if its too cold.
He called for general bailouts, which would benefit OAI far far more than others since its spending the most, and then he got backlash, he lied and said that OAI doesnt want bailouts at all.
You cant be so literalist to just take the man at his exact word and not try to determine his motivations. Especially this guy, whos made a career out of dancing around silicon valley's "no assholes" policy
mieses
7 hours ago
He might be fun to play a board game with but not in real life.
hiddencost
13 hours ago
Gary Marcus has been writing a piece like this roughly every 2 months for 20 years, and he gets a lot of attention because he seems respectable (he is not), and many papers are looking for a respectable source that has this opinion.
an0malous
13 hours ago
Why is he not respectable?
ants_everywhere
12 hours ago
Gary Marcus has a very particular view of human psychology that was popular for a while, especially in the early 2000s. Major proponents include Steven Pinker, Noam Chomsky, Jerry Fodor. This view was heavily influenced by symbolic computers when symbolic computers were new and so held some mental share leading into the dotcom boom. For several reasons, one of which is the replication crisis, this view is no longer nearly as popular.
One of the major beliefs of this view is that LLMs are essentially impossible because there's not enough information in language to learn it unless you have a special purpose language-learning module built into the brain by evolution. This is Chomsky's "poverty of the stimulus" argument.
Marcus still defends this view and because of this bias is committed to trying to prove to everyone that LLMs are not possible or at least that they're some kind of illusion. There's a sense in which they threaten his fundamental concept of how the brain works.
In proposing and defending these views he appears to me and others to be having a sort of internal crisis that's playing out publicly where his need to be right about this is coloring his judgment and objectivity.
pinnochio
12 hours ago
> LLMs are essentially impossible
> trying to prove to everyone that LLMs are not possible or at least that they're some kind of illusion
This is such poor phrasing I can't help but wonder if it was intentional. The argument is over whether LLMs are capable of AGI, not whether "LLMs are possible".
You also 100% do not have to buy into Chomsky's theory of the brain to believe LLMs won't achieve AGI.
ants_everywhere
12 hours ago
We weren't talking about AGI. It sounds like that's a topic you are interested in. It's a fine topic just not the one we're discussing.
And my phrasing was wonderful and perfect.
dvfjsdhgfv
3 hours ago
> And my phrasing was wonderful and perfect.
Do you use LLMs to post on HN? I'm asking seriously.
ants_everywhere
2 hours ago
Nope.
I would appreciate if you and the GP not personally insult me when you have a question though. You may feel that you know Marcus to be into one particular thing but some of us have been familiar with his work long before he pivoted to AI.
dvfjsdhgfv
3 hours ago
> LLMs are essentially impossible
This is the worst misrepresentation of Marcus' argument I've seen so far.
ants_everywhere
2 hours ago
Nothing in my comment is a misrepresentation. It may help to look up some of the words I mention in the comment if you're unsure what they mean.
Also Gary Marcus doesn't make an "argument" in the singular. He's a researcher with decades of public work and private work.
foldr
5 hours ago
> One of the major beliefs of this view is that LLMs are essentially impossible because there's not enough information in language to learn it unless you have a special purpose language-learning module built into the brain by evolution. This is Chomsky's "poverty of the stimulus" argument
The argument is that there is not enough information available to a child to do this. So even if we grant the dubious premise that LLMs have learned to speak languages in a manner analogous to humans, they are not a counterexample to Chomsky’s poverty of the stimulus argument because they have been trained on a vast array of linguistic data that is not available within a single human childhood.
If you want to better understand Chomsky’s position, it’s easiest to do so in relation to other animals. Why are other intelligent animals not able to learn human languages? The rather unsurprising answer, in Chomsky’s view, is that humans have a built-in linguistic capacity, rather in the way that e.g. bats have a built in capacity for echolocation. The claim that bats have a built-in capacity for echolocation is not refuted by the existence of sonar. Likewise, our ability to construct machines that mimic some aspects of human linguistic capacity does not automatically refute the hypothesis that this is a specific human capacity absent in other animals.
Imagine if sonar engineers were constantly shitting on chiropterologists because their silly theory of bats having evolved a capacity for echolocation has now been refuted by human-constructed sonar arrays. The argument makes so little sense that it’s difficult to even imagine the scenario. But the argument against Chomsky from LLMs doesn’t really make any more sense, on reflection.
Chomsky hasn’t helped his case in recent years by tacking his name on some dumb articles about LLMs that he didn’t actually write. (A warning to us all that retirement is a good idea.) So I don’t blame people who are excited about LLMs for seeing him as a bit of rube, but the supposed conflict between Chomsky and LLMs is entirely artificial. Chomsky is (was) trying to do cognitive science. People experimenting with LLMs are not, on the other hand, making any serious effort to study how humans acquire language, and so have very little of substance to argue with Chomsky about. They are just taking opportunistic pot shots at a Big Name because it’s a good way to get attention.
For the record, Chomsky himself has never made any very specific claims about a dedicated module in the brain or about the evolutionary origins of human linguistic capacity (except for some skeptical comments on it being a gradual adaptation).
ants_everywhere
2 hours ago
There was a large literature on language acquisition prior to the invention of LLMs that showed that Chomsky's argument likely wasn't correct. This is in addition to the fact that he significantly underestimated the amount of linguistic input children receive.
There's too much to hash it out here in HN. You can try to save the LAD argument by a strategic retreat, but it's been in retreat for decades now and keeps losing ground. It's clear that neural networks can learn the rules of grammar without specifically baking grammatical hierarchies into the network. You can retreat to saying it's about setting hyper parameters or priors but even the evidence for that is marginal.
There are certainly features of the brain that make language learning easier (such as size) but POS doesn't really provide anything to guide research and is mostly of historical interest now. It's a claim that something is impossible, which is a strong claim. And the evidence for it is poor. It's not clear it would have any adherents if it were proposed anew today. And this is all before LLMs enter the picture.
The research from neuroscience and learning theory and machine learning etc have all pointed toward a view of the brain as significantly different from the psychological nativism view. When many prominent results in the nativist camp failed to replicate during the replicability crisis, most big name researchers pivoted to other fields. Marcus is one of the remaining vocal holdouts for nativism. And his beliefs about AI align very closely with all the old debates about statistical learning models vs symbolic manipulation etc.
> Why are other intelligent animals not able to learn human languages?
Animals and plants do communicate with each other in structured ways. Animals can learn to communicate with humans. This is one of those areas where you can choose to try to see the continuities with communication or you can try to define a vision of language that isolates human language as completely set apart. I think human language is more like an outlier in complexity to the communication animals do rather than a fundamentally different thing. In that sense there's not much of a mystery given brain size, number of neurons, sociality etc.
> The argument is that there is not enough information available to a child to do this
Yes, but children are the humans who earn language in the typical case. So you can replace "child" with "human" especially with all the hedging I did in my first post (e.g. "essentially"). As I said above Chomsky is known to have underestimated the amount of input babies receive. Babies hear language from the moment they're born until they learn to speak. Also, as a parent, I often correct grammatical and other mistakes as toddlers learn to talk. Other parents do the same. Part of the POS is based on the premise that children don't get their grammar corrected often.
foldr
an hour ago
Yes, lots of people have argued that Chomsky is wrong about various things for various reasons and at various times. The point of my post was not to get into all of those historical arguments, but to point out that recent developments in LLMs are largely irrelevant. But I'll briefly respond to some of your broader points.
You mention 'neural networks' learning rules of grammar. Again, this is relevant to Chomsky's argument only to the extent that such devices do so on the basis of the kind of data available to a child. Here you implicitly reference a body of research that's largely non-existent. Where are the papers showing that neural networks can learn, say, ECP effects, ACD, restrictions on possible scope interpretations, etc. etc., on the basis of a realistic child linguistic corpus?
Your 'continuities' argument cuts both ways. There are continuities between human perception and bat perception and between bat communication and human communication; but we still can't echolocate, and bats still can't hold conversations. The specifics matter here. Is bat echolocation just a more complex variant of my very slight ability to sense whether I'm in an enclosed location or an outdoor space when I have my eyes closed? And is the explanation for why bats but not humans have this ability that bat cognition is just more sophisticated than human cognition? I'm sure neural networks can be trained to do echolocation too. Humans can train an artificial network to do echolocation, therefore it can't be a species-specific capacity of bats. << This seems like a terrible argument, no?
Poverty of the stimulus arguments don't really depend at all on the assumption that parents don't correct children, or that children ignore such corrections. If you look at specific examples of the kind of grammatical rules that tend to interest generative linguists (e.g. ACD, ECP effects, ...) then parents don't even know about any of these, and certainly aren't correcting their children on them.
Chomsky has never made any specific estimate of the 'amount' of input that babies receive, so he certainly can't be known to have underestimated it. Poverty of the stimulus arguments are at heart not quantitative but rather are based on the assumption that certain specific kinds of data are not likely to be available in a child's input. This assumption has been validated by experimental and corpus studies (e.g. https://sites.socsci.uci.edu/~lpearl/courses/readings/LidzWa...)
> Babies hear language from the moment they're born until they learn to speak
I can assure you that this insight is not lost on anyone who works on child language acquisition :)
ants_everywhere
5 minutes ago
A realistic child linguistic corpus for a 2 year old starting to form sentences would be about 15 million words over the course of their lifetime. Converted to LLM units that's maybe about 20 million tokens. There are small language models trained on sets that small.
Some LLMs are specifically trained on child-focused small corpora in the 10 million range, e.g. BabyLM: babylm.github.io.
Keep in mind that before age 2, children are using individual words and getting much richer feedback than LLMs are.
Humans can and do echolocate: https://en.wikipedia.org/wiki/Human_echolocation. There are also anatomical differences that are not cognitive that affect the abilities like echolocation. For example, the positioning and frequency response of sensors (e.g. ears) can affect echolocation performance.
pols45
12 hours ago
Cause he hasn't built anything or found a job since he sold his last company ages ago.
Supermancho
12 hours ago
Respect can be earned through many means, not just through goalposts set by financiers.
pols45
12 hours ago
Gary Marcus is not changing the story financiers are telling each other about AI. He has been telling the same story without making a dent in their stories. And that is because he is not in the arena. What he says, does and thinks doesn't matter.
anothernewdude
12 hours ago
So upon finding success, he has escaped the struggle that people are forced into against their will. How unreasonable of him.
nextworddev
13 hours ago
He’s playing the Nouriel Roubini game (another NYU adj prof).
Last time I checked Nouriel was partying up in the Hamptons so being a permabear is lucrative.