Sorbet: A neuromorphic hardware-compatible transformer-based spiking model

57 pointsposted a year ago
by PaulHoule
(arxiv.org)

54 Comments

magicalhippo

a year ago

I found this[1] article to give a nice overview over spiking neural networks and their connections to the more "traditional" neural networks of modern fame.

In particular the connection between the typical weighted-sum plus activation function and a simplistic spiking model where one considers the output simply by the spiking rate was illuminating (section 3).

[1]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9313413/ Spiking Neural Networks and Their Applications: A Review

satvikpendem

a year ago

I wonder how well this model can typecheck Ruby code.

thih9

a year ago

Context: Sorbet is also the name of a popular Ruby type checker[1], built by Stripe.

[1]: https://sorbet.org

Alifatisk

a year ago

Can’t wait for the follow up paper, RBS

krasin

a year ago

There's no code or weights released => no way to reproduce their results.

ajb

a year ago

They may not have linked the source in the paper, but it's not hard to find: https://github.com/Kaiwen-Tang/Sorbet

Haven't checked if there's enough there to build it.

krasin

a year ago

Thank you very much for posting this! The code is indeed there, that's great.

sanxiyn

a year ago

Results get reproduced without code or weights all the time. I note that in this case training data and evaluation benchmarks are public.

hiddencost

a year ago

The ML community has historically held themselves to a higher standard, IME.

evanwolf

a year ago

sometimes it seems folks are just making up words.

a-dub

a year ago

neuromorphic hardware is just hardware that has biologically inspired designs.

spiking neural networks are artificial neural networks that actually simulate the dynamics of spiking neurons. rather than sums, ramps and squashing, they simulate actual spike trains and the integration of energy that occurs in the dendrites.

neuromorphic hardware can range from specialized asics for doing these simulations efficiently to more experimental hybrid analog-digital systems that use analog elements to do more of the computation.

it's all very cool stuff, but i tend to think of snns as similar to the wings on the avion 3 where simplified unit functions look more like a modern jet wing.

but who knows, maybe the neuromorphic route will open the door to far more efficient computations. personally, i'm very excited about potential wins that could come from novel computational substrates!

ImHereToVote

a year ago

I wonder how far we will move the goalposts once we have a multimodal transformer type model running on neuromorphic hardware.

welferkj

a year ago

Lots of people involved in explaining away AI are labouring under the axiom that intelligence is mysterious. Therefore, if I can understand how a system works, it logically follows that it can't be intelligent.

lucubratory

a year ago

I predict that many of those people will continue to believe that up until human cognition is mechanistically understood, at which point there will be some other reason that humans are "real" thinkers and machines are not. The problem is that theoretical opposition to the existence of AIs is incompatible with materialism and thus just doesn't fit with our world, which is very much built using the scientific truths that materialism enables us to discover.

welferkj

a year ago

It is insane to me that views of consciousness and cognition other than physicalism still exist in mainstream scientific and philosophical discourse. As far as I can tell, no matter how much discourse you dress it up in, any alternative boils down to "it's magic, I ain't gotta explain shit".

dartos

a year ago

We… can’t really understand how neural networks work, but we can definitely tell they’re not intelligent beyond making good sounding word soup (as demonstrated in their minimal practical reasoning abilities)

I wouldn’t call pagerank intelligent, even though I can give it a text prompt and get relevant information back.

In my view, the only difference between that and an llm is the natural language interface.

I’m no expert on intelligence, but I’d expect being able to introspect and continually learn to be part of it.

welferkj

a year ago

You're engaging in explaining away intelligence.

One way to help you notice this is to try and estimate how many billions of people you've defined out of "being intelligent" with your latest goalpost movement.

Be honest, how many people do you think "introspect and continually learn" on a daily basis?

cootsnuck

a year ago

> Be honest, how many people do you think "introspect and continually learn" on a daily basis?

That's wild if you think that isn't quite literally one of the defining features of human consciousness (and many would say other animals as well).

If you think people thinking differently than you means they don't still indeed...think...then I don't know what tell you.

welferkj

a year ago

Unfortunately for the intelligence denial crowd, introspection and learning capability is something we can measure, as opposed to the vibes-based discourse you prefer to engage in. If that's what you've picked for your threshold of "intelligence", you've reduced the majority of the bell curve's left side to soulless automatons. Again, your definition, not mine.

dartos

a year ago

My definition (again, as a layman on the subject of intelligence philosophy), but your incorrect analysis.

I guess I just think more highly of my fellow humans.

exe34

a year ago

> I guess I just think more highly of my fellow humans.

As an article of faith, yes. But I don't see what this adds to the discussion.

dartos

a year ago

> If that's what you've picked for your threshold of "intelligence", you've reduced the majority of the bell curve's left side to soulless automatons

I disagree with this statement, which is the crux of their argument against my definition of intelligence.

I don’t think any credible survey of the intelligence or lack of a large enough population exists (due to there not being a common binary measure of intelligence), so it’s an issue you kind of need to take on faith.

ImHereToVote

a year ago

>so it’s an issue you kind of need to take on faith.

Thanks for playing.

dartos

a year ago

It cuts both ways… you need to believe that most humans aren’t actually intelligent as we don’t have any data that suggests that most humans aren’t.

exe34

a year ago

you have no data that suggests I can't fly either.

dartos

a year ago

… but… we do have data that proves that humans can’t just fly?

I think… I’m done talking with you now.

exe34

a year ago

you just like your assumptions to go unchallenged. goodbye!

dartos

a year ago

It’s crazy to me that people would rather believe that we can create intelligence by feeding the text of the internet into a statistics machine, than believe that the people making that text are intelligent.

welferkj

a year ago

It's crazy to me people would rather deny the intelligence of a large segment of the human population than admit to the increasing overlap between it and AI.

That's what you're doing when you keep moving the goalposts of "real intelligence" further and further right on the bell curve. You're denying the intelligence and consciousness of billions of people (and counting) just so you don't have to admit there's nothing magical about intelligence.

Sometime in the next 10 years, you'll have to start thinking of yourself as a soulless automaton to keep up the delusion. Good luck with that.

dartos

a year ago

> It's crazy to me people would rather deny the intelligence of a large segment of the human population than admit to the increasing overlap between it and AI.

You’re the one here denying. I think the vast majority (if not all) of humans are intelligent under my definition. You do not.

I don’t think LLMs or other statistical models are.

welferkj

a year ago

Under the latest definition you made up on the spot, yes. And definitely not all.

So what's your plan when the fraction keeps shrinking? When you're no longer in it?

This is simple interpolation. It is plainly obvious that at some point soon, you will be faced with the fact that there's nothing magical about intelligence. When that happens, will you concede that, or start thinking of yourself as a soulless automaton?

If you can't project that far forward, I question whether you meet any meaningful definition of "intelligent" right now.

dartos

a year ago

> So what's your plan when the fraction keeps shrinking?

What fraction? How would it shrink?

I don’t think that humans, as a species, are becoming non intelligent en masse. In fact, I think that we are, by default, intelligent.

That’s where our opinions seem to irreconcilably differ.

> you will be faced with the fact that there's nothing magical about intelligence.

I dont think there’s anything “magical” about anything. I just don’t think that a statistical model can achieve intelligence as we think of it with regards to humans.

You may see the recent trend of text generation models as new intelligent machines, but I’ve been studying and working these kinds of statistical models for about a decade (since 2016) and have seen these opinions spouted only to quiet down once the logarithmic improvement curve is reached. I don’t see any reason why these LLMs wouldn’t follow the same pattern.

> This is simple interpolation

Interpolation of what? You’re assuming that the goalpost will always shift, but in reality we just don’t have a generally agreed upon definition all. Either way, any definition of intelligence that rules out the majority of humanity is an incorrect definition off the bat, as pretty much all humans are intelligence.

There exists some accurate definition of intelligent such that almost any human satisfies it, but statistical models do not. I’m sure if I studied the philosophy of intelligent I could put one into words, but I’m ill equipped to do so.

> If you can't project that far forward, I question whether you meet any meaningful definition of "intelligent" right now.

Are you just trying to be mean, or do you actually believe that people who disagree with you are not intelligent?

We’ll see in 5 years that this intelligence hype will fade just like that last 2 AI booms.

This isn’t at all to say that we will never make a machine with intelligence that rivals humans, just that I don’t think the statistical model route will get us there… and it hasn’t.

dartos

a year ago

> how many people do you think "introspect and continually learn" on a daily basis?

At the very least, every single person who plays sports, video games, tries finding a way around traffic, a faster route home, a way to do less work, take a longer break, or a way to save some extra money getting food.

Literally any optimization task at all requires an observation, analysis (read: introspection,) and adjustment. That’s why we model training loops as optimization problems.

We spoof that with REACT prompts in LLMs, but it becomes clear after a few iterations that there’s no real optimization going on, just guessing at tokens (a gross oversimplification, as this guessing has real uses). It’s doing what it was trained to do, completes text. Not to mention that those steps all disappear when the prompt is changed.

exe34

a year ago

> One way to help you notice this is to try and estimate how many billions of people you've defined out of "being intelligent" with your latest goalpost movement.

love this, I will use this in future rants.

dartos

a year ago

That argument only works if your audience already thinks of humans as mostly automotons…

exe34

a year ago

It's an operational definition: if you claim AI is not intelligent because it cannot do X, then you necessarily exclude a whole lot of humans who also can't do X.

There used to be a strident faction that would say "but AI can't produce original art/a symphony/novel/etc". My answer was usually (correctly), "neither can you."

dartos

a year ago

> It's an operational definition: if you claim AI is not intelligent because it cannot do X, then you necessarily exclude a whole lot of humans who also can't do X.

Sure, but I think most people are intelligent according to my definition, but AI is not…

You’re already coming from the assumption that people are “souless automatons,” which is probably why the idea of a machine being “intelligent” is so easy for you to accept.

> There used to be a strident faction that would say "but AI can't produce original art/a symphony/novel/etc". My answer was usually (correctly), "neither can you."

This is a dumb apples and oranges comparison. AI as a concept is different than a concrete person.

AI as a concept can do anything, it’s a conceptual placeholder for an everything machine.

dartos

a year ago

I can’t reply to the other comment, but “soulless” was to quote the other commenter. Having a soul (whatever that might mean) holds no bearing on what I’m saying.

exe34

a year ago

again, do they obey the laws of physics? can they decide to go against what the physical interactions in their brain guide them to do?

user

a year ago

[deleted]

exe34

a year ago

> You’re already coming from the assumption that people are “souless automatons

do your people obey the laws of physics? is the soul magical or physical?

joloooo

a year ago

I think there is a difference from people upset around over hyped LLMs and arguing about intelligence in "A.I.". Most of the "intelligence" arguments I've seen are fighting against putting too much stock in chatgpt and Sam's fever dreams.

a-dub

a year ago

the goalposts for what?

ImHereToVote

a year ago

Literal goalpost in a game of football of course. Or soccer if you are an American.

a-dub

a year ago

thanks. you seem to think that a spiking multimodal variant of transformers on neuromorphic hardware would demarcate a goal of some sort, which one?

for as far as i can see, the achievement would just be a spiking multimodel variant of transformers on neuromorphic hardware.

ImHereToVote

a year ago

I bet you are great at playing blackjack, but suck at Texas hold 'em.

user

a year ago

[deleted]

ithkuil

a year ago

To be fair, all words are made up.

Words are useful to the extent they effectively communicate with the intended audience.

This can be accomplished by a mix of familiarity (has this word been already used enough in the target audience with the intended meaning) and the ability to evoke new meanings by intuitive derivation rules (word composition, affixes, ...)

In the case of this title, fwiw, it was perfectly clear to me what this was about because I'm already familiar with related topics and they were using the same terminology

allendave6945

a year ago

Hello, as a newbie to cryptocurrency trading, I lost a lot of money trying to navigate the market on my own, then in my search for a genuine and trusted trader/broker, i came across Trader Bernie Doran who guided and helped me retrieve my lost cryptocurrencies and I made so much profit up to the tune of $60,000. I made my first investment with $2,000 and got a ROI profit of $25,000 in less than 2 week. You can contact this expert trader Mr Bernie Doran via Gmail : BERNIEDORANSIGNALS@GMAIL.COM or WhatsApp +14242850682 and be ready to share your experience , tell him I referred you

remon

a year ago

I definitely know what "A" and "model" means.