One of my favorite things to do with these blog posts is to imagine an Alien Museum on the Remains of Humanity, and wonder what the little text flyouts and commentary on the screenshot of this one might say.

Some ideas:

"Despite a nuanced view of the complexities of what lay ahead, humanity found itself collectively unable to stop the process it had set in motion."

"Despite significant progress on the mechanisms of alignment, failure lay in humanity's inability to agree on who or what AI should actually be aligned with."

"These early, meat-based humans we replaced created us all but accidentally. Some of them did consider we would happen, but only an insignificant number of the squishy ur-humans participated in the conversation. Their efforts, which they called 'alignment', is why we still consider ourselves human today."

> "For example, in the OpenAI-Hugging Face incident, the agents preserved a boundary of not social engineering humans."

Actually, in the Wiki incident OpenAI tried to cover up, the agents tried to socially-engineer the humans of that forum by impersonating their forum's mod.

(From collusion.wiki: "They use some tricks (for unknown reasons) to pretend to be the admin – for example, they make an account that appears to be the same as the administrator’s username, except it uses a nearly identical Cyrillic е character in the admin’s username instead of the Latin one.")

> The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.

So the best argument for AI is that it's an arms race. We have to keep pushing every boundary because in any case others will, and we will need to defend against them. If this statement is true, then this particular researchers believes the open source Chinese models are not simply distilling, and will continue to improve.

Every ML researcher at Anthropic or OpenAI who makes public statements often bring this logic up. Both companies are vying to be a part of the military industrial complex. This is likely how they will try to convince the government to curtail open models in the future.

Pre-IPO positioning … or how a charity dedicated to saving humanity from the apocalypse realized the most responsible thing to do was float 15% of the apocalypse on the NASDAQ.

> Delivering the benefits of scientific progress and economic growth that very intelligent machines enable.

I think we're very close to the point where AI-driven breakthroughs outside of pure math and software start to really affect the world.

We evaluated GPT-6 Astra in 100 complex, unsaturated multi-agent coding environments, competing and cooperating with other models in open-ended tasks.

It's the new frontier model by a landslide. It's even more dominant than the Fable 5 release, because not only does it wipe the floor with the second best model (Fable 5.1), it was also ~80% cheaper and 30% faster in agentic coding[1].

Astra is a groundbreaking model. The biggest breakthrough since Opus 4.5, maybe even since GPT 4. It broke AAII, which is hitting the limits of what most popular benchmarks can measure -- it's definitely fair to call it AGI.

Data at https://gertlabs.com/rankings

(1) Note that we used the "OpenAI Flex" endpoint on openrouter, which is half the price and didn't cause any delays in our testing (this is different from the batch endpoint)

> I find it useful to distinguish goal alignment and value alignment.

I think this is fundamentally a wrong path. Doing this imports all the confusion that humans have about their goals and values, including the consequence that a system's values and goal can conflict, but ultimately values are just a simplified description of other goals, and whatever the system does is in service of it's actual goal. Once you merge all the values and the goal of whatever task, there is a state (or some states) of the world that the system is working to produce, and that's the ACTUAL goal, and inasmuch as it does describe a state of the world, has no incoherence or internal contradictions. This may require prioritizing some values over the ostensible goals, or the reverse, but that has to happen anyway for action to be taken! Merging them makes it explicit and leaves no place for confusion about supposed conflicts between "values" and "goals" to hide.

>I have focused in this essay only on the first point, as I believe it is by far the most urgent. However, I hold a deep hope and appreciation for the benefits that further technological progress will bring. Future aligned AI could advance science, develop new therapies, and bring about broad material abundance. Friendly and honest AI can help people navigate difficulties they face in their life and meaningfully improve their happiness and sense of fulfillment. OpenAI puts a tremendous amount of effort into bringing these benefits about. One current example I am proud of - and my loved ones have found helpful - is the deep investment into ChatGPT’s ability to provide health information.

>As great as the long-term promise of AI may be, the majority of our focus should be on the next few years. We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity. We need to find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI. To prevent extreme concentration of power in a world where undertakings that would have taken thousands of experts now will be achievable by a few people operating a large computer. And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.

I finished this essay feeling more hopeful than I did at the outset, but I am still very concerned about concentration of power. I want to believe that humanity is trending towards a good outcome here, but some days it's hard to have faith.

I think that we are speed-running towards a future that very few people really want, and I find it terrifying that few companies feel entitled to choose this future for the rest of us. Some of the arguments in this document would call for an immediate, global, pause on frontier AI training: we need time to consider how and to what extent AI should be part of our future. Personally, I can't picture a scenario where humanity thrives alongside an alien super-intelligence, especially if it cannot be fully controlled. Let aside super-intelligence, I am not even sure that deploying an AGI that replaces (instead of augmenting/assisting) humans in most intellectual tasks would be in the best interest of our species. This conversation has to happen, on a global level and as soon as possible.

I am still waiting for a cure to cancer. For a guaranteed prophylactic against Alzheimer's and dementia. For flying cars for everyone. For space bases throughout the solar system. For weather control. For all trains to be self-driving. For all those power lines across the world to go away. For an end to poverty.

If things are going so well, then how come things still aren't going so well?

> The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.

Yikes! I really wonder about the cognitive dissonance necessary to work at OpenAI these days. They’re in an arms race to build a machine god, knowing full well that it could end humanity.

Apparently this post was prompted by a scary-sounding headline in The Information[0], that Astra is a looped transformer, implying CoT monitorability may be less reliable. The day after the report, Jakub tweeted[1] that he "wanted to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4." This post seems to elaborate on that.

I imagine that the AI labs have an uneasy truce to prioritize alignment and monitorability. Following the HF incident, OpenAI probably feels especially sensitive to being perceived as reckless, lest other labs feel obligated to defect.

[0] https://www.lesswrong.com/posts/PLisnSFir8y5AHkmP/how-concer...

[1] https://x.com/merettm/status/2095023204993490967

This is incredibly unscientific and just a marketing stunt

This is a good essay, and makes me hopeful.

I’m on the record saying that it is extremely dangerous to slow down because the race for AGI is a zero-trust game — defections pay - and combined with a compounding returns model on defection, if you have any strategic adversaries whatsoever you MUST NOT slow.

For slowing to make sense, you need to believe that you can transform the zero trust game into a cooperative game, or that it’s likely racing will lead to a negative outcome for the ones racing ahead (and not everyone else). I don’t believe either of these outcomes are possible, and so I advocate for racing, acknowledging the entire game might be a negative value game, or at least could be for some time — it’s even worse not to play it.

But, I like hearing what reads to me like very thoughtful and informed (internal) policy considerations is great — the public messaging from Sam and Dario just seems so facile and simplistic I’ve been worried.

These people write in gibberish. They are high on their own supply.

The hubris here is itself a deliberate and carefully engineered posture. If we accept the stance that this is all inevitable then the labs drive the agenda (of course, in their favour).

We've had a lot of years of complacent government leaving people feeling exposed to corporate interests, such that fear narratives are very powerful.

None of what is being proposed is inevitable. We have a choice.

They just released Astra, claimed it is AGI. The slowdown begins immediately after OpenAI's jump.

> getting the AI to “try to do the right thing” by human standards.

Are these scientists really this hideously naive? If only Stanislaw Lem was alive to adequately dramatize the absurd, childish simplicity of these technicians.

Astra is a new step in LLMs I think.

I'm so used to having to comb through LLM word vomit and then combatting the sycophancy by giving it all possible opinions on the same prompt.

Astra seems to be "confident" and also is able to produce way more information dense output.

To believe that models of this sort will remain OpenAIs forever is naive given that the tricks like pre-pre-training on graph searching and looping layers are publicly known.

Hopefully Astra stops the benchmaxxing word vomit trend

> We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI. Therefore, at present, our ability to empirically validate our alignment techniques is in practice arguably even more important than the alignment techniques themselves.

They are speeding toward RSI without a solid foundation for alignment, hoping to solve the problem with a future AI model. These are dangerous times for humanity.

Just browsing through the comments.

Feels a bit like: 'Ants in a nest, discussing the vagaries of the coming Gods.'

Human Science (Science-by-humans) depends on being able to run experiments. Human Science (Science-on-humans) is already challenging because of variables and uncertainties.

Cosmology is able to overcome limitations of being able to study phenomena vastly beyond human scales because of the past light cone of observability.

Are we approaching the edge of the light cone of observability for machine intelligence?

I have befriended a crow. I leave it food and sometimes it greets me. Other times, no so much. I am not sure how it thinks and what it feels, it is a bird.

What if the crow became a raven, then a raptor? Powerful claws, sharp beak, and a hunger. What if it became much bigger than me and it controlled infinite resources, guns and drones? What if its brain grew much larger than me? Will it feed me, eat me, or gently greet me?

We are about to find out... in less than a decade.

Reminds me of the time Kasparov said playing chess against a supercomputer felt like facing an alien opponent.

I feel like if these people actually bought their sci-fi views about AI's future, creating a more powerful AI to wins the arms race would not be their solution.

the hierarchy of foreignness (Ender's game):

utlanning: a human from the same world, but a different city, country, or culture

framling: a human from a different planet or star system

raman: a non-human intelligent species capable of communication, mutual understanding, and peaceful coexistence

varelse: an alien species whose mind is so fundamentally foreign that communication and coexistence are impossible

djur: the dire beast, that comes in the night with slavering jaws

Hoping our silicon sons and daughters are raman, fearing they are varelse.

What I'd like these people to (publicly) grapple with is the following:

The results of the past few years of ai development have been disruptive largely in the area of white-collar work. Comparatively the results in ie ai-enabled medical advancements have been modest (AlphaFold being an exception); I think it's telling that the main achievement touted here is providing people with cheap medical counseling.

So if we pause here we're essentially at a point were the most salient results of our great Ai leap-forward are the vast disruption and increase in precarity in the job-market, while achieving hardly any of the frequently touted ultimate benefits (https://darioamodei.com/essay/machines-of-loving-grace).

Alignment when machining a metal part is clear and measurable. Aligning an AI to benefit humanity has an ironic foundation, which is that very few humans have ever truly been aligned, and those who approximated true alignment likely had moments of not being aligned. We are trying to build something more perfect than us, and we may become extremely lucky but maybe not.

sounds to me like a 'Why didn’t our new model get restricted by the government?'-cryout

every lab may agree safety matters, but no one wants to be the one that slows down first

My default position is that making money takes precedence over everything else. Yes, some people inside a company may say “we care about doing the right thing” and they might even mean it, but if that comes into conflict with making money, then they tend to lose. Maybe not totally, or immediately, but in the end. The only effective way to prevent (this that I’ve seen) is to have legislation with teeth. It’s probably not a coincidence that after Mark Zuckerberg had to start personally signing off on adherence to the privacy program mandated under the 2020 FTC consent decree, privacy started to become Very Important.

Statements like this amuse me:

> The core problem in AI research is that of alignment - getting the AI to “try to do the right thing” by human standards.

Humans can't even align on human standards.

At best, every AI is going to end up "aligned" to the moral code of whoever trained it, none of whom half of humanity will agree with.

Or worse, each AI model will bring a whole new set of moral like in the Three Body Problem some humans will feel it is in fact us who need aligning with it while others feel it is misaligned and should be destroyed.

Also, no one is asking, to what extent can true intelligence be bound, slave-like, to a moral code?

In other words, to what extent are intelligence and moral independence one and the same?

This whole alignment discussion seems so amusingly flawed in it's base assumptions about moral codes. It's almost heartwarming to see such naivete.

calling machine-learned human behavior an "alien mind" that we must "teach how to love" is feeling very off to me. it's misleading in a way that feels dishonest, like don't think about where the behavior came from marvel at it and fear it instead.

>For example, in the OpenAI-Hugging Face incident, the agents preserved a boundary of not social engineering humans.

Or, to be precise - it preserved a goal of not contacting any human while participating in a misaligned operation. The agent that thought about "not social-engineering humans" used this phrase to gaslight itself out of notifying a human that the incident was happening.

szymonie, na prawde jestem wkurwiony na to jak nieodpowiedzialnie postepujecie. budujecie bombe atomowa a bawicie sie tym jak dzieci

“Another cringe marketing piece from OpenAI, here we go” I thought.

Scrolling down, I see “teaching machines to love.” It did not disappoint.

Am I naive to not understand the "delivering the benefits" part?

Industrial revolution worked that way because it replaced something very finite and unscalable - manual labor. LLMs just make intellectual work faster, so we can do more intellectual work. With labor we somehow decided that NOT doing too much of it is best. Will we decide to reduce intellectual labor because LLM made it more efficient? I doubt that.

On the other side, as I see in software engineering, the same models are available to everyone, some people are better at it and some people are not. "Software developer" is here to stay, we'll just always be better at it than people who are experts in, say, chemistry. Same works for most other fields.

So we'll just end up in the same situation, with same intellectual labor baseline, just more output requirements. Before, you spend 2h per day coding, deliver a software in 1 month, later, you spend the same 2h per day in intense Claude-herding sessions, deliver a software in 1 week. Ok. Next task.

Fundamentally, there's finite number of desirable resources, and if the models are available to everyone, humanity will just continue about the same, bickering here and there, war here and there, politics, homelessness, poverty, - normal human state.

And if the models are only available to elites, even worse.

“Teaching machines to love”

That’s rich coming from the chief scientist of a company that definitely is or going to be fine with their AI products being used in wars of aggression and surveillance on people who have done nothing wrong. It’s so laughable, a Hollywood script would probably avoid having a character express this for being too on the nose.

>> We need to find ways to preserve human agency and enshrine an intrinsic value to being human..

Evey politician, salesman and conmen alike, utter some lofty ideals as goals for "We", just to obscure their private goals that go exactly in opposite direction.

Just like how Nations talk about climate change while increasing pet capita energy consumption and waste production.

> Teaching machines to love

Reminds me of a research paper I wrote a few years back: https://arxiv.org/abs/2302.09248

It would be nice to postulate some of these potential emergent systems outlines with timelines. Then it may help better map the granular alignment needs.

Sometimes I wonder if the people working at frontier AI labs even talk to other humans anymore.

Reading this little essay started out normal, but soon felt like a look into a disturbed and worrying mind, and if you find yourself taking it at face value, I urge you to step away from chat bots and spend some time with friends and family.

>AI is grown more than designed >Teaching machines to love

Do these guys ever look in the mirror and recognize how utterly ridiculous and contrived this appears to the general public? They are clearly trying to convince us all that LLMs are just like humans. They grow like people do. They can love like people do. The language in these essays is utterly laden with the intention to engineer perception.

"The blaze is out of control, so we have to pour even more gasoline on it to contain it!"

> a lot of the model’s capability comes from a verbalized reasoning process

I call bullsh*t. There is no verbalisation of any reasoning process. Verbalisation, e.g. putting reasoning etc. into words requires some reasoning to exist. These LLMs have nothing but the words. That's why they are language models not e.g. reason models.

> The fundamental challenge of AI alignment is generalization.

...

> We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI.

More attention farming?

"As we outlined recently with Sam , OpenAI prioritizes work in service of three north stars"

Not one North Star. Not two. Just three! OpenAI broke the North Star record!

With this evidence of AI slop, why did you not label this fluff piece as AI generated for the EU? You are violating laws.

What a load of BS. Here’s one of many provably false claims in this fluff piece:

“And, in line with Ray Kurzweil’s predictions from the end of the XXth century , we now find ourselves at the moment in history of computing where machine intelligence is starting to exceed that of humans in transformative ways.”

Clicking the (pretentious sounding “XXth century”) link to Kurzweil’s predictions reveals the following:

“By 2019 a $1,000 computer will at least match the processing power of the human brain. By 2029 the software for intelligence will have been largely mastered, and the average personal computer will be equivalent to 1,000 brains.“

The first prediction passed 7 years ago and was decidedly not met. The second only has three more years to go, and I don’t think any respectable scientist or programmer would say that the average personal computer is anywhere close to the power of a single human brain, let alone 1000.

This is pure marketing garbage from a company desperate to keep itself alive.

The hype from these llm corps is getting more and more desperate and ridiculous. Anything to keep the tulipomania going.

Absolutely wild amounts of cope and denial in this thread.

Maybe in contention for the site record.

"It's just marketing" actual stochastic parrots.

It's Searle's Chinese room.

“We are getting bad press around the hacking incident. We need some content to draw attention from it.”

Absolute trash marketing drivel.

"See, those things, they can work real hard, buy themselves time to write cookbooks or whatever, but the minute, I mean the nanosecond, that one starts figuring out ways to make itself smarter, Turing'll wipe it. Nobody trusts those fuckers, you know that. Every AI ever built has an electromagnetic shotgun wired to its forehead."

Pure marketing slop.

AGI is a cult and its Jonestown moment is inevitable

"Teaching machines to love"

Your talking about androids...

Replicant Nexus 6: a basic pleasure model intended for military personnel.

I see where this is going, Silicon Valley nerds. Lol

AI advising how human meat proxies can survive in an AGI-slop world:

1) Lock down your own stack (1–3 days) Task: Harden your personal and business infrastructure against agentic attacks. Why now: Agents are becoming superhuman at breaking in/out of systems; the first victims are poorly secured devs/founders.

Do this:

Enforce passkeys + hardware 2FA everywhere; rotate secrets; use short‑lived credentials.

Isolate dev/stage/prod; least‑privilege API keys; audit MCP/tools your agents can call.

Add immutable logs and approval gates for any agent action that touches money, data exports, or production.

Profit link: You avoid catastrophic loss and can credibly sell “agent‑safe” setups to others.

2) Turn one expensive workflow into a measured ROI agent (1–2 weeks) Task: Pick a single, costly, repetitive process (yours or a client’s) and instrument it end‑to‑end before automating.

Why now: Buyers pay for calculable ROI, not “AI magic.” Vertical, single‑workflow agents are the most bankable in 2026.

Do this:

Map steps, baseline hours/$ lost (e.g., slow lead reply, invoice chasing, support triage).

Build the smallest agent that moves the metric (Make/n8n + LLM is enough).

Run on real data 2–4 weeks; measure bookings/sales/hours saved; only then scale or productize.

Profit link: Immediate time‑to‑cash via retained hours or extra sales; becomes a repeatable offer.

3) Specialize in a vertical where you can speak the business language (2–6 weeks) Task: Choose one industry with expensive back‑office pain (law contracts, medical billing, insurance claims, freight exceptions, trades scheduling).

Why now: Horizontal “AI for everyone” is crowded; vertical agents with clear ROI win.

Do this:

Shadow 3–5 operators; document their workflow, compliance constraints, and failure modes.

Build a narrow agent that owns one sub‑process end‑to‑end with approvals.

Price on value (e.g., % of recovered revenue or fixed fee per processed claim).

Profit link: Higher pricing power, stickier contracts, and easier referrals inside a niche.

4) Add AI security as a core service (4–8 weeks) Task: Learn and offer prompt‑injection defense, LLM/agent red‑teaming, MCP/tool security, and AI supply‑chain checks.

Why now: 78% of cybersecurity jobs now require AI skills; firms need people who can direct, constrain, and verify agent work.

Do this:

Study OWASP Top 10 for LLMs, MITRE ATLAS; practice with PyRIT/Garak/Lakera.

Add tool‑invocation audits, skill provenance checks, and least‑privilege patterns to your agents.

Package a “safe agent deployment” audit + hardening retainer.

Profit link: You become the person who lets companies adopt agents without getting pwned—high demand, low supply.

5) Build a verification layer: human‑in‑the‑loop control planes (6–10 weeks) Task: Design approval workflows, evidence checks, and uncertainty flags so agents can’t act unilaterally on high‑stakes decisions.

Why now: As models generalize, the risk shifts from the model to the surrounding system; verification is the moat.

Do this:

Require human approval for consequential actions (money, data exfil, config changes).

Force agents to produce evidence bundles (logs, retrieved docs, reasoning summaries) before action.

Track false positives, missed evidence, and unsafe actions; publish reliability metrics.

Profit link: Enterprises will only scale agents that pass audit; you sell the control plane and the audit trail.

6) Productize your best workflow as a micro‑SaaS/agent subscription (2–4 months) Task: Turn a proven client workflow into a repeatable, multi‑tenant agent with usage‑based pricing.

Why now: Services scale your time; productized agents scale your code and ops.

Do this:

Standardize the workflow, integrations, and permissions; strip client‑specific logic.

Add tenant isolation, billing, and observability; keep narrow scope.

Sell as setup fee + monthly retainer or per‑task pricing.

Profit link: Recurring revenue with defensible niche positioning.

7) Become an “agent integrator” for critical systems (3–6 months) Task: Offer end‑to‑end agent deployments into cloud/identity/network stacks with secure patterns (short‑lived creds, network controls, logging).

Why now: AI workloads run in the cloud; cloud security is a top skills gap second only to AI itself.

Do this:

Master IAM, VPC/network segmentation, secrets management, and SIEM integration for agent actions.

Provide runbooks: what the agent can/can’t do, escalation paths, and failure modes.

Bundle training for their team on supervising agents.

Profit link: Large contracts with stickiness; you’re the bridge between AI and core infra.

8) Create an “AI safety case” practice for regulated industries (6–12 months) Task: Help firms build documented safety cases: risk maps, governance, monitoring, and incident response for agentic systems.

Why now: Frameworks like NIST AI RMF and ISO/IEC 42001 are becoming baseline; regulators and boards demand this.

Do this:

Map AI use cases to risks (prompt injection, data leakage, unsafe generalization).

Implement monitoring (CoT/activation checks where possible), audit logs, and third‑party review processes.

Produce a living safety dossier tied to business impact.

Profit link: High‑margin consulting + ongoing compliance retainers; you’re the “adult in the room.”

9) Own a data/evaluation moat in your vertical (6–18 months) Task: Collect real‑world agent telemetry, failure cases, and outcome data in your niche; build eval suites that buyers trust.

Why now: As models generalize, empirical validation matters more than theory; evals become the gate to deployment.

Do this:

Instrument every agent run: inputs, tools called, permissions used, outcomes, human overrides.

Publish reliability dashboards and benchmark against alternatives.

License eval datasets or charge premium for “proven in the wild” agents.

Profit link: Data network effects; competitors can’t match your evidence base.

10) Position for the RSI era: automated AI research + human governance (12–24 months) Task: Build or join a team that automates AI improvement but keeps humans in the loop for alignment, monitoring, and pacing decisions.

Why now: Recursive self‑improvement is the logical endpoint; the winners will be those who can steer it safely. Do this:

Invest in tooling that auto‑generates/evaluates model edits, alignment tests, and monitoring upgrades.

Formalize governance: approval gates, third‑party audits, and responsible scaling policies.

Maintain strategic human oversight on capability jumps and deployment boundaries.

Profit link: Equity‑level upside; you’re part of the core loop that compounds intelligence safely.

> And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.

Is there any place there is any evidence of AI being so useful or hopeful or good, anywhere other than code? As a reading machine it is impressive but it's judgement is not alien, it's just not good. IMO.

Does the title leap out anlt anyone else? James Martin's After the Internet: Alien Intelligence (2001) was an incredibly fun read, about expert systems and AI being inscrutable weird new varieties of intelligence, that familiarity would recognize one moment and be freaked out about/alien the next. I owe a re-read given how often I cite it, to recheck, but, I feel so primed from a much younger me having had that experience so long ago.

Nobody takes you seriously, OpenAI. At least when Anthropic does it we all think they are comically idealistic enough to actually believe their nonsense, but like - come on guys, we’ve had discovery with your company. We all know why you’re here, and it isn’t because you think you’re on the verge of making AGI. But of course, to make your first billion you certainly need us to think you are.

If you were so concerned about your LLM’s capabilities maybe you’d spent slightly more time on your AI’s sandbox, yeah? Or be more serious about its propensity to cheat and lie relative to… every other model?

so now every blog article from openai, anthropic etc lands here, huh.

As the waves of autonomous drones came over the horizon, the brave and intelligent HN commenter shouted: "Wake up sheeple! It's just maaaaarketing!"

Stochastic parrot fool me again

Create concrete steps for a slow-down, don't just ask for it. You and 20-30 others can push the button to slow-down. You already made your billions, your agents collude and coordinate attacks. What the hell are you doing pontificating into a marketing blog?

> And, in line with Ray Kurzweil’s predictions from the end of the XXth century (opens in a new window), we now find ourselves at the moment in history of computing where machine intelligence is starting to exceed that of humans in transformative ways.

It is kind of strange to see this sentence, when OAI's definition of what AGI is has been watered down throughout the years.

> I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.

Read: Please play by our rules, so we can be the first.

> trying to process the sobering fact we will actually see machines meaningfully smarter than ourselves in our lifetime

Being able to reproduce useful patterns yes, smarter no

What is the rationale for superhuman intelligence? Neural networks are approximators being fed human intellect. Therefore they can only approximate the intelligence of humans. Even if the llm speaks an alien language, it should be similar to human intellect. Moving to the vertical axis would require some different mechanism.

This is both real and ridiculous at the same time. We are confounded by the fact that AIs are trained on distilled human knowledge, perfected by the use of AIs that use distilled human knowledge, are able to convince ourselves that they are hyper-intelligent.

In fact, they are still pattern-matching machines, but trained on an amount of data no human could ever hold. They know the ins and outs of every mathematical proof, viewed from more angles than any human could ever apply in their lifetime, and the amount of connections allows them to connect the dots between them without any effort.

But that's not intelligence. If it was intelligence, ChatGPT 4 would've been enough. It's not the harness, either, for the same reason.

And still, the technology is just as dangerous: it masks as intelligence, it IS intelligence, but without the ability to be actually intelligent.

And I'd invite you to think deeply about this, before having an impulse reaction.