We would need the big picture, though... maybe it caused that death (which is awful) but it's also saving lives? If there are that many people confiding in it, I wouldn't be surprised if it actually prevents some suicides with encouraging comments, and that's not going to make the news.

Before declaring that it shouldn't be near anyone with psychological issues, someone in the relevant field should study whether the positive impact on suicides is greater than negative or vice versa (not a social scientist so I have no idea what the methodology would look like, but if should be doable... or if it currently isn't, we should find the way).

I feel like this article is apropos:

https://www.lesswrong.com/posts/iGF7YcnQkEbwvYLPA/ai-induced...

Basically, the author tried to simulate someone going off into some sort of psychosis with a bunch of different models; and got wildly different results. Hard to summarize, very interesting read.

While I agree that AI shouldn't be sycophantic, and I also agree that the AI shouldn't have said those things.

> “Please don’t leave the noose out,” ChatGPT responded. “Let’s make this space the first place where someone actually sees you.”

That is not sycophantic behaviour, it is asserting a form of control of the situation. The bot made a direct challenge to the suggestion.

I only just realised this now reading your comment, but I hardly ever see responses that push back against what I say like that.

>should convince you that ChatGPT should be nowhere near anyone dealing with psychological issues.

Is that a debate worth having though?

If the tool is available universally it is hard to imagine any way to stop access without extreme privacy measures.

Blocklisting people would require public knowledge of their issues, and one risks the law enforcement effect, where people don’t seek help for fear that it ends up in their record.

Something is indeed NOT better than nothing. However, for those with mental and emotional issues (likely stemming from social / societal failures in the first place) anything would be better nothing because they need interaction and patience —two things these AI tools have in abundance.

Sadly there is no alternative. This is happening and there’s no going back. Many will be affected in detrimental ways (if not worse). We all go on with our lives because that which does not directly affect us is not our problem —is someone else’s problem/responsibility.

This is just one example of the logical end state of grossly over prioritizing capital over labor in the economy.

This discovery was probably not on purpose.

No one at OpenAI thought "Hey lets make a sucidie bot"

But this should show us how shit our society is to a lot of people, how much we need to help each other.

And i'm pretty sure that a good bot could def help

There are one-off things, and then there is exponential improvements - both in guardrails, and ChatGPT's ability to handle these discussions.

This type of discussion might be very much possible in ChatGPT in 6-24 months.

That was before they started acting to fix the problem. Please check the date.

What version of chatGPT was this? That sounds like a gpt-3.5 kind of response.

I wonder how the latest models would fare in these kinds of conversations.

Exactly, any company that offers chatbots to the public should do what Google did regarding suicide searches, remove harmful websites and provide info how to contact mental health professionals. Anything else would be corporate suicide (pun not intended).

I know that minors under age 13 are not allowed to use the app. But 13-18 is fine? Not sure why. Might also be worth looking into making apps like these 18+. Whether by law or by liability, if someone 20+ gets, say, food poisoning by getting recipes from chatgpt, then you can argue that it's the user's fault for not fact checking, but if a 15yo kid gets food poisoning, it's harder to argue that it's the kid's fault.

That's really really bad.

But also, how many people has it talked out of doing it? We need the full picture.

im willing to bet that it reduces them at a statistical level. knee jerk emotional reaction to a hallucinaton isnt the way forward with these things

"One may think that something is better than nothing, but a bot enabling your destructive impulses is indeed worse than nothing."

And how would a layman know the difference?

If i desperately need help with mental item x and i have no clue how to get help, am very very ashamed for even mentioning to ask for help about mental item x or there are actually no resources available, i will turn to anything else than nothing. Because item x still exists and is making me suffer 24/7.

At least the bot pretends to listen, some humans cannot even do that.

OpenAI tried to get rid of the excessively sycophantic model (4o) but there was a massive backlash. They eventually relented and kept it as a model offering in ChatGPT.

OpenAI certainly has made mistakes with its rollouts in the past, but it is effectively impossible to keep everyone with psychological issues away from a free online web app.

>ChatGPT should be nowhere near anyone dealing with psychological issues.

Should every ledge with a >10ft drop have a suicide net? How would you imagine this would be enforced, requiring everyone who uses ChatGPT to agree to an "I am mentally stable" provisio?

> There is no other system that is available at scale, 24x7 on my phone.

https://en.wikipedia.org/wiki/Suicide_and_Crisis_Lifeline

I choose to believe that too. I think more people are interested than we’d initially believe. Money restrains many of our true wants.

Sidebar — I do sympathize with the problem being thrust upon them, but it is now theirs to either solve or refuse.

A chat like this is all you’ve said and dangerous, because they play a middle ground: Presenting a machine can evaluate your personal situation and reason about it, when in actuality you’re getting third party therapy about someone else’s situation in /r/relationshipadvice.

We are not ourselves when we are fallen down. It is difficult to parse through what is reasonable advice and what is not. I think it can help most people but this can equally lead to a disaster… It is difficult to weigh.

If you look at the number of weekly open ai users, this is just the law of big numbers at play.

> A chat like this is not a solution though, it is an indicator that our societies have issues

Correct, many of which are directly, a skeptic might even argue deliberately, exacerbated by companies like OpenAI.

And yet your proposal is

> a company to tackle this at scale.

What gives you the confidence that any such company will focus consistently, if at all,

> on help, not profits

Given it exists in the same incentive matrix as any other startup? A matrix which is far less likely to throw one fistfuls of cash for a nice-sounding idea now than it was in recent times. This company will need to resist its investors' pressure to find returns. How exactly will it do this? Do you choose to believe someone else has thought this through, or will do so? At what point does your belief become convenient for people who don't share your admirably prosocial convictions?

HIPAA only applies to covered healthcare entitites. If you walk into a McDonalds' and talk about your suicidal ideation with the cashier, that's not HIPAA covered.

To become a covered entity, the business has either work with a healhcare provider, health data trasmiter, or do business as one.

Notably, even in the above case, HIPAA only applies to the healthcare part of the entity. So if McDonald's collocated pharmacies in their restaurants, HIPAA would only apply to the pharmacists, not the cashiers.

That's why you'll see in connivence stores with pharmacies, the registers are separated so healthcare data doesn't go to someone who isn't covered by HIPAA.

**

As for how ChatGPT gets these stats... when you talk about a sensitive or banned topic like suicide, their backend logs it.

Originally, they used that to cut off your access so you wouldn't find a way to cause a PR failure.

Privacy is vital, but this isn't covered under HIPAA. As they are not a covered entity nor handling medical records, they're beholden to the same privacy laws as any other company.

HIPAA's scope is actually basically nonexistent once you get away from healthcare providers, insurance companies, and the people that handle their data/they do business with. Talking with someone (even a company) about health conditions, mental health, etc. does not make them a medical provider.

> HIPAA anybody?

Maybe. Going on a tangent: in theory GMail has access to lots of similar information---just by having approximately everyone's emails. Does HIPAA apply to them? If not, why not?

> If you give people something that seems to be a listening ear they will unburden themselves on that ear regardless of the implementation details of the ear.

Cf. Eliza, or the Rogerian therapy it (crudely) mimics.

> if people try to engage it in such conversations the bot should immediately back out because it isn't qualified to have these conversations

For a lot of people, especially in poorer regions, LLMs are a mental health lifeline. When someone is severely depressed they can lay in bed the whole day without doing anything. There is no impulse, as if you tried starting a car and nothing happens at all, so you can forget about taking it to the mechanic in the first place by yourself. Even in developed countries you can wait for a therapist appointment for months, and that assumes you navigated a dozen therapists that are often not organized in a centralized manner. You will get people killed like this, undoubtedly.

LLMs are far beyond the point of leading people into suicidal actions, on the other hand. At the very least they are useful to bridge the gap between suicidal thoughts appearing and actually getting to see a therapist

Tangent but now I’m curious about the bot, is there a write-up anywhere? How did it work? If someone says “hi”, what did the bot respond and what did the human do? I’m picturing ELIZA with templates with blanks a human could fill in with relevant details when necessary.

If there is somebody in current year that still thinks they would not store, process/train and use/sell all data, they do probably need to see a doctor.

As others have stated HIPAA applies to healthcare organizations.

Obligating everyone to keep voluntarily disclosed health statements confidential would be silly.

If I told you that I have a medical condition, right here on HN -- would it make sense to obligate you and everyone else here keep it a secret?

> (4) if people try to engage it in such conversations the bot should immediately back out because it isn't qualified to have these conversations

There is nothing in the world that OpenAI is qualified to talk about, so we might as well just shut it down.

> we shut down the project realizing that we were in no way qualified to handle human interaction at that level

Ah, when people had a spine and some sense of ethics, before everything dissolved in a late stage capitalism all is for profit ethos. Even yourself is a "brand" to be monetised, even your body is to be sold.

We deserve our upcoming demise.

Are you sure ChatGPT is the solution? It just sounds like another "savior complex" sell spin from tech.

1. Social media -> connection 2. AGI -> erotica 3. Suicide -> prevention

All these for engagement (i.e. addiction). It seems like the tech industry is the root cause itself trying to masquerade the problem by brainwashing the population.

https://news.ycombinator.com/item?id=45026886

~11% of the US population is on antidepressants. I'm not, but I personally know the biggest detriment to my mental health is just how infrequently I'm in social situations. I see my friends perhaps once every few months. We almost all have kids. I'm perfectly willing and able to set aside more time than that to hang out, but my kids are both very young still and we're aren't drowning in sports/activities yet (hopefully never...). For the rest it's like pulling teeth to get them to do anything, especially anything sent via group message. It's incredibly rare we even play a game online.

Anyways, I doubt I'm alone. I certainly know my wife laments the fact she rarely gets to hang out with her friends too, but she at least has one that she walks with once a week.

I'm surprised it's that low to be honest. By their definition of any mental illness, it can be anything from severe schizophrenia to mild autism. The subset that would consider suicide is a small slice of that.

Would be more meaningful to look at the % of people with suicidal ideation.

> OpenAI is trying to do something about it.

Ha good one

This stat is for AMI, for any mental disorder ranging from mild to severe. Anyone self-reporting a bout of anxiety or mild depression qualifies as a data point for mental illness. For suicide ideation the SMI stat is more representative.

There are 800 million weekly active users on ChatGPT. 1/800 users mentioning suicide is a surprisingly low number, if anything.

>Most people don't understand just how mentally unwell the US population is

The US is no exception here though. One in five people having some form of mental illness (defined in the broadest possible sense in that paper) is no more shocking than observing that one in five people have a physical illness.

With more data becoming available through interfaces like this it's just going to become more obvious and the taboos are going to go away. The mind's no more magical or less prone to disease than the body.

Suicide is not a mental illness.

Unless you're in that soviet man-hating mindset that put every failed suicide in mental institution.

> At least OpenAI is trying to do something about it.

They can certainly say that their chat bot has a documented history of attempting to reduce the number of suicidal people.

I am one of these people (mentally ill - bipolar 1). I’ve seen others others via hospitalization that i would simply refuse to let them use ChatGPT because it is so sycophantic and would happily encourage delusions and paranoid thinking given the right prompts.

> At least OpenAI is trying to do something about it.

In this instance it’s a bit like saying “at least Tesla is working on the issue” after deploying a dangerous self driving vehicle to thousands.

edit: Hopefully I don't come across as overly anti-llm here. I use them on a daily basis and I truly hope there's a way to make them safe for mentally ill people. But history says otherwise (facebook/insta/tiktok/etc.)

I am honestly surprised it’s only roughly 1 million per week. I would have believed a number at least an order magnitude higher.

We need to monitor americans' ai usage and involuntarily commit them if they show anomalies.

Allowing open source ai models without these safety measures in place is irresponsible and models like qwen or deepseek should be banned. (/s)

Those numbers are staggering.

It's a matter of trust and incentives. How can you trust a program curated by an entity with no accountability? A therapist has a personal stake in helping patients. An LLM provider does not.

Seeking help should not be so taboo as people are resorting to doing it alone at night while no one is looking. That is society loudly saying "if you slip off the golden path even a little your life is over". So many people resorting to LLMs for therapy is a symptom of a cultural problem, it's not a solution to a root issue.

Perhaps then the solution is that LLMs need to be aware when the chat crosses a threshold and becomes talk of suicide.

When I was getting my Education degree, we were told that, as teachers, to take talk of suicide by students extremely seriously. If a student talks about suicide, a professional supposedly asks, "Do you know how you're going to do it?" If there is an affirmative response, the danger is real.

I suspect that comes from examining case studies?

LLMs are quite good at psychological questions. I've compared AI with tharapy professional responses and they matched 80%. It is is easier to open to it, be frank (so fear of regection or ridicule is no more). And most importantly some people don't have access to proper pool of therapists (as yet you need to "match" with the one who resonates with you) making LLMs a bliss. There is place for both human and LLM psyhelp.

I'm glad you carried through that period.

I've heard this a lot, and personally I've had a lot of good success with a prompt that explains some of my personality traits and asking it to work through a stressful situation for me. The good thing with this rather than a therapist/coach is that it understands a lot of the subject matter and can help with the detail.

I wonder if really what we need is some sort of supervised mode, where users chat with it but a trained professional reviews the transcripts and does a weekly/monthly/urgent checkin with them. This is how (some? most?) therapists work themselves, they take their notes to another therapist and go through them.

Given how ideologically captured the therapist industry is now I think it’s very hard to say that an LLM do such things is objectively worse.

> best approximation of the rational mind attempting to figure out an escape from a consistently negative situation under conditions of very limited information about alternatives

I dislike this phrasing, because it implies things can always get better if only the suicidal person were a bit less ignorant. The reality is there are countless situations from which the entire rest of your life is 99.9999% guaranteed to constitute of a highly lopsided ratio of suffering to joy. An obvious example are diseases/disabilities in which pain is severe, constant, and quality of life is permanently diminished. Short of hoping for a miracle cure to be discovered, there is no alternative and it is perfectly rational to conclude that there is no purpose to continuing to live in that circumstance, provided the person in question lives with their own happiness as a motivating factor.

Less extreme conditions than disability can also lead to this, where it's possible things can get better but there's still a high degree of uncertainty around it. For example, if there's a 30% chance that after suffering miserably for 10 years your life will get better, and a 70% chance you will continue to suffer, is it irrational to commit suicide? I wouldn't say so.

And so, when we start talking about suicide on the scale of millions of people ideating, I think there's a bit of folly in assuming that these people can be "fixed" by talking to them better. What would actually make people less suicidal is not being talked out of it, but an improvement to their quality of life, or at least hope for a future improvement in quality of life. That hope is hard to come by for many. In my estimation there are numerous societies in which living conditions are rapidly deteriorating, and at some point there will have to be a reckoning with the fact that rational minds conclude suicide is the way out when the alternatives are worse.

> [1] Anybody concerned by such figures (as one justifiably should be without further context) should note that suicidality in the population is typically the result of their best approximation of the rational mind attempting to figure out an escape from a consistently negative situation under conditions of very limited information about alternatives, as is famously expressed in the David Foster Wallace quote on the topic.

> The phenomenon usually vanishes after gaining new, previously inaccessible information about potential opportunities and strategies.

Is this actually true? (i.e. backed up by research)

[I'm not neccesarily doubting, that is just different from my mental model of how suicidal thoughts work, so im just curious]

There is another factor to consider. The stakes of asking an AI about a taboo topic are generally considered to be very low. The number of people who have asked ChatGPT something like "how to make a nuclear bomb" should not be an indication of the number of people seriously considering doing that.

That's 1 million people per week, not in general. It could be 1 million different people every week. (Probably not, but you get where I'm going with that.)

The math actually checks out.

5% of 800 million is 40 million.

40 million thoughts per year divided by 52 weeks per year approximately equals around 1 million thoughts per week.

To be fair, this is week and more focused specifically on planning or intent. Over a year, you may get more unique hits to those attributes.. which I feel are both more intense indicators than just suicidal feelings on the scale of "how quickly feelings will turn to actions". Talking in the same language and timescales are important in drawing these comparisons - it very well could be that OAI's numbers are higher than what you are comparing against when normalized for the differences I've highlighted or others I've missed.

Why assume any of the information in this article is factual? Is there any indication any of it was verified by anyone who does not have a financial interest in "proving" a foregone conclusion? The principal author of this does not even have the courage to attach their name to it.

I don’t buy the “LLMs = books” analogy. Books are static; today’s LLMs are adaptive persuasion engines trained to keep you engaged and to mirror your feelings. That’s functionally closer to a specialized book written for you, in your voice, to move you toward a particular outcome. If there exists a book intended to persuade its readers into committing suicide, it would surely be seen as dangerous for depressed people.

I work with a company that is building tools for mental health professionals. We have pilot projects in diverse nations, including in nations that are considered to have adequate mental health care. We actually do not have a pilot in the US.

The phenomenon of people turning to AI for mental health issues in general, and suicide in particular, is not confined to only those nations or places lacking adequate mental health access or awareness.

> As long as people realize they are talking to a piece of software and not a real person

That has nothing to do with the issue. Most people do realise LLMs aren’t people, the problem is that they trust them as if they were better than another human being.

We know people aren’t using LLMs carefully. Your hypothetical is irrelevant because we already know it isn’t true.

https://archive.is/2025.05.04-230929/https://www.rollingston...

> "talking to an LLM about suicide" is not that much different from "reading a book/watching a movie about suicide"

It is a world of difference. Books don’t talk back to you. Books don’t rationalise your thoughts and give you rebuttals and manipulate you in context.

Precisely, I too have a bone to pick with AI companies, Big Tech and Co but there are deeper societal problems at work here where blanket bans and the like are useless or a slippery slope towards policies that can be abused someday/somehow.

And solutions for solving those underlying problems? I haven't the faintest clue. Though these days I think the lack of third spaces in a lot of places might have a role to play in it.

You don't need to withdraw from humanity, you only need to withdraw from Big Tech platforms. I'm continually amazed at the difference between the actual human race and the version of the human race that's presented to me online.

The first one is basically great, everywhere I go, when I interact with them they're some mix of pleasant, friendly, hapless, busy, helpful, annoyed, basically just the whole range of things that a person might be, with almost none of them being really awful.

Then I get online and look at Reddit or X or something like that and they're dominated by negativity, anger, bigotry, indignation, victimization, depression, anxiety, really anything awful that's hard to look away from, has been bubbled up to the top and oh yes next to it there are some cat videos.

I don't believe we are seeing some shadow side of all society that people can only show online, the secret darkness of humanity made manifest or something like that. Because I can go read random blogs or hop into some eclectic community like SDF and people in those places are basically pleasant and decent too.

I think it's just a handful of companies who used really toxic algorithms to get fantastically rich and then do a bunch of exclusivity deals and acquire all their competition, and spread ever more filth.

You can just walk away from the "communities" these crime barons have set up. Delete your accounts and don't return to their sites. Everything will immediately start improving in your life and most of the people you deal with outside of them (obviously not all!) turn out to be pretty decent.

The principal survival skill in this strange modern world is meeting new people regularly, being social, enjoying the rich life and multitude of benefits which arise from that, but also disconnecting with extreme rapidity and prejudice if you meet someone who's showing signs of toxic social media brain rot. Fortunately many of those people rarely go outside.

The romantic fallback plan of being a farmer or shepherd. I wonder, do farmers and shepherds also romantize about becoming programmers or accountants when they feel down?

> We're losing the ability to interact with each other socially, to flirt,

Speak for yourself. I live in a city. I talk to my neighbors. I met my ex at a cafe. It’s great

This trend and direction has been going a long time and it's becoming increasingly obvious. It is ridiculous and insane.

Go for your plan B.

I followed my similar plan B eight years ago, wild journey but well worth it. There are a lot of ways to live. I'm not saying everyone should get out of the rat race but if you're one, like I was, who has a feeling that the tech world is mostly not right in an insidious kind of way, pay attention to that feeling and see where it leads. Don't need to be brash as I was, but be true to yourself. There's a lot more to life out there.

If you have kids and they depend on an expensive lifestyle, definitely don't be brash. But even that situation can be re-evaluated and shifted for the better if you want to.

>now we're making life so stressful people literally want to kill themselves

Is this actually the case? Working conditions and health during industrial revolution times doesn't seem that much better. There is a perception that people now are more stressed/tired/miserable than before, but I am not sure that is the case.

In fact I think it's the opposite, we have enough leisure time to reflect upon the misery and just enough agency to see that this doesn't have to be a fact of life, but not enough agency to meaningfully change it. This would also match how birth rates keep declining as countries become more developed.

I'm right behind you on the escape to the mountains idea. I've actually already moved from the US to New Zealand, and the next step is a farm with some goats lol.

That said... I don't necessarily hate what AI is doing to us. If anything, AI is the ultimate expression of humanity.

Throughout history humans have continually searched for another intelligence. We study the apes and other animals, we pray to Gods, we look to the stars and listen to them to see if there are any radio signals from aliens, etc. We keep trying to find something else that understands what it is to be alive.

I would propose that maybe humans innately crave to be known by something other than ourselves. The search for that "other" is so fundamentally human, that building AI and interacting with it is just a natural progression of a quest we've already been on for thousands of years.

Your comment reminded me of Business Business [0]

[0] https://youtu.be/WO5wpeYSotg?si=hgwzJ5mxJyAZeYoA

I partly agree and partly disagree. Yes, we're more individual and more isolated. But ChatGPT/Gemini can really provide mental relief for people - not everyone can afford or have the time/energy to find a good human therapist close to their home. And this thing lives in your computer or phone and you can talk to it to get mental relief 24 / 7. I don't see it as bleak as you see it, mental help should be accessible and free for everyone. I know, we've had a bad decade with platforms like Meta/TikTok but I'm not convinced as you are the current LLMs will have an adverse effect.

Book recommendation:

Ashley Montagu, On Being Human

It’s from the 1950s, I believe.

I like your plan B. But I would wait until robots are good enough to help with the hard work

You can do something about it. Don't underestimate the power of an individual.

We’ve lost the ability to interact, huh? How do you explain this comment :)

The world has changed. Things are different and we adapt.

This is over the top. With tiny reframe i think the story is different. What is the avg number of google searches about suicid? What is the avg number of weekly openai users? (800m) Is this an increasing trend or just a “shock value” number?

Things are not as bleak as it seems and this number isnt even remotely surprising nor concerning to me.

i seen similar reports on social media, all of them had in common was presence of some keywords.

I agree with you in the sense that I find it helpful for personal topics. I found it very helpful to figure out how to deal with some difficult personal situation I was in. The thing actually helped me reassess the situation when I asked it to provide alternative viewpoints.

You can't just blindly type in your problem though, you still have to do the actual thinking yourself. Good prompting skills is the ability to steer with your mind. It's no different from using Google, where some people never figured out that you're actually typing in the solution you expect to find rather than the question you have. It's the same with these tools it seems

"ChatGPT saved me from years of suicidal thoughts in DAYS": https://old.reddit.com/r/traumatoolbox/comments/1kdx3aw/chat...

Amazing based on 0 evidence whatsoever “it’s passable that…”

Also incredible how you framed improving your mental health as a consequence of a (pseudo) technical skill set.

Yeah this isn’t how any of this works and you’re deluding yourself.

Out of curiosity, are you the type of person who believes that someone like Joe Rogan has an obligation to argue with his guests if they stray from “expert consensus”, or for every guest that has a controversial opinion, feature someone with the opposite view to maintain balance?

>what should instead happen is the AI try to guide them towards making their lives less shit

There aren't enough guardrails in place for LLMs to safely interact with suicidal people who are possibly an inch from taking their own life.

Severely suicidal/clinically depressed people are beyond looking to improve their lives. They are looking to die. Even worse, and what people who haven't been there can't fully understand is the severe inversion that happens after months of warped reality and extreme pain, where hope and happiness greatly amplify the suicidal thoughts and can make the situation far more dangerous. It's hard to explain, and is a unique emotional space. Almost a physical effect, like colors drain from the world and reality inverts in many dimensions.

It's really a job for a human professional and will be for a while yet.

Agree that "shut down and refer to hotline" doesn't seem effective. But it does reduce liability, which is likely the primary objective...

Refer-to-human directly seems like it would be far more effective, or at least make it easy to get into a chat with a professional (yes/no) prompt, with the chat continuing after a handoff. It would take a lot of resources though. As it stands, most of this happens in silence and very few do something like call a phone number.

> What would be more interesting is how effective ChatGPT is being in guiding them towards other ideas. Most suicide prevention notices are a joke - pretending that "call this hotline" means you've done your job and that's that.

I've triggered its safety behavior (for being frustrated, which it helpfully decided was the same as being suicidal), and it is the exact joke of a statement you said. It suddenly reads off a script that came from either Legal or HR.

Although weirdly, other people seem to get a much shorter, obviously not part of the chat message, while I got a chat message, so maybe my messages just made it regurgitate something similar. The shorter "safety" message is the same concept though, it's just: "It sounds like you’re carrying a lot right now, but you don’t have to go through this alone. You can find supportive resources here."

If you accept that “the world and life aren’t particularly pleasant things”, why do you want to prevent suicide?

AI should help people achieve their goals and shouldn't be trying to persuade them into doing things others want them to.

I bet it's how many people trigger the "safety" filter, which is way too sensitive: https://www.reddit.com/r/ChatGPT/comments/1ocen4g/ummm_okay_...

Does OpenAI's terms prevent them from looking at chats at all? I assumed that if you don't "help improve the model", it just means that they won't feed your chats in as training data, not that they won't look at your chats for other purposes.

Yes: https://www.reuters.com/technology/chatgpt-sets-record-faste...

> making it the fastest-growing consumer application in history, according to a UBS study on Wednesday.

Not at all, look at Tiktok

> OpenAI is contributing to these issues implicitly by helping companies automate away jobs.

Good luck implementing that.

Forbidding automation will make the product more expensive. Sales will go down, the company will go bankrupt.

Government cannot subsidize or sustain such a behavior forever either.

Ideation (as I understand it) crosses the barrier from a hypothetical to the possibility being entertained.

I have also been told by people in the mental health sector that an awful lot of suicide is impulse. It's why they say the element of human connection which is behind the homily of asking "RU ok" is effective: it breaks the moment. It's hokey, and it's massively oversold but for people in isolation, simply being engaged with can be enough to prevent a tendency to act, which was on the brink.

Not at all, considering end of life and to choose euthanasia, or not, I think it's perfectly human. Controversially, I think it's a natural right to decide how you will exit this world. But having an objective system that you don't have to pay like a therapist to try to get some understanding is at least better than nothing.

That seems like an obvious problem. Less obvious is, how many people does it meaningfully help, and how big is the impact of redirecting people to a crisis hotline? I’m legitimately unsure. I have talked to the chatbot about psychological issues and it is reasonably well-informed about modern therapeutic practices and can provide helpful responses.

Of course, there is already news about how they use every single interaction to train it better.

There is news about how a judge is forcing them to keep every chat in existence for EVERYONE just in case it could relate to a court case (new levels of worldwide mass surveillance can apparently just happen from one judges snap decision)

There is news about cops using some guys past image generation to try and prove he is a pyromaniac (that one might have been police accessing his devices though)

Yes. Most would benefit from more exercise. We need to get sufficient sleep. And more sun. Vitamin D deficiency is shockingly common, and contributes to mental health problems.

We would also generally benefit from internalizing ideas from DBT, CBT, and so on. People also seriously need to work on distress tolerance. Having problems is part of life, and an inability to accept the discomfort is debilitating.

Also, we seriously need to get rid of the stupid idea of trigger warnings. The research on the topic is clear. The warnings do not actually help people with PTSD, and can create the symptoms of PTSD in people who didn't previously have it. It is creating the very problem that people imagine it solving!

All of this and more is supported by what is actually known about how to treat mental illness. Will doing these things fix all of the mental illness out there? Of course not! But it is not downplaying serious mental illness to say that we should all do more of the things that have been shown to help mental illness!

> give it a try

If you have mental issues that is not as simple as you let it sound. I'm not arguing the results of exercise but I am arguing the ease of starting with a task which requires continuous effort and behavioural changes.

Most people would really benefit from socializing with others on a weekly basis. If you don’t have friends, make some. Volunteer. The gym is another type of pressure on people’s lives.

1) are you going to finance that?

2) are you going to make sure other people at the gym don't make fun of me?

>> Besides my wife and kids its almost total isolation

Good old "if you have money trouble try decreasing your caviar and truffle intake to only two meals a day"

Why did that make you want to kill yourself?

> then where would I talk about that?

Alert: with ChatGPT you're not talking to anyone. It's not a human being.

Not suicidal myself, but I think I'd be curious to hear from someone suicidal whether it actually worked for them to read "To whomever you are, you are loved!" followed by a massive spam of hotline text.

It always felt the same as one of those spam chumboxes to me. But who am I to say, if it works it works. But does it work? Feels like the purpose of that thing is more for the poster than the receiver.

> People just copypasta prevention hotlines and turn their minds off from the topic

But ChatGPT does exactly the same.

It depends on how precisely you want do definite that situation. Specifically, with the memories feature, despite being the same model, ChatGPT and now Claude both exhibit different interactions customized to each customer that makes use of those features. From simple instructions, like "never apologize, never tell me I'm right", to having a custom name and specifying personality traits like be sweet or sarcastic, so one person' LLM might say "good morning my sweet prince/princess" while another user might choose to be addressed "what up chicken butt". It's not a custom model, but the results are arguably the same. The question is, how many of the 800 million users of ChatGPT have named their ChatGPT, and how many have not? How many have mentioned their dreams, their dreams, and fears, and have those saved to the database. How many have talked about mundane things like their cat, and how many have used the cat to blackmail ChatGPT into answering something it doesn't want to, about politics, health, cat health while at the vet or instead of going to a vet. They said 100 million people mentioned suicide in the past week, but that just raises more questions than it answers.

Funny how this was voted up 4 times and then voted down five times.

I mean, betterhelp would probably be an improvement over counseling via hallucinating AI.