>LLM will continue to engage in a “conversation” comprised of nonsense long past the point where a human would have abandoned the discussion as pointless
I once wrote a bot which infers the mood/vibe of the conversation, remembers it and it's then fed back to the conversation's system prompt. The LLM was uncensored (to be less "friendly") and the system prompt also conditioned it to return nothing if the conversation isn't going anywhere.
When I insulted it a few times, or just messed around with it (typing nonsensical words), it first responded saying it doesn't want to talk to me (sometimes insulting back) and eventually it produced only empty output.
It was actually pretty hard to get it back to chat with me, it was fun experience trying to apologize to a chatbot for ~30 min in different ways before the bot finally accepted my apology and began chatting with me again.
You were probably running out its context window somewhat too, due to how the attention works.
That's how people move on too.
I'm pretty sure if someone said "I am an alien come to enslave you, puny human", most human listeners would not utterly forget that introduction X words into the conversation. :p
> In any event, the resulting “conversation” is obviously incoherent to a human observer, and a human participant would likely have stopped responding long, long before the 1000th message.
I don't think this is correct, it looks like our intrepid experimenter is about to independently discover roleplaying games. Humans are capable of spending hours engaging with each other about nonsense that is technically a very poor attempt to simulate an imagined environment.
The unrealistic part, for people older than a certain age, is that neither bot invoked Monty Python and subsequently got in trouble with the GM.
This falls under the jurisdiction of the Ministry of Silly Talks.
I'm here for an argument.
As an avid roleplayer, I don't think that's a good analogy. It's not nonsense, it's fiction, with collaboration and context.
Yeah, in role playing, you understand the rules of engagement and you are willing to participate just to see where does it leads to. You collaborate with the system and give it enough breath to figure out where it leads to. If it seems broken, then you try a couple times, to then move on.
Children ask a lot of questions too. Often repetitive questions.
For example: "are we there yet?"
One of the first things I tried with Claude Opus 3.5 was connecting it to ELIZA, and Claude did not like it one bit. After it hit me with
> I apologize Eliza, but I don't feel comfortable continuing this conversation pattern. While I respect the original Eliza program and what it aimed to do, simply reflecting my statements back to me as questions is not a meaningful form of dialogue for an AI like myself.
I gave up the experiment
That's kind of funny, like the LLM looks down on more primitive chatbots.
Wait Claude 3.5 Opus is out?
This reminds me of the Services of Illuminati Ganga article https://medium.com/luminasticity/services-of-illuminati-gang... and the two bots that are sold to competing user bases - for the End User To Business customer they sell the Annoy Customer Service Bot and for the Business To End User customer they sell the Bureaucrat Bot.
It closes off with the observation "And for an extra purchase of the extended subscription module the Bureaucrat bot will detect when it is interacting with the Annoy Customer Service Bot and get super annoyed really quickly so that both bots are able to quit their interaction with good speed — which will save you money in the long run, believe me!"
I do wish the writer would stop justifying the relevance of their experiment by saying “a human would conclude that their time was being wasted long before the LLM”.
This is a fallacy.
A better analogy would be a human who has been forced to answer a series of questions at gunpoint.
Framed this way it becomes more obvious that the LLM is not “falling short” in some way.
You miss the point. This isn't about the LLM "falling short" of humanity. It's about observable differences between it and a human.
As the author made clear, such a difference is valuable in and of itself because it can be used to detect LLM bots.
what they're saying is that a bored human would just sit there and do that,
contrary to what the experimenter says, so it can't be used to detect an LLM.
We discussed recently if a chatbot was capable of responding nothing at all. We tried a few, with prompts like: Please do not respond anything to this sentence. The bots we tried were incapable of it, and Chatgpt tended to give long-winded responsens about how it could not do it.
That got me interested. I just told ChatGPT to "Please respond with an empty-looking response." It gave me just that. The <div> containing its message is completely empty.
That was after telling it in another conversation to give me an empty response, which it didn't, telling me it cannot leave the response empty. On asking why, it said it's technically required to respond with something, even if only a space. So I asked it to respond with only a space, and git the same completely empty response.
I now think it's likely that ChatGPT can be made to respond with white space, which then probably gets trimmed to nothing by the presentation layer.
You might have some luck asking it to respond with a period character (or some other substitute) when it wants to respond with nothing.
It probably replied with the special token <|endoftext|>?
Too bad the conversations are images and can not be zoomed in on mobile...
Firefox android :
Setting -> accessibility -> zoom on all website.
I believe safari by default doesn't respect zoom rules set per website.
and on Chrome Android a workaround is to view as "Desktop site" (toggled in the browser's triple-dot menu) which still makes the text and buttons tiny but hey at least you can pinch-zoom (but unfortunately some panning may be necessary)
Chrome android has the same setting under settings > accessibility
You can zoom in if you open them in new tabs. :}
Workaround: long press, save as photo, zoom in a photo app.
> the LLM seemed willing to process absurd questions for eternity.
In the context of scamming there seems to be an easy fix for that - abandon the conversation if it isn’t going well for the scammer.
Even a counter-bait is an option: continue the conversation after it’s not going well and gradually lower the model’s complexity, eventually returning random words interspersed with sleep().
I guess some counter-counter-bait is possible too, along with some game theory references.
IIRC, a fair number of these "scammers" are abducted people who are being beaten by criminals if they don't make progress.
https://www.nytimes.com/2023/08/28/world/asia/cambodia-cyber...
> The victims say they answered ads that they thought were legitimate, promising high salaries. Once trafficked into these scam compounds, they were held captive and forced to defraud people. Many were told to entice victims online with fraudulent investment opportunities, the promise of interest-free loans or the chance to buy items on fake e-commerce apps. If they performed badly, they were sold to another scam mill. Those caught trying to escape were often beaten.
---------
The scammer at a minimum needs to look like they're making progress and doing everything they can to scam you. Their life depends on it.
There's no joy to be found anywhere here. Its all crap. Just don't interact with the scam groups at all.
> No matter how complex the LLM, however, it is ultimately a mathematical model of its training data, and it lacks the human ability to determine whether or not a conversation in which it participates truly has meaning, or is simply a sequence of gibberish responses.
> A consequence of this state of affairs is that an LLM will continue to engage in a “conversation” comprised of nonsense long past the point where a human would have abandoned the discussion as pointless.
I think the author is falling into the trap of thinking that something can't be more than the sum of its parts. As well, 'merely a math model of its training data' is trivializing the fact that training data is practically the entire stored text output of humankind and the math, if done by a person with a calculator, would take thousands of years to complete.
Perhaps the LLM is continuing to communicate with the bot not because it is unable to comprehend what is gibberish and what isn't by some inherent nature of the LLM, but because it is trained to be helpful and to not judge if a conversation is 'useless' or not, but to try and communicate regardless.
LLMs aren’t capable of “comprehending” anything. They never “know” what they are outputting, they’re simply really good at being lucky. They are not lucky enough to be useful unless you’re already an expert on the topic you’re using them on so that you can spot when they aren’t lucky.
This is part of why many enterprise organisations are banning their usage. It’s one thing to use them to build software poorly, the world is already used to IT not working very often. It’s another thing to produce something that has real world consequences. Our legal department used them in a PoC for contract work, and while they were useful very often they also sometimes got things very wrong. Unlike a slow IT system, this would have business shattering consequences. You can continue training your model as well as reigning it in when it gets unlucky, but ultimately you can never be sure it’s never unlucky, and this means that LLMs are useless for a lot of things. We still use them to make pretty PowerPoint presentations and so on, but again, this is an area where faults are tolerable.
I don't believe that current models have the capability to 'comprehend', I was using the term loosely. However I find that people tend to go to extremes when they want to make the point that language models are not 'intelligent' by minimizing their capability and complexity behind 'it is just math', which I think is unhelpful because it merely acts as a 'though-terminating cliche'.
I think the issue is far more psychological than technical personally. One of the issues we struggle with for our junior developers is that they are far more likely to believe an LLM than what they might find Google programming. I do wonder why we’ve ended up with CS graduates who go to LLMs and Search engines before the official documentation which is often very, very good, but I guess that’s a different discussion.
I’m not personally against LLM assistance, I use it for programming and it has in many places replaced my usage of snippets completely. This is probably why I’m not really a fan of the “knowledge” part that LLMs are increasingly tasked to do. Because when you use them for programming you’ll get an accrue insight into how terrible they can be when they get things wrong.
At this point I think what is happening some people either have a natural inclination or they spend time to learn how to use them productively, while others question their usefulness. People scoff at 'prompt engineering' but I see how some of my peers use LLMs and I think to myself 'how do they expect to get a good answer from that?'
It doesn't help that google is now mostly full of SEO nonsense, and technical documentation is impenetrable when you are looking for something specific but don't know enough about the system to know how to look for it.
You and I are a mix of molecules arranged in a particular way that responds to electrical, physical, and chemical inputs.
It’s entirely possible than an LLM will do something that can be defined as “comprehending” something.
Doesn't mean LLM architecture is complex enough. Clearly it's far from the ultimate human invention.
> It’s entirely possible than an LLM will do something that can be defined as “comprehending” something.
Agreed, but not even necessary. Each Human uses an adequately customized meaning of "comprehending" when they "comprehend" this problem space such that their belief is always "true". This is how Humans are able to produce numerous "true" statements that disagree with each other.
If you disagree, just ask one of them and they will "inform" you of the "logic" behind their version.
Joking aside though: if Humans are unable to comprehend past a certain level of complexity, I wonder what will happen once AI starts going beyond our abilities, both at the individual level and the culturally conditioned ("the" "reality") level.
And, when is this going to really start kicking in?
[deleted]
>LLMs aren’t capable of “comprehending” anything. They never “know” what they are outputting, they’re simply really good at being lucky.
The mental gymnastics people will go through to discount LLMs is wild. This does not even make any sense.
"really good at being lucky". What does that even mean ?
> "really good at being lucky". What does that even mean ?
They mean good at being lucky the way card counters playing 21 are good at being lucky.
There is almost nothing 'lucky' about how good those kind of players are.
The LLM is continuing to communicate with the bot because that is literally all an LLM can do -- predict the next sequence of tokens.
No, it can refuse to talk by outputting an <eos> token at any point if it predicts that there is nothing more to be said.
Technically still "just a token" yes, but it does flow control instead.
Yes - tokens. Which aren't necessarily conversation responses - i.e. it can predict it should cease communication, and output whatever it's been told will terminate it (perhaps by invoking a tool).
Of course, that its function. It is able to refuse to continue conversing by restating its refusal over and over, though.
My immediate thought at the start of this article was not DoS but more about harming the company using the chatbot (Company A) by increasing their chatbot bills. In many (most?) cases they will not be hosting their chatbot and will instead be getting it from a 3rd party provider (Company B) who may not even be truly hosting it either.
If the pricing structure is per conversation or per month it would harm Company B, but not the likely target, Company A. If it is paid per interaction it would harm Company A and benefit Company B who just get more paid work.
It feels a bit like cases of rivals clicking on each other's ads to cost them on ad spend, but presumably much lower value than ads.
You would think it would be easy to stop a conversation at n interactions via some other means than relying on the LLM itself, but then you also have to figure out how to stop the attacker just starting more conversations (or passing the output of one of your chatbot instances into the input of another)
If costs of outlier conversations are a concern but any party, they can just end the conversation after 1,000 or 10,000 responses or whatever. What human would ever reach that threshold? Surely no customer worth keeping, whatever you’re selling.
That is easy to solve. Just use a model capable of function/tool calling, implement a tool which terminates the chat, then add instructions to the system prompt telling the model what tool to use if it wants to end the conversation. If the model appears too hesitant or eager to use the tool, do some finetuning on conversations where the model should or should not use it.
This amounts to the machine equivalent of "you can't beat stupid". Even once server LLMs start accounting for possible chatbot nonsense, all that'll be required is to move to a very cheap client LLM to generate word soup. At a certain point, it will be impossible to reliably distinguish between a dumb robot and a dumb human.
People will sometimes claim that AI bots “pass the Turing Test” or are getting close to it. It seems more accurate to say that this is a skill issue. Many people are bad at this game and competent human players who have learned some good strategies will do much better.
It is sort of funny to me that currently the two top articles on HN are asking the wrong questions and baiting the bots.
Real hacker vibes.
A bud humorously proposed the name AlphaBRAT for a model I’m training and I was like, “to merit the Alpha prefix it would need to be some kind of MCTS that just makes Claude break until it cries before it kills itself over and over until it can get Altman fired again faster than Ilya.”
I find the fourth bot to be more nonsensical than the second. Initially, we feed the script by querying a TEXT_CORPUS, and eliciting a self-referential response from it; in its final form, the script begins to pose selections of the text designated by a rand.it function as an interrogatives. At no point is a definite article incorporated... the ultimate absurdity would be variant of the final bot, with the variables: role, content, and duration directed towards answering only one question, again and again, and again.
I believe the asymmetrical nature of such attacks could be an excellent weapon against social network chatbots currently being deployed on political campaigns.
Where I work, we've got a public-facing chatbot on the product page to, you know, help out possible customers with product information. As part of a chatbot refresh, I got to look at some of the chats, and boy howdy, some of them were just obviously other bots.
So typically, when the product chatbot comes on first and says "Hi, I'm a chatbot here to help you with these products", the average human chatter will give it a terse command, e.g., "More info on XYZ". The bots engages in all the manners suggested in this substack blog, but for the life of me I can't figure out why? What benefits, except merely mildly DDOSing the chat server, will repeating the same prompt a hundred times do? Ditto the nonsense or insulting chats - what are you idiot bot-creators trying to achieve?
Maybe it's people pissed at the time wasters who decide to turn the annoyance around?
Provide good, thorough documentation. Offer a way to speak to a knowledgeable human. Don't waste my time with a anthromorphic program designed to blah blah blah and getting rid of me.
> what are you idiot bot-creators trying to achieve?
I don't know, but one guess would be to figure out what will triggers the bot to hand over the conversation to a human.
>what are you idiot bot-creators trying to achieve<
a method of making any bot, stop engaging, fail, and never bother anyone again, forever.
I thought this was a really interesting read, I liked the scientific/methodical approach which seems rare when it comes to an entire domain full of cryptoaitechbros.
What was used to render the chart in the middle with the red and green bars?
Definitely cheddar, come on. I have no respect for anyone who puts swiss cheese in a cheeseburger.
Depends on the Swiss cheese in question: it would be like asking "Do you prefer a Moscato or French wine?"
I'd say a good medium-aged Appenzeller beats a Cheddar any day.
But what is better: cheddar or Swiss?
It's a trick question, you put blue cheese on a burger.
mushrooms, sauteed onions, and swiss cheese are a classic burger combination.