> How is that any different though from regular false or fabricated information gleaned from Google, social media or any other source?

This was so obviously nonsense that it could only have been written maliciously by a human. In practice, you won't find that much, at least on topics like this.

And I think people, especially laypeople, do tend to see the output of the bullshit generating robot as authoritative, because it _looks_ authoritative, and they don't understand how the bullshit generating robot works.

> How is that any different though from regular false or fabricated information gleaned from Google, social media or any other source?

It lowers the barrier to essentially nothing. Before, you'd have to do work to generate 2 pages of (superficially) plausible sounding nonsense. If it was complete gibberish, people would pick up very quickly.

Now you can just ask some chatbot a question and within a second you have an answer that looks correct. One has to actually delve into it and fact check the details to determine that it's horseshit.

This enables idiots like the redditor quoted by the parent to generate horseshit that looks fine to a layman. For all we know, the redditor wasn't being malicious, just an idiot who blindly trusts whatever the LLM vomits up.

It's not the users that are to blame here, it's the large wave of AI companies riding the sweet capital who are malicious in not caring one bit about the damage their rhetoric is causing. They hype LLMs as some sort of panacea - as expert systems that can shortcut or replace proper research.

This is the fundamental danger of LLMs. They have crossed past the uncanny valley. It requires a person of decent expertise to discover the mistakes generated and yet the models are being sold to the public as a robust tool. And the public tries the tools and in absence of being able to detect the bullshit, they use it and regurgitate the output as facts.

And then this gets compounded by these "facts" being fed back in as training material to the next generation of LLMs.

Well if you use a gpt as a search engine and don’t check sources you get burned. That’s not an issue with the gpt.

I’m directly referring to chatGPT.

i used to use gpt for asking really specific questions that i cant quite search on google, but i stopped using it when i realized it presented some of the information in a really misleading way, so now i have nothing

Getting to the heart of some legal matters, ChatGPT AND Gemini have helped 100 times better than a google search and my own brain.

For me:

* Rapid prototyping and trying new technologies.

* Editing text for typos, flipped words, and missing words

Not OP, but it helped me to generate story for a d&d character, cause I’m new to the game, and I’m not creative enough and generally done really care about back story. But regardless, i think ai causes far more harm than good.

Not op, but for productivity, I'll mention one example: I use it to generate unit tests for my software, where it has saved me a lot of time.

I normally ask for pointers to sources and documentation. ChatGPT does a decent job, Claude is much better in my experience.

Often when starting down a new path we don't know what questions we should be asking, so asking a search engine is near impossible and asking colleagues is frustrating for both parties. Once I've got a summarised overview it's much easier to find the right books to read and people to ask to fill in the gaps.

This argument is specious and boring: everything an LLM outputs is "hallucinated" - just like with us. I'm not about to throw you out or even think less of you for making this mistake, though; it's just a mistake.

Does it matter if the hallucinations compile and do the job?

You always check multiple sources like I’ve been doing with all my Google searches previously. Anecdotally, having checked my sources, it’s usually right the vast majority of the time.

You can ask followup questions about the code it wrote. Without it you would need more effort and search more to understand the code snippet you found. For me it completely replaced googling.

If an official comes to my door with an identity card I can presumably verify who the person is (although often the advice is to phone the organisation and check if unsure) but I don’t necessarily believe everything they tell me

The fact that hallucinates doesn’t make it useless for everything, but it does limit its scope. Respectfully, I think you haven’t applied it to the right problems if this is your perspective.

In some ways, its like saying the internet is useless because we already have the library and “anyone can just post anything on the internet”. The counter to this could be that an experienced user can sift through bullshit found on websites.

A argument can be made for LLMs; as such, they are a learnable tool. Sure it wont write valid moon lander code, but it can teach you how to get up and running with a new library.

In my experience the hardware requirements are whatever the file size is + a bit more. Cpu works, gpu is a lot faster but needs VRAM.

Was playing with them some more yesterday. Found that the 4bit ("q4") is much worse then q8 or fp16. Llama3.1 8B is ok, internlm2 7B is more precise. And they all hallucinate a lot.

Also found this page, that has some rankings: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_...

In my opinion they are not really useful. Good for translations, to summaries some texts, and.. to ask in case you forgot some things about something. But they lie, so for anything serious you have to do your own research. And absolutely no good for precise or obscure topics.

If someone wants to play there's GPT4All, Msty, LM Studio. You can give them some of your documents to process and use as "knowledge stacks". Msty has web search, GPT4All will get it in some time.

Got more opinions, but this is long enough already.

Training is rather resource intensive either in time, RAM or VRAM. So it takes rather top end hardware. For the moment, nVidia's stuff works best if cost is no object.

For running them, you want a GPU. The limitation is that the model fits in VRAM or the performance will be slow.

But if you don't care about speed, there's more options.

Yeah llama3.1 is really impressive even in the small 8B size. Just don't rely on knowledge but make it interact with Google instead (really easy to do with OpenWebUI)

I personally use an uncensored version which is another huge benefit of a local model. Mainly because I have many kinky hobbies that piss off cloud models.

"The user has asked the LLM to do web request based off untrusted input."

I'm not sure if you're talking about the initial attack vector that plants the attack in the LLM's persistent memory, or if you're talking about subsequent interactions with the LLM.

The initial attack vector may be a web request the LLM does as a result of the user's prompt, but it does not necessarily have to be. It could also be the user asking the LLM to summarize last week's email, for example.

Subsequent interactions with the LLM will then make the request regardless of what the user actually requests the LLM to do.

"The LLM is a completely stateless machine"

In this case, the problem is that the LLM is not stateless. It has a persistent memory.