If you're gonna go with a VS code extension and you're aiming for privacy, then I would at least recommend using the open source fork VS Codium.

https://vscodium.com/

Or https://ollama.com/

Well this was my experience...

    User: Hey, how are you?
    Llama: [object Object]

Thanks for your recommendation! I just ran Llamafile for the first time with a custom prompt on my Windows machine (i5-13600KF, RX6600) and found that it performed extremely slowly and wasn't as smart as ChatGPT. It doesn't seem suitable for productive writing. Did I do something wrong, or is there a way to improve its writing performance?

If anyone is interested in trying local AI, you can give https://recurse.chat/ a spin.

It lets you use local llama.cpp without setup, chat with PDF offline and provides chat history / nested folders chat organization, and can handle thousands of conversations. In addition you can import your ChatGPT history and continue chats with local AI.

Not only are they the only future worth living in, incentives are aligned with client-side AI. For governments and government contractors, plumbing confidential information through a network isn't an option, let alone spewing it across the internet. It's a non-starter, regardless of the productivity bumps stuff like Copilot can provide. The only solution is to put AI compute on a cleared individual's work computer.

Yeah I set up a local server with a strong GPU but even without that it's ok, just a lot slower.

The biggest benefits for me are the uncensored models. I'm pretty kinky so the regular models tend to shut me out way too much, they all enforce this prudish victorian mentality that seems to be prevalent in the US but not where I live. Censored models are just unusable to me which includes all the hosted models. It's just so annoying. And of course the privacy.

It should really be possible for the user to decide what kind of restrictions they want, not the vendor. I understand they don't want to offer violent stuff but 18+ topics should be squarely up to me.

Lately I've been using grimjim's uncensored llama3.1 which works pretty well.

idk, i have a pretty powerful laptop with 16GB VRAM and a 3080ti. when i've played with quantized llama2 and llama3 (with llamacpp), it was kinda underwhelming. inference was slow, the laptop would heat up and the results weren't as good. (is llama3.1 better?)

this was with 4bit quantization and offload of as many layers as possible to the gpu.

Many setup rely on Nvidia GPUs, Intel stuff, Windows or other stuff, that I would rather not use, or are not very clear about how to set things up.

What are some recommendations for running models locally, on decent CPUs and getting good valuable output from them? Is that llama stuff portable across CPUs and hardware vendors? And what do people use it for?

> For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it

I have now learned that my laptop is capable of a whopping 0.37 tokens per second.

11th Gen Intel® Core™ i7-1185G7 @ 3.00GHz × 8

https://old.reddit.com/r/LocalLLaMA/ is a great community for this sort of thing as well.

GPT4All for an even easier gui

https://github.com/nomic-ai/gpt4all

How do we rate whether the smaller models are any good? How many questions do we need to ask it to know that it can be trusted and we didn't waste our time on it?

Fully agree with you, it should, but after trying a few times different llamas I think they're very far from "try it out in just a few moments". Unless all you want is to see one running, for anything beyond that you'll be in dependency hell...

But how good are these models compared to gpt4o? My last experience with llama2-8b was not great at all. Are there really that good models that would fit on an average consumer hardware (mine has already 32GB ram and 16GB vram)?

I built Fluid app exactly with that in mind. You can run local AI on mac without really knowing what an LLM/ollama is. Plug&Play.

Sorry for the blatant ad, though I do hope it's useful for some ppl reading this thread: https://getfluid.app

I just tried now. Super easy indeed but slow to the point it's not usable on my PC

Isn’t there also some Firefox AI integration that’s being tested by one dev out there? I forgot the name and wonder if it got any traction.

Do you know if whisperfile is akin to whisper or the much better whisperx? Does it do diarization?

I recommend llmstudio for this usually

Same. My husky/pyr mix needs a lot of exercise, so I'm outside a minimum of a few hours a day. As a result I do a lot of dictation on my phone.

I put together a script that takes any audio file (mp3, wav), normalizes it, runs it through ggerganov's whisper, and then cleans it up using a local LLM. This has saved me a tremendous amount of time. Even modestly sized 7b parameter models can handle syntactical/grammatical work relatively easily.

Here's the gist:

https://gist.github.com/scpedicini/455409fe7656d3cca8959c123...

EDIT: I've always talked out loud through problems anyway, throw a BT earbud on and you'll look slightly less deranged.

Button-toggled voice notes in the iPhone Notes app are a godsend for taking measurements. Rather than switching your hands between probe/equipment and notes repeatedly, which sucks badly, you can just dictate your readings and maaaaybe clean out something someone said in the background. Over the last decade, the microphones + speech recognition became Good Enough for this. Wake-word/endpoint models still aren't there yet, and they aren't really close, but the stupid on/off button in the Notes app 100% solves this problem and the workflow is now viable.

I love it and I sincerely hope that "Apple Intelligence" won't kill the button and replace it with a sub-viable conversational model, but I probably ought to figure out local whisper sooner rather than later because it's probably inevitable.

This has inspired me.

I do a lot of stargazing and have experimented with voice memos for recording my observations. The problem of course is later going back and listening to the voice memo and getting organized information out of what essentially turns into me rambling to myself.

I'm going to try to use whisper + AI to transcribe my voice memos into structured notes.

For people on macOS, the free app Aiko on the App Store makes it easy to use Whisper, if you want a GUI: https://sindresorhus.com/aiko

I would be greatly interested in knowing how you set all that up if you felt like sharing the specifics.

This is exactly why I think the AI pins are a good idea. The Humane pin seems too big/too expensive/not quite there yet, but for exactly what you're doing, I would like some type of brooch.

> It feels super strange to talk to yourself

I remember the first lecture in the Theory of Communication class where the professor introduced the idea that communication by definition requires at least two different participants. We objected by saying that it can perfectly be just one and the same participant (communication is not just about space but also time), and what you say is a perfect example of that.

"before having an LLM clean up my ramblings into organized notes and todo lists."

Which local LLM do you use?

Edit:

And self talk is quite a healthy and useful thing in itself, but avoiding it in public is indeed kind of necessary, because of the stigma

https://en.m.wikipedia.org/wiki/Intrapersonal_communication

YES. I discovered this exact use case myself a few months ago. Tweeted about it even:

https://x.com/adpirz/status/1823727814191014323?s=46

I was thinking about making this the other day. Would you mind sharing what you used?

What software did you use to set all this up? Kindof interested in giving this a shot myself.

What do you use to run whisper locally? I don't think ollama can do it.

Can you give any more details on your setup? This sounds great

I found one that can isolate speakers, its just okay at that

According to Tom's (https://www.tomshardware.com/pc-components/cpus/amd-pushes-r...), those are supposed to be laptop CPUs, which makes me wonder what AMD has planned for us desktop users.

It will have around 250GB/s of bandwidth which makes it nearly unusable for 70b models. So the high amount of RAM doesn’t help with large models.

[dead]

Also, next year, there will be GPT 5. I find it fascinating how much attention small models get, when at the same time the big models just get bigger and prohibitively expensive to train. No leading lab would do that if they thought it a decent chance that small models were able to compete.

So who will be interested in a shitty assistant next year when you can have an amazing one, is what I wonder? Is this just the biggest cup of wishful thinking that we have ever seen?

It's not an open source licence, but to Meta's credit it's also not a licence you have to sell your soul to use. No more than 700 million monthly active users without requesting a licence, and don't use the model for illegal purposes and follow an AUP. It's as good as open source if you're using the models locally. If you're serious about building a commercial product with llama the AUP is almost certainly going to line up with your own terms of service for your product. Compare this with other models that are also billed as 'open source':

yi: previously non-commercial but Apache 2.0 now?

deepseek: usage policy

larger gemma models: usage policy

databricks: similar to llama: no more than 700 million MAUs, usage policy, additional restrictions on using outputs to train other models

qwen: no more than 100 million MAUs

mistral: non-commercial

command-r: CC BY-NC

starling: CC BY-NC

There are a handful of niche models released under MIT/Apache but the norm is licences similar to or more restrictive than the Llama Community Licence, and I really doubt the situation would be better if Meta wasn't first.

>"open-weights" LLMs

I doubt this is the point you're making, but the training data really isn't useful even if it could be released under a permissive licence. Most models use similar datasets: reddit (no licence afaik, copyright belongs to comment authors), stackoverflow (CC BY-SA), wikipedia (CC BY-SA), Project Gutenberg (public domain?), previously books3 (books under copyright by publishers with more money and lawyers than reddit users), etc, with various degrees of filtering to remove harmful data. You can't do much with this much data unless you have millions of dollars worth of compute laying around, and you can't rebuild llama any more than any other company using the same data have 'rebuilt llama' - all models trained in a similar manner on the same data are going to converge in outputs eventually. Compare with Linux distributions, they all use the same packages but you're not going to get the same results.

that ship sailed 13 years ago dude.

A small model necessarily is missing many facts. The large model is the one that has memorized the whole internet, the small one is just trained to mimic the big one.

You simply cannot compress the whole internet under 10gb without throwing out a lot of information.

Please be careful about what you take as fact coming from the local model output. Small models are better suited to summarization.

I'm really curious what you are doing with an LLM that can be solved 80% of the time with a 8b model.

We need browser and OS level API (mobile) integration to the local LLM.

There's actually a very popular plugin for Obsidian that integrates RAG + LLM into Obsidian called Smart Connections.

https://github.com/brianpetro/obsidian-smart-connections

Ollama is not a model, it is the sofware to run models.

which model are you using? what size/quant/etc?

thanks!

you moved the goalposts when you add 'multimodal' there; another item is, no one reads PDF tables and illustrations perfectly, at any price AFAIK

Look into Microsoft's Phi papers. The whole idea here is that if you train models on higher quality data (i.e. textbooks instead of blogspam) you get higher quality results.

The exact training is proprietary but they seem to use a lot of GPT-4 generated training data.

On that note... I've often wondered if broad memorization of trivia is really a sensible use of precious neurons. It seems like a system trained on a narrower range of high quality inputs would be much more useful (to me) than one that memorized billions of things I have no interest in.

At least at the small model scale, the general knowledge aspect seems to be very unreliable anyways -- so why not throw it out entirely?

Synthetic data (data from some kind of generative AI) has been used in some form or another for quite some time[0]. The license for LLaMA 3.1 has been updated to specifically allow its use for generation of synthetic training data. Famously, there is a ToS clause from OpenAI in terms of using them for data generation for other models but it's not enforced ATM. It's pretty common/typical to look through a model card, paper, etc and see the use of an LLM or other generative AI for some form of synthetic data generation in the development process - various stages of data prep, training, evaluation, etc.

Phi is another really good example but that's already covered from the article.

[0] - https://www.latent.space/i/146879553/synthetic-data-is-all-y...

As others point out, it's essentially distillation of a larger model to a smaller one. But you're right, it doesn't work very well. Phi's performance is high on benchmarks but not nearly as good in actual real world usage. It is extremely overfit on a narrow range of topics in a narrow format.

I would guess correctly aligned and/or finely filtered synthetic data coming from LLMs may be good.

Mode colapse theories (and simplified models used as proof of existence of said problem) assume affected LLMs are going to be trained with poor quality LLM-generated batches of text from the internet (i.e. reddit or other social networks).

That's the number one way of getting mad LLM disease. Feeding LLMs to LLMs.

Generally (not just for LLMs) this is called student-teacher training and/or knowledge distillation.

There's been efforts to train small LLM's on bigger LLM's. Ever since Llama came out the community was creating custom fine tunes this way using ChatGPT.

I think it can be a tradeoff to get to smaller models. Use larger models trained on the whole internet to produce output that would train the smaller model.

> Microsoft used LLMs to write millions of short stories and textbooks

Millions? Where are they? Where are they used?

> in the next eight months when roughly 1.5 billion iPhone users will get a local language model tied seamlessly to a mid-tier cloud based language model native in their OS.

Only iPhone 15 Pro or later will get Apple Intelligence, so the number will be wayyy smaller.

I expect people will just ship with their own model where the built-in one isn't sufficient.

When people describe it as a "critical tool" i feel like I'm missing basic information about how people use computers and interact with the world. In what way is it critical for anything? It's still just a toy at this point.

405B is beyond homelab-scale. I recently obtained a 4x4090 rig, and I am comfortable running 70B and occasionally 128B-class models. For 405B, you need 8xH100 or better. A single H100 costs around $40k.

It looks good. Thank you for sharing. Is anyone aware of a similar tool for Linux?

M4s are releasing in probably a month or two; if you’re going Apple, it might be worth waiting for either those or the price drop on the older models.

You basically need as much RAM as the size of the model.

I like [mistral-nemo](https://ollama.com/library/mistral-nemo) "A state-of-the-art 12B model with 128k context length, built by Mistral AI in collaboration with NVIDIA."

I like the latest qwen2.5 (https://nexaai.com/Qwen/Qwen2.5-0.5B-Instruct/gguf-q4_0/read...). It was just released last week. It is one of the best small langauge models right now according to benchmarks. And it is small and fast!

Qwen 2.5 has just released, with a surprising amount of sizes. The 14B and 32B look pretty promising for their size class but it's hard to tell yet.

Nada to be honest. I keep trying every new model, and invariably go back to llama 8b.

Llama8b is the new mistral.

codestral 22b, aya-23 35b, gemma2 27b-instruct.

While that's true for their ChatGPT SaaS, the API they provide doesn't impose as many restrictions.

"All you need is £10k of Apple laptops..."

Imho opinion, and I'm no expert, but this has been working well for me:

Segment the texts into chunks that make sense (i.e. into the lengths of text you'll want to find, whether this means chapters, sub-chapters, paragraphs, etc), create embeddings of each chunk, and store the resultant vectors in a vector database. Your search workflow will then be to create an embedding of your query, and perform a distance comparison (e.g. cosine similarity) which returns ranked results. This way you can now semantically search your texts.

Everything I've mentioned above is fairly easily doable with existing LLM libraries like langchain or llamaindex. For reference, this is an RAG workflow.

Look into this: https://www.anthropic.com/news/contextual-retrieval

And this: https://microsoft.github.io/graphrag/

https://khoj.dev promises this.

Look into AnythingLLM.

Gemma2 27b instruct:

Generate documentation in Rust format for use in code.

  /// An enum representing possible errors during CRUD operations.
  #[derive(Error, Debug)]
  enum CrudError {
    /// Indicates an error occurred while interacting with the SQLite database.
    /// 
    /// This variant wraps a `rusqlite::Error`, providing details about the specific SQLite error.
    #[error("SQLite error: {0}")]
    SqliteError(rusqlite::Error),

    /// Indicates an error related to input/output operations.
    /// 
    /// This variant wraps a `std::io::Error`, giving information about the specific IO problem encountered.
    #[error("IO error: {0}")]
    IoError(std::io::Error),
  }

The Continue extension for VSCode is pretty good and has native connectivity to a local install of Ollama

I use gen.nvim[1] with for small tasks, like “write a type definition for this JSON” .

Running locally avoids the concern of sending IP or PII to third parties.

[1]: https://github.com/David-Kunz/gen.nvim

There's plenty of competition in this space already, and it'll only get accelerated with time. There's not enough "moat" in building proprietary LLMs - you can tell by how the leading companies in this space are basically down to fighting over patents and regulatory capture (ie. mounting legal and technical barriers to scraping, procuring hardware, locking down datasets, releasing less information to the public about how the models actually work behind the scenes, lobbying for scary-yet-vague AI regulation, etc).

It's fizzling out.

The current incumbents are sitting on multi-billion dollar valuations and juicy funding rounds. This buys runtime for a good couple of years, but it won't last forever. There's a limit to what can be achieved with scraped datasets and deep Markov chains.

Over time, it will become difficult to judge what makes one general-purpose LLM be any better than another general-purpose LLM. A new release isn't necessarily performing better or producing better quality results, and it may even regress for many use-cases (we're already seeing this with OpenAI's latest releases).

Competitors will have caught up to eachother, and there shouldn't be any major differences between Claude, ChatGPT, Gemini, etc - after-all, they should all produce near-identical answers, given identical scenarios. Pace of innovation flattens out.

Eventually, the technology will become wide-spread, cheap and ubiquitous. Building a (basic, but functional) LLM will be condensed down to a course you take at university (the same way people build basic operating systems and basic compilers in school).

The search for AGI will continue, until the next big hype cycle comes up in 5-10 years, rinse and repeat.

You'll have products geared at lawyers, office workers, creatives, virtual assistants, support departments, etc. We're already there, and it's working great for many use-cases - but it just becomes one more tool in the toolbox, the way Visual Studio, Blender and Photoshop are.

The big money is in the datasets used to build, train and evaluate the LLMs. LLMs today are only as good as the data they were trained on. The competition on good, high-quality, up-to-date and clean data will accelerate. With time, it will become more difficult, expensive (and perhaps illegal) to obtain world-scale data, clean it up, and use it to train and evaluate new models. This is the real goldmine, and the only moat such companies can really have.

Their hope is to reach AGI and effective post-scarcity for most things that we currently view as scarce.

I know it sounds crazy but that is what they actually believe and is a regular theme of conversations in SF. They also think it is a flywheel and whoever wins the race in the next few years will be so far ahead in terms of iteration capability/synthetic data that they will be the runaway winner.

I don't have a horse in the race but wouldn't Meta be more likely to commoditize things given that they sort of already are?

Edit: I just found this. I'll give it a try today: https://github.com/0ssamaak0/SiriLLama

---

Open WebUI has a voice chat but the voices are not great. I'm sure they'd love a PR that integrates StyleTTS2.

You can give it a Serper API Key and it will search the web to use as context. It connects to ollama running on a linux box with a $300 RTX 3060 with 12GB of VRAM. The 4bit quant of Llama 3.1 8B takes up a bit more than 6GB of VRAM which means it can run embedding models and STT on the card at the same time.

12GB is the minimum I'd recommend for running quantized models. The RTX 4070 Ti Super is 3x the cost but 7 times "faster" on matmuls.

The AMD cards do inference OK but they are a constant source of frustration when trying to do anything else. I bought one and tried for 3 months before selling it. It's not worth the effort.

I don't have any interest in allowing it to run shortcuts. Open WebUI has pipelines for integrating function calling. HomeAssistant has some integrations if that's the kind of thing you are thinking about.

Apple intelligence?

IDK, 8B-class quantized models run pretty fast on commodity laptops, with CPU-only inference. Thanks to the people who figured out quantization and reimplemented everything in C++, instead of academic-grade Python.

I've not yet used any local AI, so I'm curious - what are you getting? Can you share examples?

Yes.

Either get a Mac with loads of mem or build a rig with a 3090 or three

https://groq.com

is very fast.

(this is not the same as Grok)

1) Offline connectivity — pretty cool to be able to debug technical problems while flying (or otherwise off grid) with a local LLM, and current 8B models are usually good enough for the first line of questions that you otherwise would have googled.

2) Privacy

3) Removing safety filters — there are some great “abliterated” models out there that have had their refusal behavior removed. Running these locally and never having your request refused due to corporate risk aversion is a very different experience to calling a safety-neutered API.

Depending on your use case some, all, or none of these will be relevant, but they are undeniable benefits that are very much within reach using a laptop and the current crop of models.

I agree. Even bi-lingual (English+1) small models would be very useful to process localized data, for ex english-french, english-german, etc.

Right now the small models (llama 8B) can't handle this type of task, although they could if they were trained for bi lingual data.

No ridiculous prohibitions on training on logs…

Man, imagine being OpenAI and flushing your brand down the toilet with an explicit customer noncompete rule which totally backfires and inspires 100x more competition than it prevents

An AI chip on laptop devices would be amazing!

I don't think local LLM's are being marketed "for the casual user", nor do I think the casual user will care at all about running LLM's locally so I am not sure why this comparison matters.

Yep, unfortunately those local models are noticeably worse. Also models are getting bigger, so even if a local basement rig for a higher quality model is possible right now, that might not be so in the future. Also Zuck and others might stop releasing their weights for the next gen models, then what, just hope they plateau, what if they don't?

they are the only thing you can use if you don't want to or aren't allowed to hand over your data to US corporations and intelligence agencies.

every single query to ChatGPT/Claude/Gemini/etc will be used for any purpose, by any party, at any time. shamelessly so, because this is the new normal. Welcome to 2024. I own nothing, have no privacy, and life has never been better.

>(and less useful, even illegal) things

the same illegal things you can do with Notepad, or a pencil and a piece of paper.