If you are in US, you get 1 billion tokens a DAY with Gemini (Google) completely free of cost.

Gemini Flash is fast with upto 4 million token context.

Gemini Flash 002 improved in math and logical abilities surpassing Claude and Gpt 4o

You can simply use Gemini Flash for Code Completion, git review tool and many more.

I saw that you mention https://github.com/simonw/llm/. Hadn't seen this before. What is its purpose? And why not use ollama instead?

Hi simon, is there a way to run the vision model easily on my mac locally?

Llama 3.2 vision models don't seem that great if they have to compare them to Claude 3 Haiku or GPT4o-mini. For an open alternative I would use Qwen-2-72B model, it's smaller than the 90B and seems to perform quite better. Also Qwen2-VL-7B as an alternative to Llama-3.2-11B, smaller, better in visual benchmarks and also Apache 2.0.

Molmo models: https://huggingface.co/collections/allenai/molmo-66f379e6fe3..., also seem to perform better than Llama-3.2 models while being smaller and Apache 2.0.

What are people using to check token length of code bases? I'd like to point certain app folders to a local LLM, but no idea how that stuff is calculated? Seems like some strategic prompting (eg: this is a rails app, here is the folder structure with file names, and btw here are the actual files to parse) would be more efficient than just giving it the full app folder? No point giving it stuff from /lib and /vendor for the most part I reckon.

The llama 3.0, 3.1, & 3.2 all use the TikToken tokenizer which is the open source openai tokenizer.

This obsession with using AI to help with programming is short sighted.

We discover gold and you think of gold pickaxes.

Zuckerberg has never liked having Android/iOs as gatekeepers i.e. "platforms" for his apps.

He's hoping to control AI as the next platform through which users interact with apps. Free AI is then fine if the surplus value created by not having a gatekeeper to his apps exceeds the cost of the free AI.

That's the strategy. No values here - just strategy folks.

As the Google memo (https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...) pointed out, a lot of OSS stuff/improvements are being built on top of Meta's work which somewhat benefits them as well.

But still, Kudos to Zuck/Meta for doing it anyway.

Zuckerberg probably realises the value of currying favour with engineers. Also, I think he has a personal vendetta to compete with Musk in this space.

Maybe it's cynical to think that way but maybe it's a way to crush the competition before it even begins: I would probably not invest in researching LLMs now, knowing that there is a company that will very likely produce a model close enough for free and I will likely never make back the investment.

Meta has been good about releasing their NLO work open source for a long time. Most of the open source datasets for foreign language translation were created by Facebook.

They have a hose of ad money and have nothing to lose doing this.

You can’t say that for the other guys.

They're out to fuck over the competition by killing their moat. Classic commoditize your complement.

Training data is crucial for performance and they do not (cannot) share that.

Fully second that.

Do they tell you what training data they use for alignment? As in, what biases they intentionally put in the system they’re widely deploying?

They are literally training on all the free personal data you provided, so they owe you this much

It's a great tool. Thanks!

I had to test it with Llama3.1 and was really easy. At a first glance Llama3.2 didn't seem available. The command you provided did not work, raising "An error occurred while pulling the model: not enough values to unpack (expected 2, got 1)".

or grab lmstudio

>WRONG! ENJOY MORE PENALTY, SCRUB!

Is that true tho? During training, the model predicts {"wall": 0.65, "fence": 0.25, "river": 0.03}. Then backprop modifies the weights such that it produces {"wall": 0.67, "fence": 0.24, "river": 0.02} next time.

But it does that with a much richer feedback than WRONG! because we're also telling the model how much more likely "fence" is than "wall" in an indirect way. It's likely most of the neurons that supported "wall" also supported "fence", so the average neuron that supported "river" gets penalised much more than a neuron that supported "fence".

I agree that distillation is more efficient for exactly the same reason, but I think even models as old as GPT-3 use this trick to work as well as they do.

Isn't jumping over a fence more likely than jumping over a wall?

I would have went with “moon”

Moat

yeah i mean that is exactly why distillation works. if you just were one hotting it would be the same as training on same dataset

They don't, they're playing "hide the #s" a bit. Llama 3.2 3B is definitively worse than Phi-3 from May, both on any given metric and in an hour of playing with the 2, trying to justify moving to Llama 3.2 at 3B, given I'm adding Llama 3.2 at 1B.

We're working on it. There are already draft PRs up in the GH repo. We're still working out some kinks though.

As a rule of thumb with AI stuff: it either works instantly, or wait a day or two.

ollama is "just" llama.cpp underneath, I recommend switching to LM Studio or Jan, they don't have this issue of proprietary wrapper that obfuscates, you can just use any ol GGUF

My understanding is the way the tokenization works prevents the LMM from being able to count occurrences of words or individual characters.

Does anyone know of a CoT dataset somewhere for finetuning? I would think exposing it to that type of modality during a finetune/lora would help.

What is the "think step by step" prompt? An example would be great, Is this part of the system prompt?

Of course, in many contexts, it is correct to put 9.11 after 9.9--software versioning does it that way, for example.

I question whether a 3B model can have “a lot of knowledge”.

Can it speak foreign languages like German, Spanish, Ancient Greek?

Aren't the _0 quantizations considered deprecated and _K_S or _K_M preferable?

https://github.com/ollama/ollama/issues/5425

On what basis do you use these different models?

For a second I read that as “it just removed my install of 3.1-8b” :D

https://artificialanalysis.ai/leaderboards/models

"LLM Leaderboard - Comparison of GPT-4o, Llama 3, Mistral, Gemini and over 30 models

Comparison and ranking the performance of over 30 AI models (LLMs) across key metrics including quality, price, performance and speed (output speed - tokens per second & latency - TTFT), context window & others. For more details including relating to our methodology, see our FAQs."

The Llama 3.2 11B multimodal model is a bit less than 14B but smaller models can do more these days, and Meta are not the only ones making models. The 70B model has been pruned down by NVIDIA if I recall correctly. The 405B model also will be shrunk down and can presumably be used to strengthen smaller models. I'm not convinced by your shiny hat.

You don't need an F-15 to play at least, a decent sniper rifle will do. You can still practise even with a pellet gun. I'm running 70b models on my M2 max with 96 ram. Even larger models sort of work, although I haven't really put much time into anything above 70b.

4090 has 24G

So we really need ~40B or G model (two cards) or like a ~20B with some room for context window.

5090 has ??G - still unreleased

Doesn't require a GPU, it will just be faster with a GPU.

Looking at their benchmark results and my own experience with their 11B vision model, I think while not perfect they represent the model well.

Meaning it's doing impressively bad compared to other models I've tried in similar sizes(for vision).

I use Continue on VScode, works well with Ollama and llama3.1 (but obviously not as good as Claude).

They mention tool calling in the link for the smaller models, and compare to 8B levels of function calling in benchmarks here:

https://news.ycombinator.com/item?id=41651126

Yes (rationale: 3.1 was, would be strange to rollback.)

In general, you'll do a ton of damage by constraining token generation to valid JSON - I've seen models as small as 800M handle JSON with that. It's ~impossible to train constraining into it with remotely the same reliability -- you have to erase a ton of conversational training that makes it say ex. "Sure! Here's the JSON you requested:"

yes, but only the text-only models!

https://www.llama.com/docs/model-cards-and-prompt-formats/ll...

Open webui has promising aspects, the same authors are pushing for "pipelines" which are a standard for how inputs and outputs are modified on the fly for different purposes.

ive been using https://github.com/valiantlynx/ollama-docker which comes with https://github.com/open-webui/open-webui

openwebui

https://get.big-agi.com/

Because they trained the text model. Then froze the weights. Then trained a vision model on text image pairs of progressively higher quality. Then trained an adapter to align their latent spaces. So it became smart on text then gain a new input sense magically without changing its weights

Oh, this is promising. It's not surprising to me: image models have been very oriented towards photography and scene understanding rather than understanding symbolic information in images (like text or diagrams), but I always thought that it should be possible to make the model better at the latter, for instance by training it more on historical handwritten documents.

How good it is at comic reading?

magic

Doesn’t everyone? X) it’s super easy now with ollama + openwebui or an all in 1 like mlstudio

For sure dude! Top comment thread is all about using ollama and other ways to get that done.

Absolutely! They have a billion-parameter model that will run on my first computer if we quantize it to 1.5 bits. But realistically yes, if you can fit in total ram you can run it slowly, if you can fit it in gpu ram you can probably run it fast enough to chat.

The 8B models run fine on a M1 pro 16GB.

Hoping to get this out soon w/ Ollama. Just working out a couple of last kinks. The 11b model is legit good though, particularly for tasks like OCR. It can actually read my cursive handwriting.

depends on your phone, but try a couple of these variants with ollama https://ollama.com/library/llama3.2/tags

e.g. `ollama run llama3.2:1b-instruct-q4_0`

It all depends on the amount of RAM. Since you're not mentioning it, you probably have only the default 8GB? Still, the 3B should run even on an 8GB machine.

apparently so, but im trying to find a working example / some details on what specific iOS / android devices are capable of running this

was surprised by this as well

You can run with LitServe. here is the code - https://lightning.ai/lightning-ai/studios/deploy-llama-3-2-v...

Yes, vLLM does (though marked experimental): https://docs.vllm.ai/en/latest/models/vlm.html

Wildly incorrect

If you are still curious about the architecture, from the blog:

> To add image input support, we trained a set of adapter weights that integrate the pre-trained image encoder into the pre-trained language model. The adapter consists of a series of cross-attention layers that feed image encoder representations into the language model. We trained the adapter on text-image pairs to align the image representations with the language representations. During adapter training, we also updated the parameters of the image encoder, but intentionally did not update the language-model parameters. By doing that, we keep all the text-only capabilities intact, providing developers a drop-in replacement for Llama 3.1 models.

What this crudely means is that they extended the base Llama 3.1, to include image based weights and inference. You can do that if you freeze the existing weights. add new ones which are then updated during training runs (adapter training). Then they did SFT and RLHF runs on the composite model (for lack of a better word). This is a little known technique, and very effective. I just had a paper accepted about a similar technique, will share a blog once that is published if you are interested (though it's not on this scale, and probably not as effective). Side note: That is also why you see param size of 11B and 90B as addition from the text only models.

I hereby grant license to anyone in the EU to do whatever they want with this.

Full text:

https://github.com/meta-llama/llama-models/blob/main/models/...

https://github.com/meta-llama/llama-models/blob/main/models/...

> With respect to any multimodal models included in Llama 3.2, the rights granted under Section 1(a) of the Llama 3.2 Community License Agreement are not being granted to you if you are an individual domiciled in, or a company with a principal place of business in, the European Union. This restriction does not apply to end users of a product or service that incorporates any such multimodal models.

Oh. That's sad indeed. What might be the reason for excluding Europe?

Again, we see that Llama is totally open source! Practically BSD licensed!

So the issue is privacy:

https://www.itpro.com/technology/artificial-intelligence/met...

"Meta aims to use the models in its platforms, as well as on its Ray-Ban smart glasses, according to a report from Axios."

I suppose that means that Ray Ban smart glasses surveil the environment and upload the victim's identities to Meta, presumably for further training of models. Good that the EU protects us from such schemes.

Pity, it's over. We'll never ever be able to download those ten gigabytes files, at the other side of the fence.

Fortunately, Qwen-2-VL exists, it is pretty good and under an actual open source license, Apache 2.0.

Edit: the larger 72B model is not under Apache 2.0 but https://huggingface.co/Qwen/Qwen2-VL-72B-Instruct/blob/main/...

Qwen2-VL-72B seems to perform better than llama-3.2-90B on visual tasks.

Agents are the new Apps

I mean, just because he is not doing this as a perfectly altruistic gesture does not mean the broader ecosystem does not benefit from him doing it

The main issue was that Meta quickly took down the first announcement, and the only remaining working submission was the information-sparse HuggingFace link. By the time the other links were back up, it was too late. Perfect opportunity for a rescue.

Yeah I submitted what turned out to be a dupe but I could never find the original, probably was buried at the time. Then a few hours later it miraculously (re?)appeared.

AIUI exact dupes just get counted as upvotes, which hasn’t happened in my case.

Both Llama 3.2 90B and Claude 3.5 Sonnet can find "turkey" and "spoon", probably because they're left-to-right. Llama gave approximate locations for each and Claude gave precise but slightly incorrect locations. Further prompting to look for diagonal and right-to-left words returned plausible but incorrect responses, slightly more plausible from Claude than Llama. (In this test I cropped the word search to just the letter grid, and asked the model to find any English words related to soup.)

Anyways, I think there just isn't a lot of non-right-to-left English in the training data. A word search is pretty different from the usual completion, chat, and QA tasks these models are oriented towards; you might be able to get somewhere with fine-tuning though.

Try and find where the words are in this word puzzle undefined

''' There are two words in this word puzzle: "soup" and "mix". The word "soup" is located in the top row, and the word "mix" is located in the bottom row. ''' Edit: Tried a bit more probing like asking it to find spoon or any other word. It just makes up a row and column.

Non US citizens can access the model just fine, if that's what you are implying.

This is likely because the models use OCR on images with text, and once parsed the word search doesn't make sense anymore.

Would be interesting to see a model just working on raw input though.

No, not yet. And such LLM wouldn't speak back in English or French without some "knowledge of the human kind" as you put it.

Most code is grounded in real-world concepts somehow. Imagine an engineer at Domino's asking it to write an ordering app. Now your model needs to know what goes in to a pizza.

Not yet. But, Nvidia's CEO announced a few months ago that we're about 5 years away. And, OpenAI just this week announced Super Intelligence is up to 2000 days (e.g. around 5 years) away.

Try codegemma.

Or Gemini Flash for code completion and generation.

10x relative to what? I’ve seen bad developers use AI to 10x their productivity but they still couldn’t come anywhere close to a good developer without AI (granted, this was at a hackathon on pretty advanced optimization research. Maybe there’s more impact on lower skilled tasks)

So long as you don't mind glue in your pizza...