I wasn't scientific about it, unfortunately. My searches were natural language, not token-based, though.

yeah, when i saw that they have 100% coverage with 1M token, i thought this must be a placeholder image, for when the actual results come in.

Because there is no variation, nothing.

After reading a lot of that thread, my understanding is that yarn scaling is disabled intentionally by default in the GGUFs, because it would degrade outputs for contexts that do fit in 32k. So the only change is enabling yarn scaling at 4x, which is just a configuration setting. GGUF has these configuration settings embedded in the file format for ease of use. But you should be able to override them without downloading an entire duplicate set of weights (12 to 35 GB!). (It looks like in llama.cpp the override-kv option can be used for this, but I haven't tried it yet.)

Yeah unfortunately that's the exact model I'm using (Q5 version. What I've been doing is first loading the transcript into the vector database, and then giving it a prompt thats like "summarize the transcript below: <full text of transcript>". This works surprisingly well except for one transcript I had which was of a 3 hour meeting that was per an online calculator about 38,000 tokens. Cutting the text up into 3 parts and pretending each was a seperate meeting* lead to a bunch of hallucinations for some reason.

*In theory this shouldn't matter much for my purpose of summarizing city council meetings that follow a predictable format.

Yeah describing Luo Ji as having "struggles with the ethical implications of his mission" is the biggest whopper.

He's like God's perfect sociopath. He wobbles between total indifference to his mission and interplanetary murder-suicide, and the only things that seem to really get to him are a stomachache and being ghosted by his wife.

This problem is poorly defined; what would it mean to produce a response JUST based on the text given? Should it also forgo all logic skills and intuition gained in training because it is not in the text given? Where in the N dimensional semantic space do we draw a line (or rather, a surface) between general, universal understanding and specific knowledge about the subject at hand?

That said, once you have defined what is required, I believe you will have solved the problem.

LLMs are probably better than you at tasks you're not good at. There's a huge gulf between a domain expert and an LLM though. If there weren't, all the humans in companies would be fired right now and replaced. Similarly, OpenAI and Anthropic are paying engineers a metric fuckton of money to work there. If LLMs were that big of a game changer right, they wouldn't be paying that much. Or if you make the argument that only the best humans are getting hired, they're still hiring interns & junior engineers. If that were the case those would be being replaced by LLMs and they're not.

You're basically ignoring all the experts saying "LLMs suck at all these things that even beginning domain experts don't suck at" to generate your claim & then ignoring all evidence to the contrary.

And you're ignoring the ways in which LLMs fall on their face to be creative that aren't language-based. Creative problem solving in ways they haven't been trained on is out of their domain while fully squarely in the domain of human intelligence.

> You can claim that that's not intelligence until the cows come home, but any person able to do that would be considered a savant

Computers can do arithmetic really quickly but that's not intelligence but a person computing that quickly is considered a savant. You've built up an erroneous dichotomy in your head.

I would argue the opposite actually. The same way we don't call someone who are able to do arithmetic calculations very fast a genius if they can't think in more useful mathematical way and construct novel ideas. The samething is happening here, these tools are useful in retrieving and processing current information at high speeds but intelligence is not about the ability to process some data at high speed and then recall them. This is what we actually call servant. It is the ability to build on top this knowledge retrieval and use reason to create new ideas is a closer definition of intelligence and would be better goal.

Go ask your favorite LLM to write you some code to implement the backend of the S3 API and see how well it does. Heck, just ask it to implement list iteration against some KV object store API and be amazed at the complete garbage that gets emitted.

Do you have any evidence besides anecdote?

But you just made a strong claim about something you are here saying we can't know?

> This means there are operations that are VERY HARD for the model like ranking/sorting. This requires the model to attend to everything to find the next biggest item, etc. It is very hard for the models currrently.

Ranking / sorting is O(n log n) no matter what. Given that a transformer runs in constant time before we 'force' it to output an answer, there must be an M such that beyond that length it cannot reliably sort a list. This MUST be the case and can only be solved by running the model some indeterminate number of times, but I don't believe we currently have any architecture to do that.

Note that humans have the same limitation. If you give humans a time limit, there is a maximum number of things they will be able to sort reliably in that time.

So are they human like and therefore not anything special or are they super human magic? I never get the equivocation when people complain how there is no way to objectively tell what out is right or wrong people either say they are getting better, or they work for me, or that people are just as bad. No they aren't! Not in the same way these things are bad.

I will wave my arms wildly if the claim is that LLM struggle with recall is similar to human-like struggle with recall. And since that's how we decide on truth, I win?

Seems more likely that your vacations and fridge contents aren't as novel as you hope.

> ChatGPT has synthesized my past three vacations and regularly plans my family's meals based on whatever is in my fridge. I completely disagree.

What do you mean synthetised your vacations? You seem to be very passionate with ChatGPT to entrust it with making decisions on your meals but I suspect the novelty will wear off for you too. I use ChayGPT from time to time and admit Im pleasantly surprised by the outcome when it all works out, but Im not swayed to think it is reasoning the way we do, it’s just a tool after all.

> I think this is it. LLM responses feel like the unconsidered ideas that pop into my head from nowhere.

You have to look at the LLM as the inner voice in your head. We've kind of forced them into saying whatever they think due to how we sample the output (next token prediction), but in new architectures with pause tokens, we let them 'think' and they show better judgement and ability. These systems are rapidly going to improve and it will be very interesting to see.

But this is another reason why I think they've surpassed human intelligence. You have to look at each token as a 'time step' in the inner thought process of some entity. A real 'alive' entity has more 'ticks' than what their actions would suggest. For example, human brains can process up to 10FPS (100ms response time), but most humans aren't saying 10 words a second. However, we've made LLMs whose internal processes (i.e., their intuition) is already superior. If we just gave them that final agentic ability to not say anything and ponder (which researchers are doing), their capabilities will increase exponentially

> The other thing that makes me feel like we're 'getting there' is using some of the fast models at groq.com.

Unlike perhaps many of the commentators here, I've been in this field for a bit under a decade now, and was one of the early compiler engineers at Groq. Glad you're finding it useful. It's amazing stuff.

>When I was at Cerebras, I fed in a description of the custom ISA into our own model and asked it to generate kernels (my job), and it was surprisingly good

And? Was it actually better then say the top 3 people in this field would create if they would work on it ? Because this models are better at css then me, so what? I am bad at css, but all the top models could not solve a math limit from my son homework so we had to use good old forums to have people give us some hints. But for sure models can solve more math limits then the average person who probably can't solve a single one.