It's entirely reproducible from the available documentation (which is why you see vLLM, SGLang, MLX etc all racing to produce optimized implementations).

(As an aside, this is why the "open weights are not open source" thing is a complete misunderstanding. The weights themselves along with the documentation give you enough to fine tune the LLM. You can't rebuild it from scratch, but you can't do this even with the data anyway (because of randomness!))

Implementing models directly from papers is typically pretty doable (and is of course more straightforward when the full implementation is open sourced). Often there is some amount of specific knowledge, like particular hyperparameters, that is missing and has to be trial and errored by the community, but generally speaking, getting the core model architecture implemented is a reasonable task for most well documented models.

Reproducing the exact training run, however, is basically impossible without the original dataset and training pipeline (here meaning all of the code + infra involved in actually executing the pre and post training loops). Also, it would be exorbitantly expensive to do if you weren't also a lab trying to train a similar model.

But you can still scale the architecture down and experiment as a solo researcher using the published research. There are probably some open source implementations already on GitHub for any given big open model release.

The architectures are high level concepts and the mechanics usually have enough detail for you to try and implement.

Transformers are very "mendable" in that you can permute the architecture in crazy or random ways, and still basically always end up with get a coherent LLM. The difference comes down to training efficiency, inference efficiency, and usually minor differences in performance.

Hyperparams and stuff, I mean it's standard to do a sweep anyway.

Good question, it's 100%. I.e., the developers usually share a reference implementation with e.g., Hugging Face transformers to load their weights, and from there on you can read the code and, if you have time, reimplement and check everything. It's actually a great learning exercise where you can self-check whether you reimplemented it correctly by comparing the LLM outputs to the reference implementation. Made a video about that workflow a while back here: https://www.youtube.com/watch?v=TXzQ7PGpO6w

The code is open source, and there probably several different implementations.

[flagged]

This seems like a rare gem in a space swamped with lazily LLM generated summaries. Added to my RSS feed!

I would describe him as a leading educator / explainer / teacher in this space.

Never heard of him before but I was amazed how straight to the point yet very easy to grasp it was

I second that. I've always enjoyed his work!

The primary author of RoPE, Jianlin Su, is currently working at Moonshot (KIMI)

As a sibling comment points out you don't strictly need positional embeddings for decoder-only causal transformers. You definitely need it for non-causal ones (e.g. the encoder of the original transformer paper!).

And yes accumulation is a good intuition for what's going on. You could imagine a part of the attention head that just kept writing to the same part of the residual stream causing that to keep accumulating (simply via attention summation) as more input tokens come in thereby functioning as a kind of index without the need for any positional encoding.

Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.

Linear layers use decays (like IIR filters) that naturally provide relative positions. Full attention layers can then be free to develop concepts that attend to each other regardless of distance.

When you have recurrent blocks in your model, you implicitly have a timestep T(amount of recurrent steps). Similar to Diffusion Transformers, it then becomes valuable to encode the knowledge of where you are in this chain somehow. NoPE is more flexible than RoPE for this.

At least the SSM layers should have relative PE built in via recurrence, in a hybrid attention model like this.

They need to work on token cost for unrestrained problem solving.

The assertions doing the rounds that Kimi K3 and GLM 5.2 are way cheaper than Claude/GPT are not true -- DeepSeek V4 Pro is a lot cheaper but K3 and 5.2 ain't. Turns out that you actually have to fork out some cash for frontier-esque models, be they Chinese or American. Hope that helps.

Source: my bank balance

Oh no! First time dealing with an American corporation?

Better than Opus 4.8 on complex tasks but tends to overthink. It found a bunch of bugs and architecture issues that only 5.6 Sol Max and Fable on my C++ projects.

Agreed. Dropped my $200/mo Claude I was saturating for the $100/mo moonshot sub and have no complaints

Plus I can actually see the thinking traces and I’m not getting downgraded/denied by anthropic constsntly…

threat?

that's a weird way of describing a near frontier open weights un-crippled useful coding buddy

do you work for OpenAI or Anthropic per chance?

Even if they are distilling, I don't particularly care. Anthropic and others have been distilling copyrighted material by to build these models, largely without permission.

No! It's just not fair!! What about our financial bubble!

Moreover, Distillation is a misused term here. Distillation means training a smaller student model using a larger teacher model to completely mimic its behavior. As in you take a base model and create its smaller “turbo” version. Like distillation in Chemistry it means it’s %100 purified version of its teacher.

"Kimi is largely a byproduct of distillation" and "Kimi is introducing new and novel approaches" are not mutually exclusive, and I'm not sure it's clear from the paper how much of the improvement comes from the new approaches. So I wouldn't take the new approaches to be much evidence about whether the distillation attacks occurred.

"attack"

Yeah something is up. I have the same problem with K3 as I had with earlier kimis. I ask it to write <bash>code</bash> every turn, that does not seem very difficult, but kimi gets this wrong a large percentage of the time. Yet, the code it writes is pretty good.

Deepseek is also struggling with simple tasks for the last two days. Claude is perfect, with the additional credits.

Kimi Delta Attention (KDA), despite having "Attention" in the name, isn't really attention at all in any conventional sense. It's more like an RNN which can be efficiently parallelized during training. It's a very small modification to Gated DeltaNet, which can be described as an RNN whose hidden state acts like a small, editable attention memory.

Because it's RNN-like, it has an inherent idea that X comes before Y which comes before Z in the sequence XYZ. Transformers, by default, don't have that. They operate on sets, unordered collections of unique items. They have no idea where those items are in relation to each-other so you have to clue them in.

Because There are 3 KDA layers per attention layer, and 3 KDA layers before the first attention layer, every single token position is going to be able to learn information about where it is in the sequence before the first actual attention layer.

RoPE is actually a bit destructive, so being able to omit it like this is very convenient. Models like Gemma-4 have a similar structure with 5:1 Sliding Window Attention (SWA) layers for every global attention layer. These are cheap, shitty attention layers which handle local information and which go in-between the big powerful ones, KDA serves the same role in this model. In Gemma only the SWA layers have RoPE while the global attention layers omit it. SWA is actual attention, even if it only operates on a small sliding window, so it needs the positional embedding. KDA isn't, so it doesn't.

>Curious to see if it holds up at frontier scale.

I don't know how much more frontier scale you can get than this, but yes, there's no reason why that wouldn't work at larger scales. Honestly, more parameters just makes it easier for the KDA layers to communicate that positional information better.

Is frontier scale larger than this? Kimi K3 seems to benchmark in the same range as Opus and Fable. I would have expected they are all in the 2-4T range, with quality of the training and architecture differences as the major differentiators

It has some weird side effects though. for example KV-caches are implemented in fixed incremental token blocks (1024 from the providers I used) instead of simply caching up to the most recent input prompt input. It results in up to 1023 additional input (cache miss) tokens per inference.

This comment smells LLM generated.

Ask Kimi