Mirror: https://xcancel.com/jietang/status/2089941544581403107

Maybe just me, but I have a feeling that we are currently stuck at 100-200k effective context window due to a parameter limitation, specifically the hidden dimension of the model, which would be an incredibly expensive dimension to increase vs adding optimizations elsewhere. I don’t think we are done scaling parameters yet.