magic 8 ball as a service.

Sure it is.

The prefill is the only blocking part, and you can prefill the whole context up to the point where they diverge, then prefill each question and decode the one token in parallel for each question.

If you batch vLLM calls with the same prompt prefix to the same process, it'll deduplicate the prompt prefix across batched requests (+/- the block size) and decode in parallel for each.

That's with a vanilla LLM. If you modify the LLM you can pull that in-graph, but it isn't really necessary.

I'm fairly sure it is 100%, and has been available for "ages". That's how every tool call and whatnot works:

https://developers.openai.com/api/docs/guides/structured-out...

yes, it does, with appropriate tuning/testing of the llm's parameters (temperature, top p, top k, using the right model, and the right prompt). You have to give it a rigid and very specific prompt to only answer in the JSON form. Also test it with LLMs that will handle being given a very low temperature to be very 'literal'. You're not asking for creative writing.

I should also add that the source comes from one of about 400 possible places and in a variety of messed up formats, it's the raw feed from a news scraper...

Yes, you can use constrained decoding

Also, while it's clearly got a lot of training on some use cases, others that probably weren't in the training set have worse good decision rates than a random number generator. If you can rebuild the architecture, you can train it on your use case.

Whether Jev is something architecturally different from an LLM (bidirectional vs auto-regressive? different type of parallel readout head?) remains to be seen, or guessed, but LLMs are certainly fungible and are competing on price - they leapfrog each other from release to release, but overall they are all progressing in unison.

When it comes to the high volume market of business automation, it seems that ultra-low cost, rather than expensive frontier intelligence, is exactly what you want, and low latency is also nice to have for customer-facing applications like customer service chatbots.

> it may raise some questions about the necessity of Jev's architecture

When I hear “architecture” I am thinking number of parameters and latency.

When I hear “accuracy” I think training recipe, data, and (later) number of parameters.

So when you say that Jev’s architecture may not be necessary, the evidence I expect to see is comparable quality at comparable latency. Not equal quality at 2x latency and 4x the cost.

I mean just from what's known of the funding and timeline it pretty much has to be based on open weights.

But it is likely more than just a fine tune + novel training. At the very least the LM head is swapped out for a classifier one and then or also idk, bidirectional attention for the encoding pass I'm out of my depth at this point and will stop guessing. The training is probably where they have the biggest moat though, not that it's necessarily huge.

I have a project that fits jev as advertised almost comically well and I've been playing with it, and the various hacks and open versions. Jev doesn't necessarily perform better overall but it is quite different. It's sensitive to prompt phrasing in ways the others aren't, it's easy to generate questions where all the other models cluster in confidence but jev is an outlier. Not necessarily more correct, but it does feel like it's getting its answers in a different way.

I'm guessing just as much as anyone else but I've been spending a ton of time on this the last couple weeks, it landed right when I was most ready to dig into it.

Great article, thanks for posting.

I think I'm gradually getting a picture of what jev does, and why it might be good:

1) one shot classifiers are great, and are well-known.

2) LLMs are also great classifiers, but are slow at output - and not great at guaranteed structured output.

3) everyone focused on LLMs, and forgot about classifiers.

4) any decent one shot classifier should be easy to run "fast"/"in realtime".

5) combining 4) with LLMs and a harness (a loop) open up some clever use-cases that LLMs alone are too slow for.

As I understand it, jev isn't long for this world (we'll have foss one shot classifiers we can run on modest hardware) - but it seems likely the legacy will be that classifiers will re-gain some prominence alongside generative models.

I am just this far from adding Jev posts to my blocklist.

I cannot fathom how people are not seeing the multiple daily posts as anything but the spam they are.

don’t worry once they get acquired for 30 billion usd then the spam will stop.

Right, so it is double the latency and will no longer feel real-time to the end user.

I have a bridge to sell you.