Because what they did is kinda trivial. Its basically like the Dropbox comment really[0], except here you don't need petabytes of storage and infinite VC pockets.

After chatgpt everything in AI mostly became LLMs and building wrappers around them. It's like people forgot how to do ML.

To those of us who actually trained models back in the day, its kind of cute to see people wowed by a classifier. Yes, this is 0 shot and doesn't need training (most people wanting this would've used structured output, this is cool because it's cheaper and faster). But anyone with basic ML knowledge could've built this in a few hours.

The question is mostly why wasn't this productized. And it's interesting indeed that it took this long to become a finished product.

[0] https://news.ycombinator.com/item?id=9224

I think the idea of a "feature startup" is dead. What used to be a niche subscription business is now an individual Epic level of work. The smallest viable business becomes what two or three years ago was a mid tier enterprise. It is no longer "look at this tool I maintain", but "we take this specific approach using these hundreds of tools merged together to solve a problem in a specific way that nobody is going to compete with. Not because they can't compete if they wanted to, but that the competitions approach diverges in fifty different chosen ways that they are targeting a different market segment essentially."

I adhere to the idea that this is software's "Tower of Babel" moment where everyone just fundamentally ships things in completely diverging architectures, because creating a ground up architecture is no longer something that needs to be avoided for an economically viable business mode that in the past two decades would have otherwise incentivized people into industry standards. In a world where "taste" is the focus, single ingredients in the recipe aren't enough.

Are you saying laya copied from jev, and released in two weeks? If so I don’t thinks it’s quite as simple a story as that. https://xtxinversexty.com/layas-prior-art-claim-is-absurd/

Presumably the training recipe and training dataset itself cannot be easily copied in a week or two. So if they want to shut down these competitor models they need to make it obvious how they are better than them.

> I'm not sure what this means for AI startups if their innovations can be copied by OSS so quickly

This particular "innovative" concept already has a rich, open research background. What Jev appears to have done is scale that up a bit and isolate good training data, which results in a great product but not really something impossible to imitate. The only major difference currently is that the open source decision models need to be fine-tuned as they're not trained off of the entire internet yet.

This is precisely why we need strict government regulation of open weight models. :wink:

Wait so a startup that didn’t innovate much should have a bigger moat? If their work can be copied in 2 weeks maybe they don’t have anything?

Didn't llaya come first? So the entire Jev's innovation is encapsulating idea in a cheap service?

Today we might need to evaluate “innovation” in a new standard, and have a different expectation for what innovator would be awarded. Getting public attention in such an era where innovation happens every a few days could’ve already been something precious. And that attention would allow TypeSafe to be heard easily next time. Like OpenAI, Anthropic, or any others, they launch frequently but still each time they launch something new, that would hit headlines. I think that’s the “surplus” flown to innovators today.

The moat is the RL synthetic data pipeline they set up to train jev. Open sourcing that would be the coup, not the model architecture and training scripts, which are trivial.

> I'm not sure what this means for AI startups if their innovations can be copied by OSS so quickly (what, like 2 weeks?).

My thoughts too. It sounds like a minor feature being framed as a whole new business.

Then again, Dropbox and Docker are too.

Just because it is zero-shot does not mean that Jev is an architectural innovation, especially in the year 2026.

Anyone fitting MNIST in 2019 was already outdated by several years at the very least. GPT-2 was 2019! We already had zero-shot classifiers then. In fact, the paper for GPT-3 was literally

"Large Language Models are Zero-Shot Reasoners"

These kinds of zero-shot classifiers were already developed and used in-house for many years. They just weren't commercialized as a separate product, because anyone who could use an LLM proper could build layers around it to fulfill any classification task like this.

Hah you’re probably dating yourself. Keras came out in 2015 and that was one of the early examples with Theano backend. You could train MNIST since 2015 pretty straight forward. But comparing Jev to image classification is an unfaithful argument. Comparing it to ELmo or BERT is analogously better.

I've been following jevbench twice a day for the past week and that's been a lot of fun. Latest update:

Rank System Score Public / sealed accuracy Evidence

1 decider-4b v2 64.13 83.5% / 34.7% Evaluator-run, offline

2 Jev 1.13 63.29 86.6% / 36.7% Evaluator-run API

3 JevK5 v0.2 62.04 85.3% / 33.1% Evaluator-run

4 Cygnet 12B 61.76 87.9% / 33.8% Evaluator-run, offline

5 Hopper 59.43 82.3% / 34.1% Evaluator-run

28 Kev 4B 36.14 66.2% / 22.4% Evaluator-run

41 Laya 421M 30.25 58.4% / 30.8% Evaluator-run

https://benchmarkheaven.com/jev-models

Yes. JEV generalizes better because they probably have an enormous corpus and trained on it for a long time. Laya's out of the box model is much weaker. However, in the age of LLM's it's incredibly easy and cheap to generate large datasets to fine tune laya for your task, and the training loop is pretty quick and cheap too.

It's so easy that I question why I would ever pay for JEV when eventually I'll have done enough random things that I will also have a large corpus and likely a general model as well.

Developer here. You're right, Laya is a lot weaker than Jev, especially on harder queries. It's a small model, so it's fast, but that's the trade-off. The open models that get close to Jev are much bigger, and running those is what I'm working on next.

This is just anecdotal and I might be doing it wrong but I made jev and laya versions of a simple semantic grep tool (https://github.com/lgastako/jevplay) and played with them a bit, and at first it seemed like laya was comparable (eg on queries like "this is a mans name" or "this is a womans name" on names.txt) but the more I played with it, eg. "this is a vegetable" on foods.txt the further the gap widened in favor of jev. Then I started trying variations of the query eg simply "mans name" and for the most part laya just fell apart and didn't return anything useful for a lot of stuff. I was hoping to find that laya was competitive because it's much faster to have the model running locally but it's just not, yet.

I've been testing, for my use case laya didn't come close to decider was just as good. My experience with the 3 models matches the result here

https://benchmarkheaven.com/jev-models

In my experience it's not close and the benchmarks I've seen don't reflect my experience at all.

But I'm guessing people will find the right training regime and data mix soon to close the gap.

But big things I see are instability and inaccuracy - like pick a random problem.

one day, perhaps people will click through to the laya author's arxiv paper content and the why may become clearer, you won't have to read it, a skim will suffice

That's the thing I think some people keep missing.

I mean, anyone here could have a decision maker just return a random number. Fast and easy.

The question really is how GOOD is it?

Nothing yet. Unfortunately it sometimes feels like our industry has been overrun by grifters and chancers.

I’m sure this has been a gradual and long decline. Maybe it even started with the dot com boom and accelerated with crypto. With AI it seems to have got worse.

> difference between an instruct based re-ranker and laya/jev I just don't see it

Main difference is that laya/jev/et-al give you a zero-shot classifier that requires no training. You can prompt engineer your way to a quick fairly reliable cheap enough decision engine that you can use to iterate quickly (by prompt engineering).

Right now a lot of people are doing this with LLMs and it's too slow and expensive.

Imo the right iterative approach to productionizing these systems is something like:

    1. Build it with an LLM. Iterate on the prompt
    2. Start building a real-world dataset
    3. When the prompt works, turn it into a clear rubric for Jev or similar
    4. Keep iterating until desired accuracy achieved
    5. Use the real-world evals you've built to train a custom classifier fine-tuned to your needs

Jev's value becomes more apparent when the task is a moving target. eg an auto-mode classifier.

Calibrated probability across multi task with zero shot I guess. A reranker is single task and tuning it make it even more narrow. And I guess some piping to make multiclass efficient since you cannot mask logprob for independent questions in the same output space without throwing calibration away.

I have a lot of semi-practical examples of how you can use this model wrapped in unix-ish tools - https://github.com/aurorainfra/grev (readme links to docs of each tool with some more or less practical examples)

Really I think "smart grep" is a pretty good one ('look for an error looking vaguely like this'). Also I think sql-based shell history + decision model is quite good to make the last 'which one of those choices is best fit given users past few commands' etc.

Is for when you want an AI to make a decision. If you have been using gpt or claude or open source models for that, than it’s a way cheaper alternative.

And if you have not been, it’s for when you have to extract the context from text. When you have numbers or fixed options, it’s just a matter of code.

So if you find yourself having to decide if a given user comment is a refund_request, that’s for that.

It’s not perfect, you still have to fine-tune (or calibrate) using examples you have (and keep those examples updated over time). But it’s way better than trying to parse text with regexes.

If you don't want to spend a lot and want low latency, e.g. home automation. "It's cold and dark in here, do something about it", it will then turn on the lights and heater nearly instantly.

"Decision model" is just marketing jargon.

decision model = classifier

system one model = small non-reasoning LLM

noul = boolean

confidence = f(probabilities)

It's sad to see how gullible engineers are today.

text classification is equivalente to decision. This is exactly the same thing Jev does.

Fair point, that example is basically classification. I'll change it to something that looks more like a real decision.

Fair, and I'd be happy if they did. Ollaya uses the same API as Jev, so your code isn't tied to it either way

and that ollama is go-llama and not rust, so it's not really the ollama of anything

next vLLM release will have this

if you use gateways, GoModel support the S1 endpoints, my favorite feature is the virtual models, stable name, I can swap out the backing model(s)

https://gomodel.enterpilot.io/docs/getting-started/quickstar...

(the "kev" in the docs is my fault, I should have said Jev / System1 in my feature request)

[dead]

Depends on the model. The small ones I support today are well below Jev on harder queries, but fine for simple, well-defined questions. The open models that get close to Jev are bigger, and I'm adding support for those next.

there's this thing with a bunch of similar models https://huggingface.co/spaces/multimodalart/jev-decision-ind...

top open one is trained by perplexity cto for $3k, kinda cool https://x.com/denisyarats/status/2102252088067850507

The link rgbrgb posted is a good overview. The best open ones are close to Jev now, but they're big models. And I agree, if you have an eval set for a fixed task, a trained classifier is the better choice.

[dead]

open-source is the only way for safe AI development. whoever doesn’t share the weights/code will lag behind.

Thanks!

wait another week or so for vLLM's next release

hey Claude, download and run vllm nightly for me

(already merged)

GoModel (gateway) already supports Jev like endpoints too

https://gomodel.enterpilot.io/docs/providers/jev