First impressions of Decision Models

Hey 👋,

Unless you have been living under a rock, you might have already heard about the shiny new AI thing: Jev from TypeSafe AI. Or the so-called Decision Models, or System One models, or Jev-like models.

I haven’t personally played a lot with them yet, but I wanted to share my immediate thoughts as I learn more about them.

My impression is that they are useful and hyped. Yes, both at the same time.

What’s that again?

The key differentiator from a traditional (the irony) LLM is the interface used to interact with the underlying model.

With LLMs, the output is always language being generated by the LLM token by token. With Decision Models, it’s the resulting weights themselves that are translated into a combination of the three primitives that are defined.

Not really surprising, because we already knew that generating a JSON with an LLM was a bad idea. That’s why TOON has been a thing.

Sometimes you don’t need the model to talk to you. You just want a very straightforward response.

Why does it work?

I won’t pretend I know what this all does internally, but it makes sense for it to work. After all, LLMs contain a huge amount of knowledge and also language processing capabilities.

You can feed them the input, and just check how the output weights change. Then expose those outputs more directly to the outside.

This is a simplification of how these work, but it makes sense that you don’t need the LLM to talk back to you token by token. Right?

The hype

My impression is that it’s easy to fall on the survivor bias once again. There’s an infinite list of impressive things filling all of Twitter (or any other feeds of your liking). And most of them are pretty amazing and look super cool, but I haven’t found many that are really that useful.

I’m not saying there aren’t any, I’m just saying that there seems to be more of a focus on “showing some shiny thing you did” than really showing how to apply it realistically.

I see the value, but it’s in less impressive things than the ones that are being shared and re-shared everywhere.

Open-weights and fine-tuning

Looks like this trend has brought attention back to open-weight models. And the community is up-to-speed fighting to become the equivalent of Jev but open. It’s exciting to follow! People are doing an amazing job and I am loving the impressive and fast results that are appearing. This is the benchmark I am tracking right now.

And the attention is not only back to the open-weight models, it’s also back to fine-tuning especially around small models.

That’s because some of the Jev competitors are worse initially, but once you fine-tune them for your specific task, they jump to similar levels (sometimes they even outperform Jev!).

I think this is based on two premises: 1) open-weight models have progressed a lot recently and 2) it looks like speaking back in a coherent language was penalizing the smaller models a lot, so skipping that part is making them much more useful.

It is now really cheap (free even) to fine-tune a small 4B or 9B model for a very specific decision task.

Where are we going?

I expect very fast progress in this area. On one side I would expect Jev (The First of Its Name) to continue improving. On the other side I expect open models to catch up mostly at the same level pretty soon.

Then, I would also expect Anthropic or OpenAI to have something to say and release their own. Hopefully they will release in a format compatible with TypeSafe AI’s, but one never knows 🤞.

Additionally, the ecosystem for open models doesn’t seem very mature yet. In order to test different models, you need to run different runtimes. Almost one per different model family out there. This is what is preventing me from doing more testing.

There are a bunch of ongoing projects fighting to become the equivalent of ollama or llama.cpp to these Decision Models. But none seem to be outstanding from the rest yet.

As a final note, the most promising models right now seem to be Kev and Laya. I am keeping an eye on them.

If this is any interesting to you please click the heart button below, to let me know. As soon as I invest some more time playing with them, I will share more definitive impressions grounded in data.