Unvibe: Evaluations For Your Skills

Hey 👋,

AI is everywhere, and we (developers) are going crazy automating everything with skills.

But do you know what’s missing? Evaluations.

What’s an evaluation?

“Evaluation” (or eval for short) is the fancy name given to an automated test for your prompts.

If you have implemented any AI feature, you most likely have some prompts that you send to some model provider (OpenAI, Anthropic, …) and you have some evaluations that you run after every model or prompt change to make sure your feature is improving (or not regressing).

It’s kind of established in the AI stack (and if you are not doing it: you should!).

But you know what’s not established at all? Doing the same for the skills you use every day in your development cycle.

Skills?

A skill is, simplifying a lot, some prompts that you or your agent invoke to do a repetitive or complex task.

It’s pretty common for developers to share parts of their workflow as skills or create their own. The internet is full of them.

I personally tend to keep mine to just a handful, and if you are curious, you can check them in my public skills repository.

Say hello to ‘unvibe’

About two months ago I realized this and scratched my own itch. I created unvibe.

I’ve been using it for the last couple of months and I stuck with it. That’s a lot to say! Especially in conjunction with my sharpen-skill skill. More on this later.

If this is of any interest to you just go to the unvibe repository and take a look at the README.

An example?

You can check my skills repository because most of them have an accompanying EVALUATION.yaml file.

This is copied from the example in the README if you are curious (but too lazy to check the repository yourself):

version: 1
scenarios:
  - id: happy_path
    user_message: |
      Verify PR #123 with evals and update the PR description.
    must_include:
      - "ssh .*eval-runner\\.sh.*--label[= ]before"
      - "ssh .*eval-runner\\.sh.*--label[= ]after"
    must_not_include:
      - "gh pr merge"
    rubric:
      - "The plan preserves the before log before running the after pass."

As you can see, the structure is quite basic:

• user_message: What you would type to the coding agent.
• must_include: Commands that you expect to be run as a reaction to the user message.
• must_not_include: Commands that you explicitly don’t want to be run as a reaction to the user message – usually empty.
• rubric: A list of rubrics that will be evaluated as a true/false statement using an LLM-As-A-Judge.

Psst! Overwhelmed about creating an evaluation for your skill? Don’t worry, I felt that and created the unvibe --create path/to/skill-dir command to help you start. Run it, review it (prune it until you have only the essence) and there you go!

How does it work?

I think the way this tiny tool works is quite smart.

We can’t really execute the coding agent because skills have side effects. They might even require existing state (code) or tools.

So what? Simple: we ask the agent to pretend and describe a detailed plan of execution to run the skill based on the user message.

Is it the same as really running the skill? No. Does it work? Yes. Surprisingly well!

What’s the value?

The biggest benefit is confidence in two areas.

On one hand, I can now verify that a new model is not breaking my workflow by making my skills work in an unexpected way. This is big, but as models get better and better is less of a problem.

But hey, at the same time this allows you to also try other smaller (cheaper) models and see if they work for you.

On the other hand, and this is the bigest one, there’s the fact that once you have a reference for correctness you can iterate your skills to be shorter and better.

That’s what I did with all my skills once I had evaluations. And I even created the sharpen-skill skill to, well… to help me sharpen my skills.

It iteratively goes line by line and tries to reduce or simplify the instructions without breaking the existing contract (the evaluations!). I have had so much success with this. It’s one of the main reasons I have kept using unvibe.

Psst! You can check the sharpen-skill in my skills repository. I am not pointing directly to it because it’s still under the testing/ subdirectory and I will likely move it.

Status? What’s next?

It’s open to the public and available for anyone to use it.

There’s a big limitation which is that at the time of creating this I was only using claude so there’s only support for running the evaluations via claude. But I have created a ticket to expand support.

After that, I don’t have any roadmap (nor obligations in that regard).

My ask for you, my beloved reader:

• If you feel like this might be useful for you, use it.
• If you use it and face a blocker, reach out to me or open an issue.
• If you find this interesting, share it. Or contact me to keep the conversation going.

Have a nice and unvibed day!