Learn to build evals, folks!

You are not RSI-pilled enough if you don't know how to properly measure the thing you want to improve.

It doesn't matter if it's a model, complex codebase, or a business function. This is extremely hard in the real world.

You ever wonder why most of what AI is great at today is around verifiable problems?

The open-ended world requires a completely different approach to improving and measuring models.

We need to go back to hardcore science and evals. Enough of this short-term benchmaxxing.

This space offers so much opportunity. We are doing research and building evals in this space @dair_ai. I am currently chatting with folks building in this space too. Feel free to reach out for potential collab opportunities or just share perspectives.