Evaluating evaluation

All About Evaluating LLM Applications // Shahul Es // MLOps Podcast

Some things just make you break in to a sweat.

When there’s 5 minutes left in your exam and you notice the page is actually double-sided.

Realizing you clicked ‘Reply All’ on that email.

Evaluating your system.

But worry not!

This week’s podcast with Shahul Es will help settle those heart palpitations as he gets into the crucial role of evaluation in debugging and troubleshooting. He talks through challenges with evaluating models using open data sets, as well as the limitations of current benchmarks.

As well as discussing solutions like finetuning and evaluating in a custom data distribution, he has some interesting insights into the future of evaluation and finetuning.

I’m not sure he can help with the ‘Reply All’ email you’ve sent though.

Video || Spotify || Apple

Job of the week

Multiple positions // Dream11 (Mumbai, India)

Dream11 is the world’s largest fantasy sports platform with 190 million+ users. We have over 150 micro-services written in Java and backed by a Vert.x framework. These work with isolated product features with discrete architectures to cater to the respective use cases.

We have a number of positions open:

Lead ML Scientist - https://lnkd.in/dWhi63wT
Sr. ML Scientist - https://lnkd.in/dDMS_Ejk
Sr. ML Scientist (Generative AI) - https://lnkd.in/dG78wRWp

Virtual meetups are back!

LLM Security is so hot right now.

And this is a great one to start with as Ben Epstein is joined by three great guests:

Sankalp Gilda introduces tsbootstrap, an open-source Python library implementing state-of-the-art bootstrapping tailored for time series.
Uri Shamay introduces a new OSS we've developed named GeniA, and discusses its role as an enabler for the future of platform engineering using Gen AI.
Raahul Dutta discusses LLM security and the risks of prompt injection, denial of service, and model theft and strategies for tackling these issues.

LLM Security // MLOps Meetup #125

Vizro combines benefits of the speed and user-friendliness of low-code visual analytics tools with the flexibility and power of high-code tools - whilst providing inbuilt visual design, application architecture and coding standardization which also facilitates scaling and reusability. You get inbuilt visual styling in minutes and reduce the amount of “glue code” required.

Vizro’s users have reported it as being a game changer, helping them to bring their vision to life more easily, and accelerate their existing processes — such as being able to visualize and discover raw data quality issues ten times faster than before.

Check it out on GitHub and consult the Vizro documentation.