The Meme King Cometh

Application Programming Interface

The Long and the short of it - Hypervector is an API for writing test fixtures for data-driven features and components. It provides synthetic data (generated from distributions the user defines programmatically) over dedicated endpoints. It also allows you to benchmark the output of any function or model to detect changes and regressions as part of your wider test suites.

So I caught up with the creator of Hypervector Jason Costello to talk about how and why this tool materialized. Enter Jason.

I started off as a data scientist around 2014 after working in applied ML research for a bit, but I found myself more interested in becoming a better engineer as time went on. I've been fortunate enough to work with some superb teams, and one of the areas I've learned the most as a developer has been in writing useful tests. I find these help me contribute with less stress & uncertainty, can often aid in my understanding of a problem, and help make the experience of fast-moving shared codebases a little less chaotic.

A surprisingly common scenario I've encountered in multiple data teams has been at the interface between the data scientists and wider engineering folks when it comes time to ship something.

Engineers love to verify equivalence and consistency - over and over again with every incremental change. I remember being asked after pushing an improvement to a model: "How do we know it's doing what it did before, plus some bits better?". My answer would have been explaining the train-test-validate cycle, the various model selection metrics we'd used to make the decision to ship an improvement, and I might have pulled up a Jupyter Notebook to show some graphs (and probably waved my hands about a lot).

Now I can see that was sort of missing the point. The question was more like "How do WE (the engineers) know its doing what it did before, plus some bits better?". Why can't we run a test on every build of this project that ensures even seemingly unrelated changes have not somehow broke a small but important part of the model's output?

I didn't have a great response for this at the time, and eventually settled on using some of the training data used to build the feature as a test resource in the project repo - not a very elegant solution, and then a real pain to maintain going forward.

Hypervector tries to help in this area by providing a set of tools you can access via Python (or REST if you'd prefer). These tools allow you to define test resources centrally for such scenarios.

It began as a side project I was working on during evenings and weekends, and I decided to focus on it full-time towards the end of 2020. You can try out the early adopters Alpha version here, and please feel free to reach out on the MLOps Slack at any time. Feedback Appreciated.

Explainability and Maturing Ideas to Data Products

Lex Bettie from Spotify, Michael Munn from Google and Mike Moran from Skyscanner sat down with us last Wednesday for a round table discussion.

Topics of interest - We started out talking about some of the main bottlenecks they have encountered over the years of trying to push data products into production environments. Then things started to heat up as we dove into the topic of monitoring ML and inevitably the word explainability started being thrown around.

Turns out Lex is currently doing a PhD on the subject so there was much to talk about. I had to ask if explainability is now tablestakes when it comes to monitoring solutions on the market? Short answer from the team. Yes

Check out the whole conversation here on youtube and here in podcast land. Please excuse the bit of sound trouble we had with google mike at the beginning.

Meta Talk Around Memes

Let us all just take a moment to marvel at this collection of MLOps memes. Ariel is the undisputed meme king of the MLOps Community and we got a chance to chat with him!

This discussion was really fun. We had an expansive discussion with Ariel about his background as a research scientist, how he got into making memes about MLOps and what his perspective on the MLOps landscape really is. I highly recommend watching this on YouTube so you can see the memes that Ariel made and know what Demetrios and I are giggling at. A couple things stood out to me in this chat.

First, the radically simple nature of communicating through memes on MLOps was something both Demetrios and I really appreciated. It's very easy to get caught up in all the jargon in our industry, but memes, visuals, and humor make it much simple to understand how people actually feel about different topics and how they think. We talked about how Ariel stumbled into this form of communication and how it fits into his work as an evangelist now.

Next, Ariel made an amazing point: "we are in the Stone Age for MLOps." This is a recurring theme in our chats that Ariel summarized succinctly. No one has had enough experience with all the possible ML systems to say that one way is exactly right. In particular, the tools we use still have a long way to go, just like stone preceded bronze. Ariel gave us great insight into how these tools can evolve and get even better.

Thanks to Ariel for spending time with us and giving us lots of laughs! Check out the video here and the podcast here.

Till next time,
Vishnu

PoC Hell

Starting the AI adoption with AI Proof-of-Concepts (PoCs) is the most common choice for most companies. Yet, a significant percentage of AI PoCs do not make it into production whether they were successful or not. Furthermore, running yet another AI PoC follows the law of diminishing returns in various aspects.

On Wednesday we will talk with Oguzhan Gencoglu, Co-founder & Head of AI at Top Data Science about how not to get stuck in PoC hell and really create value at your company.

Bio
Oguzhan (Ouz) Gencoglu is the Co-founder and Head of AI at Top Data Science, a Helsinki-based AI consultancy. With his team, he has delivered more than 70 machine learning solutions in numerous industries for the past 5 years. Before that, he used to conduct machine learning research in several countries including USA, Czech Republic, Turkey, Denmark, and Finland. Oguzhan has given more than 40 talks on machine learning to audiences of various backgrounds.

See you on Wednesday at 5pm UK / 9am California. Click the button below to jump into the event, or subscribe to our public google calendar.

Perfectionist

An issue that can knock data initiatives off the rails is over-scoping — trying to wrap models around business processes that are simply too big or complex and overloaded with variables.

Maybe in these cases, there's pressure from senior leadership to use "AI" as a sort of computing crystal ball that can take all business data and use it to predict business outcomes. It's OK to reach for the stars, but like striving for perfection, practical barriers could mean you're destined to fall short.

I wanted to write about some more recurring themes I've been hearing. So we now have a new blog post about how Perfect is the enemy of good.