Orchestrating Machine Learning Workflows

Kevin Kho is an Open Source Community Engineer at Prefect, an open-source workflow orchestration management system. Previously, he was a data scientist for four years working in the energy and HR spaces.

In this workshop, Kevin will demonstrate how to orchestrate a full ML hyperparameter grid search pipeline over a Dask cluster. Other tools included will be Pandera, Tune, and Evidently AI.

He will go through:
1. How to orchestrate workflows with Prefect
2. How to scale workflows on top of Dask (grid search)
3. Setting up notifications for Slack
4. Prefect as a glue for other machine learning tools

Come and Prefect with us the machine learning workflow!

Over the last week, the Community blog was quite productive. We managed to publish two new posts.

The first post is the follow up to that good ol' survey we sent out in conjunction with Arize at the end of last year. We compiled data from over 300 participants to give us a better view of the MLOps landscape, straight from the horse's mouth! Read it here.

The next up was a follow up to the conversation we had with Ernest Chan on the podcast. In that conversation, we spoke about how he had researched a process to standardize deploying hundreds of thousands of ML models.

We’ve had on a number of people from the Googleverse (Todd Underwood, D. Sculley, Sara Robinson, Lak Lakshmanan) and they’ve universally been regarded as some of the best episodes of our podcast. This week, we had another set of Googlers, Donna Schut and Christos Aniftos, both of whom have a very unique perspective at Google: customer facing Google Cloud solutions engineering!

Donna and Christos talked to us about their work on helping customers get the most of GCP and its powerful product. As authors of the excellent “Practitioners Guide to MLOps”, they shared with us how they think about identifying the core challenges companies face in adopting MLOps and resolving it. Christos walked us through how this looks like on a very practical level. He shared an example of identifying and developing MLOps capabilities for a customer service call center. Donna explained how she works with her team to distill the lessons they learn into crucial thought leadership documents that help everyone better implement MLOps.

Google needs no explanation as a top shop for M! Listen to this episode for a great summary of how teams at Google Cloud learn and apply MLOps best practices!

Chad Sanderson is the Product Lead for Convoy's Data Platform team, which includes the data warehouse, streaming, BI & visualization, experimentation, machine learning, and data discovery.

Chad built everything from feature stores, experimentation platforms, metrics layers, streaming platforms, analytics tools, data discovery systems, and workflow development platforms. He’s implemented open source, SaaS products (early and late-stage) and has built cutting-edge technology from the ground up.

In this Meetup, Chad talks us through the benefits his team has had since implementing data contracts at Convoy. Data Contract - an agreement between the producer and consumer that says what data is wanted, in what shape and context. The engineering team then produces this data with this specific use case in mind.

We have talked before about Data Mesh concepts. Chad does a great job of bringing the abstract Data Mesh concepts to reality. How have they structured the data teams at Convoy? What does this actually look like in practice in an ML context? Check out the session to hear all about it.