DIY ML Platform

Nowadays, building ML platforms entirely yourself might seem like an expensive decision. There are tons of vendors for different components of the stack, as well as complete off-the-shelf platforms that address most challenges. At the same time, getting your own ML platform right can be an accelerant for a business.

The build vs. buy decision is tantalizing!

Joseph Haaga, the lead MLOps engineer at Interos, joins us to walk through how his company overcame the challenge and created The Shipyard.

The Shipyard is Interos’s custom-built, heavily open-source-based ML platform. It addresses Interos’s unique data requirements for helping customers manage supply chain risks. The blog posts that outline this platform (part 1 linked here) prompted us to ask Joseph to come on the pod.

It’s rare that you have an engineer with as much as much freedom at such a large organization. He shared with us some of the well thought out reasons why Interos went with the approach they did and what benefits they’ve experienced.

For all of you interested in VC, startups, and how to build a company in the data + ML space, this is the podcast for you! Pete Soderling is the founder of the Data Council, a pre-eminent data engineering conference and community. He also leads the Data Community Fund, a new VC that writes the first check for companies changing the way we work with data and ML.

Pete has been building communities around data since 2013, when he ran meetups in NYC focused on the then-nascent field of data engineering (how far we’ve come).

In this episode, he shares some of the lessons he’s learned about how to grow community the right way and then leverage them to build a resonant product. This is a powerful formula for business that companies like dbt have beautifully used. We discuss the opportunities and pitfalls of this.

Other points: how open source projects can grow, what aspects of current data tooling hype he doesn’t believe in, and much more.

Over the last few months, we have been collaborating with our customers and community users to create the first of its kind model monitoring policy library for common monitors across ML use cases and industries.

📚 20+ use case templates with more added each month.
🎯 Auto-configured with your model's relevant metrics and features.
🪛 Fully customizable so you can apply your domain expertise and business logic.

If you’re not a community edition user yet, head over to the Superwise platform and get started with easy, customizable, scalable, and secure model observability for free.

Get ready for some hands on the keyboard practice sessions. Jovan and Maarten will showcase Vaex, an open-source DataFrame library in Python, tailor-made to allow fast, interactive workflows with datasets that are too large to fit in RAM on a single node.

Vaex makes this possible by leveraging lazy evaluations, efficient out-of-core algorithms, memory mapping, and computational graphs, all mostly behind the scenes and out of the user's way.

Using data from the New York City YellowCab taxi service comprising 1.1 billion samples and taking up over 100 GB on disk, Jovan and Maarten will show how one can conduct an exploratory data analysis, complete with filtering, grouping, calculations of statistics, and interactive visualizations on a single laptop in real-time.

Jovan and Maarten will also demonstrate how one can automatically build a machine learning pipeline as a by-product of the exploratory data analysis using the computational graphs in Vaex.