LLM in Production III starting now!

Stephen Batifol MLOps Podcast #178

Regular listeners to the podcast (so, all of you, right??) will know I love having Stephen Batifol on as a co-host.

Not this week though.

Don’t worry, we haven’t fallen out, even though he did go to see Barbie without me. He’s not co-hosting because he’s on as a guest this week.

He’s much too friendly to fall out with, and that really comes across in his work too. As an internal ML platform developer advocate at Wolt he focuses on making a positive developer experience:

creating high-quality onboarding guides, documentation, and tutorials,
getting input from data scientists and product engineers to understand their needs and pain points,
simplifying complex topics rather than gatekeeping knowledge.

This positivity also fuels his work in the community giving talks and helping to restart the meetup scene in Berlin after Covid.

Now I just need to get him to organize a meetup at the Watergate club…

Video || Spotify || Apple

Job of the week

Multiple positions // Tecton (United States or remote)

3 positions are available for people who want to own a technical focus and grow into leaders. You’ll get to work at an established Series C startup, taking on first-of-their-kind challenges as we build a category from scratch.

Multicloud SWE - Work with complex distributed systems to expand our offerings to GCP, Azure, and drive the growth of our product.
Realtime Execution SWE - Work on the architecture that makes online inference possible, enabling subsecond predictions and decisioning.
Lead Security Engineer - Take ownership of our cloud security roadmap and lead efforts to implement security projects and uplevel our security capabilities.

[Exclusive] Tecton Round-table // Get your ML Application Into Production

Rick Rubin. Quincy Jones. Sylvia Robinson.

Legendary producers. But they’d struggle to get an ML app into production. If you’ve got 99 problems and this is one, then this special Roundtable podcast is for you.

I was joined by Kevin Stumpf, Derek Salama, Eddie Esquivel, and Isaac Cameron from Tecton. With over 35 years combined these guys know what they’re talking about!

So when they talk about issues like transitioning from batch processing to online inference, or not investing enough time in the design and requirement gathering phases vs talking about solutions, you know it’s worth the listen - click the link below.

Video || Spotify || Apple

LLMs in Production: From language models to successful products by Christopher Brousseau and Matthew Sharp

This book is a thorough guide covering the lifecycle of LLMs, from concept to deployment. It delves into dataset preparation, cost-efficient training methods, model evaluation, and deployment infrastructure setup.

But it stands out from other for 3 reasons:

It includes hands-on projects like a chatbot, VSCode extension, and deploying LLMs to edge devices.
It’s accessible now as it’s being written, which means it’ll be up to date, and you can give feedback and suggestions on what else to include in the book.
You can 45% off until October 11th with the code: mlbrousseau

Click the link below to order:

Order now

How Instacart Modernized the Prediction of Real Time Availability for Hundreds of Millions of Items While Saving Costs

For Instacart, accurately predicting the inventory status—whether a product is in-stock, low on stock, or out-of-stock—is a hard challenge. Several years ago, they built an item availability model that has since served as the basis of their approach to tackling this problem on a high scale.

However, this model has not fully solved the issue and came with a set of challenges after the constant growth of Instacart:

🔹Interpretability: It was particularly problematic when the model made a wrong prediction about an item in stock. Understanding why the model came up with these signals was even more challenging.

🔹Sparse Data: The majority of the items in their catalog have very limited or even no shopper signals. This came with the challenge of finding better features to accurately predict the availability of tail items.

🔹Business use case mismatch: The different ways customers can order on Instacart have increased in the past years. For example, customers can now schedule an order for the next day. This means that item availability should account for the scheduled time and not the current time. Adjusting the model to these varying scenarios was like hitting a moving target.

🔹Legacy Infrastructure: Instacart's inventory has grown significantly in the past years and it has become economically unfeasible to score frequently hundreds of millions of items using their current infrastructure.
The article then dives deep into how they solved these issues using a new model and a new infrastructure. It’s a great read if you’re looking to get your feet wet in the world of ML scalability.

Thanks to Mohamed Sadek for the contribution.