MLOps in training

Aren't they the same?

If you have been in the community for long enough this topic has come up. Ther have been some amazing coffee sessions we've done around this very topic. Most notably the one with Damian, and with Ryan.

I have been thinking lots about it and how things get real messy when data is involved. I tried to synthesize a few of the key learnings into a blog post so I could 'think out loud' as it were.

This is not a super technical blog post so if you were looking for code snippets and a 'how to' you ain't gonna find it here. This is one of those posts you can pass around the office and show both your DevOps compadres and 'non-tech' people. May they understand, the struggle is real.

I defer to my favorite quote when things are similar: "Same same, but different!"

Workshopping Ideas

We have gotten so many rave reviews after the session with Alon last week that two things have become obvious.

1. Alon is going to be back on a meetup soon to finish what he started
2. We are going to start having more live coding tutorial sessions.

The workshop lasted 2 hours. Alon took us through how to use tools like pulumi, mlflow, and dvc. He also showed us how to deploy the model using FastAPI.

I Highly recommend watching how this guy works and debugs issues that come up. Full session is out!

Transitioning from DS to ML Platform Engineer

This week, Demetrios and I had the pleasure of chatting with Kyle Gallatin, a software engineer focused on machine learning infrastructure at Etsy. Lately, our community has seen a lot of discussion about how to manage the code, skills, and contributions of the diverse professionals that make up ML organizations. For example, the #production-code channel has seen great tactical discussion about ways to marry data science insight with software engineering deliverables. Kyle joined as the right time, since he personally made the transition from a pure model developer to a platform-oriented software engineer and is exceptionally thoughtful about that progression.

We dived deep into this perspective on the challenges of platform engineering and how Kyle uses his perspective from his past roles to help guide his decision making. On the more technical front, Kyle took us through some of his favorite tips for coding more effectively, how he upskilled himself to tackle software engineering problems, and which engineering challenges in MLOps he's most interested in today. Kyle dished out some lessons on building model serving solutions that are well worth your time to hear. We also got to jam about future trends in MLOps Kyle sees as most pressing; I can't give any spoilers here.

Check out the video and podcast to join us on this wide-ranging and deep discussion with Kyle.

Till next time,
Vishnu

Shopify Q & A

I Caught up with Diego Castaneda and Jennifer Bader who will be presenting this week at MLOps World to ask them a few questions of how they see the current state of affairs.

What are some main challenges you’ve seen running ML in Production?

There are many examples and resources online that present ideal case scenarios of how to run your ML projects in production. However, this is almost never as simple for companies with a specific technical infrastructure setup already. In Shopify's case, there’s no strict guide for how to productionize a ML project. Our technology stack includes off-the-shelf and in-house tools by different vendors.

To successfully deploy our ML system in production we brought together different teams across the company, coordinated work with them, proposed new tools, and connected everything to our existing infrastructure. We hope this experience can serve as a template for other teams to deploy similar ML systems to production!

Another tricky point about running modern real-time inference NLP models in production is the need to fully understand the technical requirements you need. There are not many specific guides available. The ones that are available address specific use cases only. Many of the state-of-the-art NLP models are also huge and require expensive technical setups. We had to carefully research how to deploy an efficient solution that fit our specific needs.

What are some of the most critical or common challenges of Developing a Data-Centric NLP Machine Learning Pipeline?

In our message classification project, we developed our own topic taxonomy and also setup an internal annotation campaign to create a solid training dataset from scratch. Because Shopify’s is a global commerce platform that powers over 1.7 million businesses worldwide, we had to make sure we covered a large set of messages to successfully reflect the diversity of products and services offered by these businesses. This training dataset will continuously grow and evolve, so we need to be ready to keep evolving and growing our taxonomy and training dataset.
Choosing a great team to support the annotation campaign is critical. Two main options to approach this are:
outsource the annotation effort and hire an external team to work on this full-time
have an in-house team annotating messages

We chose the latter because it allowed us to put together a team of product domain experts that understood Shopify’s merchants and their products, and it also aligned well with the timeline we had in mind. These annotators helped us improve the taxonomy during the annotation campaign. We believe this is one of the main reasons why we were able to produce a very accurate model.
Aligning the interpretation of our new taxonomy with all annotators was also not trivial. Some classes/topics were broad and we needed to make sure we were all in agreement to label every message as consistently as possible.
During our annotation campaign some instances we labelled were not always easy to interpret or read. They contained typos, grammatical mistakes, missing words/sentences and sometimes didn’t make sense but we still had to make a choice since the model has to deal with this in production.

What's the most recent bottleneck you’ve come across with your work at Shopify?

Shopify is still a fairly young company that is growing at an amazing pace. Sometimes, to sustain that growth, we need to change and move quickly as we develop projects. This can sometimes be a bottleneck since it impacts the teams involved, tools, and frameworks we use during development. Fortunately, it also forces us to learn how to thrive in environments of constant change.

Product management or Problem management?

Releasing data products is not easy. If there is one thing the marketing teams of MLOps tools have taught us, its that. Since having on Lazslo a few months ago to talk about Product Management in ML, the theme has come up regularly enough to warrant its own slack channel.

What to expect?
Three of the best PM's in the community agreed to come together and chat about what they have learned, over their collective decades of experience.

Who is coming?

Vesi Staneva - Over the past few years, Vesi work at a product company called CloudStrap.io, where together with her team they are simplifying cloud technologies and crafting modern solutions that lay a solid foundation for digital transformation at scale. Vesi's main focus currently is their new product TeachableHub.com - an ML deployment and serving platform for teams, where she heads Product and Customer Development.

Korri Jones - Sr Lead Machine Learning Engineer and Innovation Coach at Chick-fil-A, Inc. in Atlanta, Georgia where he is focused on MLOps. Prior to his work at Chick-fil-A, he worked as a Business Analyst and product trainer for NavMD, Inc., was an adjunct professor at Roane State Community College, and instructor for the Project GRAD summer program at Pellissippi State Community College and the University of Tennessee, Knoxville.

Simarpal Khaira - Simarpal is the product manager driving product strategy for Feature Management and Machine Learning tools at Intuit. Prior to Intuit, he was at Ayasdi, a machine learning startup, leading product efforts for machine learning solutions in the financial services space. Before that he worked at Adobe as a product manager for Audience Manager, a data management platform for digital marketing

Sneak Peek - You can expect some questions like these:

The mindset of organizations who treat ML as products vs ones who treat it like projects
Platform Vs Point Solutions for ML and how does that impact what role the product manager plays.
Product vs Project management in ML and what works when.
The difference between delivering a product that solves today’s problem and delivering one that is ready to solve tomorrow’s problems.

See you on Wednesday aka tomorrow at 5pm UK / 9am California. Click the button below to jump into the event, or subscribe to our public google calendar.

Production Code: The stars of the community this week were Laszlo, Ben, Alex, Larysa, Ray, Henri, and so many others who contributed to the #production-code channel. Make sure to join and see the great suggestions in threads like this one about how to do code review.
Amazing Clubhouse Sessions with some pretty big names!
PyTorch at Facebook: An introduction to the way PyTorch is used at Facebook. Fascinating to see how the framework is used at scale for the massive company.
Lean Machine Learning: Thought-provoking blog post by community member Tim Wolodzko about Lean Manufacturing principles can be applied to machine learning engineering. I like the framework of the 7 wastes quite a bit, as MLE can be bloated sometimes.