Monitoring Month

This past meetup we had Oren Razon, co-founder and CEO of Superwise. Oren pointed out that model monitoring is not only a technology challenge, but since models drive business decisions, they also can create organizational issues.

Key takeaways:

Scale means nothing without context.
Retraining is not always the answer: we do want automation but whenever you detect issues you need to be careful with taking the right actions. Retraining with fresh data is not always the answer.
Succesful model implementation requires a multidisciplinary approach.
Modeling is an iterative task.
Real model observability implies knowing when, what, why, who, where and how before making decisions.

If you want to learn more about what Oren has to say click the blue button below.

When I joined the MLOps Community in May 2020, I was really struggling to understand how companies that weren't Pinterest, Google, Uber, Facebook, etc. were managing to build and productionize machine learning models. I wanted to talk to a community of peers who understood the challenges of my work, my organization, and the limited resources (and skills!) available to me. I truly found that in the MLOps Community: a group of understanding practitioners of production ML at a diverse range of a companies.

The reason I am reminded of this is Jacopo Tagliabue, our guest this week and the Director of AI at Coveo, personifies what it means to be a meaningful MLOps practitioner "at reasonable scale". Most ML professionals don't work at a FAANG size company. Jacopo has seized on this idea, and repeatedly put out world class content for non-FAANG ML professionals looking for solutions. His famous repo/paper/blog "You Don't Need a Bigger Boat" made waves, as has his convincing definition of what a "reasonable scale" company is. Long story short, Jacopo knows how to do MLOps in the kinds of settings that most of us actually work in.

It was inspiring to hear Jacopo share his enthusiasm and optimism for where we are in the MLOps revolution. We also learned useful technical concepts, such as the power of ELT over ETL and what Jacopo views is the minimum technical stack for MLOps.

Find Jacobo on slack and ask him if you have any questions.

Till next time,
Vishnu

Today at 9am PST/5pm BST we have Amy Hodler, from Fiddler, and based on her poll popularity we hope it's the first of several appearances on the meetup.

Over time, our AI predictions degrade. Full Stop. Whether it's concept drift where the relationships of our data to what we're trying to predict has changed or data drift where our production data no longer resembles the historical training data, identifying meaningful ML drift versus spurious or acceptable drift is tedious. Not to mention the difficulty of uncovering which ML features are the source of poorer accuracy.

Attend this meetup to understand the key types of machine learning drift and how to catch them before they become problems.

Sub to our public calendar and click the button below to join the meetup. Jump in and learn more about ML Drift!

Happening very very soon! come join us!

Skylar Payne came to our last reading and provided valuable information on how he helped LinkedIn in putting the ML Test Score into action. Keep in mind that this recipe can be a bit coarse-grained, so if you want to bring it to your company: use it wisely, as a tool. Always apply it to your domain to give priority to the most important tests so that you can obtain a higher ROI.

The most common tests from the ML test score across ML teams at LinkedIn were the following:

Training/serving aren't skewed (monitoring test)
Prediction quality has not regressed (monitoring test)
Serving models can be rolled back (infrastructure test)
Offline & online metrics correlate (model test)
New features can be added quickly (data test)

Throughout our reading group we discussed several subjects. For example, how to tackle Pipeline jungles (defined in the well-known Hidden Technical Debt in Machine Learning Systems paper). Moreover, Skylar gave us several directions on how to do it: 1) start by having confidence in your pre-processed/feature engineered data; 2) apply the refactoring strategies explained in the Laszlo Sragner blog series; 3) Start with a baseline and keep adding a feature 1 by 1, see the performance go up and assure that the data is well behaved as you go along. Slowly evolve this baseline to a full pipeline where you have more control.

In addition, we also discussed on the importance of being careful with FPs & FNs feature alerts (e.g. feature value above a threshold) will lead to Engineers losing trust in your testing system. Remember: "trust is hard to gain, easy to lose". Furthermore, we also talked about the CI/CD principles of LinkedIn like using Canary deployments for every deployed model, and that the Andrej Karparthy blog post on how to train Neural Networks is a good recipe for you can test your ML algorithm correctness.

The full interview will be soon uploaded to the MLOps Community channel, Our next reading group session is Friday the 17th and we are currently voting on what to read now! jump into the #read-group channel to vote and stay up to date!