Embrace the chaos

Connie Yang, lone data science + ML extraordinar wolf at Pallet HQ came on the meetup last week to talk about how they serve millions of job recommendations to their customers per week.

Connie was kind enough to dive deep into the infrastructure side at Pallet and the why behind their design choices. She echoes what many have said in the community before - start manual then automate, and at any point possible, don't use ML unless its totally needed. My favorite takeaway was the part where she dove into how they deal with misclassification, and the system she set up to solve them.

In this episode, we spoke to Emmanuel Ameisen a Machine Learning Engineer at Stripe and author of ‘Building Machine Learning Powered Applications (published by O’Reilly). It quickly became clear to Demetrios and Adam that Emmanuel really knew his stuff!

‘The shortest iteration cycle wins’

Emmanuel talked about the best way to develop complex, ML powered applications and services within an organisation. He stressed the importance of time-to-insight and speed of iteration for really unlocking the progress of your MLOps and data science teams.

‘Models suffer from a curse of success’

We’re introduced to the curse of success - often the most useful models end up becoming an integral part of more and more workflows. The need to maintain and support the model grows with each new user. This also makes it difficult to support without breaking the myriad downstream use cases.

‘You develop operational excellence by exercising it’

This was a great nugget of wisdom - the idea that you only get slicker processes by running through them enough times to smooth out all the rough edges. Going through the process and systematically automating and rebuilding things that are too involved or difficult leads to a better system overall. Emmanuel talked about scheduling the ETL for training sets just to ensure things haven’t broken.

It’s clear from this conversation that Stripe are at a mature place with their MLOps. Listen to this episode if you’re interested in gaining some insights from someone that’s done the hard yards.

For this coffee session we had Julien Bisconti, Head of Data Infrastructure at Hypertype.co and we had a blast.

We dived into a variety of subjects from: how to manage your career (technical and non-technical advise), the famous buy vs. build decision and introduced (at least for me) the concept of chaos engineering and what's its role in creating useful complex ML systems.

He dropped some great quotes and harsh truths:

`It's harder to change a process than to change code.`

'Every problem can be broken down into simple UNIX commands'

If you want to learn more of what Julien told us, click the link below and listen to the complete session. You won't regret it!