Making software better

Today the CEO of Neptune AI, Piotr, shares his insights on the importance of confidence and control in production, the need for regulations and audits, and the significance of order and reproducibility in engineering processes.

He also discusses Neptune's mission to provide ML/AI teams with the same level of control and confidence in building and testing models as in software development.

According to everyone that's used the, large language models still have many challenges when using them in production. Much of Piotr’s brain power has been focused on discovering and supporting ways to test and validate new LLMs.

We also chat about “classical” machine learning vs. deep learning, the role of AutoML, and the use of numerical methods to test and validate models.

Piotr shares a few gems about sustaining a business in the MLOps space, making strategic decisions, and staying focused on solving core problems, AKA how he thinks about building a product developers will love.

Video || Spotify || Apple

Job of the week
Developer Advocate // Weaviate (US Based) - As a Developer Advocate, you will wear many hats, from speaking at events, participating in hackathons, writing blog posts, designing tutorials, and creating engaging videos to fostering our community.

This role has a strong focus on the Python developer community and ecosystem. We need someone who is part of the Python community, with proven public speaking experience, a great overview of the hot conferences and engaging meetups, and a way into the Python developer’s heart.

How Lemonade Optimized Its Feature Store

How We Migrated from Python Multithreading to Asyncio
In this article, Dor Indivo, Senior Machine Learning Engineer at Lemonade, explains his journey migrating Lemonade feature store from Python multithreading to asyncio.

Indivo highlights the three main reasons for migration: Performance, Scale, and Resource consumption. Next, he goes deeper into the migration process and the lessons learned along the way building with asyncio.

Indivo puts the icing on the cake by detailing their process of testing and benchmarking the code. Overall, It's a great piece to dip your toes in the water of asyncio.

Overcoming Performance Bottlenecks with Async Python: A Deep Dive into CPU-Bound Code
This article is a sequel to the previous one above. Dor Indivo takes us on a deep dive into CPU-Bound challenges he faced while maintaining Lemonade’s in-house feature store. The battle started with latency degradation in one of their core models in production, with p90 and p95 metrics nearly tripling. Indivo shows how they used event loop lag metrics to pinpoint the issue. He even goes the extra mile by explaining the metric implementation in-depth.

Using the event loop lag metric, Lemonade managed to zero in on the prime bottleneck - CPU-bound code, primarily triggered by processing large JSON files. If you ever find yourself grappling with similar challenges, I recommend giving this article a read.

Making Software Better ⧸⧸ Thorben Louw

Thorben shares his expertise on how to do things right - such as testing model performance, handling data quality issues, and the importance of having a backup plan in case of model failure.

He explains the concept of drift detection, its different types, and how it affects the performance of machine learning models.

What practical strategies and techniques can be implemented to ensure optimal model performance? Conduct "chaos days" to simulate failure scenarios, analyze logs and monitor concept drift.

Mr. Louw delves into the tools and best practices needed to keep your software running smoothly in this talk from the Bristol community meetup.

Video

Warning: in-depth post about the use of vector databases! If you are doing anything with LLMs or recsys, I can guarantee you will find it useful!

Let's look at some of the topics covered:

Importance of vectors in language model applications
The role of vector databases in Language Model Applications
Key features and capabilities of vector databases
How vector databases enhance language model applications
Understanding the different types of vector databases
Comparison of vector databases
Evaluating vector databases in production
Benchmarking vector databases
Evaluating graph-based databases for recommendation systems
Evaluating document-based databases for text clustering
Best practices for vector database integration

Empowering Language Model Applications: Understanding and Evaluating Vector Databases in Production - By Stephen Oladele