Designing forward compatibility – AI Gateways to the future

Designing for Forward Compatibility in Gen AI // Rohit Agarwal // MLOps Podcast #189

With the rapid rise of LLMs, it’s often said we’re living in the future.
If that’s the case, then I want to know where my self-drying jacket and hoverboard are.

Perhaps they’ve been hampered by the worry about forward compatibility – being scared to work on something because you don't know if in a month you're going to get a new open source model that comes out that renders everything that you're working on obsolete.

Well, my guest Rohit Agarwal, founder of Portkey AI, has a solution to that.

He tells me all about how using an AI gateway and decoupling APIs from the code, companies can rapidly adopt new AI models, databases, frameworks – you name it. A fantastic way to overcome that FOMO of the future.

He also talks about the parallels between the traditional engineering world and the AI space and the challenges of controlling data access and ensuring security.

Now that it’s sorted, I’m off to buy my power-lace Nikes.

Video || Spotify || Apple

Join Union’s next Flyte School session, "Learn Your Codebase: Fine-Tuning CodeLlama with Flyte," led by Niels Bantilan, Chief ML Engineer at Union. In this session, you will gain hands-on experience using Flyte to leverage state-of-the-art deep learning tools such as `torchrun` distributed training, LoRA, 4/8-bit quantization, and FSDP, while benefiting from Flyte’s reproducibility, versioning, and cost-management capabilities.

🗓️ Date/Time: Thursday, November 30, at 9 AM PST
📍 Location: Virtual

At the end of this session, you will acquire the knowledge and code to apply the same fine-tuning principles to your own codebase to:
Answer user support questions
Generate boilerplate starter code
Tackle any other downstream tasks

Register now and be part of this exciting and informative session that will enhance your knowledge of LLMs. You don’t want to miss this!

Resources:
Check out the Flyte repo on GitHub.
Join our Slack community, follow us on X, and subscribe to our YouTube channel.
Sign up for our newsletter.
Learn more about Union and Flyte.

Guarding LLM and NLP APIs: A Trailblazing Odyssey for Enhanced Security // Ads Dawson // MLOps Podcast #190

If there’s a Top 10 list of Top 10 lists, would it include itself?

These are the uncertainties that keep me awake at night.

What’s more certain is that it would include the OWASP’s Top 10 for LLM Applications.

My guest, Ads Dawson from Cohere, talks to me about his work on it, and explains that it’s a list of vulnerabilities to be aware of when building models. And, just like the Billboard, there are new entries and chart movers as the community votes and updates the list.

We also talk about how to bridge this pipeline between the securities world and the ML world, what were some of the wildest vulnerabilities that he's seen on the job, and what he thinks is a potential danger that is not being talked about enough.

One of the main vulnerabilities that got me was split view data poisoning, and he shared a story of how his friend bought up domains to do this as an experiment.

Great, now there’s even more to keep me awake at night.

Video || Spotify || Apple

Cloudera - for Large Scale MLOps // Erik Steinholtz // IRL Meetup #53 Stockholm

Fine-tuning is fine, but could it be finer?

Erik from Cloudera shares what could be the next step, Parameter Efficient Fine-Tuning (PEF) with Lora. He starts by introducing CML, Cloudera's specific tool for machine learning, and highlights how it caters to data scientists and data engineers.

He then talks about how Lora enables fine-tuning LLMs with significantly fewer parameters compared to traditional methods, emphasizing its applications in LLMs. He even provides a prototype within Cloudera's platform for users to explore and become proficient in using Lora for their own fine-tuning tasks.

It's sure to be a fine watch!

Watch it here

Mini-summit meetup on Nov 16 brought to you by Weights & Biases.

We’ve had to extend this mini-summit! Usually, they’re an hour, but we’ve got so much to share we needed to add on an extra 30 minutes!

Join Ben Epstein as he hosts Jonathan Whitaker from Data Science Castnet, Boris Dayma from Craiyon, Thomas Capelle from Weights & Biases, and Robbie McCorkell from Leap Labs as they share their secrets of maximizing language models' potential.

These seasoned experts will share practical experiences, invaluable tips, and unique insights into the intricacies of the fine-tuning process, explore the best practices crucial for training large models, and witness real-world applications through the lens of the prestigious Kaggle competition.

Be sure to register here.

Job of the week

Senior ML Engineer // Prem
We are currently looking for an experienced ML Engineer to shape and drive the future of AI by fine-tuning open-source models, and training foundational models using the latest techniques.

Key Responsibilities:

Fine-tuning LLMs
Training Foundational Models
Leading the current Data Science team to the next breakthrough.

Skills needed:

Deep understanding of Transformers architecture and underlying components
5 years of Python development experience, 5 years of hands-on experience with PyTorch, 3 years of Docker experience, 2 years of FastAPI experience
Experience with git

Don’t Build AI Products The Way Everyone Else Is Doing It

When there's a rush of people doing something, it's hard not to follow - remember the ice bucket challenge?

Well, following the crowd is kinda what's happening now in the rush to get an AI product out.

But, this piece by Steve Sewell gives a great rundown on the issues you'll face and how to build differently, including creating your own toolchain.

So remember, don't just follow people! Unless of course, they're shouting something about an escaped lion...

Thanks to Soham Chatterjee for sharing.