On this podcast, we had Melissa Barr and Michael Mui, Technical Project Manager, and Engineering Manager at Uber AI. We had an exciting conversation about their ML education program at Uber.

Uber developed a program to educate engineers on doing ML at Uber using Uber's internal ML platform. It's focused on designing a product with engineering principles and best practices. The course is structured to cater to a lot of different engineers depending on where they are in their skill or experience level, from experienced incoming engineers to beginning engineers at Uber.

The course also serves as a support tool in helping engineers use Uber's ML platform tools, which is a two-way street. It empowers users to use Uber's internal tool sets and ecosystem. It also enables the platform team to get feedback on things that are exciting to users, which they use to make the ML platform better. This way the platform team is able to have direct communication with its users and engage with their target audience.

Meryem Arik, a co-founder of TitanML talked about making NLP deployment much easier using specialization and compression methods.

Over the last five years, fantastic foundation models like BERT and GPT have been driving NLP development currently known as LLMs. LLMs are big neural networks that know many things but can be customized for specific use cases.

The low data requirement when building NLP applications is a major benefit of LLMs. Given that they're derived from the foundational models which are pre-trained on almost every single piece of data on the internet, the state-of-the-art accuracy is good. This makes them really easy to build into version one applications.

However, these foundation models and LLMs come at a cost. Due to their big size, they are expensive, slow, and difficult to deploy.

With Titan's idea of specialization, they take parts of large language models that are relevant to a specific task and build a much smaller model, which is equally as accurate and much easier to deploy.

This blog was written by Soham Chatterjee.

"How do we serve a response with confidence, if we don’t know how confident we should be about the response?"

This blog points out some of his key takeaways after building an LLM product, chrome extension aimed at improving English writing skills for non-native speakers, to see the challenges of taking an LLM to production. The takeaways include:

• LLMs, specifically LLM APIs, make it easy to build complex applications.
• However, to take those applications to production, you need to make them reliable, scalable, and trustworthy. Therein lies the biggest challenge with building LLM products: making them production-ready.
• The lack of reliability is mainly due to the type of problem you are solving, your LLM API provider, and the vague nature of prompt engineering.
• A lack of good practices and patterns for building LLM applications.

The extension is free to use and open-source. All you need is an OpenAI API key.