Measuring Business Results of LLMs with Abi Aryan

The Prompt Desk

May 22 2024 • 43 mins

In this insightful episode, hosts Justin Macorin and Bradley Arsenault chat with AI expert Abi Aryan to dive deep into the challenges of measuring the business impact of large language model applications. Abby draws from her 8 years of experience deploying and productizing LLMs to provide a comprehensive framework for defining relevant metrics.

She emphasizes the importance of bridging the gap between engineering and business objectives, highlighting that 76% of machine learning projects fail due to this disconnect. Abby shares practical advice on evaluating LLM performance through a combination of language skills assessment, task-specific metrics, human evaluation, and aligning with overarching business goals.

The conversation covers key topics such as measuring the efficacy of retrieval versus generation components, incorporating user feedback beyond simple thumbs-up/thumbs-downs, detecting and mitigating hallucinations, and tying metrics to concrete business KPIs like sales funnels and revenue generation.

---

Please check out Abi on LinkedIn at https://www.linkedin.com/in/goabiaryan/
Visit her website here: https://abiaryan.com/
Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.
Check out PromptDesk.ai for an open-source prompt management tool.
Check out Brad’s AI Consultancy at bradleyarsenault.me
Add Justin Macorin and Bradley Arsenault on LinkedIn.

Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_link


Hosted by Ausha. See ausha.co/privacy-policy for more information.

You Might Like

Acquired
Acquired
Ben Gilbert and David Rosenthal
Darknet Diaries
Darknet Diaries
Jack Rhysider
Hard Fork
Hard Fork
The New York Times
Marketplace Tech
Marketplace Tech
Marketplace
WSJ’s The Future of Everything
WSJ’s The Future of Everything
The Wall Street Journal
Search Engine
Search Engine
PJ Vogt, Audacy, Jigsaw
TechStuff
TechStuff
iHeartPodcasts
Rich On Tech
Rich On Tech
Rich DeMuro
The Vergecast
The Vergecast
The Verge
Waveform: The MKBHD Podcast
Waveform: The MKBHD Podcast
Vox Media Podcast Network