In this insightful episode, hosts Justin Macorin and Bradley Arsenault chat with AI expert Abi Aryan to dive deep into the challenges of measuring the business impact of large language model applications. Abby draws from her 8 years of experience deploying and productizing LLMs to provide a comprehensive framework for defining relevant metrics.She emphasizes the importance of bridging the gap between engineering and business objectives, highlighting that 76% of machine learning projects fail due to this disconnect. Abby shares practical advice on evaluating LLM performance through a combination of language skills assessment, task-specific metrics, human evaluation, and aligning with overarching business goals.The conversation covers key topics such as measuring the efficacy of retrieval versus generation components, incorporating user feedback beyond simple thumbs-up/thumbs-downs, detecting and mitigating hallucinations, and tying metrics to concrete business KPIs like sales funnels and revenue generation.---Please check out Abi on LinkedIn at https://www.linkedin.com/in/goabiaryan/Visit her website here: https://abiaryan.com/Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

[Bonus] This is a bonus episode. We had too many unreleased episodes in our backlog, so we decided to give you an extra treat this week! Hope you enjoy!This episode explores the idea of making large language model interactions more transparent and immersive for users. The hosts discuss the potential of showing intermediate steps and outputs instead of just providing a final result. They envision an interface where the AI agent's process is visualized, such as displaying the generated search terms, rendered search results, visited web pages, and other intermediary steps. This approach aims to build user trust, provide insights for engineers to optimize the systems, and allow varying degrees of user control akin to management styles.The conversation highlights the need to move beyond the current "magic" of inputting text and receiving an output. By presenting the AI's workflow in a visually intuitive manner, users can better understand and even interact with the process. The hosts propose combining traditional UI elements like search bars and web page renderings with conversational AI interfaces. This transparency could pave the way for more advanced and reliable AI applications that meet users' needs while fostering trust through insight into the underlying mechanisms.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

Did you immediately try out the GPT-4o model as soon as it came splashing across your news feed? If you were anything like your show hosts, you quickly realized that it's just one step in the ever-evolving world of AI. This realization might have sparked your curiosity about the future potential of language models and their impact on various industries.In the latest episode of The Prompt Desk, your hosts discuss how our expectations need to be recalibrated —Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

Note! This is an episode that we found in our archive unreleased. It is one of the earliest episodes we ever recorded, but has so far gone not been listened to. It's not quite the same style as our current episodes, but we thought we'd release it anyway as a special treat for interested listeners!The podcast episode explores the realm of large language models, prompt engineering, and best practices, with a emphasis on discussing word embeddings and their applications. The hosts, Bradley and Justin, reminisce about their initial encounters with word embeddings and how they transformed the field of natural language processing.They then examine the advantages and potential drawbacks of word embeddings, highlighting their usefulness in various tasks such as retrieval, named entity recognition, and deduplication. The conversation also touches on practical tips and tricks for working with word embeddings, stressing the importance of vectorizing the right information and choosing the appropriate model for the task at hand.Throughout the discussion, the hosts underscore the need for product involvement and a thorough understanding of the problem at hand when deciding how to best utilize word embeddings.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

In this insightful episode, hosts Bradley and Justin tackle the challenge of managing expectations when using large language models (LLMs) and AI agents for tasks like content creation and website interactions. They dispel the common misconception that these systems can produce perfect outputs right out of the gate without any human guidance.Instead, the hosts emphasize the importance of breaking down complex workflows into structured components that can be fed to the AI incrementally, just as you would with training an intern. They explore illuminating examples that illustrate how mimicking human thought processes, like scanning website navigation links, yields better results than expecting the AI to seamlessly parse pages from pixels alone.Bradley and Justin provide a pragmatic perspective on getting optimal performance from today's AI capabilities by being realistic about their strengths and limitations. You'll walk away with valuable insights on the nuanced approach needed to effectively leverage LLMs and AI agents for robust task execution. Don't miss this engaging discussion on right-sizing your expectations.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

In this episode, we discuss, how we might protect prompt-based applications and LLMs from prompt injection. We discuss how data validation was done in the 1960s and modern libraries and techniques that can successfully act as a first line of defense against prompt injection. We touch on the idea that using other types of models, such as decision trees, conventional NLP pipelines, embedding models, or neural networks trained on datasets different from typical LLM training data, might be used to validate inputs before sending them to an LLM.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

If you are anything like us, for 36 hours your Twitter and LinkedIn feeds were set ablaze by claims that Llama 3 is the greatest LLM since sliced bread. But how do we know that is actually the case? How do we separate the hype and noise from the reality? On what basis can we credibly claim that one LLM is better than another?On this episode of The Prompt Desk, Justin Macorin and I brought on LLM expert Maxime Labonne, who has developed the open-source package LLM-autoeval. In the episode, he shares with us the techniques researchers are using to compare and contrast different LLM models.Check out the LLM-autoeval Github here: https://github.com/mlabonne/llm-autoevalFind Maxime Labonne on LinkedIn here: https://www.linkedin.com/in/maxime-labonne/Follow Maxime on Twitter here: https://twitter.com/maximelabonne?lang=en—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

As AI language models become increasingly powerful, more and more people are discovering the potential of well-crafted prompts to boost productivity and solve complex problems. Justin and Brad share their expertise on designing effective prompts that adhere to rigorous engineering principles. They highlight real-world examples of prompts that have yielded impressive results and provide valuable insights into the art and science of prompt engineering. We discuss decomposing problems into manageable steps, offering strategies to optimize this process for maximum efficiency.Check out the latest episode of The Prompt Desk to learn more about decomposition and prompts.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

What is LLM and ML Ops? How has the field of MLOps changed with the introduction of LLM models? What is the emerging field of Prompt Ops? How should we measure the effectiveness of our models in production? To answer these questions, we invited MLOps expert Andrew McMahon to chat.Check out Andrew's LinkedIn, Twitter, and book using these links:Linkedin - https://www.linkedin.com/in/andrew-p-mcmahon/ Twitter/X - @electricweegieBook - https://packt.link/w3JKL—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

Dive into the exciting world of prompt experimentation and uncover the secrets to crafting the perfect prompts! In this episode, we share our tried-and-true techniques for iterative prompt engineering, from running prompts on realistic inputs to making creative modifications that can transform a prompt's performance. We also explore the tantalizing possibilities of using datasets to generate prompts and the challenges of integrating AI conditional logic into source code - it's a wild ride you won't want to miss!—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

Prompt Injections might be the single biggest risk to the deployment of LLM systems. In this episode of The Prompt Desk, your host Bradley Arsenault sits down with researcher Jonas Geiping to talk about his research on LLM security.Jonas and his colleagues have found optimization techniques that allow them to find prompts that they can put into any LLM model that can make it do anything. The open-ended nature of the AI security flaws they have found pose serious risks to pretty much any company adopting LLM technology.See the paper: "Coercing LLMs to do and reveal (almost) anything": https://arxiv.org/abs/2402.14020Add Jonas Geiping on LinkedIn: https://www.linkedin.com/in/jonas-geiping-9684441b5/—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

Explore the six essential components that define an AI agent in 2024, including input/output, memory, loops and recursion, third-party actions, and comprehension. Join Brad and Justin as they discuss the criteria needed for a sophisticated AI system to be considered an agent, and how these requirements contribute to realizing the sci-fi vision of AI depicted in movies.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

Are you wondering how LLM Agents can be applied in large enterprises? Are we all going to become managers to teams of AI systems, cobbled together like a rag-tag group of robot butlers? In the latest episode of The Prompt Desk, hosts Justin Macorin and Bradley Arsenault sit down with Alan Wunsche to talk about the deployment and rollout of AI agents in enterprise business environments.Check out Alans' Book Agentware here: https://www.amazon.com/AGENTWARE-Unveiling-Future-Humanitys-Tomorrow/dp/1738297403—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

In this episode of The Prompt Desk, hosts Justin Macorin and Bradley Arsenault discuss the various ways they have been using coding assistants, such as Github Copilot, in order to boost their productivity as software developers. The two exchange ideas, thoughts and techniques on how to better use these tools.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

Join us with Sachin Kamath as we explore how to use AI in creative projects effectively and how we may effectively integrate AI into existing workflows, significantly enhancing productivity and creative output. We discuss frameworks for advanced prompting, ethical considerations, and best practices in AI.Consistent Character GPT:https://chat.openai.com/g/g-a9JivI0y2-consistent-character-gpt-fast-high-qualityAI Bootcamp for Creatives:https://maven.com/ai-lemon-academy/generative-ai-training-for-creativesYouTube:https://www.youtube.com/@ailemonacademy—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

Does Voodo in prompt engineering exist? The evolution from manual prompt crafting to automated refinement and optimization in AI interactions marks a transition from art to science. Let's discuss how initial natural language inputs transform, improving communication with large language models for more precise and effective outputs. Join us as we explore the fascinating process of refining prompts for AI, highlighting the blend of creativity and technical precision in advancing AI interactions.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

This episode discusses improving code generation with LLMs by dynamically extracting functions and context, using techniques like Python's inspect modules and JavaScript's toString. The goal is to produce code that integrates well with existing projects, enhancing customization and practicality.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

Join us as we explore the advancements and challenges in implementing Retriever-Augmented Generation (RAG) systems for document retrieval and summarization, examining selective information inclusion, re-ranking strategies, and the importance of document parsing and query pre-processing. We discuss with Denys Linkov the practical applications, benefits, and considerations of RAG-based solutions in improving information retrieval accuracy and relevance.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

Don't miss out on our discussion about the critical factors of expert domain knowledge, intellectual property, and the impact of first-mover advantage. These factors remain the most important competitive advantages, even in today's rapidly evolving technological landscape.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.

In today's world, application security is more important than ever before. One of the most significant risks to application security is prompt injection. It can cause irreparable damage to your business. That's why it's critical to understand the various injection techniques, their impact on your business, and how you can mitigate the risks.—Continue listening to The Prompt Desk Podcast for everything LLM & GPT, Prompt Engineering, Generative AI, and LLM Security.Check out PromptDesk.ai for an open-source prompt management tool.Check out Brad’s AI Consultancy at bradleyarsenault.meAdd Justin Macorin and Bradley Arsenault on LinkedIn.Please fill out our listener survey here to help us create a better podcast: https://docs.google.com/forms/d/e/1FAIpQLSfNjWlWyg8zROYmGX745a56AtagX_7cS16jyhjV2u_ebgc-tw/viewform?usp=sf_linkHosted by Ausha. See ausha.co/privacy-policy for more information.