Experiencing Data w/ Brian T. O’Neill - Data Products, Product Management, & UX Design

Brian T. O’Neill from Designing for Analytics

If you’re a leader tasked with generating business and org. value through ML/AI and analytics, you’ve probably struggled with low user adoption. Making the tech gets easier, but getting users to use, and buyers to buy, remains difficult—but you’ve heard a ”data product” approach can help. Can it? My name is Brian T. O’Neill, and on Experiencing Data—one of the top 2% of podcasts in the world—I offer you a consulting designer’s perspective on why creating ML and analytics outputs isn’t enough to create business and UX outcomes. How can UX design and product management help you create innovative ML/AI and analytical data products? What exactly are data products—and how can data product management help you increase user adoption of ML/analytics—so that stakeholders can finally see the business value of your data? Every 2 weeks, I answer these questions via solo episodes and interviews with innovative chief data officers, data product management leaders, and top UX professionals. Hashtag: #ExperiencingData. PODCAST HOMEPAGE: Get 1-page summaries, text transcripts, and join my Insights mailing list: https://designingforanalytics.com/ed ABOUT THE HOST, BRIAN T. O’NEILL: https://designingforanalytics.com/bio/ read less
TechnologyTechnology

Episodes

141 - How They’re Adopting a Producty Approach to Data Products at RBC with Duncan Milne
Apr 16 2024
141 - How They’re Adopting a Producty Approach to Data Products at RBC with Duncan Milne
In this week's episode of Experiencing Data, I'm joined by Duncan Milne, a Director, Data Investment & Product Management at the Royal Bank of Canada (RBC). Today, Duncan (who is also a member of the DPLC) gives a preview of his upcoming webinar on April 24, 2024 entitled, “Is that Data Product Worth Building? Estimating Economic Value…Before You Build It!”  Duncan shares his experience of implementing a product mindset within RBC's Chief Data Office, and he explains some of the challenges, successes, and insights gained along the way. He emphasizes the critical role of understanding user needs and evaluating the economic impact of data products—before they are built. Duncan was gracious to let us peek inside and see a transformation that is currently in progress and I’m excited to check out his webinar this month! Highlights/ Skip to: I introduce Duncan Milne from RBC (00:00)Duncan outlines the Chief Data Office's function at RBC  (01:01)We discuss data products and how they are used to improve business process (04:05)The genesis behind RBC's move towards a product-centric approach in handling data, highlighting initial challenges and strategies for fostering a product mindset (07:26)Duncan discusses developing a framework to guide the lifecycle of data products at RBC (09:29)Duncan addresses initial resistance and adaptation strategies for engaging teams in a new product-centric methodology (12:04)The scaling challenges of applying a product mindset across a large organization like RBC (22:02)Insights into the framework for evaluating and prioritizing data product ideas based on their desirability, usability, feasibility, and viability. (26:30)Measuring success and value in data product management (30:45)Duncan explores process mapping challenges in banking (34:13)Duncan shares creating specialized training for data product management at RBC (36:39)Duncan offers advice and closing thoughts on data product management (41:38) Quotes from Today’s Episode “We think about data products as anything that solves a problem using data... it's helping someone do something they already do or want to do faster and better using data." - Duncan Milne (04:29)“The transition to data product management involves overcoming initial resistance by demonstrating the tangible value of this approach." - Duncan Milne (08:38)"You have to want to show up and do this kind of work [adopting a product mindset in data product management]…even if you do a product the right way, it doesn’t always work, right? The thing you make may not be desirable, it may not be as usable as it needs to be. It can be technically right and still fail. It’s not a guarantee, it’s just a better way of working.” - Brian T. O’Neill (15:03)“[Product management]... it's like baking versus cooking. Baking is a science... cooking is much more flexible. It’s about... did we produce a benefit for users? Did we produce an economic benefit? ...It’s a multivariate problem... a lot of it is experimentation and figuring out what works." - Brian T. O'Neill (23:03)"The easy thing to measure [in product management] is did you follow the process or not? That is not the point of product management at all. It's about delivering benefits to the stakeholders and to the customer." - Brian O'Neill (25:16)“Data product is not something that is set in stone... You can leverage learnings from a more traditional product approach, but don’t be afraid to improvise." - Duncan Milne (41:38)“Data products are fundamentally different from digital products, so even the traditional approach to product management in that space doesn’t necessarily work within the data products construct.” - Duncan Milne (41:55)“There is no textbook for data product management; the field is still being developed…don’t be afraid to create your own answer if what exists out there doesn’t necessarily work within your context.”- Duncan Milne (42:17) Links Duncan’s Linkedin: https://www.linkedin.com/in/duncanwmilne/?originalSubdomain=ca
140 - Why Data Visualization Alone Doesn’t Fix UI/UX Design Problems in Analytical Data Products with T from Data Rocks NZ
Apr 2 2024
140 - Why Data Visualization Alone Doesn’t Fix UI/UX Design Problems in Analytical Data Products with T from Data Rocks NZ
This week on Experiencing Data, I chat with a new kindred spirit! Recently, I connected with Thabata Romanowski—better known as "T from Data Rocks NZ"—to discuss her experience applying UX design principles to modern analytical data products and dashboards. T walks us through her experience working as a data analyst in the mining sector, sharing the journey of how these experiences laid the foundation for her transition to data visualization. Now, she specializes in transforming complex, industry-specific data sets into intuitive, user-friendly visual representations, and addresses the challenges faced by the analytics teams she supports through her design business. T and I tackle common misconceptions about design in the analytics field, discuss how we communicate and educate non-designers on applying UX design principles to their dashboard and application design work, and address the problem with "pretty charts." We also explore some of the core ideas in T's Design Manifesto, including principles like being purposeful, context-sensitive, collaborative, and humanistic—all aimed at increasing user adoption and business value by improving UX.   Highlights/ Skip to: I welcome T from Data Rocks NZ onto the show (00:00)T's transition from mining to leading an information design and data visualization consultancy. (01:43)T discusses the critical role of clear communication in data design solutions. (03:39)We address the misconceptions around the role of design in data analytics. (06:54) T explains the importance of journey mapping in understanding users' needs. (15:25)We discuss the challenges of accurately capturing end-user needs. (19:00) T and I discuss the importance of talking directly to end-users when developing data products. (25:56) T shares her 'I like, I wish, I wonder' method for eliciting genuine user feedback. (33:03)T discusses her Data Design Manifesto for creating purposeful, context-aware, collaborative, and human-centered design principles in data. (36:37)We wrap up the conversation and share ways to connect with T. (40:49) Quotes from Today’s Episode "It's not so much that people…don't know what design is, it's more that they understand it differently from what it can actually do..." - T from Data Rocks NZ (06:59)"I think [misconception about design in technology] is rooted mainly in the fact that data has been very tied to IT teams, to technology teams, and they’re not always up to what design actually does.” - T from Data Rocks NZ (07:42) “If you strip design of function, it becomes art. So, it’s not art… it’s about being functional and being useful in helping people.” - T from Data Rocks NZ (09:06) "It’s not that people don’t know, really, that the word design exists, or that design applies to analytics and whatnot; it’s more that they have this misunderstanding that it’s about making things look a certain way, when in fact... It’s about function. It’s about helping people do stuff better." - T from Data Rocks NZ (09:19)“Journey Mapping means that you have to talk to people...  Data is an inherently human thing. It is something that we create ourselves. So, it’s biased from the start. You can’t fully remove the human from the data" - T from Data Rocks NZ (15:36) “The biggest part of your data product success…happens outside of your technology and outside of your actual analysis. It’s defining who your audience is, what the context of this audience is, and to which purpose do they need that product. - T from Data Rocks NZ (19:08)“[In UX research], a tight, empowered product team needs regular exposure to end customers; there’s nothing that can replace that." - Brian O'Neill (25:58) “You have two sides [end-users and data team]  that are frustrated with the same thing. The side who asked wasn’t really sure what to ask. And then the data team gets frustrated because the users don’t know what they want…Nobody really understood what the problem is. There’s a lot of assumptions happening there. And this is one of the hardest things to let go.” - T from Data Rocks NZ (29:38)“No piece of data product exists in isolation, so understanding what people do with it… is really important.” - T from Data Rocks NZ (38:51) Links Design Matters Newsletter: https://buttondown.email/datarocksnz Website: https://www.datarocks.co.nz/LinkedIn: https://www.linkedin.com/company/datarocksnz/BlueSky: https://bsky.app/profile/datarocksnz.bsky.socialMastodon: https://me.dm/@datarocksnz
139 - Monetizing SAAS Analytics and The Challenges of Designing a Successful Embedded BI Product (Promoted Episode)
Mar 19 2024
139 - Monetizing SAAS Analytics and The Challenges of Designing a Successful Embedded BI Product (Promoted Episode)
This week on Experiencing Data, something new as promised at the beginning of the year. Today, I’m exploring the world of embedded analytics with Zalak Trivedi from Sigma Computing—and this is also the first approved Promoted Episode on the podcast. In today’s episode, Zalak shares his journey as the product lead for Sigma’s embedded analytics and reporting solution which seeks to accelerate and simplify the deployment of decision support dashboards to their SAAS companies’ customers. Right there, we have the first challenge that Zalak was willing to dig into with me: designing a platform UX when we have multiple stakeholder and user types. In Sigma’s case, this means Sigma’s buyers, the developers that work at these SAAS companies to integrate Sigma into their products, and then the actual customers of these SAAS companies who will be the final end users of the resulting dashboards.  also discuss the challenges of creating products that serve both beginners and experts and how AI is being used in the BI industry.     Highlights/ Skip to: I introduce Zalak Trivedi from Sigma Computing onto the show (03:15)Zalak shares his journey leading the vision for embedded analytics at Sigma and explains what Sigma looks like when implemented into a customer’s SAAS product . (03:54)Zalak and I discuss the challenge of integrating Sigma's analytics into various companies' software, since they need to account for a variety of stakeholders. (09:53)We explore Sigma's team approach to user experience with product management, design, and technical writing (15:14)Zalak reveals how Sigma leverages telemetry to understand and improve user interactions with their products (19:54)Zalak outlines why Sigma is a faster and more supportive alternative to building your own analytics (27:21)We cover data monetization, specifically looking at how SAAS companies can monetize analytics and insights (32:05)Zalak highlights how Sigma is integratingAI into their BI solution (36:15)Zalak share his customers' current pain points and interests (40:25) We wrap up with final thoughts and ways to connect with Zalak and learn more about Sigma (49:41)  Quotes from Today’s Episode "Something I’m really excited about personally that we are working on is [moving] beyond analytics to help customers build entire data applications within Sigma. This is something we are really excited about as a company, and marching towards [achieving] this year." - Zalak Trivedi (04:04) “The whole point of an embedded analytics application is that it should look and feel exactly like the application it’s embedded in, and the workflow should be seamless.” - Zalak Trivedi (09:29)  “We [at Sigma] had to switch the way that we were thinking about personas. It was not just about the analysts or the data teams; it was more about how do we give the right tools to the [SAAS] product managers and developers to embed Sigma into their product.” - Zalak Trivedi (11:30) “You can’t not have a design, and you can’t not have a user experience. There’s always an experience with every tool, solution, product that we use, whether it emerged organically as a byproduct, or it was intentionally created through knowledge data... it was intentional” - Brian O’Neill (14:52)  “If we find that [in] certain user experiences,people are tripping up, and they’re not able to complete an entire workflow, we flag that, and then we work with the product managers, or [with] our customers essentially, and figure out how we can actually simplify these experiences.” - Zalak Trivedi (20:54) “We were able to convince many small to medium businesses and startups to sign up with Sigma. The success they experienced after embedding Sigma was tremendous. Many of our customers managed to monetize their existing data within weeks, or at most, a couple of months, with lean development teams of two to three developers and a few business-side personnel, generating seven-figure income streams from that.” - Zalak Trivedi (32:05) “At Sigma, our stance is, let’s not just add AI for the sake of adding AI. Let’s really identify [where] in the entire user journey does the intelligence really lie, and where are the different friction points, and let’s enhance those experiences.” - Zalak Trivedi (37:38) “Every time [we at Sigma Computing] think about a new feature or functionality, we have to ensure it works for both the first-degree persona and the second-degree persona, and consider how it will be viewed by these different personas, because that is not the primary persona for which the foundation of the product was built." - Zalak Trivedi (48:08) Links Sigma Computing: https://sigmacomputing.com Email: zalak@sigmacomputing.com  LinkedIn: https://www.linkedin.com/in/trivedizalak/ Sigma Computing Embedded: https://sigmacomputing.com/embedded About Promoted Episodes on Experiencing Data: https://designingforanalytics.com/promoted
138 - VC Spotlight: The Impact of AI on SAAS and Data/Developer Products in 2024 w/ Ellen Chisa of BoldStart Ventures
Mar 5 2024
138 - VC Spotlight: The Impact of AI on SAAS and Data/Developer Products in 2024 w/ Ellen Chisa of BoldStart Ventures
In this episode of Experiencing Data, I speak with Ellen Chisa, Partner at BoldStart Ventures, about what she’s seeing in the venture capital space around AI-driven products and companies—particularly with all the new GenAI capabilities that have emerged in the last year. Ellen and I first met when we were both engaged in travel tech startups in Boston over a decade ago, so it was great to get her current perspective being on the “other side” of products and companies working as a VC.  Ellen draws on her experience in product management and design to discuss how AI could democratize software creation and streamline backend coding, design integration, and analytics. We also delve into her work at Dark and the future prospects for developer tools and SaaS platforms. Given Ellen’s background in product management, human-centered design, and now VC, I thought she would have a lot to share—and she did! Highlights/ Skip to: I introduce the show and my guest, Ellen Chisa (00:00)Ellen discusses her transition from product and design to venture capital with BoldStart Ventures. (01:15)Ellen notes a shift from initial AI prototypes to more refined products, focusing on building and testing with minimal data. (03:22)Ellen mentions BoldStart Ventures' focus on early-stage companies providing developer and data tooling for businesses.  (07:00)Ellen discusses what she learned from her time at Dark and Lola about narrowing target user groups for technology products (11:54)Ellen's Insights into the importance of user experience is in product design and the process venture capitalists endure to make sure it meets user needs (15:50)Ellen gives us her take on the impact of AI on creating new opportunities for data tools and engineering solutions, (20:00)Ellen and I explore the future of user interfaces, and how AI tools could enhance UI/UX for end users. (25:28)Closing remarks and the best way to find Ellen on online (32:07) Quotes from Today’s Episode “It's a really interesting time in the venture market because on top of the Gen AI wave, we obviously had the macroeconomic shift. And so we've seen a lot of people are saying the companies that come out now are going to be great companies because they're a little bit more capital-constrained from the beginning, typically, and they'll grow more thoughtfully and really be thinking about how do they build an efficient business.”- Ellen Chisa (03: 22)  “We have this big technological shift around AI-enabled companies, and I think one of the things I’ve seen is, if you think back to a year ago, we saw a lot of early prototyping, and so there were like a couple of use cases that came up again and again.”-Ellen Chisa (3:42) “I don't think I've heard many pitches from founders who consider themselves data scientists first. We definitely get some from ML engineers and people who think about data architecture, for sure..”- Ellen Chisa (05:06)   “I still prefer GUI interfaces to voice or text usually, but I think that might be an uncanny valley sort of thing where if you think of people who didn’t have technology growing up, they’re more comfortable with the more human interaction, and then you get, like, a chunk of people who are digital natives who prefer it.”- Ellen Chisa (24:51) [Citing some excellent Boston-area restaurants!] “The Arc browser just shipped a bunch of new functionality, where instead of opening a bunch of tabs, you can say, “Open the recipe pages for Oleana and Sarma,” and it just opens both of them, and so it’s like multiple search queries at once.” - Ellen Chisa (27:22) “The AI wave of  technology biases towards people who already have products [in the market] and have existing datasets, and so I think everyone [at tech companies] is getting this big, top-down mandate from their executive team, like, ‘Oh, hey, you have to do something with AI now.’”- Ellen Chisa (28:37) “I think it’s hard to really grasp what an LLM is until you do a fair amount of experimentation on your own. The experience of asking ChatGPT a simple search question compared to the experience of trying to train it to do something specific for you are quite different experiences. Even beyond that, there’s a tool called superwhisper that I like that you can take audio content and end up with transcripts, but you can give it prompts to change your transcripts as you’re going. So, you can record something, and it will give you a different output if you say you’re recording an email compared to [if] you’re recording a journal entry compared to [if] you’re recording the transcript for a podcast.”- Ellen Chisa (30:11) Links Boldstart ventures: https://boldstart.vc/LinkedIn: https://www.linkedin.com/in/ellenchisa/Personal website: https://ellenchisa.comEmail: ellen@boldstart.vc
137 - Immature Data, Immature Clients: When Are Data Products the Right Approach? feat. Data Product Architect, Karen Meppen
Feb 20 2024
137 - Immature Data, Immature Clients: When Are Data Products the Right Approach? feat. Data Product Architect, Karen Meppen
This week, I'm chatting with Karen Meppen, a founding member of the Data Product Leadership Community and a Data Product Architect and Client Services Director at Hakkoda. Today, we're tackling the difficult topic of developing data products in situations where a product-oriented culture and data infrastructures may still be emerging or “at odds” with a human-centered approach. Karen brings extensive experience and a strong belief in how to effectively negotiate the early stages of data maturity. Together we look at the major hurdles that businesses encounter when trying to properly exploit data products, as well as the necessity of leadership support and strategy alignment in these initiatives. Karen's insights offer a roadmap for those seeking to adopt a product and UX-driven methodology when significant tech or cultural hurdles may exist. Highlights/ Skip to: I Introduce Karen Meppen and the challenges of dealing with data products in places where the data and tech aren't quite there yet (00:00)Karen shares her thoughts on what it's like working with "immature data" (02:27)Karen breaks down what a data product actually is (04:20)Karen and I discuss why having executive buy-in is crucial for moving forward with data products (07:48)The sometimes fuzzy definition of "data products." (12:09)Karen defines “shadow data teams” and explains how they sometimes conflict with tech teams (17:35)How Karen identifies the nature of each team to overcome common hurdles of connecting tech teams with business units (18:47)How she navigates conversations with tech leaders who think they already understand the requirements of business users (22:48)Using design prototypes and design reviews with different teams to make sure everyone is on the same page about UX (24:00)Karen shares stories from earlier in her career that led her to embrace human-centered design to ensure data products actually meet user needs (28:29)We reflect on our chat about UX, data products, and the “producty” approach to ML and analytics solutions (42:11)  Quotes from Today’s Episode "It’s not really fair to get really excited about what we hear about or see on LinkedIn, at conferences, etc. We get excited about the shiny things, and then want to go straight to it when [our] organization [may not be ] ready to do that, for a lot of reasons." - Karen Meppen (03:00) "If you do not have support from leadership and this is not something [they are]  passionate about, you probably aren’t a great candidate for pursuing data products as a way of working." - Karen Meppen (08:30) "Requirements are just friendly lies." - Karen, quoting Brian about how data teams need to interpret stakeholder requests  (13:27) "The greatest challenge that we have in technology is not technology, it’s the people, and understanding how we’re using the technology to meet our needs." - Karen Meppen (24:04) "You can’t automate something that you haven’t defined. For example, if you don’t have clarity on your tagging approach for your PII, or just the nature of all the metadata that you’re capturing for your data assets and what it means or how it’s handled—to make it good, then how could you possibly automate any of this that hasn’t been defined?" - Karen Meppen (38:35) "Nothing upsets an end-user more than lifting-and-shifting an existing report with the same problems it had in a new solution that now they’ve never used before." - Karen Meppen (40:13) “Early maturity may look different in many ways depending upon the nature of  business you’re doing, the structure of your data team, and how it interacts with folks.” (42:46)  Links  Data Product Leadership Community https://designingforanalytics.com/community/Karen Meppen on LinkedIn: ​​https://www.linkedin.com/in/karen--m/Hakkōda, Karen's company, for more insights on data products and services:https://hakkoda.io/
136 - Navigating the Politics of UX Research and Data Product Design with Caroline Zimmerman
Feb 6 2024
136 - Navigating the Politics of UX Research and Data Product Design with Caroline Zimmerman
This week I’m chatting with Caroline Zimmerman, Director of Data Products and Strategy at Profusion. Caroline shares her journey through the school of hard knocks that led to her discovery that incorporating more extensive UX research into the data product design process improves outcomes. We explore the complicated nature of discovering and building a better design process, how to engage end users so they actually make time for research, and why understanding how to navigate interdepartmental politics is necessary in the world of data and product design. Caroline reveals the pivotal moment that changed her approach to data product design, as well as her learnings from evolving data products with the users as their needs and business strategies change. Lastly, Caroline and I explore what the future of data product leadership looks like and Caroline shares why there's never been a better time to work in data. Highlights/ Skip to: Intros and Caroline describes how she learned crucial lessons on building data products the hard way (00:36)The fundamental moment that helped Caroline to realize that she needed to find a different way to uncover user needs (03:51)How working with great UX researchers influenced Caroline’s approach to building data products (08:31)Why Caroline feels that exploring the ‘why’ is foundational to designing a data product that gets adopted (10:25)Caroline’s experience building a data model for a client and what she learned from that experience when the client’s business model changed (14:34)How Caroline addresses the challenge of end users not making time for user research (18:00)A high-level overview of the UX research process when Caroline’s team starts working with a new client (22:28)The biggest challenges that Caroline faces as a Director of Data Products, and why data products require the ability to navigate company politics and interests (29:58)Caroline describes the nuances of working with different stakeholder personas (35:15)Why data teams need to embrace a more human-led approach to designing data products and focus less on metrics and the technical aspects (38:10)Caroline’s closing thoughts on what she’d like to share with other data leaders and how you can connect with her (40:48) Quotes from Today’s Episode “When I was first starting out, I thought that you could essentially take notes on what someone was asking for, go off and build it to their exact specs, and be successful. And it turns out that you can build something to exact specs and suffer from poor adoption and just not be solving problems because I did it as a wish fulfillment, laundry-list exercise rather than really thinking through user needs.” — Caroline Zimmerman (01:11) “People want a thing. They’re paying for a thing, right? And so, just really having that reflex to try to gently come back to that why and spending sufficient time exploring it before going into solution build, even when people are under a lot of deadline pressure and are paying you to deliver a thing [is the most important element of designing a data product].” – Caroline Zimmerman (11:53) “A data product evolves because user needs change, business models change, and business priorities change, and we need to evolve with it. It’s not like you got it right once, and then you’re good for life. At all.” – Caroline Zimmerman (17:48) “I continue to have lots to learn about stakeholder management and understanding the interplay between what the organization needs to be successful, but also, organizations are made up of people with personal interests, and you need to understand both.” – Caroline Zimmerman (30:18) “Data products are built in a political context. And just being aware of that context is important.” – Caroline Zimmerman (32:33) “I think that data, maybe more than any other function, is transversal. I think data brings up politics because, especially with larger organizations, there are those departmental and team silos. And the whole thing about data is it cuts through those because it touches all the different teams. It touches all the different processes. And so in order to build great data products, you have to be navigating that political context to understand how to get things done transversely in organizations where most stuff gets done vertically.” – Caroline Zimmerman (34:37) “Data leadership positions are data product expertise roles. And I think that often it’s been more technical people that have advanced into those roles. If you follow the LinkedIn-verse in data, it’s very much on every data leader’s mind at the moment:  how do you articulate benefits to your CEO and your board and try to do that before it’s too late? So, I’d say that’s really the main thing and that there’s just never been a better time to be a data product person.” – Caroline Zimmerman (37:16) Links Profusion: https://profusion.com/Caroline Zimmerman LinkedIn: https://www.linkedin.com/in/caroline-zimmerman-4a531640/Nick Zervoudis LinkedIn: https://www.linkedin.com/in/nzervoudis/Email: mailto:carolinez@profusion.com
135 - “No Time for That:” Enabling Effective Data Product UX Research in Product-Immature Organizations
Jan 23 2024
135 - “No Time for That:” Enabling Effective Data Product UX Research in Product-Immature Organizations
This week, I’m chatting with Steve Portigal, who is the Principal of Portigal Consulting and the Author of Interviewing Users. We discuss the changes that prompted him to release a second version of his book 10 years after its initial release, and dive into the best practices that any team can implement to start unlocking the value of data product UX research. Steve explains that the key to making time for user research is knowing what business value you’re after, not simply having a list of research questions. We then role-play through some in-depth examples of real-life experiences we’ve seen from both end users and leadership when it comes to implementing a user research strategy. Thhroughout our conversation, we come back to the idea that even taking imperfect action towards doing user research can lead to increased data product adoption and business value.  Highlights/ Skip to: I introduce Steve Portigal, Principal of Portigal Consulting and Author of Interviewing Users (00:38)What changes caused Steve to release a second edition of his book (00:58)Steve and I discuss the importance of understanding how to conduct effective user research (03:44)Steve explains why it’s crucial to understand that the business challenge and the research questions are two different things (08:16)Brian and Steve role-play a common scenario that comes up in user research, and Steve explains an optimal workflow for user research (11:50)The importance of provocation in performing user research (21:02)How Steve would handle a situation where a member of leadership is preventing research being done with end users (24:23)Why a consultative approach is valuable when getting buy-in for conducting user research (35:04)Steve shares some of the major benefits of taking imperfect action towards starting user research (36:59)The impact and value even easy wins in user research can have (42:54)Steve describes the exploratory nature of user research and how to maximize the chance of finding the most valuable insights (46:57)Where you can connect with Steve and get a copy of v2 of his book, Interviewing Users (49:35) Quotes from Today’s Episode “If you don’t know what you’re doing, and you don’t know what you should be investing effort-wise, that’s the inexperience in the approach. If you don’t know how to plan, what should we be trying to solve in this research? What are we trying to learn? What are we going to do with it in the organization? Who should we be talking to? How do we find them? What do we ask them? And then a really good one: how do we make sense of that information so that it has impact that we can take away?” — Steve Portigal (07:15)“What do people get [from user research]? I think the chance for a team to align around something that comes in from the outside.” – Steve Portigal (41:36)On the impact user research can have if teams embrace it: “They had a product that did a thing that no one [understood], and they had to change the product, but also change how they talked about it, change how they built it, and change how they packaged it. And that was a really dramatic turnaround. And it came out of our research, but [mostly] because they really leaned into making use of this stuff.” – Steve Portigal (42:35)"If we knew all the questions to ask, we would just write a survey, right? It’s a lower time commitment from the participant to do that. But we’re trying to get at what we don’t know that we don’t know. For some of us, that’s fun!" – Steve Portigal (48:36) Links Interviewing Users (use code DATA20 to get 20% off the list price): https://rosenfeldmedia.com/books/interviewing-users-second-edition/Personal website: https://portigal.comPublisher website: https://rosenfeldmedia.comLinkedIn: https://www.linkedin.com/in/steveportigal/
134 - What Sanjeev Mohan Learned Co-Authoring “Data Products for Dummies”
Jan 9 2024
134 - What Sanjeev Mohan Learned Co-Authoring “Data Products for Dummies”
In this episode, I’m chatting with former Gartner analyst Sanjeev Mohan who is the Co-Author of Data Products for Dummies. Throughout our conversation, Sanjeev shares his expertise on the evolution of data products, and what he’s seen as a result of implementing practices that prioritize solving for use cases and business value. Sanjeev also shares a new approach of structuring organizations to best implement ownership and accountability of data product outcomes. Sanjeev and I also explore the common challenges of product adoption and who is responsible for user experience. I purposefully had Sanjeev on the show because I think we have pretty different perspectives from which we see the data product space. Highlights/ Skip to: I introduce Sanjeev Mohan, co-author of Data Products for Dummies (00:39)Sanjeev expands more on the concept of writing a “for Dummies” book   (00:53)Sanjeev shares his definition of a data product, including both a technical and a business definition (01:59)Why Sanjeev believes organizational changes and accountability are the keys to preventing the acceleration of shipping data products with little to no tangible value (05:45)How Sanjeev recommends getting buy-in for data product ownership from other departments in an organization (11:05)Sanjeev and I explore adoption challenges and the topic of user experience (13:23)Sanjeev explains what role is responsible for user experience and design (19:03)Who should be responsible for defining the metrics that determine business value (28:58)Sanjeev shares some case studies of companies who have adopted this approach to data products and their outcomes (30:29)Where companies are finding data product managers currently (34:19)Sanjeev expands on his perspective regarding the importance of prioritizing business value and use cases (40:52)Where listeners can get Data Products for Dummies, and learn more about Sanjeev’s work (44:33) Quotes from Today’s Episode “You may slap a label of data product on existing artifact; it does not make it a data product because there’s no sense of accountability. In a data product, because they are following product management best practices, there must be a data product owner or a data product manager. There’s a single person [responsible for the result]. — Sanjeev Mohan (09:31) “I haven’t even mentioned the word data mesh because data mesh and data products, they don’t always have to go hand-in-hand. I can build data products, but I don’t need to go into the—do all of data mesh principles.” – Sanjeev Mohan (26:45) “We need to have the right organization, we need to have a set of processes, and then we need a simplified technology which is standardized across different teams. So, this way, we have the benefit of reusing the same technology. Maybe it is Snowflake for storage, DBT for modeling, and so on. And the idea is that different teams should have the ability to bring their own analytical engine.” – Sanjeev Mohan (27:58) “Generative AI, right now as we are recording, is still in a prototyping phase. Maybe in 2024, it’ll go heavy-duty production. We are not in prototyping phase for data products for a lot of companies. They’ve already been experimenting for a year or two, and now they’re actually using them in production. So, we’ve crossed that tipping point for data products.” – Sanjeev Mohan (33:15) “Low adoption is a problem that’s not just limited to data products. How long have we had data catalogs, but they have low adoption. So, it’s a common problem.” – Sanjeev Mohan (39:10) “That emphasis on technology first is a wrong approach. I tell people that I’m sorry to burst your bubble, but there are no technology projects, there are only business projects. Technology is an enabler. You don’t do technology for the sake of technology; you have to serve a business cause, so let’s start with that and keep that front and center.” – Sanjeev Mohan (43:03) Links Data Products for Dummies: https://www.dataops.live/dataproductsfordummies“What Exactly is A Data Product” article: https://medium.com/data-mesh-learning/what-exactly-is-a-data-product-7f6935a17912It Depends: https://www.youtube.com/@SanjeevMohanChief Data Analytics and Product Officer of Equifax: https://www.youtube.com/watch?v=kFY7WGc-jFMSanjMo Consulting: https://www.sanjmo.com/dataops.live: https://dataops.livedataops.live/dataproductsfordummies: https://dataops.live/dataproductsfordummiesLinkedIn: https://www.linkedin.com/in/sanjmo/Medium articles: https://sanjmo.medium.com
133 - New Experiencing Data Interviews Coming in January 2024
Dec 26 2023
133 - New Experiencing Data Interviews Coming in January 2024
Today I am sharing some highlights for 2023 from the podcast, and also letting you all know I’ll be taking a break from the podcast for the rest of December, but I’ll be back with a new episode on January 9th, 2024. I’ve also got two links to share with you—details inside!   Transcript Greetings everyone - I’m taking a little break from Experiencing Data over December of 2023, but I’ll be back in January with more interviews and insights on leveraging UX design and product management to create indispensable data products, machine learning apps, and decision support tools.  Experiencing Data turned this year five years old back in November, with over 130 episodes to date! I still can’t believe it’s been going that long and how far we’ve come.  Some highlights for me in 2023 included launching the Data Product Leadership Community, finding out that the show is now in the top 2% of all podcasts worldwide according to ListenNotes, and most of all, hearing from you that the podcast, and my writing, and the guests that  I have brought on are having an impact on your work, your careers, and hopefully the lives of your customers, users, and stakeholders as well!  So, for now, I’ve got just two links for you: If you’re wondering how to either: support the show yourself with a really fast review on Apple Podcasts,to record a quick audio question for me to answer on the show, or if you want to join my free Insights mailing lists where I share my bi-weekly ideas and thoughts and 1-page episode summaries of all the show drops that I put out here on Experiencing Data. …just head over to designingforanalytics.com/podcast and you’ll get links to all those things there. And secondly, if you need help increasing customer adoption, delight, the business value, or the usability of your analytics and machine learning applications in 2024, I invite you to set up a free discovery call with me 1 on 1.  You bring the questions, I’ll bring my ears, and by the end of the call, I’ll give you my best advice on how to move forward with your situation – whether it’s working with me or not. To schedule one of those free discovery calls, visit designingforanalytics.com/go And finally, there will be some news coming out next year with the show, as well as my business, so I hope you’ll hop on the mailing list and stay tuned, that’s probably the best place to do that. And if you celebrate holidays in December and January, I hope they’re safe, enjoyable, and rejuvenating. Until 2024, stay tuned right here - and in the words of the great Arnold Schwarzenegger, I’ll be back.
132 - Leveraging Behavioral Science to Increase Data Product Adoption with Klara Lindner
Dec 12 2023
132 - Leveraging Behavioral Science to Increase Data Product Adoption with Klara Lindner
In this conversation with Klara Lindner, Service Designer at diconium data, we explore how behavioral science and UX can be used to increase adoption of data products. Klara describes how she went from having a highly technical career as an electrical engineer and being the founder of a solar startup to her current role in service design for data products. Klara shares powerful insights into the value of user research and human-centered design, including one which stopped me in my tracks during this episode: how the people making data products and evangelizing data-driven decision making aren’t actually following their own advice when it comes to designing their data products. Klara and I also explore some easy user research techniques that data professionals can use, and discuss who should ultimately be responsible for user adoption of data products. Lastly, Klara gives us a peek at her upcoming December 19th, 2023 webinar with the The Data Product Leadership Community (DPLC) where she will be going deeper on two frameworks from psychology and behavioral science that teams can use to increase adoption of data products. Klara is also a founding member of the DPLC and was one of—if not the very first—design/UX professionals to join.   Highlights/ Skip to: I introduce Klara, and she explains the role of Service Design to our audience (00:49)Klara explains how she realized she’s been doing design work longer than she thought by reflecting on the company she founded, Mobisol (02:09)How Klara balances the desire to design great dashboards with the mission of helping end users (06:15)Klara describes the psychology behind user research and her upcoming talk on December 19th at The Data Product Leadership Community (08:32)What data product teams can do as a starting point to begin implementing user research principles (10:52) Klara gives a powerful example of the type of insight and value even basic user research can provide (12:49)Klara and I discuss a key revelation when it comes to designing data products for users, which is the irony that even developers use intuition as well as quantitative data when building (16:43)What adjustments Klara had to make in her thinking when moving from a highly technical background to doing human-centered design (21:08)Klara describes the two frameworks for driving adoption that she’ll be sharing in her talk at the DPLC on December 19th (24:23)An example of how understanding and addressing adoption blockers is important for product and design teams (30:44)How Klara has seen her teams adopt a new way of thinking about product & service design (32:55)Klara gives her take on the Jobs to be Done framework, which she will also be sharing in her talk at the DPLC on December 19th (35:26)Klara’s advice to teams that are looking to build products around generative AI (39:28)Where listeners can connect with Klara to learn more (41:37)   Links diconium data: http://www.diconium.com/LinkedIn: https://www.linkedin.com/in/klaralindner/Personal Website: https://magic-investigations.com/Hear Klara speak on Dec 19, 2023 at 10am ET here: https://designingforanalytics.com/community/
131 - 15 Ways to Increase User Adoption of Data Products (Without Handcuffs, Threats and Mandates) with Brian T. O’Neill
Nov 28 2023
131 - 15 Ways to Increase User Adoption of Data Products (Without Handcuffs, Threats and Mandates) with Brian T. O’Neill
This week I’m covering Part 1 of the 15 Ways to Increase User Adoption of Data Products, which is based on an article I wrote for subscribers of my mailing list. Throughout this episode, I describe why focusing on empathy, outcomes, and user experience leads to not only better data products, but also better business outcomes. The focus of this episode is to show you that it’s completely possible to take a human-centered approach to data product development without mandating behavioral changes, and to show how this approach benefits not just end users, but also the businesses and employees creating these data products.    Highlights/ Skip to: Design behavior change into the data product. (05:34)Establish a weekly habit of exposing technical and non-technical members of the data team directly to end users of solutions - no gatekeepers allowed. (08:12)Change funding models to fund problems, not specific solutions, so that your data product teams are invested in solving real problems. (13:30)Hold teams accountable for writing down and agreeing to the intended benefits and outcomes for both users and business stakeholders. Reject projects that have vague outcomes defined. (16:49)Approach the creation of data products as “user experiences” instead of a “thing” that is being built that has different quality attributes. (20:16)If the team is tasked with being “innovative,” leaders need to understand the innoficiency problem, shortened iterations, and the importance of generating a volume of ideas (bad and good) before committing to a final direction. (23:08)Co-design solutions with [not for!] end users in low, throw-away fidelity, refining success criteria for usability and utility as the solution evolves. Embrace the idea that research/design/build/test is not a linear process. (28:13)Test (validate) solutions with users early, before committing to releasing them, but with a pre-commitment to react to the insights you get back from the test. (31:50) Links: 15 Ways to Increase Adoption of Data Products: https://designingforanalytics.com/resources/15-ways-to-increase-adoption-of-data-products-using-techniques-from-ux-design-product-management-and-beyond/Company website: https://designingforanalytics.comEpisode 54: https://designingforanalytics.com/resources/episodes/054-jared-spool-on-designing-innovative-ml-ai-and-analytics-user-experiences/Episode 106: https://designingforanalytics.com/resources/episodes/106-ideaflow-applying-the-practice-of-design-and-innovation-to-internal-data-products-w-jeremy-utley/Ideaflow: https://www.amazon.com/Ideaflow-Only-Business-Metric-Matters/dp/0593420586/Podcast website: https://designingforanalytics.com/podcast
130 - Nick Zervoudis on Data Product Management, UX Design Training and Overcoming Imposter Syndrome
Nov 14 2023
130 - Nick Zervoudis on Data Product Management, UX Design Training and Overcoming Imposter Syndrome
Today I’m joined by Nick Zervoudis, Data Product Manager at CKDelta. As we dive into his career and background, Nick shares insights into his approach when it comes to developing both internal and external data products. Nick explains why he feels that a software engineering approach is the best way to develop a product that could have multiple applications, as well as the unique way his team is structured to best handle the needs of both internal and external customers. He also talks about the UX design course he took, how that affected his data product work and research with users, and his thoughts on dashboard design. We discuss common themes he’s observed when data product teams get it wrong, and how he manages feelings of imposter syndrome in his career as a DPM.  Highlights/ Skip to: I introduce Nick, who is a Data Product Manager at CKDelta (00:35)Nick’s mindset around data products and how his early career in consulting shaped his approach (01:30)How Nick defines a data product and why he focuses more on the process rather than the end product (03:59)The types of data products that Nick has helped design and his work on both internal and external projects at CKDelta (07:57)The similarities and differences of working with internal versus external stakeholders (12:37)Nick dives into the details of the data products he has built and how they feed into complex use cases (14:21)The role that Nick plays in the Delta Power SaaS application and how the CKDelta team is structured around that product (17:14)Where Nick sees data products going wrong and how he’s found value in filling those gaps (23:30)Nick’s view on how a digital-first mindset affects the scalability of data products (26:15)Why Nick is often heavily involved in the design element of data product development and the course he took that helped shape his design work (28:55)The imposter syndrome that Nick has experienced when implementing this new strategy to data product design (36:51)Why Nick feels that figuring things out yourself is an inherent part of the DPM role (44:53)Nick shares the origins and information on the London Data Product Management meetup (46:08) Quotes from Today’s Episode “What I’m always trying to do is see, how can we best balance the customer’s need to get exactly the data point or insight that they’re after to the business need. ... There’s that constant tug of war between customization and standardization that I have the joy of adjudicating. I think it’s quite fun.” — Nick Zervoudis (16:40) “I’ve had times where I was hired, told, 'You’re going to be the product manager for this data product that we have,' as if it’s already, to some extent built and maybe the challenge is scaling it or bringing it to more customers or improving it, and then within a couple of weeks of starting to peek under the hood, realizing that this thing that is being branded a product is actually a bunch of projects hiding under a trench coat.” — Nick Zervoudis (24:04) “If I just speak to five users because they’re the users, they’ll give me the insight I need. […] Even when you have a massive product with a huge user base, people face the same issues.” — Nick Zervoudis (33:49) “For me, it’s more about making sure that you’re bringing that more software engineering way of building things, but also, before you do that, knowing that your users' needs are going to [be varied]. So, it’s a combination of both, are we building the right thing—in other words, a product that’s flexible enough to meet the different needs of different users—but also, are we building it in the right way?” – Nick Zervoudis (27:51) “It’s not to say I’m the only person thinking about [UX design], but very often, I’m the one driving it.” – Nick Zervoudis (30:55) “You’re never going to be as good at the thing your colleague does because their job almost certainly is to be a specialist: they’re an architect, they’re a designer, they’re a developer, they’re a salesperson, whereas your job [as a DPM] is to just understand it enough that you can then pass information across other people.” – Nick Zervoudis (41:12) “Every time I feel like an imposter, good. I need to embrace that, because I need to be working with people that understand something better than me. If I’m not, then maybe something’s gone wrong there. That’s how I’ve actually embraced impostor syndrome.” – Nick Zervoudis (41:35) Links CKDelta: https://www.ckdelta.ieLinkedIn: https://www.linkedin.com/in/nzervoudis/
129 - Why We Stopped, Deleted 18 Months of ML Work, and Shifted to a Data Product Mindset at Coolblue
Oct 31 2023
129 - Why We Stopped, Deleted 18 Months of ML Work, and Shifted to a Data Product Mindset at Coolblue
Today I’m joined by Marnix van de Stolpe, Product Owner at Coolblue in the area of data science. Throughout our conversation, Marnix shares the story of how he joined a data science team that was developing a solution that was too focused on the delivery of a data-science metric that was not on track to solve a clear customer problem. We discuss how Marnix came to the difficult decision to throw out 18 months of data science work, what it was like to switch to a human-centered, product approach, and the challenges that came with it. Marnix shares the impact this decision had on his team and the stakeholders involved, as well as the impact on his personal career and the advice he would give to others who find themselves in the same position. Marnix is also a Founding Member of the Data Product Leadership Community and will be going much more into the details and his experience live on Zoom on November 16 @ 2pm ET for members.   Highlights/ Skip to: I introduce Marnix, Product Owner at Coolblue and one of the original members of the Data Product Leadership Community (00:35)Marnix describes what Coolblue does and his role there (01:20)Why and how Marnix decided to throw away 18 months of machine learning work (02:51)How Marnix determined that the KPI (metric) being created wasn’t enough to deliver a valuable product (07:56)Marnix describes the conversation with his data science team on mapping the solution back to the desired outcome (11:57)What the culture is like at Coolblue now when developing data products (17:17)Marnix’s advice for data product managers who are coming into an environment where existing work is not tied to a desired outcome (18:43)Marnix and I discuss why data literacy is not the solution to making more impactful data products (21:00)The impact that Marnix’s human-centered approach to data product development has had on the stakeholders at Coolblue (24:54)Marnix shares the ultimate outcome of the product his team was developing to measure product returns (31:05)How you can get in touch with Marnix (33:45) Links Coolblue: https://www.coolblue.nlLinkedIn: https://www.linkedin.com/in/marnixvdstolpe/
128 - Data Products for Dummies and The Importance of Data Product Management with Vishal Singh of Starburst
Oct 17 2023
128 - Data Products for Dummies and The Importance of Data Product Management with Vishal Singh of Starburst
Today I’m joined by Vishal Singh, Head of Data Products at Starburst and co-author of the newly published e-book, Data Products for Dummies. Throughout our conversation, Vishal explains how the variations in definitions for a data product actually led to the creation of the e-book, and we discuss the differences between our two definitions. Vishal gives a detailed description of how he believes Data Product Managers should be conducting their discovery and gathering feedback from end users, and how his team evaluates whether their data products are truly successful and user-friendly.   Highlights/ Skip to: I introduce Vishal, the Head of Data Products at Starburst and contributor of the e-book Data Products for Dummies (00:37)Vishal describes how his customers at Starburst all had a common problem, but differing definitions of a data product, which led to the creation of his e-book (01:15)Vishal shares his one-sentence definition of a data product (02:50)How Vishal’s definition of a data product differs from mine, and we both expand on the possibilities between the two (05:33)The tactics Vishal uses to useful feedback to ensure the data products he develops are valuable for end users (07:48)Why Vishal finds it difficult to get one on one feedback from users during the iteration phase of data product development (11:07)The danger of sunk cost bias in the iteration phase of data product development (13:10)Vishal describes how he views the role of a DPM when it comes to doing effective initial discovery (15:27)How Vishal structures his teams and their interactions with each other and their end users (21:34)Vishal’s thoughts on how design affects both data scientists and end users (24:16)How DPMs at Starburst evaluate if the data product design is user-friendly (28:45)Vishal’s views on where Designers are valuable in the data product development process (35:00)Vishal and I discuss the importance of ensuring your products truly solve your user’s problems (44:44)Where you can learn more about Vishal’s upcoming events and the e-book, Data Products for Dummies (49:48) Links Starburst: https://www.starburst.io/Data Products for Dummies: https://www.starburst.io/info/data-products-for-dummies/“How to Measure the Impact of Data Products with Doug Hubbard”: https://designingforanalytics.com/resources/episodes/080-how-to-measure-the-impact-of-data-productsand-anything-else-with-forecasting-and-measurement-expert-doug-hubbard/Trino Summit: https://www.starburst.io/info/trinosummit2023/Galaxy Platform: https://www.starburst.io/platform/starburst-galaxy/Datanova Summit: https://www.starburst.io/datanova/LinkedIn: https://www.linkedin.com/in/singhsvishal/Twitter: https://twitter.com/vishal_singh
127 - On the Road to Adopting a “Producty” Approach to Data Products at the UK’s Care Quality Commission with Jonathan Cairns-Terry
Oct 3 2023
127 - On the Road to Adopting a “Producty” Approach to Data Products at the UK’s Care Quality Commission with Jonathan Cairns-Terry
Today I’m joined by Jonathan Cairns-Terry, who is the Head of Insight Products at the Care Quality Commission. The Care Quality Commission is the the regulator for England for health and social care, and Jonathan recently joined their data team and is working to transform their approach to be more product-led and user-centric. Throughout our conversation, Jonathan shares valuable insights into what the first year of that type of shift looks like, and why it’s important to focus on outcomes, and how he measures progress. Jonathan and I explore the signals that told Jonathan it’s time for his team to invest in a designer, the benefits he’s gotten from UX research on his team, and the recent successes that Jonathan’s team is seeing as a result of implementing this approach. Jonathan is also a Founding Member of the Data Product Leadership Community and we discuss his upcoming webinar for the group on Oct 12, 2023.   Highlights/ Skip to: I introduce Jonathan, who is the Head of Insight Products at the Care Quality Commission in the UK (00:37)How Jonathan went from being a “maths person” to being a “product person” (01:02)Who uses the data products that Jonthan makes at the Care Quality Commission (02:44)Jonathan describes the recent transition towards a product focus (03:45)How Jonathan expresses and measures the benefit and purpose of a product-led orientation, and how the team has embraced the transformation (07:08)The nuance between evaluating outcomes and measuring outputs in a product-led approach, and how UX research has impacted Jonathan’s team (12:53)What signals Jonathan received that told him it’s time to hire a designer (17:05)How Jonathan’s team approaches shadowing users (21:20)Some of the recent successes of the product-led approach Jonathan is implementing on his team (25:28)What Jonathan would change if he had to start the process of moving to outcomes over outputs with his team all over again (30:04)Get the full scoop on the topics discussed in this episode on October 12, 2023 when Jonathan presents his deep-dive webinar to the Data Product Leadership Community. Available to members only. Apply today. Links Care Quality Commission: https://www.cqc.org.uk/LinkedIn: https://www.linkedin.com/in/jcairnsterry
126 - Designing a Product for Making Better Data Products with Anthony Deighton
Sep 19 2023
126 - Designing a Product for Making Better Data Products with Anthony Deighton
Today I’m joined by Anthony Deighton, General Manager of Data Products at Tamr. Throughout our conversation, Anthony unpacks his definition of a data product and we discuss whether or not he feels that Tamr itself is actually a data product. Anthony shares his views on why it’s so critical to focus on solving for customer needs and not simply the newest and shiniest technology. We also discuss the challenges that come with building a product that’s designed to facilitate the creation of better internal data products, as well as where we are in this new wave of data product management, and the evolution of the role.   Highlights/ Skip to: I introduce Anthony, General Manager of Data Products at Tamr, and the topics we’ll be discussing today (00:37)Anthony shares his observations on how BI analytics are an inch deep and a mile wide due to the data that’s being input (02:31)Tamr’s focus on data products and how that reflects in Anthony’s recent job change from Chief Product Officer to General Manager of Data Products (04:35)Anthony’s definition of a data product (07:42)Anthony and I explore whether he feels that decision support is necessary for a data product (13:48)Whether or not Anthony feels that Tamr qualifies as a data product (17:08)Anthony speaks to the importance of focusing on outcomes and benefits as opposed to endlessly knitting together features and products (19:42)The challenges Anthony sees with metrics like Propensity to Churn (21:56)How Anthony thinks about design in a product like Tamr (30:43)Anthony shares how data science at Tamr is a tool in his toolkit and not viewed as a “fourth” leg of the product triad/stool (36:01)Anthony’s views on where we are in the evolution of the DPM role (41:25)What Anthony would do differently if he could start over at Tamr knowing what he knows now (43:43) Links Tamr: https://www.tamr.com/Innovating: https://www.amazon.com/Innovating-short-guide-making-things/dp/B0C8R79PVBThe Mom Test: https://www.amazon.com/The-Mom-Test-Rob-Fitzpatrick-audiobook/dp/B07RJZKZ7FLinkedIn: https://www.linkedin.com/in/anthonydeighton/
125 - Human-Centered XAI: Moving from Algorithms to Explainable ML UX with Microsoft Researcher Vera Liao
Sep 5 2023
125 - Human-Centered XAI: Moving from Algorithms to Explainable ML UX with Microsoft Researcher Vera Liao
Today I’m joined by Vera Liao, Principal Researcher at Microsoft. Vera is a part of the FATE (Fairness, Accountability, Transparency, and Ethics of AI) group, and her research centers around the ethics, explainability, and interpretability of AI products. She is particularly focused on how designers design for explainability. Throughout our conversation, we focus on the importance of taking a human-centered approach to rendering model explainability within a UI, and why incorporating users during the design process informs the data science work and leads to better outcomes. Vera also shares some research on why example-based explanations tend to out-perform [model] feature-based explanations, and why traditional XAI methods LIME and SHAP aren’t the solution to every explainability problem a user may have.   Highlights/ Skip to: I introduce Vera, who is Principal Researcher at Microsoft and whose research mainly focuses on the ethics, explainability, and interpretability of AI (00:35)Vera expands on her view that explainability should be at the core of ML applications (02:36)An example of the non-human approach to explainability that Vera is advocating against (05:35)Vera shares where practitioners can start the process of responsible AI (09:32)Why Vera advocates for doing qualitative research in tandem with model work in order to improve outcomes (13:51)I summarize the slides I saw in Vera’s deck on Human-Centered XAI and Vera expands on my understanding (16:06)Vera’s success criteria for explainability (19:45)The various applications of AI explainability that Vera has seen evolve over the years (21:52)Why Vera is a proponent of example-based explanations over model feature ones (26:15)Strategies Vera recommends for getting feedback from users to determine what the right explainability experience might be (32:07)The research trends Vera would most like to see technical practitioners apply to their work (36:47)Summary of the four-step process Vera outlines for Question-Driven XAI design (39:14)   Links “Human-Centered XAI: From Algorithms to User Experiences” Presentation“Human-Centered XAI: From Algorithms to User Experiences” Slide Deck “Human-Centered AI Transparency in the Age of Large Language Models”MSR Microsoft ResearchVera's Personal Website
123 - Learnings From the CDOIQ Symposium and How Data Product Definitions are Evolving with Brian T. O’Neill
Aug 8 2023
123 - Learnings From the CDOIQ Symposium and How Data Product Definitions are Evolving with Brian T. O’Neill
Today I’m wrapping up my observations from the CDOIQ Symposium and sharing what’s new in the world of data. I was only able to attend a handful of sessions, but they were primarily ones tied to the topic of data products, which, of course, brings us to “What’s a data product?” During this episode, I cover some of what I’ve been hearing about the definition of this word, and I also share my revised v2 definition. I also walk through some of the questions that CDOs and fellow attendees were asking at the sessions I went to and a few reactions to those questions. Finally, I announce an exciting development on the launch of the Data Product Leadership Community.   Highlights/ Skip to:   Brian introduces the topic for this episode, including his wrap-up of the CDOIQ Symposium (00:29)The general impressions Brian heard at the Symposium, including a focus on people & culture and an emphasis on data products (01:51)The three main areas the definition of a data product covers according to Brian’s observations (04:43)Brian describes how companies are looking for successful data product development models to follow and explores where new Data Product Managers are coming from (07:17)A methodology that Brian feels leads to a successful data product team (10:14)How Brian feels digital-native folks see the world of data products differently (11:29)The topic of Data Mesh and Human-Centered Design and how it came up in two presentations at the CDOIQ Symposium (13:24)The rarity of design and UX being talked about at data conferences, and why Brian feels that is the case (15:24)Brian’s current definition of a data product and how it’s evolved from his V1 definition (18:43)Brian lists the main questions that were being asked at CDOIQ sessions he attended around data products (22:19)Where to find answers to many of the questions being asked about data products and an update on the Data Product Leader Community that he will launch in August 2023 (24:28) Quotes from Today’s Episode “I think generally what’s happening is the technology continues to evolve, I think it generally continues to get easier, and all of the people and cultural parts and the change management and all of that, that problem just persists no matter what. And so, I guess the question is, what are we going to do about it?” — Brian T. O’Neill (03:11) “The feeling I got from the questions [at the CDOIQ Symposium], … and particularly the ones that were talking about the role of data product management and the value of these things was, it’s like they’re looking for a recipe to follow.” — Brian T. O’Neill (07:17) “My guess is people are just kind of reading up about it, self-training a bit, and trying to learn how to do product on their own. I think that’s how you learn how to do stuff is largely through trial and error. You can read books, you can do all that stuff, but beginning to do it is part of it.” — Brian T. O’Neill (08:57) “I think the most important thing is that data is a raw ingredient here; it’s a foundation piece for the solution that we’re going to make that’s so good, someone might pay to use it or trade something of value to use it. And as long as that’s intact, I think you’re kind of checking the box as to whether it’s a data product.” — Brian T. O’Neill (12:13)   “I also would say on the data mesh topic, the feeling I got from people who had been to this conference before was that was quite a hyped thing the last couple years. Now, it was not talked about as much, but I think now they’re actually seeing some examples of this working.” — Brian T. O’Neill (16:25)   “My current v2 definition right now is, ‘A data product is a managed, end-to-end software solution that organizes, refines, or transforms data to solve a problem that’s so important customers would pay for it or exchange something of value to use it.’” — Brian T. O’Neill (19:47)   “We know [the product is] of value because someone was willing to pay for it or exchange their time or switch from their old way of doing things to the new way because it has that inherent benefit baked in. That’s really the most important part here that I think any data product manager should fully be aligned with.” — Brian T. O’Neill (21:35)   Links Episode 67Episode 110The Definition of Data ProductThe Data Product Leadership CommunityAsk me a question (below the recent episodes)
122 - Listener Questions Answered: Conducting Effective Discovery for Data Products with Brian T. O’Neill
Jul 25 2023
122 - Listener Questions Answered: Conducting Effective Discovery for Data Products with Brian T. O’Neill
Today I’m answering a question that was submitted to the show by listener Will Angel, who asks how he can prioritize and scale effective discovery throughout the data product development process. Throughout this episode, I explain why discovery work is a process that should be taking place throughout the lifecycle of a project, rather than a defined period at the start of the project. I also emphasize the value of understanding the benefit users will see from the product as the main goal, and how to streamline the effectiveness of the discovery process.  Highlights/ Skip to: Brian introduces today’s topic, Discovery with Data Products, with a listener question (00:28)Why Brian sees discovery work as something that is ongoing throughout the lifecycle of a project (01:53)Brian tackles the first question of how to avoid getting killed by the process overhead of discovery and prioritization (03:38)Brian discusses his take on the question, “What are the ultimate business and user benefits that the beneficiaries hope to get from the product?”(06:02)The value Brian sees in stating anti-goals and anti-personas (07:47)How creative work is valuable despite the discomfort of not being execution-oriented (09:35)Why customer and stakeholder research activities need to be ongoing efforts (11:20)The two modes of design that Brian uses and their distinct purposes (15:09)Brian explains why a clear strategy is critical to proper prioritization (19:36)Why doing a few things really well usually beats out delivering a bunch of features and products that don’t get used (23:24)Brian on why saying “no” can be a gift when used correctly (27:18)How you can join the Data Product Leadership Community for more dialog like this and how to submit your own questions to the show (32:25) Quotes from Today’s Episode “Discovery work, to me is something that largely happens up front at the beginning of a project, but it doesn’t end at the beginning of the project or product initiative, or whatever it is that you’re working on. Instead, I think discovery is a continual thing that’s going on all the time.” — Brian T. O’Neill (01:57) “As tooling gets easier and easier and we need to stand up less infrastructure and basic pipelining in order to get from nothing to something, I think more of the work simply does become the discovery part of the work. And that is always going to feel somewhat inefficient because by definition it is.” — Brian T. O’Neill (04:48) “Measuring [project management metrics] does not tell us whether or not the product is going to be valuable. It just tells us how fast are we writing the code and doing execution against something that may or may not actually have any value to the business at all.” — Brian T. O’Neill (07:33) “How would you measure an improvement in the beneficiaries' lives? Because if you can improve their life in some way—and this often means me at work— the business value is likely to follow there.” — Brian T. O’Neill (18:42) “Without a clear strategy, you’re not going to be able to do prioritization work efficiently because you don’t know what success looks like.” — Brian T. O’Neill (19:49) “Doing a few things really well probably beats delivering a lot of stuff that doesn’t get used. There’s little point in a portfolio of data products that is really wide, but it’s very shallow in terms of value.” — Brian T. O’Neill (23:27) “Anytime you’re going to be changing behavior or major workflows, the non-technical costs and work increase. And we have to figure out, ‘How are we going to market this and evangelize it and make people see the value of it?’ These types of behavior changes are really hard to implement and they need to be figured out during the design of the solution — not afterwards.” — Brian T. O’Neill (26:25) Links designingforanalytics.com/podcast: https://designingforanalytics.com/podcastdesigningforanalytics.com/community: https://designingforanalytics.com/community