326. Rethinks: The Secrets to Getting the Information You Need from AI

[00:00:00] Matt Abrahams: AI can generate answers in seconds, but the quality of those answers depends on the quality of the conversation. I'm Matt Abrahams, and I teach Strategic Communication at Stanford Graduate School of Business. Welcome to this Rethinks episode of Think Fast, Talk Smart, the podcast. We're revisiting my conversation with Jeremy Utley and Kian Gohar to explore how conversational AI can become a true thinking partner, helping individuals and teams solve problems more creatively and communicate more effectively. Enjoy this journey into our communication vault.

[00:00:41] The promise of AI is tremendous. Yet most of us, if we're using it at all, are using it incorrectly. It's not about a transaction. It's about a conversation and interaction. I'm Matt Abrahams, and I teach Strategic Communication at Stanford Graduate School of Business. Welcome to Think Fast, Talk Smart, the podcast. Today, I am really excited to chat with Jeremy Utley and Kian Gohar. Jeremy is a repeat guest to Think Fast, Talk Smart. He's an adjunct professor specializing in creativity and entrepreneurship at Stanford, and the author of Idea Flow: The Only Business Metric That Matters. Kian is founder of Geolab and former executive director at Singularity University. He's the bestselling author of Competing in the New World of Work. Welcome, Jeremy and Kian. Thanks for being here.

[00:01:37] Jeremy Utley: Thanks for having me back. 

[00:01:38] Kian Gohar: Such a joy to be here with you. 

[00:01:40] Matt Abrahams: All right. Shall we get going?

[00:01:42] Jeremy Utley: Let's do it.

[00:01:42] Kian Gohar: Sounds good.

[00:01:43] Matt Abrahams: To get us started, Kian, I'm curious, what motivated you and Jeremy to look at the impact of AI on creativity and problem-solving? 

[00:01:51] Kian Gohar: So in my business, we coach teams to achieve complex business goals through team transformation practices. Sometimes this involves tackling a difficult innovation project, sometimes it's struggling with change management, but it always boils down to this one issue, which is: How can we solve this problem? And that usually entails human ideation and prioritization. And I wanted to see if we can bring a new technology tool into the mix to get better ideas. And given that I've been teaching AI in Silicon Valley to executives for a long time, that was always my hunch.

[00:02:26] But it wasn't until ChatGPT became publicly available that I had the ah-ha, and I didn't quite know how it could be useful to the issue of problem-solving on teams, but I had a hunch. And so we designed the study together, uh, to learn and from real world practice of how teams might use AI to get better ideas to solving problems. And what we did was we recruited hundreds of participants from many companies in Europe and the US, and we asked them to identify a particular pain point within their organization or a problem that they needed to solve for, something that was real.

[00:03:02] And then we actually gave half of the participants in each of these problem-solving ideation workshops had access to ChatGPT, and the other half didn't have access to it. So they were just thinking on their own as humans. And then the other half had humans plus AI. And we went through a problem-solving ideation exercise, and at the end of it, we asked the problem owners or the executives who shared with us the particular pain point to grade all the various ideas from A, being spectacular, all the way to D, not worth pursuing further. And so we ran this blind study to understand how generative AI can facilitate problem-solving, ideation, and collaboration with real world examples.

[00:03:47] Jeremy Utley: The other thing that we did was we asked participants before and after the session how they felt about collaboration and problem-solving and ideation because we wanted to see what was the impact of just a standard brainstorming session on someone's attitude towards problem-solving and collaboration and innovation. And what was the impact of a session that included generative AI? Did it have a differential impact on participant sentiment? 

[00:04:13] Matt Abrahams: And the research to me is fascinating. Jeremy, can you summarize the results of the research you and Kian did and relate it to the quality of idea generation and the feelings people have about the ideas that were created?

[00:04:27] Jeremy Utley: Very broadly speaking, we found that teams, if they want to outperform using AI, they need to adhere to certain practices. And sadly, most teams do not. So despite the potential to dramatically outperform, most teams leave the vast majority of their innovation potential on the table when they use AI. When you approach AI like an oracle or like a search engine just looking for the answer, it may feel like magic, but the data shows you end up underperforming. If you want to get world-class results and outperform, you can't approach it like an oracle. You need to instead approach AI like a conversation partner, and that doesn't feel nearly as magical. It feels more like work. And for that reason, the teams that outperform actually feel worse than the teams that underperform. It's very counterintuitive. 

[00:05:18] Matt Abrahams: I find that absolutely fascinating, and I also find that interestingly, the ability to have a good conversation with AI is what makes the difference for creating valuable ideas and solving problems. Kian, can you go into a little more depth with the counterintuitive findings that Jeremy just shared? Why do you think it is that AI-assisted teams that delivered worse solutions actually felt better about their work, and the AI-assisted teams who delivered better outcomes actually felt worse related to their non-AI-assisted counterparts? 

[00:05:50] Kian Gohar: Because rewiring our brains to act differently takes practice. And we professionals have decades of experience or certain ways of doing things, and we're asking them to work differently now with a different workflow, and that's hard. It's like going to the gym after not having exercised for a long time. It really sucks, and you don't particularly enjoy it at first, but it has long-term benefits. And so we found that teams that were able to use AI effectively, they actually had to rewire their workflow and how they went about to try to incorporate AI as a co-pilot on the team. And that's different, and it was exhausting, and they felt like it was just a lot of work, but ultimately, they had better responses.

[00:06:36] Matt Abrahams: Would you add something to that, Jeremy? 

[00:06:38] Jeremy Utley: Part of the challenge is, I almost picture a cartoon strip with a thought bubble that's like, "But I was promised superpowers." The premise and the promise of AI, I think it's intimidating, and it maybe strikes fear into a lot of people's hearts. That's one thing. I don't think there's a reason to be afraid. I think there's a lot of reason for enthusiasm and fun and seeking fluency, obviously. But I think there's a premise of this is gonna be like magic. But when you start there and you think, "All I have to do is ask a question, and then I get answers," well, guess what?

[00:07:07] You do. You ask a question, and you get three pages of, call it B+ documentation. And teams go, "It is kind of magical. Well, let's go get coffee." Like, we're kind of done, right? If you're expecting magic, you will get it, but you won't do that well. If you've gotten practice being a conversation partner to an AI copilot, what you realize is the first stuff that you get out isn't great unless you're really thoughtful about framing and providing context and digging deeper and pushing back and asking questions and treating it much more like a conversation.

[00:07:41] And then the more you treat it like a conversation, the less magical it feels, and the more it feels like AI is actually getting you to work and pulling your best thinking out of you. And so I think part of it is just the mindset people have when they hear, "Oh, we get ChatGPT." The people who go, "Oh, magic," are almost always the underperformers. The people who roll up their sleeves and go, "Oh, now we have work to do. It's maybe a different kind of work to do, but I'm excited to do it," those are the people who outperform. 

[00:08:10] Kian Gohar: I would also add that there is a reason why ChatGPT gives you generic answers, and it's designed mathematically to give you the most likely answer that it thinks you want to hear. And so as a result, the first few suggestions it offers are supposed to be generic and average. And so unless you have this conversational back and forth with it like you would with a colleague or a friend to push it and to give it context, you're just gonna get average answers and average ideas, and that's really not good enough if you're trying to solve complex problems.

[00:08:40] Matt Abrahams: It comes down to mindset shift and work. There is no magic. It's not for free. If you really want to solve problems well and get creative, you actually have to have a conversation. So Jeremy, can you share your FIXIT methodology and describe the different components and how they work? 

[00:08:57] Jeremy Utley: Absolutely. So FIXIT is a five-step methodology to turbocharge your collaboration. F stands for you have a focused problem, right? So you don't want to boil the ocean, but you want to be very focused in the kind of challenge that you are bringing to ChatGPT. So in this example, instead of saying what recommendations would you make for a human interfacing, I would say, "I'm joining a podcast, and I'd like to give concrete, tangible suggestions to professionals who haven't had much experience with ChatGPT.

[00:09:30] If you had three or four, or maybe I'd start with ten suggestions for someone who's unfamiliar but wants to be more familiar, start by asking me three questions to better understand the kind of professional I'm describing." Right? That's what we mean by F, kind of focused. So that's a very focused prompt, right? I is ideate individually. So before coming to the team, 'cause remember our study was conducted in the context of teams trying to solve problems. Before coming to your team and even before coming to ChatGPT, think for yourself about what do you know and what's your point of view.

[00:10:03] Again, this kind of hones the context that you bring to the conversation, right? It's critical. Research has shown that the best way to brainstorm is to alternate kind of individual ideation and group ideation. Well, the same is true actually in the context of collaborating with generative AI. You want to alternate between individual personal thinking and thinking assisted and amplified by AI, right? So that's the I. X is context, so providing sufficient background context. A lot of times we recommend that folks upload documents, right?

[00:10:35] When we were conducting our study, we would have a problem owner actually do a dossier that we would upload to ChatGPT to provide context. If you don't know what your context is, here's an amazing hack. Ask ChatGPT to ask you for the context. So as an example, I've got a friend who's negotiating a lease trying to build a gym, and he's trying to come to an agreement with the property owner, and there's a gap between where they need to be. And I said, "Hey, why don't you ask ChatGPT for help?" He said, "Well, how would I do that?"

[00:11:03] I said, "Well, have ChatGPT interview you about your objectives and then interview you about the counterparty's objectives and then make some recommendations." Well, that interview effectively becomes a means by which you can provide context. So the X is make sure that you're giving ChatGPT context on the problem. The second I is interactive iterative conversation, right? So you're never just taking the first response that ChatGPT gives you. You wanna be having a back and forth and a dialogue.

[00:11:34] So for example, if Matt and Kian and I are going to try to title this episode, we might say to ChatGPT, "Hey, we'd love ten titles for an episode." And then ChatGPT comes back with ten. Almost everyone's default is to think, "Which of these ten do I like best?" What we'd recommend is just immediately say, "I'd like another ten." And then read all twenty and say, "Here are the ones that I like. Would you give me ten more like these?" Chances are you're not gonna get as good. And then if you think, "Wait, why did I like number two?

[00:12:02] Oh, there was a funny alliteration, and number seven had a funny pun, and number nine made reference to..." Okay, use those as design principles for the next ten, right? That's an iterative back and forth that yields. We'd probably find that the fourth tranche of ten title suggestions would be radically exponentially better than the first ten, right? But it's by that going back and forth. F-I-X-I-T. T is for team incubation. So it's important to then bring the ideas that you've generated individually and with ChatGPT or whatever LLM you're using back to your team, and then importantly, commission some experiments.

[00:12:42] So this is where this work dovetails with traditional innovation methodology, but you never just wanna select one idea and move forward. It's impossible for an LLM or for a human to a priori know which solution is gonna be the best fit. So as a team, you wanna have a practice and a process around incubating or low-resolution prototyping a handful of the high-potential solutions that you've generated in order to determine which one actually solves the problem the best.

[00:13:10] Matt Abrahams: This methodology, upon hearing it, makes a lot of intuitive sense, but I can definitely see the effort involved. I like that it involves individual work and collaborative work, not just with the ChatGPT LLM, but also with others on the team. Kian, not surprisingly, I'd like to dive deeper into the fourth step that Jeremy introduced us to: interactive conversations. We've spent a lot of time on this podcast talking about how to have better conversations, but of course, focusing on humans. What specific advice can you provide to us for improving our conversations and our communication with LLMs to make sure we maximize the potential goodness that can come from collaborating with a tool like ChatGPT?

[00:14:00] Kian Gohar: Yeah. So the first thing I'd say is to make sure you've downloaded a LLM app on your phone and interact with an LLM on the phone instead of doing it on the web browser. If you don't see a app on your phone, it's very unlikely that you'll use it on a consistent basis. And the more you use it, the more familiar you become with it, the more likely it'll actually be part of your workflow, whether it's individually or whether it's as a team. So that's really the first step. Download one of these apps on your phone.

[00:14:30] And part of that is because we have historically been, for the last twenty-plus years in the internet era and the browser era, we see a text box, and we know exactly what to do with it. We type in a particular word or particular question. And now that we have, uh, large language models, the user interfaces look pretty much exactly the same, like the Google search engine box. And it's oftentimes difficult to figure out the right kind of question to ask it or to word it in the right way to get the right answer to suggestions.

[00:14:58] And so we actually think it's a lot better if you start interacting with these large language models through conversation, through spoken audio, rather than just trying to type it into a search engine box and trying to figure out the exact right prompt. The second thing is that you should be uploading your prompts with audio messages, voice messages, and talk to it just like you would talk to a friend. So whether you're talking to a friend about a particular problem or talking to a colleague about a particular problem, just record it literally on the phone on audio to text, and then the large language model will transcribe that and then offer out some suggestions.

[00:15:35] And then the third thing I'd say is think about using different models. There are several different kinds of large language models that you could use, from ChatGPT to Claude to Bing and to others, and they all have their different flavors and different kinds of personalities. And you might want to use one of those models for a particular exercise or activity, and then maybe you should go talk to another model, just like you would talk to two or three different friends and get their opinions or two or three different colleagues and get their opinions, because they're gonna give you different kinds of nuance and different kinds of answers.

[00:16:07] Matt Abrahams: I find that really helpful advice. I mean, we approach the communication with technology based on the interface that we have with it. And if we move to our phones, which we're used to communicating with very differently than we are, let's say, a browser and a text box, it can really change it. And it strikes me that it's not just the actual interaction interface, but it's also the curiosity we bring to it.

[00:16:30] When I go to enter information into a search engine, I just want to get the answer. I don't necessarily see it as a conversation and a dialogue. So that curiosity that I bring with the follow-up questions or the doubting and the exploration and expansion, I think, is really important. And I like the advice to check with different LLMs, just like you would check with different friends. It gives you different ideas and inputs 

[00:16:55] Jeremy Utley: You can even feed different LLMs answers into one another, right? 

[00:16:58] Matt Abrahams: Oh my goodness. We can facilitate and broker a conversation amongst LLMs. 

[00:17:03] Jeremy Utley: I, absolutely. I mean, I plug in ChatGPT's answer into Claude and say, "What do you think of this?" all the time, and vice versa. Take Claude's answer and plug it into ChatGPT. Ask ChatGPT what it thinks of it's own. So this is, this is why Googling is such a bad metaphor for what we're talking about here, because you'd never query Google about a Google query to, to get meta for a second, right? But after a sales call, I hop on my ChatGPT voice app while I'm stretching for a run. Instead of sitting at the screen, I'm stretching for a run. I say, "Hey, I just talked with Matt and Kion about our research, and a couple follow-ups.

[00:17:37] Would you craft a quick memo that I could send to the team just thanking them for their time today and how much I enjoyed the conversation," right? Well, it's gonna do it. Well, then my next thing is I just look at it and I say, "If I were to upload this memo to you, and I were to ask you for advice on how to make sure that they read it and respond, what three changes would you recommend?" And then immediately, I had this happen the other day.

[00:17:58] I was doing a voice vomit on a post-sales call, and I had ChatGPT write a memo, and then I asked ChatGPT, "How would you criticize this memo?" And ChatGPT said, "It's far too long for today's busy professionals. No one's gonna read this memo." And I said, "Would you please go ahead and shorten it to a point where you think that people will actually read it?" You can ask ChatGPT to evaluate its own work, and it will do so dispassionately, right? And that's the fun, but that requires iteration and a back and forth.

[00:18:26] Matt Abrahams: In many ways, the advice that you all are giving are advice that we give to people when they are engaged in empathetic dialogue in human-to-human interaction. It's about curiosity, questioning, about challenging in certain ways, and it changes the metaphor completely, as you mentioned, Jeremy.

[00:18:45] So before we end, I like to ask my guests a series of questions. Two are similar to everybody, and one that's very unique. So Jeremy, for those who haven't started with ChatGPT and large language models, where should they start?

[00:19:00] Jeremy Utley: I think for a lot of people, if, if they say, "Where should I start?" We've talked to so many professionals who say, "I've been meaning to try ChatGPT. I've been meaning to try generative AI." And that to us is a shame. Every single listener to this podcast could have at least ten hours of ChatGPT under their belt. And if you find yourself going, "Oh man, I'm behind," well, don't worry. You can get up to speed quickly. Here's a simple place to start. This is something that every single listener can do right now. Think of an emotional human decision you're trying to make in your life, something that you would ordinarily ask a partner or a friend or a spouse or a colleague about, okay?

[00:19:44] Think about what that is. Go to ChatGPT and say, "Hey, I'm trying to make this decision. Will you please ask me three or four questions before giving me your recommendation?" If every single listener will do that one simple activity, they're gonna have what we call a personal epiphany. That's gonna have cascading impact. All of a sudden they're gonna start thinking, "Could ChatGPT do this? Could I ask ChatGPT about that?" Right? But it must be emotional, it must be deeply personal, it must be the kind of thing that you would ordinarily ask another human being about. Get ChatGPT to ask you three or four questions about it before giving you advice, and start the snowball rolling that way. 

[00:20:28] Matt Abrahams: Question number two and three. Kian, I'm gonna ask you because Jeremy has previously answered these on an earlier episode, and I encourage everybody to listen to that episode where Jeremy and I talk about his book, Idea Flow. So Kian, who's a communicator that you admire and why? And you can't say ChatGPT. 

[00:20:47] Kian Gohar: I'm gonna give you two answers. One is Peggy Noonan, who is a columnist for The Wall Street Journal, and I so deeply admire her elegant prose and her personal anecdotes that make complex topics readily understandable. She was a speechwriter for George Bush Senior, and she developed the phrases that we became very familiar with during his presidency, like "Kindler, gentler nation," or "Read my lips, no new taxes."

[00:21:18] And the politics and the policy aside, the ability to translate big ideas into a few short words is just so masterful that I always enjoy reading her articles in The Wall Street Journal, regardless of the politics or the policy. And the second person I'd recommend is Sam Horn. She is an author of a book called Tongue Fu, and she is a complete master at teaching which words to lose and which words to use to convey meaning without tripping over yourself and creating unintended arguments. And these two are role models of how I think about communication. 

[00:22:00] Matt Abrahams: Thank you. Both of those recommendations, I hear you talking about words and the ideas that those words bring about. Both of those are individuals who, who have mastered that craft. 

[00:22:11] Kian Gohar: Because words are really conversations that are the seeds of innovation and how we think about solving big problems. And if we don't get the words, then we won't be able to actually get to the meaning and connection of what we're trying to solve for. 

[00:22:24] Matt Abrahams: Final question for you, Kian. What are the first three ingredients that go into a successful communication recipe?

[00:22:31] Kian Gohar: The first ingredient is something that might seem counterfactual when we're talking about a communication recipe, and that is listening. Listening to the environment, listening to the context, and listening to what your counterparty might be thinking. And so trying to understand the context by listening is super important. The second one is to know your audience. Who are you speaking to? Who's on the other side of the table? Who's in the room? What are their priorities, and what are their goals?

[00:23:01] And so that you can think about crafting your message in a way that resonates with them. And the third thing is to know your end goal. How do you want the audience of your intended communication to feel after the communication is over? Whether that's in person, whether that's in a meeting or whether that's online. How do you want the audience receiving the communication to feel? And then you work backwards from that to structure the flow and the necessary ingredients to make your communication easily relatable, understandable, and to land. 

[00:23:38] Matt Abrahams: We have certainly heard the notion of knowing your audience and being critical. I really appreciate and like this idea of backward mapping. Start from the outcome, and then we have to build and do and say and emote to get to that outcome. Thank you for that. And thank you both, Kian and Jeremy, for the conversation. I find your work fascinating and super helpful as generative AI becomes more and more commonplace. Further, I love how the concepts of effective communication and conversation and critical thinking can help our partnering with AI to be better, more creative problem solvers. To learn more about the results from Kian and Jeremy's work, go to howtofixit.ai and check out their article in HBR. Thank you so much.

[00:24:32] Jeremy Utley: Thanks for having us.

[00:24:35] Matt Abrahams: Thank you for joining us for another episode of Think Fast, Talk Smart, the podcast. It was great learning from Jeremy and Kian. Among the many things they shared, the one that stands out most to me is that AI is most powerful when we treat it as a collaborative thinking partner rather than simply a tool for finding answers. To learn more, please be sure to dive into our library of past episodes wherever you get your podcasts. Our back catalog is full of helpful communication tips and tools to help you be a better communicator. This episode was produced by Katherine Reed, Ryan Campos, and me, Matt Abrahams. Our music is from Floyd Wonder, with special thanks to Podium Podcast Company. Please find us on YouTube and wherever you get your podcasts. Be sure to subscribe and rate us. Also, follow us on LinkedIn, TikTok, and Instagram.