Transforming Care Through AI: CX Innovation at City of Hope

Something that we found a lot of success with is we build with our clinicians, not for them. So HopeLLM or any other product that we have in production right now, we start from ideation. Usually, the idea comes from a doctor or a nurse or a clinician.

Welcome to Simply CX where we bring you customer voices and expert insights. I'm Nicole McKinley, leader of global customer experience at Microsoft. Today, I get to introduce you to someone whose work is truly changing lives, Dr. Nasim Eftekhari from City of Hope. Nasim is a visionary and a relentless problem-solver. She and her team are tackling some of the toughest challenges in cancer care, harnessing the power of AI to give doctors and nurses more time with their patients and to give hope to families searching for answers. Welcome, Nasim.

Thank you, Nicole. Glad to be here.

What inspired you to pursue a career applying AI in the healthcare space?

So I did not have any life sciences or healthcare background. My background is purely in computer science, machine learning, AI. As part of my grad school thesis, I came up with an algorithm to predict the outcome of political issues. And from there, we started a company that was focusing on stock market, commodity market predictions, political predictions, which was amazing. But at the same time, I always woke up every morning thinking about if this is adding anything positive to the world. So that's when I decided that I wanted to be in the healthcare space.

Very impressive. I mean, clearly, you're a purpose-driven leader and super inspiring. Is this leadership role at City of Hope the first role that you've had in healthcare?

City of Hope is kind of my first venture into healthcare. I did have a little bit of exposure to healthcare data because my advisor, who I did my thesis with and started a company with, he also had a healthcare company. And that was maybe where I could see things, how you could put these technologies into good use. And that was what inspired me initially. But I didn't have any healthcare AI experience before City of Hope.

I know City of Hope's legacy of innovation is pretty remarkable. And obviously, you've played a key role in bringing AI to the forefront. I love learning just now about how your expertise in AI broadly in a completely different industry kind of propelled you to have a much bigger, impactful purpose. What was the problem that you set out to solve with AI at City of Hope?

So lots of problems. Even before me, there were researchers at City of Hope doing a lot of cool stuff with AI. But when I joined City of Hope, I joined in the clinical informatics department, specifically to build predictive models for predicting adverse events or unwanted outcomes that we wanted to prevent. But at some point, a couple of years in, we were able to show so much impact that the organization decided we want this impact in more areas. So we started the department that I'm leading now, the department of applied AI and data science, to bring these technologies anywhere across City of Hope enterprise that we can meaningfully make an impact. That could be revenue cycle, philanthropy, that could be marketing, patient [inaudible] optimization. So lots of things in the operational efficiency space, but also even more importantly in research and precision medicine. So we've been collaborating very closely with our researchers. City of Hope has a very strong research arm. We have a world-class Bankman Research Institute. We have an affiliated genomics testing company called TGEN. So you can imagine that we have a lot of data, not just clinical, but also imaging, genomics, socioeconomics, variables, assessors. And in our research and precision medicine team, we focus on bringing all these data together and trying to make sense of things like, "Why does this therapy work for one patient but not for another patient with exactly the same disease and demographics?" So if you look at it, we are very much horizontally positioned in the organization, impacting operations, patient care, as well as research and precision medicine.

Wow. So many opportunities, right, to deliver significant impact to patients, to doctors, to how the business, the operations is running. How do you and your team decide where to start, where to prioritize?

We have a process for selecting which projects we work on. And that obviously has to do with, is it feasible, first of all? Can we solve this problem? Do we have the data to be able to do it? But also, what impact does it make? Is it really worth our time and effort and investment that we are putting into it? And when it comes to that question, is it worth it? A lot of people talk about AI ROI or return on investment. For us, return on investment is not just financial. We have six different impact dimensions that we look at for the projects that we work on to be able to, ahead of time, estimate the impact. Most important for us at CTO for patient outcomes, is this something that is going to impact our patients in a positive way that is very important? And there is patient experience, provider experience. There is direct and indirect financial impact. Obviously, the economists have to make sense as well. And then some of the things that we do have an impact on CT of our reputation and brand. So there is all these different impact dimensions that we look at. Sometimes they're not all applicable to a use case. But when there is ties, something may have a slightly negative financial impact, but positive patient care impact, that's what we care about most.

I love that. Yeah. And just putting customer outcomes first, having that really guide everything that you do, and the fact that you have such a disciplined six-dimensional framework that you use, not just for planning and prioritizing, but also assessing, did you actually achieve the intended outcome? What was the measurable impact? One of the most impactful innovations that you've been leading the charge on is HopeLLM. What is the HopeLLM initiative, and why has this been such a game-changer for CX in your environment?

HopeLLM is an oncology assistant. So think of your favorite AI assistants, these large language models that you can ask to point up your emails for you, or write something for you, or do something for you. HopeLLM is very similar. And the difference is that it can do oncology tasks very well because it has been trained on a lot of oncology data. So it's a large language model framework. It is not something that we trained from scratch. We're using a lot of multiple foundation models, both commercially available and open-source. And we fine-tuned it using a lot of oncology data. So City of Hope has been around for more than 100 years, and we have a lot of oncology data. We have data on rare diseases. We have a lot of genomics, imaging, pathology, progress notes, even psychology notes, socioeconomic discussions, and data. And so we've used all of that to make the underlying models be really tailored to oncology. So it can do oncology tasks very well, like clinical trial matching, document summarization, a lot of answering oncology-related questions, anything really related to oncology care, and sometimes even research.

So this is a tool specifically for doctors and aiding doctors to better help patients.

Yes. All the current applications of HopeLLM of are clinician-facing, and not just doctors, doctors, nursing, and the rest of the care team.

Yeah, I know from my research, one of the teams that partnered with you was precision medicine. Together, you really shaped some of the drivers for better customer experience.

We had a really good foundation, right? We've been doing this for many years. We've been doing machine learning, deep learning. One of the very last projects we worked on before HopeLLM was our precision medicine team actually came to us and said, "We have this precision medicine program where we are doing a lot of genomic testing for our patients and even sometimes their families to guide better treatments to improve the outcomes." And they were like, "We want to see if the precision medicine program is effective." And to be able to see if it's effective, we want to compare patients who are part of the program with the patients who are not part of the program in terms of their progression-free survival.

And progression-free survival is essentially the length of time someone lives with a disease like cancer without it getting worse, right?

Yes. So after they got a line of treatment, how long they were-- line of treatment that was informed by precision medicine, how long they were progression-free, and if that is more than the patients who did not get it. In order to do that, they needed to extract all lines of treatment and associated progression-free survival with it. And it may sound very simple. But this is something that is a very highly manual process that they had to read all the patient charts, sometimes thousands of pages, to extract those lines of treatments and then progression-free survival associated with it.

And so they turned to us, and they said, "For one patient, we are spending up to two days. And we want to do this for 20,000 patients, and this is not sustainable." So we started working on a deep learning model, an NLP deep learning model that would do exactly that, identify all the lines of treatments in our structured and unstructured data, and then read all the notes and see when progression happened.

And NLP, or Natural Language Processing is how AI learns to understand and work with human language.

Yes, this was very successful. We were able to identify progression or classify it in machine learning terms with like 99-plus percent accuracy. And we were like, "Okay, where do we go from now? This is progression-free survival. But there's all these other outcomes that people are interested in, researchers are interested in. And should we go and build deep learning models for each one of them?"

And this is when LLMs were just coming on the scene.

Yes. Absolutely. Suddenly, these foundation models emerged. And we were immediately able to tell that this is the direction we should be going. We started-- through a tech partnership that we had, we immediately had access to the [oken?] AI models. We started experimenting. And we saw that there's a lot of promise for us to be able to build and fine-tune something that can understand oncology context really well. So we set out to build GPT, but on steroids for oncology. And because of the foundation we had, the data that we had already pre-processed, and the expertise that we had, we were able to do it very quickly. But I should say that we didn't sleep for two, three months because how excited we all were. Those first few months, it's all a blur to me because we were all so excited, and we wanted to do something and not to be left behind this train. So that's how we started working on [inaudible]. And in a matter of few months, we had something that not only could extract all the data elements you wanted from patient history, but also could summarize them, could do clinical trials matching, and then a dozen different applications that used to be huge pain points for us before the foundation models.

I mean, the scale of what you're tackling is incredible. And I mean, helping doctors manage the overwhelming administrative burden, preventing them from having more time with patients and being able to identify methods and approaches for how to best help their patients and giving them more time with patients. Wow. I mean, again, incredible and super impressive. You talked about the early days of no sleep for weeks. What stood out for you in those early days as the breakthrough moment? Was there one where you're like, "Oh, we're onto something. We have something really special here."

Well, the first time I was able to talk to it, knowing that this is something we built internally that I can interact with in natural language, and it could do things given our patient histories. That was a very sweet moment. And I still enjoy it the same amount every time I do it, I enjoyed it the first time I started asking questions.

We've talked a lot about doctors and clinicians as your customers, but let's talk about their customers, patients, and their families, who in many cases may be navigating a critical moment in their lives. Do you find that you apply your knowledge and resources differently?

You're absolutely correct. It's a different bar for patient care. We do this in many different spaces, but when it comes to patient care, it's a different bar for accuracy. So the first version of [inaudible] for the application of document summarization was like maybe 80% accurate or 80% information coverage and accuracy and all of that. And with generative models, it's very easy to go from 0 to 80% almost overnight. But then we've spent years. We've had [inaudible] for more than two years now, and we've spent years putting in improvements, guardrails, making sure that the accuracy is enough for us to be able to put this into patient care practices. So the accuracy of our summaries now are more than 99.9% in terms of the information that it surfaces to our clinicians.

As you've been navigating that and recognizing that there's always a small percentage of errors, no human, no technology is perfect. How have you balanced or integrated human intuition or human brainpower into the process, into the system that you're building, if at all, to help the technology get smarter and be less error prone?

First of all, something that we found a lot of success with is we build with our clinicians, not for them. So [inaudible] or any other product that we have in production right now, we start from ideation. Usually the idea comes from a doctor or a nurse or a clinician. The idea for [inaudible] came from our chief medical officer, Dr. Trisok, who told me, "If you solve this problem, your statue is going to be on the main campus in front the main hospital." So I'm still waiting for that statue to be placed. But something that was such a huge pain point, according to him, and he's been in the oncology space forever. So we've got a lot of feedback, and we're still getting it. So HopeLLM for document summarization for onboarding our new patients, it was in Pilots with 40-something clinicians for 13 months before we made the decision to go live with it system-wide for all our clinicians to be able to use it.

Getting feedback from your users is so critical to gaining adoption and creating value. How does that play out?

We received feedback on a daily basis, improved the system on a daily basis. So a lot of credit for this performance goes to those clinicians who helped us perfect it. And then when it went live, we still have a very simple five-star scale feedback loop on the user interface. When they see the summary, they give us one star, five star, four star. And there is a very simple input box where they can comment and say, "This is not accurate," or, "This is too much information," or, "It's just too little information. It's missing this." And we read those feedback. That's the first thing I do when I open my laptop in the morning. And the last thing I do, when I close my laptop at night, we read these feedback entries religiously, all of us in my team, and we improve the system based on the feedback. So it is really something they have built. And I think when you do that, when you take that approach, people are also more likely to give you a pass if it's not exactly because they feel ownership, right? If it's not exactly like the format, or the look and feel is not exactly what they want, it helps with engagement. It helps with adoption.

Absolutely. Yeah. The build with model, I think, is a powerful one and one that's extensible, again, across industries, across the board in terms of a best practice to innovating and ensuring that you're using technology to really serve end customer, end user outcomes and their needs. I'm curious, were there any skeptics along the way? You're gathering all this feedback. You started small, which makes sense before you scaled, but were there skeptics? And how did you overcome the fear or those that maybe were doubting what's possible?

I'm proud to say that we've turned most of our skeptics to our biggest supporters by going to them without any ego and saying, "We're here to help you. What is it that you don't like about it? Is it inaccurate? Do you not like the formatting?" I mean, I take these one-star reviews that are very few. Because on average, we are four out of five. But we reach out to each user who gave us those reviews individually and say, "What is it that made you give us a one-star?" And just for them to see that, the methodology that's behind all of this, trying to explain to them, "Here's what happened, here's why HopeLLM made this mistake or missed this information, maybe because the source data wasn't even there for it to pick up and summarize," I think that communication has helped a lot. And then, they go into the system, and they do go to the system. Because it's such a hard thing to do manually, going through 1,000 pages of documents and trying to build a summary on your own, that even if HopeLLM isn't perfect, they go back to it. And they go back next time and see that exactly what they gave us feedback for is now addressed. They said, "I want dates in this format, not that format," and they see that it's fixed. And they give us a five-star and say, "Excellent, this is now exactly what I wanted." So some of those skeptics have turned to be our best supporters that now go and spread the word with other colleagues at City of Hope or even outside City of Hope. So that's great. But some people just don't like using new tools. And there's always going to be a percentage of people who won't use it, who think that AI is here to replace people and things like that. But for every one of them, I receive many emails, usually in the middle of the night, 2:00, 3:00 AM of a doctor who sends me an email and says, "Thank you. Because of this, I get to get more sleep. Something I would have had to spend five hours. It's 2:00 AM to prep for my next day visit, but thanks to you, I'm doing it in five minutes, and now I can go to bed." So we try to focus more on that and trying to bring that kind of impact to more people.

So fulfilling, I'm sure. And just again, back to your core purpose and what fuels you every day as an AI innovator and technology leader, really transforming the experience in such amazing ways. You mentioned the star rating system. I think it's fantastic that you're living and your solution is living this value of the customer's voice. In this case, it's the doctors and the clinicians. For those one or two stars that come in, on average, what kind of turnaround time do you and the team strive to act on that feedback?

So I can tell you that 90% of the non-perfect reviews that we receive is either because when the doctor is seeing the patient and when the HopeLLM summary was generated, there wasn't enough information for it to summarize. So we don't want a summary that says the patient is a 75-year-old woman and that's it, because they didn't have more access to more data about their diagnosis and everything else. So those are the things that are impacting a lot of people we address immediately.

And another example of that is a lot of times human errors happen in uploading the documents. Maybe there is a document that should not even be part of the summary. We have put now, again, AI agents in place that look to see every document is for the same patient. It's relevant to the summary. So we address those systematic issues immediately and first. The second thing we really prioritize is if there is an accuracy issue, which is very unlikely, but sometimes happens because the underlying document's quality is so bad. And then last but not least is people want things in different formats. Someone says, "I want this to be in this format. I want my dates to have months first." Someone says exactly the opposite.

Again, we are really focusing on user experience, so we are introducing a lot of customization into HopLLM. So everyone can customize it. They can say, "I'm a surgeon. I don't care about all this cancer history stuff. I only care about the last year. And surgery-related information first. I want the first sentence to be bold. I want bullet points." I don't want this and that. And they can do that. It will remember their style and able surface summaries to tailor to them. They can go back to default if they don't like it. They can continue making changes.

So those are the features that we are introducing in the next versions, making sure that we address-- again, we don't want to address feedback one-off as much as possible. We want to do something that will address large amounts of feedback.

Incredible. Was there a piece of input that changed how you think about feedback and how to action it?

So yes, there is one that jumps out because it was something that impact us and our approach to feedback. Every morning, I go directly to PowerPoint Activity Dashboard to the feedback logs and read each one of them. And there was this one-star feedback that someone gave us, and they went into lengths of also leaving a comment, and the comment said, "It sucks." So that was all that it was. And I was looking at it for a few seconds, and I was like, "Okay, this is not very helpful. I mean, how do we know what to do with this?"

Not super actionable, that's for sure.

Exactly.

At least on the surface.

Yes. And I don't know. Maybe this person who gave us this feedback, maybe they didn't know that we see who it's coming from. So we were not sure how to approach it. So we had to sit down with my team and say, "Seems like this person is not very happy with what they saw." So first, we started examining the summary to see if there is anything wrong with it. We didn't find that. And then we had to strategize how to approach them and how to really because this feedback is really important to us. So we had to strategize how to approach them and how to really get to the bottom of the issue. But when you say if you'd like that jumps at you, this was the one that left me thinking about like, "Okay, now what do we do with this?"

Well, what's incredible about your approach and what you and your team lean into is I think many would dismiss that piece of feedback, right? I think many would say, "Oh, nothing we can do here." And instead, you're taking every piece of feedback, even the most simplistic feedback and ambiguous feedback to a certain degree of like, "This sucks," and actually digging in and seeking to understand what is the real lived experience in this case that a doctor or clinician is having and what's behind the initial veneer of a few simple words and seeking to really uncover and then go solve for that experience and that set of challenges.

And I want to add that it had nothing to do with the technology. We did meet with them. We talked about what it was that they were not happy with, and it was this philosophical challenge of you're building these tools, this would replace people and our amazing nurses. And we were like, "No, no. So there was also an educational moment that came out of it that we were able to educate about what is it that we are trying to do?" And that these tools are just mental decision support tools, just to make people's lives easier. I showed them a graph that is something, again, as a point of pride for me and the team. You can see how many hours people save from 7:00 PM all the way to 5:00 AM. Time that they should not be spending-- they call it pajama time in healthcare, the time that they should be spending with their families and resting or sleeping. People have been spending this time prepping, reading through these crazy pods of documents. So I showed him and said, "This is what we are doing. These are all your colleagues. We can see their names, 2:00 AM, 3:00 AM, 4:00 AM. They use this tool to be able to get more sleep and rest, and time with their family. And I think that was a turning moment for them to see and realize what value this brings at scale to a lot of people. And again, one of the biggest skeptics that is now a big supporter is this person who told us it sucks.

Amazing. Amazing.

You talked about the impact this clearly has had on doctors and clinicians, saving them time, giving them back their evenings, and the ability to be with their families, to sleep, to re-energize, which is super important, and having more time with patients. What's been the impact on patients?

So the document summarization for patient referrals is inherently a doctor-facing tool. However, doctors tell me my patients love this because they go through these summaries with the patients, making sure that everything is captured. One of the doctors that we work with very closely, he says, "I go through the summaries with my patients. I tell them to use AI to gather everything from your history, from all the five different organizations you've been before and 20 years of cancer history. And I want to go through it with you." And he said that, "My patients love it." Because if you think about it, if you are someone who's had 20 years of cancer history, the patients and doctors share that burden of making sure nothing is missed, making sure that the doctor knows everything the patient has tried before, not to try a therapy they tried before again, or not to give them something they might have an adverse reaction to. So the patients historically will share that burden just in their most vulnerable time in their lives, making sure that the doctor has every detail that will impact their care moving forward. So that's one aspect of it. The other aspect is now we are also rolling out clinical trial matching at point of care using HopeLLM. It's already saving lives, detecting trials that a patient can be eligible for. Clinical trial matching is another extremely manual, time-consuming, very important thing, especially for cancer. And a lot of times, studies close because they cannot find enough patients. And patients cannot find the life-saving therapies that they need to be on that will save their lives. HopeLLM does this seamlessly. So when the doctor sees the patient, they can see two trials that the patient might be eligible for. So if the standard of care is not enough for them, or if they're out of options, they can consider the clinical trials. They can talk to the patient live when they are there, and they can expedite putting a patient on a trial.

That, I think, from a patient-facing perspective, I have gone with friends and family through this painful process of trying to find them an option when they are told they have no other options. And it's impossible to navigate clinicaltrials.gov if you don't have medical knowledge. Even when you do have medical knowledge, trying to find something that may save your life, a needle in a haystack of thousands of trials. So that is something that I am personally very passionate about, and I'm happy that we finally have the technology that we can enable this nationally for all patients.

Phenomenal. I mean, I literally have chills. So many of us have been impacted by a cancer diagnosis. I really appreciate the value this brings to find potential treatments and a path forward for those individuals and families. You mentioned scale earlier. How do we scale solutions like this?

So we tried this at City of Hope, validated it. We know it works. We know it's helping people saving lives. And not only we have a perspective, we have a plan. It will be available to more cancer centers, more healthcare systems, but also hopefully at some point to patients who can go somewhere and see which trials they can be a match for with a very patient, non-clinician-friendly summary. We already have the technology. It's just a matter of rolling it out, putting it out there. And City of Hope, our mission is to increase access to care. So this aligns very well with our mission. Our mission is to help everyone who's touched by cancer and diabetes. And I just can imagine a world where we have a very easy, just something like a website that people can go and say, "Here I am. Here's my medical records," or, "Here's the type of cancer that I have." And they can see what trials they have, what trials they might be a match for. They can filter based on distance, but everywhere in the world. The technology HopeLLM can tap into any database, clinicaltrials.gov, our internal database, or any other oncology database, and it can match a patient down to every inclusion/exclusion criteria. So we have the technology, and we have the desire to make it available to more patients. And I can only tell you that it will be coming soon.

Nassim, wow, such an amazing journey you've been on from your decision to give back to the world by then parlaying your talents to the healthcare industry. It's truly inspiring to see the impact you've made on your clinician stakeholders, doctors, and their patient customers. We are all counting on you to keep going. Truly, thank you for all that you are doing to improve the experience of those touched by cancer.

Thank you for having me.

Thank you for listening to Simply CX. I'm Nicole McKinley. Our show is produced by Larj Media. That's L-A-R-J Media. Special thanks to Luya Polka and Elizabeth Machado and to our video partner, Specular Studios. To find out more about today's guest, check out the links in the show notes. If you have your own challenges or questions, email us at simplycx@microsoft.com. And please share today's episode with someone you think could learn from it, and follow me on LinkedIn to keep the conversation going.