Broken Record Presents: A Musical Episode of Solvable

00:00:15Speaker 1: Pushkin. Hey, y'all, this is justin Richmond. I hope you're enjoying a nice and safe holiday season. We're looking forward to coming back to you with new episodes of Broken Record in twenty twenty one, starting on January twelve. But in the meantime, we wanted to gift you an interview Malcolm Gladwell did for another podcast called Solvable. Solvable is hosted by Pushkin CEO Jacob Weisberg and occasionally Malcolm as well. In this episode, Malcolm speaks with a software engineer who solved awkward video conference calls by reducing the delay time that occurs enough that musicians can now jam over Zoom in real time. Enjoy this episode. Here's Malcolm Gladwell. Was Solvable. This is Solvable. I'm Malcolm Gladwell. Occasionally we would get to go on break out rooms on Zoom and just discuss our feelings about what's happening at the time, and when the lockdown began, we were all devastated because singing was our life. Sam is twelve years old and like many kids, he's struggling with the restrictions COVID nineteen has imposed on his life. There's a lot of talking to news about the impact of COVID on classroom education. But for Sam and his friends, what they're missing is after school. Ragotsi was like a sanctuary to go to at the end of the school day. Rogatsi is a boy's choir based in California. Sam's been singing in the group for seven years. In pre COVID times, he would rehearse twice a week for several hours each day, but all of that came to a halt when COVID arrived in the US. The choir was basically stuck. If you've ever tried to sing something like Happy Birthday during a zoom meeting, you know what I'm talking about. Making music sync up with other people in real time over the Internet is really tough. The connect he delays, what they call latency, can be pretty awkward. Give us the technical explanation of the problem your son was facing. It's rooted in latency. Fundamentally, the time that it takes the sound you make to reach another musician that you're performing with. You need to have a latency of about less than twenty five milliseconds. Otherwise, the asense of the pulse or the beat of the music is just lost across those musicians. Mike Dickie is a software engineer, and after months of seeing his son Sam bummed out about not singing, he realized he could do something about it now for far less than the cost of a year of private music lessons. Dickie created a tool that allows groups of musicians to make music online. And it might be just the beginning. So we could have real time cheering and fan reaction in a basketball game. Yeah, absolutely, Performing in rehearsing with others in real time is really difficult or even impossible for many people. But this problem is solvable. I talked with Mike Dickie about Virtual Studio. It's a low cost device that works with the open source tool called jack Trip to make remote music collaboration possible in real time. Here is our conversation. Mike tell me about the first time this that Rugazzi rehearse using virtual Studio. What was that like? At first? It was a bit terrifying progress out. Let's progress down, okay and keep go. I thought of all the things that could go wrong, and I knew that this was all very new and untested, and I was also worried just of what people would think of me and my son if this was a huge disaster. But when we when we started getting people online and it's started to work, and the boys started to sing right. It was the first time they were singing together in a long time, and I could see through the zoom the faces on the different boys start to light up. I could see the smiles, I could see it in their eyes, and I could see it in the director's eyes, thinking, hey, this actually is working. That experience of seeing people and having them so happy and having their experience with others impact their lives, that's really what led me to put more and more time into this and ultimately to devote all my time towards it. Prior to the lockdown, people must have thought about this problem and tried to solve it. Were there existing technical solutions to the problem of how to sing remotely? There were, thankfully. I looked at several of them when I started out, and I came across this small project from out of Stanford University called jack Trip, which is an open source technology that was developed over a year ago by the Computer Music department. It seemed very promising. It had a high quality, lossless audio. It focused on low latency and over the Internet, and there were a lot of people already using it. The challenge that I found with that is that it was a command line tool. You had to open a terminal on your computer. In many cases, you had to build it from the source code, the C plus plus source code, and then you had to know the command lines and figure out the right command line options in order to use it with other people. That's great for somebody who's very technically oriented, but in my son's chorus, I was trying to figure out how how can we roll this out to a large number of young boys that don't have this advanced technical expertise. And it was actually a challenge that I thought I was well suited for because I understood the technical side and I also had the product background in order to try to make something that was fairly difficult to use fairly easy for most of the people that were out there. So we're used to is it a normal sonic collaboration, a conversation up with people singing, We're already used to a degree of latency. We've incorporated that into our definition of normal. You don't have to you don't have to take latency to zero to make it sound normal. In other words, Yes, it's all about tricking your brain. Sound travels at roughly one foot per millisecond, and so if you're ten feet away from someone, it takes you ten milliseconds to actually hear something that they say or a note that they play on their instrument, and ours have evolved to adapt to that. And when we play music with a group, our brains are used to having people that are a certain distance away. Once they get further than twenty five feet away from you, it becomes really hard for you to stay in sync with them. So the trick here with these software solutions is simply to get under that twenty five threshold with a way that the sound is being transmitted exactly. And there's many steps along that path when you think about it. There's there's the path from let's say you're singing. There's the path from your voice your vocal chords to the microphone. Then there's the path from the microphone to your computer or whatever audio device you're using. There's the process of digitizing that audio, turning it from an analog sound into digital samples that can be buffered and transmitted electronically. And then there's the process of sending all that data from your computer to somebody else's computer and then essentially reversing the process in order to get into the headphones that someone's listening to you on. And each of those steps adds latency, and the key is just minimizing the latency of as many of those steps as possible along the way. Can you give a layperson's explanation of what we mean when we talk about buffer size? Sure? Buffer size is similar to if you're a shipping company and you're trying to send something from China to the United States. You have to load all of these containers onto a ship and then send that ship across the ocean to its destination port. All those containers then get unloaded and delivered to someone else. So each of those containers could be considered a buffer. The ship itself could also be considered a buffer. So you're kind of grouping things into these larger buckets and then transmitting them from one location to another. Describe virtual studio to make what is this thing that you have created? So the device it's basically using a Raspberry pie microcomputer or a single board computer that fits in the palm of your hand, and that this board only costs about thirty five dollars, and so I combine that with a sound card from a company called Hi Fi Berry, which is extremely low latency, and it plugs directly into this single board computer, so you don't have to go through a USB bus, which by itself adds latency by the way, it just talks directly to the CPU and other components on that board, and that allows it to achieve a latency as low as about one millisecond. But when I realized that I could do that, and I tested it and measured it and everything else, then I realized, Hey, this is really possible. We can really do something with this. Wait back up for a moment. So there is this thing Jack Trip, which comes out of Stanford. How do they, without without getting two swallowed up in the technical but explain to a computer dummy like me, how they propose to solve this latency problem. So you have this lag how does Jack Trip solve that problem? There's really no rocket science to it. What Jack Trip does is it just tries to take the audio packets, to extract them from the chip sets that are in your sound card and transmit them over the internet as quickly and as efficiently as possible. A large part of that is just having small buffer sizes, so jack Trip effectively just tries to minimize the overhead of transmitting this data over the networks. How long did it take you to build the first kind of prototype of this? Building the raspberry pie itself was fairly easy. It just took a few days to convert that over. And I had a few people helping test things with me. And I remember the first time I tested it myself, just as a loop back to the server and back to hear myself, and I was certain I did something wrong because it was instantaneous, or at least to me, it sounded instantaneous, and I thought, now, this can't possibly be sending. But yeah, the more I dug into it, the more I convinced myself that this was really happening. Were you Do you remember what? Were you singing or speaking? Do you remember why? I can't say you don't want to hear anything? No, I was just I was whistling a lot. For usually I whistled from my test. Sometimes I do try to sing if I know nobody's listening or in the room. If you do try to sing, what do you try to sing, I sing about my dog. This is like didn't what is it that Alexander Graham Bell said to his assistant on the first telephone transmission. Was it like, you know, doctor Watson, come quick or something like that? This is like up there with that? And when was that when you did that first test and you couldn't believe it? When was that? That was probably in May? Oh wow, this whole thing went really quickly. Yeah. Yeah. We shut things down here in California in mid March, late March. And I started out just you know, some evenings I would look at what's out there and spend a little time here and there. And the amount of time I spent grew more and more every week, until finally I started really putting things together and building things and testing things. By may, So, walk me through how Rigazzi does their rehearsals. Now, how do you put this into practical service? Well, interestingly, Rigassi still starts all of their rehearsals with Zoom, and we still use that as a tool to carry the video and connect everyone visually with one another. And then they in parallel to running Zoom, they run jack Trip and they use jack Trip for the audio and that means that the video is very heavily delayed compared to the audio. It takes a lot longer for you to see something versus when you hear it. And one of the downsides of that is that conducting, we've learned, is just not possible. You can't conduct visually, in particular with the gesticulations that you would normally see a conductor doing throughout a performance. So what the conductors have done and learned over time is that they have started conducting through audio. They give audio cues throughout a performance to either become a little bit louder, a little bit softer, a little bit faster, and when to cut off. All these things ended up moving from the visual side to the audio. So we still we still use the video just to help everybody seeing one another, helps give them a little bit more sense of presence. So everyone does. Every boy in the choir have to have his own little virtual studio. Yes, early on a major donor that allowed us to distribute this out to the entire chorus. It's about one hundred and fifty dollars for the device, and that doesn't include the accessories, which would be the microphone and some cables. So altogether it's about two hundred dollars. Did you when you set out to solve this particular rob you quit your old job to focus on this? Was that was? That? Was that a big step? Did that seem like a rash step? It still seems like a rash step. It was. It was certainly a monumental and life changing step for me. Um I also realized that this isn't something that's unique to Ragatzi. This is a problem that's really general across the entire world. And to me that was that was one of those moments where I realized that I could do something to help, and I may not ever have an opportunity to have an impact like I could at that moment. How do people I mean, so you have this thing that's now being used by Rigazzi, how do you get the Rigazzi cam't be the only choir out there? Or people who might I mean, have I come from a family where three members of my family are in church choirs and they've been grounded since March. I don't think they think it's even possible to be able to sing together. How have you gotten the word out about virtual studio? Different mechanisms I'm certainly in the Within the Bay area itself, we had a number of connections to other choruses and performing arts organizations. We've also been working a lot with Chorus America, which is a great organization of many choruses across the country. So we're slowly getting the information and the word out that this is possible through through not just our mechanism, but different mechanisms. So it'll be interesting to see as this progresses over the next several months. But wait, I would go even further. There's a whole category of live experiences which could be recreated using this. So I'm thinking, I mean, I don't know whether you're a basketball fan, but imagine an NBA game where everyone who's watching, every fan who's watching the game, has one of these devices in their house, So we could have real time cheering and fan reaction in a basketball game. We can recreate a reel. Right now, they're just doing fake crowd noise. Right it's a total phony experience. But if I have if I elimit, eliminate the latency for fans to react to a sporting event, I've I've got a virtual I've got a realistic virtual fan experience. Do I not. Yeah, absolutely, Honestly, I haven't even thought about sports. But I think you're totally correct, Mike, Mike, this is this is you can this is the future of this, This is That's the first thing I thought about when when I was thinking of because this is this huge because every single I don't know why I'm going on like this, but every single sporting event right now and that goes on television is suffering huge ratings declines. And that is because the experience of watching sports doesn't seem real. And if you're telling me, if it's this easy, I just have to I gotta buy it. I gotta buy a device if I want to play. But people are they were they were paying one hundred and fifty bucks for their ticket to the to one game before. I mean, it's not like people aren't used to investing in a fan experience. Yeah, it's a it's a great point. I think what one challenge we have is just with the video side of it is we can make the audio very quick, but you're still going to see things delayed on the video side. Potentially, if the delay of the video is the same for everyone who's watching, that may not be an issue, like they'll still hear each other in sync with what's happening. Yeah. Yeah. What's fascinating to me is how you start with a very very specific and very very personal problem, which is your son is bombed out that he can't sing with his chorus, And now you know, we're six months later and you're thinking about recreating the way in which we experience live music, and I'm going on and on about rescuing the NBA, And is it I mean, it's this sort of funny and fascinating picture of how innovation works. It is. It's It's amazing, isn't it. Yeah, that whole process of innovation being this this thing that builds and builds and changes over time and ways you don't even expect it to. Yeah. That's That's what I've always loved about technology. It's always drawn me into the world of entrepreneurism and creating products. And now I have the opportunity to do that in a way that involves something I really love, which is music, but also is helping people. Where could you go to listen in on something that's been performance has been conducted or recorded over virtual studio. The Rigassi Voye Chorus has recorded and published a number of our rehearsals online on YouTube, so the best place to see a few examples is probably at Rigazzi's YouTube channel. One last thing, can you give us a little sense of what listeners can do if they want to make use of this new idea or virtual studio or they want to spread the word. Do you have any practical some practical words of advice to our listeners. Yes, I'd suggest going to our website at jacktrip dot org, where we have a lot of different resources and people bowl who are volunteers available to help answer questions. Mike Dickie is co founder and chairman of the Jack Tripp Foundation and the creator of Virtual Studio. And we're hearing the Ragatzi Boys Choir from California rehearsing from over thirty locations at once using jack Trip and Virtual Studio. Be sure to check out our show notes to learn more about online music collaboration and ways you can get involved. Next week, Unsolvable, just ahead of Thanksgiving, retired NASA astronaut Scott Kelly. We'll talk about how social isolation is solvable even when feels like you're trapped alone so many miles and miles away from those who love most. I hope you'll join us. Solvable is brought to you by Pushkin Industries. Our show is produced by Camille Baptista, Senior producer jos and Frank. Catherine Gerardo is our managing producer and our Executive producer is Miilabell. Special thanks to Kobe Gilford, Heather Fane, Eric Sandler, Carle Migliori and Kadisha Holland. I'm Malcolm Bobber