Leaders Shaping the Digital Landscape
Dec. 21, 2023

Mitigating the Cost of Change

Much of what happens in IT optimizes for day 0 - or initial release or change. However, the downstream effects of those changes are rarely brought into consideration.

The risks we introduce.
The costs of audit and compliance.
The costs risk mitigation.
The costs of observability.

In this conversation with Michael Wood of HashiCorp, host Wade Erickson, digs into the ripple effects of change and how we seek to mitigate those downstream challenges via automation, forming platform teams, and more.

#automationengineering #automation #informationtechnology #softwaretesting #softwaredevelopment

Join host Wade Erickson in an insightful conversation with Michael Wood of HashiCorp as they delve into the often overlooked downstream effects of IT changes and strategies to mitigate them.

Key Takeaways:

  • Explore the risks introduced by IT changes and the importance of considering downstream effects.
  • Understand the costs associated with audit, compliance, risk mitigation, and observability.
  • Learn about automation strategies and the formation of platform teams to address downstream challenges in IT.
Transcript

Carlos Ponce (00:10):

Good morning everyone. Welcome to another episode of Tech Leaders Unplugged. And as we are getting ready to wrap up the year 2023, well today we're going to be having right here on Tech Leaders Unplugged a new guest, Michael Wood, who is the field, field, CTO at HashiCorp. Alright. And of course, thank you Wade as ever for being here, for graciously co-hosting, Tech Leaders Unplugged, and for playing a leading role on the show. So, alright, let's start with Michael. So Michael, tell us a little bit about you first. Tell us a little bit about your background, and your story, and then we'll go from there and then we're going to talk about HashiCorp.

Michael Wood (01:04):

Yeah, certainly. So a little bit about me, I suppose around, I, you know, after getting out of high school, I went to college and enjoyed probably the rec center a little more than I did courses. And ended up moving over to the military to help me pay for things like college. And ultimately ended up getting into management information systems, got into programming did a couple of startups early where I would do everything from networking to coding to storage, to running down to Office Max for office supplies, like whatever, whatever the office needed. We did everything to try to keep things going. Eventually, I moved into more of a field-facing or technical sales role where I was an engineer implementing things on the customer side. Moved into enterprise architecture agile coaching. As, as we made all these generational shifts into you probably remember some of these movements from three-tier applications to EAI enterprise application integration patterns, which eventually turned into a service-oriented architecture, which eventually turned into microservices and cloud, and so on. And through that time, I I operated as a systems engineer and enterprise architect helping some of the largest companies in the world. And then suddenly there was this big bunch bump where I think I was promoted four times in about 18 months and found myself an executive over distributed engineering teams, scattered across the globe led a couple of go-to markets, built some best practice tech labs. So I went from a practitioner, very hands-on delivery, focused, those types of things into the executive realm. Looking at the possibilities of user-centric development practices, test-driven development so on and so forth. We just, we tried to put all of the core best practices that a lot of people feel are sort of a nirvana state, right? Paired programming, tight-knit innovation groups, two pizza teams releasing every five minutes, new features into production to drive user satisfaction, those types of things. And then eventually moved into more of this advisory role. As you can imagine, in that journey, you make a whole bunch of mistakes. I joke that I learned from my mistakes very well. And that's the type of thing that I look to share when I work with executive or technical decision-makers who are, who are working on their technical strategy. I work with them and just share my experiences. What did I do badly that led to problems? And then what were the things that led to tremendous victory? And then I hope that we help guide a program, budgeting, legal, and how do we get products to market. All those things I'm very interested in. I've, I've also pivoted into organizational incentives. So I became fascinated with how teams work. We, a lot of the mistakes that we make in it project work are, our incentives are just misaligned. People aren't inclined to consume what we're building. We're sort of at each other's throat sometimes when, when the organizations pit us against each other. And so I became fascinated with industrial organizational psychology, right? How do we form teams that produce high-quality work, right? I think we've got more tools than we know what to deal with. So the question is, it, and, and I guess our progress in, in our ability to deliver is more of a human challenge than it is a technical challenge. And so I became fascinated with workflow, which brought me over to HashiCorp. You know, HashiCorp was not looking to replace clouds or compete with VMware or compete, you know, like it wasn't, it wasn't trying to be the tech to rule them all. It was more about how we orchestrate workflow. How do we get teams to work together via producer-consumer models where, you know, I'm producing a web application firewall on your behalf, and you just kind of consume it? But I don't want you, I don't want you concerned with the level of threat or risk that you're introducing by punching holes in our firewall. Or, you know, not properly restricting bot traffic or what, you know, all the various mistakes I can make with a WAF or a network defense posture. I can make a whole bunch of mistakes. So how do we, how do we get a producer-consumer model that allows people to accomplish their jobs without being under constant threat and or cognitively overloaded? Like, there are too many things for me to think about to release an application. Anyway, that's a lot. But hopefully, that gives you a sense of how I got where I am.

Carlos Ponce (05:45):

Oh, absolutely. Well, that's quite a journey. So I can imagine that there have been some you're somehow can I say this? There's a lot of learning and, iterations I assume in producing products like, you know, Terraform, nomad Packer, Waypoint. So tell us a little bit about this. Tell us a little bit about your journey behind the magic of, of all these products on the suite and all that that HashiCorp has. And tell us a little bit about Hashi Corp in general.

Michael Wood (06:22):

Yeah. Hashi Corp is extremely interesting to me. I used to work for a company back in the day called BEA, which was like WebLogic server and Java. It was one of the early success stories. A quick a quick open source company that ran up to a billion dollars very, very quickly. Well, I don't guess we were open-source, but we were focused on Java. And it was a very powerful startup that was eventually acquired by Oracle. But we used to joke that BEA STA stood for built entirely by acquisition. Every product we had, every technology was purchased someplace else. HashiCorp is very much the opposite of that ethos. Every single one of the tools that exists were built you know, internally in order to drive an improved pattern. So if we look at I think at the time Mitchell and Armand, who were the co-founders saw things like cloud formation and said, wow, cloud formation's awesome. This infrastructure approach to, you know, codifying a set of instructions for the cloud to deliver is much more predictable than putting people's hands on a keyboard, on a UI, you know, requesting an S3 bucket and, you know, all of that type of stuff. I'd rather fire a set of instructions and get an outcome. And cloud formation's great. We, if you go back in the Twitter archives, you'll see a handful of tweets where we're just like, this is awesome. Congrats AWS we're super happy for what you're doing. But when we realized that it was restricted to only AWS-based workloads, and then, you know, Azure ARM is really geared at, at what Azure does and so on and so forth, we really saw the need to have something that was more ubiquitous as an infrastructure, as code layer. And that led to Terraforms development. Like, let's look at the best practices and patterns of what people are doing and say, what is a more optimal approach to tackling this from a multi-cloud perspective? So one of the core theses that we have is the world is not a single cloud, it's multi-cloud. And I need to prepare my IT operations for a world where I want to be flexible. I want to pivot. And especially with things like AI, you know, I'm going to want to use different capabilities from different clouds, probably, right? Like, I don't want to be so beholden to a single cloud contract that I'm inflexible. I can't move, I can't take advantage of innovation. And so how do we think about this stuff? And, and, and very, very quickly after things like Terraform, we realized we're looking at secrets in the clear in our config files, in our, in our configuration code within, within Terraform. And so we built a little utility called Vault to mask that from me. Give me a token, give me a key to the ticket, but don't show me the, you know, don't, don't show sorry, give me a ticket for the key. Don't show me the actual key and let me make an exchange someplace. And you, you take care of that identity stuff away from me. 'cause I don't want, you know, I don't want to have the bank's credentials on my desktop anywhere. I, I need to be more secure than that. So then Vault emerges as a product, and then, and then people ask us, how do you do not just machine identity, but how do you do human identity? And so, we take a look at the privilege access management space. We look at all the various players in the space, and we see some things that we think could possibly be improved. And so we begin to build something like boundary, you know, how do we, how do we think of privilege access management, or that pattern of least privilege access in a highly ephemeral cloud environment. So I would say at a high level, a lot of what we do at HashiCorp is in response to a particular problem. We do look at who's out there and consider acquisition. But if we see that there's a, there's an anti-pattern, or we think that we can answer a question that, that isn't being properly answered by the field of competitors, then we typically just build it. Now we have, just to be clear, we do, we did acquire a company called Blue Bracket not too long ago, which added things like credential scanning to what we can do with Vault. So, you know, if you're a CISO and you want to know how at risk your environment is, can I scan my GI repos? Can I scan my, you know, like my code repos, my any, any place that I'm distributing this information, my CACD pipelines, logging, et cetera can I see if there are any passwords out there? And rather than build that, that was, that was something that we saw some other folks were doing very, very well. And so we acquired a company and added it to the portfolio. So we're not opposed to acquisition or adding, you know, capabilities to improve what we do from an operational perspective. It just so happens that almost everything that we do, we build from the ground up.

Carlos Ponce (10:53):

I see. Well, that leads me to the actual topic of today's conversation. Yeah. So today we're gonna be talking about, as the topic chosen by you is mitigating the cost of change. Yeah. So we're gonna be discussing optimizing it for day zero and beyond, right? Mitigating downstream risks through automation. Let's start there. Tell us why you chose this particular topic and why you felt it was relevant for today's day and age. Let's, let's say that.

Michael Wood (11:30):

Yeah. I think in the cloud world we have gotten so very, very complex. I was joking with some of my friends at Microsoft and, and we were talking about, you know, Hey, you've got 30 plus services that fall into your security portfolio. How many of your customers fully understand all 30? You know? And the answer is, very few. You know, there, there's probably a handful, but most developers are not thinking about every single way that I can run an Azure Sentinel policy, or where I could apply Azure purview to scan for sensitive data. So, so we have this plethora of tools, and that's just the Microsoft-built tools. That's not the marketplace. The marketplace is another thousand vendors that, you know, sit in there to solve security issues for you. And so I think most organizations are moving from kind of this artisanal approach to consuming cloud. You know, I have a set of Azure people, and Azure people know Azure, and you just work with whatever tools Azure provides, and you have some AWS people that do the same thing over in AWS, and in GCP, we were doing image analytics over there. We're doing some kind of, you know, there's, there's various things that we're doing in these various clouds. And I just think it, it, it has become an audit nightmare a cost challenge, right? Can we apply best practices across all of these clouds in a clean, seamless sort of manner? And what ends up happening is organizations begin to centralize things like the audit function. We establish a cloud center of excellence, we build out a platform team, and we do something to try to get our arms around how much risk these changes that are being introduced at a very rapid pace. How much risk does this impose for us, right? I make a change to a network boundary control. What's the ripple effects? How much east-west traffic can you know, if I, if I allow nefarious traffic onto this network, do they have the right to bounce across to another AZ? Can they, can they float between accounts? Because I have trusted connectivity between different accounts within GCP. You know, I have all these various questions of the ripple effect of the risk associated with what I would call artisanal change smart people on keyboard making changes 24/7. And so a lot of what we've started to see is, is the need to systemically approach this through automation through and automation does not just mean, how do I write some code to fire at a set of cloud APIs? It's, I want to codify policy. I want to codify how a secret gets issued and under what conditions. You know, like, how, what, what's my authentication? How does my authentication handle, how, what are the authorization policies that govern when I issue or min a new secret for, for a particular workload or human or what have you? So a lot of the thought there was just I think the cost of change tends to be high in certain areas partially because we've introduced a whole bunch of complexity. But even things like that we know, are things like best practices. As we know we want to rotate database secrets fairly often, but if in order for me to get a database secret to something that's sensitive, I have to argue with an enterprise architecture group for a couple of weeks to figure out, you know, what level of privilege I should have and whether this application workload warrants the access and so on. And then I turn around and I wrestle with the DBAs on, you know, Hey, I need this. I need the secret. No, you don't need this much secret. You just need to read only. No, I need, to create a read update. You know, I need, I need, I need a full credential here for the following reasons. So if it takes me two or three weeks to get through the per missioning cycle to get a credential, and you, as the CISO say, I want to best practice that, we rotate these secrets every 45 days. I'm sorry, you're not going to, it's not going to happen. The, cost of the issuance of that secret is so high, you're not going to hit that target. The developers are going to rebel. Like, nobody's going to follow that. They're going to ask for a 10-year credential, and they're going to say, please forget you, you know me. Like, forget my number. 'cause I don't, I don't wanna go back through this, this process again. It's just too painful. So when I talk about things, that's, that's kind of a, I know it's kind of a quirky example, but it, I think it's fairly real. Even I see Snowflake credentials that are 10 years to live. And I'm, and I'm wondering to myself, you're in a dynamic cloud world. Why, why is that so big? It's normally there's a human approver in the way, and I don't want to deal with the human approver for whatever that access looks like. So at HashiCorp, we try to look at how to automate these things so that we lower that cost. In this case, we're talking about credential issuance or secrets issuance. Can I lower the cost of that workflow via an automation authorization policy? You know what I mean? Like, how do we automate this so that you have an assurance that credentials are not being issued in a, in too big of a too big of a scope but also allow me to do things like hit my best, best practices like the rotation of secrets regularly? Does that make sense? At a high level?

Carlos Ponce (16:29):

Oh yeah. Absolutely.

Wade Erickson (16:31):

I've been there. Then they're doing release nights and we wanted to update some credentials and, you know, somebody forgot something. And all of a sudden databases are locked out. Applications aren't logging in. Yeah. Very, very real situation.

Carlos Ponce (16:48):

Well, I'm sure that, you know, questions are starting to pop into my head, but I'm going to pass on the mic to Wade. So Wade, please.

Wade Erickson (16:56):

Yeah. You know we talked in about, you know, the multi-cloud environments that most companies are, like you said, presented with, I think, you know, I remember back in the day when everything was on-prem and we had stacks of servers to manage. And yeah. You know, for some of them, the vision of going to the cloud was to reduce the complexity of our infrastructure management and let the vendor take care of that, right? Yep. And so now what I think we're finding is that I think it's more complex. 'cause At least we were controlled by the tech stack that we had the servers, you know, whether it's VMware or whatever. And now with having the requirement to be multi-cloud, not only for risk management, but just because the technologies are available, and absolutely what I've found is that you have specialists in each of these cloud environments, and you, you've got to now almost duplicate or triplicate, your staffing just to support that. So I definitely can see the market that you're targeting here with, your tool sets to now kind of try to bring that back under control. 'cause It has gone pretty wild Yeah. In the last five, or six years. So yeah, you talked about so in, in you know, I was reading one of your white papers and it talks about, you know, you know, we, I don't know, we've probably been in this transition of heavy cloud migration about 10 years, I would say. I think I remember it about 2015. It was kind of, 2014, it was starting to get pushed. And it says you that, you know, you, there's three areas of adoption, people, process, and tools. You guys obviously are working on tools. We talked about the people aspect. Yep. What would you say are some of the challenges? We have tech, we have change management, like you said, the willingness to actually, you know, go towards the easy route and do that 10-year credential, right? What would you say now as the tools get better, obviously it should make the people's side get a little better. What are some of the challenges that you see are still out there for those aspects of it?

Michael Wood (19:07):

And I think this does bring us back into the people space. It's interesting because a lot of folks want to talk to us about how we structure our platform teams, how we release software, what we put in the hands of developers versus res, you know, keep from the hands of developers, right? They're, and, and whenever I say that, I think developers get a little bit nervous because, you know, they're, they don't want to, they want to be able to self-serve as much as possible. Our goal is to provide things like self-service but to remove things that should be outside of your purview, you know, away from you. Oftentimes, I'll ask like the C-suite if I'm talking to like the CISO, I'll ask, how creative do you want developers being with? And I'll, I'll come back to like, WAFs, how creative do you want a developer to be with their WAF policies? You know, bot detection, prevention, signature, denial of service attacks. You know what I mean? Like, there's a whole bunch of stuff that a WAF provides me that I can consume via the cloud, and the answer is not creative. I don't want them to be creative at all. I want a set of known, well-known configurations, and well-known defense policies that just get asserted every single time. And I don't want the developer to think about it. So we know there's a line to be drawn for where a developer, like, here's where the developer focuses. When it comes to things like credentials, if you're in Kubernetes, a lot of Kubernetes folks want to play with their credentials and get that. But oftentimes at a CISO, I recognize that that's a social engineering threat right? Now, if I can phish one of your developers who are managing their secrets and Kubernetes, that's an opening for me that I need to figure out how to allow them to move very, very quickly, but bring those credentials under management, very, very quickly, rotate 'em so that the developer doesn't no longer have access to the key, still have the social engineering threat. But I want to bring secrets under management very, very quickly. As an example, I'm just throwing out a handful of examples. So a lot of what we've seen here is, and Gartner's picked up on this, which is the notion of a platform team, a centralized group that works on a producer, consumer-style model. I want, I want to infrastructure assets. I want developers to consume vented assets. I don't want them handcrafting, you know, an environment within AWS or handweaving an AKS cluster. And, you know, or in e you know, if I'm over in Azure and I'm playing around with Azure functions, I don't necessarily want them hand weaving that whole environment because it's, it's too easy to leave something open for breach and open service, API on your QEs cluster, ET CDs not properly encrypted. It's too easy for me to make a mistake. And these aren't, these aren't I'm not talking about individuals who don't know what they're doing. They could be rock stars, but you just make one little mistake. Your discipline falls one time and now you're breachable. And so we want to think about like, the risk that those changes are going to imply. And then think of it through this operational model, which typically we're referring to as a platform team. The platform team is going to orchestrate the delivery of primarily infrastructure services, and focus the developers closer to that application stack. What are the core things that you need to be building? And you take all the risk and authority for the application and things like that. But we're gonna centralize the delivery of secrets, networking, storage, infrastructure, compute, like, all of that stuff needs to be handled as much as we can by experts who have automated it, tested it to the nth degree, made sure we're not introducing any undue risk. My deployment platforms will scan your source for open-source vulnerabilities as a last-mile check before allowing it to go into production. So we're gonna provide you with those safeguards. If you ever read like the I wanna say it was the CIO dilemma, or one of the, one of those books, we talk about it in terms of like, play with fences. How do I build a nice walled garden kind of experience for developers to go really, really fast? And Google will refer to that as psychological safety. I can execute a lot faster if I'm not worried. About the ripple effects of my changes. So how do I build that safe environment? And typically we're seeing it as an organizational response is a centralization around the platform team, and then automation with a consistent set of tools, like let's be prescriptive.

Wade Erickson (23:24):

Great. Great. You know, I know we're coming on time, and at the end, I like to try to spin it to a little more of a personal aspect to you. Yeah. You said in the beginning you went into the military and you, you know, you pivoted into a pretty sophisticated role in tech right after your military time. And then also you jumped from management, like you said where you were doing a lot of the work in the field and you jumped into a role with some quick promotions you had there. Can you tell me a little bit about that transition from military to tech? Because I know there are a lot of people out there who are looking for the best better ways to get into tech. Yeah. And then second, a little bit about your CTO journey, how you jumped into this executive role from being a practitioner. Yeah, I think in the audience I'd love to hear that story and maybe can help them in their career.

Michael Wood (24:16):

Yeah. Would love to. So in the military, I there's, there's a truism in the military, especially when you're enlisted. So you're not an officer, you're, you're a doer. I, I went in as an infantry soldier, so I was not in the technical field. I was I carried a rifle and a bunch of gear on my back and hiked through the woods. That was primarily my job, moved onto some vehicles there. But a trim is, is that you're typically broke as a soldier. You never have any money, so you find things to do to occupy your time. And I was building computers in my in my barracks room, just cause you need a while away, a bunch of free time, you know? And then there was a shortage in a group called Parts Labor and Logistics, which was a computer job where you were, you were scheduling vehicles for maintenance, you were ordering parts, you were, you know, it was, it was a logistics system on, on keeping vehicles healthy. Apparently, it's a year-long school. I learned the systems in about a week. But I realized I didn't understand how the Army filed things. So I built a, few macro-based, you know this is back in the day when Attach Mate, this is pre Microsoft Office. So I built a few programmatic answers to tell me how our spending was, what was the risk and what was our effective combat effectiveness. So if I took another vehicle out and pull this engine, what would that do to our ability to respond to the threat? Was, so I built a couple of reports. And then, the commander came through and said, that's not standard. Why do you have these printouts dot matrix printouts on the wall? And I explained, well, our combat effectiveness right now is 87%, and our, our spending for the month, quarter year is trending like this. And I just wanted to be aware that the next time I press a button, I have an idea of what I'm affecting. And he was like, come with me. And all of a sudden I was advising like the colonel on, you know, financial disposition, combat effectiveness, and things like that. And that's when I caught the bug of, Hey, there's something in this. If I can rationalize data, if I can make sense of the world through an application, there might be a career for me in this, you know? And then I got out and went into the, went into school for management information systems. I would say, in terms of moving from practitioner into leadership roles I always had a view toward the systemic. Like, I never, so in other words, I was, I was looking above, I, I was working on a tree for sure, but I had my eyes up to see the forest. So if I if I take this tree out, what does this do to the larger execution of this overarching program? And I was much more fascinated by the bigger picture than I was by the small picture. I would execute the small picture as quickly as I could, but I needed to stay. I, I was just very passionate about the big picture. What are we trying to accomplish? How are we going to get there? What's this going to look like? As I was an engineer I started building a couple of programs to help others optimize what they were doing. Not just what I was doing, but hey, let me take some of these core practices and establish a program or two, just cause it wasn't in my job description, but I felt the need to answer a larger systemic problem. Naturally, leaders began to recognize that I was, that I was trying to answer a systemic problem that was on leadership's minds. And, and I was doing it from the execution level. And then all of a sudden I was a senior manager with a team trying to deliver this larger systemic answer. And then all of a sudden I was a director with multiple teams and all of a sudden I was senior director. You know, it was just like, and all of a sudden we were delivering systemic change over a larger swath of business. And that's ultimately, that's how I ended up in, in the executive level role is that accurate

Wade Erickson (28:05):

Solving business problems, getting seen by executives, and not just basically doing what you're asked, but looking for gaps and filling those gaps. And, being able to see where there's a need and executing without being asked, I think is kind of a summary that I heard.

Michael Wood (28:26):

Yeah. And, and my wife, it always bothered my wife. cause I, I've never really cared about my incentives, and I know that sounds funny, but I always exceeded my incentives, but I, I was never driven by them. I don't care. You can give me a number to go hit and I'll, I'll hit the number. That's no big deal. You know, like it's, it's okay. But what I really care about is team health, company health, you know what I mean? Like, like what are we trying to accomplish and can we, can we make that happen? And ultimately, if I'm, if I'm going to do something off to the side, which is not, not in alignment with my immediate incentives, it's okay, it's going to be better for the larger system if I do this thing, if I step outside my incentive and do this thing, then everybody else is better off because of it. And I never did it for the recognition. It, it never even occurred to me until all of a sudden I had I think I skipped reported when one of my bosses left to go work for another company. And, and I had a director who was like, what are you doing? This is awesome. We need to, we need to showcase this a little bit more. And then all of a sudden it, it broke out.

Carlos Ponce (29:32):

Alright, well Michael, we're unfortunately, we're coming up on time, but the first, well, but I want to, one of the first things that I wanted to say before we go is that I find it particularly commendable. You know, how you describe the way that you use your free time in the military to build a career in Tech. So it's like, it's like, it's, it's I mean, it's, it's a very interesting journey, I mean, to say the least. So I congratulate you for that. And, and also I thank you for sharing your journey right here on Tech Leaders Unplugged with us and with the audience. So that being said, I want to thank you for having been with us again on Tech Leaders Unplugged. Before we go, please stay us, stay with us as we go off the air. I got just a quick announcement and that is going to be about tomorrow's guest. Let me share with you that we're going to be having a conversation with Rob Harrop, the CTO of Bitso. And the topic is going to be The Fintechs of the Future, Anywhere and Everywhere: enabling FinTech innovation in remote-first organizations. So I look forward to that conversation tomorrow, right here on Tech Leaders Unplugged, same time. And please be here or be square. That being said again, thank you, Michael. Thank you Wade as ever. And I'll see you next time.

 

Michael WoodProfile Photo

Michael Wood

Field CTO

Michael Wood, serving as the Field CTO of HashiCorp, brings a profound understanding of organizational and technical transformations. With a keen interest in the intricate dynamics of change, he explores its effects on individuals and teams. Michael is particularly fascinated by the influence of change on our sense of self-worth and our capacity to reshape the world through the adoption of transformative technologies. His expertise lies in navigating the intersection of technology and human experience, making him a valuable leader at the forefront of HashiCorp's innovation journey.