Recorded live at Black Hat 2021 in Las Vegas! Our vaxxed and masked podcast co-hosts, Brandon and Derick, take the show on the road to talk to real network and security folks about Zero Trust. Is it more than just a marketing buzzword? Is it even achievable? Listen to this one-of-a-kind episode to get the straight scoop.
Show notes
- Black Hat 2021: https://www.blackhat.com/us-21/schedule.html
- Zero Trust Definition: https://www.csoonline.com/article/3247848/what-is-zero-trust-a-model-for-more-effective-security.html
- Seeking Truth in Networking: From Testing to Verification (blog): https://forwardnetworks.com/seeking-truth-in-networking-from-testing-to-verification/
- Zeno’s Dichotomy Paradox: https://en.wikipedia.org/wiki/Zeno%27s_paradoxes
TRANSCRIPT
Brandon: What does zero trust mean to you?
Stranger 13: Trust no-one and verify everything.
Stranger 10: Re verifying every single time.
Stranger 2: Kind of a weird term.
Stranger 3: Marketing hype.
Stranger 6: So I think we’re getting there.
Stranger 1: I think there’s a few that do a decent job at it.
Stranger 8: Absolutely not. It is extremely difficult to implement.
Stranger 2: The industry always has a ways further to go.
Stranger 3: We’re running, we’re running faster, but we’re running against a car.
Stranger 5: I don’t know.
Derick: All right. Well thank you for your time.
Brandon2: If you couldn’t tell from the intro, I think this is going to be a fun episode. Today, we take our little show outside the zoom dungeon, masked, and vaxxed, and into the real world. Not exactly the real world, but the world of security. Yeah. Yes.
Brandon: What happened in Vegas is coming to your ears with our stories from the Black Hat conference, from Derick breaking things, to skewering a buzzword on the show floor, to an interesting breach from the past. And whether you’re networking or security or a little bit column A and a little bit column B, there’s something for you. But first, enjoy this ad from our sponsor and visualize the air quotes I’m making.
[00:03:21] Intro
Brandon: Last week Derick and I went to the Black Hat conference in Las Vegas. It was our first time in god-knows-how-long to be at a conference, to fly on a plane, even to meet in person. Yes. Despite having been on a whole bunch of shows together, I had no idea what Derek looked like in person.
Derick: I’m amazing. Of course.
Brandon: So anyway, I didn’t know what to expect , but let’s just say when a conference is named for the dark kind of hacker, you tend to want to take a few precautions. We’re talking, leave the laptop and your credit cards in the room, shut off your phone WiFi and any NFC, bring burner laptops and wipe them after, those kinds of a uh – trust-no-one precautions.
Derick: Yes. So why, why did we go here again?
Brandon: Well on the surface, it was a work trip to give product demos, but honestly, we went there to learn. I’m not a security expert, but I do know a few things. And one thing I know is that you really can’t detach security from networking. Because so much of what security teams have to do depends on an understanding of network connectivity.
And that could be daily tasks for SecOps and InfoSec teams, like finding out where an attacker lives, as in what IP address lives where, or where they’re going, or how they go there, or even just making firewall changes for application access.
So I was there to test a simple hypothesis – that if information on network connectivity were made more accessible, it could help security teams get things done. Seems reasonable, right? This is a new crowd for us.
Derick: Yeah, that that was my biggest worry. Like security people are very smart, right? Very clever, super brilliant. And, uh, also very unfiltered and very full of opinions, like everyone in tech. So credibility was on my mind going into this.
Brandon: Yeah, me too. And if people don’t know you, you want to give them a reason to talk to you. So our marketing team hatched a plan for how we might grab people, even if they had no idea who we were or what we did, or why in the world a company with the word networks in its name would be at a security conference.
So step one of the plan was to get noticed. We went for a bright yellow booth in a sea of what I’d call trustworthy colors, things like blues and blacks and dark grays, you know, enterprise colors, and step two is to make it fun. So it turns out my lead IT engineer is really into retro gaming. So he got some old Ataris off of eBay, figured out how we’d procure an old school CRT, the authentic kind from a Vegas pawn shop the day before, so we could play fully authentically with these ancient Ataris.
All right. So anyway, that’s the plan. We get to Vegas. It is hot. It is scorching. I think it had cooled down to a high of 112 from 120 Fahrenheit in the days before.
And it’s the first day of the show, Wednesday. And I’m trying to get there early for setup, but I hit construction. I hit traffic, the tram’s not working. I get there frantically late and others on the team are going out the door that I’m coming in. And they’re equally frantic and they’re saying we’re headed to Best Buy because as our luck would have it, the first thing that happened when Derick went to connect the TV’s co-ax jack was that it broke off in his hand.
Oops. And the show floor is going to open soon and no, we don’t have backups. We didn’t think of this particular eventuality.
Derick: So I’m like, Hi everyone. It’s a great meeting you for the first time. I’ve been, I’ve been here a year. I have not been to any of you in person. So I’m going to go ahead and destroy this TV real quick.
Brandon: Thanks Derick.
Derick: You’re welcome.
Brandon: So fast forward an hour, and our luck starts to turn around. Somehow 40 years later, an Atari can connect to the co-ax jack on a modern LCD …who’d have thunk? And then at 10:00 AM the fence that’s holding back this horde of hackers. It goes down, people start streaming in and operation Atari Drop is in full effect.
So here’s my pitch. Hey. Want to play a game? High score for the day gets to keep the Atari and we’ll even ship it to you in its original box. Really? Yeah.
Uh, Okay.
I’ll give it a try.
And it seemed to work! All kinds of people came by.
One guy came by and he talked about – his eyes just lit up – he talked about Christmas morning in 1982, when he got his Atari 400 for, I think it was $900.
And one person came by and really wanted to talk about retro gaming. I think this was like the first 10 minutes of the show and showed me this thing that I thought was an original chonker GameBoy. Like the one from 1989, that I actually have one and occasionally I play Tetris on, uh, but it was much cooler.
It was this case for a Raspberry Pi that ran all kinds of games and some people would come by and. Is that real. Seriously? Can I, can I touch it? Well, yes. it’s real. And yes, you can touch it. The Atari, that is.
So we had centipede the first day and then Pac-Man and Space Invaders…. people were having fun.
Derick: Yeah. So there was a 15 year old kid who showed up. He was on a, uh, on a team. They were all teenagers from Europe. They were on one of the CTF teams at the event.
Brandon: Capture flag, that is.
Derick: Yeah. Capture the flag. Yeah. And he came up very excited about playing this game and he had never played Centipede before. And especially not on that awkward controller.
Right. That’s not the rollerball from the arcade. So he stepped up and lost all three of his lives in the first like minute and a half. But then he, he sat there staring at the screen. I got to go again. I got to go again. And on the second try, he got 40,000 points. And while he’s sitting there playing right, he’s super-focused, and then he just blurts out, it’s all about the patterns, right?
Like it, is all about the patterns. That’s exactly what it is with these old games.
You think he was a savant, or…. ?
I Maybe, like, that’s incredible that if he, for real, never played Centipede, that’s pretty incredible – 40,000 points on the second try.
Brandon: So some of them we’d give a demo, we’d start to chat with, we’d say, Hey, here’s a minute about what we do. You want to see what we do in action? And we learned pretty quickly that no, it is not exactly a wretched hive of scum and villainy where you’re the target. Or at least, I haven’t seen any unexpected charges on my credit card – just honestly it reminded me of Cisco Live, just a bit less densely packed and in a different place.
So it’s, it’s all kinds of people who have jobs in this space, want to know what can help them, want to know what’s possible, want to stay on the cutting edge, and they were open to talking. Even if I did not believe some of their affiliations on the name tags, based on the questions they asked, if you know what I mean, there, there is no way they were working at those companies.
So at the end of those two days, when the flood of demos was starting to slow down, when we barely had our voices left from speaking for 12 hours at that point, we wanted to take a break, walk the floor, feel the pulse, especially around one topic that we wanted to learn more about. So we grabbed a mic, we asked two questions and here’s a sample.
Hi there. Uh, I’m Brandon I’m with Seeking Truth in Networking, Derick and I host a podcast. Today, we’re at Black Hat. We’re interviewing people to learn what they think about zero trust, what it means to them. Do you have a moment to answer our question? Great.
What does zero trust mean to you?
Derick: So without naming names, do you think the industry is delivering products that can actually help with this?
Brandon: All right. The first thing we found out, nobody wanted to be on the record. Nobody.
Derick: Yeah. Yeah, that’s true. And you know what, um, a few people were like, yeah, they were kind of iffy about it and then they agreed to it, but as soon as we held the microphone up – as they had like this weird black box adapter to go from the microphone to the phone that we’re using to, to actually store what we were recording.
As soon as they saw that they were like, no, I changed my mind, I’m out, I’m out. And they, and they bail, they’d walk away. That happened a few times.
Brandon: Yeah. nobody trusted anyone there. I think a random person on the street would have been more likely to stay and chat, even though, you know, you’d think we all have similar background and interest, we should be open to talking and being social. But no, that was, that was not the case. So we kept asking and eventually we started, our luck started to turn around and there was, there was this one guy and Derek said, we have to get him.. I’m sure he has a lot to say.
Derick: As soon as I saw this guy, just something about his vibe, right? He had a ponytail, he had this black t-shirt on. I couldn’t tell if if, what was written on it, was about InfoSec or maybe some kind of like industrial metal band, everything about him said, this is the guy. And I was right. I totally nailed it. And he gave one of the best responses, actually.
Brandon: So credit where it’s due to Derick. Uh, once, once the ball was rolling, we loosened up, they loosened up, people would see someone else speaking, and that little bit of credibility took us to some good content over the next hour.
And instead of just playing the raw tape, we wanted to package it up into a more entertaining form and share what they said and what themes emerged.
And of course, we’re going to connect it back to networking too. So definitely hang around until the end for the host thoughts. There’s definitely something deeper here than just a marketing phrase that you might react strongly to. And I want you to listen to the end for that.
[00:11:47] Attendees Share Their Thoughts
Brandon: All right. So back to the characters, here’s a quick sampler of 10 of the responses.
Stranger 2: Zero trust is, kind of a weird term that keeps getting bandied about.
Stranger 6: It means to trust absolutely no one and nothing. And you’re going to get hacked. It’s gonna happen. And you just got to figure it out after the fact.
Stranger 8: Uh, zero trust means to me that at any point within a given network, When you’re accessing a resource, performing a task, right, uh, privilege is checked.
Stranger 3: I think it is a branding, marketing hype.
Stranger 7: Well, what that would mean to me is just, uh, trusting absolutely nobody, especially on your own network, but of course you wouldn’t trust people off your network.
Stranger 9: Talking about that at the keynote. Right. And, uh, they asked the question, everybody, what does it mean? Is it a rehash of, uh, the same old principles that we’ve always had? You know, don’t trust third party applications, what, you know, you don’t trust anything. So I don’t think it’s a new concept.
Stranger 10: If I could give a, uh, a rough estimate it’s, um, re verifying every single time, anytime something wants to access any level of your security stack.
Stranger 13: Trust no-one and verify everything. From data ingress, you know, user input, anything just don’t trust that you’re getting anything legitimate at any given time. Because most of the time, it’s probably not.
Stranger 5: I don’t know.
Derick: All right. Well thank you for your time. All right.
Brandon: Some knew they had no idea, but there are definitely some themes that emerged.
The first one is there were some really strong negative emotions related to a buzzword and any marketing connotations. In fact, our first recording captured this one really nicely. Let me play that one for ya.
What does zero trust mean to you?
Stranger 1: Um, zero trust, I mean, applied should mean that we, uh, no, you know what, I, I can’t even take it seriously at this conference. Everybody says zero trust. I can make fun of it, or I can give you an actual definition. But I have no idea. I have no idea what any vendor in this room actually means by zero trust when they say it, because it has become a buzzword.
Brandon: All right. I gotta pause here. I love this guy. I love his unfiltered honesty.
Derick2: Yeah, this is the guy with the ponytail. I loved, he was, he was so ready to say this, right? He just wanted someone to ask those questions.
Stranger 1: I mean, in effect, I’m a director of threat intelligence. Zero trust means force authentication. We force checks. We make sure we’re covering all our bases, uh, applied security, right? Really, I don’t know what these guys mean though. So I am unfortunately unable to give you a straight answer in the current, uh, venue. How’s that for an answer?
Brandon: That is a good answer.
Here’s another answer.
Stranger 5: It means we’ve run out of exciting marketing material and it’s time to spin it back up. I say that, but ultimately I liked some of the concepts there, but really it’s been spun out of control and into a marketing thing. When is this going to stop being like a huge buzzword? Whenever we find our next one and move on to, you know, three to five years from now.
Derick: Yeah, that was a consistent theme, right? There was, um, a lot of, a lot of leaning in on it’s just marketing, hype, right and people, I think some people are just don’t want to hear about it anymore.
Brandon: And if you didn’t get that message…
What does zero trust mean to you?
Stranger 3: I think it is a branding, marketing hype. And that C-suite type people want to throw around because that is their desire in life.
Brandon: That’s pretty jaded. Isn’t it? This guy had some, nice quotables too.
Derick: Yeah.
Stranger 3: As a former sys admin, it means that we’re going to be putting in policies and procedures that will preclude me from actually doing my work because I won’t be given the level of authority I need to do it. And anywhere in between those two, if you do something smart that makes your system a little bit more secure and actually trusts internal east-west movement, make sure that people have an identity before they open the database, that all sounds great. If that falls under a zero trust, umbrella, even better, I’m just not buying the t-shirt.
Brandon: That’s nice. I got to use that phrase,
“Derek. I love what you’ve done there, But I’m just not buying the t-shirt.”
Derick: But the t-shirt is how I monetize my opinions.
Brandon: Nice. Uh, and so for some, their issues were more about whether it even made sense at all, whether it was achievable…
Stranger 2: I mean, is anything ever truly going to be sufficient that there someone’s going to come up with a way around it at some point. So it’s, it’s one of the technologies that we have at the moment. Is at the end-all be-all? No.
Stranger 5: I mean, what’s the threshold really? Uh, cause obviously it’s a unattainable ultimate goal.
Stranger 3: We’re running, we’re running faster, but we’re running against a car.
Stranger 7: I don’t think that anyone’s all the way. I don’t know if you ever could truly be all the way without being so absolutely authoritarian with everyone’s, you know, usage of the internet. And I don’t think that it’d be a very good working environment if you tried that.
Stranger 12: So we spent a lot of time on doing things, but I always think of zero as you can get halfway there and you keep getting halfway, halfway, halfway, halfway, but you’re never at zero. So I don’t believe that zero will ever come because things need to flow.
Derick: Yeah. what is the actual word for that? It’s like asymptotic to zero right? Or something like that.
Brandon: Yeah. You never get there. In the limit, you get there, but you never really get to the limit. I think there’s a paradox about getting halfway there. Yeah. Zeno’s Paradox that says motion is nothing but an illusion. Um,
But there were some more positive reactions too, among those who thought they knew and could see some value and they tended to follow themes of minimizing default trust and continuously re-verifying identity and access.
You can start to see the theme emerging.
What does zero trust mean to you?
Stranger 8: Uh zero trust means to me that at any point within a given network, when you’re accessing a resource, performing a task, privilege is checked, you know, it’s not assumed that the previous place where you were that may have validated identity, that that’s passed along. It’s basically not implicit trust. So
Brandon: So I thought this one was a pretty nice analogy .
Stranger 8: Effectively, once you go inside, it’s no different to me than scanning your badge right at Black Hat. Right? You go up to a, vendor station. They’re going to want to scan your badge to know who you are. Right. When you walk up, you can tell them who you are, but they’re going to want to scan your badge.
Brandon: So kind of a deep, persistent authentication / authorization everywhere?
Stranger 8: I think that’s probably a good general high level.
Brandon: In fact, there’s, there’s a good external definition, zero trust is a security concept centered on the belief that organizations should not automatically trust anything inside or outside of its perimeters, and instead must verify anything and everything trying to connect to its systems before granting access.
That’s pretty crisp, but not everyone has really internalized that or, or believes in it.
All Right.
So now we’re going to move on to our second question, which was all about whether industry is really in a position to deliver. We’re going to play a few clips to give you a sense of, of the responses.
Derick: Do you think the industry is actually introducing products and services that are helping people in a significant way? Or does it have a long way to go?
Stranger 2: So yeah, no, the industry always has a ways further to go.
Stranger 5: Possibly. Yeah. I think so.
Stranger 8: Absolutely not. It is extremely difficult to implement and I think that the industry has a long way to go to, make that move.
Stranger 1: I think there’s a few that do a decent job at it and certainly speak to it. There’s not many, but there are a few.
Stranger 7: Uh, from what I’ve seen? I think that they’re, they’re pretty close.
Stranger 6: So I think we’re getting there, but we’re not there yet.
Brandon: So that’s, that’s a range of answers. Pretty clearly, we’re not there yet to, we’re actually closer than you think.
Another theme that emerged is that zero trust is hard because it’s not some single product you can buy. Is it even about a product or is it more about a larger change at an org or is it even a cultural change?
Stranger 8: I think there’s a significant, bridge or gap out there between what needs to be in place and what should be in place. And most of the time it’s, technology that is difficult to implement in, in reality. In a whiteboard sense it’s really easy and it’s like, oh yeah, just go do the things. Right. Um, but when you actually get into a working environment and you have legacy applications, right? You have people that have not changed processes for long periods of time. It is extremely difficult to implement.
Stranger 1: Products don’t fix this. Culture fixes this. I mean, you don’t fix this by buying a product. I mean, you can certainly make it a little easier to live while trusting sources. Right? But you’re not going to fix it that way. Not permanently. Someone’s always going to find a work around. There’s always going to be a way to evade. Uh, and, and so that doesn’t fix it. That, band-aids it pretty good sometimes. Yeah. Uh, and it might even be worth the money you’re paying for it. I’m not saying it’s a bad play necessarily, but, uh, you know, if you can’t change the culture of how you’re developing or how you’re building a network or whatever you were trying to secure, there’s no advancement there.
Were there any kind of thoughts you wanted to give about mindset or product?
Derick2: Yeah. You need tools and platforms, right? to help you achieve really any goal, right. Whether it’s zero trust or not. But mindset is important. You have to recognize the importance of the tool and you have to get everyone to, accept it and to work it into your, your workflows.
Otherwise, spending money doesn’t achieve anything. Right.
Brandon: And as luck would have it, we came across a software engineer who was very different than the tools operators and the managers and the InfoSec directors and incident response leads and the more security-first people that we had seen. So let me play a slightly different perspective here.
Stranger 4: Um, so I’m probably the least like security-based person. I’m a software engineer.
So
Brandon: Do you have any kind of recommendations as a software developer to those who are more operational with the tools? ?
Stranger 4: Um, I would say like those who are more operational be in contact with your software developers Because if you know what they’re building and what they’re building into your software, you can be more secure. Nobody’s perfect. There’s always mistakes to be made, hence the zero trust. Be a part of that design with those developers, ask the questions. I love questions. I love knowing what the user actually wants to see versus what’s coming down the pike from some agile scrum master who’s never developed before.
Brandon: And without naming any names, do you feel like industry is delivering on the solutions or products you need to be successful with security, with the software that you develop?
Stranger 4: I think, I think we’re keeping up or at least trying to keep up. There’s so much growth in technology right now. And so many good guys trying to do good things and bad guys trying to do bad things that it’s always going to be a chase, a give, and a go. We’ll catch up and they’ll come out with something even better. But I mean, that’s the game we play, right. That’s the game we like to play here.
Derick: Yeah, that’s a good one. The part where she says, be a part of that design process with the developers – that’s another industry buzz word called shift-left. In the, uh, timeline of a project, start thinking about the security, in the very beginning.
Brandon: Yeah, I thought it was interesting to hear a software developer who wants more feedback from the security types that are going to have to operate the products she’s building.
One of the challenges seems to be the number of tools.
There’s not a booth in the sight of my voice that doesn’t have a tool. The problem is, is that you’ve got a tool box with hundreds of different tools that are all thrown in there. And very, very few people are providing a comprehensive solution
Stranger 3: We’ve decided to put grandma’s email and our private banking all on the same network, plug it into really bad countries and hope for the best. That is never going to be solvable. I think it is probably mathematically provable it is unsolvable.
So this tool here, or that tool there… are band-aids on a cracking dam. So what we need is someone who is looking forward 10 years and saying, there’s going to be something different.
Brandon: So I got to jump in with this one. I hear this all the time. Tools overload. So many tools out there. Everyone at this conference was seeing all kinds of tools thinking, how do these play together? How do these help me ultimately deliver a secure network? And I don’t think it’s clear to a lot of people, how they fit together, which ones they really need, or whether what they really need is one tool that does the work for them to pull these together and give them that central view.
Derick: Yeah, it’s an operational issue too, right? Like a lot of times people in the field are busy with what their main job is and trying to, to grok all of that it takes time and it takes brain power. And you know, they’re tired, right? They’re done thinking sometimes at the end of the day, they’re not going to be trying to understand the, the nuances of, of all the various options they have in front of them.
Brandon: Yeah. I felt that too.
[00:23:59] The Networking Perspective
Brandon: All right. So let’s, let’s switch gears a little bit. Derek, what do you think people are really trying to get at? What’s your truth from a networking lens and how does this connect to you?
What I see over and over it’s about users and access to resources. There’s like a mindset or there’s a philosophy behind zero trust, which is don’t trust, anything, right?
Derick: You know, when you have changes in your network, like during a change window or, emergency changes, whatever it is, when you have changes in your network, how do you know that the changes you’re making, aren’t expanding your attack surface, right? How do you know you’re not introducing vulnerabilities?
And that’s, that’s a really hard thing to answer, and we sort of trust that we don’t – like we make the changes we make and we run a bunch of ping tests and we go home, right? Hopefully we haven’t introduced something that, you know, that’s going to be trouble for us later, and I, I have actually a great story about this.
You know, I, I worked at a company who, uh, they, acquire companies all the time. They buy companies that build software products, and then they try to integrate that technology into their existing suite of technologies and the way they do network integrations was, you know, Slap a firewall in between the company you’re buying and yourself, put in some NAT rules and start opening up ports, so that application components can talk to each other. Well, somewhere along the line, um, this company that had got purchased, they did a redesign, and subnets were removed from their environment, but there were NAT rules on the firewalls that were not removed. There was an incident, like eight months, this hacker had been inside of this, our environment… they discovered that they could go after one of the old NAT rules on this firewall. The destination address would get translated, to the address of a resource that they were trying to hit someplace completely else in the network.
But when it went behind that firewall, it would hit a default route because that subnet had been removed and would go back through the firewall, get its source address translated, and then routed towards the resource elsewhere that they were trying to reach.
Brandon: Okay. So this is kind of like a hidden worm hole.
Derick: Yeah. Yeah. It was like, I, what do you call that? A firewall hairpin NAT attack. Like, I, I, I, it was crazy to me that this, they had found this. Right. And the thing is that, altogether, I think it was something like 18 months between the time that, that change had been made, they removed those subnets, but not the NAT rules, and when the incident occurred. Right.
Who would have imagined that… that scenario, right? I think connecting back to networking, you know, we’ve done a pretty good job on the podcast of not breaking the fourth wall. Right. And talking about Forward, our sponsor, right? The company we work for, they make this podcast possible, but, a platform that can detect when that kind of thing occurs , and notify you about it, you can actually say, you know, these, subnets over here, these LAN subnets or whatever it may be, should never have access directly to this production server.
And if that ever happens, tell me, , and to have a platform that understands all of the routing in the whole network, all of the NAT translations in the whole network, all of the firewall rules and the whole network, and be able to detect when that path occurs and notify… that that would have saved that network from that incident. That would have been a lifesaver, right? I mean, like I said, how can you imagine that? That’s important, right? So that’s the connection I make from that, and I think part of that is, you know, what our definition of zero trust is as network engineers.
Brandon: Interesting. Yeah. Well, let me connect it back to our yellow booth. There is a phrase printed there, zero trust but verify. And the way I’m thinking about it is especially after what Derick said, connectivity is a form of trust. It’s also a form of access control, It’s something that we can control in the network.
We’re not dependent on application owners and engineers and their configuration and others’ timescales to change software. It’s, it’s something we can change today. We still need to have every service implement their authentication and authorization, but this ability to discover the paths that are there, that’s, that’s a meaningful step.
Every time you find one of these attack paths and you shut it off and you had no idea about it, you are now more secure … you’ve taken a concrete step to prevent a future breach. And the more of these you can find, the more secure that network really is, especially in the ways that you would otherwise have no idea about.
So anyway, we were, we were there to show some new features, uh, that our engineers have built in the last few weeks on top of the same data that our product has access to – all the network config and state and analyzed paths. And so one of those features we demo’ed was policy matrix. So Derick, how would you describe that one?
Derick: Yeah. So this is, um, this is kind of awesome, you can select a set of security zones in your network, um, whatever 7, 8, 9, 10, 20 of them. And we list them both vertically and horizontally, , in order to make a zone-to-zone matrix, and you can click on the intersection of two zones and you can see what can pass between those two zones.
And what’s really cool about it again, is that, it’s not just comparing two different firewalls’ rule sets to see what, combined, they would permit, between those two zones. We actually take into account everything in the path, all the forwarding rules, all of the NAT rules, all the translation rules, anywhere, a source, destination, port, or address, um, can be translated. We take all that into account, to, show what actually can today pass through the network between those two zones. That’s super important because there’s, really three reasons why.
First, you know, we live in an SDN world, people are doing VRFs now, commonly in the enterprise is very easy to set up virtual routing that bypasses enforcement points, right. This is by the way, a common issue in the cloud. So that’s reason number one, right? it’s easy to do that.
Two, , there could be translation points in the middle that actually, change the packet, so that between Zone A and Zone B, payloads can be delivered.
And third it’s also very possible that the network is just not set up to pass that traffic anyway, even if the firewalls would permit it, there is no path in the network for that traffic. That’s what makes that feature very cool. That, that policy matrix feature.
Brandon: You’re kind of saying the firewall-only security-oriented view is incomplete. It just doesn’t, it doesn’t give you the information to know what connectivity is really there? What reachability really exists?
Derick: Exactly. It’s going to give you false positives or false negatives because it has no idea what the rest of the network is doing.
Brandon: All right. And there was, there’s a second feature that you were demoing, was this called blast
Derick: radius?
Yeah, and it’s again, predicated on this same, same idea, of you’ve got to include the network. If you think a host or an end point, is potentially infected or compromised in some way, you can put in their IP address, you can specify the entire RFC 1918 range and other subnets, whatever subnets you want to put into the list of destinations and taking again, everything into account, we can tell you what are all the places that this end point can deliver a payload to and on what ports.
And it’s accurate. You know, we, we like to say it’s. It’s mathematically accurate and that’s, you know, that’s true. It is. That’s, it’s literally, it’s based on Header Space Analysis, which is a mathematical way of looking at how the network treats packets, as they traverse.
Brandon: And then that third feature you were, you were sharing – once you could look at the destinations from a blast radius analysis, how’s traffic actually able to get there. I didn’t expect this. What’s going on? Why, why?
Derick: You can see why, you can get a, very detailed path layout that tells you every device that the traffic is going to pass through, every network function being applied to the packet, right? Policy-based routing, filtering, NAT rules, forwarding rules, et cetera. And you can see exactly, how this is possible. And, you know, if you think about that, that’s, that’s great. You can get that back from that query, but that query is also kind of a true-false question and, uh, at a high level, are there any paths available?
And so you can sort of say that as an expectation, you can say: there should not be any paths available between these two zones and if one becomes available, for instance, you delete a subnet behind a firewall, but not the NAT rules. You can be notified about it, instead of being notified that there was an incident that exploited that hole you created 8, you know, 18 months later…
Brandon: so we’ve gone from, you had no idea, to, you have a way of detecting it, to, it’s pretty much automatically detectable and it comes to you and you can detect every such instance of these examples of unexpected connectivity that no single person, no security engineer, no network engineer would have, with their partial view of the world, right? You’ve got to bring those views together. I would put it as we’re taking useful info and we’re making it available, useful network info, and we’re making it available to security operators.
Derick: Yeah, that, so this is a good point. Like in the booth, we heard several times that the Forward platform, it can be a bridge between the network and security team, because it has to be , when we were talking earlier at the intro, you can’t have one without the other, the network and security team relying on each other and having, having a platform that can sort of, Allow them to visualize in a common way, maybe speak a common language, is extremely helpful, right?
Brandon: You can’t solve security alone and you should have some help. Who would argue with that?
[00:33:15] The Software Perspective
Derick: So Brandon, uh, you’re on the software side, what were your takeaways, right? What do you think? the people we talked to were getting at here through the lens of software, if you will.
Brandon: Yeah. this is, this is a tough question, but I’m going to, I’m going to try to give a good answer here and I’m gonna apply a software developer mindset.
So something you said really resonated, which is we need to verify that the work that network engineers do is not increasing the footprint of attack. This is a hard to discover, easy to get wrong, big deal kind of event that becomes possible if you get it wrong and no engineer wants to be responsible for the thing that brings down their company or puts them in the news. Like these are the worst kind.
And I like thinking more in terms of invariants and guarantees. So instead of individual problems, think about classes of problems that you can solve broadly . And when I was running engineering at Forward a few years ago, whenever I would hear a bug on a post-mortem, the question would always be: how do I prevent the category of that bug? Cause it’s always better to prevent an issue, whether it’s some software bug or it’s some unwanted access, security issue. It’s just always better to get ahead than try to catch up and react to problems. It’s too late at that point.
And so one question on my mind, it was always, how do we get ahead of the bugs our customers might see. And I, I personally put a lot of time into infrastructure so that we could automatically find bugs that might occur in customer environments, to where we could say – there is no possibility for that kind of bug. Those aren’t a few lines of Java code. They’re complete systems you have to build, or they’re language frameworks that you have to custom-build. And I would, I would love to talk about this on another episode some time, but I won’t go too, too far into it. Anytime there’s a complex system, software comes into place to make sure you’re doing the right things all the time. And in fact, there’s a blog post… if you type seeking truth in networking blog. At least at this time, the first hit on Google is a blog post I wrote – it’s almost five years old now. And it talks about two other examples wherein complex systems, there was an enormous mistake. We’re talking billion dollar mistake, and it stuck out in everyone’s mind.
The Ariane 5 rocket didn’t even have any software changes relative to the 4, but it was much more powerful, went higher and it entered a software regime where all the computers failed the same way. It turned 90 degrees. It blew up 37 second in. It’s a fascinating video and story. The other example is if anyone remembers- this was the mid-nineties, uh, Intel almost went under.
Definitely go to that webpage. We’ll add a link in the show notes.
Y’know, we don’t have these kinds of bugs today because of verification software. And, you know, we’re eight years into this journey with Forward, and that was always one of those things that motivated us. The smartest, most experienced people are always going to make mistakes and the most prudent thing you can do is give them support to catch those mistakes that they wouldn’t otherwise ever have the ability to see, especially when a system is so complex that no one human can reason about it, which I would say is the network. And this is the difference between, you know, you’ve got the go-ahead, you’re confident – versus hope and pray, cross your fingers, depend on the right people, having the right focus every day.
And the side benefit of verification of, you know, complete, complete understanding of everything you’ve got is the visibility you get, even when things are mostly working. So you’re trying to trace a path for an application. Where is the traffic going? Why is it taking that path? What are the lines of config and state that someone else changed or added years ago that explain why this is here? You know, connecting it back to Derick’s story… It was an 18- months old- change. It probably wasn’t the thing that was on people’s original mind until they saw that path and they saw what it would have changed.
So you do the best with what you have and if what you’ve got gives you complete, god’s-eye visibility into the path of every possible packet, that’s really helpful.
[00:36:48] Outro
Brandon: So I don’t know about you Derick, but I found that trip pretty motivating. Cause you want to know that what you’re doing has value and we got some great feedback. We also got some great ideas from the attendees on where to go next. So thanks to everyone who gave us their time there.
Derick: Yeah. I loved this trip. The response was really positive from the security community. People really love the demo. In fact, if anyone is interested in seeing that demo, you can just reach out to me, at cloudtoad C L O U D T O A D , on Twitter. You know, I’d be more than happy to give the demos to you, one-on-one, I’d love to do it.
Brandon: You’re a pro now. You’re a world expert.
Derick: Um, or even Brandon, uh, at
Brandon: Every why guy.
Derick: every why guy.
I gave that demo about 50 times in a row. It’s burned on my brain. Like I, my voice was blown by the end of the show.
Brandon: Oh man, the next day, people used the phrase, Barry White. That was the only thing they mentioned the next week after my voice recovered. Wow, you sounded so different. I couldn’t believe that was you. Yes, that’s what it’s like to be in the real world, going to conferences. All right.
So on that note, I want to thank everyone for taking this break from the usual with us. We’re going to be back next week with more content. See ya then.
Brandon: This episode was brought to you by a passionate and creative team. Blame me for much of the editing and audio work, but thank Derick Winkworth for the stories, thank Charlie Elliott for production support and thank Aditya Chakraborty for additional audio editing.
