Turning CMDB horror stories into IT success strategies

Sean Dawson: Welcome, everybody, to the Cask Distillery Podcast, where we unlock the full potential of ServiceNow with expert insights and practical strategies, only here on the Cask Distillery Podcast. And today, we have a spooktacular episode, because Halloween's coming up!

And so we wanted to take a moment and meet with Chris Padmore, who's my special guest today-who is our ITOM practice principal here at Cask-and talk about some of the scary stories that can happen within CMDB. So, welcome, Chris. Thanks for taking time out of your busy schedule, even though this actually turned out to be kind of last minute. So thank you. 

Chris Padmore: Thanks for having me, Sean. 

Sean Dawson: Oh, no worries. So, let's get right into it, because I want to get you to talk a little bit about some stories and some things that can go scarily bad with CMDB, and the first one that we had thought of was the "Phantom Update" incident story, so talk a little bit about that for us.

Chris Padmore: Oh yeah, this is a good one, and a lot of people probably have come across this at one point in time, but I want you to imagine this: You have a large enterprise IT team, right? And they're doing their typical updates to the CMDB. And then they have a new person, and they've done their changes. They're making their updates. But they make a mistake-a very small mistake. You don't think it would be a big deal. But then, tragedy starts to happen. One application goes down. No one knows why. Another application goes down. No one knows that. And they're trying to figure it out. This never happened before. And people are looking, and they're looking. They're checking the CMDB, and they're checking tickets, and they do not know why all these things are going down. 

And it turns out the culprit was an error in the update to the CMDB. So when they made that last change to the CMDB, they accidentally removed one of the core dependencies of the server they were running on. So all of these things were happening on the server side, and it wasn't related back to all of the upstream services. So they could clearly see that that server was having a problem; that database wasn't working. All of those applications didn't realize that they were dependent on that database. And all because someone unwillingly changed a dependency without knowing.

Sean Dawson: Wow. So how, in your eyes, how do you ensure accurate dependency mapping in CMDB to kind of prevent that from happening? 

Chris Padmore: So, your friend there is going to be the Discovery tool. It's going to make sure that as it's scanning and it sees dependencies between IT components, that those dependencies are properly associated in the CMDB. So even if I were to go in and remove that dependency manually, when Discovery runs next time, it'll just put it right back, and you won't have to worry about missing configurations and not knowing what the full upstream and downstream impacts to something are. 

Sean Dawson: Awesome. Thanks for that. So, the next scary situation we can run into is the ghost server. Talk a little bit about what that is all about and what can happen. 

Chris Padmore: Oh, yeah, but lots of people have seen this too. You're sitting at your help desk, and you get a call in saying, "Hey, it looks like server 123 is down." And you go and you look up server 123, and it's in the CMDB, but it's not pinging. You don't see any tickets associated to it. You don't see any applications running on it. You don't see anybody requesting it except for this one phantom caller. And you're like, "Well, where did this server come from? Why is it in here?" And you try scanning it with Discovery and nothing's happening. You ask the server team to log into it, and they say, "Well, that IP address doesn't exist. It's saying, 'Nothing found.'" Turns out that server used to be a server, used to exist in the CMDB. But it was decommissioned. And somewhere along the lines, when someone was doing their job of decommissioning it, breaking its dependencies, they left something intact.

So even though the server is there, it's not talking to anything, nobody can access it, and someone has to physically go down into the basement and disconnect that server from the premises. But it's scary because you have your risk people saying, "Well, who put the server here? Who has access to it? Did somebody walk in and plug something in when nobody was looking?"-kind of thing. Turns out it's just a leftover server that's supposed to be there. And now somebody can go down and actually turn it off. 

Sean Dawson: Nice. So, what's a process that someone can ensure that decommissioned assets are really fully, truly removed?

Chris Padmore: So, there's a couple of things that we try and do. We have something called a "closed-loop change process." So, that's where we are doing something like updating or removing a component. The last thing that you want to do is scan that component. And then make sure the CMDB accurately reflects its updates.

In this case, with the decommission, what should have happened is Discovery should have seen that that server was completely gone. Someone should have marked it "decommissioned" in the CMDB. And then someone from the server team should have actually gone down and had a task saying "unplug from the wall" kind of thing. And then you would have had that complete. Not only fully executed decommission, but then documentation and tickets saying that, "Yep, this person went down and unplugged it during this date. And we confirmed that Discovery is no longer seeing it."

Sean Dawson: Great. So let's talk about the next one. The "Unseen Configuration Drift Catastrophe."

Chris Padmore: Oh yeah, this is, again, these are things that people probably have nightmares about if they've been involved with them. But picture this: We have our core systems, and we're scanning them. The team knows how they're set up, and we don't usually have any issues with them. They are humming. They are running along just fine. And then, you start to see changes, start to see them behaving differently. And then, just to make sure it's not you, you go and look and you see it, like, "Hey, this says that we're using two Linux servers, but I see two Windows servers running these applications." "Hey, this says that we're on patch 14, but I see we're on patch 8. What's going on?" And so, what can happen is, for whatever reason, the core configurations of that system are starting to change. Things are getting out of date. Things are mismatching with what people have had planned. And someone's concerned that somebody's in the system doing something they're not supposed to do. 

But it's not the case. This could usually happen when the wrong code gets pushed out during a routine update. And so those are things that we can catch easily with the Discovery tool. We can track those core config files. We can track when they're being updated, what's updating them, and what the differences are between the last time Discovery ran. And then we can have automated alerts generated, saying, "Hey, this looks like this config file changed. It didn't change during one of its core change windows. Let's have someone take a look at it and see what's really going on."

Sean Dawson: Oh, great. And you actually answered my question there, was "How can we prevent it?" All right. So what about the Dependency Domino-Effect Disaster? 

Chris Padmore: Oh yeah. This one is one that I personally experienced many, many moons ago, but imagine that you and your team are just doing your normal work, and you're just doing minor updates. You have ten servers that you patch on a regular basis, and it's that time again. So, it's a Friday afternoon, you're doing your patches, you finish for the day, you close up, and you go home. You go to bed thinking you're going to enjoy the weekend. And then you wake up and you have 50 unread phone calls, and you have hundreds and hundreds of Slacks and an email saying, "Hey, this system is down. This system is crashing. What's going on?" And you're panicking because you remembered that you just pushed an update in before you went home. And so you start thinking, "Well, it can't be my update. Mine's an isolated system. It's just a small update." And you log in, and this application's, down this application's down, and you're freaking out. You're like, "There's no way it could be me. There's no way it could be me." And then you happen to go and look and see that of the 10 servers that you updated, the last server was supposed to be CMDB server 22, and you actually updated CMDB server 2022, which is not your server. It's somebody else's server. 

So you've unwittingly released a monster in your organization, and you don't have the know-how and knowledge to go in and fix it. So you're getting on the calls with people. You're in those war room calls trying to figure out the problem. And you have to stand up in front of everybody and say, "I think it might be this server, and I think this might be the change that caused it. Let's have somebody go and take a look at it."

Sean Dawson: So what are the best practices for managing dependencies in a CMDB? 

Chris Padmore: Again, you want to follow your change management process. You want to make sure that you are reviewing and validating all the work that you're doing.

When you're doing any changes, you're going to have a list of CIs that you're going to be working on. You want to ensure that you're actually logging into those CIs to push your changes to-that the tasks that you're committing during those tasks, you want to validate, like, "Hey, is this a-can you screenshot the server number so that we can confirm that you're actually logged into the right server?" Again, you want to have those closed-loop changes using Discovery. So after you're done making your updates, you want to have Discovery re-update everything so that it can make sure that the configs that you were doing actually apply to the CIs listed on the change. So, if you were supposed to do server 20 and you rediscover server 20, and it says, "No changes detected," then maybe you were in the wrong server, and you're not gonna be able to enjoy that weekend like you were planning to. 

Sean Dawson: Yeah. What about the AI Gone Wrong CMDB update? Now, this is a newer one, but it could be very scary.

Chris Padmore: Yeah, so AI itself is pretty scary if you're not used to it. But let's imagine that you've initiated one of ServiceNow's machine learning modules, and you're using it to flag inconsistencies in your CMDB. And so you've just released the next version of it. It's doing its updates based upon some recent changes that went in. But the AI has become self-aware and thinks it knows better than you. And so it starts going in and marking IT assets as invalid, incorrect, nondiscoverable, nonoperational, based upon those changes that were made earlier. And by the time you have a chance to catch on, you have hundreds and hundreds of servers and databases in the wrong operational statuses. You have lots and lots of relationships removed that shouldn't have been, and then you have lots and lots of resources tied to services that they have no business being tied to. 

Sean Dawson: Yeah. So a good question would be, well, how can AI be safely integrated into CMDB management then? 

Chris Padmore: So you want to make sure that you are not going full unsupervised learning to start out with what your CMDB AI updates. You want some validation. It learns by observing what you're doing and seeing what's in the system. But you do have to coach it, especially if you're just starting out. You want to go and make sure that the updates it thinks it's making are valid. You want to provide that feedback saying, "No, under these circumstances, this is not the right type of update that we want to do." And you want to test these things out in the lower environment as well. You want to make sure that it's behaving as expected in the dev environment and that you can use those models up in production without having to, you know, roll back a bunch of changes that it thought it should be making. 

Sean Dawson: Great. Well, thanks, Chris, again, for the time. I know this was last minute, but we wanted to provide value to our watchers and listeners, however they're combining or consuming the podcast here. So thank you so much, Chris.

And I want to point out also that Chris has been involved with a series of CMDB webinars that we'll put the link to in the bottom or above or however, whatever you're watching this on, that we have a series on CMDB, which there's been tons of great information. So we'll send you to the registration page so you can get on those and view more.

But that's it for now. Have a safe and happy Halloween and take care. Bye bye. 

Chris Padmore: Thanks, Sean. See you soon.