Cybersecurity in Manufacturing - Ep. 4

Cybersecurity on the Frontlines

‍

‍

How do you build out a sophisticated, enterprise-level cybersecurity posture when your mid-sized organization doesn’t even have a dedicated cybersecurity team, and even your vendors treat you like a small business? Turns out, the answer is pretty old school. It’s all about building the right relationships and setting the right priorities.

Today we’re talking with Tom Knutson, the self-described jack-of-all-trades for a Wisconsin-based manufacturing firm. He’s going to walk us through how he leverages his team and his own diverse skill set to keep his company safe.

Episode Table of Contents

• [01:21] IT and cybersecurity in Manufacturing
• [04:49] Selling cybersecurity to the C-suite
• [6:30] Developing a cybersecurity roadmap
• [17:02] COVID’s impact on the security stack
• [19:54] The threat landscape is real

IT and Cybersecurity in Manufacturing

Tom Knutson: My name is Tom Knutson. I am a jack of all trades, master of none, for lack of better words. Responsible for engineering and design of our security stack, our applications, and how we are going to run them or the platforms that we run them on.

I’m the central escalation point for most of my team’s technical problems as well as our desktop teams, technical problems. Traditionally, if a problem comes up or somebody doesn’t know how to do it, I wind up getting the escalation path, and then we figure it out together.

Ted Gruenloh: Can you talk about maybe the size of your organization a little bit and some of the challenges there?

Tom Knutson: We are a manufacturing organization in the Upper Midwest, in Wisconsin. We are technically a medium sized or an SMB, but we have enterprise expectations and ambitions. We’ve got about 300 to 400 employees. We’re spread out primarily across the US and Canada, with some international employees.

Ted Gruenloh: When you say SMB, you’re on the larger side of the SMB, I would assume, there in terms of your technology. Let’s dive right into that a little bit. Before we dig into the stack itself and some of the technologies you use, I’m interested in the team as well. How many people do you have in IT specifically and then maybe security specific people?

Tom Knutson: Looking at IT, we’ve got 20 or 30 people in IT, from DevOps to business managers, to infrastructure and security and support. We run a bit of a smaller team and do a lot with less, so some challenges there.

In our infrastructure and security stack team, it’s a team of a few. All of us do a sizable amount of security work or escalation work. Then we’ve got a single person that’s responsible for the choices that are made or are advised by us.

Ted Gruenloh: When you say your security team is small, and you yourself call yourself a jack of all trades, does that mean that the security responsibility falls to you and a small number of people that also do other IT related things, or is the organization separated into a security team and an IT team?

Tom Knutson: It falls into we do security plus other things. We have a single team member that specializes in just security and is responsible for internal security messaging, reviews, and escalations of either incidents or incident response, or policy and procedure for how we’re going to move forward.

Some of that, though, is expanding or breaking out in the organization. Currently, it’s a single person responsible for those escalations, with me and my team handling a large amount of the day to day operations of review and reporting and escalating up to our security specialist.

Ted Gruenloh: I’m sorry, I’m harping on this, Tom. Does that single person…What level in the organization are they? Are they a CISO that reports directly to somebody high up, or are they part of the IT structure?

Tom Knutson: They’re part of the IT structure, or more specifically, part of our infrastructure team. We are still growing quite a bit in terms of our IT structure and whether it’s a CISO or other types of C level type employees. I expect that that’ll probably come about here in the coming years, but we’re just not there yet internally now.

Ted Gruenloh: Makes sense. That stuff always interests me in terms of the security details of where people report to and that sort of thing and maybe some of the headaches and pitfalls around those types of organizations. Do you feel like you have a pretty good support from your executive team in terms of providing security and the tools you need?

Selling Cybersecurity to the C-Suite

Tom Knutson: Oh, absolutely. The security and the spend and the choices are always a moderately difficult one, no matter where you’re at or you’re what you’re doing. Our leadership team, our executive team, owners, or senior leadership really appreciate what we do and how we do it in keeping the organization safe.

It comes down to we’ve learned a lot over the years in how we approach or present the problems, the plan to move forward, or the technology stack that we’re going to implement. If we use what we’ve learned to present it, and make sure that we present it correctly, and everyone understands it, it’s typically less of a problem to either get support, get buy off, or move forward on things.

Ted Gruenloh: That’s an interesting lesson. We hear that a lot, the ability to sell those things upstream. That’s an important skill set. You talked about being a jack of all trades. That’s an important skill set for you I imagine.

Tom Knutson: You can’t just be the salesman to be a salesman. You have to understand what it is that you’re looking for and why you need it. The technology team knows that you need it or that we should have it. The executive team might not be able to relate on how it supports, empowers, enables, or helps the business.

Being able to turn it around into a valid business case and how it will either mitigate, prevent, or enable, is key.

Ted Gruenloh: How do you make your decisions on what the next tool is you’re going to implement maybe, what the next hire you’re going to make, the next managed service you’re going to add on?

Those decisions, you’re leaning on something there, some sort of strategy, maybe a framework, something like that. Can you talk more about that a little bit?

Developing a Cybersecurity Roadmap

Tom Knutson: In our growth or where we’re at and how we do it, in the past or previously, it was less of a strategy or a plan, less of a firefighting thing, and more of we regularly have this type of problems, or we’re unable to do these things, or we don’t necessarily have this dataset, how do we get it?

Then, we develop a roadmap or a plan for that specific purpose and decide what technology, how and why it would get implemented, and then get an approval to move forward.

Over the past few years, with growth and change in the security landscape being as hostile as it’s been, has seen some of that develop into a true plan and a true roadmap of what can we do today in our timeline to accomplish this year? What can we accomplish next year? What do we need to get there?

The biggest thing for us is time. There’s a lot times that you think, “Man, I just wish that we had another four hours in a day. How do we get these hundred or a thousand dollar projects down into a manageable amount of time so that we can get the next project done?” That’s a recurring theme.

Ted Gruenloh: It makes sense. In your industry, is there any framework or compliance issues that drive some of these decisions?

Tom Knutson: There’s of course PCI compliance. In terms of safety, control, and quality assurance, whether it’s ISO or others, some of those drive what we do and why we do it, but they’re less of a direct contributor. It’s more of we’re doing these things in these areas, what do we need to do to support it? How do we enable and empower it?

Ted Gruenloh: Why don’t we dive into the weeds a little bit? Guide us through…Technically speaking, we know how your team works. We have a little bit of idea on the strategy behind how you make your decisions. Now, we’re in the weeds.

Why don’t you talk a little bit about your stack? You could start on one end or the other. You could start at the cloud and move all the way to the endpoint or vice versa, however way you want to do it. Then, maybe we’ll talk about those tools a little bit.

Tom Knutson: I feel that we’ve got a relatively good stack encompassing cloud, off prem/on prem infrastructure, workstations, and even our users to a certain extent. Of course, there’s always room for growth, for change, or improvements. With the security landscape changing practically daily, sometimes it feels difficult to keep up with.

We have worked from the outside, back. It’s been an evolution over the years. Starting, we realized we’ve got these public facing websites and resources. How do we protect and secure them and in a cost effective way? Whether it’s leveraging tools like Akamai, Cloudflare, or other public WAF type services, a big focus for us has been time.

What impacts some of those choices is how do we do more with less? What do these tools allow us to do, whether it’s a full managed service or a partial managed service? Going further back into IPS/IDS at the edge, we’ve covered some of our cloud or services at the edge edge.

Then IPS/IDS, on prem or before it gets to our gateways and our services, where we do it, how do we do it? We, of course, are very happy with the services that we have there from Sentinel.

Going behind that, we also have to ensure a multi vendor, multi strategy approach if maybe there’s a weakness or a discrepancy in that one platform, putting all the keys in one place.

Going into firewalls or IPS on firewalls is challenging and resource and cost intensive. We’ve made some choices there on how do we protect behind our edge IPS and have various manufacturer’s firewalls and IPS/IDS systems in content inspection.

Ted Gruenloh: That’s by design? That’s not just because you piecemealed it together over the years? That’s by design, to have a couple different vendors like at that firewall there?

Tom Knutson: Less by design and more by necessity, whether it’s virtual and physical combinations. Maybe we weren’t happy with vendor one’s physical hardware choices and we were happy with their virtual hardware choices or cloud firewalls and choices of that.

Also, part of that reasoning has been how do we protect these different types of resources? It’s been, one provider does these specific things. It’s been more of we need these particular feature sets and we need to be able to do this thing, has caused us to choose those vendors.

As a side thing, has been multi vendor in terms of inspection in IDS/IPS and content manipulation, needing to have multiple people or choices in that stack to ensure that we don’t let anything by.

Ted Gruenloh: That’s a common thing we see. It’s a classic defense in layers. A lot of times, people don’t consider the layers to be different vendors, which is a healthy interpretation.

Tom Knutson: Some of the vendors or platforms have multiple providers in them. Rather than saying we’re going to just use one provider in this inspection program, we’ll instead use two or three, whether it’s malware detection or proxy type security.

A good example would be, what am I doing for the virus inspection in our proxy? Could I just use one provider such as Klam, Sophos, McAfee, Cisco, or what have you, or do I use a few? There are choices to be made there and some form of trade offs.

Ted Gruenloh: You talked about money and time. Some of these things are relatively inexpensive to implement so you might have the luxury of providing two or three different vendors for some redundancy there.

Tom Knutson: Absolutely. Going so far as some recent problems with the onset of COVID and work from home, we realized, how do we protect all of our remote employees? All of these people that might be off net or not on VPN, how do we ensure that they comply with our policy or what we’re doing?

We had to branch out further from we talked about edge and the cloud security and then protecting on prem and hardware and firewalls IPS/IDS. Then it went to, what do we do for these people that are at their home or traveling?

It was less of a problem pre COVID when the entire workforce was working at the office and you just had the traditional traveling roles. Requirements were easier to make and establish. We had to make some choices on security to protect our remote stack as well and off VPN and on VPN.

We’ve been very happy with that. Some of them are DNS based security providers. Others are just threat reporting, data collection like network visibility, and how do we report on and collect the datasets. A key tool for us that we’ve really started to appreciate is Splunk.

A problem that we had in threat hunting or finding out what was going on was who did it and where was it at. With all of these platforms, you wound up having to look through logs, or a UI, or something from a whole bunch of different sources, or pulling all the logs and use some tool to grep through all the information. It was challenging.

Leveraging a SIEM, or data collector, or CIS log, we’ve in this case chosen Splunk. It’s been a night and day difference for how we are able to report on incidents or activities.

Ted Gruenloh: Is Splunk a relatively recent thing related to the COVID changes you made or you’ve been running a SIEM for a while?

Tom Knutson: We previously ran another SIEM, but we weren’t able to collect a lot of the data in how we wanted. With COVID and with some of the technology stack that we implemented to support and enable our security while remote, caused us to take a look at is on prem security enough? How do I know what these people are doing, if something is not going on and they’re not on VPN?

A good example is a user working from home. They don’t necessarily need VPN so they’re not connected to our corpnet. We didn’t want to expose our CIS log or our SIEM to the Internet for whatever reason. How do we get the data and know that something happened right now, rather than waiting until they choose to connect to VPN?

We maybe didn’t want to require them to always be on VPN. We looked at what tools are there for this. It caused us to look at Splunk Cloud specifically because we could report a lot of data via collectors, cloud services, or servers that we would collect and not be on prem. It empowered us to do more there.

Yes. I guess it was part of COVID, but it was more COVID was the catalyst to get there and how it has always been something that we looked at, just didn’t have the justification for.

Ted Gruenloh: Does that mean there’s an agent running or there’s some data collection happening on the particular endpoint that is remote, the remote endpoint? Then, that is simply feeding into the Splunk cloud directly?

Tom Knutson: A technology stack that we’re using there to collect some of that data is Cisco’s NBM module. We collect their network stack and network visibility for what they do, how they do it based on some configuration.

Then we have that check in to a server or a collector that’s running in a public cloud. Then that reports and feeds that data into Splunk. We’ve got some other tools as well that run on the endpoints for data collection that do similar things and are also providing that dataset.

COVID's Impact on the Security Stack

Ted Gruenloh: What do you have in terms of protection? Maybe next gen antivirus, EDR, or something like that on the endpoints as well, and did that change during COVID? Did you make any changes there?

Tom Knutson: No. It didn’t change because of COVID. We’ve been running our current next gen AV for a while. A big challenge that we had with this particular provider was when Windows and Windows Defender, was released, this provider wasn’t listed as a known provider at the time.

Our end users would get notifications before we changed, “You don’t have AV running. You’re not protected.” We generated a high amount of help desk tickets. We wound up disabling the display options so our end users didn’t feel insecure.

We do have a next gen antivirus that runs both a not Internet connected component and then also sends all of the datasets and the SHA signatures to a cloud resource.

We regularly look at the providers that we’re using and keep an eye on industry trends, who’s doing what in their technology stack and why. We don’t just sit on, “We’re happy with this provider. We don’t need another one.”

Because the landscape is changing faster than anybody can keep up, so we have to be ready to make changes to continue to protect ourselves. If another provider has a significant toolset or a change in their toolset, then we continue to evaluate that as well. Come renewal time or change time, we bring that into consideration.

Ted Gruenloh: You mentioned DNS as well. Do you have a security piece in place there, like a DNS sinkhole? I’m trying to hone in on that. You mentioned it in passing, but I thought I’d ask that specifically.

Tom Knutson: I suppose if you read between the lines, you could figure out the provider. We have a DNS based filtering security service for our edge clients and our workstations. We also use it for servers. We found that we can protect better if we don’t wait for the browser or the device to reach out and then inspect it.

Instead, we can say, “Hey, are you malicious?” or, “Are you bad?” right when we make the DNS request. If our DNS security provider knows that, “”Hey, this is a bad resource,” it will send a deny statement or redirect to their particular block resource.

We do also have some DNS sinkholes for on net. We haven’t yet really done much for policy based configuration for sinkholes when you’re off net. It is on our to do list. Again, time is critical.

The Threat Landscape is Real

Ted Gruenloh: Sure. Let’s focus in on maybe backups, disaster recovery. If anybody’s listening to this podcast, they probably want us to say the word ransomware at some point. I am curious to know, when I have these conversations, what our clients are doing with backups and disaster recovery.

Tom Knutson: Oh, my gosh. Ransomware, it’s in the news. You hear something about it every day. Everybody is on edge about it, whether it’s school shutdowns or manufacturing shutdowns. How does it impact you? I think ransomware has played a big part in our evolution and what we do.

For DR, it’s really enabled us to take a lot of new steps for where we’re at, where before, maybe we were backing up and had a DR site, and maybe weren’t keeping things for as long as we were keeping, or doing as much replication. With the onset of the daily influx of ransomware, it’s really caused us to take a look at what we’re doing and how we’re doing it.

From having multiple copies of the same dataset in multiple locations. In addition to multiple locations, maybe replicating to one or more cloud providers, as well as whatever virtual or physical stack that we’ve got replicating that. Keeping that data longer.

If maybe you’re hit by ransomware and the trigger’s not executed. They don’t lock you out. You don’t know that something’s happened. You’re only keeping your backups for a month or a few weeks. What do you do when it executes, and you go back to your backup sets, and you find, “Oh, geez, this is corrupted as well”?

It’s cost us keep it for a lot longer using WORM tape media, or cloud based WORM media, in addition to retaining our DR replication at multiple sites and keeping it a lot longer.

Ted Gruenloh: Now, back to the way the organization is put together and who’s in charge of what, being the jack of all trades, sometimes when we have conversations with people, they talk about how maybe the infrastructure people aren’t necessarily on the same page as the security people.

It sounds to me like if you’re trying to put together a backup or a DR plan, it actually might be easier for you to get things done since you’ve got your hands in several of those places. You’ve got your hands in the security stack and you’ve got your hands in the DR. That sort of thing.

Tom Knutson: It does make it a lot easier. A big challenge is communicating with all the sub departments or areas, whether it’s DevOps and our application team to a business unit that needs a particular application or a service, or marketing that is setting up a new application.

It might be cloud based or rely on some of our internal resources. Making sure that we secure all of that. The change with ransomware has empowered us to have better internal communication. Who’s doing what? Why are you doing it? What are your needs?

How do we ensure that you have the DR or the availability of this in the event that we do have a catastrophic event, whether it’s ransomware, a fire, or some hardware failure?

Communicating with the business units has started to be a lot more regular. With that communication, it has allowed us to ensure that everybody is on the same page.

When you don’t communicate, you don’t talk about it, or you don’t update your documentation, maybe you have some leadership that thinks that a few years ago, this is what you had, and you’ve changed it so much over time that maybe that perception is wrong.

Maybe you’ve got team members that read through the documentation, but you didn’t have a meeting, a conversation, or talk about it, and they interpreted it wrong. It’s been very important for us to have candid conversations with the respective business units or leadership about what are we doing and why are we doing this?

Reviewing our policy, procedure, and what we have. We’ve started to do it a lot more regular. It’s been a night and day difference in terms of what we’re able to do, how we’re able to do it, and the expectations set out.

Ted Gruenloh: That’s an excellent segue, actually, to the next thing I was going to try to hone in on, which was the people themselves. How you feel your relationship is with them and the trust that you build with them over time. If you want to comment on that a little bit.

Maybe also if there’s anything in terms of training that you might be doing, cybersecurity training. Anything to do with the people that directly impact the security of the organization.

Tom Knutson: People are realistically our weakest link. Doesn’t matter the organization or where you’re at. You can protect from the outside and set all these things up. All it takes is one person to do the wrong thing to compromise everything. We’ve realized that. We’ve talked about it.

We’ve got a very good relationship with the people that we serve and that we train and help to protect us. We, of course, sometimes ruffle some feathers when we do internal phishing campaigns or whatever, spoof email from trusted sources, or do pen tests with social engineering involved.

People get upset. They think, “Hey, you shouldn’t have impersonated this respected person or this individual.” If it wasn’t us and somebody else did that, what might happen? We’ve started to get a lot of positive feedback on that, where in the past, it wasn’t discussed, talked about, or it was forbidden.

Now people are seeing, and with the training that we do, they see that the threat landscape is real. It’s not just this imagined thing. The news that they see and the articles that they read are all close to home. We’ve started to over the years have a much better relationship with people in those regards.

It’s no longer just waving a red flag saying, “Hey, the sky is falling.” It’s instead saying, “We have these things. We do need to be protective of it.”

Some things that have helped have been internal security meetings with team leads from various departments and areas. We do weekly or bi weekly training with these team leads and managers. They go out and talk about these important issues with their employees and their peers.

It’s developed a groundswell campaign in terms of what we’re doing and how we’re doing it. Where before, somebody would be upset or bothered if they were blocked or something happened, but now, with this training, we have the ability to talk about these things are going to happen. Here’s why you’re seeing this. People are less frustrated.

Ted Gruenloh: That common communication is so critical. I’ve seen it happen both ways. The trust of those people is so critical. It sounds like you’ve built, over time, a pretty solid relationship with those people.

Tom Knutson: Absolutely. It’s a very easy trust to break, too. Sometimes you wind up having to repair that trust because an event or a training seminar and somebody didn’t agree with. We’ve learned to develop then and make improvements there.

Ted Gruenloh: I didn’t mean to move on to that without discussing anything else in the stack that you wanted to mention. I wanted to give you a chance if there are any tools or anything else you wanted to add there.

Tom Knutson: It’s big for us to look at more than just a centralized thing of in the past, it was, “Hey, I got a firewall. I’m OK. Nobody can get in.” Or, “How do I secure applications? What do we do?” With a big security stack, it’s so easy to be complacent in, “I have this stack. It’s good. I’m happy with the vendor or vendors that provide this.”

The technology is changing so rapidly, so fast that if you don’t evaluate new things, whether it’s from the same vendor or other vendors, you might miss out on what’s going on and these changes in how to protect your organization or your infrastructure.

The big thing that has saved us or has allowed us to adapt and overcome has been the constant evaluation of both threats, as well as how people are protecting from things, and what these software and hardware components are doing.

Ted Gruenloh: It sounds to me, like you mentioned earlier, technically an SMB, but it sounds to me like you do have a pretty enterprise level stack built out and a solid understanding of what’s important. It’s always good to hear when we get to talk to a customer and they’ve considered everything from top to bottom.

Tom Knutson: It’s a challenge. When you talk with partners, vendors, or establish new relationships, it’s, “Well, how many employees do you have?” Or, “What kind of things are you doing?” Or, “How many sites do you have?” You wind up in that SMB queue or talking about SMB products. We talk about, “Well, we want to do all of these things, too.” It’s a regular comical discourse in some of our discussions.

Ted Gruenloh: “Hey, wait, we’re bigger than that.”

Ted Gruenloh: “Take our money. We’re willing to spend the money.” [laughs] That’s great, Tom. I appreciate the time today and look forward to working with you in the future.

Tom Knutson: Thank you for having me to discuss. It’s been a pleasure.

Ted Gruenloh: Awesome. Thanks, Tom.

Tom Knutson: Thanks Ted.

We hope you enjoyed this unique perspective on cybersecurity from the manufacturing industry. For more information on how some of our clients prioritize their budget decisions and use various tools across their networks, check out our monthly podcast. We share honest, objective takes from real people fighting bad actors on the front lines.

‍

‍