Pay now or pay later: Why you can't afford to skimp on your security budget

by Jonathan Yarden | Sep 16, 2005 7:00:00 AM

Tags: Strategy, Jonathan Yarden, security, e-mail, e-mail system, IT Staff, information technology, e-mail server, Internet Security Focus Newsletter

6 comment(s)

Takeaway: Many corporations have stretched their IT staff and budgets so thin that it interferes with the department's abilities to support the corporate enterprise�much less keep it secure. But a large-scale incident will cost much more than preventive measures. How do you convince the ones holding the purse strings? Jonathan Yarden has this cautionary tale to share.

Economic business models have traditionally focused on supply and demand. And while this is a long-respected approach, I think it's time to consider a new model based on downtime and money. Somehow, many companies still aren't getting the message that modern business depends on technology�particularly when it comes to communication.

In my experience, many corporations seem to think that the IT department is the best place to focus their cost-savings efforts. Of course, they are dead wrong. Technology runs the corporate machine, and it's time to adjust traditional models of corporate economics to account for technology costs.

Many corporations have stretched their IT staff and budgets so thin that it interferes with the department's abilities to support the corporate enterprise�much less keep it secure. But when a malfunction strikes a critical system, it's rather amazing how quickly the powers-that-be forget the word budget.

Of course, you know and I know that technology departments require adequate funding to function properly�that's not the problem. How do you convince the ones holding the purse strings? Here's a cautionary tale to share.

About a month ago, a Fortune 500 company encountered a worst-case system failure. Its e-mail server crashed�the result of a combination of bad hardware, corrupt data, and e-mail worms. The entire e-mail system came to a grinding halt, and there was no backup system to bring online.

So, while the IT staff scrambled to get the e-mail system operational, work throughout the entire company came to a standstill. As e-mail delivery failed, customers began calling in�resulting in a brand-new problem. With incoming phone lines jammed to capacity from customers, the system dropped or failed to complete calls.

The massive call volume also made it difficult for employees to get an outbound line or use fax machines. The voice mail system was yet another casualty of the e-mail server problem; it didn't have the ability to process all of the calls coming in�or even allow employees to pick up their voice mail from customers.

While the IT staff focused its efforts entirely on finding out the cause of the problem and getting the e-mail system operational as fast as possible, the company's management was busy rushing to blame someone. Questions on how this problem occurred were the first to crop up and led directly to questions about the IT department's capability.

Of course, the IT staff was well aware of the possibility of such a problem. However, because management hadn't seen IT as a "profit center," a redundant e-mail system wasn't in the budget.

Fortunately, the e-mail server stored its data on a Fibre Channel RAID array. Unfortunately, the failed hardware turned out to be the Fibre Channel controller, which the IT staff had to order. Two days after the initial e-mail system crash, the company told employees to take days off as everyone waited for the necessary hardware to arrive.

During that time, the IT staff struggled to justify its existence, as work at the entire company ground to a halt. More than a few IT employees simply quit�both from abuse and from working excessive hours without overtime.

When the Fibre Channel adaptor arrived during day two of the outage, the IT staff quickly discovered that it would need to restore the entire system from backup due to data corruption. With more than a terabyte of data to restore, it was evident this wouldn't be a quick process. As a result, the finger-pointing continued.

The IT staff worked around the clock to bring the e-mail system back online and restore the data, with its efforts culminating in success on day four. But within an hour, both the e-mail system and the Internet were unusable again�the company didn't have enough Internet bandwidth to begin with.

While the e-mail server was down, management instructed the IT staff to open access on the firewall so employees could use free Web-based e-mail services. Open access to the Internet led to more than a few incidents of viruses, spyware, and more e-mail worms. In addition, a lot of existing viruses and spyware were present on computers, few of which even had desktop virus protection.

After disconnecting infected systems and a few tense hours, the IT staff managed to somewhat recover the e-mail server, and the Internet connection was no longer flooded with incoming SMTP traffic. The Internet connection was reportedly "slow as always" but usable.

This is about the point where I came in. Management asked me what they could do to prevent this from happening again. I told them to invest in the IT department in order to replace the employees who quit in disgust, purchase a backup e-mail server system, and increase Internet bandwidth.

While I can't tell you exactly what the incident cost this company, I can tell you that it would have cost much less to prevent it in the first place. In this case, to avoid almost a full work week of downtime, this Fortune 500 company could have spent approximately $25,000.

Without a doubt, this system failure cost the company much more than that�not just in dollars, but also in the loss of intellectual knowledge as well as customer satisfaction and trust. Even worse, this entire situation was preventable had the powers-that-be understood that you can't rely on traditional economic models to dictate technology investments.

Miss an issue?

Check out the Internet Security Focus Archive, and catch up on the most recent editions of Jonathan Yarden's column.

Want more advice for locking down your network? Stay on top of the latest security issues and industry trends by automatically signing up for our free Internet Security Focus newsletter, delivered each Monday.

Jonathan Yarden is the senior UNIX system administrator, network security manager, and senior software architect for a regional ISP.

6 comment(s)