If you're responsible for a corporate network then you'll be no stranger to logs: records of events that occur on your network, generated by anti-virus and other security software; devices like firewalls, intrusion detection systems (IDS), routers and other networking equipment; server and workstation operating systems; and applications running on your network.
In a large network the number of logs generated every second can run in to the hundreds of thousands, which begs a few important questions: Should you be closely monitoring these logs, and if so, why? And, perhaps most importantly, how?
The question of how logs can be managed is an important one because if it was easy then it already would be as a matter of course. But the fact is log management is far from straightforward. Ultimately, it comes down to finding a way to wade through a continuous stream of logs generated by different systems and spot the ones that are important, using limited log management resources. This is made more difficult by a number of factors, including:
- Large numbers of log sources;
- Inconsistent log content generated by different devices;
- Different log formats;
- Inconsistent time stamps on logs;
- Huge volumes of log data; and
- The need to maintain the confidentiality and integrity of logs.
Luckily, the "how" question is the easiest one to answer. There is no shortage of companies offering log management solutions. What these products do is centralize the logs by collecting them from the many different log sources on your network, normalize the logs so that they are consistent in terms of format and timestamp, and then convert the log data into decision support information: dashboards, charts and so on, to make the log information comprehensible and actionable.
It's also possible to cook up your own log management system, but it's likely to be far more efficient and cost effective to buy one off the shelf or to make use of one operating in the cloud either monitored by a managed service provider, or by your own IT staff.
The 'why' of log management
This brings us to the question of why logs should be monitored in the first place. The obvious answer is for security purposes; to spot suspicious events such as repeated failed login attempts or port scans. But Mandeep Khera, chief marketing officer at LogLogic, said that, for many companies, (especially SMBs) compliance is the key driver for adopting a log management product.
That's because they may have no choice but to store and analyze certain logs in order to comply with Federal legislation and regulations including the Federal Information Security Management Act of 2002 (FISMA), the Health Insurance Portability and Accountability Act of 1996 (HIPAA), the Sarbanes-Oxley Act of 2002 (SOX), the Gramm-Leach-Bliley Act (GLBA), and the Payment Card Industry Data Security Standard (PCI DSS).
Of course, some compliance regulations -- PCI DSS is a good example -- are all about security anyway. But would companies use log management products if it wasn't for compliance regulations? "Bigger companies, yes, but for the smaller companies, probably not," Khera admits.
For these larger companies, Khera said log management systems offer significant security benefits including the ability to spot advanced persistent threat (APT) type attacks, which it may not be possible to detect any other way.
"There will be signs in the logs, but you need to be able to get your data into a form that makes intelligent decision making possible, or that will raise an alarm automatically," he said. Systems like LogLogic's are able to accept 150,000 logs per second, search 100 million logs per second, and handle networks generating 60 billion logs per day.
But John Kindervag, a principal analyst at Forrester, said that most companies are misled into thinking that a log management system will help them with their security efforts.
"Vendors talk up the value of a log management system for threat protection or incidence response, but the real value is not that at all," he said. "The main reason to buy one is simply for compliance reporting; to provide compliance reports to auditors. "
The second reason is one of internal politics. Specifically, to help security teams justify their own value, he said.