Fast networks are harder to monitor than slow ones, and if monitoring systems can't handle the speed they will end up dropping packets. It's a simple fact, but one which is becoming increasingly relevant as organizations move from 1Gb Ethernet (GbE) networks to 10GbE, 40GbE and even 100GbE infrastructure.
"There is increasing demand for products that can do very rapid packet capture," said Dianna Kelley, a partner at security consultancy SecurityCurve. "Network pipes are getting bigger than ever, and companies need to monitor their networks for compliance reasons, to ensure that policies are being followed and, above all, for security reasons. If they can't capture all their traffic then they simply can't do that."
One company that is addressing this problem is New Zealand-based Endace. The company sells network monitoring appliances equipped with its own data acquisition and generation (DAG) cards, which the company claims guarantees 100 percent packet capture at full line rate.
Traditional NIC cards are unable to do this, but DAG cards' on-board field-programmable gate arrays enable smart packet-processing to be managed in the hardware. A large first-in-first-out (FIFO) memory buffer allows packets to be captured off the wire at full line rate without a single packet being dropped, according to Tim Nichols, Endace's vice president of Marketing. The company's appliances have been proven to work without dropping packets at speeds of 12Gbps, he said.
Endace appliances rely on the company's Linux-based OSm operating system, and can run monitoring and security applications from Endace's bundled Application Suite, or most other applications that customers want, running in a virtualized environment, which the company calls the Endace Application Dock.
"This Application Dock is very important as it allows Endace customers to use their own software if they want," said Kelley. "Configuration can be very hard and takes a long time, so if a company buys an Endace appliance they can keep all the nice rules they have written. They don't need to rip and replace."
The Application Suite that is bundled with every device includes a latency monitor, a NetFlow generator, a packet access program, and a SNORT based intrusion detection system. A gap detector application that monitors multi-cast market data feeds for duplicate packets, out of sequence packets and other anomalies is also available at an extra cost.
Later this quarter, the company intends to introduce a new version of OSm. This will come with Endace Vision, a tablet optimized, browser-based application that replaces most of the functionality of the Application Suite. Nichols said this will enable customers to:
- Set up and monitor a range of alarms for network performance, network latency and security anomalies through a single consolidated dashboard.
- Visualize traffic related to any kind of event or anomaly by bandwidth utilization down to 100 microsecond resolution and application type.
- Drill down into traffic to access higher resolution information about a particular host, user or application.
- Overlay detected anomalies on top of traffic visualizations to add color and context to an investigation.
- Share visualizations among users utilizing integrated screen sharing capabilities and save standard visualizations for future reference.
- Download packets of interest associated with an event from any point across the network for a full forensic investigation inside a protocol decoder.
The company sells two appliances: EndaceSensors and EndaceProbes. The Sensors can run a single application, contain no on-board traffic storage, and are designed for deployment near the network edge. The more complex Probes are designed for deployment nearer the network core, and enable captured traffic to be written to disk. They can also run multiple applications at once: ones from the Endace Application Suite as well as third party and custom applications. Both the Sensors and the Probes come in a range of different models with varying port and storage configurations.
"After a security breach, in order to work out what has happened, you need the packets, and you need them quickly," said Nichols. "We are selling the ability to search and mine packets on high speed networks with a high level of accuracy and get any packets in a few seconds. We are the 'Google of packets', if you like."
Endace doesn't have the market to itself, however. It competes with other products from established vendors like Solera Networks , Niksun, NetWitness and NetScout Systems. But Kelley believes Endace's strength is hardware; ensuring that its DAG cards are optimized to work with the rest of the appliance.
"Hardware is very important and Endace's focus on it makes them unique," she said.
The cheapest EndaceSensor costs $20,000, while more highly spec 'ed appliances can cost over $250,000. Companies typically start by buying a couple, Nichols said. "What generally happens is that they plug them in to the network and say, 'Holy Cow, I never knew that that was going on my network,' and they buy more."
The actually number of appliances that a given organization might need depends on what it needs to monitor and the size and topography of its network, but spending over $1 million on the devices is not unusual, he said.
That might seem like a lot of money, but the cost of network downtime for some organizations can be millions of dollars per hour. Put in that context, spending a few hundred thousand dollars on devices which can provide tools for minimizing downtime due to security problems starts to look very attractive indeed.
Paul Rubens has been covering IT security for over 20 years. In that time he has written for leading UK and international publications including The Economist, The Times, Financial Times, the BBC, Computing and ServerWatch.