Recently we had the privilege of having one of our clients, a Head of Infrastructure of a Tier 1 Global Bank, come talk with us at a Corvil offsite. He talked with us about the everyday problems and large scale challenges he faces as Head of Infrastructure, and his insights were fascinating. But one of his statements stuck with me, as I find it succinctly articulates the benefits of taking network data from the network. Here’s what he told us:
Imagine being able to make all your important life decisions using data that is undisputed, independent and authoritative. Imagine you had that data supplied to you in real-time. Imagine if all the relevant decision makers and influencers were looking at the same data, a single source of truth. Problem solving and decision making would be a whole lot simpler!
In essence, this is what network data provides to IT professionals. Gartner defines network data as "the data contained in the headers and payloads of packets and their associated flow data as traffic moves from one node to another across a distributed IT system." It is complementary to (but distinct from) machine-generated data, which is self-reported, may add overhead, and is time consuming to parse or mine. Because all applications transact on the network, user experience and application performance are seen and understood in their full context, rather than concealed within siloed views.
Let’s imagine again we were making life decisions from this data. Using machine data is like making decisions from information gleaned from snippets of a conversation. Network data gives you the whole dialogue and its context. Machine data is like a well-meaning friend or colleague who inserts themselves in your problem, and therefore influences the outcome. Network data remains passive and unbiased, and doesn’t impact the data. Therefore, it doesn’t affect the decision-making process.
Network data is data in motion and, as such, can be challenging to deal with. It is high-volume, high-velocity and high-variety. It is unstructured and difficult to harvest in a scalable manner. Accurate timing is essential and synchronized hardware timestamps of the packets are necessary to provide granular visibility and correlate operational events across the enterprise. Many switch and aggregation tap vendors have recognized the value of hardware timestamping and synchronized network data and have built this capability into their products. At Corvil, we capture this network data is via span ports, or the aforementioned aggregation switches and taps. It’s decoded, normalised, analysed, and presented back to you in real-time.
Companies need independent and authoritative data that drives real-time monitoring, troubleshooting, reporting and decision making. You must have a full, comprehensive view of how your network is performing and be able to understand the intricacies of overlaying, inter-connected applications. You need this data for alerts for immediate troubleshooting and problem resolution. You need this data for proactive network, application and End User monitoring. You need this data to hold service providers accountable; for ongoing IT requirements such as bandwidth measurement, QoS policy design, and for long term planning and trending. This normalised version of network data is also invaluable for 3rd party applications. The client mentioned above uses it to feed Risk and Compliance tools. Other clients correlate it against or feed it into SQL databases such as Splunk, KDB+, MongoDB, ElasticSearch, Tableau, etc, to make the data more and more meaningful.
The Bottom Line: Information has nowhere to hide on the network, so why wouldn’t you want to be making your business decisions based on that data? Fragmented, incomplete data can lead to ill-informed decisions. Network data is the true source of data for your company - the wire don’t lie!