The End of the Beginning for Traditional IT Operations Monitoring?

Gartner have just published a new report with the descriptive title Use Data- and Analytics-Centric Processes With a Focus on Wire Data to Future-Proof Availability and Performance Management.

The End of the Beginning for Traditional IT Operations Monitoring?By Donal O'Sullivan    30 March 2016      Product

In 1942, Winston Churchill uttered these famous words in a speech in London:

”Now this is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning.”

The immediate reasons for Churchill to make this claim are not so important for our purpose – suffice it to say that with his deep military knowledge and strategic insight, it was clear to him that a shift in the fundamental dynamic and likely outcome of the war had occurred. This shift was not obvious to others. On the surface, the war looked much the same as before. So, wisely, Churchill tempered his optimism with a deep caution, but made it clear that at some fundamental level a corner had been turned.

Has a similar fundamental corner just been turned in the world of IT Operations Monitoring?

Gartner have just published a new report with the descriptive title Use Data- and Analytics-Centric Processes With a Focus on Wire Data to Future-Proof Availability and Performance Management.

Written by Will Cappelli and Vivek Bhalla, this clear and quite concise document signals some significant, perhaps seismic, changes in how Gartner clients and Gartner itself views the challenge of availability and performance management in today's IT world. Previously, Gartner placed a heavy emphasis on topology discovery and service delivery mapping. Monitoring solutions were required to build and maintain these maps, and IT Ops departments used them to infer end user experience and system performance, and to identify and resolve problems. The monitoring solutions were complex and expensive, but presumably necessary and worthwhile, since they were the only way to ensure proper operation of the IT systems and delivery of an optimal end user experience.

The primary data sources consumed and used by these systems were API and intra-code data, with log data coming up in third. Wire or packet data brought up the rear, valuable mostly to network teams wanting to understand and troubleshoot network performance.

This report signals several key changes in that understanding and in that approach. A few example quotes of the more radical departures signaled:

  • While log data will have a role […], it is wire data that will prove to be the most critical source of data for availability and performance monitoring…
  • Modern IT systems […] are particularly difficult to monitor using traditional technologies [which use] element instrumentation…
  • For tradition monitoring technologies to yield an accurate picture [you need to] instrument a very large number of system elements [and extract] the data at a very high frequency…
  • Increasing hybridization […] puts a large number of the system elements out of the reach of the monitoring system…
  • Exploit wire data as a Source of Information and Analysis of the Entire IT Infrastructure and Application Portfolio in production, Not Only the Network…

And perhaps most tellingly:

  • Gartner believes the importance of wire data will continue to increase in most organizations over the next few years…

You can access the full report here.

So, wire data is coming out of the network and providing value across the organization. There are a few key reasons according to Gartner:

  • It’s pretty much real time, and that’s where we are all headed.
  • The network communication starts to dominate over the compute in increasingly distributed systems.
  • System components are becoming individually too small to instrument.
  • Wire data gets you closer to the end user experience, the ultimate performance arbiter.

Based on my experience with customers over the last several years, extracting deep insights from wire data in real time, I would add a few more:

  • It’s cheaper in terms of coverage and operational complexity.
  • It’s ubiquitous: it doesn’t matter if you wrote that particular piece of software or not, wire data can monitor it.
  • It’s passive: it doesn’t impose any performance burden on the systems it monitors, and for some critical or very busy nodes like large databases, using logging or any API method is simply impossible.

The traditional big monitoring vendors are largely ignoring this. The likes of CA, IBM, HP and others are largely oblivious to what seems, to them, to be a small shift in the sands. Even Splunk has paid only polite attention to the value of wire data. But for those of us who have been at the forefront of working with wire data for years, this feels like the end of something old, and the beginning of something new.

You can read my earlier blogs on the Democratization of Network Data, and The Problem with APM.

Gartner, Inc., Use Data- and Analytics-Centric Processes With a Focus on Wire Data to Future-Proof Availability and Performance Management, Will Cappelli, Vivek Bhalla, 10 March 2016.

The End of the Beginning for Traditional IT Operations Monitoring?

Donal O'Sullivan, Vice President, Product Management, Corvil
Corvil safeguards business in a machine world. We see a future where all businesses trust digital machines to algorithmically conduct transactions on their behalf. For some businesses, this future is now.
@corvilinc