XClarity & Splunk: Scalable Systems Management & Log Analytics

Cowritten with Jeff Van Heuklon.

Data is Everywhere!

Organizations are grappling with data generated every day from a variety of sources. Humans and machines are producing huge amounts of data every minute, something in the order of terabytes to petabytes. While the usefulness of data to help make smart business decisions is not always immediately obvious, there is growing investment by organizations in harnessing data and gathering intelligence. They need tools to enable the extraction of insights hidden inside of data to leverage it as a source of competitive advantage.

Data Types and Intelligence Tools

Data comes in a variety of formats — text, binary, structured, unstructured, human-readable, machine-generated, encrypted, etc. Data also has other characteristics. For instance, data generated periodically over time (time series data), data from security events (audit log data), social media data, blog posts and so on. Depending upon the specific personalities of data, the tools to extract intelligence from it need to be adopted and optimized for the respective profile and usage.

While humans create large volumes of data every day, machines generate many times more data and they do it faster. Think about real-time data from sensor devices, debug and performance data from thousands of running applications, audit log data from servers and other devices and security data generated by intrusion detection devices, firewalls, monitoring tools, etc., to name a few. Extracting intelligence from machine-generated data can help in many scenarios, but it is also quite challenging just given the sheer volume of information and the variety of data sources and formats involved. This would be a great example of the classic big data problems. While there are numerous tools available to deal with big data, Splunk has specifically focused on and optimized the extraction of operational analytics from machine-generated data. I will talk more about Splunk in a minute.

Scalable Systems Monitoring

Lenovo recently introduced XClarity, a great tool for managing and monitoring the Lenovo server platforms. XClarity is a powerful utility for consolidating management of hundreds of server systems from a single place, performing discovery and inventory of end points, mass-application of firmware updates, automating system configurations through patterns, deployment of operating systems on bare-metal servers and scalable monitoring of system events and alerts. In addition, XClarity provides an extensible REST API, which enables easy integration upstream into other management tools such as VMware vCenter.

If we focus on the monitoring aspects of XClarity, the tool continuously listens for events from all the resources it manages. Most of these are received via standard protocols such a CIM (common information model) or SNMP (simple network management protocol). Users can either view a log of all these events in the XClarity GUI console, or configure “event forwarders”, which enable them to forward events to another external visualization or management tool.

XClarity Integration with Splunk

Let’s talk about Splunk again. Splunk is a business-analytics platform focused on operational analytics. One of the key applications of Splunk is analyzing time-series, machine-generated data such as system event logs, audit events, etc., and enabling very detailed and highly customizable operational analytics dashboards. Users can quickly extract information that is interesting to them and analyze it via the Splunk engine.

Lenovo worked with Splunk in optimizing the tool to run on our System x servers, plus we integrated the XClarity tool with Splunk by developing an XClarity Splunk App. The result is a combination of scalable systems management and monitoring, and integrated analytics on top of the monitored data. 

We developed rich dashboards in the XClarity Splunk App to provide detailed and easy-to-understand graphs from a variety of hardware and system event logs, in real-time.

To use the XClarity Splunk App, users can configure a Syslog event forwarder to point to the Splunk server listener. From that point on, all of the events that match what the user wants to forward will be received by the XClarity Splunk App.

A few examples of the critical insights that can be gained from the XClarity Splunk App:

  • The volume and types of events generated over time from all monitored hardware. This will help administrators quickly identify problem hardware and take actions.
  • Percentage of total events being surfaced by each end point type such as the chassis management module (CMM), switch module, blade, etc.
  • Number of times when a power threshold has been exceeded for any XClarity-managed resource, over time. This can help identify environmental issues in the data center. If exceeding of power thresholds caused power capping, this could also explain performance slowdowns.
  • Number of user accounts that were created on XClarity instances over time. Spikes in the number of new accounts could help identify uncommon security activities for audit purposes.
  • User IDs that attempted to authenticate to XClarity, but failed. Seeing which unauthorized user IDs were used to attempt access would be useful in system audits.
  • Number of login attempts made outside of normal business hours. This may help identify uncommon user account activity, like a large number of login attempts in the middle of the night or on a weekend.

As is evident, XClarity and Splunk together provide a powerful and comprehensive tool for managing and monitoring Lenovo server platforms in the data center and helping you to gain complete control over the infrastructure. The XClarity App for Splunk will be made available on the Splunk base soon and users of XClarity and Splunk can download the application from the marketplace. We are also in the process of publishing a Solution brief for the product.

What’s Next?

Our pursuit to help customers deploy robust data center infrastructure via Lenovo hardware platforms and provide them full control over their infrastructure will continue. We are working on integrating XClarity with other popular analytics tools that I will discuss in future articles. Please stay tuned!