Log Analytics is DEAD
December 03, 2015

Albert Mavashev
jKool

Log Analytics is DEAD. Did I really say that?? Yes I did. Log Analytics is a process of investigating logs and hoping to derive actionable information that might be useful to the business. Many log analytics tools are used to gain visibility into web traffic, security, application behavior, etc. But how valuable and practical is log analytics in reality?

One basic precondition for log analytics is that information to be delved into must be in log files and here lies the basic problem:

In order to derive useful analytics from logs one must have proper logging instrumentation and have it enabled everywhere, all the time.

Not only is this approach impractical and very expensive, except in a few limited cases, but it is also burdensome, imposing a significant performance overhead on the systems that produce these logs.

One must log gigabytes and gigabytes of data, store this data and then analyze it in order to detect a problem. I would call this a brute force approach. As most brute force approaches, it is expensive, slow and unwieldy. In many cases log analytics is used to catch occasional errors or exceptions. Do we really need to have all these logs to catch a few outliers?

Log analytics quickly turns into a Big Data problem – store and analyze everything, everywhere, all the time. Is that really needed? Maybe, or maybe not …

Simple Example

You deploy log analytics and it tells you've got 100 errors or exceptions in the past hour. Typically, you will want to investigate this and start with a specific exception.

Your next question would be “is what am I looking and noise or something that requires attention?” Then you will ask “what else happened” and “why?”. There is a series of questions you would ask might include the following:

■ What was my application doing?

■ What was the response time?

■ What was CPU, memory utilization?

■ What were the I/O rates and network utilization?

■ What was Java GC doing?

■ What other abnormal conditions occurred that I should be looking at?

There are so many variables. There are too many to look at and too much to analyze.

What do you do? Unfortunately this is where log analytics stops, you have to jump elsewhere. The path to root-cause becomes lengthy and painful. You may know that there is a problem, but why you have a problem in many cases is not clear.

We have all this data (big data) yet I don’t know what it means or where to look to find meaning. Of course one can say that you can parse out the log entries and extract metrics. Who will write the parsers? Who maintains the rules? Who writes complex regular expressions? What if the required metrics are not in the log files? In most cases they won’t be.

The biggest problem with log analytics is that what can be analyzed must be always logged. You need to know what information you need for root cause in advance. How often do you know what you need in advance? It is what you don’t know, have not thought about, did not instrument, did not log. It is unlikely you will have the information you will need.

Customers don’t want log analytics; customers want solutions to their problems. So what do I propose? I think log analytics is really morphing into a larger discipline.

The Post Log Analytics World

It is Application Analytics that combines logs, metrics, transactions, topology, changes, and more, along with machine learning techniques: where asking about quality of service, application performance, business and IT KPIs is a click away.

This approach must be combined with smart instrumentation, heuristics and even crowd-sourced knowledge that points to anomalies, suppresses noise and reveals important attributes without constantly collecting terabytes of data.

How do I understand what I don’t know or have not collected yet? How do I know what questions to ask?

Essentially Application Analytics is about managing risks lurking within application and IT infrastructures which are inherently complex and “broken”.

Log Analytics is dead, not because is not useful, but because it must quickly evolve into the next level.

Albert Mavashev is Chief Technology Officer at jKool.

Share this

Industry News

February 28, 2024

JFrog announced a new technology integration with Qwak, a fully managed ML Platform, that brings machine learning models alongside traditional software development processes to streamline, accelerate, and scale the secure delivery of ML applications.

February 28, 2024

ServiceNow, Hugging Face, and NVIDIA, announced the release of StarCoder2, a family of open‑access large language models (LLMs) for code generation that sets new standards for performance, transparency, and cost‑effectiveness.

February 28, 2024

GMO GlobalSign announced the availability of an Issuer for Kubernetes cert-manager.

February 27, 2024

MacStadium announced the launch of its online community to deepen the connections of application developers through knowledge sharing and collaboration.

February 27, 2024

Octopus Deploy announced the acquisition of Codefresh Inc.

February 26, 2024

Intel announced its new Edge Platform, a modular, open software platform enabling enterprises to develop, deploy, run, secure, and manage edge and AI applications at scale with cloud-like simplicity.

February 26, 2024

Tray.io announced AI-augmented API Management, a new Tray Universal Automation Cloud capability that turns any new or existing workflow into a reusable API, significantly decreasing the technical debt associated with the operational effort and costs of traditional API management (APIM).

February 26, 2024

Bitwarden Secrets Manager is now integrated with Ansible Playbook.

February 22, 2024

Check Point® Software Technologies Ltd. introduces Check Point Quantum Force series: an innovative lineup of ten high-performance firewalls designed to meet and exceed the stringent security demands of enterprise data centers, network perimeters, campuses, and businesses of all dimensions.

February 22, 2024

Tabnine announced that Tabnine Chat — the enterprise-grade, code-centric chat application that allows developers to interact with Tabnine AI models using natural language — is now available to all users.

February 22, 2024

Avaamo released Avaamo LLaMB™, a new low-code framework for building generative AI applications in the enterprise safely, securely, and fast.

February 21, 2024

CAST announced the winter release of CAST Imaging, an imaging system for software applications, with significant user experience (UX) enhancements and new features designed to simplify and accelerate processes for engineers who develop, maintain, modernize, complex software applications.

February 21, 2024

Pulumi now offers native ways to manage Pinecone indexes, including its latest serverless indexes.

February 21, 2024

Orkes, whose platform offers the fastest way to scale distributed systems, has raised $20 million in new funding.

February 20, 2024

JFrog and Carahsoft Technology announced a partnership that empowers U.S. Government organizations to safeguard their software supply chains with automated DevSecOps workflows to secure software services consumed by citizens.