A Smarter, Data Science-Driven Approach to Operations
December 04, 2015

Amit Sasturkar
OpsClarity

Data science and machine learning algorithms have become pervasive throughout the modern consumer world. There are many successful applications of machine learning in consumer products that we use on a daily basis, including:

■ Movie, music and product recommendations

■ Ad targeting

■ Web search

But, when we look at the Ops world, we find that there is no breakthrough product that incorporates these same machine learning innovations.

A relevant comparison is with page-level and host-level features (for example, page rank for URLs or host rank for hosts) used in search ranking. These features are typically a function of the WebMap (the massive graph where nodes are URLs, and edges are hyperlinks between URLs). The page rank algorithm allows the ranking of URLs in the WebMap based on the hyperlinks between them. It is a very effective way to get a reasonable estimate of the overall importance of a URL.

What if we used similar ideas to rank the hundreds or sometimes thousands of alerts that operations engineers receive, especially when they are managing hundreds of machines? What is the equivalent of the WebMap in the Ops world?

Another relevant example is provided by duplicate web page detection. These algorithms run as MapReduce jobs on massive Hadoop clusters (thousands of machines) and detect duplicate pages across tens of billions of web pages. When the mappers or reducers fail or when there are performance degradations, hundreds of alerts are generated, many of them for the same underlying root cause.

What if we applied the techniques of web page duplicate detection to eliminate the duplicate and unnecessary alerts received by Ops?

A third big challenge is personalization of content. Personalization is a well-studied problem in the consumer space, with user feedback — both implicit (clicks and actions) and explicit (reviews and ratings) — contributing critical inputs to the learning algorithms. Employing this type of machine learning means that the more time a user spends with a product, the better their user experience will be.

What if we incorporated feedback to learn Ops users’ preferences and continuously improve the accuracy of alert generation and alert ranking?

The answers to these questions will become evident as we bring the innovations in data science and machine learning that are commonplace in the consumer world to the Ops world. DevOps teams need, in effect, an “expert assistant” that can learn their application and system environment, detect and correlate failures, and make recommendations that drive increased focus and productivity — even as everything is continuously changing. It’s time for Ops to get smarter.

Amit Sasturkar is Co-Founder and CTO of OpsClarity.

Share this

Industry News

October 17, 2019

Acquia announced the availability of its new Developer Studio, a suite of tools designed to improve the productivity of Drupal developers.

October 17, 2019

Talend announced Talend Cloud is now available on Microsoft Azure, offering a secure and scalable Integration Platform-as-a-Service for collecting, transforming and cleaning data.

With embedded data quality and native integration performance, Talend Cloud on Microsoft Azure delivers the trusted data companies need to make real-time business decisions, accelerate advanced analytics, and meet regulatory compliance requirements.

October 17, 2019

Cognizant entered into an agreement to acquire Contino, a privately-held technology consulting firm.

October 16, 2019

Red Hat announced Red Hat OpenShift 4.2, the latest version of Red Hat’s enterprise Kubernetes platform designed to deliver a more powerful developer experience.

October 16, 2019

Gluware announced Gluware Automation v3.6, which extends the platform API capabilities including integrations with the Mist and Ansible platforms and introduces lifecycle management and infrastructure integration enhancements.

October 16, 2019

XebiaLabs announced that Wipro has renewed and extended its partnership with XebiaLabs as their Strategic Enterprise DevOps Partner across the globe.

October 15, 2019

Puppet announced enhancements to its current product portfolio and the public beta of a new project focused on providing a simplified continuous deployment workflow.

October 15, 2019

DBmaestro expanded its database automation platform to enable CI/CD and release automation for MySQL, MariaDB and Amazon RDS with DBmaestro DevOps Platform v2019.4.

October 15, 2019

Radware announced the launch of Radware Kubernetes Web Application Firewall (WAF), a comprehensive and highly scalable application security solution for Kubernetes-based environments.

October 10, 2019

CloudBees launched a new partner program that expands ISV partners’ ability to align with CloudBees offerings and the global Jenkins community.

October 08, 2019

Nureva announced a key update to the Jira Software integration with Span Workspace, Nureva’s cloud-based digital canvas for visual planning and collaboration.

October 08, 2019

Fugue announced support for Open Policy Agent (OPA), an open source general-purpose policy engine and language for cloud infrastructure.

October 03, 2019

Redgate announced the launch of SQL Compare v14, the latest version of its industry standard tool for quickly and accurately comparing and deploying SQL Server databases.

October 03, 2019

Harness announced the release of Continuous Insights, a new capability of its CD platform that enables organizations to see clearly into software delivery performance across their engineering and development teams without needing to manually collect, correlate, and report metrics that might take days or weeks.

October 03, 2019

OutSystems and Workato announced a partnership aimed at allowing organizations to rapidly realize innovation, time to value, productivity, and mission-critical objectives through readily available application connectors.