A Smarter, Data Science-Driven Approach to Operations
December 04, 2015

Amit Sasturkar
OpsClarity

Data science and machine learning algorithms have become pervasive throughout the modern consumer world. There are many successful applications of machine learning in consumer products that we use on a daily basis, including:

■ Movie, music and product recommendations

■ Ad targeting

■ Web search

But, when we look at the Ops world, we find that there is no breakthrough product that incorporates these same machine learning innovations.

A relevant comparison is with page-level and host-level features (for example, page rank for URLs or host rank for hosts) used in search ranking. These features are typically a function of the WebMap (the massive graph where nodes are URLs, and edges are hyperlinks between URLs). The page rank algorithm allows the ranking of URLs in the WebMap based on the hyperlinks between them. It is a very effective way to get a reasonable estimate of the overall importance of a URL.

What if we used similar ideas to rank the hundreds or sometimes thousands of alerts that operations engineers receive, especially when they are managing hundreds of machines? What is the equivalent of the WebMap in the Ops world?

Another relevant example is provided by duplicate web page detection. These algorithms run as MapReduce jobs on massive Hadoop clusters (thousands of machines) and detect duplicate pages across tens of billions of web pages. When the mappers or reducers fail or when there are performance degradations, hundreds of alerts are generated, many of them for the same underlying root cause.

What if we applied the techniques of web page duplicate detection to eliminate the duplicate and unnecessary alerts received by Ops?

A third big challenge is personalization of content. Personalization is a well-studied problem in the consumer space, with user feedback — both implicit (clicks and actions) and explicit (reviews and ratings) — contributing critical inputs to the learning algorithms. Employing this type of machine learning means that the more time a user spends with a product, the better their user experience will be.

What if we incorporated feedback to learn Ops users’ preferences and continuously improve the accuracy of alert generation and alert ranking?

The answers to these questions will become evident as we bring the innovations in data science and machine learning that are commonplace in the consumer world to the Ops world. DevOps teams need, in effect, an “expert assistant” that can learn their application and system environment, detect and correlate failures, and make recommendations that drive increased focus and productivity — even as everything is continuously changing. It’s time for Ops to get smarter.

Amit Sasturkar is Co-Founder and CTO of OpsClarity.

The Latest

January 17, 2019

To better align business and IT objectives, enterprise organizations should focus on the core "problems" that individual business units face today in driving out real consumer value. Until the roadblocks and inhibitors — and, ultimately, the resultant technical debt — are removed from the equation, large enterprise organizations will continue struggling to succeed ...

January 16, 2019

Technical debt is what results when legacy platforms or highly integrated and dependent systems and processes inhibit large enterprise organizations from meeting the needs of internal business stakeholders. In many cases, the core objectives that drive real, monetizable business value are not aligned to the esoteric IT goals of "automation" and "Agile development." This creates a fundamental disconnect between business and IT ...

January 14, 2019

Budget season is an important time of the year for businesses because it gives senior IT and security leaders time to reflect on what went right this year and what initiatives need to be given priority in the new year. Recent research from Threat Stack shows security budgets are expected to increase by 19 percent over the next two years, but business leaders are still facing challenges determining where to allocate this budget in the face of rapidly evolving infrastructure ...

January 10, 2019

As organizations of all sizes are embracing hybrid and multi-cloud infrastructures, they are experiencing the many benefits of a more agile, distributed and high-speed environment where new applications and services can be built and delivered in days and weeks, rather than months and years. But as the adoption of these next generation architectures continues to grow, so do the complexities of securing the cloud workloads running on them ...

January 09, 2019

DEVOPSdigest invited DevOps experts for their predictions on how DevOps and related technologies will evolve and impact business in 2019. Part 9, the final installment, covers microservices, containers and APIs ...

January 08, 2019

DEVOPSdigest invited DevOps experts for their predictions on how DevOps and related technologies will evolve and impact business in 2019. Part 8, covers microservices and containers ...

January 07, 2019

DEVOPSdigest invited DevOps experts for their predictions on how DevOps and related technologies will evolve and impact business in 2019. Part 7, covers the Cloud ...

December 20, 2018

DEVOPSdigest invited DevOps experts for their predictions on how DevOps and related technologies will evolve and impact business in 2019. Part 6, covers DevOps Analytics, including AI and Machine Learning ...

December 19, 2018

DEVOPSdigest invited DevOps experts for their predictions on how DevOps and related technologies will evolve and impact business in 2019. Part 5 is all about testing ...

December 18, 2018

DEVOPSdigest invited DevOps experts for their predictions on how DevOps and related technologies will evolve and impact business in 2019. Part 4 covers Agile, CI/CD and automation ...

Share this