3 DevOps Lessons Learned While Scaling
July 15, 2016

Adam Serediuk
xMatters

Throughout my 15 years in operations I've noticed the same dilemma pop up within most organizations that practice traditional software development – many companies have gotten into the habit of "triage" development. They react to problems defensively, and because the strategy is reactive, triage development occupies most of the ops team's time.

When the ops team is constantly putting out fires, they have no time, people, or tools left for building the actual product. Make no mistake, the operational component is equally important to the product. Recognizing the cost of constant fire-fighting, many teams take a more proactive approach and try to predict breakdowns.

Failures are inevitable. However by implementing a few best practices, operations professionals can take back control of their output and build more resilient systems. Here are three lessons I've learned that have allowed my teams to spend more time building up, rather than hunkering down:

1. Automate the B.S.

We've already established that spending your time putting out fires is a sure-fire way to guarantee you'll never be productive. So, should you automate your system completely and eliminate the need for human intervention in incidents? Maybe, but that's an awfully lofty goal to start. You may not need total automation.

The key is to find the right areas to automize. This should be determined strategically, but there will also be an element of trial and error. If you put automation in the wrong place or discover a newer, better way of doing things, don't be afraid to throw code out.

Some people think that constant change causes lines and logic to become muddled. I happen to believe the opposite. Constant change can eliminate problems. Ops teams should not be intimidated by change. Embracing change on a regular basis will make it less scary, and you'll see fewer fires as a result.

This must be balanced with automated testing and QA of Ops code and infrastructure. The same software development approach to unit testing and test plans can and should be applied to Ops, to eliminate regression and enable confidence in change.

2. Go Beyond IT Automation

Repeat after me: Deploying an enterprise IT automation platform is not the same as adopting DevOps. Developers, systems administrators and operations professionals use these platforms to manage the continuous integration/delivery pipeline that defines agile software development and manage system environments. While IT automation platforms are important for DevOps practitioners, they are in no way the foundation of the model.

Give equal focus to the process – the build, test, release, deploy and monitoring lifecycle – so you can iterate quickly on changes. I've seen far too many DevOps teams focus only on their automation code without giving adequate attention to the software development process and how this code fits into the larger picture. This means having ongoing conversations with your teams, QA, Development and Ops alike.

By ensuring conversations are ongoing, DevOps teams can deploy IT automation without making the situation more complicated for themselves. The team should be able to understand each deployment framework or tool selected to run automation code and where it fits into the big picture. This may mean team meetings and regular messaging, but, hey, communication is what DevOps is all about.

3. Reset Your Definition of Done

As we've seen, constant change is essential for avoiding problems. That's why startups are passing over traditional development for soft releases and continuous, everyday delivery. It's also more stimulating for the team when every day is different, and releasing small, incremental changes is safer than large monolithic releases. Recognizing the changing tide of software development, the industry has developed deployment tools that have unit and integration testing baked into them.

Thanks to these new tools, IT professionals are able to complete tasks to a fuller extent. Not only do they build, they also test and launch. With this power comes responsibility; there's no excuse for leaving anything short of "done." This is where product owners can enable and support the process, by giving equal importance to uptime, continuous delivery and testing as part of story planning.

One of my past roles was Lead Operations Engineer at a mobile and online gaming company, where my team and I built the software and cloud infrastructure. Recently, I spoke to my former boss and he noted how the code I built performed reliably in the two years since I left. The secret was developing a cohesive system, a complete package through continuous iteration. Investing in this process allowed us to build a product that adhered to the new definition of "done", which made it good enough to last.

Which brings us to what might be the most important lesson of all: software code should not exist separately from infrastructure code. The reason is simple – infrastructure without software is pointless and software can't exist without infrastructure.

Conway's Law states that "organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations." In short, if teams don't talk to each other, parts won't talk to each other. At the same time, if a product is built to work as a whole, it will work. So, why not put the pieces together?

Adam Serediuk is Director of Operations at xMatters.

Share this

Industry News

March 18, 2024

Kubiya.ai announces the launch of its DevOps Digital Agents.

March 18, 2024

Aviatrix® introduced Aviatrix Distributed Cloud Firewall for Kubernetes, a distributed cloud networking and network security solution for containerized enterprise applications and workloads.

March 18, 2024

Stride announces the general availability of Stride Conductor, its new autonomous coding product that transforms the software development landscape.

March 14, 2024

CircleCI unveiled CircleCI releases, which enables developers to automate the release orchestration process directly from the CircleCI UI.

March 13, 2024

Fermyon™ Technologies announces Fermyon Platform for Kubernetes, a WebAssembly platform for Kubernetes.

March 13, 2024

Akuity announced a new offer targeted at Enterprises and businesses where security and compliance are key.

March 13, 2024

New Relic launched new capabilities for New Relic IAST (Interactive Application Security Testing), including proof-of-exploit reporting for application security testing.

March 12, 2024

OutSystems announced AI Agent Builder, a new solution in the OutSystems Developer Cloud platform that makes it easy for IT leaders to incorporate generative AI (GenAI) powered applications into their digital transformation strategy, as well as govern the use of AI to ensure standardization and security.

March 12, 2024

Mirantis announced significant updates to Lens Desktop that makes working with Kubernetes easier by simplifying operations, improving efficiency, and increasing productivity. Lens 2024 Early Access is now available to Lens users.

March 12, 2024

Codezero announced a $3.5 million seed-funding round led by Ballistic Ventures, the venture capital firm dedicated exclusively to funding entrepreneurs and innovations in cybersecurity.

March 11, 2024

Prismatic launched a code-native integration building experience.

March 07, 2024

Check Point® Software Technologies Ltd. announced its Check Point Infinity Platform has been ranked as the #1 Zero Trust Platform in the latest Miercom Zero Trust Platform Assessment.

March 07, 2024

Tricentis announced the launch and availability of SAP Test Automation by Tricentis as an SAP Solution Extension.

March 07, 2024

Netlify announced the general availability of the AI-enabled deploy assist.

March 07, 2024

DataStax announced a new integration with Airbyte that simplifies the process of building production-ready GenAI applications with structured and unstructured data.