3 DevOps Lessons Learned While Scaling
July 15, 2016

Adam Serediuk
xMatters

Throughout my 15 years in operations I've noticed the same dilemma pop up within most organizations that practice traditional software development – many companies have gotten into the habit of "triage" development. They react to problems defensively, and because the strategy is reactive, triage development occupies most of the ops team's time.

When the ops team is constantly putting out fires, they have no time, people, or tools left for building the actual product. Make no mistake, the operational component is equally important to the product. Recognizing the cost of constant fire-fighting, many teams take a more proactive approach and try to predict breakdowns.

Failures are inevitable. However by implementing a few best practices, operations professionals can take back control of their output and build more resilient systems. Here are three lessons I've learned that have allowed my teams to spend more time building up, rather than hunkering down:

1. Automate the B.S.

We've already established that spending your time putting out fires is a sure-fire way to guarantee you'll never be productive. So, should you automate your system completely and eliminate the need for human intervention in incidents? Maybe, but that's an awfully lofty goal to start. You may not need total automation.

The key is to find the right areas to automize. This should be determined strategically, but there will also be an element of trial and error. If you put automation in the wrong place or discover a newer, better way of doing things, don't be afraid to throw code out.

Some people think that constant change causes lines and logic to become muddled. I happen to believe the opposite. Constant change can eliminate problems. Ops teams should not be intimidated by change. Embracing change on a regular basis will make it less scary, and you'll see fewer fires as a result.

This must be balanced with automated testing and QA of Ops code and infrastructure. The same software development approach to unit testing and test plans can and should be applied to Ops, to eliminate regression and enable confidence in change.

2. Go Beyond IT Automation

Repeat after me: Deploying an enterprise IT automation platform is not the same as adopting DevOps. Developers, systems administrators and operations professionals use these platforms to manage the continuous integration/delivery pipeline that defines agile software development and manage system environments. While IT automation platforms are important for DevOps practitioners, they are in no way the foundation of the model.

Give equal focus to the process – the build, test, release, deploy and monitoring lifecycle – so you can iterate quickly on changes. I've seen far too many DevOps teams focus only on their automation code without giving adequate attention to the software development process and how this code fits into the larger picture. This means having ongoing conversations with your teams, QA, Development and Ops alike.

By ensuring conversations are ongoing, DevOps teams can deploy IT automation without making the situation more complicated for themselves. The team should be able to understand each deployment framework or tool selected to run automation code and where it fits into the big picture. This may mean team meetings and regular messaging, but, hey, communication is what DevOps is all about.

3. Reset Your Definition of Done

As we've seen, constant change is essential for avoiding problems. That's why startups are passing over traditional development for soft releases and continuous, everyday delivery. It's also more stimulating for the team when every day is different, and releasing small, incremental changes is safer than large monolithic releases. Recognizing the changing tide of software development, the industry has developed deployment tools that have unit and integration testing baked into them.

Thanks to these new tools, IT professionals are able to complete tasks to a fuller extent. Not only do they build, they also test and launch. With this power comes responsibility; there's no excuse for leaving anything short of "done." This is where product owners can enable and support the process, by giving equal importance to uptime, continuous delivery and testing as part of story planning.

One of my past roles was Lead Operations Engineer at a mobile and online gaming company, where my team and I built the software and cloud infrastructure. Recently, I spoke to my former boss and he noted how the code I built performed reliably in the two years since I left. The secret was developing a cohesive system, a complete package through continuous iteration. Investing in this process allowed us to build a product that adhered to the new definition of "done", which made it good enough to last.

Which brings us to what might be the most important lesson of all: software code should not exist separately from infrastructure code. The reason is simple – infrastructure without software is pointless and software can't exist without infrastructure.

Conway's Law states that "organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations." In short, if teams don't talk to each other, parts won't talk to each other. At the same time, if a product is built to work as a whole, it will work. So, why not put the pieces together?

Adam Serediuk is Director of Operations at xMatters.

Share this

Industry News

October 22, 2020

Puppet announced Puppet Comply, a new product built to work with Puppet Enterprise aimed at assessing, remediating, and enforcing infrastructure configuration compliance policies at scale across traditional and cloud environments.

October 22, 2020

Harness announced two new modules: Continuous Integration Enterprise and Continuous Features.

October 22, 2020

Render announced automatic preview environments which are essential for rapid and collaborative development of modern applications.

October 21, 2020

Conducto is launching a toolkit for simplifying complex CI/CD and data science pipelines, having raised $3 million in seed funding led by Jump Capital.

October 21, 2020

Snyk Intel vulnerability database will be integrated into IBM Cloud security capabilities to enhance security for enterprise workloads.

October 21, 2020

Accurics announced $20 million across seed and series A financing raised in the past six months, with Intel Capital leading the Series A and ClearSky leading the seed.

October 20, 2020

Splunk announced the Splunk Observability Suite, the most comprehensive and powerful combination of monitoring, investigation, and troubleshooting solutions designed to help organizations become cloud-ready and accelerate their digital transformation.

October 20, 2020

Tricentis announced Vision AI, the core technology that will now power Tosca.

October 20, 2020

MuseDev has extended its code analysis platform to deliver bug reports via Github's code scanning UI.

October 20, 2020

Digital Shadows announced the ability to detect exposed access keys.

October 19, 2020

StackRox and Robin.io announced a new partnership bringing together Robin’s application-focused approach to Kubernetes data management with StackRox’s Kubernetes-native security and compliance capabilities.

October 19, 2020

PubNub announced new Chat UI Kits to streamline chat development.

October 19, 2020

Secure Code Warrior announced support for GitHub’s new code scanning functionality in conjunction with a new collaboration with Snyk.

October 15, 2020

Couchbase announced version 2.8 of Couchbase Lite and Couchbase Sync Gateway for mobile and edge computing applications.

October 15, 2020

Kong unveiled the private beta release of Kong Konnect, a full-stack platform for cloud native applications delivered as a service.