Crisis Management: What Should You Do When Things Go Wrong?
June 12, 2017

Joyce Lin
Postman

What if you discover a fatal error or an exploit in your app? What if your app is down during a crucial time? As a developer, how you react to a crisis can mean the difference between minor blip and an embarrassing or costly company blunder.

Your organization is depending on you to save the day. Here's a crisis management plan to get things right when they go wrong:

1. Stay calm and think clearly

This is obvious, but easier said than done. Staying calm will allow you to think more clearly. Working long hours under stress frequently leads to subpar code, and may be the reason why your app is down in the first place.

Don't underestimate the value of taking a walk, grabbing a snack or something else that changes your outlook to see the issue in a different light. And don't worry about wasting time, your brain will still be working on the issue in the background.

2. Revert to working code

If the appropriate solution might take a while to implement, or you have no idea how long the fix will take, roll back to a previous version of the code as a temporary measure. This is the last time you know your service was working, and it's a stable build.

Reverting to working code can provide the extra time you need to thoughtfully address the issue. When you're no longer rushed, you can gather the information you need to more effectively resolve the problem.

3. Monitor and alert stakeholders

You're a responsible developer, so hopefully you've set up some monitoring and alerting for your app. Before you ever find yourself in crisis mode, make sure you set up the right triggers and intervals for monitoring.

Once you receive the initial alert of a failure, notify the appropriate engineers of the outage so they can get started on a fix right away. Also notify other internal stakeholders so they're aware that their services might be impacted. As an alternative to notifications, you can set up a status page to inform those who rely on your app of updates on performance and availability.

4. Debug the issue

Now that you're set up for success, it's time to dig into the issue – starting with your logs. Logs are only as helpful as you make them. You should be logging the right activities with descriptive log statements. In addition to your existing log statements, add temporary log statements to guide the debugging process.

If you're still uncertain about the cause of the outage, focus on isolating the issue. Some code bases are structured in a way that is easy to see where the code is failing, but some dependencies and abstractions make it tricky to pin down the root culprit. If you're at a loss, you can try the debugging variation of a binary search by dividing and conquering to pin down where the code is failing in the most efficient manner.

Finally, don't work in a silo. You might benefit from talking through the problem or getting another perspective. Some developers like to pair program, rely on another teammate for rubber duck debugging, or use a literal rubber duck to slow down and articulate code line by line.

5. Push fixes with continuous deployment

When you're making code changes under pressure, you might be hacking together a solution. As a result, you might also be cutting corners and incurring technical debt in exchange for a quick turnaround.

Running an automated test suite guarantees consistent code coverage, and rigorous regression testing hedges against unintentionally affecting another dependency. Automating the build, testing and deployment process ensures you deliver patches in the fastest and most efficient manner possible.

6. Communicate changes to the team

Once you've patched the fix, communicate the status and diffs to the team. Just because your world came to a standstill with the problem doesn't mean the same happened for the rest of your team. They are continuing to work on their own features and issues.

Communicate what is necessary to keep them up to speed on the changes, and how it might impact what they're working on. Some teams keep track of an activity feed to stay up-to-date with the latest code changes, while others prefer to set up an integration with their preferred messaging platforms.

Joyce Lin is a Developer Evangelist for Postman

The Latest

October 16, 2017

A survey of more than 750 development team leaders in the US and UK, revealed that 68 percent plan to build more apps during the next 12 months. At the same time as reporting increased volumes of development, 91 percent of developers surveyed agree that user expectations for innovation and quality have increased, but app deliveries continue to fail ...

October 12, 2017

Today, organizations must digitally evolve or they risk becoming irrelevant. One area that’s been growing in adoption is a shift to developing and deploying modern applications in the cloud, which requires software and IT architects to rethink how to architect and manage these apps ...

October 10, 2017

Designing and deploying complete software-defined data centers (SDDCs) can be complicated because each implementation requires a broad range of infrastructure to support heavy demands for compute, networking, storage, applications and security ...

October 05, 2017

According to LogiGear's State of Software Testing Survey, almost one-third of the respondents are experiencing classic test automation issues. One problem commonly cited among respondents was that management didn’t fully understand what it takes to have a successful automation program ...

October 04, 2017

Load balancing at the DNS (Domain Name System) level has been around for a few decades now, but it didn't become crucial until recently as technology is moving to the cloud. DNS is the perfect solution for managing cloud systems ...

October 02, 2017

QualiTest recently compiled a data report analyzing software testers globally. The report details the Quality Assurance and Software Testing job market, one of the fastest growing job markets and a bellwether of tech employment due to QA's involved in nearly every conceivable industry ...

September 28, 2017

API use is exploding among developers, as APIs are an essential part of software development for the web, IoT, mobile and AI applications. APIs allow a developer to create programs or apps that can successfully request services or data from other applications or operating system. This connectivity, though powerful, is complex, and that complexity grows with new apps, new hardware such as the new iPhone and Echo, and the creation of new APIs ...

September 26, 2017

Companies are placing a greater value on high performing IT professionals as IT demands continue to escalate, according to Puppet's DevOps Salary Report ...

September 25, 2017

Code working perfectly doesn't matter much if apps aren't reaching customers, or are negatively impacted by network latency or outages. All the customer cares about is how they are enjoying an app. To effectively guarantee application availability and usage satisfaction, DevOps teams need to leverage three important application assurance data sets into their delivery automation logic ...

September 21, 2017

There are many options when it comes to container orchestration platforms and services. Figuring out which one is best for a particular organization’s needs and applications can be a challenge. Which platform for containerization you choose can significantly influence your business success, so the selection process should be carefully considered ...

Share this