Crisis Management: What Should You Do When Things Go Wrong?
June 12, 2017

Joyce Lin
Postman

What if you discover a fatal error or an exploit in your app? What if your app is down during a crucial time? As a developer, how you react to a crisis can mean the difference between minor blip and an embarrassing or costly company blunder.

Your organization is depending on you to save the day. Here's a crisis management plan to get things right when they go wrong:

1. Stay calm and think clearly

This is obvious, but easier said than done. Staying calm will allow you to think more clearly. Working long hours under stress frequently leads to subpar code, and may be the reason why your app is down in the first place.

Don't underestimate the value of taking a walk, grabbing a snack or something else that changes your outlook to see the issue in a different light. And don't worry about wasting time, your brain will still be working on the issue in the background.

2. Revert to working code

If the appropriate solution might take a while to implement, or you have no idea how long the fix will take, roll back to a previous version of the code as a temporary measure. This is the last time you know your service was working, and it's a stable build.

Reverting to working code can provide the extra time you need to thoughtfully address the issue. When you're no longer rushed, you can gather the information you need to more effectively resolve the problem.

3. Monitor and alert stakeholders

You're a responsible developer, so hopefully you've set up some monitoring and alerting for your app. Before you ever find yourself in crisis mode, make sure you set up the right triggers and intervals for monitoring.

Once you receive the initial alert of a failure, notify the appropriate engineers of the outage so they can get started on a fix right away. Also notify other internal stakeholders so they're aware that their services might be impacted. As an alternative to notifications, you can set up a status page to inform those who rely on your app of updates on performance and availability.

4. Debug the issue

Now that you're set up for success, it's time to dig into the issue – starting with your logs. Logs are only as helpful as you make them. You should be logging the right activities with descriptive log statements. In addition to your existing log statements, add temporary log statements to guide the debugging process.

If you're still uncertain about the cause of the outage, focus on isolating the issue. Some code bases are structured in a way that is easy to see where the code is failing, but some dependencies and abstractions make it tricky to pin down the root culprit. If you're at a loss, you can try the debugging variation of a binary search by dividing and conquering to pin down where the code is failing in the most efficient manner.

Finally, don't work in a silo. You might benefit from talking through the problem or getting another perspective. Some developers like to pair program, rely on another teammate for rubber duck debugging, or use a literal rubber duck to slow down and articulate code line by line.

5. Push fixes with continuous deployment

When you're making code changes under pressure, you might be hacking together a solution. As a result, you might also be cutting corners and incurring technical debt in exchange for a quick turnaround.

Running an automated test suite guarantees consistent code coverage, and rigorous regression testing hedges against unintentionally affecting another dependency. Automating the build, testing and deployment process ensures you deliver patches in the fastest and most efficient manner possible.

6. Communicate changes to the team

Once you've patched the fix, communicate the status and diffs to the team. Just because your world came to a standstill with the problem doesn't mean the same happened for the rest of your team. They are continuing to work on their own features and issues.

Communicate what is necessary to keep them up to speed on the changes, and how it might impact what they're working on. Some teams keep track of an activity feed to stay up-to-date with the latest code changes, while others prefer to set up an integration with their preferred messaging platforms.

Joyce Lin is a Developer Evangelist for Postman
Share this

Industry News

October 06, 2022

Platform.sh announced it has partnered with MongoDB.

October 06, 2022

Veracode announced the enhancement of its Continuous Software Security Platform to include container security.

This early access program for Veracode Container Security is now underway for existing customers.

The new Veracode Container Security offering, designed to meet the needs of cloud-native software engineering teams, addresses vulnerability scanning, secure configuration, and secrets management requirements for container images.

October 06, 2022

Mirantis announced that Mirantis Container Runtime – latest generation of the Docker Enterprise Engine, the secure container runtime that forms the foundation of Mirantis Container Cloud and Mirantis Kubernetes Engine and is used at the heart of many other Kubernetes deployments – is now available in the Microsoft Azure Marketplace.

October 05, 2022

Perforce Software announced enhanced support for automated testing with the release of Helix ALM 2022.2.

October 05, 2022

Parasoft announced the latest releases of its API and microservices testing tools, including SOAtest, Virtualize, CTP, and Selenic.

October 05, 2022

Vaadin announced the release of four Acceleration Kits designed to make it faster and easier to build and modernize Java applications for enterprise use.

October 04, 2022

Pegasystems announced the latest release of Robot Studio, the robotic process automation (RPA) low-code authoring environment for Pega's intelligent automation platform.

October 04, 2022

EvolveWare announced the Agile Business Rules Extraction (Agile BRE) solution on its Intellisys platform.

October 04, 2022

Mabl announced new features that empower quality professionals to easily validate APIs as part of their integrated end-to-end tests.

October 03, 2022

Spectro Cloud announced a major new release of its Palette Edge platform.

October 03, 2022

Arcion announced agentless change data capture (CDC) for all of its supported databases and applications.

September 29, 2022

CloudBees announced the acquisition of ReleaseIQ to expand the company’s DevSecOps capabilities, empowering customers with a low-code, end-to-end release orchestration and visibility solution.

September 29, 2022

SmartBear continues expanding its commitment to the Atlassian Marketplace, adding Bugsnag for Jira and SwaggerHub Integration for Confluence.

Bugsnag developers monitoring application stability and documenting in Jira no longer need to interrupt their workflow to access the app. Developers working in SwaggerHub can use the macro to push API definitions and changes directly to other teams and business stakeholders that work within Confluence. By increasing the presence of SmartBear tools on the Atlassian Marketplace, the company continues meeting developers where they are.

September 29, 2022

Ox Security exited stealth today with $34M in funding led by Evolution Equity Partners, Team8, and M12, Microsoft's venture fund, with participation from Rain Capital.

September 29, 2022

cnvrg.io announced that the new Intel Developer Cloud is now available via the cnvrg.io Metacloud platform, providing a fully integrated software and hardware solution.