Crisis management

Perhaps work feels like a crisis every day to you and your team. The software business changes rapidly. Management, partners, and customers shift requirements frequently. And issues arise when you expect them the least. However, there’s everyday chaos, and then there’s a real crisis.

You can reduce everyday chaos: Don’t panic, acknowledge that You can’t have it all, allocate significant time to Debt and investment, let partners know that You can depend on me, and engage stakeholders when You’re late. While these practices will improve your daily life, they can’t completely protect you from a real crisis when one comes for you.

A real crisis could be a broad service outage, a security breach, a highly public bug, or a failure to meet government and/or contractual obligations. It could be the kind of crisis that draws the attention of executives and, potentially, mass media. The sort of crisis you feel down in your soul. What do you do? Don’t panic—I’ve got you.

Eye of the storm

The first and most important thing to do in a crisis is stay calm and think clearly. Feeling anxious and filling your mind with worries won’t resolve the crisis. It will only cloud your judgement and cause you to make rash decisions. Instead, accept the current situation. Perhaps people could have avoided the crisis by making different choices, but the past is past. You must think and act in the present.

Regardless of what got you into the current crisis, you will now be evaluated by how you respond. Being calm, clear, and well-reasoned will earn you respect and give you the greatest chance of resolving the issues quickly and effectively. (Panicked decisions often make things worse.)

A thought that might help calm you is that you never face a crisis alone. In fact, no one involved wants you to face it alone (seriously). You will have all the help you need—any reasonable request will be met. Stay calm, think things through, bring in the right folks to help (more on that shortly), and ask for whatever is necessary to resolve the issue. It’s remarkable how quickly and positively people respond to thoughtful requests in a crisis.

Why doesn’t he take him to the Oracle?

You never (and shouldn’t ever) face a crisis alone. Who should tackle it with you? Naturally, there are the stakeholders who care deeply about the outcome: your management chain plus impacted partners and customers. There are those responsible for handling the issue: you and your teammates. But the most crucial people to involve are the experts who can quickly diagnose problems and provide concrete steps to resolve them. How do you find these experts?

At first, everyone seems like an expert—they each have an opinion about what to do, including yourself. However, this is a crisis. Mistakes get compounded. Speculation is rampant. You need clarity, conviction, and confidence in your actions. You need to find the sources of problems and then fix them.

Instead of trying a bunch of solutions in parallel, start a bunch of investigations in parallel. If the issue is going to take hours or days to resolve, set up an online call and chat (a “bridge”). If the issue is going to take days or weeks to resolve, reserve a conference room and communications channel devoted to the issue and schedule frequent regular meetings until all problems are solved (a “war room”). Have your parallel investigations report results immediately to your bridge or war room.

Each investigation is seeking a potential root cause of the issue (there may be many). Each investigation has the authority to bring in whatever experts it needs immediately. Insist on collaborating with the sources of knowledge: the expert network admin who can track down that misconfigured router; the cybersecurity expert who can recognize that pattern of attack on your system; or the expert attorney who can interpret that section of the agreement.

You can tell the speculators from the experts by what they say. Speculators say, “It could be this or that. I’m not sure.” Experts say, “It could be this or that. Try this. If it doesn’t work, we know it’s that.” They are confident and take control. Experts are invaluable. Follow their instructions, let them work, and be generous in your thanks and recognition when the issue is resolved.

Write this down

Clear, written communication is crucial during a crisis. Stakeholders typically can’t all be on the bridge or attend all the war room meetings, yet they must be kept up to date. In addition, there will likely be future audits and/or analyses to document what went wrong, what the root causes were, and what was done to immediately mitigate the crisis and later fully resolve it. Writing down what’s happening as the situation progresses is essential to capturing and communicating this information. These records include bug reports, incident reports, root cause analyses (RCAs), and status updates.

If you haven’t been through an audit before, it can feel intimidating. An audit is a formal process performed by dedicated professionals. Their reports can be sent all the way up to the board of directors. You might be worried about how you and your work will be judged. Relax. As was true with the crisis, accepting the situation and responding well to it will garner you the most appreciation and respect. Auditors are nice individuals who are good at their jobs. If you are open, honest, and transparent, you will make their jobs easier, and they, in turn, will ensure you are represented fairly and respectfully.

Writing documentation and collaborating with experts can be stressful, but it’s also remarkably educational. Real crises aren’t that common, so remind yourself that you’re gaining rare and valuable experience.

The truth will set you free

Emotions can run high during a crisis. If you or your team contributed to the crisis, or you are worried about being blamed for it, you might be tempted to conceal prejudicial information or deemphasize unflattering details. That’s a mistake. A real crisis will get plenty of attention. The information will come out. You can provide the information as a responsible professional with high integrity, or others can uncover it despite you being a bad actor (possibly facing disciplinary action). Choose to be a responsible professional.

Likewise, you want to be upfront about what you and your coworkers did well. There’s no need to embellish. State the facts, good and bad, as you understand them, and provide proper credit to those who performed admirably, even if that includes yourself.

It’s gonna be a bright, sunshiny day

A broad service outage, security breach, highly publicized bug, or any other crisis that draws the attention of executives and, potentially, mass media can be stressful and unsettling, as well as a remarkable learning opportunity.

Stay calm and think clearly, knowing that many others will assist you in facing the situation. Avoid speculating on what went wrong or what you should do, regardless of how many suggestions you receive. Instead, initiate parallel investigations into potential root causes of the problem (there may be many). Set up a bridge or war room to coordinate and track progress, and regularly provide written status reports to stakeholders. Ensure each investigation has the authority to bring in the experts needed: people who are the source of knowledge about the problem and can discern the true cause. Once the crisis is resolved, be open, honest, and transparent about the good and bad of what happened, providing generous thanks and recognition for all those who helped resolve the issue.

As scary as crises can be, they all pass, leaving you with compelling stories to tell your friends and family. Use each one to hone your skills and grow yourself and your teammates. Take pride in your professionalism, transparency, and calm thoughtful response. The hero of the day could be you.