How to run a fire drill

Fire drills help align your incident response team and prepare them for real incidents. We recommend running a fire drill to gain familiarity with the FireHydrant platform. We also recommend running fire drills often (once per month or once per quarter) to refresh your team's skills .

Here are some best practices for a successful fire drill:

1. Create a low-stress environment for the drill.
2. Focus on the incident management process and workflows.
3. Encourage participation!

Six tips for running a fire drill with FireHydrant

1. Put a plan in place

For this first fire drill, it's probably best to have a plan in place and to put it on the calendar. This fire drill is really an opportunity for your team to get more familiar with the FireHydrant platform. For future fire drills, you may consider a more unannounced approach.

Make sure your FireHydrant runbook(s) are good to go. Review them to ensure all of the steps you want are in place but don't worry about perfection. Chances are you'll learn things from your first fire drill and make some changes to your runbooks.

2. Create a scenario

We recommend taking a recent incident your team worked through and recreating that scenario for your fire drill. Set the stage briefly so you have a situation to base your fire drill on.

3. Declare an incident

Now that the stage is set, declare an incident in FireHydrant. There are a number of ways you can do this depending on how you've configured things. The simplest way would be to enter /firehydrant new into Slack. Use the "game day" severity to signify this incident is a fire drill.

Tip: We support the following aliases for all slash commands:
/firehydrant
/fh
/incident

As you work through the incident, star important messages in Slack so that they're automatically captured in the timeline and the retrospective.

Here are some common Slack commands you'll find useful during an incident:

Add note /firehydrant add note [note text] Think of notes as a piece of information more important than a normal chat message. You are able to filter specifically for notes on the FireHydrant incident timeline. Add impact /firehydrant add impact This will open a text box that you can type in to start filtering through all available services, environments, and functionalities in your application. Update incident /firehydrant update The update command will allow you to change the milestone, component status conditions, and include a comment as to why you are changing the status. If you have active status pages you can choose to post your update to the status pages as well. Assign teams /firehydrant assign team Assigning a team will automatically assign roles to predefined users or on-call schedules. If the users have linked their Slack account FireHydrant will also send that user a DM on Slack informing them that they have been assigned a role to an active incident. Insert artifacts (graphs, screenshots, etc.) You can star any messages or images in Slack and they'll appear in your retrospective.

Tip: Bookmark our Slack commands cheat sheet!

4. Resolve the incident

Once the incident is deemed "resolved", resolve it in FireHydrant. The easiest way to do that is by entering /firehydrant resolve into the Slack channel. This will close an incident and set its status to resolved. A resolved incident will continue to collect chat messages.

5. Run a retrospective

Now that the incident is resolved, it's time to run a retrospective! For the most part, you can run your incidents entirely out of Slack, however, retrospectives need to be done inside the FireHydrant UI. Learn more about running retrospectives in FireHydrant here.

Tip: you can re-order and modify the retrospective questions.

6. Review your analytics

Once you've completed the retrospective, head over to the analytics section of the FireHydrant UI to see what types of metrics we track. Learn more about analytics in FireHydrant here.