Part Two: The product · Trouble and change
Admitting a failure feels risky. Silence and blame cost more, because customers judge a company by how it behaves when it fails.
Every product fails sometimes. The server goes down, a bad release ships, a bill is sent twice. What happens in the next hour decides what the failure costs.
The instinct inside a company is to say little. Legal worries about liability. Marketing worries about the headline. Engineers are busy and do not yet know the cause. So the status page stays green, support is told to say "we are looking into it," and the eventual statement speaks of "an issue affecting a small number of customers."
Meanwhile the customer is doing their own investigation. Is it me? Is it my network? Did I lose anything? Should I tell my own customers? The company knows the answer to each of those questions and is not giving it. People fill the silence with the worst guess, and they are right to, because a company that will not say what happened might not say when it is worse.
Plain speaking works the other way. In January 2017 GitLab lost production data after an engineer deleted the wrong database directory during an incident. The company said so within hours, kept a public running document of the recovery, and afterward published a full account of what went wrong, including the backups that had not been working. Customers were angry about the loss and still respected the telling.
What a company says in a crisis is usually decided above the people who build the product. What they can do is have the words ready, so that telling the truth is also the quickest thing to do.
Therefore:
When the product fails, say so quickly and plainly: what happened, who is affected, and what people should do meanwhile. Afterward, explain the cause and what you changed. Never blame the customer.
In practice. Write the message for your most likely failure now, while nothing is on fire. Settle today who is allowed to send it without asking permission.
Goes with. The status you show during an incident must follow HONEST STATUS (82). People in real trouble need A HUMAN WHEN IT MATTERS (52). The less a failure can destroy, the less there is to confess, so NOTHING IS LOST (55).
Source: GitLab, "Postmortem of database outage of January 31" (2017).