Enable paging for GA

[BigLep]

Done Criteria

1. The FOC WG has a paging tool they're onboarded with
2. sev2.x alerts cut a page (rather than relying exclusively on slack)

Why Important

When we're GA, it's important that we treat sev2.x events as higher priority than everything else and requiring immediately engagement.

Notes

1. We deferred this for M4.1 as we work on baking in our alarms / thresholds.
2. We need to investigate/decide between using BetterStack vs. FF PagerDuty etc.
3. We should ideally be timezone sensitive. Lets page people who are most likely already awake.
4. Ideally we have a way to differentiate between sev2 (page instantly) and sev2.5 (page during business hours)

[rjan90]

We need to investigate/decide between using BetterStack vs. FF PagerDuty etc.

Based on initial investigation Betterstack pricing seems expensive. $39/month per responder. That means we would very quickly end up in the $500–$700/month range once we onboard the relevant people that should be able to be paged for FOC.

I will investigate how much it is per responder in FF PagerDuty, or if they have a special pricing over there for XYZ-amount of people.

[rjan90]

We have landed on using PagerDuty for now / and leverage what FF already is paying for. For GA I do not plan to hook up alerts just yet, because I would like to see some stability in alerts (and them not misfiring too much), before enabling automatic page.

That said, I will create a connection to the #fs-wg-alerts channel, so its easy to page from Slack in the event of a real 🔥