Use custom log filter to send warning logs and above to slack

[bram-vdberg]

This PR updates the logging to send warning logs and above to slack for monitoring.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[bram-vdberg]

I have read the CLA Document and I hereby sign the CLA

[fhenneke]

Should that function be used e.g. in error and critical (and warning)?

[bram-vdberg]

Yes it should, sorry about that.

[fhenneke]

Is this not sent to stderr? Or did the config of which messages appear on slack change somehow?

[bram-vdberg]

Yes, we changed the config so that doesn't matter anymore. But that also means that warning, error, and critical logs are no longer sent to slack automatically.

[fhenneke]

I think it is a good idea to use channel names instead of channel ids 👍
We do the latter in other repos but it is always a pain to reconstruct where messages are sent.

[fhenneke]

One alternative would be the following, which uses python loggers directly.

class SlackHandler(logging.Handler):
    def __init__(self, name):
        super().__init__(name)
        self.slack_client = WebClient(token=os.environ["SLACK_BOT_TOKEN"])

    def emit(self, record):
        if record.level >= logging.WARNING:
            self._post_to_slack(record.msg)

    def _post_to_slack(self, msg):
        """Post log to slack"""
        self.slack_client.chat_postMessage(
            channel=os.environ.get("SLACK_CHANNEL", "#alerts-ebbo"), text=msg
        )

def get_logger(filename: Optional[str] = None) -> logging.Logger:
    ...
    if "SLACK_BOT_TOKEN" in os.environ:
        std_handler = StreamHandler()
        ...
        slack_handler = SlackHandler()
        ...
        self._logger.addHandler(std_handler)
        self._logger.addHandler(slack_handler)