Alerting notifications delay

Incident Report for Kentik SaaS EMEA Cluster

Resolved

This incident has been resolved.

Posted Dec 04, 2020 - 02:15 UTC

Update

We've deployed more verbose logging in the platform and collecting diagnostic info. Kentik engineers will continue monitoring the situation closely until we find the root cause.

Posted Dec 03, 2020 - 15:02 UTC

Monitoring

We have introduced a more verbose logging / diagnostic solution to the alerting subsystem which required a restart of those services. In doing so, the delays appear to be temporarily resolved. We will continue to investigate the situation with the newly introduced diagnostics.

Posted Dec 03, 2020 - 12:35 UTC

Update

We are continuing to investigate this issue.

Posted Dec 03, 2020 - 12:01 UTC

Investigating

Alerting notifications are presently severely delayed (12+ hours)

Posted Dec 03, 2020 - 11:28 UTC

This incident affected: Alerting and Mitigation Services.