Degraded Ingest/Tagging

Incident Report for Kentik SaaS EMEA Cluster

Postmortem

ROOT CAUSE

Due to ingest capacity constraints, one of our ingest servers began to run out of resources, causing software restarts and load balancing issues.

RESOLUTION

Kentik Operations manually rebalanced our ingest layer to allow this server to fully come back online. We also provisioned some emergency compute resources to provide additional ingest capacity while new hardware is being brought online that will effectively double the ingest and storage capacity of this cluster (ETA 2022-10-14).

Posted Oct 07, 2022 - 16:27 UTC

Resolved

This incident has been resolved.

Posted Sep 28, 2022 - 03:13 UTC

Monitoring

A fix has been rolled out and we are monitoring the situation.

Posted Sep 28, 2022 - 02:39 UTC

Investigating

Flow ingest and tagging is currently degraded for a subset of customers. We are investigating and working quickly to resolve.

Posted Sep 28, 2022 - 01:03 UTC

This incident affected: Flow Ingest.