Processing delay affecting 5% of items
Resolved

One of the shards of our processing pipeline was stalled for about 35 minutes, causing processing and notifications to be paused for about 5% of items. We've cleared the stall and events are flowing through. The pipeline has caught up and is fully operational.

Additionally, we've found and corrected the monitoring misconfiguration that caused us to not notice this sooner.

Tue, Oct 17, 2017, 01:29 AM
(7 years ago)
·
Affected components

No components marked as affected

Updates

Resolved

One of the shards of our processing pipeline was stalled for about 35 minutes, causing processing and notifications to be paused for about 5% of items. We've cleared the stall and events are flowing through. The pipeline has caught up and is fully operational.

Additionally, we've found and corrected the monitoring misconfiguration that caused us to not notice this sooner.

Tue, Oct 17, 2017, 01:29 AM