Increased occurrence processing pipeline latency
Resolved·Partial outage

We have finished processing the backlog of occurrences. Again, apologies for interruptions this caused you today. Please expect a post-mortem next week.

Thu, Nov 17, 2022, 10:57 PM
(2 years ago)
·
Affected components
Updates

Resolved

We have finished processing the backlog of occurrences. Again, apologies for interruptions this caused you today. Please expect a post-mortem next week.

Thu, Nov 17, 2022, 10:57 PM

Monitoring

We are continuing to monitor for any further issues.

Thu, Nov 17, 2022, 10:56 PM

Monitoring

We expect to finish processing the backlog of occurrences by 7 PM PT.

Thu, Nov 17, 2022, 06:58 PM(3 hours earlier)

Monitoring

We are continuing to drain the backlog of occurrences. During this time, you may see out-of-date timestamps in the UI, and it may take time for new items to show up.

We apologize for this outage. Please expect a postmortem next week.

Thu, Nov 17, 2022, 06:35 PM(23 minutes earlier)

Monitoring

We are now processing the backlog of occurrences.

Thu, Nov 17, 2022, 05:13 PM(1 hour earlier)

Identified

We are nearly done with the fix. We expect to begin processing occurrences in 15 minutes.

Thu, Nov 17, 2022, 04:59 PM(14 minutes earlier)

Identified

We are continuing to work to resolve the issue. We still expect to be done by 9 AM PT.

Thu, Nov 17, 2022, 03:56 PM(1 hour earlier)

Identified

We are currently unable to process incoming occurrences. That is, we are storing them – there is no data loss – but unable to fully process them yet. We are working on a remediation. We expect the issue to be resolved by 9 AM PT. Please expect the next update at 8 AM PT.

Thu, Nov 17, 2022, 03:10 PM(45 minutes earlier)

Identified

We are experiencing a processing delay in our occurrence pipeline. We are investigating actively and will post when we have identified the issue.

Thu, Nov 17, 2022, 02:30 PM(40 minutes earlier)