Web outage and processing pipeline delay

Incident Report for Rollbar

Resolved

The delay in the processing pipeline has resolved.

Posted May 29, 2020 - 11:14 PDT

Monitoring

A fix has been implemented. The Processing Pipeline is working through backlog of events received during the outage; we now expect that backlog to be cleared about 2 hours, but prioritizing processing recent events.

Posted May 29, 2020 - 10:07 PDT

Identified

We have identified the issue. The Web App is operational, the Processing Pipeline is stopped while we implement a fix. Currently the real time notifications are delayed by 45 minutes.

Posted May 29, 2020 - 09:42 PDT

Investigating

We are experiencing web outage and delay in the processing pipeline. We are investigating the issue.

Posted May 29, 2020 - 09:27 PDT

This incident affected: Web App (rollbar.com) and Processing pipeline (Core Processing Pipeline).