Rollbar values transparency and understands that our growth is dependent on our ability to honestly reflect on where we can do better.

Between December 16th and December 18th, we experienced a system-wide degradation. As a result, error occurrences sent to us over a span of 48 hours from December 16th to December 18th were processed but with significant delays.

The incident resulted in knock-on effects that impacted our ability to deliver the best real-time experience. For the duration of the incident, even though the majority of errors sent to us appeared in the web app in real time, notifications were significantly delayed and some aspects of the UI were affected \(such as occurrence counts\).

Around 11:30 AM PT on December 16th, an unexpected interaction between our data retention deletion system and the rest of our pipeline caused one partition of a job queue to begin building up.

As soon as we detected the issue, we began our incident response process. Within 24 hours, on December 17th we isolated the issue and released a fix.

Due to the large queue of jobs to process, this placed major strain on the rest of our system since we received historically high traffic on top of the backlog of jobs to process. This meant that we did not finish processing the entire backlog until 11 AM PT on December 18th.

We will review and update our incident response process with the learnings from this event, no later than Jan 31 2022, and we will fold the learnings from this event into our upcoming infrastructure scaling initiatives.

Thank you for being a Rollbar customer, and happy holidays!

This incident has been resolved.

We have finished processing the pipeline. All operations are back to normal.

Thank you for your patience during this incident. Please expect a postmortem during the next week.

We expect the entire backlog of events to be processed within the next 13 hours (by 6 PM Pacific). At that time, we will consider this incident resolved.

We are seeing that some customers are not getting alerted about errors in real time. While we investigate the issue, please know that you can still see errors appear in real time on the web app.

Our maintenance is complete. The pipeline is once again processing new errors sent to our API in real time.

Regarding the processing of our backlog from the past two days – we estimate that it could take a few more days to process the entire backlog. What this means is that while new errors sent to us are processed in real time, some of the notifications you will receive over the next few days will be notifications about errors you sent us up to two days ago.

Additionally, we would like to note that realtime update functionality on the web app is affected. This does not mean new items are not available. Instead, you may have to refresh the page manually to see them.

We apologize for the continued effects on your workflows.

This update replaces the planned update at 3 PM PT. You can expect another update tomorrow.

Due to additional required maintenance there is still a delay in the processing pipeline but work is ongoing to resolve with minimal interruptions. A further update will be made at 3pm PT.

The backlog of items are still being processed, a further update to the status will be posted at midday PT.

The entire backlog of items to process will take up to 12 further hours. During this time, pipeline performance will be somewhat affected though most items continue to be processed in real time.

Please expect a further update no later than 5 AM Pacific. We will of course update sooner if there is pertinent new information that we need to communicate with you. Thank you.

We have resolved the recent issue with our pipeline. There was no data loss and our API layer was not affected.

We are continuing to investigate this issue.

Our pipeline has hit an issue while processing the backlog of items. We are investigating.

We are now processing item reactivations and updating certain metadata like occurrences per item.

It will take some hours to process the entire backlog of items.

Instead of posting an update at 1 PM PT, we will post another update at 3 PM PT.

We have identified the issue and have implemented a fix. We are monitoring the results. Expect another update at 1 PM PT.

We are now validating our fix and will be releasing it in the next hour. We will post another update at 1 PM PT.

We are continuing to work to fix the issue. We will provide another update by 12:00 PM Pacific.

We are continuing to investigate the issue. We are experiencing an issue with Kafka caused by yesterday's load, and we are preparing for a fix.