Web app, API, and pipeline outage
Incident Report for Rollbar
Postmortem

On Nov 25, 2021 from 11:08 AM PT to 11:29 AM PT our API, web app, and item processing pipeline were sporadically unavailable. Unfortunately, errors sent from 11:08 AM to 11:17 AM were not processed.

The root cause was an infrastructure change intended to better handle traffic spikes to our API. Unfortunately, this change caused us to briefly overload one of our application databases, causing the unavailability.

When we detected the database issues, we reverted the work and our services recovered.

To prevent this incident from recurring, we have paused the aforementioned infrastructure change work. Meanwhile, we are continuing work to scale our application databases. As mentioned in a previous postmortem, we have significantly increased resources for database work. The first database projects have been completed, but we have more projects planned for this quarter and next to continue to improve our databases.

As always, thank you for being a Rollbar customer.

Posted Dec 01, 2021 - 13:37 PST

Resolved
This incident has been resolved. Thank you for your patience. Please expect a post-mortem by Tuesday 5 PM PT.
Posted Nov 25, 2021 - 11:51 PST
Update
The web app and API are functional again. The pipeline is processing a backlog of items.
Posted Nov 25, 2021 - 11:34 PST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Nov 25, 2021 - 11:19 PST
Identified
An issue with one of our application databases has caused a web app, API, and pipeline outage. We will update when the database is available again.
Posted Nov 25, 2021 - 11:15 PST
This incident affected: Web App (rollbar.com), API Tier (api.rollbar.com) and Processing pipeline (iOS Symbolication pipeline, Core Processing Pipeline).