This incident was resolved. Rollbar processing pipeline latency (measured at p99) was elevated above our 30-second p99 SLO today during the following windows:
02:02AM - 02:35AM PDT: 30s - 7.0 minutes latency
06:02AM - 07:03AM PDT: 30s - 3.9 minutes latency
08:00AM - 09:02AM PDT: 30s - 37.2 minutes latency
09:02AM - 09:52AM PDT: 30s - 23.1 minutes latency
p50 stayed at SLO (~5s) during the entire time period, and no data was lost. The initial cause analysis is a particular traffic pattern. We're investigating the root cause to ensure the system is resilient to this traffic pattern going forward.
Resolved
This incident was resolved. Rollbar processing pipeline latency (measured at p99) was elevated above our 30-second p99 SLO today during the following windows:
02:02AM - 02:35AM PDT: 30s - 7.0 minutes latency
06:02AM - 07:03AM PDT: 30s - 3.9 minutes latency
08:00AM - 09:02AM PDT: 30s - 37.2 minutes latency
09:02AM - 09:52AM PDT: 30s - 23.1 minutes latency
p50 stayed at SLO (~5s) during the entire time period, and no data was lost. The initial cause analysis is a particular traffic pattern. We're investigating the root cause to ensure the system is resilient to this traffic pattern going forward.
Investigating
Rollbar processing pipeline latency (measured at p99) was elevated above our 30-second p99 SLO today beginning at 02:02AM PDT.
