Quick incident summary for this morning's processing delay:
Delay began at approximately 8:00:00am PDT
The delay peaked at 141 seconds at approx 8:06:50am PDT
The delay then slowly dropped and cleared at approx 8:20:40am PDT
The immediate cause appears to be high load on the database during this time. In the short term, we're investigating how to move this workload so that it doesn't affect the processing pipeline.
Posted Sep 13, 2017 - 08:27 PDT
Resolved
Processing delay has returned to normal levels (approx 1 second).
Posted Sep 13, 2017 - 08:23 PDT
Monitoring
Processing delay is at 47 seconds; taking a bit longer than expected to return to normal levels but still continuing to clear. We're watching the system closely to understand the bottleneck.
Posted Sep 13, 2017 - 08:17 PDT
Identified
Processing is currently delayed about 2 minutes. The delay is clearing and we expect to clear in about 2 more minutes.