Postmortem -
Read details
Jun 24, 14:07 PDT
Resolved -
The pipeline is fully recovered and the system is stable. We'll do a postmortem on this incident in the near future.
Jun 19, 19:59 PDT
Update -
The pipeline pause is complete and data is being processed. We expect to be back to real-time within a few minutes.
Jun 19, 19:53 PDT
Update -
We're starting a ~15-minute pause in the processing pipeline now. Once the pause is complete, data received during the pause will be processed.
Jun 19, 19:42 PDT
Update -
We're beginning the maintenance; in approx 10 minutes, the processing pipeline will pause for approximately 15 minutes. Once the pause is complete, data received during the pause will be processed.
Jun 19, 19:26 PDT
Update -
We'll begin the maintenance in approximately 30 minutes. The maintenance will result in a ~15-minute pause in the processing pipeline. We'll post another update when the pause is about to begin.
Jun 19, 19:07 PDT
Update -
We've completed processing the backlog of data collected during the pipeline outage.
The health of one of our DB clusters is currently degraded such that we'll need to put a new cluster into service later this evening. This will require another brief pause of the processing pipeline. We'll keep you update as we prepare for that cut-over.
Jun 19, 18:17 PDT
Monitoring -
During the outage the pipeline fell approximately an hour behind. We're once again processing NEW data coming into pipeline in near-real time and we're simultaneously processing the backlog of data we collected during pipeline outage.
Jun 19, 17:13 PDT
Identified -
We've identified and resolved the issue that caused the pipeline failure. The system is recovering. We'll post ETA until recovery and processing delay data as we have them. We had an extended outage because of a failure in our monitoring system and we'll be performing a complete post-mortem as soon as possible.
Jun 19, 16:07 PDT
Investigating -
We're receiving data via the API, but the processing pipeline is down. We're investigating.
Jun 19, 15:44 PDT
