e This incident has been resolved: the affected components (the Occurrences counts on the Account-Level Dashboard, Project Level dashboard, and Reports API) are again at near-real-time.
As noted earlier: as part of the fix, we have implemented a performance optimization that, in rare cases, may cause occurrences to be counted as the framework/level at time of processing instead of the framework/level at time of occurrence. This can affect 1) items whose levels are changed (i.e., manually or upon reactivation) and 2) merged items where the framework changes as a result of the merge (technically possible but very rare). For occurrences of such items, if the level/framework of the item changes between time of occurrence and time the occurrence is processed by the internal service that prepares the dashboard summary data, then the level/framework reported in these summaries will be the value at time of processing rather than the value at time of occurrence.
Posted May 20, 2020 - 23:41 PDT
Monitoring
We've implemented a fix for this issue. We expect the affected components (the Occurrences counts on the Account-Level Dashboard, Project Level dashboard, and Reports API) recover to near-real-time within about 45 minutes.
Note: as part of the fix, we have implemented a performance optimization that in rare cases may cause occurrences to be counted as the framework/level at time of processing instead of the framework/level at time of occurrence. This can affect 1) items whose levels are changed (i.e. manually or upon reactivation) and 2) merged items where the framework changes as a result of the merge (technically possible but very rare). For occurrences of such items, if the level/framework of the item changes between time of occurrence and time the occurrence is processed by the internal service that prepares the dashboard summary data, then the level/framework reported in these summaries will be value at time of processing rather than the value at time of occurrence.
Posted May 20, 2020 - 22:49 PDT
Update
Account level dashboard and project dashboards are still showing data that might be outdated by at most 6 hours
Posted May 20, 2020 - 04:42 PDT
Investigating
There is a processing delay in parts of the pipeline. Account level dashboards, project dashboards pages may show data that is outdated by a few hours.
Posted May 19, 2020 - 17:59 PDT
This incident affected: Processing pipeline (Core Processing Pipeline).