e This incident has been resolved: the affected components (the Occurrences counts on the Account-Level Dashboard, Project Level dashboard, and Reports API) are again at near-real-time.
As noted earlier: as part of the fix, we have implemented a performance optimization that, in rare cases, may cause occurrences to be counted as the framework/level at time of processing instead of the framework/level at time of occurrence. This can affect 1) items whose levels are changed (i.e., manually or upon reactivation) and 2) merged items where the framework changes as a result of the merge (technically possible but very rare). For occurrences of such items, if the level/framework of the item changes between time of occurrence and time the occurrence is processed by the internal service that prepares the dashboard summary data, then the level/framework reported in these summaries will be the value at time of processing rather than the value at time of occurrence.
Resolved
e This incident has been resolved: the affected components (the Occurrences counts on the Account-Level Dashboard, Project Level dashboard, and Reports API) are again at near-real-time.
As noted earlier: as part of the fix, we have implemented a performance optimization that, in rare cases, may cause occurrences to be counted as the framework/level at time of processing instead of the framework/level at time of occurrence. This can affect 1) items whose levels are changed (i.e., manually or upon reactivation) and 2) merged items where the framework changes as a result of the merge (technically possible but very rare). For occurrences of such items, if the level/framework of the item changes between time of occurrence and time the occurrence is processed by the internal service that prepares the dashboard summary data, then the level/framework reported in these summaries will be the value at time of processing rather than the value at time of occurrence.
Monitoring
We've implemented a fix for this issue. We expect the affected components (the Occurrences counts on the Account-Level Dashboard, Project Level dashboard, and Reports API) recover to near-real-time within about 45 minutes.
Note: as part of the fix, we have implemented a performance optimization that in rare cases may cause occurrences to be counted as the framework/level at time of processing instead of the framework/level at time of occurrence. This can affect 1) items whose levels are changed (i.e. manually or upon reactivation) and 2) merged items where the framework changes as a result of the merge (technically possible but very rare). For occurrences of such items, if the level/framework of the item changes between time of occurrence and time the occurrence is processed by the internal service that prepares the dashboard summary data, then the level/framework reported in these summaries will be value at time of processing rather than the value at time of occurrence.
Investigating
Account level dashboard and project dashboards are still showing data that might be outdated by at most 6 hours
Investigating
There is a processing delay in parts of the pipeline. Account level dashboards, project dashboards pages may show data that is outdated by a few hours.