We attempted to rollout DNSSEC as part of an effort to improve the security of our DNS records and secure our customer data. While our rollout in our testing environments succeeded, our use of multiple providers (both Amazon Route 53 and Google Cloud DNS) in production for redundancy caused issues. This inadvertently resulted in a DNS “split-brain” effect, with sporadic DNS failures until we rolled back DNSSEC. Unfortunately due to network caching some of our customers continued to experience issues long after the rollback of DNSSEC.
All Times in Pacific Standard Time
Tuesday, March 23:
Wednesday, March 25: