One customer, in particular, was credited as the cause of the recent major internet outage that led to the crashing of a group of well-known websites this past week, as reported by the cloud service that was at the root of the issue.
As reported by the Daily Wire, “The problems were reportedly due to issues experienced by the cloud computing provider Fastly, which identified concerns within its global content delivery network (CDN) and was in the process of implementing a fix.”
“We identified a service configuration that triggered disruptions across our POPs globally and have disabled that configuration,” Stated Fastly in an announcement recently released.
“The issues began at around 11am BST and lasted for an hour,” reported the BBC. Other websites impacted included The New York Times, PayPal, CNN, Twitch, Hulu, Vimeo, BBC.com, Shopify, the Financial Times, and The Guardian, along with certain features of other applications such as Twitter’s emoji services.
Nick Rockwell, Fastly’s Senior Vice President of Engineering, attempted to explain what had happened in a blog post that was titled, “Summary of June 8 outage.”
“This outage was broad and severe, and we’re truly sorry for the impact to our customers and everyone who relies on them,” Rockwell stated, before issuing the claim that the crash was caused by a single customer’s choices.
“On May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances,” stated Rockwell. “Early June 8, a customer pushed a valid configuration change that included the specific circumstances that triggered the bug, which caused 85% of our network to return errors.”
Once he had laid out a timeline of the events of the outage, between “Initial onset of global disruption” at 09:47 UTC and “Bug fix deployment began” at 17:25 UTC, Rockwell continued on.
“Once the immediate effects were mitigated, we turned our attention to fixing the bug and communicating with our customers. We created a permanent fix for the bug and began deploying it at 17:25,” he continued.
Part of a list of short-term steps laid out by Fastly was a “complete post mortem of the processes and practices we followed during this incident” in order to “figure out why we didn’t detect the bug during our software quality assurance and testing processes,” and to “evaluate ways to improve our remediation time.”
“Even though there were specific conditions that triggered this outage, we should have anticipated it. We provide mission critical services, and we treat any action that can cause service issues with the utmost sensitivity and priority. We apologize to our customers and those who rely on them for the outage and sincerely thank the community for its support,” concluded Rockwell in the blog post.