On February 23, CloudFlare announced the company had spent about a week dealing with a bug that leaked sensitive information into the net. The company explained they solved the situation, but a lot of things happened before they could make that claim. Things that reflect how the tech industry works.
In short, CloudFlare (CF) provides a broad range of online services including domain names, firewall security, reverse proxy among others. According to 2016 numbers, the organization has around 30% of all domain names. So, their job affects millions of people.
The company has been using Ragel parser to offer some of its services, but the programmers realized they could write a better piece of software. They started doing so a year ago and named it cf-HTML.
Last year, they began to implement it, but a mistake in the code created a bug in Ragel that resulted in a leak of information that could lead to people’s identities (PII) among many other things. The event was dubbed “Cloudbleed” because of a similar leakage called “Heartbleed.”
Now, this story has two main sources: CoudFlare, the affected, and Tavis Ormandy, the hero who discovered the problem. Cloudbleed exposes how good these companies are at what they do, and the shady nature of some of their procedures.
The good side of the first big security problem of 2017
Tavis Ormandy is one of the many tech experts that works with Google. He is part of a team that checks the Internet for irregularities such as DDoS attacks.
On February 17, he realized the search engine had picked up a weird pattern of information. He and the people who were at the office got interested and started looking at it. After a little while, they realized it was leaked information and that the source was CloudFlare.
They saw passwords, private messages from popular date sites and chat apps, cookies and more. So, they were dealing with an emergency. Tavis assertedly recurred to Twitter and asked for a contact from CloudFlare’s security team.
Once they got in touch, Google handled CloudFlare’s team a detailed bug report, and it was as bad as it seemed. The company didn’t lose a second and summoned the right people to deal with the situation. Within hours, they had pinpointed and fixed the bug. However, different search engines had already cached many of the data.
The bug compromised protocols that inserted a “kill-switch” to the information in case something like this happened. However, there was an old software that did not have the countermeasure and was affected by it.
Most of the information was automatically erased, but that wasn’t enough. So, CF teamed up with Google, Bing, Yahoo, and similar companies to start hunting for the leaked information.
As Tavis explained, millions of users had downloaded part of the data, but they did not understand what was in front of them. Everybody got busy and found 770 unique URls and 161 domains containing personal information.
In the end, CloudFlare and company saved the day, and preemptive features were set in place. According to the company, the leak did not contain SSL Security Keys, and as far as everybody knows, no malicious third parties exploited the bug.
The ugly implications of Cloudbleed
In the beginning, Tavis and the CloudFlare’s security team were working as well-oiled machine dealing with the problem quickly and efficiently. However, things started going south when the next logical step became imminent.
When a security breach like this happens, companies have to come clean to the users. Customers need to know their information has been compromised so that they can take the necessary measures. Of course, that represents a PR nightmare as the guys at Yahoo can tell.
In this kind of cases, there is a time when the parties involved cannot disclose the issue. If black hat hackers realize there is a bunch of data out there in the wild, the already bad situation can turn into an apocalyptic disaster.
The days passed, and Tavis started asking CloudFlare when they were going to make the issue public, but the organization did everything in its power to delay the announcement including feeding collaborators with confusing information.
Mr. Ormandy used the Chromium blog to keep tabs on the progress. He praised the dedication of CloudFlare’s team, but a couple of days later, he questioned why the staff was withholding the information from the users.
There was some back and forth between CloudFlare’s team and Tavis, but in the end, the company ended up releasing a detailed technical explanation on its official website. However, as Google’s employee points out, the press release didn’t explain how bad the problem is. Moreover, the writer used technical language that most readers wouldn’t understand without a good investigation.
The conclusion
Cloudbleed was really bad. During the bug hunt, researchers found whole private messages, passwords, and even CloudFlare Authentication protocols that could have compromised further data.
In fact, the security team said they did not know how long the bug had been running by the time they addressed it. So, time will tell the real extent of the damaged caused by the leakage.
On top of that, it seems CF doesn’t take this kind of problems very seriously. The team rewarded the collaborators, who were cleaning up after the huge mess they did, with a T-shirt bearing the company’s logo. Not really motivating.
In CF’s defense, if they chose their words carelessly, it could have incited unnecessary panic among users. However, as Adrian Thurston points out, that can’t stop news outlets (He only mentions The Verge) from misunderstanding the press release and say Ragel parser was at fault, and that was NOT the case.
Mr. Thurston explains that “legacy code” needs to a lot of testing before implementing any change, something that the guys at CF should have considered. Nevertheless, Cloudbleed is not a reason to stop using the service.
Similar problems happen more often than what the average user thinks, which is why there is such a vast infrastructure in place to deal with them.
Deviating from the topic, many Gmail users have reported their mobile app showed a signed off notification on their devices, myself included, but Tavis explained it had nothing to do with Cloudbleed.
Disclaimer: This article intends to explain the Cloudbleed event in an easy way. For a more technical explanation, please read both the CF press release and Tavis Ormandy’s blog posts as there are too many angles to approach this incident.
Sources: CloudFlare / Chromium Blog Post