Post-mortem on yesterday’s (false) malware report – and a request for Google

by John Ramey on January 16, 2013

Around 8:20am Pacific on Tuesday January 15th, Google’s automated threat system wrongly identified isocket as serving malware. It impacted about 10-15% of isocket’s publishers, including TechCrunch and dozens of other legitimate, prominent, innocent websites.

First and foremost: we’re sorry. isocket is made up of people who work really hard to delight customers and make advertising suck a little less. Regardless of what happened (even if we’re 100% innocent), it’s the biggest punch in the gut we’ve ever felt (personally, the worst day of my professional life) and we’re sorry to see people upset. People we care about were hurt and that sucks.

Put simply: we did not deliver malware on the range of sites blocked by Google.

In summary:

  1. Google flagged isocket’s ad server as malware, which put up the scary red “Warning!” blocker for a lot of websites.
  2. Google manually removed their automated “isocket is bad” warning around 9am PT, about an hour after it triggered. They wouldn’t have done that if there was actually a problem.
  3. We see zero evidence that isocket actually did/delivered anything bad – no malware, no human reports of problems, no traces of a hack or security breach, etc.
  4. Third party threat detection services could find nothing wrong with isocket’s system or the ads we were serving either.
  5. So we believe Google committed a false positive, incorrectly labeling isocket as bad. Then, Google wrongly applied that warning to lots of websites through “guilt by association.”
  6. There is no rhyme or reason why Google blocked the sites that it did, as far as we can tell. No common ad campaigns across the websites affected, etc etc.
  7. It’s very slow at best and impossible at worst to get Google to fix these problems. As a result, people were down for hours and there wasn’t anything we could do to fix it.
  8. We’re asking Google to improve this system, so that even when false positives happen they are properly contained and don’t create such massive unnecessary ripple effects.
  9. isocket will do whatever is possible to prevent this from happening again, even though we can’t identify specific things that caused this. Even if we’re “innocent” in this event, it’s a kick in the ass to make things even more secure – because it’s unacceptable that good publishers are punished for no reason.

What we think happened – a false positive warning, grossly misapplied by Google.

Google continually crawls the web and looks for bad stuff, like malware. Advertising is often a vehicle for bad stuff. When Google detects a threat it automatically tries to quarantine the site and protect the users, usually via the big scary red warning page. Web browsers like Chrome, Firefox, and Safari use this service provided by Google to know when to block things at the browser level.

The code put in the HTML of websites that make online ad services work are often daisy chained, so that when Network 1 doesn’t have an ad to show, it hands the ad space over to Network 2, etc. Many isocket publishers run Google’s AdSense and other networks as remnant, and sometimes they put that 3rd party ad network code into isocket’s ad server so that we can do the daisy chaining (i.e. if we don’t have a premium ad to show, punt it to the 3rd party.)

While isocket can control and protect the ads through our ad server, when the impression is handed off to a 3rd party, we (and the publisher) lose that control. It’s a common problem in the industry. It’s basically an open back door that you can’t control what comes through.

We have zero evidence so far that isocket actually served something bad – either directly through an advertisement bought/hosted through us, or indirectly through these 3rd party networks. No one has been able to produce a report of someone being infected, or to even point to whatever it is that Google didn’t like.

So we’ve been doing our own scans to find what, if any, malware came through. Both our internal scans and those done by professional 3rd party security/verification services have come back with zero problems.

There’s also zero evidence that our systems were hacked or compromised in any way. When ad servers get hacked, there are signs and breadcrumbs. We found none, and we looked through everything including the source code of our ad server. There are additional (pretty severe) security measures we have in place that we don’t disclose publicly, but none of them were compromised.

There are three theories about why Google first created a threat warning:

  1. Somewhere, on one of the isocket enabled websites, Google falsely identified an advertisement served by isocket as bad. There’s a lot of reasons why a false positive could happen. Google looks for certain “evil” things, but sometimes they can seem evil but really be innocent.
  2. isocket did serve something bad on a specific website, such as an ad with a clickthrough URL that was a phishy website. But Google messed up by applying their threat judgement across our entire isocket marketplace, instead of just limiting the warning to the website in question.
  3. On some specific website, we handed the ad call off to a 3rd party remnant ad network daisy chained through us, and that 3rd party served something Google didn’t like, but then Google erred on the side of broad-caution and inappropriately blamed isocket’s ad server as well.

Then, Google applied that warning way too broadly.

It’s bad enough that a false positive warning occurred – but then, instead of limiting the warning to just the website that showed the ad that Google didn’t like, Google applied the warning to all of isocket’s ad server. So that any website that was running isocket could be guilty by association.

Basically, double whammy. In no way did isocket serve malware on the range of sites Google blocked. There was no common pattern, no shared ads run across the sites blocked, no rhyme or reason to it, nothing.

Google reversed their “isocket is bad” decision within an hour of it occurring. But because their system had applied the consequences across a large set of publishers, it then took their system a long time to propagate through the web and unblock all the innocent properties affected due to the broad sword ripple effects.

This “internet police” system must work better.

I love Google, Chrome, and the fact they try to make the web better by helping prevent spam and malware. But when you are in a position to police the web, where people so widely depend upon your word and assume it’s infallible, where one of your algorithms can literally destroy legitimate businesses, you have the moral obligation to do it better.

The first problem: threat detection systems are not perfect.

In the likely event that this was a false positive, or that the true positive happened from another 3rd party source, the method that Google uses to detect problems is not fool proof. It can make mistakes. And we’ve learned and seen first hand that there are often contradictions or inconsistencies in these systems.

The second problem: Google is quick to punish, but too slow to fix mistakes. 

Everything that happens on Google’s end is automated. And while it’s very quick to bring the hammer down (with no warning), the ability to fix mistakes is almost nonexistent. There is no easy remedy. No one you can contact. It’s slow, frustrating, painful, and vague.

So when we found ourselves in this situation, the best we could do is help publishers remove isocket from their webpages and submit their URL to an automated “re-test” machine from Google. It was only later in the day we found out Google had manually reversed the isocket warning within an hour of it occurring.

Many people asked if we had a red nuclear emergency phone to call into Google, assuming there was someone we could call to fix this. Unfortunately there isn’t. It’s a little bit of a black box in that regard.

I’m sure this is a relatively unusual situation for Google. They aren’t trying to hurt websites, and there aren’t a ton of products like isocket out there (so maybe their system was designed for apples but isocket is oranges). But while everyone can make mistakes and that’s OK, when you act as the policeman for the internet it comes with a higher level of responsibility. Good people were hurt, the punishment was unfair, and we all need to do better to prevent that. We will do anything we can to help Google improve in that regard.

What isocket will do from here.

In a weird way, I almost wish this was something we did wrong.  Because then we could own it, directly fix it, and take the punch on the chin. But since so far we can’t identify something specific to fix, that makes it harder to articulate a plan. Regardless, it’s a kick in the ass to always be vigilant for ways to prevent these situations, even if it’s to try and protect our publishers from the wrath of third parties. And we will do whatever we can to work with Google on maturing these systems. If we identify specific problems to fix inside isocket or opportunities to be proactively better, we will be transparent about those plans so we can continue to earn your trust and business.

Please contact us if you have any questions or concerns. We’re sorry, we love you, we love Google, and we love ads. Except when ads break the internet. Then we don’t love ads so much ;)

{ 5 comments… read them below or add one }

Leave a Comment

{ 2 trackbacks }

Previous post:

Next post: