Clicky

Post-mortem on yesterday’s (false) malware report – and a request for Google

by John Ramey on January 16, 2013

Around 8:20am Pacific on Tuesday January 15th, Google’s automated threat system wrongly identified isocket as serving malware. It impacted about 10-15% of isocket’s publishers, including TechCrunch and dozens of other legitimate, prominent, innocent websites.

First and foremost: we’re sorry. isocket is made up of people who work really hard to delight customers and make advertising suck a little less. Regardless of what happened (even if we’re 100% innocent), it’s the biggest punch in the gut we’ve ever felt (personally, the worst day of my professional life) and we’re sorry to see people upset. People we care about were hurt and that sucks.

Put simply: we did not deliver malware on the range of sites blocked by Google.

In summary:

  1. Google flagged isocket’s ad server as malware, which put up the scary red “Warning!” blocker for a lot of websites.
  2. Google manually removed their automated “isocket is bad” warning around 9am PT, about an hour after it triggered. They wouldn’t have done that if there was actually a problem.
  3. We see zero evidence that isocket actually did/delivered anything bad – no malware, no human reports of problems, no traces of a hack or security breach, etc.
  4. Third party threat detection services could find nothing wrong with isocket’s system or the ads we were serving either.
  5. So we believe Google committed a false positive, incorrectly labeling isocket as bad. Then, Google wrongly applied that warning to lots of websites through “guilt by association.”
  6. There is no rhyme or reason why Google blocked the sites that it did, as far as we can tell. No common ad campaigns across the websites affected, etc etc.
  7. It’s very slow at best and impossible at worst to get Google to fix these problems. As a result, people were down for hours and there wasn’t anything we could do to fix it.
  8. We’re asking Google to improve this system, so that even when false positives happen they are properly contained and don’t create such massive unnecessary ripple effects.
  9. isocket will do whatever is possible to prevent this from happening again, even though we can’t identify specific things that caused this. Even if we’re “innocent” in this event, it’s a kick in the ass to make things even more secure – because it’s unacceptable that good publishers are punished for no reason.

What we think happened – a false positive warning, grossly misapplied by Google.

Google continually crawls the web and looks for bad stuff, like malware. Advertising is often a vehicle for bad stuff. When Google detects a threat it automatically tries to quarantine the site and protect the users, usually via the big scary red warning page. Web browsers like Chrome, Firefox, and Safari use this service provided by Google to know when to block things at the browser level.

The code put in the HTML of websites that make online ad services work are often daisy chained, so that when Network 1 doesn’t have an ad to show, it hands the ad space over to Network 2, etc. Many isocket publishers run Google’s AdSense and other networks as remnant, and sometimes they put that 3rd party ad network code into isocket’s ad server so that we can do the daisy chaining (i.e. if we don’t have a premium ad to show, punt it to the 3rd party.)

While isocket can control and protect the ads through our ad server, when the impression is handed off to a 3rd party, we (and the publisher) lose that control. It’s a common problem in the industry. It’s basically an open back door that you can’t control what comes through.

We have zero evidence so far that isocket actually served something bad – either directly through an advertisement bought/hosted through us, or indirectly through these 3rd party networks. No one has been able to produce a report of someone being infected, or to even point to whatever it is that Google didn’t like.

So we’ve been doing our own scans to find what, if any, malware came through. Both our internal scans and those done by professional 3rd party security/verification services have come back with zero problems.

There’s also zero evidence that our systems were hacked or compromised in any way. When ad servers get hacked, there are signs and breadcrumbs. We found none, and we looked through everything including the source code of our ad server. There are additional (pretty severe) security measures we have in place that we don’t disclose publicly, but none of them were compromised.

There are three theories about why Google first created a threat warning:

  1. Somewhere, on one of the isocket enabled websites, Google falsely identified an advertisement served by isocket as bad. There’s a lot of reasons why a false positive could happen. Google looks for certain “evil” things, but sometimes they can seem evil but really be innocent.
  2. isocket did serve something bad on a specific website, such as an ad with a clickthrough URL that was a phishy website. But Google messed up by applying their threat judgement across our entire isocket marketplace, instead of just limiting the warning to the website in question.
  3. On some specific website, we handed the ad call off to a 3rd party remnant ad network daisy chained through us, and that 3rd party served something Google didn’t like, but then Google erred on the side of broad-caution and inappropriately blamed isocket’s ad server as well.

Then, Google applied that warning way too broadly.

It’s bad enough that a false positive warning occurred – but then, instead of limiting the warning to just the website that showed the ad that Google didn’t like, Google applied the warning to all of isocket’s ad server. So that any website that was running isocket could be guilty by association.

Basically, double whammy. In no way did isocket serve malware on the range of sites Google blocked. There was no common pattern, no shared ads run across the sites blocked, no rhyme or reason to it, nothing.

Google reversed their “isocket is bad” decision within an hour of it occurring. But because their system had applied the consequences across a large set of publishers, it then took their system a long time to propagate through the web and unblock all the innocent properties affected due to the broad sword ripple effects.

This “internet police” system must work better.

I love Google, Chrome, and the fact they try to make the web better by helping prevent spam and malware. But when you are in a position to police the web, where people so widely depend upon your word and assume it’s infallible, where one of your algorithms can literally destroy legitimate businesses, you have the moral obligation to do it better.

The first problem: threat detection systems are not perfect.

In the likely event that this was a false positive, or that the true positive happened from another 3rd party source, the method that Google uses to detect problems is not fool proof. It can make mistakes. And we’ve learned and seen first hand that there are often contradictions or inconsistencies in these systems.

The second problem: Google is quick to punish, but too slow to fix mistakes. 

Everything that happens on Google’s end is automated. And while it’s very quick to bring the hammer down (with no warning), the ability to fix mistakes is almost nonexistent. There is no easy remedy. No one you can contact. It’s slow, frustrating, painful, and vague.

So when we found ourselves in this situation, the best we could do is help publishers remove isocket from their webpages and submit their URL to an automated “re-test” machine from Google. It was only later in the day we found out Google had manually reversed the isocket warning within an hour of it occurring.

Many people asked if we had a red nuclear emergency phone to call into Google, assuming there was someone we could call to fix this. Unfortunately there isn’t. It’s a little bit of a black box in that regard.

I’m sure this is a relatively unusual situation for Google. They aren’t trying to hurt websites, and there aren’t a ton of products like isocket out there (so maybe their system was designed for apples but isocket is oranges). But while everyone can make mistakes and that’s OK, when you act as the policeman for the internet it comes with a higher level of responsibility. Good people were hurt, the punishment was unfair, and we all need to do better to prevent that. We will do anything we can to help Google improve in that regard.

What isocket will do from here.

In a weird way, I almost wish this was something we did wrong.  Because then we could own it, directly fix it, and take the punch on the chin. But since so far we can’t identify something specific to fix, that makes it harder to articulate a plan. Regardless, it’s a kick in the ass to always be vigilant for ways to prevent these situations, even if it’s to try and protect our publishers from the wrath of third parties. And we will do whatever we can to work with Google on maturing these systems. If we identify specific problems to fix inside isocket or opportunities to be proactively better, we will be transparent about those plans so we can continue to earn your trust and business.

Please contact us if you have any questions or concerns. We’re sorry, we love you, we love Google, and we love ads. Except when ads break the internet. Then we don’t love ads so much ;)

{ 5 comments… read them below or add one }

Amy Jurries January 17, 2013 at 10:53 AM

Thanks John. I understand how hard it can be to work with Google at times and really appreciate your transparency on the matter. As one of the sites that was brought down from the false malware threat, it would be great in the future if you could email or somehow contact your customers to let them know what is happening. It took me an entire day of calls to Godaddy, Typepad, etc. to finally figure out it was the isocket code and not some attack on my site. I am assuming it is now ok for us to replace the isocket code?
Thanks again for your fantastic ad service! I remain an isocket fan.

Amy

Reply

John Ramey January 17, 2013 at 1:44 PM

Thanks for the note Amy. Regarding the notification – we did provide notice almost immediately through the normal channels: facebook, twitter, and a notification when you logged into isocket.com.

Please contact support@isocket(dot)com if you need any assistance putting things back together. Although it’s not possible for us to give 100% assurance on things, given what I’ve described above with Google, reinstalling your ad code should be fine.

Happy advertising =)

Marc Perry January 17, 2013 at 7:24 PM

Thanks for the clarification, John. It does sound like it was a brutally difficult day for you guys. The truth is you’ve created a very awesome product and you and your customers are innocent victims.

I agree with Amy, an email to all of isocket customers would have been the right move. I’m actually curious why you didn’t do that? I’m sure tons of your customers were scrambling like us to figure out what was going on. Fortunately, we searched twitter for “malware” and the TNW article came up. I’m not sure it’s fair to expect all of your customers to follow you on twitter, facebook, and be logged in to get some type of warning message, which I didn’ t even notice when I logged in.

Going forward, if something like this were to happen again, how will you handle it? Will you let everyone know ASAP via email and give specific instructions on actions to take?

Reply

John Ramey January 18, 2013 at 9:24 AM

Thanks for the kind words, Marc.

The email vs other communication methods in this scenario is an interesting debate within web product circles, actually. The thought process is that when things are happening in real time, you use the real time channels to get the word out quickly. I think we did a great job quickly saying something, even before we had confirmation something was wrong. There’s also a tough balance of what to officially communicate versus what we actually know internally – because the problem here is this vague black box from Google, it took a while to triangulate what was happening / what steps folks should take.

Your question makes me realize a flaw in my thinking – that all depended on people knowing there was a problem related to isocket. Even though on the red malware warning page when you click into “more details” it would say isocket’s the cause, a lot of people didn’t dig that deep and just thought “crap, I have malware! Why?!” So if someone didn’t know that isocket was involved, they wouldn’t have checked our twitter/facebook/web app. Thanks for the feedback, we’ll be more proactive on email next time.

FWIW, we are definitely planning an email to all customers with the post-mortem.

IanR September 3, 2013 at 3:33 PM

Nine months on and Google still haven’t cleaned up their act. Recently they blacklisted our site, claiming that there was malware present. After hours of wasted work downloading and scanning the entire site contents, no malware was found.

When quizzed about it at length , a Google rep then claimed that the malware wasn’t actually on our site but on a site we carried a link to. This other site reportedly had three infected files, one of which was a css stylesheet. I don’t know of any way in which a stylesheet could carry malware, after all it’s basically a text file. In any event I downloaded these three files and sent them to virustotal. Nothing found.

In principle this whole website-scanning business is about as reliable and useful as a burglar alarm which goes off at random all night. If they can’t make the thing work properly, and it still keeps giving false alarms, then they need to be given a court order to shut the thing off.

Reply

Leave a Comment

{ 2 trackbacks }

Previous post:

Next post: