Selenium based malware (malvertising) checking - A few questions

Question

We recently had an issue where an advertiser who purchased advertisments via a 3rd party was distributing malware via the ads they purchased.

This led to google black listing our web property for a short period of time.

This issue is now resolved.

After this happened we decided that we will self-aduit our advertisers.

After searching the web for services that provide this service, we found a few... Armorize (www.armorize.com), amongst others, provides this type of service. But after speaking with their sales on the telephone we found that they charge aprox 10K-15K USD / year for this service. Way out of our price range.

We dont have that kind of cake.

What we do have is a smart head on our (err, my) shoulders.

So, here is what I have developed.

A) Selenium running firefox.
B) Firefox proxying all requests via a locally hosted squid proxy.

The result?

Pipe in advertisers URL -> Selenium Firefox -> Squid access log -> A nice clean list of all URLS hit by the advertisment(s).

The next step was to test these against some sort of malware list. We are now testing them againts googles safebrowsing API ( https://developers.google.com/safe-browsing/ ).

The result is exactly what we wanted. A way to test via "real browser" each of the URLS hit by our advertisers.

So, the questions are as follow:

a) Is using their (googles) API like this acceptable as far as google is concerned? We will be keeping this 100% in house, and will not be reselling this service. Its 100% for us.

b) Does the google safe browser API allow checking of FULL URLs, or does it work only on a per-domain basis?

c) Does anyone know any other APIs where we can test these URLs? Free / low cost would be great :)

Thanks!

Answer 1

a. Reviewing the Safe Browsing API Terms of Service together with the Google APIs Terms of Service I cannot find anything that you are doing that falls outside of these.

b. The docs consistently refer to URL rather than domain - having performed some tests (e.g. liderlab.ru / absa/ vs. liderlab.ru /absa / page/ 1) the first is a phising site and gives the appropriate warning whereas the second doesn't).

c. PhishTank is good and free and seems to be a little more current than Google (from a brief investigation). BrightCloud is a reasonably priced pay for service. URL Blacklist is a pay for service that works on a honour system so you can see their data.