Cronjob: Web Service query

Question

I have a cronjob that runs every hours and parse 150,000+ records. Each record is summarized individually in a MySQL tables. I use two web services to retrieve the user information.

1. User demographic (ip, country, city etc.)
2. Phone information (if landline or cell phone and if cell phone what is the carrier)

Every time I get 1 record I check if I have information and if not I call these web services. After tracing my code I found out both of these calls takes 2 to 4 seconds and it makes my cronjob very slow and I can't compile statistics on time.

Is there a way to make these web service faster?

Thanks

Answer 1

simple:

get the data locally and use mellissa data:

1. for ip: http://w10.melissadata.com/dqt/websmart/ip-locator.htm
2. for phone: http://www.melissadata.com/fonedata.html

you can also cache them using memcache or APC which will make it faster since he does not have to request the data from the api or database.

Answer 2

A couple of ideas... if the same users are returning, caching the data in another table would be very helpful... you would only look it up once and have it for returning users. Upon re-reading the question it looks like you are doing that.

Another option would be to spawn new threads when you need to do the look-ups. This could be a new thread for each request, or if this is not feasible you could have n service threads ready to do the look-ups and update the results.