PHP: How does Trivago, Hotelscombined get data faster

Question

Hello everyone greetings, I am developing website which is similar to Trivago and Hotelscombined. I am able to get the data from all apis like expedia, skoosh, lateroom, getaroom, venere, hotel pronto. I am stuck at the point where I have to maintain the performance of the website. I want to know how to make the website faster just like http://www.trivago.com , http://hotelscombined.com.

How to work with pagination and compare the prices with all apis together. If I have a cron job running and saving the data in the database then it would be easier for me to show the prices of all the apis and compare them and it will be faster. But it is the bad idea for me to run the cron job, specially when we are talking about hotels api because the content changes every time for hotels. Problem with the cron is that, there are thousand of cities, how much data should be fetched in the database and how many times because hotels are booked in next 1 hour approx . If we run the cron job, and suppose hotel was booked which was fetched and saved in the database, in that case how should i manage this. If cron is the answer to the question then how should we fetch 10 to 20 apis data containing hotels description, images etc in the database and manage them because there are over 1 million hotels in the world or may be more than 10 million. Also there are Cases such as:

i) Search for hotel in New York for 2 adults, 1 children from 1st August check in time till 8th August check out time.

ii) Search for hotel in London for 1 adults, 2 children from 2nd August check in time till 5th August check out time. And there can be many cases in search. So how many times and for how many cases we should run the cron?

Within the specification of Expedia (http://developer.ean.com/spec/) it is stated that there is a limitations of the number of requests: - Hotel List Requests: 5000:1 - Room Availability Requests: 500:1

It will be very much helpful for me if I am able to know how to speed up the website.

Thanks in advance

Answer 1

I think you answered your own question. Cron jobs and caching to local database sounds like a good idea to me. Just set the cron to a similar interval that the content changes at the different hotel sites. If performance is crucial, perhaps have the cache database on it's own server.

Answer 2

Amuk, try to use curl_multi. It's a great way to process multiple HTTP requests in parallel in PHP. curl_multi is particularly handy when working with large data sets (like fetching thousands of RSS feeds at one time) http://www.onlineaspect.com/2009/01/26/how-to-use-curl_multi-without-blocking/