Scaling Software/Hardware for a Large # of External API Requests?

Question

We have a system that given a batch of requests, makes an equivalent number of calls to an external 3rd Party API. Given that this is an I/O bound task, we currently use a cached thread-pool of size 20 to service these requests. Other than above, is the solution to:

Use fewer machines with more cores (less context-switching, capable of supporting more concurrent threads)

or

Use more machines by leveraging commodity/cheap hardware (pizza boxes)

The number of requests we receive a day is on the order of millions.

We're using Java, so the threads here are kernel, not "green".

Other Points/Thoughts:

• Hadoop is commonly used for problems of this nature, but this needs to be real-time vs. the stereotypical offline data mining.
• The API requests take anywhere from 200ms to 2 seconds on average
• There is no shared state between requests
• The 3rd Party in question is capable of servicing more requests than we can possibly fire (payments vendor).

Answer 1

It's not obvious to me that you need more resources at all (larger machines or more machines). If you're talking about at most 10 million requests in a day taking at most 2 seconds each, that means:

• ~110 requests per second. That's not so fast. Are the requests particularly large? Or are there big bursts? Are you doing heavy processing besides dispatching to the third-party API? You haven't given me any information so far that leads me to believe it's not possible to run your whole service on a single core. (Call it three of the smallest possible machines if you want to have n+2 redundancy.)
• on average, ~220 active requests. Again, that seems like no problem for a single machine, even with a (pooled) thread-per-request model. Why don't you just expand your pool size and call it a day? Are these really bursty? (And do you have really tight latency/reliability requirements?) Do they need a huge amount of RAM while active?

Could you give some more information on why you think you have to make this choice?