A good way to bulk download images over http with Java

Question

We have a web application that needs to import 10-20 images from a partner site via http. If I have a list of strings that represent the urls I want to download does anybody have a suggestion for how to download them as fast as possible?

I could just put them in a for loop but if there is a simple way to parallelize this it would be probably be good for the end user. I would like to avoid using straight Java threads, although the executor framework might be a good idea.

Any ideas?

Answer 1

Here is my take using Resty. (DISCLAIMER: I'm the author of Resty) Downloads all URLs given on the command line and prints out the file names.

package us.monoid.web.parallel;

import java.io.File;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;

import us.monoid.web.Resty;

public class Downloader {

    public static void main(String[] args) throws InterruptedException, ExecutionException {
        ExecutorService pool = Executors.newFixedThreadPool(10);
        List<Callable<File>> tasks = new ArrayList<Callable<File>>(args.length);
        for (final String url : args) {
            tasks.add(new Callable<File>() {
                public File call() throws Exception {
                    return new Resty().bytes(url).save(File.createTempFile("img", ".png"));
                }               
            });
        }
        List<Future<File>> results = pool.invokeAll(tasks);
        for (Future<File> ff : results) {
            System.out.println(ff.get());
        }
    }

}