How to find optimal number of mappers when running Sqoop import and export?

Question

I'm using Sqoop version 1.4.2 and Oracle database.

When running Sqoop command. For example like this:

./sqoop import                               \
    --fs <name node>                         \
    --jt <job tracker>                       \
    --connect <JDBC string>                  \
    --username <user> --password <password>  \
    --table <table> --split-by <cool column> \
    --target-dir <where>                     \
    --verbose --m 2

We can specify --m - how many parallel tasks do we want Sqoop to run (also they might be accessing Database at same time). Same option is available for ./sqoop export <...>

Is there some heuristic (probably based on size of data) which will help to guess what is optimal number of task to use?

Thank you!

Answer 1

This is taken from Apache Sqoop Cookbook by O'Reilly Media, and seems to be the most logical answer.

The optimal number of mappers depends on many variables: you need to take into account your database type, the hardware that is used for your database server, and the impact to other requests that your database needs to serve. There is no optimal number of mappers that works for all scenarios. Instead, you’re encouraged to experiment to find the optimal degree of parallelism for your environment and use case. It’s a good idea to start with a small number of mappers, slowly ramping up, rather than to start with a large number of mappers, working your way down.