Limit to number of files to cp in parallel

Question

Im running the gsutil cp command in parallel (with the -m option) on a directory with 25 4gb json files (that i am also compressing with the -z option).

gsutil -m cp -z json -R dir_with_4g_chunks gs://my_bucket/

When I run it, it will print out to terminal that it is copying all but one of the files. By this I mean that it prints one of these lines per file:

Copying file://dir_with_4g_chunks/a_4g_chunk [Content-Type=application/octet-stream]...

Once the transfer for one of them is complete, it says that it'll be copying the last file.

The result of this is that there is one file that only starts to copy only when one of the others finishes copying, significantly slowing down the process

Is there a limit to the number of files I can upload with the -m option? Is this configurable in the boto config file?

Answer 1

I was not able to find the .boto file on my Mac (as per jterrace's answer above), instead I specified these values using the -o switch:

gsutil -m -o "Boto:parallel_thread_count=4" cp directory1/* gs://my-bucket/

This seemed to control the rate of transfer.

Answer 2

From the description of the -m option:

gsutil performs the specified operation using a combination of multi-threading and multi-processing, using a number of threads and processors determined by the parallel_thread_count and parallel_process_count values set in the boto configuration file. You might want to experiment with these value, as the best value can vary based on a number of factors, including network speed, number of CPUs, and available memory.

If you take a look at your .boto file, you should see this generated comment:

# 'parallel_process_count' and 'parallel_thread_count' specify the number
# of OS processes and Python threads, respectively, to use when executing
# operations in parallel. The default settings should work well as configured,
# however, to enhance performance for transfers involving large numbers of
# files, you may experiment with hand tuning these values to optimize
# performance for your particular system configuration.
# MacOS and Windows users should see
# https://github.com/GoogleCloudPlatform/gsutil/issues/77 before attempting
# to experiment with these values.
#parallel_process_count = 12
#parallel_thread_count = 10

I'm guessing that you're on Windows or Mac, because the default values for non-Linux machines is 24 threads and 1 process. This would result in copying 24 of your files first, then the last 1 file afterward. Try experimenting with increasing these values to transfer all 25 files at once.