GNU Parallel: suppress warning about increasing blocksize

Question

I'm using GNU Parallel to execute a command concurrently over each line of a file. Though it's not causing any issues with the results, GNU parallel emits a lot of the following type of warnings, cluttering the output:

parallel: Warning: A full record was not matched in a block. Increasing to --blocksize 1363150

I assume this just means that the blocksize I elected to use didn't quite match nicely at an input boundary and so parallel is increasing the blocksize so it does. As I said, I don't believe this is causing any problems with the output result, it's just not a warning I need to see repeated over and over, or even once. Is there any way to suppress these warnings specifically? I don't really want to redirect stderr altogether and miss any important messages.

Answer 1

Your assumption is wrong. What GNU Parallel tells you is that it has read a full block and your record could not fit in that size. So increase the size. GNU Parallel tells you that '--blocksize 1363150' would be sufficient, but you may want to increase it even further to, say, '--block 3M'.

If you set '--blocksize' to at least 2 times the max record size, you will never see the warning.

Technically what it going on is that GNU Parallel reads a full block (for this example let us say 7 bytes) and appends it to its buffer. It tries to find a full record (for this example let us say a full line) in its buffer. If it cannot, it gives you the warning, increases the blocksize (slowly) exponentially and tries again:

Input data:

12345\n
123456\n
1234567\n
12345678\n
123456789\n
1234567890\n

GNU Parallel reads 7 bytes at a time:

12345\n
1

The buffer contains a full line, so everything is good.

23456\n
1

The buffer contains a full line, so everything is good.

234567

The buffer does not contain a full line, so you get a warning and the --blocksize is increased to 8:

\n
1234567

The buffer now contains a full line, so everything is good.

8\n
123456

The buffer contains a full line, so everything is good.

789\n
1234

The buffer contains a full line, so everything is good.

567890\n

The buffer contains a full line, so everything is good.