Unhandled internal error. Java heap space when using EXPLAIN in pig

Question

When I try to use the following statement

explain -brief A;

I got the following error

2013-02-06 19:18:34,250 [Low Memory Detector] INFO     org.apache.pig.impl.util.SpillableMemoryManager - first memory handler call - Collection threshold init = 526123008(513792K) used = 349984528(341781K) committed = 699072512(682688K) max = 699072512(682688K)
2013-02-06 19:18:34,942 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. Java heap space

I tried to increase the heap space size by adding option

-Dmapred.child.java.opts=-Xmx4096m

but it didn't work. Any idea?

Thanks. Dan

Answer 1

1. Increase the PIG heap space using

   export PIG_HEAPSIZE=4096

ALTHOUGH, if you're running the PIG in a Virtual Machine, make sure you have allocated more than 4GB (or whatever amount you want to set) of memory to the Virtual Machine. Say if you've allocated X Gb's to the VM, then increasing the heap size more than X Gb's won't do anything.

2. ALSO, it may be an issue with the PIG release you are using.

Try using an older pig release. I was getting the same reproducible issue with PIG-0.14.0 but not with PIG-0.12.1