Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

executing pig script file with multiple stores in embedded pig program

Tags:

apache-pig

I want to execute a pig script file in embedded pig program which is shown below

----testPig.pig-----

A = load '/user/biadmin/student' using PigStorage() as (name:chararray);
B = foreach A generate name; 
store B into '/user/biadmin/myoutput001';

for this I have written code as shown below

> PigServer pigServer = new PigServer(ExecType.MAPREDUCE);
> pigServer.registerScript("testPig.pig");

but it is not working.I have checked this in grunt-shell mode. there it is working fine. So I made changes like this

---testPig.pig -----

A = load '/user/biadmin/student' using PigStorage() as (name:chararray);
B = foreach A generate name;
--store B into '/user/biadmin/myoutput001';

Embedded pig code for this is

> PigServer pigServer = new PigServer(ExecType.MAPREDUCE,prt);
> pigServer.registerScript(path);
> pigServer.store("B","/user/biadmin/myoutput20");

Now the modified code is working fine. So now my doubt is

  1. why I was not able to execute pig script which is having store command?

  2. How can I execute pig script file which is having store command?

like image 971
Rajesh Barri Avatar asked Aug 24 '26 09:08

Rajesh Barri


1 Answers

Your PigServer code is not working because; when you call .registerScript(), by default, PigServer sets the interactive mode flag on GruntParser to false. From the PigServer source code:

public void registerScript(InputStream in, Map<String,String> params,List<String> paramsFiles) throws IOException {
    try {
        String substituted = doParamSubstitution(in, params, paramsFiles);
        GruntParser grunt = new GruntParser(new StringReader(substituted));
    /********************************************/
        grunt.setInteractive(false);
    /********************************************/
        grunt.setParams(this);
        grunt.parseStopOnError(true);
    } catch (org.apache.pig.tools.pigscript.parser.ParseException e) {
        log.error(e.getLocalizedMessage());
        throw new IOException(e.getCause());
    }
}

Quoting from the GruntParser source code:

In interactive mode, executes the plan right away whenever a STORE command is encountered.

This means that when interactive mode is not active, STOREcommands will be ignored (that is they won't run automatically) until further PigServer.openIterator or PigServer.store calls (that is you explicitly make a call requiring the STORE line).

As for your second question, you might want to have a look at PigRunner class.

like image 61
Cihan Keser Avatar answered Aug 26 '26 23:08

Cihan Keser



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!