Running command in perl6, commands that work in shell produce failure when run inside perl6

Question

I'm trying to run a series of shell commands with Perl6 to the variable $cmd, which look like

databricks jobs run-now --job-id 35 --notebook-params '{"directory": "s3://bucket", "output": "s3://bucket/extension", "sampleID_to_canonical_id_map": "s3://somefile.csv"}'

1. Splitting the command by everything after notebook-params

my $cmd0 = 'databricks jobs run-now --job-id 35 --notebook-params '; my $args = "'{\"directory\": \"$in-dir\", \"output\": \"$out-dir\", \"sampleID_to_canonical_id_map\": \"$map\"}'"; my $run = run $cmd0, $args, :err, :out;

Fails. No answer given either by Databricks or the shell. Stdout and stderr are empty.

2. Splitting the entire command by white space

   my @cmd = $cmd.split(/\s+/); my $run = run $cmd, :err, :out

Error: Got unexpected extra arguments ("s3://bucket", "output": "s3://bucket/extension", "sampleID_to_canonical_id_map": "s3://somefile.csv"}'

3. Submitting the command as a string my $cmd = "$cmd0\"$in-dir\", \"output\": \"$out-dir\", \"sampleID_to_canonical_id_map\": \"$map\"}'";

again, stdout and stderr are empty. Exit code 1.

this is something about how run can only accept arrays, and not strings (I'm curious why)

If I copy and paste the command that was given to Perl6's run, it works when given from the shell. It doesn't work when given through perl6. This isn't good, because I have to execute this command hundreds of times.

Perhaps Perl6's shell https://docs.perl6.org/routine/shell would be better? I didn't use that, because the manual suggests that run is safer. I want to capture both stdout and stderr inside a Proc class

EDIT: I've gotten this running with shell but have encountered other problems not related to what I originally posted. I'm not sure if this qualifies as being answered then. I just decided to use backticks with perl5. Yes, backticks are deprecated, but they get the job done.

Answer 1

I'm trying to run a series of shell commands

To run shell commands, call the shell routine. It passes the positional argument you provide it, coerced to a single string, to the shell of the system you're running the P6 program on.

For running commands without involving a shell, call the run routine. The first positional argument is coerced to a string and passed to the operating system as the filename of the program you want run. The remaining arguments are concatenated together with a space in between each argument to form a single string that is passed as a command line to the program being run.

my $cmd0 = 'databricks jobs run-now --job-id 35 --notebook-params ';

That's wrong for both shell and run:

• shell only accepts one argument and $cmd0 is incomplete.

• The first argument for run is a string interpreted by the OS as the filename of a program to be run and $cmd0 isn't a filename.

So in both cases you'll get either no result or nonsense results.

Your other two experiments are also invalid in their own ways as you discovered.

this is something about how run can only accept arrays, and not strings (I'm curious why)

run can accept a single argument. It would be passed to the OS as the name of the program to be run.

It can accept two arguments. The first would be the program name, the second the command line passed to the program.

It can accept three or more arguments. The first would be the program name, the rest would be concatenated to form the command line passed to the program. (There are cases where this is more convenient coding wise than the two argument form.)

run can also accept a single array. The first element would the program name and the rest the command line passed to it. (There are cases where this is more convenient.)

I just decided to use backticks with perl5. Yes, backticks are deprecated, but they get the job done.

Backticks are subject to code injection and shell interpolation attacks and errors. But yes, if they work, they work.

P6 has direct equivalents of most P5 features. This includes backticks. P6 has two variants:

• The safer P6 alternative to backticks is qx. The qx quoting construct calls the shell but does not interpolate P6 variables so it has the same sort of level of danger as using shell with a single quoted string.

• The qqx variant is the direct equivalent of P5 backticks or using shell with a double quoted string so it suffers from the same security dangers.

Answer 2

Two mistakes:

• the simplistic split cuts up the last, single parameter into multiple arguments
• you are passing $cmd to run, not @cmd

use strict;

my @cmd = ('/tmp/dummy.sh', '--param1', 'param2 with spaces');
my $run = run @cmd, :err, :out;

print(@cmd ~ "\n");
print("EXIT_CODE:\t" ~ $run.exitcode  ~ "\n");
print("STDOUT:\t"    ~ $run.out.slurp ~ "\n");
print("STDERR:\t"    ~ $run.err.slurp ~ "\n");

output:

$ cat /tmp/dummy.sh
#!/bin/bash
echo "prog: '$0'"
echo "arg1: '$1'"
echo "arg2: '$2'"
exit 0

$ perl6 dummy.pl
/tmp/dummy.sh --param1 param2 with spaces
EXIT_CODE:      0
STDOUT: prog: '/tmp/dummy.sh'
arg1: '--param1'
arg2: 'param2 with spaces'

STDERR:

If you can avoid generating $cmd as single string, I would generate it into @cmd directly. Otherwise you'll have to implement complex split operation that handles quoting.