/usr/bin/env questions regarding shebang line pecularities

Question

Questions:

• What does the kernel do if you stick a shell-script into the shebang line?
• How does the Kernel know which interpreter to launch?

Explanation:

I recently wanted to write a wrapper around /usr/bin/env because my CGI Environment does not allow me to set the PATH variable, except globally (which of course sucks!).

So I thought, "OK. Let's set PREPENDPATH and set PATH in a wrapper around env.". The resulting script (here called env.1) looked like this:

#!/bin/bash
/usr/bin/env PATH=$PREPENDPATH:$PATH $*

which looks like it should work. I checked how they both react, after setting PREPENDPATH:

$ which /usr/bin/env python
/usr/bin/env
/usr/bin/python

$ which /usr/bin/env.1 python
/usr/bin/env
/home/pi/prepend/bin/python

Look absolutely perfect! So far, so good. But look what happens to "Hello World!".

# Shebang is #!/usr/bin/env python
$ test-env.py
Hello World!

# Shebang is #!/usr/bin/env.1 python
$ test-env.1.py
Warning: unknown mime-type for "Hello World!" -- using "application/*"
Error: no such file "Hello World!"

I guess I am missing something pretty fundamental about UNIX.

I'm pretty lost, even after looking at the source code of the original env. It sets the environment and launches the program (or so it seems to me...).

Answer 1

First of all, you should very seldom use $* and you should almost always use "$@" instead. There are a number of questions here on SO which explain the ins and outs of why.

Second - the env command has two main uses. One is to print the current environment; the other is to completely control the environment of a command when it is run. The third use, which you are demonstrating, is to modify the environment, but frankly there's no need for that - the shells are quite capable of handling that for you.

Mode 1:

env

Mode 2:

env -i HOME=$HOME PATH=$PREPENDPATH:$PATH ... command args

This version cancels all inherited environment variables and runs command with precisely the environment set by the ENVVAR=value options.

The third mode - amending the environment - is less important because you can do that fine with regular (civilized) shells. (That means "not C shell" - again, there are other questions on SO with answers that explain that.) For example, you could perfectly well do:

#!/bin/bash
export PATH=${PREPENDPATH:?}:$PATH
exec python "$@"

This insists that $PREPENDPATH is set to a non-empty string in the environment, and then prepends it to $PATH, and exports the new PATH setting. Then, using that new PATH, it executes the python program with the relevant arguments. The exec replaces the shell script with python. Note that this is quite different from:

#!/bin/bash
PATH=${PREPENDPATH:?}:$PATH exec python "$@"

Superficially, this is the same. However, this will execute the python found on the pre-existing PATH, albeit with the new value of PATH in the process's environment. So, in the example, you'd end up executing Python from /usr/bin and not the one from /home/pi/prepend/bin.

In your situation, I would probably not use env and would just use an appropriate variant of the script with the explicit export.

The env command is unusual because it does not recognize the double-dash to separate options from the rest of the command. This is in part because it does not take many options, and in part because it is not clear whether the ENVVAR=value options should come before or after the double dash.

I actually have a series of scripts for running (different versions of) a database server. These scripts really use env (and a bunch of home-grown programs) to control the environment of the server:

#!/bin/ksh
#
# @(#)$Id: boot.black_19.sh,v 1.3 2008/06/25 15:44:44 jleffler Exp $
#
# Boot server black_19 - IDS 11.50.FC1

IXD=/usr/informix/11.50.FC1
IXS=black_19
cd $IXD || exit 1

IXF=$IXD/do.not.start.$IXS
if [ -f $IXF ]
then
    echo "$0: will not start server $IXS because file $IXF exists" 1>&2
    exit 1
fi

ONINIT=$IXD/bin/oninit.$IXS
if [ ! -f $ONINIT ]
then ONINIT=$IXD/bin/oninit
fi

tmpdir=$IXD/tmp
DAEMONIZE=/work1/jleffler/bin/daemonize
stdout=$tmpdir/$IXS.stdout
stderr=$tmpdir/$IXS.stderr

if [ ! -d $tmpdir ]
then asroot -u informix -g informix -C -- mkdir -p $tmpdir
fi

# Specialized programs carried to extremes:
#   * asroot sets UID and GID values and then executes
#   * env, which sets the environment precisely and then executes
#   * daemonize, which makes the process into a daemon and then executes
#   * oninit, which is what we really wanted to run in the first place!
# NB: daemonize defaults stdin to /dev/null and could set umask but
#     oninit dinks with it all the time so there is no real point.
# NB: daemonize should not be necessary, but oninit doesn't close its
#     controlling terminal and therefore causes cron-jobs that restart
#     it to hang, and interactive shells that started it to hang, and
#     tracing programs.
# ??? Anyone want to integrate truss into this sequence?

asroot -u informix -g informix -C -a dbaao -a dbsso -- \
    env -i HOME=$IXD \
        INFORMIXDIR=$IXD \
        INFORMIXSERVER=$IXS \
        INFORMIXCONCSMCFG=$IXD/etc/concsm.$IXS \
        IFX_LISTEN_TIMEOUT=3 \
        ONCONFIG=onconfig.$IXS \
        PATH=/usr/bin:$IXD/bin \
        SHELL=/usr/bin/ksh \
        TZ=UTC0 \
    $DAEMONIZE -act -d $IXD -o $stdout -e $stderr -- \
    $ONINIT "$@"

case "$*" in
(*v*) track-oninit-v $stdout;;
esac

Answer 2

You should carefully read the wikipedia article about shebang.

When your system sees the magic number corresponding to the shebang, it does an execve on the given path after the shebang and gives the script itself as an argument.

Your script fails because the file you give (/usr/bin/env.1) is not an executable, but begins itself by a shebang....

Ideally, you could resolve it using... env on your script with this line as a shebang:

#!/usr/bin/env /usr/bin/env.1 python

It won't work though on linux as it treats "/usr/bin/env.1 python" as a path (it doesn't split arguments)

So the only way I see is to write your env.1 in C

EDIT: seems like no one belives me ^^, so I've written a simple and dirty env.1.c:

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>


const  char* prependpath = "/your/prepend/path/here:";

int main(int argc, char** argv){
  int args_len = argc + 1;
  char* args[args_len];
  const char* env = "/usr/bin/env";
  int i;

  /* arguments: the same */
  args[0] = env;
  for(i=1; i<argc; i++)
    args[i] = argv[i];
  args[argc] = NULL;

  /* environment */
  char* p = getenv("PATH");
  char* newpath = (char*) malloc(strlen(p)
                 + strlen(prependpath));
  sprintf(newpath, "%s%s", prependpath, p);
  setenv("PATH", newpath, 1);

  execv(env, args);
  return 0;
}