user data scripts fails without giving reason

Question

I am starting a Amazon Linux instance (ami-fb8e9292) using the web console, pasting data into the user data box to run a script upon startup. If I use the example given by amazon to start a web server, it works. But when I run my own script (also a #!/bin/bash script), it does not get run.

If I look in var/log/cloud-init.log, it gives no useful information on the topic:

May 22 21:06:12 cloud-init[1286]: util.py[DEBUG]: Running command ['/var/lib/cloud/instance/scripts/part-001'] with allowed return codes [0] (shell=True, capture=False)
May 22 21:06:16 cloud-init[1286]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [2]
May 22 21:06:16 cloud-init[1286]: util.py[DEBUG]: Failed running /var/lib/cloud/instance/scripts/part-001 [2]
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 637, in runparts
    subp([exe_path], capture=False, shell=True)
  File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 1528, in subp
    cmd=args)
ProcessExecutionError: Unexpected error while running command.
Command: ['/var/lib/cloud/instance/scripts/part-001']
Exit code: 2
Reason: -
Stdout: ''
Stderr: ''

If I ssh into the instance and sudo su and execute the shell script directly:

/var/lib/cloud/instance/scripts/part-001

then it runs fine. Also, it works if I emulate the way cloud-init runs it:

python
>>> import cloudinit.util
>>> cloudinit.util.runparts("/var/lib/cloud/instance/scripts/")

Using either of those methods, if I intentionally introduce errors into the script then it produces error messages. How can I debug the selective absence of useful debugging output?

Answer 1

I'm not sure if this is going to be the case for everyone, but I was having this issue and was able to fix it by changing my first line from this:

#!/bin/bash -e -v

to just this:

#!/bin/bash

Of course, now my script is failing and I have no idea how far it's getting, but at least I got past it not running it at. :)

Answer 2

Instead of /var/log/cloud-init.log consider searcing for keywords like "Failed", "ERROR" "WARNING" or "/var/lib/cloud/instance/scripts/" inside /var/log/cloud-init-output.log - which in most cases, contains very clear error messages.

For example - running a bad command will produce the following error in /var/log/cloud-init-output.log:

/var/lib/cloud/instance/scripts/part-001: line 10: vncpasswd: command not found
cp: cannot stat '/lib/systemd/system/[email protected]': No such file or directory
sed: can't read /etc/systemd/system/[email protected]: No such file or directory
Failed to execute operation: No such file or directory
Failed to start vncserver@:1.service: Unit not found.
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Cleaning repos: amzn2-core amzn2extra-docker amzn2extra-epel

And at the end of /var/log/cloud-init.log you'll receive a quiet general error message:

Aug 31 15:14:00 cloud-init[3532]: util.py[DEBUG]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 910, in runparts
        subp(prefix + [exe_path], capture=False, shell=True)
      File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 2105, in subp
        cmd=args)
    ProcessExecutionError: Unexpected error while running command.
    Command: ['/var/lib/cloud/instance/scripts/part-001']
    Exit code: 1
    Reason: -
    Stdout: -
    Stderr: -
    cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)

---

(*) Try to grep just the relevant error message with:

grep -C 10 '<search-keyword>' cloud-init-output.log

Answer 3

Hope it will reduce the debugging time for someone. I didn't have any explicit error messages in my /var/log/cloud-init-output.log, just this:

2021-04-07 10:36:57,748 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts) 2021-04-07 10:36:57,748 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed

After some investigation, I've realized that the cause was a typo in the shebang string: #!?bin/bash instead of #!/bin/bash.

Answer 4

I had a similar issue and I was able to get around it. I realized that the environment variables EC2_HOME would not be setup for the sudo. I was doing a bunch of stuff in my configset which uses aws cli and for these to work, the EC2_HOME needs to be setup. So, I went in and removed sudo everywhere in my configset and UserData. Earlier when I was hitting the issue, my UserData looked like:

"UserData"       : { "Fn::Base64" : { "Fn::Join" : ["", [
                                "#!/bin/bash\n",
                                "sudo yum update -y aws-cfn-bootstrap\n",

                                "# Install the files and packages and run the commands from the metadata\n",
                                "sudo /opt/aws/bin/cfn-init -v --access-key ", { "Ref" : "IAMUserAccessKey" }, " --secret-key ", { "Ref" : "SecretAccessKey" },  
                                "         --stack ", { "Ref" : "AWS::StackName" },
                                "         --resource NAT2 ",
                                "         --configsets config ",
                                "         --region ", { "Ref" : "AWS::Region" }, "\n"
                        ]]}}

My UserData after the changes looked like:

"UserData"       : { "Fn::Base64" : { "Fn::Join" : ["", [
                                "#!/bin/bash -xe\n",
                                "yum update -y aws-cfn-bootstrap\n",

                                "# Install the files and packages and run the commands from the metadata\n",
                                "/opt/aws/bin/cfn-init -v --access-key ", { "Ref" : "IAMUserAccessKey" }, " --secret-key ", { "Ref" : "SecretAccessKey" },  
                                "         --stack ", { "Ref" : "AWS::StackName" },
                                "         --resource NAT2 ",
                                "         --configsets config ",
                                "         --region ", { "Ref" : "AWS::Region" }, "\n"
                        ]]}}

Similarly, I removed all the sudo calls I was doing in my configsets