Ubuntu AMI not loading user-data

Question

I am attempting to load user data into an Ubuntu 12.04 LTS AMI (ami-a29943cb, but I've tried a few others to no avail) via boto's ec2.run_instances(..., user_data=USER_DATA). Similarly, I have had no success with manually supplying the user data while launching the instances via the AWS console. There are no results or messages in /var/logs/syslog for any of the methods I've tried.

USER_DATA looks something like the following, read in as a string from a file:

#!/usr/bin/env python

import boto

AWS_BOOTSTRAP_BUCKET  = ''
AWS_ACCESS_KEY_ID     = ''
AWS_SECRET_ACCESS_KEY = ''

# Pull processing script from S3.
print 'Bootstrapping started.....'
print 'Connecting to S3...'
s3     = boto.connect_s3(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
bucket = s3.get_bucket(AWS_BOOTSTRAP_BUCKET)
print 'Downloading bootstrap file...'
key    = bucket.get_key('xxx')
key.get_contents_to_filename('xxx')

print 'Importing Bootstrap file...'
import xxx
xxx.process()

# Shut down the EC2 instance running this process.
print 'Shutting down this instance...'
import socket
desired_hostname = socket.gethostname()
ec2 = boto.connect_ec2(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
reservations = ec2.get_all_instances()
for reservation in reservations:
    for instance in reservation.instances:
        if desired_hostname in instance.private_dns_name:
            instance.terminate()

I have furthermore tried uploading the file to a public S3 bucket and loading it in this manner, once again to no avail:

#include
https://s3.amazonaws.com/bucket/file.py

Does anyone have any advice in this regard? Am I completely misunderstanding the purpose of user-data/cloud-init or is the technology merely broken in the AMI I am trying to utilize?

Answer 1

It's hard to know what happened without an error message, but there are a few places you can look:

1. The file /var/log/cloud-init.log will usually contain any errors (e.g. boto import failure) that occurred during instance bootstrapping.
2. The directory /var/lib/cloud/instance will contain the raw scripts and user-data downloaded to the instance
3. You can View/Edit USER_DATA inside the AWS console by right-clicking the instance, to see if boto populated it correctly.

Looking in those places should help provide clarity.

I know Ubuntu 12.04 comes with boto 2.2.2:

root@vanilla-562c:/var/lib/cloud/instance# python
Python 2.7.3 (default, Apr 20 2012, 22:44:07) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto
>>> boto.__version__
'2.2.2'

..but I wonder if it's actually accessible in your PYTHONPATH at runtime.

Answer 2

Quick checklist:

1 ) SSH into your instance.
2 ) Run cat on /var/log/cloud-init-output.log and dump it into a nice text file.
3 ) You'll see multiple block which contains the structure below:

Dependencies Resolved

================================================================================
 Package                    Arch      Version               Repository     Size
================================================================================
Installing:


Transaction Summary
================================================================================
Install  X Package (+Y Dependent packages)

Each block is created according to the user data which you provided to the EC2 instance (All scripts are stored under /var/lib/cloud/instance/scripts).

4 ) Search for keywords like Failed, Error, WARNING or /var/lib/cloud/instance/scripts/.

For example in case of the error below:

/var/lib/cloud/instance/scripts/part-001: line 10: vncpasswd: command not found
cp: cannot stat '/lib/systemd/system/[email protected]': No such file or directory
sed: can't read /etc/systemd/system/[email protected]: No such file or directory
Failed to execute operation: No such file or directory
Failed to start vncserver@:1.service: Unit not found.
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Cleaning repos: amzn2-core amzn2extra-docker amzn2extra-epel

---

(*) You can also quickly review the end of /var/log/cloud-init-output.log or /var/log/cloud-init.log which contain general error messages which will inform you about the fact that an error has occurred while executing the user data scripts.

The end of /var/log/cloud-init-output.log in the error above:

Is this ok [y/d/N]: Exiting on user command
Your transaction was saved, rerun it with:
 yum load-transaction /tmp/yum_save_tx.2020-08-31.15-14.VpTvV1.yumtx
Aug 31 15:14:00 cloud-init[3532]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
Aug 31 15:14:00 cloud-init[3532]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
Aug 31 15:14:00 cloud-init[3532]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
Cloud-init v. 19.3-3.amzn2 finished at Mon, 31 Aug 2020 15:14:00 +0000. Datasource DataSourceEc2.  Up 87.44 seconds

And the end of /var/log/cloud-init.log:

Aug 31 15:14:00 cloud-init[3532]: util.py[DEBUG]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 910, in runparts
        subp(prefix + [exe_path], capture=False, shell=True)
      File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 2105, in subp
        cmd=args)
    ProcessExecutionError: Unexpected error while running command.
    Command: ['/var/lib/cloud/instance/scripts/part-001']
    Exit code: 1
    Reason: -
    Stdout: -
    Stderr: -
    cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)