aws s3 command does not work in a batch file triggered by a Windows task scheduler

Question

I have a batch file C:\upload_to_s3.bat. In this file, there is a line:

aws s3 sync D:\S3\batch1\ s3://MyBucket/batch1 --exclude *.bat

I have Windows task scheduler "S3 Hourly Sync" that runs every hour to trigger to run C:\upload_to_s3.bat. But this command does not do anything - the file upload never happened.

It runs perfectly if I double click on C:\upload_to_s3.bat.

This is Windows 2008 Standard server. I have installed AWS CLI and configured with the command "aws configure", and entered my access key and secret key. That is why it runs if I double click the batch file.

The Windows Task Scheduler is run as "[MyServer]/Administrator" account.

So why doesn't the "aws s3 sync" command work when triggered by the task scheduler?

Do I need to add a line before "aws s3 sync" to set credentials first? If so, how?

Thank you!

Answer 1

The reason this is happening, is because aws command is not finding the the config or the credentials file! And it happens for Win2012 too.

The problem is that Windows Task Schedule sets the %USERPROFILE% to C:\Users\Default instead of the actual user the task is running under!! ??? And AWS is using that to find the config and credentials files.

So the fix is to put something like this in your script (either BAT or PowerShell)

:: in a BAT script
set userprofile=C:\Users\%username%

# in a PowerShell
$env:USERPROFILE = "c:\users\$env:USERNAME"

Even the %homepath% variable is blank when running from a Scheduled Task.

I just discovered this after procrastinating trying to figure out why this was happening for us.

set AWS_DEFAULT_PROFILE=namedprofile
set AWS_CONFIG_FILE=C:\Users\xxxx\.aws\config

Only until I added the AWS_CONFIG_FILE line above in my script was AWS able to actually see the config file! However it can not find the credentials now, and I don't see anywhere how to set that.

This is the error I get without telling where the config file is:

Traceback (most recent call last):
  File "aws", line 27, in <module>
  File "aws", line 23, in main
  File "awscli\clidriver.pyc", line 50, in main
  File "awscli\clidriver.pyc", line 176, in main
  File "awscli\clidriver.pyc", line 157, in _create_parser
  File "awscli\clidriver.pyc", line 91, in _get_command_table
  File "awscli\clidriver.pyc", line 111, in _build_command_table
  File "botocore\session.pyc", line 680, in emit
  File "botocore\hooks.pyc", line 226, in emit
  File "botocore\hooks.pyc", line 209, in _emit
  File "awscli\customizations\preview.pyc", line 71, in mark_as_preview
  File "awscli\clidriver.pyc", line 349, in service_model
  File "awscli\clidriver.pyc", line 366, in _get_service_model
  File "botocore\session.pyc", line 256, in get_config_variable
  File "botocore\session.pyc", line 277, in _found_in_config_file
  File "botocore\session.pyc", line 349, in get_scoped_config
botocore.exceptions.ProfileNotFound: The config profile (backup) could not be found

And then this is the error I get after setting the AWS_CONFIG_FILE variable:

Unable to locate credentials
Completed 1 part(s) with ... file(s) remaining

Answer 2

i was able to solve the problem by adding the following 3 lines in the batch file BEFORE your s3 command

aws configure set AWS_ACCESS_KEY_ID <your acess key id>
aws configure set AWS_SECRET_ACCESS_KEY <your secret access key here>
aws configure set default.region eu-west-1

replace the stuff between angled brackets with your data and the region with your bucket's region. and yes ,there is no equal sign between "AWS_ACCESS_KEY_ID" and the key.

Answer 3

Include the following lines in your batch file:

set AWS_ACCESS_KEY_ID=your_access_key_id

set AWS_SECRET_ACCESS_KEY=your_secret_access_key

AWS Documentation