Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Connecting to AWS Transfer for SFTP

I am having trouble connecting to AWS Transfer for SFTP. I successfully set up a server and tried to connect using WinSCP.

I set up an IAM role with trust relationships like follows:

{   "Version": "2012-10-17",   "Statement": [     {       "Effect": "Allow",       "Principal": {         "Service": "transfer.amazonaws.com"       },       "Action": "sts:AssumeRole"     }   ] } 

I paired this with a scope down policy as described in the documentation using a home directory homebucket and home directory homedir

{     "Version": "2012-10-17",     "Statement": [         {             "Sid": "ListHomeDir",             "Effect": "Allow",             "Action": [                 "s3:ListBucket",                 "s3:GetBucketAcl"             ],             "Resource": "arn:aws:s3:::${transfer:HomeBucket}"         },         {             "Sid": "AWSTransferRequirements",             "Effect": "Allow",             "Action": [                 "s3:ListAllMyBuckets",                 "s3:GetBucketLocation"             ],             "Resource": "*"         },         {             "Sid": "HomeDirObjectAccess",             "Effect": "Allow",             "Action": [                 "s3:DeleteObjectVersion",                 "s3:DeleteObject",                 "s3:PutObject",                 "s3:GetObjectAcl",                 "s3:GetObject",                 "s3:GetObjectVersionAcl",                 "s3:GetObjectTagging",                 "s3:PutObjectTagging",                 "s3:PutObjectAcl",                 "s3:GetObjectVersion"             ],             "Resource": "arn:aws:s3:::${transfer:HomeDirectory}*"         }     ] } 

I was able to authenticate using an ssh key, but when it came to actually reading/writing files I just kept getting opaque errors like "Error looking up homedir" and failed "readdir". This all smells very much like problems with my IAM policy but I haven't been able to figure it out.

like image 888
ChristopherTull Avatar asked Dec 05 '18 23:12

ChristopherTull


People also ask

How does SFTP work with AWS?

AWS Transfer for SFTP is a member of the AWS Transfer Family. It is a secure transfer service that you can use to transfer files into and out of AWS storage services over SFTP. You can use AWS Transfer for SFTP with Amazon Simple Storage Service (Amazon S3) or Amazon Elastic File System (Amazon EFS).

Can AWS glue connect to SFTP?

ConnectionType – UTF-8 string (valid values: JDBC | SFTP | MONGODB | KAFKA | NETWORK | MARKETPLACE | CUSTOM ). The type of the connection. Currently, SFTP is not supported.

How do I transfer from SFTP to AWS S3?

Download the AWS CLI onto the sFTP server and copy the files via the AWS s3 cp command. Write a script using the AWS SDK that takes the files and copies them. You may need to use the multi-part upload with the size of your files.


2 Answers

We had similar issues getting the scope down policy to work with our users on AWS Transfer. The solution that worked for us, was creating two different kinds of policies.

  • Policy to attach to the role which has general rights on the whole bucket.
  • Scope down policy to apply to the user which makes use of the transfer service variables like {transfer:UserName}.

We concluded that maybe only the extra attached policy is able to resolve the transfer service variables. We are not sure if this is correct and if this is the best solution, because this opens the possible risk when forgiving to attach the scope down policy to create a kind of "admin" user. So I'd be glad to get input to further lock this down a little bit.

Here is how it looks in my console when looking at the transfer user details: Transfer user detail view with extra policy attached

Here are our two policies we use:
General policy to attach to IAM role

{     "Version": "2012-10-17",     "Statement": [         {             "Sid": "AllowListingOfUserFolder",             "Action": [                 "s3:ListBucket",                 "s3:GetBucketLocation"             ],             "Effect": "Allow",             "Resource": [                 "arn:aws:s3:::my-s3-bucket"             ]         },         {             "Sid": "HomeDirObjectAccess",             "Effect": "Allow",             "Action": [                 "s3:PutObject",                 "s3:GetObject",                 "s3:DeleteObjectVersion",                 "s3:DeleteObject",                 "s3:GetObjectVersion"             ],             "Resource": "arn:aws:s3::: my-s3-bucket/*"         }     ] } 

Scope down policy to apply to transfer user

{     "Version": "2012-10-17",     "Statement": [         {             "Sid": "AllowListingOfUserFolder",             "Action": [                 "s3:ListBucket"             ],             "Effect": "Allow",             "Resource": [                 "arn:aws:s3:::${transfer:HomeBucket}"             ],             "Condition": {                 "StringLike": {                     "s3:prefix": [                         "${transfer:UserName}/*",                         "${transfer:UserName}"                     ]                 }             }         },         {             "Sid": "AWSTransferRequirements",             "Effect": "Allow",             "Action": [                 "s3:ListAllMyBuckets",                 "s3:GetBucketLocation"             ],             "Resource": "*"         },         {             "Sid": "HomeDirObjectAccess",             "Effect": "Allow",             "Action": [                 "s3:PutObject",                 "s3:GetObject",                 "s3:DeleteObjectVersion",                 "s3:DeleteObject",                 "s3:GetObjectVersion"             ],             "Resource": "arn:aws:s3:::${transfer:HomeDirectory}*"         }     ] } 
like image 53
limfinity Avatar answered Oct 17 '22 07:10

limfinity


I had a similar problem but with a different error behavior. I managed to log in successfully, but then the connection was almost immediately closed. I did the following things:

  • Make sure that the IAM role that allows bucket access also contains KMS access if your bucket is encrypted.
  • Make sure that the trust relationship is also part of that role.
  • Make sure that the server itself has a Cloudwatch role also with a trust relationship to transfer.amazonaws.com! This was the solution for me. I don't get why this is needed but without the trust relationship in the Cloudwatch role, my connection get's closed.

I hope that helps. Edit: Added a picture for the settings of the CloudWatch role: enter image description here

The bucket policy for the IAM user role can look like this:

{ "Version": "2012-10-17", "Statement": [     {         "Effect": "Allow",         "Action": [             "s3:ListBucket"         ],         "Resource": [             "arn:aws:s3:::<your bucket>"         ]     },     {         "Effect": "Allow",         "Action": [             "s3:PutObject",             "s3:GetObject",             "s3:DeleteObject"         ],         "Resource": [             "arn:aws:s3:::<your bucket>/*"         ]     } ] 

}

Finally, also add a Trust Relationship as shown above for the user IAM role.

If you can connect to your sftp but then get a readdir error when trying to list contents, e.g. with the command "ls", then that's a sign that you have no bucket permission. If your connection get's closed right away it seems to be a Trust Relationship issue or a KMS issue.

like image 28
Uwe Bretschneider Avatar answered Oct 17 '22 09:10

Uwe Bretschneider