AWS Transfer for SFTP
The Secure File Transfer Protocol (SFTP) is a widely used method to exchange data between third parties and inside enterprises into and out of AWS S3 storage. A fully managed AWS service that helps to migrate SFTP workflows to AWS, and enables authorized clients to transfer files directly into your Amazon S3 bucket. Reference: AWS Docs; User Guide.
Possible Use Cases:
1) External transfers from third parties: Vendors, Partners.
2) Internal transfers within the Organization (compare with Storage Gateway use case)
Benefits:
Considerations:
1) Very expensive. $200-$400 a month. Negotiate with AWS.
Implementation Pre-requisites:
1) S3 Bucket: AWS Transfer for SFTP accesses your Amazon S3 bucket to service your users' transfer requests.
Enable "Default Encryption" from bucket properties and choose your desired encryption method either "AES-256" or "AWS-KMS" for which the IAM Policy should be updated with the appropriate key.
Remove the Deny statement to upload (PutObject) requests to this bucket from the bucket policy since "Transfer for SFTP" service doesn't have a mechanism to select the encryption method. Refer to AWS Support Case on this.
2) Create IAM Policies and Roles for SFTP
a) Create an IAM Role and establish a trust relationship with AWS Transfer for SFTP.
Note: When you create a role, select EC2 at first since AWS Transfer for SFTP is not available, then you can modify the "Trust Relationships" by clicking on the Tab and selecting the "Edit Trust Relationship"
{
"Version": "2012-10-17",
"Statement": [
{ "Effect": "Allow",
"Principal": { "
Service": "transfer.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
b) Create an IAM Policy that grants read-write access to objects in the S3 bucket created in step 1.
c) Add the Policy to the role.
3) SSH Keys
a) Create SSH Keys as described here.
Example:
Create/Utilize an EC2 Instance that has SFTP Client installed by default. SSH to the instance.
If you don't want a passphrase use -P ""
If you don’t want to store the key files under the default location use the -f option. Apart from storing it in a different directory, you can also specify your own name for the key files.
[root@ip-10-154-183-227 keys]# ssh-keygen -P "" -f "test-sftp-key"
b. cat the public key "test-sftp-key.pub" and copy the contents to notepad to edit the key in a single line and remove the extragenous characters
Recommended by LinkedIn
Manual setup of SFTP Server
1) Login to the AWS Console and select "AWS Transfer for SFTP" Service.
2) Select "Create Server"
3) Select the below configurations for a Public End Point Type
4) While the server is staring, click on the "Server ID" and add a user with the below configuration. The home directory is the corresponding S3 Bucket, the role and the Public key created in Step 1, 2 and 3 in the Pre-requisites section above.
Test SFTP
SSH (e.g. from NoMachine: ssh -i ./EMR/keys/anaconda/anaconda-key.pem centos@10.154.183.153) to the EC2 Linux Instance where the private key is held and: (Tested on a Centos Machine)
a) [root@ip-10-154-183-227 keys]# ls -ltr
total 16
-rw-------. 1 root root 1675 Sep 5 10:09 test-sftp-key
-rw-r--r--. 1 root root 435 Sep 5 10:09 test-sftp-key.pub
-rwxr--r--. 1 root root 21 Sep 5 11:47 test.txt
-rwxrwxr-x. 1 root root 27 Sep 5 12:36 sftp
Update the permissions on the private key:
chmod 600 ./test-sftp-key
b) SFTP → Remember this action has to be performed as a root user.
UserId: sftptestuser
[root@ip-10-154-183-227 keys]# sftp -i ./test-sftp-key sftptestuser@s-2891724dd72840a2b.server.transfer.us-east-1.amazonaws.com
for RHLS: sftp -oIdentityFile=./test-sftp-key sftptestuser@s-2891724dd72840a2b.server.transfer.us-east-1.amazonaws.com
Connected to s-2891724dd72840a2b.server.transfer.us-east-1.amazonaws.com .
c) sftp> pwd
Remote working directory: /anaconda-vgi-ops-int-us-east-1-poc
d) sftp> lpwd
Local working directory: /keys
e) sftp> help
f) sftp> put sftp
Uploading sftp to /anaconda-vgi-ops-int-us-east-1-poc/sftp sftp 100% 27 2.5K s
g) sftp> rm sftp
Removing /anaconda-vgi-ops-int-us-east-1-poc/sftp
h) sftp> bye
Other Commands:
sftp> get { File on the remote server }, Local commands: lls, lcd, lpwd