AWS Transfer for SFTP

The Secure File Transfer Protocol (SFTP) is a widely used method to exchange data between third parties and inside enterprises into and out of AWS S3 storage. A fully managed AWS service that helps to migrate SFTP workflows to AWS, and enables authorized clients to transfer files directly into your Amazon S3 bucket. Reference: AWS Docs; User Guide.

Possible Use Cases:

1) External transfers from third parties: Vendors, Partners.

2) Internal transfers within the Organization (compare with Storage Gateway use case)

Benefits:

1.  Fully Managed. No Infrastructure setup.
2.  Scales in Real time to meet the transfer needs.
3.  The data in S3 can be used for processing, analytics, reporting, archival etc.
4.  Cost: Pay per use
5.  Recently enabled with PrivateLink, where traffic movement is within your network and your customer's VPC and not over the internet making it more secure.
6.  User creds can be established with an identity management system.

Considerations:

1) Very expensive. $200-$400 a month. Negotiate with AWS.

Implementation Pre-requisites:

1) S3 Bucket: AWS Transfer for SFTP accesses your Amazon S3 bucket to service your users' transfer requests.

                      Enable "Default Encryption" from bucket properties and choose your desired encryption method either  "AES-256" or "AWS-KMS" for which the IAM Policy should be updated with the appropriate key.

                      Remove the Deny statement to upload (PutObject) requests to this bucket from the bucket policy since "Transfer for SFTP" service doesn't have a mechanism to select the encryption method. Refer to AWS Support Case on this.

2) Create IAM Policies and Roles for SFTP

                     a) Create an IAM Role and establish a trust relationship with AWS Transfer for SFTP.

                     Note: When you create a role, select EC2 at first since AWS Transfer for SFTP is not available, then you can modify the "Trust Relationships" by clicking on the Tab and selecting the "Edit Trust Relationship"

{  

  "Version": "2012-10-17",  

  "Statement": [    

{       "Effect": "Allow",      

"Principal": {         "

Service": "transfer.amazonaws.com"      

 },      

"Action": "sts:AssumeRole"    

}  

  ]

}      

b) Create an IAM Policy that grants read-write access to objects in the S3 bucket created in step 1. 

c) Add the Policy to the role.       

3)     SSH Keys

        a) Create SSH Keys as described here.

Example:

Create/Utilize an EC2 Instance that has SFTP Client installed  by default. SSH to the instance.

If you don't want a passphrase use -P ""

If you don’t want to store the key files under the default location use the -f option. Apart from storing it in a different directory, you can also specify your own name for the key files.

[root@ip-10-154-183-227 keys]# ssh-keygen -P "" -f "test-sftp-key"

   b. cat the public key "test-sftp-key.pub" and copy the contents to notepad to edit the key in a single line and remove the extragenous characters

Manual setup of SFTP Server

1) Login to the AWS Console and select "AWS Transfer for SFTP" Service.

2) Select "Create Server"

3) Select the below configurations for a Public End Point Type

4) While the server is staring, click on the "Server ID" and add a user with the below configuration. The home directory is the corresponding S3 Bucket, the role and the Public key created in Step 1, 2 and 3 in the Pre-requisites section above.

Test SFTP

 SSH (e.g. from NoMachine: ssh -i ./EMR/keys/anaconda/anaconda-key.pem centos@10.154.183.153) to the EC2 Linux Instance where the private key is held and: (Tested on a Centos Machine)

        a)  [root@ip-10-154-183-227 keys]# ls -ltr

total 16

-rw-------. 1 root root 1675 Sep  5 10:09 test-sftp-key

-rw-r--r--. 1 root root  435 Sep  5 10:09 test-sftp-key.pub 

 -rwxr--r--. 1 root root   21 Sep  5 11:47 test.txt

-rwxrwxr-x. 1 root root   27 Sep  5 12:36 sftp

Update the permissions on the private key:

chmod 600 ./test-sftp-key

b) SFTP → Remember this action has to be performed as a root user.

UserId: sftptestuser

EndPoint: s-2891724dd72840a2b.server.transfer.us-east-1.amazonaws.com

[root@ip-10-154-183-227 keys]# sftp -i ./test-sftp-key sftptestuser@s-2891724dd72840a2b.server.transfer.us-east-1.amazonaws.com

for RHLS:  sftp -oIdentityFile=./test-sftp-key sftptestuser@s-2891724dd72840a2b.server.transfer.us-east-1.amazonaws.com

Connected to s-2891724dd72840a2b.server.transfer.us-east-1.amazonaws.com .

c) sftp> pwd

Remote working directory: /anaconda-vgi-ops-int-us-east-1-poc

d) sftp> lpwd

Local working directory: /keys

                e) sftp> help

                f) sftp> put sftp

Uploading sftp to /anaconda-vgi-ops-int-us-east-1-poc/sftp sftp            100%   27     2.5K s

g) sftp> rm sftp

Removing /anaconda-vgi-ops-int-us-east-1-poc/sftp

h) sftp> bye

Other Commands:

sftp> get { File on the remote server }, Local commands: lls, lcd, lpwd