Upload Files Into Aws From Folder on Desktop

Photograph past Daniel Eledut on Unsplash

How To Upload and Sync Files To An AWS S3 Bucket

Comparing AWS CLI and AWS PowerShell Tools for data migration

The AWS CLI and AWS PowerShell Tools are powerful open-source tools that permit you to interact with AWS services via your command-line trounce.

Beneath I volition provide links to the official documentation for each tool and the virtually basic commands y'all will demand to go started. I volition also include some possible roadblocks and solutions or workarounds when using these tools to upload files.

If you wish to know how they compare, please scroll to the last section of the article.

The goal is to provide a quick reference for anyone who is looking to install and use these tools as well as assistance with the decision-making process in choosing which ane is better suited for a data migration.

Prerequisites

• AWS S3 bucket and the saucepan name

• An Admission Fundamental and Secret Cardinal pair to your AWS account

AWS CLI

Installation

Open up your concluding (Mac) or command prompt (Windows).

Mac

Run the following commands. When it prompts you for your password, enter the password for your computer. Annotation: using sudo will install the CLI for all users, simply avoids path problems.

          $            curl "            https://awscli.amazonaws.com/AWSCLIV2.pkg            " -o "AWSCLIV2.pkg"            
            % Total % Received % Xferd Boilerplate Speed Time Time Time Current
            Dload Upload Total Spent Left Speed
100 23.8M 100 23.8M 0 0 x.2M 0 0:00:02 0:00:02 — : — : — 10.2M          $            sudo installer -pkg AWSCLIV2.pkg -target /            
Password:
installer: Parcel name is AWS Command Line Interface
installer: Installing at base of operations path /
installer: The install was successful.        

Windows

Click on the link below and download the .msi file listed in Stride one. It should be fairly quick. Once downloaded, click on the install package and a screen will pop up to guide you through the installation. You can keep all the default options.

Once installed, for either operating system, run this command and run into if it throws an error.

          $            aws --version
            aws-cli/2.i.28 Python/three.8.8 Darwin/20.iii.0 exe/x86_64 prompt/off        

If there is an error:

• Windows: close and reopen your command prompt to refresh the path
• Mac: run which aws to find out where it was installed and add it to your path
• If information technology still doesn't piece of work, copy and paste the error into your Google search bar, hit enter, and click on the first few links. One of them hopefully contains the answer. 🙏🏼

If it tells you which version you have, congrats!

Now it's time to configure your AWS bucket with your access central and undercover access key. Run aws configure and it will prompt y'all for everything it needs. (You won't need the bucket name just nevertheless. This is simply so that the CLI knows which AWS account it'due south using.)

          $            aws configure            
AWS Access Cardinal ID [None]:            AKFIE8F0DL2LK0EXAMPLE            
AWS Secret Admission Key [None]:wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY            Default region name [None]:            united states of america-west-2            
Default output format [None]:            json (tin leave blank, simply hitting enter)                  

Upload and Sync Files

To upload and sync your files, run the command beneath. You volition need your saucepan proper noun and the file path of your local directory.

Tip: Make sure your command line is in the correct binder or use an absolute path. If you are getting confused, using an absolute path is the easiest because it contains the root directory.

Instance of absolute paths:
Mac: '~/Desktop/FolderToUpload'
Windows: 'C:\Users\Venessa\Desktop\FolderToUpload'

Mac

          $            aws s3 sync ./FOLDER_PATH s3://BUCKET_NAME/SUB_DIRECTORY                  

Windows

          $            aws s3 sync .\FOLDER_PATH s3://BUCKET_NAME/SUB_DIRECTORY                  

Too, if you noticed, y'all don't need the slash at the end of the path for your local directory (Windows or Mac), but it also doesn't hurt.

Tip: Yous will know when it is washed when yous regain your control of the control-line session and when you don't see "Completed…" anymore. This confused me at first when information technology was uploading a large file and it was on "completed" for a long fourth dimension. I thought maybe information technology was done, only it was nevertheless syncing.

Nonetheless syncing…

Complete!

And…that'due south information technology for uploading and syncing files with AWS CLI! Super piece of cake, right?

If it breaks for some reason or your estimator reboots unexpectedly, you can easily restart the sync without duplicating files by running the full aws s3 sync command above. Information technology will option up where it left off!

Here is the documentation for all possible parameters:

AWS PowerShell Tools

Installation

Mac

To install PowerShell:

          $            brew install --cask powershell                  

Open up Powershell:

          $              pwsh                  

Windows

Click Start → type "powershell" → click on Windows PowerShell. Open up not every bit an administrator. Amazon explains:

We recommend that y'all don't run PowerShell as an administrator with elevated permissions except when required by the job at paw. This is because of the potential security take a chance and is inconsistent with the principle of least privilege.

One time yous have PowerShell open for either one, run the post-obit command:

          $            Install-Module -Name AWS.Tools.Installer                  

Then, fix credentials by running the command below. Notation: unlike with aws configure, there is no prompt here if you just run Set-AWSCredential, y'all will need to take everything typed out in one command.

          $            Set-AWSCredential `
              -AccessKey AKIA0123456787EXAMPLE `
              -SecretKey e51zRpSvNYzYe/Qmb6SHvC+FdNm6EXAMPLEKEY `
              -StoreAs default                  

Upload and Sync

To upload a file to the root folder of the S3 saucepan, you will need the bucket proper noun and the -File parameter, which will be the name of the file on your local automobile:

          $            Write-S3Object -BucketName saucepan -File file.txt                  

To upload to a specific location inside the S3 bucket, you lot tin use the -Fundamental parameter and specify the path as a string (this is non the access fundamental or secret admission primal).

          $            Write-S3Object -BucketName bucket -Key "SUBFOLDER/FILE.txt" -File file.txt                  

To sync a whole folder, use the -Binder parameter, which volition exist the path on your local machine. Once more, make certain you are in the correct folder or apply an absolute path.

Optional

Upload the binder to a subdirectory inside the S3 bucket by using the
-KeyPrefixparameter (this is not the access key or clandestine access cardinal).

Write-S3Object simply uploads the files within a folder. If you accept folders within a folder and desire to catch the files within those as well, you tin use the -Recurseparameter and set it to True.

Hither is what the full command may look like with all those options included:

          $            Write-S3Object -BucketName bucket -Folder .\Scripts -KeyPrefix Scripts\ -Recurse True                  

Hither is the documentation for all possible parameters:

So which one should you use for data migrations?

I would say the AWS CLI. Here's why:

Both are easy to install and get started. Y'all can apply either 1 for uploading single and batch files.

However, when it comes to syncing much larger quantities of files, the PowerShell tools may fall brusk compared to the AWS CLI.

They both sync in the sense that, at the stop of the day, your S3 bucket volition look like your local directory. The principal difference is that the PowerShell tool will re-upload them regardless if they accept been updated or imported already; whereas the CLI tool will only import a file if it hasn't been imported or it has been updated — this is especially advantageous if you are migrating data over time and new files are existence added.

The CLI tool is more efficient for large data migrations over time because it tin can pick upwardly correct where information technology left off while the PowerShell tool cannot.

One of the most useful features of the PowerShell tool is that you tin can write crush scripts to manage your AWS resources. Therefore, you could write a beat out script to overcome it, but it becomes deadening and inefficient, peculiarly if you could practice information technology with a single line of code using the built-in feature of the AWS CLI: aws sync.

I hope that I was able to help you lot get bound started with using these tools to sync your data. If you have any questions, please don't hesitate to ask!

wattsondur1976.blogspot.com

Source: https://levelup.gitconnected.com/how-to-upload-and-sync-files-to-an-aws-s3-bucket-52ddfcec5bc1

Share this post