Rclone Amazon Photos

Rclone is an open-source command-line program used to sync files and directories (folders) to and from over 20 cloud storages ranging from Google Drive, Google Cloud Storage, Microsoft OneDrive, Google Photos, MEGA, OpenDrive, Box, ownCloud, webDAV, Yandex Disk, OVH, Local filesystem, Dropbox, Amazon S3, Backblaze B2, Amazon Drive, Alibaba. Apr 14, 2020 Amazon Cloud Drive is very similar to services like SkyDrive, Google Drive and Dropbox, all of which offer a limited amount of free storage. The Amazon Cloud Drive client allows you to easily upload photos, documents, videos and other files via a simple drag and drop interface. I'd really prefer Amazon Drive (which is the same storage mechanism as Amazon Photo Storage, free unlimited photos with amazon prime), but of course the current fiasco of rclone being banned will have to be sorted. If I can help I hope to figure out how. In the mean time, it would be nice to consolidate all the duplicate issues.

Rclone is a great tool. From the developer “Rclone is a command line program to sync files and directories to and from different cloud services”. It is like rsync, but supports cloud storage services. Here are a few options:. Amazon S3. Dropbox.

What's Amazon Cloud Drive?

Do you have Amazon Prime? Do you have hundreds of gigs of raw photos or photoshop files? Back them up to Amazon Cloud Drive for free!

What's the catch?

Sound great? Here's the catch- it's a pain to upload files to Amazon Cloud Drive. On iOS and Android you can sink your photos automatically, and there's a PC app, but for Mac you're dragging and dropping files on to your browser. No fun if you're talking about ten thousand files.

Okay, how can I upload them then?

That's where drivesink comes in- it's a command line tool that will synchronize a directory from your computer to Cloud Drive. Run it whenever you want and it will upload any new or changed files.

What else is this site?

Cloud Drive has a pretty rich API available. However, if you want to use it at all, even for your own use, you have to jump through several hoops and submit an app for approval. Or you can just get your access token right here.

Get started

  • First download the script:
  • Then get your access configuration by logging in with Amazon
  • Finally, synchronize your files whenever you feel like it:

Rclone Overview

Rclone provides a modern alternative to rsync. It is able to communicate with any S3 compatible cloud storage provider as well as other storage platforms, and can be used to migrate data from one bucket to another, even if those buckets are in different regions.

Requirements

  • You have an account and are logged into console.scaleway.com
  • You have configured your SSH Key
  • You have generated your API Key
  • You have at least two object storage buckets

Installing Rclone

1 . Connect to your server as root via SSH.

2 . Update the APT packet cache and the software already installed on the instance:

3 . Download and install Rclone with the following sequence of commands:

Configuring Rclone

1 . Begin rclone configuration with the following command:

If you do not have any existing remotes, the following output displays:

If you have previously configured rclone you may see a slightly different output. However, that does not affect the following steps.

2 . Type n to make a new remote. You are then prompted to type a name - here we type remote-sw-paris:

The following output displays:

3 . Type s3 and hit enter to confirm this storage type. The following output displays:

Photos

4 . Type Scaleway and hit enter to confirm this S3 provider. Sourcetree cheat sheet. The following output displays:

5 . Type false and hit enter, to be able to enter your credentials in the next step.

The following output displays:

6 . Enter your API Access Key and hit enter.

The following output displays:

7 . Enter your API Secret Key and hit enter.

The following output displays:

Amazon

Amazon Photos Desktop App

8 . Enter your chosen region and hit enter. Here we choose fr-par.

The following output displays:

9 . Enter your chosen endpoint and hit enter. Here we choose s3.fr-par.scw.cloud.

The following output displays:

10 . Enter your chosen ACL and hit enter. Here we choose private (1).

The following output displays:

11 . Enter your chosen stoage class and hit enter. Here we choose STANDARD (2).

The following output displays:

12 . Type n and hit enter. A summary of your config displays:

Rclone Amazon Photos Free

13 . Type y to confirm that this remote config is OK, and hit enter.

The following output displays:

14 . Type q to quit the config, and hit enter.

15 . If you want to be able to transfer data to or from a bucket in a different region to the one you just set up, repeat steps 1-14 again to set up a new remote in the required region. Simply enter the required region at steps 7 and 8. Similarly, you may wish to set up a new remote for a different object storage provider.

Note: For further information, please refer to the official RClone S3 Object Storage Documentation. Official documentation also exists for other storage backends

Migrating data

There are two commands that can be used to migrate data from one backend to another.

  • The copy command copies data from source to destination.

For example, the following command copies data from a bucket named my-first-bucket in the remote-sw-paris remote backend that we previously set up, to another bucket named my-second-bucket in the same remote backend. The --progress flag allows us to follow the progress of the transfer:

  • The sync command copies data from one one backend to another, but also deletes files/objects in the destination that are not present in the source:

For example, the following command copies data from a bucket named my-first-bucket in the remote-sw-paris remote backend that we previously set up, to another bucket named my-third-bucket in a different remote backend that we configured for the nl-ams region and named remote-sw-ams. It also deletes any data that present in my-third-bucket that isn’t also present in my-first-bucket:

Note: this migration may incur some costs from the object storage you are migrating from since they may or may not bill egress bandwitdth.

There are other commands such as move, which progressively deletes data from the source backend.

Transferring data to C14 Cold Storage

When you copy or sync, your can determine which storage class you wish to transfer your data as.

At Scaleway Elements you can choose from two classes:

  • STANDARD: The Standard class for any upload; suitable for on-demand content like streaming or CDN.
  • GLACIER: Archived, long-term retention storage;

Amazon Drive

If the storage class is not specified, the data will be transferred as STANDARD by default.

To transfer data to C14 Cold Storage class, add

to your command, as such:

Rclone Amazon Photos Online

You can verify the storage class of the transferred data by accessing your bucket on the Scaleway Elements console.