File transferring with SCP and Rclone

Transferring files to and from your cloud instances is a common task when working with cloud computing. In this guide, we'll cover two powerful methods for file transfers:

• SCP (Secure Copy Protocol): A simple, built-in utility that comes with SSH for transferring individual files or directories
• Rclone: A more advanced tool that supports many cloud storage providers and offers more features for synchronizing directories

SCP (Secure Copy Protocol) is a command-line utility that allows you to securely transfer files between computers using SSH encryption.

Prerequisites

• SSH access to your instance (see How to SSH into your instance)
• The public IP address of your instance
• Username and SSH key or password for your instance

Basic SCP Usage

Copying a file from your local machine to the instance

scp /path/to/local/file username@instance_ip:/path/on/instance

Example:

scp data.csv ubuntu@192.168.1.100:/home/ubuntu/data/

Copying a file from the instance to your local machine

scp username@instance_ip:/path/on/instance /path/to/local/destination

Example:

scp ubuntu@192.168.1.100:/home/ubuntu/results.csv ./downloads/

Copying an entire directory recursively

scp -r /path/to/local/directory username@instance_ip:/path/on/instance

Example:

scp -r ./project ubuntu@192.168.1.100:/home/ubuntu/

Advanced SCP Options

• -P port: Specify a custom SSH port (e.g., scp -P 2222 file.txt user@host:/path)
• -i identity_file: Specify an SSH private key file (e.g., scp -i ~/.ssh/my_key file.txt user@host:/path)
• -C: Enable compression for faster transfers over slow connections
• -l limit: Limit bandwidth in Kbit/s

Rclone is a command-line program to sync files and directories between cloud storage services and your local machine. It's more powerful than SCP and supports many cloud providers.

Installing Rclone

On Linux

curl https://rclone.org/install.sh | sudo bash

On macOS (with Homebrew)

brew install rclone

On Windows

Download the Windows installer from rclone.org/downloads

Configuring Rclone for SFTP

To transfer files to your instance using SFTP via Rclone:

1. Run rclone config to create a new remote
2. Choose a name for your remote (e.g., "my-instance")
3. Select "sftp" as the storage type
4. Enter your instance's IP address as the host
5. Enter your username
6. Choose your authentication method (key file or password)
7. Save the configuration

Basic Rclone Usage

Copying files to your instance

rclone copy /local/path remote-name:/remote/path

Example:

rclone copy ./data my-instance:/home/ubuntu/data

Copying files from your instance

rclone copy remote-name:/remote/path /local/path

Example:

rclone copy my-instance:/home/ubuntu/results ./downloads

Synchronizing directories

rclone sync /local/path remote-name:/remote/path

Example (this will make the remote directory identical to the local one):

rclone sync ./project my-instance:/home/ubuntu/project

• Compress large files or directories before transferring to save bandwidth and time
• Use bandwidth limiting when transferring large amounts of data to avoid network saturation
• For repeated transfers, consider setting up SSH keys to avoid typing passwords
• When transferring sensitive data, ensure your connection is secure (both SCP and Rclone with SFTP use SSH encryption)
• For very large datasets, consider using object storage solutions like AWS S3 or Google Cloud Storage
• Schedule large transfers during off-peak hours to minimize impact on other users