Setting up rsync backups with the backup plan

Knowledgebase

38Steadfast Cloud Platform 3Full Management 37Dedicated Servers & Colocation 5Control Panels 19Other

Knowledgebase: Other

Posted by - NA -, Last modified by Kevin Stange on 03 November 2010 04:32 PM

For the incremental backup of a large number of files, rsync is often an ideal method. Each time the process runs, it will only copy the files that have changed since the last sync was run. This saves time and bandwidth and can be run in both directions relatively easily, making it fairly simple to restore the data onto a new server. It also optionally preserves file permissions, links, and other special data.

For the security of everyone's data, we have decided not to allow plain (unencrypted) rsync connections to our backup server. Instead, we encourage users to use rsync over SSH. This is relatively easy to set up, and it ensures that users are logged in as themselves when they upload data, preventing any possibility of corruption of other users' data. Since SSH is encrypted, all data transferred has the added benefit of being secure.

On your server, you will want to set up a command such as the following to be run on a schedule, such as with cron. You can run multiple commands on different sets of files (rather than backing up your entire system), though we recommend that you space apart the timing on each run, so that you do not bog down your server or our backup server too much.

rsync -au --stats --delete --rsh=ssh /local/path/to/files/ ftpuser@backup.steadfast.net:subdir/

You can customize the parameters however you prefer to properly sync your data. The ones we have given are the ones we use for our own backup processes. For more information about the options you can use with rsync, run "man rsync" from your server. The last parameter "--rsh=ssh" is required. It directs your server to connect via SSH.

You should replace the text "ftpuser" with the FTP username contained in the welcome email you received when you signed up for a backup account.

The "subdir/" part is optional, and will place the synced files in that subdirectory of your home directory, which is ideal if you are backing up sets of data from various places. The ending ":" is required even if you leave out the subdirectory part.

When you connect with rsync via SSH, you are prompted for your password. To work around this "problem" for a scheduled task, you need to create and upload an SSH key owned by the user that will run rsync. As that user on your server, run:

ssh-keygen -t dsa

Make sure this creates .ssh/id_dsa and .ssh/id_dsa.pub in the user's home directory.

Do not set a password for the key, or you'll still have to enter a password each time you sync. Now, use FTP or SFTP to connect to the backup server using the details you received in your welcome email. Upload the "id_dsa.pub" file you created earlier as a file named "authorized_keys2" inside the .ssh directory on the backup server. You may need to tell your FTP client to show hidden files, or run "ls -a" via SSH to see the .ssh directory, or create it as follows via SSH if it does not exist.

mkdir .ssh

chmod 700 .ssh

You should now be able to run the rsync command successfully. If you have any problems completing these steps, please contact support for assistance.

(415 vote(s))

Helpful

Not helpful

Comments (4)

Jon Chambers

25 October 2006 01:11 AM

I had a little trouble getting the SSH key authentication thing to work. I needed to do this on the backup box:

$ mkdir .ssh
$ mv ~/id_dsa.pub ~/.ssh/authorized_keys

After that, everything was great.

Doug B

19 February 2007 01:12 AM

Here is what worked for me - I knew NOTHING about rsync when I started -
The Steadfast Support staff was very helpful

Follow the directions above to create the key needed to be able to schedule the rsync

Login to your server as user that has rights to copy all needed files, create key ssh-keygen -t dsa
Copy the CONTENTS of .ssh/id_dsa.pub to the root of your home directory on the backup server as a file called authorized_keys2 in a root folder called .ssh

Then I used this rsync command -( make sure its all one one line )
/usr/bin/rsync -au --stats --delete --exclude-from=rsync_exclude.txt --rsh=ssh / username@backup.steadfast.net:folder_name/

Its designed to copy all the files on the server to the backup server EXCEPT the folders listed in the rsync_exclude.txt file. You need to put the username that Steadfast sent you in place of â€œusernameâ€ . The files will be copied to a folder in your home directory on the backup server called folder_name/

The format of the rsync_exclude.txt is

/home/
/sys/bus/pci
/tmp/
#/dev/

The folders /home and /sys/bus/pci will be skipped and the #/dev folder will not

In my case I was doing it simple so the rsync_exclude.txt is in the root
I saved the rsync command above to a file in the root called backup_other_than_home.txt

Then I added a file to /etc/cron.d called serverbackup that contained the following lines

MAILTO=root
SHELL=/bin/sh
4 4 * * Sun root /backup_other_than_home.txt > /dev/null
1 4 * * Sat root /backuphome.txt > /dev/null

The files listed backup half the server on sat night at 4:01 am and the other half on Sunday at 4:04 am

The quantities of files can be tweaked by editing the rsync_exclude.txt

All but the format of the cron has been tested.

Frank Coetzee

06 August 2007 11:15 PM

I struggle for hours until I got the problem.

The file authorized_keys2 on backup server must be chmod 600 otherwise it will not work.

It is also easier NOT to type any name if you generate the key and just hit enter for name, enter for passphrase and again enter.

Ivan Zhekov

25 June 2010 03:37 AM

The easiest way to upload the the key file directly in place at the backupserver is:
scp ~/.ssh/id_dsa.pub ftpuser@backup.steadfast.net:.ssh/authorized_keys2