Setting up rsync backups with the backup plan
Posted by - NA -, Last modified by Kevin Stange on 03 November 2010 04:32 PM
|
|
For the incremental backup of a large number of files, rsync is often an ideal method. Each time the process runs, it will only copy the files that have changed since the last sync was run. This saves time and bandwidth and can be run in both directions relatively easily, making it fairly simple to restore the data onto a new server. It also optionally preserves file permissions, links, and other special data.
For the security of everyone's data, we have decided not to allow plain (unencrypted) rsync connections to our backup server. Instead, we encourage users to use rsync over SSH. This is relatively easy to set up, and it ensures that users are logged in as themselves when they upload data, preventing any possibility of corruption of other users' data. Since SSH is encrypted, all data transferred has the added benefit of being secure. On your server, you will want to set up a command such as the following to be run on a schedule, such as with cron. You can run multiple commands on different sets of files (rather than backing up your entire system), though we recommend that you space apart the timing on each run, so that you do not bog down your server or our backup server too much. rsync -au --stats --delete --rsh=ssh /local/path/to/files/ ftpuser@backup.steadfast.net:subdir/
You can customize the parameters however you prefer to properly sync your data. The ones we have given are the ones we use for our own backup processes. For more information about the options you can use with rsync, run "man rsync" from your server. The last parameter "--rsh=ssh" is required. It directs your server to connect via SSH. You should replace the text "ftpuser" with the FTP username contained in the welcome email you received when you signed up for a backup account. The "subdir/" part is optional, and will place the synced files in that subdirectory of your home directory, which is ideal if you are backing up sets of data from various places. The ending ":" is required even if you leave out the subdirectory part. When you connect with rsync via SSH, you are prompted for your password. To work around this "problem" for a scheduled task, you need to create and upload an SSH key owned by the user that will run rsync. As that user on your server, run: ssh-keygen -t dsa
Make sure this creates .ssh/id_dsa and .ssh/id_dsa.pub in the user's home directory. Do not set a password for the key, or you'll still have to enter a password each time you sync. Now, use FTP or SFTP to connect to the backup server using the details you received in your welcome email. Upload the "id_dsa.pub" file you created earlier as a file named "authorized_keys2" inside the .ssh directory on the backup server. You may need to tell your FTP client to show hidden files, or run "ls -a" via SSH to see the .ssh directory, or create it as follows via SSH if it does not exist. mkdir .ssh
You should now be able to run the rsync command successfully. If you have any problems completing these steps, please contact support for assistance. | |
|
$ mkdir .ssh
$ mv ~/id_dsa.pub ~/.ssh/authorized_keys
After that, everything was great.
The Steadfast Support staff was very helpful
Follow the directions above to create the key needed to be able to schedule the rsync
Login to your server as user that has rights to copy all needed files, create key ssh-keygen -t dsa
Copy the CONTENTS of .ssh/id_dsa.pub to the root of your home directory on the backup server as a file called authorized_keys2 in a root folder called .ssh
Then I used this rsync command -( make sure its all one one line )
/usr/bin/rsync -au --stats --delete --exclude-from=rsync_exclude.txt --rsh=ssh / username@backup.steadfast.net:folder_name/
Its designed to copy all the files on the server to the backup server EXCEPT the folders listed in the rsync_exclude.txt file. You need to put the username that Steadfast sent you in place of “username†. The files will be copied to a folder in your home directory on the backup server called folder_name/
The format of the rsync_exclude.txt is
/home/
/sys/bus/pci
/tmp/
#/dev/
The folders /home and /sys/bus/pci will be skipped and the #/dev folder will not
In my case I was doing it simple so the rsync_exclude.txt is in the root
I saved the rsync command above to a file in the root called backup_other_than_home.txt
Then I added a file to /etc/cron.d called serverbackup that contained the following lines
MAILTO=root
SHELL=/bin/sh
4 4 * * Sun root /backup_other_than_home.txt > /dev/null
1 4 * * Sat root /backuphome.txt > /dev/null
The files listed backup half the server on sat night at 4:01 am and the other half on Sunday at 4:04 am
The quantities of files can be tweaked by editing the rsync_exclude.txt
All but the format of the cron has been tested.
The file authorized_keys2 on backup server must be chmod 600 otherwise it will not work.
It is also easier NOT to type any name if you generate the key and just hit enter for name, enter for passphrase and again enter.
scp ~/.ssh/id_dsa.pub ftpuser@backup.steadfast.net:.ssh/authorized_keys2