RSYNC command – sync the data between two directory

Introduction

• I have to sync up the two directory many times on Unix boxes.The simple steps would be to delete the directory from destination and take a backup on source destination  and copy it to the destination directory.
• It was good for small directories but This process will consume lot of time  as directory size grow up.
• I searched around and find that The fastest way to do this process would be to use RSYNC utility
• It uses ‘rsync algorithm’ which provides a very fast method for syncing the directories or file systems. An important feature of rsync is that the mirroring takes place with only one transmission in each direction and which is not available in other similar programs.
• Rsync’s default port is 873 and its an opensource software. Rsync is available for Unix,Linux and windows operating systems. You can freely download rsync source code from rsync.samba web portal.

Key features of RSYNC command

• Support for copying links, devices, owners, groups and permissions
• Exclude and exclude-from options similar to GNU tar
• A CVS exclude mode for ignoring the same files that CVS would ignore
• Does not require root privileges
• Pipelining of file transfers to minimize latency costs
• Support for anonymous or authenticated rsync servers (ideal for mirroring)

I would be giving the basics on RSYNC and how to use it effectively to sync the directories/code

How to use RSYNC – sync the data between two directory

SOURCE_PATH='/u900/oracle/'
SOURCE_SERVER='myserver1'
DESTINATION_PATH='/u900/oracle-bck/'
DESTINATION_HOST='myserver2'
DESTINATION_USER='oracle'
rsync -av --rsh=ssh $SOURCE_PATH [email protected]$DESTINATION_HOST:$DESTINATION_PATH

Here

-a option is called the archive option,It means following things
Recursive mode
Preserves symbolic links
Preserves permissions
Preserves timestamp
Preserves owner and group
-v  Verbose option

This command once runs will ask for the password of the oracle user on the Destination machine. Once the password is given it will sync the change files in the directory to the destination server

If u need to enable this sync through an automated job, then this script should be working without prompting for a password.

We  can passwordless ssh setup between the machine to achieve it

The automated script would be  like

$ cat rsync_oracle.sh
#!/bin/ksh
SOURCE_PATH='/u900/oracle/'
SOURCE_SERVER='myserver1'
DESTINATION_PATH='/u900/oracle-bck/'
DESTINATION_HOST='myserver2'
DESTINATION_USER='oracle'
LOGFILE='rsync_oracle.log'
echo $'\n\n' >> $LOGFILE
rsync -av --rsh=ssh $SOURCE_PATH [email protected]$DESTINATION_HOST:$DESTINATION_PATH 2>&1 >> $LOGFILE
echo "Sync Completed at:`/bin/date`" >> $LOGFILE
$chmod 700 rsync_oracle.sh

The  same  command of RSYNC can be used in the same server also

SOURCE_PATH='/u900/oracle/'
SOURCE_SERVER='myserver1'
DESTINATION_PATH='/u900/oracle-bck/'
DESTINATION_HOST='myserver1'
rsync -av  $SOURCE_PATH $DESTINATION_PATH

Some more points to remember

(1)If the directories contain multiple owners, then I would suggest doing it using root user

(2) We can delete the files at the destination site which are deleted on the source site by adding –delete in RSYNC

rsync -av --rsh=ssh --delete $SOURCE_PATH $DESTINATION_PATH

(3) it can also be achieved using RSYNC daemon which is configured through /etc/rsyncd.conf on Solaris. But it is not necessary to use that.

Benefits of RSYNC command

(1) First time, rsync replicates the whole content between the source and destination directories. Next time, rsync transfers only the changed blocks or bytes to the destination location, which makes the transfer really fast.
(2) We can use ssh protocol for rsync which allows encryption of data during transfer. So it is quite secure
(3) Less Bandwidth: rsync uses compression and decompression of data block by block at the sending and receiving end respectively. So the bandwidth used by rsync will be always less compared to other file transfer protocols.

Practical RSYNC Command

1.   Synchronizing the directories on the local server

rsync -av  /app/src/ /app1/dest/

2.  Synchronizing the directories on the remote  server

rsync -av  /app/src/ [email protected]:/app1/dest/

3. Synchronizing the  directory structure on the remote  server

rsync -v  -d  /app/src/ [email protected]:/app1/dest/

4. How to Include and Exclude Pattern during File Transfer

rsync -av  --exclude 'U*'  /app/src/ [email protected]:/app1/dest/

The above command exclude any file starting with U

5. How to delete files at the destination

rsync -av --rsh=ssh --delete /app/src/ [email protected]:/app1/dest/

I hope  you like this article on RSYNC command and  add value to the technical work you are doing  in your organization

Please do provide feedback for this article on the RSYNC command

Related Articles
SSH Putty commands : Putty is a widely used tool to connect with a Linux server. Here are 41 Useful SSH Putty commands to help you manage Linux with examples.
Unix Background Jobs : what is foreground and background process.How to use bg, fg, CtrlZ Unix command to manage the Unix background jobs with examples
how to use putty: Check out this post on how to use putty for ssh connection to Linux or Unix server.Various settings of terminal, how to download the client and install it
ssh passwordless login: This post has the detailed instruction for setting up Passwordless Authentication with ssh between two servers
chmod recursive: File permissions decoded, how to change the permission of file using chmod,chmod recursive, Linux chmod -R
copy directory Linux: Check out this post for a detailed description of how to copy files/directories in Linux. Examples of copy directory Linux command is also given