I have to sync up the two directory many times on Unix boxes.The simple steps would be to delete the directory from destination and take a backup on source destination and copy it to the destination directory.
It was good for small directories but This process will consume lot of time as directory size grow up.
I searched around and find that The fastest way to do this process would be to use RSYNC utilty
It uses ‘rsync algorithm’ which provides a very fast method for syncing the directories or filesystems. An important feature of rsync is that the mirroring takes place with only one transmission in each direction and which is not available in other similar programs.
Rsync’s default port is 873 and its an opensource software. Rsync is available for Unix,Linux and windows operating systems. You can freely download rsync source code from rsync.samba web portal.
Key features of RSYNC command;
Support for copying links, devices, owners, groups and permissions
Exclude and exclude-from options similar to GNU tar
A CVS exclude mode for ignoring the same files that CVS would ignore
Does not require root privileges
Pipelining of file transfers to minimize latency costs
Support for anonymous or authenticated rsync servers (ideal for mirroring)
I would be giving the basics on RSYNC and how to use it effectively to sync the directories/code
How to use RSYNC – sync the data between two directory
SOURCE_PATH=’/u900/oracle/’
SOURCE_SERVER=’myserver1′
DESTINATION_PATH=’/u900/oracle-bck/’
DESTINATION_HOST=’myserver2′
DESTINATION_USER=’oracle’
rsync -av –rsh=ssh $SOURCE_PATH [email protected]$DESTINATION_HOST:$DESTINATION_PATH
Here
-a option is called the archive option,It means following things
Recursive mode
Preserves symbolic links
Preserves permissions
Preserves timestamp
Preserves owner and group
-v Verbose option
This command once run will ask for the password of oracle user on Destination machine.Once password is given it will sync the change files in the directory to the destination server
If u need to enable this sync through a automated job, then this script should be working without prompting for password.
We can password less ssh setup between the machine to achieve it
How to Enable Password less Authentication with SSH between two servers
Automated script would be like
$ cat rsync_oracle.sh
#!/bin/ksh
SOURCE_PATH=’/u900/oracle/’
SOURCE_SERVER=’myserver1′
DESTINATION_PATH=’/u900/oracle-bck/’
DESTINATION_HOST=’myserver2′
DESTINATION_USER=’oracle’
LOGFILE=’rsync_oracle.log’
echo $’\n\n’ >> $LOGFILE
rsync -av –rsh=ssh $SOURCE_PATH [email protected]$DESTINATION_HOST:$DESTINATION_PATH 2>&1 >> $LOGFILE
echo “Sync Completed at:`/bin/date`” >> $LOGFILE
$chmod 700 rsync_oracle.sh
The same command of RSYNC can be used in the same server also
SOURCE_PATH=’/u900/oracle/’
SOURCE_SERVER=’myserver1′
DESTINATION_PATH=’/u900/oracle-bck/’
DESTINATION_HOST=’myserver1′
rsync -av $SOURCE_PATH $DESTINATION_PATH
Some more points to remember
1)If the directories contain mulitple owner,then I would suggest doing it using root user
2) We can delete the files at destination site which are deleted on source site by adding –delete in RSYNC
rsync -av –rsh=ssh –delete $SOURCE_PATH $DESTINATION_PATH
3) it can also be acheived using RSYNC deamon which is configured through /etc/rsyncd.conf on Solaris.But it is not necessary to use that.
Benefits of RSYNC command
1) First time, rsync replicates the whole content between the source and destination directories. Next time, rsync transfers only the changed blocks or bytes to the destination location, which makes the transfer really fast.
2) We can use ssh protocol for rsync which allows encryption of data during transfer.So it is quite secure
3) Less Bandwidth: rsync uses compression and decompression of data block by block at the sending and receiving end respectively. So the bandwidth used by rsync will be always less compared to other file transfer protocols.
Some Practical RSYNC Command
- Synchronizing the directories on the local server
rsync -av /app/src/ /app1/dest/
2. Synchronizing the directories on the remote server
rsync -av /app/src/ [email protected]:/app1/dest/
3. Synchronizing the directory structure on the remote server
rsync -v -d /app/src/ [email protected]:/app1/dest/
4. How to Include and Exclude Pattern during File Transfer
rsync -av –exclude ‘U*’ /app/src/ [email protected]:/app1/dest/
The above command exclude any file starting with U
5. How to delete files at the destination
rsync -av –rsh=ssh –delete /app/src/ [email protected]:/app1/dest/
I hope you like this article on RSYNC command and add value to the technical work you are doing in your organization
Please do provide feedback
Leave a Reply