DSM-G600, DNS-3xx and NSA-220 Hack Forum

Unfortunately no one can be told what fun_plug is - you have to see it for yourself.

You are not logged in.

Announcement

#1 2011-04-24 20:41:11

mfpockets
Member
Registered: 2011-01-10
Posts: 44

Rsync drive to drive copy

Hello,

I copy files from drive 1 to drive 2 sometimes to do backups.  When I run Rsync through putty from my windows machine I only get around 4.5mb a second. 

Now to my understanding this isnt using the network to copy the files since the command is being run in a remote shell and executed locally on the NAS unit. 

When I write files to the NAS unit from my windows box, I get around 9.5mb a sec on a 10/100 network...

Is this normal?

Offline

 

#2 2011-04-27 13:07:52

mfpockets
Member
Registered: 2011-01-10
Posts: 44

Re: Rsync drive to drive copy

Can anyone post their speed for comparison?  use --progress switch to view copy speed

Offline

 

#3 2011-04-27 16:56:07

karlrado
Member
Registered: 2009-12-07
Posts: 229

Re: Rsync drive to drive copy

There are other threads that discuss rsync and disk copy speeds.

But you are comparing apples (rsync) and oranges (file copy).

Rsync is a file synchronization system and protocol designed to sync files across networks, often over low bandwidth connections.  It is not designed or optimized to simply copy files, even when the copy is local. 

Even when you run rsync with a local source and a local destination, rsync will start two additional rsync processes, one to send the source files, and the other to receive the files.  The file data gets sent over a pipe between the two processes using IPC (inter process communication) and there is probably a fair amount of "difference processing" going on in rsync, as rsync analyzes each block of the files to determine which blocks of the file need to be transferred.  The whole point of rsync is to minimize the data going over the IPC connection (local or not) even at the expense of doing a lot of work to get that minimization.  Usually rsync is run between two different machines, with the IPC going over an external network.

A simple file copy, even over a LAN, involves much less overhead.

A lot of people who use rsync to do drive-to-drive copies like this often do an initial manual copy of any new files on the source disk to the destination disk, precisely to avoid the initial slow rsync transfer.

But once the files are established in both places, rsync is very quick and efficient in resolving any small incremental differences in the files.

So, yes, the speeds that you are seeing with rsync are nominal when you are doing an initial sync.  If you run rsync again with little or no new files, your effective rsync bandwidth as reported by rsync would be extremely high.


DNS-323 FW 1.07 : 2 1TB WD Caviar Green SATA : fun_plug: utelnet + optware (no ffp)

Offline

 

#4 2011-04-27 21:54:48

chriso
Member
Registered: 2009-03-29
Posts: 74

Re: Rsync drive to drive copy

My process is:
If first time copy, use cp -a
After that link last copy of backup to new directory
Rsync primary disk data to new directory.
Use list of of changed files to drive copying them to offline.

Also I use the -W option (whole file) on rsync.  For disk to disk transfers rsync would spend much more time trying to figure out what changed then to just copy the file.

Note that rsync has a linking option, but I don't use it because if you do it will not produce a list of the files that changed, that I need to feed into my offline copy.

Offline

 

#5 2011-04-28 04:11:11

bjby
Member
Registered: 2009-02-22
Posts: 265

Re: Rsync drive to drive copy

I have somewhat similiar io running rsync between disks.

root@alice:~# iostat -m
Linux 2.6.32-5-orion5x (alice)     04/28/2011     _armv5tel_    (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          31.16    0.00   56.12    1.33    0.00   11.38

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
mtdblock0         0.00         0.00         0.00          0          0
mtdblock1         0.00         0.00         0.00          0          0
mtdblock2         0.00         0.00         0.00          0          0
mtdblock3         0.00         0.00         0.00          0          0
mtdblock4         0.00         0.00         0.00          0          0
sda               1.68         0.02         0.00       5489        844
sdb              15.07         0.00         3.63        305    1114814
sdc              34.96         3.63         0.02    1113205       4716

Offline

 

#6 2011-05-01 18:19:29

mfpockets
Member
Registered: 2011-01-10
Posts: 44

Re: Rsync drive to drive copy

Thanks for the replies, it has been a while since I checked the forum. 

I already did the initial backup and I am doing subsequent backups.  For example I copied my entire TV folder, and now updating the backup with new episodes/seasons downloaded. 

"If you run rsync again with little or no new files, your effective rsync bandwidth as reported by rsync would be extremely high."

I am still seeing slow speeds on new updates.

Offline

 

#7 2011-05-02 17:23:34

karlrado
Member
Registered: 2009-12-07
Posts: 229

Re: Rsync drive to drive copy

Sure, any new file is going to be copied over at the same rate achieved by the initial rsync.  Rsync checksums the files during transfer for validating the transfer, which takes longer.  It also checks that the rest of the fileset is in sync, and that's what is meant by the effective sync rate.

If you had a huge file already on the source and destination and the source had a one-byte change in it, rsync would update the file on the destination much faster than it would take to copy the entire file over.  This also contributes to a high effective sync rate.  And this is the type of case where rsync shines, especially over a remote network connection.

If you are just plunking down new half-GB files into the source periodically, rsync will take some time to copy them over.  And that's not the optimum use case for rsync.  This shouldn't be a huge concern since rsync should just do its thing in the background and not bother anything else.  If getting the files synced with rsync is too slow for you, then perhaps rsync is not the tool for the data you are managing.

In my case, as an example, I rsync a lot of small files that change frequently over a remote network connection. So, rsync is ideal.  I also sync media files like yours that don't ever get modified, but may get deleted or moved occasionally.  Rsync is not the greatest for these files, but I still use it for its convenience and since my DNS-323 is a Server, I don't care too much about how long it takes.


DNS-323 FW 1.07 : 2 1TB WD Caviar Green SATA : fun_plug: utelnet + optware (no ffp)

Offline

 

#8 2011-05-02 19:02:27

scaramanga
Member
Registered: 2010-08-04
Posts: 251

Re: Rsync drive to drive copy

karlrado wrote:

If you had a huge file already on the source and destination and the source had a one-byte change in it, rsync would update the file on the destination much faster than it would take to copy the entire file over.  This also contributes to a high effective sync rate.  And this is the type of case where rsync shines, especially over a remote network connection.

rsync uses the delta transfer algorithm, by default, only when transferring files over the network, not when copying files locally.
See --whole-file in the rsync manual


DNS-323 HW Rev. C1 FW 1.10 fun-plug 0.5
2 x WD10EARS-00Y5B1 in Standard mode (LCC set to 5 min; Aligned to 4K)
Transmission with Transmission Remote GUI

Offline

 

Board footer

Powered by PunBB
© Copyright 2002–2010 PunBB