DSM-G600, DNS-3xx and NSA-220 Hack Forum

It depends on how you're doing the copy - if it is through Windows, cut & paste or similar - then you're transferring data from disk #1 across your network to the Windows PC and then back across the network to the NAS - 5.xx MB/sec is a little less than the theoretical maximum for a 100 mbps LAN but it sounds about right given my "non-scientific" tests.

File size also plays a part - a single 2GB file transfers a heck of a lot faster than 1000 x 2MB files, even though it's the same volume of raw data - this is due to the "housekeeping" required to keep directory and block allocation listings etc. up to date.

I did an internal copy of a 220MB file from /mnt/HD_a2 to /mnt/HD_b2 in 9.21 seconds.  That is a transfer rate of roughly 24MB/s.


# # time cp /mnt/HD_b2/tmp/test1 .

real    0m 9.21s
user    0m 0.21s
sys     0m 7.62s


Regards,
BSPvette

Pejn wrote:

real    0m 10.90s
user    0m 0.12s
sys     0m 7.55s

with a 230mb file but in windows i get 31sec with same file

Like I said earlier - if you do it from Windows the data has to corss the network twice, once coming and once going - it is not an efficient way to transfer data.

When I copy a 477102080 byte file (exactly 455 MB) from HD_a2 to HD_b2 these are my results:
/mnt/HD_b2/temp # time cp /mnt/HD_a2/Backup/WinXPHome6.tib .
real    0m 19.29s
user    0m 0.28s
sys     0m 15.72s
Giving a speed of 24733130 bytes/sec (23.6 MB/s) - similar to everyone else's results.

However, nightly I use rsync to copy from HD_a2 to HD_b2.  This is a typical output log when doing so:
2008/09/02 02:34:15 [3264] Number of files: 20025
2008/09/02 02:34:15 [3264] Number of files transferred: 26
2008/09/02 02:34:15 [3264] Total file size: 450206222593 bytes
2008/09/02 02:34:15 [3264] Total transferred file size: 6223343983 bytes
2008/09/02 02:34:15 [3264] Literal data: 6223343983 bytes
2008/09/02 02:34:15 [3264] Matched data: 0 bytes
2008/09/02 02:34:15 [3264] File list size: 322655
2008/09/02 02:34:15 [3264] File list generation time: 0.001 seconds
2008/09/02 02:34:15 [3264] File list transfer time: 0.000 seconds
2008/09/02 02:34:15 [3264] Total bytes sent: 6224427974
2008/09/02 02:34:15 [3264] Total bytes received: 1083
2008/09/02 02:34:15 [3264] sent 6224427974 bytes  received 1083 bytes  3601058.18 bytes/sec
2008/09/02 02:34:15 [3264] total size is 450206222593  speedup is 72.33
2008/09/02 02:34:15 [3264] sent 6224427974 bytes  received 1084 bytes  total size 450206222593

The transfer speed is here is only 3601058.18 bytes/sec (~3.4 MB/s)

Any ideas why rsync is dramatically slower?

I agree that it is probably a s/w issue with rsync, but unfortunately I can argue against most of the points fordem mentions.

To clarify, most of the files I'm transferring are compressed backups of 455MB in size.  They are also "new" files, so there should be little or no comparison required.  This is the exact listing of what was transferred (26 of 20025 files):
2008/09/02 02:05:27 [3264] building file list
2008/09/02 02:05:28 [3264] cd..t...... ./
2008/09/02 02:05:28 [3264] >f.st...... ffp.log
2008/09/02 02:05:28 [3264] >f.st...... fun_plug
2008/09/02 02:05:28 [3264] cd+++++++++ .bootstrap/
2008/09/02 02:05:28 [3264] >f+++++++++ .bootstrap/dmesg.out
2008/09/02 02:05:28 [3264] >f+++++++++ .bootstrap/part.conf
2008/09/02 02:05:28 [3264] >f+++++++++ .bootstrap/part_table.sh
2008/09/02 02:05:28 [3264] >f+++++++++ .bootstrap/read_me.txt
2008/09/02 02:05:28 [3264] >f+++++++++ .bootstrap/setup.sh
2008/09/02 02:05:28 [3264] >f+++++++++ .bootstrap/usb-storage.ko
2008/09/02 02:05:28 [3264] >f+++++++++ .bootstrap/usbmount.sh
2008/09/02 02:05:28 [3264] cd..t...... Backup/
2008/09/02 02:05:49 [3264] >f+++++++++ Backup/Documents12.tib
2008/09/02 02:07:59 [3264] >f+++++++++ Backup/Photos140.tib
2008/09/02 02:10:09 [3264] >f+++++++++ Backup/Photos141.tib
2008/09/02 02:12:18 [3264] >f+++++++++ Backup/Photos142.tib
2008/09/02 02:14:27 [3264] >f+++++++++ Backup/Photos143.tib
2008/09/02 02:16:36 [3264] >f+++++++++ Backup/Photos144.tib
2008/09/02 02:18:46 [3264] >f+++++++++ Backup/Photos145.tib
2008/09/02 02:20:55 [3264] >f+++++++++ Backup/Photos146.tib
2008/09/02 02:23:04 [3264] >f+++++++++ Backup/Photos147.tib
2008/09/02 02:25:14 [3264] >f+++++++++ Backup/Photos148.tib
2008/09/02 02:27:23 [3264] >f+++++++++ Backup/Photos149.tib
2008/09/02 02:29:33 [3264] >f+++++++++ Backup/Photos150.tib
2008/09/02 02:31:42 [3264] >f+++++++++ Backup/Photos151.tib
2008/09/02 02:32:17 [3264] >f+++++++++ Backup/Photos152.tib
2008/09/02 02:32:43 [3264] >f+++++++++ Backup/Programs18.tib
2008/09/02 02:33:37 [3264] >f+++++++++ Backup/WinXPHome12.tib
2008/09/02 02:34:15 [3264] cd..t...... usb-storage/
2008/09/02 02:34:15 [3264] >f.st...... usb-storage/snapshot.sh

The .bootstrap, ffp, and usb-storage files are tiny text files, and I agree these will take longer per byte to transfer.  The files in the Backup directory are the compressed backups.  From the timestamps you can see that the large files (e.g. Photos147.tib) take ~2:10 to transfer - compared to 20 seconds using "time cp".

On a night where no changes have been made to HD_a2, the "transfer" takes approximately 30 seconds.  This I can assume is the overhead to generate the file list on both drives and compare the size / timestamp / attributes to determine nothing has changed.  In the example I posted earlier the total time taken was just under 30 minutes.  Although significant, 30 seconds should not cause a 10 fold slowdown.

Any thoughts / disagreements with my reasoning, or ideas for further debugging?

bq041 wrote:

Does rsync look internal to the archive (compressed) file to look for changes internal to it?

No, it doesn't.

I'd ask for the exact rsync command. There are options that can slow down rsync dramatically.

DaveN wrote:

$ffp_path/rsync -aix --stats --link-dest=$dstpath/current --delete --log-file=$logdir/rsync.log $srcpath/ $dstpath/$date/

Looks ok. The candidates would have been -c, -z, -e ssh.