DSM-G600, DNS-3xx and NSA-220 Hack Forum

Unfortunately no one can be told what fun_plug is - you have to see it for yourself.

You are not logged in.

Announcement

#1 2011-09-25 02:11:54

GabrieleV
Member
From: Como
Registered: 2008-12-08
Posts: 17
Website

Perfomance : 350Gb/24hours

With Alt-F-0.7, from an unaligned WD10EARS to an aligned WD20EARX, it took 24 hours to rsync 350Gb.
Is this normal ?
I have other 350+ Gb to rsync ...

Last edited by GabrieleV (2011-09-25 02:13:40)

Offline

 

#2 2011-09-25 16:39:45

bjby
Member
Registered: 2009-02-22
Posts: 265

Re: Perfomance : 350Gb/24hours

There are threads on this. Summary is use cp -a first it is way faster then run a rsync afterwards.

Offline

 

#3 2011-09-25 17:57:00

jcard
Member
From: Portugal
Registered: 2008-09-21
Posts: 289

Re: Perfomance : 350Gb/24hours

bjby wrote:

There are threads on this. Summary is use cp -a first it is way faster then run a rsync afterwards.

On the other side, at least with the busybox shipped with Alt-F, and with deep or large directories structures, 'cp -a' uses a huge amount of memory and starts swapping, leading to a big performance decrease...
Alt-F backup and directory copy scripts uses find/cpio, which has a constant (not even linear!) memory usage.

    # cp -a, piped tar and rsync uses too much memory (that keeps growing) for big trees and start swapping
    # cpio has constante memory usage but is limited to 4GB files...

Further comments appreciated.


Please consider discussing Alt-F at http://groups.google.com/group/alt-f/topics
Please consider filling Alt-F bugs at http://code.google.com/p/alt-f/issues/list

Offline

 

#4 2011-09-25 21:31:15

GabrieleV
Member
From: Como
Registered: 2008-12-08
Posts: 17
Website

Re: Perfomance : 350Gb/24hours

Ouch !
Never thinked that simply copying files involves such problems =:]
The rsync of the other disk that I have upgraded was done with two usb2sata adapters, and took 12 hours for the same amount of data.
I thinked that doing the rsync on the dns-323 would be faster: I'm really wrong ...
Do you think that it was better doing an rsync from my notebook with:
rsync -aHv -e ssh dns323:/mnt/sdadata  dns323:/mnt/sdbdata

I don't think it's a memory hog problem. With the rsyng running from hours I have
$ free
                     total         used                free       shared        buffers
Mem:          62176        61132            1044              0        13712
Swap:     1060536        18028      1042508
Total:      1122712        79160      1043552

Stunned about this ...

Offline

 

#5 2011-09-25 22:03:26

GabrieleV
Member
From: Como
Registered: 2008-12-08
Posts: 17
Website

Re: Perfomance : 350Gb/24hours

In the meantime, I've tried copying a file from the source disk via smb share: 50Mbit/s over wireless N ...

Offline

 

#6 2011-09-25 22:03:58

jcard
Member
From: Portugal
Registered: 2008-09-21
Posts: 289

Re: Perfomance : 350Gb/24hours

GabrieleV wrote:

Ouch !
Never thinked that simply copying files involves such problems =:]
The rsync of the other disk that I have upgraded was done with two usb2sata adapters, and took 12 hours for the same amount of data.
I thinked that doing the rsync on the dns-323 would be faster: I'm really wrong ...

The dns is really slow, if you can do the first rsync/copy on a "real" computer.

Alt-F- RC1 Backup uses find/cpio for the first copy, then uses rsync. Not faster by itself, but avoids the swap problem, that really degrades disk operations.

Do you think that it was better doing an rsync from my notebook with:
rsync -aHv -e ssh dns323:/mnt/sdadata  dns323:/mnt/sdbdata

I don't think it's a memory hog problem. With the rsyng running from hours I have
$ free
                     total         used                free       shared        buffers
Mem:          62176        61132            1044              0        13712
Swap:     1060536        18028      1042508
Total:      1122712        79160      1043552

Stunned about this ...

But swap is being used. Try 'top' and watch the 'vsize' and '%MEM' columns.


Please consider discussing Alt-F at http://groups.google.com/group/alt-f/topics
Please consider filling Alt-F bugs at http://code.google.com/p/alt-f/issues/list

Offline

 

#7 2011-09-26 16:17:36

GabrieleV
Member
From: Como
Registered: 2008-12-08
Posts: 17
Website

Re: Perfomance : 350Gb/24hours

jcard wrote:

Alt-F- RC1 Backup uses find/cpio for the first copy, then uses rsync.

RC1 ? Where is RC1 ? wink

But swap is being used. Try 'top' and watch the 'vsize' and '%MEM' columns.

(Un)fortunately the copy has finished :-(.
Yes I saw that swap has been in use, but i thought that swapped memory was the one of "unused" tasks, parked into virtual memory and not needed at the moment.
Empirically, i thought that swap was not "stressed" because hdd lights were not flashing constantly ...

Offline

 

#8 2011-09-26 18:41:26

jcard
Member
From: Portugal
Registered: 2008-09-21
Posts: 289

Re: Perfomance : 350Gb/24hours

GabrieleV wrote:

jcard wrote:

Alt-F- RC1 Backup uses find/cpio for the first copy, then uses rsync.

RC1 ? Where is RC1 ? wink

In the forge :-)

But swap is being used. Try 'top' and watch the 'vsize' and '%MEM' columns.

(Un)fortunately the copy has finished :-(.
Yes I saw that swap has been in use, but i thought that swapped memory was the one of "unused" tasks, parked into virtual memory and not needed at the moment.

That is generally true, but if you watch 'top' output after running 'cp -a' for some tens of minutes you will see that the program virtual memory size is greater than 64MB, the box physical memory. This means that portions of the running program data or code are in disk, and must be retrieved into memory when needed, (which happens frequently), which means that other portions of the same running program data or code will have to go to disk...

Empirically, i thought that swap was not "stressed" because hdd lights were not flashing constantly ...

And how can that happen if 'cp -a' is running?

What initially called my attention was exactly seeing both hdd lights flashing when I was copying directories only on one disk.
Alt-F uses swap on both disks with equal priorities, so swapping is interleaved between them, which improves I/O (like in RAID-0).

PS-The same applies to 'rsync', not only 'cp -a', if my code comments are correct.


Please consider discussing Alt-F at http://groups.google.com/group/alt-f/topics
Please consider filling Alt-F bugs at http://code.google.com/p/alt-f/issues/list

Offline

 

#9 2011-09-26 22:02:10

GabrieleV
Member
From: Como
Registered: 2008-12-08
Posts: 17
Website

Re: Perfomance : 350Gb/24hours

Excellent explanation !
We are used to deal with systems with gadzillion of bytes Ram, and the world of embedded systems is not like the world of network server ...
But is it right to name the dns-323 an "embedded system" ? wink
Yesterday night I 've red the documentation of freenas: works greatly only with 6Gb of ram (yes SixGigaBytesOfRam !).
Another time I'm thinking that the big trouble with dns323 is the limited amount of RAM. Even with the 128k hack, problems will not be solved, may be ...

Offline

 

Board footer

Powered by PunBB
© Copyright 2002–2010 PunBB