Unfortunately no one can be told what fun_plug is - you have to see it for yourself.
You are not logged in.
With Alt-F-0.7, from an unaligned WD10EARS to an aligned WD20EARX, it took 24 hours to rsync 350Gb.
Is this normal ?
I have other 350+ Gb to rsync ...
Last edited by GabrieleV (2011-09-25 02:13:40)
Offline
bjby wrote:
There are threads on this. Summary is use cp -a first it is way faster then run a rsync afterwards.
On the other side, at least with the busybox shipped with Alt-F, and with deep or large directories structures, 'cp -a' uses a huge amount of memory and starts swapping, leading to a big performance decrease...
Alt-F backup and directory copy scripts uses find/cpio, which has a constant (not even linear!) memory usage.
# cp -a, piped tar and rsync uses too much memory (that keeps growing) for big trees and start swapping
# cpio has constante memory usage but is limited to 4GB files...
Further comments appreciated.
Offline
Ouch !
Never thinked that simply copying files involves such problems =:]
The rsync of the other disk that I have upgraded was done with two usb2sata adapters, and took 12 hours for the same amount of data.
I thinked that doing the rsync on the dns-323 would be faster: I'm really wrong ...
Do you think that it was better doing an rsync from my notebook with:
rsync -aHv -e ssh dns323:/mnt/sdadata dns323:/mnt/sdbdata
I don't think it's a memory hog problem. With the rsyng running from hours I have
$ free
total used free shared buffers
Mem: 62176 61132 1044 0 13712
Swap: 1060536 18028 1042508
Total: 1122712 79160 1043552
Stunned about this ...
Offline
In the meantime, I've tried copying a file from the source disk via smb share: 50Mbit/s over wireless N ...
Offline
GabrieleV wrote:
Ouch !
Never thinked that simply copying files involves such problems =:]
The rsync of the other disk that I have upgraded was done with two usb2sata adapters, and took 12 hours for the same amount of data.
I thinked that doing the rsync on the dns-323 would be faster: I'm really wrong ...
The dns is really slow, if you can do the first rsync/copy on a "real" computer.
Alt-F- RC1 Backup uses find/cpio for the first copy, then uses rsync. Not faster by itself, but avoids the swap problem, that really degrades disk operations.
Do you think that it was better doing an rsync from my notebook with:
rsync -aHv -e ssh dns323:/mnt/sdadata dns323:/mnt/sdbdata
I don't think it's a memory hog problem. With the rsyng running from hours I have
$ free
total used free shared buffers
Mem: 62176 61132 1044 0 13712
Swap: 1060536 18028 1042508
Total: 1122712 79160 1043552
Stunned about this ...
But swap is being used. Try 'top' and watch the 'vsize' and '%MEM' columns.
Offline
jcard wrote:
Alt-F- RC1 Backup uses find/cpio for the first copy, then uses rsync.
RC1 ? Where is RC1 ?
But swap is being used. Try 'top' and watch the 'vsize' and '%MEM' columns.
(Un)fortunately the copy has finished :-(.
Yes I saw that swap has been in use, but i thought that swapped memory was the one of "unused" tasks, parked into virtual memory and not needed at the moment.
Empirically, i thought that swap was not "stressed" because hdd lights were not flashing constantly ...
Offline
GabrieleV wrote:
jcard wrote:
Alt-F- RC1 Backup uses find/cpio for the first copy, then uses rsync.
RC1 ? Where is RC1 ?
In the forge :-)
But swap is being used. Try 'top' and watch the 'vsize' and '%MEM' columns.
(Un)fortunately the copy has finished :-(.
Yes I saw that swap has been in use, but i thought that swapped memory was the one of "unused" tasks, parked into virtual memory and not needed at the moment.
That is generally true, but if you watch 'top' output after running 'cp -a' for some tens of minutes you will see that the program virtual memory size is greater than 64MB, the box physical memory. This means that portions of the running program data or code are in disk, and must be retrieved into memory when needed, (which happens frequently), which means that other portions of the same running program data or code will have to go to disk...
Empirically, i thought that swap was not "stressed" because hdd lights were not flashing constantly ...
And how can that happen if 'cp -a' is running?
What initially called my attention was exactly seeing both hdd lights flashing when I was copying directories only on one disk.
Alt-F uses swap on both disks with equal priorities, so swapping is interleaved between them, which improves I/O (like in RAID-0).
PS-The same applies to 'rsync', not only 'cp -a', if my code comments are correct.
Offline
Excellent explanation !
We are used to deal with systems with gadzillion of bytes Ram, and the world of embedded systems is not like the world of network server ...
But is it right to name the dns-323 an "embedded system" ?
Yesterday night I 've red the documentation of freenas: works greatly only with 6Gb of ram (yes SixGigaBytesOfRam !).
Another time I'm thinking that the big trouble with dns323 is the limited amount of RAM. Even with the 128k hack, problems will not be solved, may be ...
Offline