DSM-G600, DNS-3xx and NSA-220 Hack Forum

Is there a command to see what speed the Rsync is happening at?  I am at 24 hours in & its still not done.

This is from post #190

Code:

# +---------------- minute (0 - 59)
# |  +------------- hour (0 - 23)
# |  |  +---------- day of month (1 - 31)
# |  |  |  +------- month (1 - 12)
# |  |  |  |  +---- day of week (0 - 6) (Sunday=0 or 7)
# |  |  |  |  |
  *  *  *  *  *  command to be executed

also, after i had enough of this code not working i took fun_plug and ffp out completely (using telnet and rm fun_plug, rm ffp, ect).  i moved the two fun_plug files back over and rebooted.  i followed help from page 10 on checking the code ( and with help on the time format), it finally ran at 11am this morning.  I changed my time to 11pm (my network is off peak time then).  ive added a firefly server and a few other items.  it should move all that new stuff over to HD_b2 tonight.  if that works im using help from the time information above to run the backup once a week, like saturday nights.

RichTJ99 wrote:

Will Rsync keep both the source & data in sync?  Meaning, if I delete something from the source (Disk 1) will Disk 2 have that data deleted?

If you use the option "--delete" in your rsync command it will delete the files on disk 2 if you remove those files from disk 1.

did you vi the file for editcron?  what does crontab -l show? (lowercase "L")

halfsoul wrote:

greg,
I recently had a Vol1 drive fail.  My recovery procedure was as follows (with power-cycles in between each step):
1. remove bad Vol1 & good Vol2, insert new Vol1 in RHS
2. format new Vol1
3. insert old Vol2 in LHS
4. copy fun_plug script to Vol1 (I run fun_plug off USB, so all I needed was the file to call it)
5. copy data from Vol2 to Vol1

I too wish it were as simple as putting the Vol2 drive in the RHS, but alas... 'tis not.

the only tricky thing is to make sure your copy command is recursive and preserves attributes.  In order to put all of your data from the Vol2/HD_a2/ folder to the root of Vol1... you need to start in the Vol2/HD_a2/ folder and copy *.* recursively.  If you define a source directory then the destination (Vol1) will also have said sorce directory as a subdir.  <-- BTW, this is the same reason your Vol2 has a /HD_a2/ subdir instead of retaining the same root folder structure.

i just had my RHS drive fail and i need go get the data back to my new vol1.  so what command do i use to get vol 2's backup folder (HD_a2) back to vol 1's root.

i put ffp back on vol1 and telnet'd in and tried cd /mnt/HD_b2/HD_a2

then ls, all of my files were there (perfect).  so what command copies everything with the right attributes back to the new vol 1.

i typed in cp -r *.* /mnt/HD_a2 but nothing happened.

halfsoul wrote:

@roth: You're on the right path.

Code:

cd /mnt/HD_b2/HD_a2
cp -a *.* /mnt/HD_a2

The -a preserves links, attributes, and recurses directories.
If you do -af, then it will do all of -a plus forcing overwrite (only do this if you're sure)

What do you mean by "nothing happened"?  If it's working correctly, it will sit there for quite awhile with "nothing happening" until the operation finishes.  If it immediately gave you another command prompt, then something likely went wrong.

yea, something went wrong then.  i only had a few (large folders) so i typed the commande for each folder (instead of *.*) and it worked.  it even copied my fun_plug and ffp.  i rebooted the NAS and everything came up like it should, except with a new 1TB hdd instead of an old 80gb.

thx

I would like to add a suggestion to this thread. The "backup" process shown here is a good replacement for RAID1 configuration (although the needs are not exactly the same). Some people here argued, as well as on other threads, that RAID1, synchronization and backups are all different things. That's very true.

In the implementation shown here, if a file is deleted by accident, and if you do not react soon enough (i.e on the very same day), the file is also deleted from the "saved" copy during the nightly synchronisation, which is a pity. There is a very easy way to change this.

One interesting feature of rsync allows to have something very similar to "backup". This is the "--backup / backup_dir" option. In short, any deletion/modification of any file in the source directory is "synchronised" (ie. the file is also deleted from the destination directory); but this option  makes a copy of the deleted or modified file. The copy is stored in the "backup_dir" directory.
Example : if you delete a file by accident  (or somebody else, or a virus.... does it for you), the file is NOT deleted from the file system, but stored in a backup directory. One separate backup directory can be created each night, for example. The same happens if you modify a file, The "previous" version is stored in the "backup_dir"

So, if your file was deleted/modified one week or one month ago, you can still find it. Using "find" will allow you to retrieve any "deleted" file.

The drawback is that all copies of all "changed files" are kept on the disk. You then have to decide how long you want to keep your "deleted" files, and delete them either manually, either through an automatic cron  (for example, you might want to keep all modifications lasting more than 3, or 6 months, depends on the disk storage you may use for this).

Just  have a look at  this command:

/mnt/HD_a2/fun_plug.d/bin/rsync -rlptDv --delete  --backup --backup-dir=../backup_rsync_save/`date +%Y-%m-%d` /mnt/HD_a2/* /mnt/HD_b2/rsync_save >> /mnt/HD_a2/fun_plug.d/log/rsync.last.log 2>&1

You now have something VERY similar to a true backup tool. Each synchronization will create a new separate directory, like:

drwxr-xr-x 3 root root 4096 2009-04-24 20:42 2009-04-24

Hope this helps.

Hello Chumby,

Well, at least, this "new code" works on mine... This backup feature is something I use on several "rsync'ed" environments, I never encountered any problem with it (but I must also admit I had no disk crash, or virus, or accidental deletions for a while! But I can check that the files I suppress are obvioulsy stored in these backup dirs, so it works OK for me).

Some more issues:

A way to "clean up" this directories and save some disk space, is to use "find" with +mtime. Something like:

/mnt/HD_a2/fun_plug.d/bin/find /mnt/HD_b2/backup_rsync_save -name "[0-9]*-[0-9]*-[0-9]*" +mtime 60 | xargs rm -rf

will clean up anything older than 60 days.

The name pattern should allow to retrieve the directories built by `date +%Y-%m-%d`. This should be strictly something like "[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}"

Unfortunately, the regular expression understood by find does not support this "extended" notation.  One way is to pipe this to "grep", like:

find /mnt/HD_b2/backup_rsync_save -name "[0-9]*-[0-9]*-[0-9]*" | grep [0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}

. Of course, the "grep" command should appear between find and xargs.

Any "regexp" guru (I'm certainly not....) who could give a better (or nicer/smarter) solution is welcome.

Another point is that "find" provided by fun-plug (I'm still on version 3....) does not support the "maxdepth" option, which would be very useful in this case (maxdepth=1 for our need). I use myself the chrooted "etch" find command for this, but I know a lot of people use only "ffp" commands, so...

At last, I have also created a separate script file, named "synchronise.sh" which contains all this code (call to rsync and call to find, comments, etc...). My crontab contains then only one line for this (call to "synchronise.sh"). I find it simpler to manage this way.

  Cheers

Last edited by dkl (2009-04-29 12:21:46)

Is there any way to speed up the data transfer to the other drive during rsync? 

When I try the rsync I get about 16GB/h.  So for 320GB its taking about 20 hours.  When I did a mirror it could resync in about 2.5 hours the same stuff.  Considering the huge speed difference using the RAID1 seems to be the faster way to back up information.  If I change more than 40GB on the first disk it will take the same time as the resync.

Also, what would happen if the first cron job is not finished before the cron job of the next day begins?  When I was watching network file tansfer I estimated about 40GB/h.  Shouldn't it at least be that high?

Last edited by shawngiese (2009-05-01 21:16:04)

DKL... thanks for this extra script.  I'll see if I can get it in the tutorial soon...

If you get a chance, can you double check the rsync line I have added in the tutorial... doesn't seem to be copying for me at the moment.  I presume you need to create the rsync save folder manually?

Cheers
Chumby

dkl wrote:

yes, I have manually created the "backup" dir. Works ok here

I can't get it to work either, so i want to create the directory. But what are the .. in '--backup-dir=../backup_rsync_save' standing for?

EDIT:

I think it creates a ramdisk? Anyway the process stopped after copying just a few files. Then i changed the location to /mnt/HD_b2/backup_rsync_save and now it works fine. Didn't even have to create the destination directory.

Last edited by knireis (2009-05-10 15:29:17)

Hello guys, am having trouble with the backup running every night, i did everything in the instructions and when i did the # ls -al to verify it gave me errors about the log folder and BACKupnas folder , that they did not exits, did the mkdir on both and after that runned again ls -al and it stated backing up everything , but i added a new movie to my volume_1 and it never backed up to the volume_2 last night, i went to see the log and thats the error that i see, can anyone tell me what that means?
there only 1 new movie on volume1, but is not been backup on volumen 2.

in the log snapshot at the 1st line it says

sending incremental file list
--link-dest arg does not exist: /mnt/HD_b2/Backup_NAS/current

At the last line

rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(541) [sender=3.0.4]
rsync: writefd_unbuffered failed to write 78 bytes [generator]: Broken pipe (32)


is because there is no current folder inside backupnas folder?

dkl wrote:

can you post here your rsync command (the one you use in crontab?)

#!/bin/sh

CRONTXT=/mnt/HD_a2/crontab.txt

# start with existing crontab
/bin/crontab -l > $CRONTXT

# add the Rsync job to execute at 2:00 am
/bin/echo "0 2 * * * /ffp/bin/snapshot.sh" >> $CRONTXT

# install the new crontab
/bin/crontab $CRONTXT

# clean up
/bin/rm $CRONTXT


Here my snaphot.sh

#!/bin/sh

# Set Source Path
# Back up the A drive (HD_a2)  by identifying the source path as /mnt/HD_a2
srcpath=/mnt/HD_a2
# Back up multiple directories by surrounding a list with single quotes (i.e., srcpath='/mnt/HD_a2/Music /mnt/HD_a2/Docs')
# srcpath='/mnt/HD_a2/Music /mnt/HD_a2/Docs'

# Set the Destination Path
dstpath=/mnt/HD_b2/Backup_NAS

# Set path to Fun_Plug files
# Fun_Plug 3.0 or 4.0
# ffppath=/mnt/HD_a2/fun_plug.d
# Fun_Plug 5.0
ffppath=/ffp

date=`date "+%Y%m%d_%H%M%S"`
mkdir $dstpath/$date

$ffppath/bin/rsync -aivx --link-dest=$dstpath/current $srcpath $dstpath/$date > $ffppath/log/snapshot.log 2>&1

var=`ls -1A $dstpath/$date | wc -l`

if [ $var -ne 0 ]
then
  rm $dstpath/current
  ln -s $date $dstpath/current
fi

Last edited by lokitoorg (2009-05-21 22:14:09)