DSM-G600, DNS-3xx and NSA-220 Hack Forum

This is also what BackupNetClone (http://backupnetclone.sourceforge.net/) does.  Additionally, BNC uses SSH to encrypt transfers in case you want to do the backup over the Internet, and it sends you status emails, and it cleans up old stuff as the disk gets full (avail in the next version which I'm working on releasing).

Very Good tuturial, great for me too... but

It is possible to copy same folders, and not the full disk ?

Like This :

srcpath=/mnt/HD_a2/Videos
srcpath=/mnt/HD_a2/Data
srcpath=/mnt/HD_a2/Pictures
dstpath=/mnt/HD_b2/Pictures
dstpath=/mnt/HD_b2/Data
dstpath=/mnt/HD_b2/Videos

?????

Tks.

By the way, I have modified my script slightly since my last post.  The first version works, but here are some of the updates I did:

- The first version stored a copy of the log file for each backup.  To save space, this new version will just keep a single copy of the log under fun_plug.d/log/snapshot.log.  Instead of checking for the existence of the log file before updating the current link, it checks to make sure the directory is not empty before updating the "current" link.

- Added a ffppath so that the fun_plug directory is more easily configurable.  For example, fun_plug 0.5 users may find it in /mnt/HD_a2/ffp, and USB users may find it in /mnt/usb_1/fun_plug.d or /mnt/usb_1/ffp


Here's the new snapshot.sh:

=======================================
#!/bin/sh

srcpath=/mnt/HD_a2
dstpath=/mnt/HD_b2
ffppath=/mnt/HD_a2/fun_plug.d
date=`date "+%Y%m%d_%H%M%S"`
mkdir $dstpath/$date
$ffppath/bin/rsync -aivx --link-dest=$dstpath/current --exclude='workspace/' $srcpath $dstpath/$date > $ffppath/log/snapshot.log 2>&1
if [ $(ls -1A $dstpath/$date | wc -l) -ne 0 ]
then
  rm $dstpath/current
  ln -s $date $dstpath/current
fi
========================================

Well.. i have problems...

My Snapshot :

#!/bin/sh

srcpath="/mnt/HD_a2/NETWORK_DISK_01/ANIME /mnt/HD_a2/NETWORK_DISK_01/DATA /mnt/HD_a2/NETWORK_DISK_01/MUSICA /mnt/HD_a2/NETWORK_DISK_01/VIDEOS"
dstpath=/mnt/HD_b2/Backups
date=`date "+%Y%m%d_%H%M%S"`
mkdir $dstpath/$date
/mnt/HD_a2/ffp/bin/rsync -aivx --link-dest=$dstpath/current $srcpath $dstpath/$date > $dstpath/$date/rsync.log 2>&1
if [ -s $dstpath/$date/rsync.log ]
then
  rm $dstpath/current
  ln -s $date $dstpath/current
fi

The Directory DATA ( have 23956 itens / 24,3 GB ) and it this directory the task always stop. If i remove this directory from the snaphots all works great.

The total backups with all directories is 80 a 90 GB, but the directory DATA have most of the files, small files and many "23956"...

memory problem ? Solution ? I have only transmission runing......

Tks.

Last edited by Nasp (2008-05-08 16:53:23)

raid123 wrote:

Let's start by making sure there's plenty of disk space in the backup drive, just in case.  Then take a look at the rsync.log file in the backup folder to see where it stopped.  Try moving that file or directory out of the DATA folder, run snapshot.sh and see if it works.  Maybe a single file or directory name is having problems with rsync.

Well, it stop at a file named .pureftp-upload....................................................  what file is that ? Maybe I have to stopped the FTP server or the backups dont work and i used the beta firmware 1.04a in my CH3SNAS............ but for now i only removed the folder from DATA, tonight i will test again.

tks from the help again, raid123.

Last edited by Nasp (2008-05-09 02:33:54)

raid123,
Thanks for the post. I however ran into a problem with your latest script.
if [ $(ls -1A $dstpath/$date | wc -l) -ne 0 ]

error output with set -x enabled

+ ls -1A /mnt/HD_b2/20080514_215811
+ wc -l
wc: No such file or directory
+ [ -ne 0 ]
[: 0: unknown operand



Im a newbie when it comes to scripts,
I see some places that talk about adding "" around wc command
if [ "$(ls -1A $dstpath/$date | wc -l)" -ne 0 ]

but this does not seem to be right/working either.

Thanks for the help
Newbie2

Last edited by newbie2 (2008-05-15 08:35:59)

That was it. I thought wc was giving the error not the system stating it couldn't find wc .
Great post and thanks for the help.
Newbie2

Last edited by newbie2 (2008-05-16 06:48:34)

For those who don't know rsnapshot, have a look here Rsnapshot for local and remote backup,

It does the same as the script proposed here in this post, but rsnapshot is now quite stable and very well tested.

Just a hint,
marinalink

fonz wrote:

marinalink wrote:

For those who don't know rsnapshot

There's always more than one way to do things. Personally, I like the simplicity of raid's script.

Absolutly! I fully agree. As I said, just a hint...

Just one comment in general for those who use rsync:
rsync needs approximately 100byte of RAM for each file to be synchronized (since it builds up a list of files to be synchroniyed). This means, if you have 100.000 files to be backuped, this results in a memory usage of at least 10MB.

Maybe someone already encountered this problem. Tonight I realized, that my DNS ran out of RAM and started swapping to disk. This of course slows down the backup process... A solution to this problem is to run rsync on sub directories to reduce the amount of files to be synchronized and kept in RAM.

marinalink

fonz wrote:

That's certainly true for rsync version < 3. Memory usage is much smaller with with rsync 3. From the NEWS file:

ENHANCEMENTS:

    - A new incremental-recursion algorithm is now used when rsync is talking
      to another 3.x version.  This starts the transfer going more quickly
      (before all the files have been found), and requires much less memory.
      See the --recursive option in the manpage for some restrictions.

    - Lowered memory use in the non-incremental-recursion algorithm for typical
      option values (usually saving from 21-29 bytes per file).

Hm, you are right... strange, that rsync needed 117% of my memory for 95GB and 81.500 files. For 36GB and 10.000 files it needs 15MB. Its just an observation. But unfortunately I can't explain it...

Anybody explainations for that?

marinalink

Feel free to add the -a back and use any other flags that make sense to you.  The reason they have -a on that post was because that post was actually on a full blown linux system where user accounts are used to log in.  I'm not sure the usage of the DNS-323 user account is the same.  In my case, I did not use the accounts on the DNS-323 other than to provide access, and I wanted my backup image to be read-only so that a virus that deletes all files on all my shares will not delete my backup image as well for extra protection.  That said, I've updated the wiki to use the -a flag once again.

What command are you using to restore and why would you run into problems?

Last edited by raid123 (2008-06-15 18:56:30)

I had this working, and then I decided to clear out my snapshots folder and start over because I had restructured the files. However, now it isn't creating the "current" folder. It is giving the following error:

"--link-dest arg does not exist: /mnt/HD_b2/v1_snapshots/current"

Any ideas? Thanks