DSM-G600, DNS-3xx and NSA-220 Hack Forum

Unfortunately no one can be told what fun_plug is - you have to see it for yourself.

You are not logged in.

Announcement

#176 2008-10-26 18:38:52

silversurfer
Member
Registered: 2008-07-20
Posts: 95

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

Hey Chumby, thank you for this tutorial.

I tried it out and it works quite well. I only found a problem with the ownership of the folders and files on the backup drive. Using your suggested options "-rlptDv --delete" will cause all newly copied folders and files to belong to root:root on the backup drive. That is NOT what I want and probably a lot of other people agree with me and didn't even notice this behaviour so far.

I think that a much better way would be to use option "-av --delete" because this will also keep the ownership of the copied folders and files. This is a backup that we are talking about and backups are not about changing ownership, aren't they? wink


By the way, I like being informed about the results of the backup and since there is sendmail in ffp, you might as well add an option to send an email notification with the results.


My sync script looks like this at the moment:

Code:

#!/bin/sh
#this script synchronizes disk 1 (right slot) with disk 2 (left slot)
#it can be used by cron to automatically sync disks over night

DISK1=/mnt/HD_a2
DISK2=/mnt/HD_b2
SYNCOPT="-av --delete"
LOGFILE=/USB/var/log/sync-ab.log
PASS=`cat $DISK1/mailcred`

#write the current date and time to the log
date > $LOGFILE

#sync all the folders that we like
/ffp/bin/rsync $SYNCOPT $DISK1/Folder1 $DISK2 >> $LOGFILE
/ffp/bin/rsync $SYNCOPT $DISK1/Folder2 $DISK2 >> $LOGFILE

#send email notification
sendmail -U mailusername -P $PASS -t receiver@domain -s "Backup Details" \
servername sender@domain $LOGFILE

Of course you will have to put in the correct data for mailusername etc.

"sendmail --help" provides answers about the different options.

The script reads the password of the email account from the file "mailcred" which has permissions of 600.

Offline

 

#177 2008-10-27 20:49:43

WildSioux
Member
Registered: 2007-03-05
Posts: 71

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

Does anyone know how to change this script to backup once a week instead of nightly?

Offline

 

#178 2008-10-27 22:15:27

nurunet
Member
From: Germany
Registered: 2008-08-31
Posts: 44

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

Hi silversurfer,

this looks nice. I've been desperately trying to get my CH3SNAS to send mail. However, my sendmail reacts strangely:

root@NAS:/mnt/HD_a2/usbstorage/ffp# sendmail --help
sendmail: invalid option -- -
Invalid option ?
root@NAS:/mnt/HD_a2/usbstorage/ffp#

When I tried to use your code to test it, I got this:

root@NAS:~# sendmail -U test@example.com -P password -t test2@example.com -s "Backup Details" post.example.com test@example.com "Test"
sendmail: invalid option -- U
Invalid option ?
root@NAS:~#

What am I doing wrong? Why doesn't sendmail (it *does* exist in my bin directory) know its options?

Do I have to configure sendmail anyplace else before using it?

Greets
Ben


Conceptronic CH3SNAS FW 1.03, ffp from USB key.

Offline

 

#179 2008-10-28 07:15:29

yojoe007
New member
Registered: 2008-09-04
Posts: 4

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

blbrown wrote:

Dexter Tones wrote:

I followed the tutorial and everything went smoothly. However, I was looking at the rsync FAQ and it says that one should "expect rsync to consume about 100 bytes per file" to "hold the list of files being transferred." I don't have many files on my DNS right now, but given the amount of RAM it has, I fear that getting an "out of memory" error may become an issue in the future (when I have many more files stored on the DNS). Is there a workaround for this at all?

Yes, I have noticed this issue as well.  There are two work-arounds:

1. Do some scripting magic to walk your directory structure and run rsync on each one to make sure the file list doesn't get too long.

2. Wait until rsync (http://samba.anu.edu.au/rsync/) version 3.0 is complete and someone (fonz?) compiles it for the DNS-323.  Version 3 addresses this specific issue and it's currently in pre-release which gives me hope it will be complete some time soon.

That said, I have BNC running on a directory structure with ~26000 files (on a Windows PC backup client).  I've had no problems so far with the DNS-323 and BNC nicely tells me this file count in the email status (on the line that says "Backup location total file/directory count: xxx (xxxGB)").  On my DNS-323 that backs itself up (it's both the backup client and the backup server) I'm up to ~9000 files without problem so far.

At 100 bytes per file, you should have no problem using rsync with 320,000+ files, which should only consume half the physical memory of the DNS-323.  And I'm guessing swap space would be used automatically if you used up the physical memory, though this would be stressing the unit I think.

Rsync is now up to version 3.04, with 3.05 in pre-release test - was this problem addressed? Can I install ffp 0.4 and manual install rsync 3.04, or as is written above does it need to be "compiled" (I don't know what this means) to work correctly? And since ffp .05 includes rsync, does anyone know which version - as in, is it version 3.0 or above? I couldn't find that information anywhere, and have looked as best I can.

I am concerned about the physical memory being used up as I use my DNS for backing up thousands of digital photos from my job.

Thanks, James

Offline

 

#180 2008-10-28 16:04:02

blbrown
Member
Registered: 2007-11-02
Posts: 88
Website

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

yojoe007 wrote:

blbrown wrote:

Dexter Tones wrote:

I followed the tutorial and everything went smoothly. However, I was looking at the rsync FAQ and it says that one should "expect rsync to consume about 100 bytes per file" to "hold the list of files being transferred." I don't have many files on my DNS right now, but given the amount of RAM it has, I fear that getting an "out of memory" error may become an issue in the future (when I have many more files stored on the DNS). Is there a workaround for this at all?

Yes, I have noticed this issue as well.  There are two work-arounds:

1. Do some scripting magic to walk your directory structure and run rsync on each one to make sure the file list doesn't get too long.

2. Wait until rsync (http://samba.anu.edu.au/rsync/) version 3.0 is complete and someone (fonz?) compiles it for the DNS-323.  Version 3 addresses this specific issue and it's currently in pre-release which gives me hope it will be complete some time soon.

That said, I have BNC running on a directory structure with ~26000 files (on a Windows PC backup client).  I've had no problems so far with the DNS-323 and BNC nicely tells me this file count in the email status (on the line that says "Backup location total file/directory count: xxx (xxxGB)").  On my DNS-323 that backs itself up (it's both the backup client and the backup server) I'm up to ~9000 files without problem so far.

At 100 bytes per file, you should have no problem using rsync with 320,000+ files, which should only consume half the physical memory of the DNS-323.  And I'm guessing swap space would be used automatically if you used up the physical memory, though this would be stressing the unit I think.

Rsync is now up to version 3.04, with 3.05 in pre-release test - was this problem addressed? Can I install ffp 0.4 and manual install rsync 3.04, or as is written above does it need to be "compiled" (I don't know what this means) to work correctly? And since ffp .05 includes rsync, does anyone know which version - as in, is it version 3.0 or above? I couldn't find that information anywhere, and have looked as best I can.

I am concerned about the physical memory being used up as I use my DNS for backing up thousands of digital photos from my job.

Thanks, James

Check post #158 in this thread.  Fonz has compiled a version of rsync since 3.0, which should help with memory usage in rsync.  I haven't bothered going to this version yet, but I trust fonz that it should work fine with ffp 0.5.

Offline

 

#181 2008-10-28 16:17:09

fonz
Member / Developer
From: Berlin
Registered: 2007-02-06
Posts: 1716
Website

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

yojoe007 wrote:

And since ffp .05 includes rsync, does anyone know which version

http://www.inreto.de/dns323/fun-plug/0. … html#rsync

Offline

 

#182 2008-10-28 21:52:57

silversurfer
Member
Registered: 2008-07-20
Posts: 95

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

nurunet wrote:

I've been desperately trying to get my CH3SNAS to send mail. However, my sendmail reacts strangely:

root@NAS:/mnt/HD_a2/usbstorage/ffp# sendmail --help
sendmail: invalid option -- -
Invalid option ?
root@NAS:/mnt/HD_a2/usbstorage/ffp#

What am I doing wrong? Why doesn't sendmail (it *does* exist in my bin directory) know its options?

Do I have to configure sendmail anyplace else before using it?

Greets
Ben

Maybe you have a different version of busybox? Running "which sendmail" tells me that it is located in "/ffp/bin". Actually it is part of busybox. Running "sendmail --help" says:

Code:

BusyBox v1.11.1 (2008-07-13 20:04:00 CEST) multi-call binary

Usage: sendmail [-w timeout] [-U user] [-P password] [-X]
-t to [-t to]... [-n] [-s subject] [-c charset] server[:port] from [body] [attachment ...]

Send an email.

Options:
        -w timeout      Set timeout on network operations
        -U username     Authenticate with specified username/password
        -P password
        -t address      Recipient(s). May be repeated
        -X              Use openssl connection helper for secured servers
        -n              Request delivery notification to sender
        -s subject      Subject
        -c charset      Assumed charset for body and subject [utf-8]

You can see that it is busybox version 1.11.1. It is part of the current release of ffp 0.5.
Do a "busybox --help" to see what version you have got.

At the moment I have a problem with the mail notification anyway. If I run the script manually as root it sends the mail and if cron runs the script I get nothing. Cron seems to run as root, so that shouldn't be the cause of the problem. I need to investigate when I find the time.



@WildSioux

Have a look in the Wiki -> http://en.wikipedia.org/wiki/Cron  The fields in front of the command are explained there.

Offline

 

#183 2008-10-28 22:56:08

nurunet
Member
From: Germany
Registered: 2008-08-31
Posts: 44

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

root@NAS:~# which sendmail
/ffp/sbin/sendmail
root@NAS:~# busybox --help
BusyBox v1.12.1 (2008-09-29 20:38:04 CEST) multi-call binary
Copyright (C) 1998-2008 Erik Andersen, Rob Landley, Denys Vlasenko
and others. Licensed under GPLv2.
See source distribution for full notice.

Usage: busybox [function] [arguments]...
   or: function [arguments]...

        BusyBox is a multi-call binary that combines many common Unix
        utilities into a single executable.  Most people will create a
        link to busybox for each function they wish to use and BusyBox
        will act like whatever it was invoked as!

Currently defined functions:
        [, [[, adduser, adjtimex, arp, arping, ash, awk, basename, bbconfig,
        bunzip2, bzcat, bzip2, cal, cat, catv, chgrp, chmod, chown, chroot,
        chrt, chvt, cksum, clear, cmp, comm, cp, crond, crontab, cryptpw,
        cut, date, dc, dd, deallocvt, deluser, depmod, df, dirname, dmesg,
        dos2unix, du, echo, ed, egrep, eject, env, ether-wake, expand,
        expr, fakeidentd, false, fdisk, fgrep, find, fold, free, freeramdisk,
        fsck.minix, ftpget, ftpput, fuser, getopt, getty, grep, gunzip,
        gzip, halt, hdparm, head, hexdump, hostid, hostname, hwclock,
        id, ifconfig, ifdown, ifenslave, ifup, inetd, init, insmod, install,
        ip, ipaddr, ipcalc, ipcrm, ipcs, iplink, iproute, iprule, iptunnel,
        kill, killall, killall5, klogd, last, length, ln, logger, logname,
        logread, losetup, ls, lsmod, lzmacat, md5sum, mdev, mesg, microcom,
        mkdir, mkfifo, mkfs.minix, mknod, mkswap, mktemp, modprobe, more,
        mount, mountpoint, mv, nameif, nc, netstat, nice, nmeter, nohup,
        nslookup, od, openvt, pgrep, pidof, ping, pivot_root, pkill, poweroff,
        printenv, printf, ps, pwd, raidautorun, rdate, rdev, readahead,
        readlink, readprofile, realpath, reboot, renice, reset, resize,
        rm, rmdir, rmmod, route, run-parts, runlevel, rx, script, sed,
        seq, setconsole, setlogcons, setsid, sh, sha1sum, slattach, sleep,
        sort, split, stat, strings, stty, su, sulogin, sum, swapoff, swapon,
        switch_root, sync, sysctl, syslogd, tac, tail, tar, taskset, tee,
        telnet, telnetd, test, time, top, touch, tr, traceroute, true,
        tty, ttysize, udhcpc, umount, uname, uncompress, unexpand, uniq,
        unix2dos, unlzma, unzip, uptime, usleep, uudecode, uuencode, vconfig,
        vi, vlock, watch, watchdog, wc, wget, which, who, whoami, xargs,
        yes, zcat, zcip

root@NAS:~# sendmail --help
sendmail: invalid option -- -
Invalid option ?
root@NAS:~#

Seems like some things changed here, although I do have ffp 0.5. As I do have sendmail installed too, I still wonder why it won't take the --help argument.


Conceptronic CH3SNAS FW 1.03, ffp from USB key.

Offline

 

#184 2008-10-28 23:34:13

fonz
Member / Developer
From: Berlin
Registered: 2007-02-06
Posts: 1716
Website

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

nurunet wrote:

BusyBox v1.12.1 (2008-09-29 20:38:04 CEST) multi-call binary

sendmail is pretty new in busybox, and because I don't know how well it works, I removed it from busybox 1.12. There's esmtp as a more mature replacement. Use 'man sendmail' to get help.

Offline

 

#185 2008-10-29 22:47:34

nurunet
Member
From: Germany
Registered: 2008-08-31
Posts: 44

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

This explains a lot. For esmtp: I tried it, but it is too powerful for me, I guess. Couldn't get it to work.


Conceptronic CH3SNAS FW 1.03, ffp from USB key.

Offline

 

#186 2008-10-29 22:49:59

fonz
Member / Developer
From: Berlin
Registered: 2007-02-06
Posts: 1716
Website

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

nurunet wrote:

This explains a lot. For esmtp: I tried it, but it is too powerful for me, I guess. Couldn't get it to work.

It's not too difficult. There's good documentation with examples available at:
http://esmtp.sourceforge.net/manual.html

PS: And the best thing about esmtp is that it supports authentication (which the busybox sendmail cannot do).

Offline

 

#187 2008-10-29 23:28:03

nurunet
Member
From: Germany
Registered: 2008-08-31
Posts: 44

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

Thanks for your help, fonz, but I read that. Tried it, to no avail. See http://dns323.kood.org/forum/t3081-Send … esmtp.html for details.

I just wished someone got this to work and posted some kind of tutorial, it simply exceeds my Linux skills.

Have a nice evening!
Ben


Conceptronic CH3SNAS FW 1.03, ffp from USB key.

Offline

 

#188 2008-10-31 00:31:44

WildSioux
Member
Registered: 2007-03-05
Posts: 71

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

Chumby or anyone else...

Is there a way to set this script up to backup once a week instead of nightly?

If not, are there any other backup scripts for this box that would do that?

Thanks!

Offline

 

#189 2008-10-31 02:33:44

halfsoul
Member
Registered: 2008-01-28
Posts: 57

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

WildSioux wrote:

Chumby or anyone else...

Is there a way to set this script up to backup once a week instead of nightly?

If not, are there any other backup scripts for this box that would do that?

Thanks!

http://en.wikipedia.org/wiki/Crontab#Fields

Offline

 

#190 2008-10-31 12:49:13

silversurfer
Member
Registered: 2008-07-20
Posts: 95

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

Code:

# +---------------- minute (0 - 59)
# |  +------------- hour (0 - 23)
# |  |  +---------- day of month (1 - 31)
# |  |  |  +------- month (1 - 12)
# |  |  |  |  +---- day of week (0 - 6) (Sunday=0 or 7)
# |  |  |  |  |
  *  *  *  *  *  command to be executed

Example:

30 3 * * 6 command to be executed

This would run the command every Saturday at 03:30 o'clock.

Offline

 

#191 2008-11-01 07:16:09

Chumby
Member
From: Tasmania, Australia
Registered: 2007-08-01
Posts: 96
Website

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

Hey Silver... thanks for that ownership note.... have updated the tutorial to reflect the changes you suggest wink

Offline

 

#192 2008-11-01 22:57:56

halfsoul
Member
Registered: 2008-01-28
Posts: 57

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

Chumby wrote:

Hey Silver... thanks for that ownership note.... have updated the tutorial to reflect the changes you suggest wink

So if I change the rsync params to preserve ownership, then will it change files that already have been copied?  Or should I create a fresh sync?

Offline

 

#193 2008-11-02 00:17:50

silversurfer
Member
Registered: 2008-07-20
Posts: 95

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

halfsoul wrote:

So if I change the rsync params to preserve ownership, then will it change files that already have been copied?  Or should I create a fresh sync?

Well, I did a little test.

1. created a new file with ownership user:group
2. ran my sync script with -av once
3. checked that the file was on the target disk - it was there with user:group and the log file said that it was copied
4. changed ownership on target disk manually to root:root
5. ran the sync script again
6. checked the ownership of the file on the target disk - it was back to user:group

From this experience I would say that running rsync with the -av parameter will correct any ownership issues. Rsync didn't show anything in the log after step 5 because it didn't copy anything but as long as it fixes the ownership, that behaviour is fine for me.

Offline

 

#194 2008-11-02 15:44:45

silversurfer
Member
Registered: 2008-07-20
Posts: 95

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

By the way, I found out why my cron wouldn't want to email me about the results. I'm so silly, I forgot to provide the path to sendmail. roll lol

This is what it looks like right now. I added another two lines to backup my ffp to the disks too, just in case my USB stick stops working.

Code:

#!/bin/sh
#this script synchronizes disk 1 (right slot) with disk 2 (left slot)
#it can be used by cron to automatically sync disks over night

DISK1=/mnt/HD_a2
DISK2=/mnt/HD_b2
USB=/mnt/USB
SYNCOPT="-av --delete"
LOGFILE=/USB/var/log/sync-ab.log

#write the current date and time to the log
date > $LOGFILE

#backup ffp from USB to DISK1
/ffp/bin/rsync $SYNCOPT $USB/ $DISK1/USB_Backup >> $LOGFILE

#sync all the folders that we like
/ffp/bin/rsync $SYNCOPT $DISK1/Frank $DISK2 >> $LOGFILE
/ffp/bin/rsync $SYNCOPT $DISK1/share $DISK2 >> $LOGFILE
/ffp/bin/rsync $SYNCOPT $DISK1/USB_Backup $DISK2 >> $LOGFILE

#send email notification
/ffp/bin/sendmail -t receiver@domain -s "Rsync run" \
mailserver sender@domain $LOGFILE

Offline

 

#195 2008-11-08 16:15:57

nurunet
Member
From: Germany
Registered: 2008-08-31
Posts: 44

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

Chumby wrote:

2) Telnet in and from the # prompt, type crontab –l (lowercase L).  If everything has worked as it should you will see your rsync line as a scheduled task: (funplug 0.5 = ffp directory)

Code:

5 2 * * * /mnt/HD_a2/fun_plug.d/bin/rsync -rlptDv --delete /mnt/HD_a2 /mnt/HD_b2 >> /mnt/HD_a2/fun_plug.d/log/rsync.last.log 2>&1

This still shows the old parameters.

One question. I changed the parameters now. Will it have to copy everything again or can rsync just change the attributes?

Cheers
Ben


Conceptronic CH3SNAS FW 1.03, ffp from USB key.

Offline

 

#196 2008-11-08 21:00:55

halfsoul
Member
Registered: 2008-01-28
Posts: 57

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

@nurunet: The last word on this is from silversurfer two posts up:

silversurfer wrote:

Well, I did a little test.

1. created a new file with ownership user:group
2. ran my sync script with -av once
3. checked that the file was on the target disk - it was there with user:group and the log file said that it was copied
4. changed ownership on target disk manually to root:root
5. ran the sync script again
6. checked the ownership of the file on the target disk - it was back to user:group

From this experience I would say that running rsync with the -av parameter will correct any ownership issues. Rsync didn't show anything in the log after step 5 because it didn't copy anything but as long as it fixes the ownership, that behaviour is fine for me.

Offline

 

#197 2008-11-09 11:38:59

nurunet
Member
From: Germany
Registered: 2008-08-31
Posts: 44

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

Hey!

I hadn't understood, whether or not rsync would correct ownership by modifying the existing file or simply overwriting it with the new version - which would take far longer. So I changed the user and group manually and recursively for all files in the "HD_a2" directory on "HD_a1". Then I ran the new rsync task. Everything went smooth and fast.

Greetz
Ben


Conceptronic CH3SNAS FW 1.03, ffp from USB key.

Offline

 

#198 2008-11-10 19:30:49

yo_adrian_eh
Member
Registered: 2008-11-08
Posts: 12

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

I've been unable to get the script to work until this morning. I spent a good part of yesterday mucking around with this, this morning I deleted fun-plug completely and started from scratch, everything seemed fine but the backup still failed. As usual the fix was right in front on my nose. The file editcron.sh contains a reference for rsync to write to a log [/mnt/HD_a2/ffp/log/rsync.last.log]. I went to check the log, that file and folder didn't exist. I created the folder manually and ran the script again (changed the time in editcron.sh). All is well, the script ran as scheduled! Can anyone tell me if there fun-plug .05 install included the ffp/log folder? Might be worth adding to the instructions?

Yo!Adrian, eh?
Toronto

Offline

 

#199 2008-11-11 00:00:33

halfsoul
Member
Registered: 2008-01-28
Posts: 57

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

@yo_adrian_eh: You're not the first person to have that same issue.  crontab seems to fail without generating an error when the destination directory for a log file doesn't exist.  I agree that it would be nice to add to the instructions.

@newbie73: Assuming you've searched the forums and can't find a solution, perhaps you could start a new thread for your question, since it seems to be off-topic.

Offline

 

#200 2008-11-11 00:15:42

Chumby
Member
From: Tasmania, Australia
Registered: 2007-08-01
Posts: 96
Website

Re: Tutorial: Backup Everything from Vol A to Vol B once a night

Hi Halfsoul and yo_adrian_eh ... have updated the tutorial to show how to create the log directory if not already there (I can't remember if it is or isn't there by default... seems not sad )

Cheers
Chumby

Offline

 

Board footer

Powered by PunBB
© Copyright 2002–2010 PunBB