DSM-G600, DNS-3xx and NSA-220 Hack Forum

fonz wrote:

http://dns323.kood.org/forum/p21857-Yesterday-16%3A47%3A33.html#p21857

Thanks!  :-)  Have started setting up time-based rsnapshot to remote NAS.  :-)

ozziegt wrote:

My script isn't updating the symlink when it is run. If I run it manually, the symlink updates but when the cron runs it, the symlink is not changed. Any ideas what could cause this?

In the "if" statement wc (wordcount) is called. There are (at least) two wc applications on your NAS:

/ffp/bin/wc (This one works)
/bin/wc (This one doesn't)

When you run the script manually, wc is used from your ffp environment. This one works --> the "if" is true -_> the link is created.
When crond is running the script, wc is used from the cron environment. This one doesn't --> If fails --> no link.

The solution is to add
   export PATH=/ffp/sbin:/ffp/bin:$PATH
as the second line in your script setting up the ffp environment.

Hi all.  I'm a Linux infant so please be patient with me...

I setup my DSN323 as per the wiki's how-to guide (http://dns323.kood.org/howto:backup) based on raid123's thread here.  It's doing a daily backup but it doesn't appear to be linking files.  What it's doing is a full backup each time.  I believe this is what's happening because a) each backup is taking about 12 hrs (200+ gb of data), b) drive space on my backup drive is decreasing about about 200 gb after each backup, and c) when I do a ls -ln on the backup drive, I see all the directories made up of date_time but none are listed as ->current.

Can someone please offer some suggestions as to what might  be wrong?  How can I get it to start making virtual copies of the unchanged files?  I compared my snapshot.sh file to the one on the wiki many, many times and don't see any differences.  Same thing with my editcron file.  And I've restarted my DSN323.

Loose Gravel wrote:

btw: you are using firmware 1.5, ffp 0.5, aren't you?

Yep, firmware 1.5 and ffp 0.5.

EDIT: As your backup job runs 12 hours it may be wise to disable the cronjob wile running snapshot.sh manually (or run the cronjob only and wait two nights...)

Yeah, good idea. 

Thx for the suggestions.  I'll give 'em a shot when I get home.

shepherd wong wrote:

Loose Gravel, thanks very, very much for your help.

Glad I could help.

shepherd wong wrote:

That path statement really should be added to the wiki, I think.

Good idea. Added a troubleshooting section.  I didn't modify the script, because this could be used with an old ffp version also, and I do not know the correct path for e.g. ffp 0.4.

ozziegt wrote:

I had this working, and then I decided to clear out my snapshots folder and start over because I had restructured the files. However, now it isn't creating the "current" folder. It is giving the following error:

"--link-dest arg does not exist: /mnt/HD_b2/v1_snapshots/current"

Any ideas? Thanks

Just experienced the same thing though I did not see anyone respond to this. Anyone?

Loose Gravel wrote:

Thats the magic of this time machine script. You see full Backups every day (you will see all files in the backup directory - so its a full backup from a users perspective), but an unchanged file will be stored only once on disk (using diskspace - so its an incremental backup from a systems perspective). This is done by using hardlinks.

What you can do:
- You can burn a Backup-day-folder to dvd or copy the Backup to another disk --> you get your files (not the links) automatically
- You can delete any backup-day-folder - the file will remain on disk as long as it is used in another backup-day-folder.
- If you delete the last backup-day-folder using a file, the file is removed from disk.

You can check this out:

ls -al in the day2-folder. (-al shows a lot of information about the file. One is the link-count). 1.txt will show a linkcount of 2 (because it is used in day1 and in day2 folder), but 2.txt will show a linkcount of 1 (used only in day 2). ls -al in the day1-folder. 1.txt will show a linkcount of 2 (because its the same 1.txt used in the day2-folder).
If you delete the day1-folder, 1.txt will stay in the day2 folder. Linkcount will be 1 (because it is only used in day2 now)

So it IS some kind of magic

EDIT: Typo

Not sure if this thread is active however, I believe that I am mistaken in understanding this - that is because I am windows guy with some knowledge of linux. Due to my limited knoledge I am not sure if this is right or wrong.

If I run du -d 1 I get this

785616  ./20090819_020011
785616  ./20090818_020010
786172  ./20090820_020010
786172  ./20090821_020010
787640  ./20090821_173527
3931220 .

See each directory is 785MB and over and adding to 3.7G shown above. How is this possible? This means the script is creating full backups everyday. Am I missing something - I must be. I will appreciate if someone can shed some light please.

I have DNS-323 with FW 1.07 and fun plug version 0.5 and raid123's script that works very well. Thanks raid123 for such simplicity.

Last edited by lagreat (2009-08-22 15:31:25)

There is an easy test: Do a

df

(diskfree) before a backup and afterwards. Look at "used" on your Backup device.  It should increase only slightly. (Remember: Linking files will use diskspace also, but only some bytes per file.). If you are backing up lets say 10 GB unchanged files, and the used diskspace increases 10 GB, then something is going wrong. In that case: Post your sript.

Gravel

I haven't checked back to this forum for a while, and looks like some people had problems with future versions of ffp and wc.  I think at one point I made the suggestion of removing the if check, which is what my current version has:

#!/bin/sh

srcpath='/mnt/HD_a2'
dstpath=/mnt/HD_b2
ffppath=/ffp
date=`date "+%Y%m%d_%H%M%S"`
mkdir $dstpath/$date
$ffppath/bin/rsync -aivx --link-dest=$dstpath/current $srcpath $dstpath/$date > $ffppath/log/snapshot.log 2>&1
rm -f $dstpath/current
ln -s $date $dstpath/current

Others have mentioned the desire to clean up the backups.  I have a script that I wrote, but it's very specific to my own needs.  My backup scheme involves keeping daily backups for the 2 latest months, and then keeping monthly backups for 6 months prior to that.  If this meets your needs, feel free to take a look at my script.

** DISCLAIMER: THE SCRIPT HAS NOT BEEN EXTENSIVELY TESTED, SO USE AT YOUR OWN RISK **

That said, I've been using it for quite a while with the standard snapshot.sh script and it works for me.  It does use wildcards, so if you keep other files in the backup drive, you should understand what the script does before deciding whether or not to use it.  You may also want to backup your backup first.

Here's the script I use:

#!/bin/sh

path=/mnt/HD_b2/
months=6

decrement_month()
{
    let month=$(($month-1))
    if [ $month -eq 0 ]
    then
        month=12
        let year=$(($year-1))
    fi
}

year=$(date "+%Y")
month=$(date "+%m")
filetail="_*"
filetaildate="??_*"

decrement_month

while [ $months -gt 0 ]
do
    decrement_month
    let yearmonth=$(($year*100+$month))
    let yearmonthfirst=$(($year*10000+$month*100+1))
    if [ -d $path$yearmonth ]
    then
        echo Skipping $path$yearmonth
    else
        echo Processing $path$yearmonth
        mv $path$yearmonthfirst$filetail $path$yearmonth
        rm -rf $path$yearmonth$filetaildate
    fi
    let months=$(($months-1))
done

Obviously I haven't put in the time to make it customizable, mainly because it takes time to test that the customizations indeed work.  The first decrement_month, for example, is to make it keep 2 months of daily backups.  That way, on the 1st of every month, there's still at least 28 days of daily backup available.  This script also assumes that rm -rf will complete.  If somehow the program crashes or the device resets, you might end up with some orphan directories that needs to be cleaned up manually.  I haven't had the time to make it more robust...  sorry! 

Note that I'm on an older version of ffp, so some modifications may be necessary to use this on the latest version.

I believe I have fixed the issue after reading this thread all over again. Boy that was some 8+ hours of work but if I have indeed fixed it for my case I am happy to have put in that time. Now to what I did.

- I read a reply from loose gravel suggesting to add 'export PATH=/ffp/sbin:/ffp/bin:$PATH' and further suggests that there are two wc commands. However on searching my disks using find -name wc* at '/' on /mnt/HD_a2 I found just one in /ffp/bin/wc, /bin/wc was not found. I still added the command and ran ./snapshot.sh and it gave me another 16.6 GB folder

- I commented the line 2nd line 'export PATH=/ffp/sbin:/ffp/bin:$PATH' and ran again - this created 16.6GB folder again. du and df proved that the sizes are indeed real.

- I read from last comment by raid123 not requiring the 'if then fi' combination and left out the two lines after 'then'.

- Chose a smaller folder with 600+ mb as srcpath. It started to work as I ran it on schedule every 5 minutes. Each folder will increase by just few KB instead of 600+ MB. df and du agreed too.

Present script looks like this:
=========================
#!/bin/sh

# export PATH=/ffp/sbin:/ffp/bin:$PATH
srcpath=/mnt/HD_a2/homedrives
dstpath=/mnt/HD_b2/daily-backup
ffppath=/mnt/HD_a2/ffp
date=`date "+%Y%m%d_%H%M%S"`
mkdir $dstpath/$date
$ffppath/bin/rsync -aivx --link-dest=$dstpath/current --exclude='workspace/' $srcpath $dstpath/$date > $ffppath/log/snapshot-$date.log 2>&1
# if [ $(ls -1A $dstpath/$date | wc -l) -ne 0 ]
# then
    rm $dstpath/current
    ln -s $date $dstpath/current
# fi
==========================

Removed the 5 minute schedule and manually backed up the original 16.6 GB. I will confirm how it is after the backup finishes at 2AM tonight and post it here. However, if any of you believe that I have made any mistake please correct me. I am M$ trained so it is easy for me to overlook things in Linux. Your guidance has helped me, and I will appreciate your comments for my own edification.

Once this is settled, I will work with raid123's script however not sure how I'll test as I won't have folders for last 60 days.

Thanks to all.

Hi,

I am reading this forum and found a lot of useful stuff.

I have a problem. The snapshot.sh script works, but i don't know how to get files from the windows server which is in a domain.
I'm a newbie in linux and don't know if it only the problem in the path (srcpath) or you must in a way first log on to the domain and then execute the script.

Thanks in advance

Just a small thoughts of mine about purging old backups.

I'm using weekly incremental backup. The rsync does not have the feature to delete old backups before disk became full, so I use these two additional lines in my script before starting rsync:

---

dateold=`date -d '-5 week' +%Y%m%d`
rm -r /mnt/HD_b2/backup/back-${dateold}*

---

I have the actual and the last 4 weeks' backup. Of course if you have daily backup you have to use '-x days'. In case you have a more complicated way of backup, this method won't work.

Iknow this is not a "clear" way: it not verifies whether -5 week date exists or not, rm will send a "not found" error message, but it does what I want...

I hope I have helped.

Last edited by Rival (2010-02-02 12:19:13)

I'm posting here my version for the editcron.sh script. It has some additional "features":
1. Can be used like other /ffp/start scripts to start, stop, restart or check status of cron job.
2. Safe to call start multiple times - job will only be scheduled once.
3. separated "data" (the schedule, the cron job) from the code for easy editing.
4. Made all commands path's super-easy to modify to accommodate different fun-plug versions.

Your inputs are always welcomed.

P.S.: I thought of turning this into a whole system that supports multiple cron jobs. If anyone finds that interesting please let me know.

#!/bin/sh

# PROVIDE: Editcron

#
# Schedule job
# Important: Always run "./editcron.sh stop" BEFORE making changes to job.
#            Otherwise, the old job will remain scheduled until you restart.
#
schedule="0 2 * * *"
job="/ffp/bin/snapshot.sh"

#
# Commands
#
grepcmd="/bin/grep"
crontabcmd="/bin/crontab"
rmcmd="/bin/rm"
echocmd="/bin/echo"

#
# Tempporary files
#
TMPDIR="/tmp"
TMPCRONTXT="${TMPDIR}/crontab.txt"
TMPOTHERCRONTXT="${TMPDIR}/othercrontab.txt"

#
# FFP Start Functions
#
. /ffp/etc/ffp.subr
start_cmd="editcron_start"
stop_cmd="editcron_stop"
status_cmd="editcron_status"

editcron_start()
{
    # grab existing crontab
    ${crontabcmd} -l > ${TMPCRONTXT}

    # check if already scheduled
    cronjobs=$(${grepcmd} "${job}" ${TMPCRONTXT})
    if test -n "${cronjobs}"; then
        ${echocmd} "${job} already scheduled:"
        ${echocmd} "${cronjobs}"
    else
        # add the cron job and install the new one
        ${echocmd} "Scheduling ${schedule} ${job}"
        ${echocmd} "${schedule} ${job}" >> ${TMPCRONTXT}
        ${crontabcmd} ${TMPCRONTXT}
    fi

    # clean up
    ${rmcmd} ${TMPCRONTXT}
}

editcron_stop()
{
    # grab existing crontab
    ${crontabcmd} -l > ${TMPCRONTXT}

    # check if already scheduled
    cronjobs=$(${grepcmd} "${job}" ${TMPCRONTXT})
    if test -z "${cronjobs}"; then
        ${echocmd} "${job} not scheduled"
    else
        # remove the cron job(s) and install the new one
        ${grepcmd} -v "${job}" ${TMPCRONTXT} > ${TMPOTHERCRONTXT} 
        ${crontabcmd} ${TMPOTHERCRONTXT} 
        ${echocmd} "${job} not longer scheduled"

        # clean up
        ${rmcmd} ${TMPOTHERCRONTXT} 
    fi

    # clean up
    ${rmcmd} ${TMPCRONTXT}
}

editcron_status()
{
    # grab existing crontab
    ${crontabcmd} -l > ${TMPCRONTXT}
    cronjobs=$(${grepcmd} "${job}" ${TMPCRONTXT})

    # check if already scheduled
    if test -n "${cronjobs}"; then
        ${echocmd} "${job} is scheduled:"
        ${echocmd} "${cronjobs}"
    else
        ${echocmd} "${job} not scheduled"
    fi

    # clean up
    ${rmcmd} ${TMPCRONTXT}
}

run_rc_command "$1"

EDIT: Only tested with FW 1.08 and fun-plug 0.5

Last edited by scaramanga (2010-08-12 13:58:39)