DSM-G600, DNS-3xx and NSA-220 Hack Forum

rysncd is included in ffp. If not in the basic ffp install, as an additional package.
The wiki is your friend. See here and here.
And also this: http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp

Last edited by scaramanga (2010-08-26 23:07:39)

TangoMan wrote:

Thanks!

I've been searching the Wiki, but I'm still getting used to everything.  I had seen a number of pages about using rsync, but some (including a page or two on other sites) were talking about running rsyncd on the client computer (making it, really, a server).  It's up and running now, which is a huge relief.

With rsync it's up to you to decide how the backup/snapshot happens. Either your PC pushes the data to the DNS-323 (you'll need rsyncd configured and running for that) or the DNS-323 pulls the data from the PC. In the latter case (which I think is better) the PC is, indeed, the server.
The reason I think the second option is better is because it allows you to control (timing, load on DNS-323, network load) the backup/snapshot of multiple PCs from a single place.

Last edited by scaramanga (2010-08-27 12:01:07)

After reading this thread I was prompted to go and setup rsync on my 2 DNS323s as well. No issues as Fonz includes it in ffp.

I use both "push" and "pull" for various tasks and having the flexibility of running the daemon is great.

I decided to setup to check for updated ffp packages once a week automatically. The only issue I had was with adding the crontab jobs. On a reboot there seems to be some sort of timing issue. I added a script to append the new jobs but if I telnet in during bootup, I see the new jobs are added, then a few seconds later they are gone again. I added a 5 minute sleep to my script and that seems to have worked  but I didn't notice anyone else running into this issue.

Last edited by FunFiler (2010-08-28 11:46:08)

FunFiler wrote:

...I added a script to append the new jobs but if I telnet in during bootup, I see the new jobs are added, then a few seconds later they are gone again. I added a 5 minute sleep to my script and that seems to have worked  but I didn't notice anyone else running into this issue.

Isn't this the same situation with other config files being re-written at boot?  I noticed that if I edit /etc/exports, then reboot, it's been reset to what it was.

Does anyone know what script/program resets system files like this or if there's a way to stop it?

Fun Filer, I can see that with a lot of files, they're copying them from RAM, but /etc/exports HAS to be re-created from config info somewhere, since that gets modified by what a user enters in the web control panel.  It's not a dire need, but I do wish I could modify that file.  I guess, to do it, I'll have to do something like what you have: add a script that changes it back to what I want on boot.

Here's the latest iteration of my rsync script, which has some improvements/enhancements:
1. I added a post-backup command, which I use to email me the status of the backup and the log. If you'd like to set things up like that you'll need to configure msmtp/mutt (that's what I use). It's worth a different post. If anyone is interested let me know.
2. If a backup fails (=is incomplete) for any reason, the script will rename the backup directory so that it will be easy to see that. Also, now the script will keep the latest 7 complete backups. Previously, if a backup failed the oldest backup would not be deleted. But the next day, if the backup completes the two oldest backups were deleted. Now, the 7 latest complete backups would be kept.

#!/bin/sh

############
# Settings #
############

# Set source and exclude rules to govern what should be backed-up.
srcPaths="/mnt/HD_a2/Photos /mnt/HD_a2/Documents /mnt/HD_a2/ffp /mnt/HD_a2/downloads"
excludeRules="--exclude=/downloads/**.part"

# Set destination path, i.e. where should the backup be placed.
dstBasePath="/mnt/HD_b2/Backup_NAS"

rsyncOpts="-ahHix --fuzzy --stats"

# What is the maximum number of complete backups to keep. If there are more, 
# the oldest will be deleted.
maxSnapshotCount=7

# A command to-be executed after the backup operation is done. You can use the
# following variables in the command: 
#  RSYNC_EXIT_CODE  - The return code of rsync. Zero means success.
#  LOG_FILE         - The full-path to the log file.
#  DESTINATION_PATH - The full-path to the backup directory.
postBackupCmd='echo "Attached: snaptool log." | /usr/bin/mutt -n -F /ffp/usr/local/share/mutt/Muttrc -s "nas snapshot completed result[$RSYNC_EXIT_CODE]" -a $LOG_FILE my_email@gmail.com'

###################
# Run Environment #
###################

# Set path to Fun_Plug files.
# Fun_Plug 3.0 or 4.0
# ffpPath=/mnt/HD_a2/fun_plug.d
# Fun_Plug 5.0
ffpPath=/ffp


#
# Commands
#
rsyncCmd="${ffpPath}/bin/rsync"
dateCmd="/bin/date"
lsCmd="/bin/ls"
touchCmd="/bin/touch"
rmCmd="/bin/rm"
mvCmd="/bin/mv"
lnCmd="/bin/ln"
grepCmd="/bin/grep"
wcCmd="${ffpPath}/bin/wc"
headCmd="/bin/head"
echoCmd="/bin/echo"
mkdirCmd="/ffp/bin/mkdir"
awkCmd="${ffpPath}/bin/awk"

#
# Backup locations and file names
#
currentLinkName="current"
logFileName="snapshot.log"
failedBackupSufix="incomplete"

dstName=$(${dateCmd} "+%Y%m%d_%H%M%S")
dstPath="${dstBasePath}/${dstName}"
dstLog="${dstPath}/${logFileName}"
currLink="${dstBasePath}/${currentLinkName}"


##########
# Backup #
##########

${mkdirCmd} -p ${dstPath}

${echoCmd} "Executing ${rsyncCmd} ${rsyncOpts} --link-dest=${currLink} ${excludeRules} ${srcPaths} ${dstPath}" >> ${dstLog}
${echoCmd} "== ${rsyncCmd} log start ===================================" >> ${dstLog}

${rsyncCmd} ${rsyncOpts} --link-dest=${currLink} ${excludeRules} ${srcPaths} ${dstPath} >> ${dstLog} 2>&1
rsyncRC=$?

${echoCmd} "== ${rsyncCmd} log end =====================================" >> ${dstLog}

# This touch is not redundant. When getting files from a remote rsync daemon I noticed it sometimes modified the time of the
# destination directory to a value in the past. This screwed with the removal of old backups
${touchCmd} ${dstPath}

#######################
# Update Current Link #
#######################
if [ ${rsyncRC} -eq 0 ]; then
    ${rmCmd} ${currLink}
    ${lnCmd} -s ${dstName} ${currLink}
else
    ${echoCmd} "Something went wrong during backup (rc=${rsyncRC}). Current won't be updated." >> ${dstLog}
    ${echoCmd} "No snapshot(s) were removed." >> ${dstLog}

    # rename destination to mark it as an incomplete snapshot
    origDstPath=${dstPath}
    dstPath="${origDstPath}.${failedBackupSufix}"
    ${mvCmd} ${origDstPath} ${dstPath}
    dstLog="${dstPath}/${logFileName}"
fi

######################
# Remove Old Backups #
######################

cd ${dstBasePath}

# Count how many good backups there are
snapshotCount=$(${lsCmd} -1Atr . | ${grepCmd} -v ${currentLinkName} | ${grepCmd} -v "${failedBackupSufix}" | ${wcCmd} -l)

if [ ${snapshotCount} -gt ${maxSnapshotCount} ]; then

    # Please note that the output from ls is sorted in the correct order
    snapshotRemoveCount=$(( ${snapshotCount} - ${maxSnapshotCount} ))

    # What the code below does is list all snapshots ordered by date, then grabs
    # the N oldest (=first) good snapsthos and all incomplete snapshots
    # leading to them, where N is the number of snapshots to-be removed
    snapshotRemove=$(${lsCmd} -1Atr . | ${grepCmd} -v ${currentLinkName} | ${awkCmd} -v count=${snapshotRemoveCount} "! /${failedBackupSufix}/ { count--} {print} {if (count==0) exit 0};")

    ${echoCmd} "Removing the ${snapshotRemoveCount} oldest snapshot(s):" >> ${dstLog}
    ${echoCmd} "${snapshotRemove}" >> ${dstLog}
    ${rmCmd} -r ${snapshotRemove} >> ${dstLog} 2>&1
else
    ${echoCmd} "Found ${snapshotCount} snapshot(s) where ${maxSnapshotCount} allowed. None removed." >> ${dstLog}
fi

###########################
# Run Post Backup Command #
###########################

if [ -n "${postBackupCmd}" ]; then

    # Setup the post backup command run environment
    DESTINATION_PATH="${dstPath}"
    LOG_FILE="${dstLog}"
    RSYNC_EXIT_CODE="${rsyncRC}"

    ${echoCmd} "Executing ${postBackupCmd}" >> ${dstLog}
    eval ${postBackupCmd}
fi

exit ${rsyncRC}

Last edited by scaramanga (2011-09-11 21:02:51)

after a long LONG L-O-N-G time of use without any issues, I ran into a rare/weird problem using my rsync script. I can safely say this rare/weird problem can happen to you too, even if you're using the wiki script or some other tool.

What happened is that after cron started the snapshot backup, the built-in process that adjust the system clock to account for HW clock drift kicked-in and moved the system clock back in time. What happened next is that cron started the snapshot backup process a second time.
That caused two rsync processes to run at the same time trying to backup to the same location.

The rsync log(s) contained errors like:

rsync: failed to hard-link /mnt/HD_b2/Backup_NAS/current/ffp/share/man/man3/ASN1_OBJECT_free.3 with ffp/share/man/man3/ASN1_OBJECT_free.3: File exists (17)
rsync: symlink "/mnt/HD_b2/Backup_NAS/20120903_035001/ffp/share/man/man3/ASN1_OBJECT_free.3" -> "ASN1_OBJECT_new.3" failed: File exists (17)
...
rsync error: errors with program diagnostics (code 13) at hlink.c(548) [generator=3.0.9]
rsync: connection unexpectedly closed (14994 bytes received so far) [sender]
rsync error: errors with program diagnostics (code 13) at io.c(605) [sender=3.0.9]

(yes, I backup my ffp library)

and the rsync exit code(s) were 13 and 23

I've updated the snapshot script (added a simple lock file) to prevent this from happening. I'm testing it and will post the updated script soon.

Last edited by scaramanga (2012-09-05 11:27:51)

bjby wrote:

I usually guard against this by listing running processes. Lock files often get into the "wrong" state unless you have some robust logic to clear them.

#exit if already running
RESULT=$(ps -ef | egrep 'lftp' | grep -v grep )
if [ -n "$RESULT" ] ; then
        echo already running...;
        exit 0;
fi

You've got a point. I'll do that instead.
Thanks!

There are lots of examples people writing lock file templates in bash, doing it more solid than my example maybe this one will work in ffp environment.

http://www.franzone.com/2007/09/23/how- … y-running/

I do not use it, i do a weekly mirror rsync one disk to other.  I keep the mirror disk mounted read-only except when writing the mirror. Keeps backups somewhat safe against accidental recursive deletes.