DSM-G600, DNS-3xx and NSA-220 Hack Forum

hooligan · 2011-12-18 17:23:56

Please bear with me., Linux is not even close to being my 432nd language...

Using Alt-F (RC1)

I have two 323's with matching drives (JBOD). I currently have them setup with my media on disk 1 of NAS 1 and my users are setup on disk 2 of NAS 2. Every night I run a Windows Robocopy and mirror disk1/NAS1 to disk1/NAS2 and disk2/NAS2 to disk2/NAS1. Both NASs have duplicate shares, users, security, etc. Sort of a poor mans cluster. All my PC's have a login script that can be changed should a disk or NAS go down. This helps with throughput and also gives me a single nights backup...

What I'm looking for is a method to mirror these without the use of the Windows box. I had been hoping the Backup function in ALT-F would allow for some kind of rsync between the NASs but it only seems to allow for local disks. I understand that rsync is in the default install package but I can't quite sort out where to start to get some kind of scheduled script to accomplish this. Any kind of pointers would be greatly appreciated.

Warren

Last edited by hooligan (2011-12-18 17:25:23)

jcard · 2011-12-19 19:10:23

hooligan wrote:
Please bear with me., Linux is not even close to being my 432nd language...

Using Alt-F (RC1)

I have two 323's with matching drives (JBOD). I currently have them setup with my media on disk 1 of NAS 1 and my users are setup on disk 2 of NAS 2. Every night I run a Windows Robocopy and mirror disk1/NAS1 to disk1/NAS2 and disk2/NAS2 to disk2/NAS1. Both NASs have duplicate shares, users, security, etc. Sort of a poor mans cluster. All my PC's have a login script that can be changed should a disk or NAS go down. This helps with throughput and also gives me a single nights backup...

What I'm looking for is a method to mirror these without the use of the Windows box. I had been hoping the Backup function in ALT-F would allow for some kind of rsync between the NASs but it only seems to allow for local disks.

From the 0.1RC1 announcement:

-The Backup Service can now backup remote hosts using ftp/http/smb/nfs/rsync/rsyncd.
It can even wakeup the remote host at backup time if the remote host is configured
to be awaked-up by a network magic packet, WOL.
Files greater than 4.2GB are not supported.

And there is online help for the backup module (Services->System->backup->Configure) just hit the (?) icon near the page title.
And of course it is possible to backup a dns box from another dns box.

But I'm afraid that pure mirroring is not possible, the Backup module creates backups on the Backup directory, not what you want (although you could easily move a given backup to a directory of your choice -- if the move is done within the same filesystem it will be almost instantaneous).

I understand that rsync is in the default install package but I can't quite sort out where to start to get some kind of scheduled script to accomplish this. Any kind of pointers would be greatly appreciated.

Warren

hooligan · 2011-12-19 23:27:38

Ok. Thanx for that... I guess I will stay with the current method. I want each NAS to be the same with no work other than remapping... I also have quite a few files over 4 gig...

Thanx again...
Warren

bjby · 2011-12-21 07:53:23

Some pointers. Google the details

Mount the filesystem (samba share) of one dns323 into the other.

Create a rsync command script to sync folders.

Schedule it using cron (crontab)

hooligan · 2011-12-23 13:16:56

Thanx... I had just tried the mounting idea yesterday. But I didn't know if it world work. I haven't taken it any further yet but I do appreciate the short step by step and knowing I was kind of heading in the right direction...

Warren

bjby · 2011-12-24 03:06:23

I dont have your exact environment therefore its hard to make detailed instructions.

But to break down the first step of mounting the one dns323 into another.

1. log in to one dns323 with telnet and ssh.
2. Google a guide or three for "mounting a samba share"
3. Try retry and pokearound

hooligan · 2011-12-24 14:56:07

I have question from jcard's original response:
"-The Backup Service can now backup remote hosts using ftp/http/smb/nfs/rsync/rsyncd.
It can even wakeup the remote host at backup time if the remote host is configured
to be awaked-up by a network magic packet, WOL.
Files greater than 4.2GB are not supported. "

Does this mean the built in rsync will not deal properly with files larger then 4.2 gig or is this only related to the interaction of the Backup Service with rsync? This will be between to 323's running matching ALT-F's (RC1 at the moment).

Assuming the rsync can deal with the larger files, my plan is a script that:

makes the mount point directory(s)
mounts the remote share(s)
removes the old log
rsync the data from the mount(s) to the local drive, writing a new log
unmount the remote share(s)
remove the mount point directory(s)

This will allow for me to forget how I've done all this in a couple or three years and only have to delete/change the script to undo/change things with no bits and pieces to unravel later. Nothing worse then looking at a server and wondering what all the undocumented left over software, bit, pieces, and folders were/are for...

Then scheduling it in some fashion. I had no luck playing with cron yesterday but I assume its a "stupid user trick" on my part...

Do this on both 323's

Thanx again for the help and the pointers...

Warren

Last edited by hooligan (2011-12-24 15:54:49)

hooligan · 2011-12-29 01:14:28

Well I tried an 8 gig video file and its seemed to rsync properly so I tried an entire directory. Rsync seems to have problems with some characters in my file names (things like ' and such) Nothing I couldn't deal with but when I went and did a full rsync of my media folder It regularly reported vanished files and then aborted with I/O errors. It wasn't copying very many files as they were already synced up. Just the ones I renamed to get rid of the characters rsync couldn't deal with. I tried this multiple times with the same result. So... No joy... This is what I used in a telnet session...

mkdir /mnt/sda2/srv01media
mount -t cifs -o username=???,password=????? //192.168.xxx.xxx/Media /mnt/sda2/srv01media
rsync -avz -stats --delete-after /mnt/sda2/srv01media/ /mnt/sda2/Media/ > /mnt/sdb2/Users/wquin/Backups/rsync.log
umount /mnt/sda2/srv01media
rmdir /mnt/sda2/srv01media

Afterwards, to be sure it wasn't really an I/O error, I deleted the backup SDA2 drive and recreated it using EXT4 and used robocopy to mirror it again. No problems... So... No idea what the issue is...

Thanx for all the help with this experiment...

Warren

scaramanga · 2011-12-29 14:34:37

try running rsync with

Code:

-ai --stats

The extra output might help figure out what's going on.
The "-z" option does not make much sense to me in your case, since it's all happening locally.
Also, note that there's a - missing in front of "-stats".

Last edited by scaramanga (2011-12-29 14:42:35)

hooligan · 2012-01-04 13:50:59

Thanx for catching the missing "-"...

I tried the the change (-ai) but no luck... Same thing. Random "vanishing" files etc...

Going to give up on this for now and maybe revisit it after I get a little more experience. It's probably, as I mentioned earlier, as stupid user trick....

I now have a new issue but it needs a new post...

Thanx again...

Last edited by hooligan (2012-01-08 16:39:38)

jcard · 2012-01-10 19:21:37

The backup 4GB file limit is Alt-F fault, as it uses 'cpio' to do the initial, first, copy. This has been already discussed.

As for the errors, probably you have to supply some "character set" mount options, the default is utf-8. Does your files have "funny names"?
After mounting, can you list and see the files correct names? Try a "ls -R <mount-path>" command.
Can you copy files? Does a "cp -a <mount-path> <dest-path>" works? Start with a small subdirectory

For a faster "service" please use Alt-F google group

hooligan · 2012-01-11 14:16:16

yes some have funny names. Not a problem as i could rename them. My real problem looks like security.

Using the Web GUI I create a folder with a group called admin having read/write/browse with owner root. I robocopy /mir /copy:dt a folder from c:\users\wquin to \\fileserver\users\wquin as myself (its actually a scheduled task) on my Windows box (matching users/passwords on Win7 and all 323's). The entire contents end up as owner wquin/group user (okay a linux thing, not inheriting from the root). On the next robocopy pass I find random files that get an access denied. I manually try to overwrite them. No go but I can delete them. I delete them for the next pass of robocopy but I still end up with random files with no overwrite. I try using the web interface to change the security recursively on files. This only works if I select one option at a time. Change the group. it works. Change the group and other users, no effect. And yet I'm pretty sure this worked when I started out on this adventure. Now - Change the group. OK. Change the other users. OK. Change security on folders recursively. OK. Change security on Files. OK. Any combination and no effect. And still sometimes the files do not allow overwrite. I've started doing a chown -R and chmod -R 777 on the folders every morning so I can get a complete backup that night. This is probably related to the rsync issues I started with...

I have the same issue copying from nas to nas. Random files losing the ability to overwrite.

Of course it could also be I don't understand...

So I'm still playing...

Warren

Last edited by hooligan (2012-01-13 04:03:46)

hooligan · 2012-01-15 19:26:44

All fixed... Different group and user UIDs between the 2 323s... I recreated the users and groups on one unit to match the first. On both boxes I then did a chown -R to set the permissions on the shared folders the way I wanted them and then a chmod -R 2775 and chgmod -R 2770 so that all the user and group permissions on existing data matched on both boxes. I then modified the smb.conf on both units:

In the share sections I added:
create mask = 0770
directory mode = 0770

Restarted samba and away we went...

Now all new files/folders are created with the appropriate group access and "rsync" worked as expected except for a character set issue. "Rsync" was replacing any special characters (french etc) with an underline. In the end I tracked this down to the "mount" command not "rsync". I added "iocharset=utf8" to the "-o" in the "mount" command line and all is good.

For reference the web interface did not appear to work as expected so I did it through telnet and will probably keep on that way...

Thanx again to jcard for a great bit of kit for this little box...

Warren

Last edited by hooligan (2012-01-16 04:08:07)

DSM-G600, DNS-3xx and NSA-220 Hack Forum

Announcement

#1 2011-12-18 17:23:56

Need help with a backup/mirror

#2 2011-12-19 19:10:23

Re: Need help with a backup/mirror

hooligan wrote:

#3 2011-12-19 23:27:38

Re: Need help with a backup/mirror

#4 2011-12-21 07:53:23

Re: Need help with a backup/mirror

#5 2011-12-23 13:16:56

Re: Need help with a backup/mirror

#6 2011-12-24 03:06:23

Re: Need help with a backup/mirror

#7 2011-12-24 14:56:07

Re: Need help with a backup/mirror

#8 2011-12-29 01:14:28

Re: Need help with a backup/mirror

#9 2011-12-29 14:34:37

Re: Need help with a backup/mirror

Code:

#10 2012-01-04 13:50:59

Re: Need help with a backup/mirror

#11 2012-01-10 19:21:37

Re: Need help with a backup/mirror

#12 2012-01-11 14:16:16

Re: Need help with a backup/mirror

#13 2012-01-15 19:26:44

Re: Need help with a backup/mirror

Board footer