DSM-G600, DNS-3xx and NSA-220 Hack Forum

The DNS-323 has no raid controller. It is purely software. Did you power off the unit before you swapped the drive?

@ frodo - yeah I did power off. Didn't realise that it was purely software. In that case the software isn't up to scratch.

@ fordem - I think you should change you name to Captain Backup - that seems to be your response to just about every post that I have looked at in this forum. Most people do know the benefits of backing up. Just not everybody does backups. In this case, you should expect that a unit that sells itself as a RAID should operate properly - it doesn't, it screws up your data - I know, you're just gonna say - "Should have backed up" - Yep, but didn't and now am paying the price and hoping that my experience will warn others of this unit.

Hope you guys have a happy error free Christmas.

A.

fordem wrote:

Actually - as long as your data remained available up to the point that you inserted the replacement drive the RAID array fullfilled it's intended purpose - which is to reduce the impact of downtime caused by a failed drive.

SO.....

In a nut shell, your RAID array did what it was designed to do, and if you had done what you were supposed to do, you would still have your data.

And in response to that Captain Backup remark - I'm probably one of the reasons the people who don't back up know that they should have - so like you I'm hoping my experience will warn others - so that they know, even if they have RAID that they still need to back up - because that's not what RAID is for.

Okay fordem, I'll bite: what are you supposed to do once a drive has failed?

I'm a cautious engineer.  I purchased this box, and I've been running tests on it to determine what the correct use cases and what I can expect to happen are.  Using firmware 1.03 I've run into a very similar problem to what Aiden mentioned above, in that given my two disk array, when I remove one disk, make some changes, and then add the disk back in, the changes aren't synced correctly.

So what's the correct use case?  Does failure necessitate a full backup to another disk, manual reformat+rebuild, and then a copy back?

fordem -

Thanks for the details.  I think there is a lot of misunderstandings and erroneous expectations going around about what a RAID1 configuration is useful for.  Your points about user error (or, my personal fav example, viruses attacking available mounts) should be taken to heart by everyone.

My question is a little more specific.  I was under the impression that making a change on the drive, and then later adding an out-of-date member of the same RAID1 cluster should cause it to re-sync.  Is that an erroneous expectation on my part?  Or is that a valid use case that the current 1.03 firmware doesn't support (though I'm seeing some hints that 1.04 will)?

Mostly because I was looking at it as a change event.  RAID1 sees that the drives are not "in sync" and forces the old one to resync to the master.  I guess that's an erroneous way of looking at it

When I first purchased my DNS-323 , I tested the same scenario on a RAID1
DNS-323 as Gambit:

   remove a drive
   change some data
   replace drive to re-sync

and, sadly ,  I had the same results. 
I was quite disappointed with the D-Link firmware and if is wasn't for Fonz's
fun_plug, I would have returned my DNS-323 back then!

The only way I was able to successfully re-sync the removed drive was to use telnet access
and mdadm commands from the command prompt to fail and remove the "bad" drive. 
I plan to use this technique if a drive ever fails on my DNS-323.  I will not reboot my DNS-323
or allow D-Link's scripts to attempt to rebuild the array.

1) #cat /proc/mdstat   (to see which drive /dev/sda2 or /dev/sdb2 has failed)
2) #mdadm /dev/md0 --fail /dev/sdb2 --remove /dev/sdb2   (if /dev/sdb2 fails)
3) remove failed drive
4) data changes as I wait for new drive replacement, daily backups continue as usual
5) add the new drive, make sure it is added as /dev/sdb   (I once saw it was added as /dev/sdc??!)
6) use fdisk to examine partition on /dev/sda [working disk] 
    and create identical partitions on /dev/sdb [new disk]
7) #mdadm /dev/md0 --add /dev/sdb2   (add new disk to array) 
8) #cat /proc/mdstat   (to see progress of the re-sync)

Last edited by mig (2008-01-08 07:23:31)

gambit - You're right, I am SO disappointed with D-Link's firmware, I try not to rely on their
programming as much a possible.  I see it as fighting with D-Link's scripts rather than fighting
with the OS (Linux kernel / utilities). 

IMHO, the DNS-323 hardware is very impressive... small, quiet, low-power, dual SATAII,
fast and relatively cheap.  I think the concept of restoring all the configuration files from
flash on boot-up is brilliant!  But the lack of a journaling file system, old version of samba,
no UPS support, miserable clock drift makes this device a mediocre NAS. 

I've been commenting on these issues at the "official" D-Link forum, but none of the moderators
seem to want to post comments about them.  This coupled with the fact that the last firmware
update has been over 9 months ago, make me feel like D-Link doesn't plan to make many
improvements to the firmware.

If it wasn't for the fun_plug exploit and Fonz's software development, I would have given up
on this device long ago.   I would love to see a native Debian (or other distribution) kernel
loader.  There was some talk about this at http://dns323.kood.org/forum/t347-Loader-DNS-323.html
but no current progress has been reported.  I afraid kernel cross compilation is a bit beyond
my current Linux skills, but I always learn a lot when playing with these NAS devices (that is
until I actually use the device to provide a function in my network, then I have to stop tinkering
with is and just let it be)

About the root partition, I believe this exists in the RAM, so writing root is a bit space limited.
I have created several links (recreated from startup scripts each boot) from the root that point
to directories on the RAID to keep config files and users home directories.  This seems to work well.

fordem wrote:

I've discovered that the mods will delete your posts simply because you express an opinion challenging theirs - so it's not a forum for open discussion.

Yes, I saw your post disappear on the D-Link forum, too.  I though my post mentioning a
competing product to the DNS-323 in the "Defrag with Firmware 1.03" thread was susceptible
to removal.  Either the moderator didn't read it or let it slip by.

By the way, that device is a Buffalo Tech Linkstation Pro Duo, which could prove to be a
good replacement for my DNS-323, since is has several features I'm looking for... XFS filesystem,
and UPS support.  Although, perterjb seems to be making progress with NUT for UPS support on
the DNS-323

Thanks for the interesting discussion guys!

I stumbled on this while searching for the keyword "Input/output Error" when using "ls". The general "net conclusion" is drive failure. 

In my case I have a similar problem to another poster on this forum.  I have a mysterious "loss of data" in one of my directories which comes out with an Input/Output Error when doing a listing.  I am also running RAID1.  I've setup an rsync backup system of my RAID1 DNS to do remote backups to an offsite storage location.  In my debug stage I decided to just backup (using rsync) one test directory to another backup directory on the same RAID1 array. 

When I do this, the backup goes well and things look really good.  I have hourly backups showing up and changes I make are reflected in my most recent snapshot and older snapshots have hardlinks correctly made.  This morning I made an update to my fun_plug and then rebooted the DNS.  I checked my "backup" directory and all my snapshots were gone!  Totally gone!  I did an ls through telnet and it showed me the "input/output" error.  Very odd.  How does backed up snapshot data just "go missing". 

My only guess is that perhaps the DNS had not finished mirroring the data to both drives when i performed the reboot?  Is the D-Link software that dumb to not even clear out any syncing jobs before rebooting through the web interface? 

The story gets more interesting...

I went out for about 2 hours and came home.  I checked the backup directory and "magically" all my snapshots are back!  Was the DNS resyncing?  and if so it looked like it took a while!  Now I'm not sure if I should trust what I see but I certainly don't feel too keen on the DNS RAID implementation.  I've used hardware raid at work and the syncing of drives when writing a single 5MB directory is instantaneous.  I can't believe it would take more than minutes to do the same on the DNS.  Certainly I am still perplexed on what happened here but it seems like when I write to y RAID1 I should wait a while before rebooting it! 

Thought I'd add that in since my "missing directories" ended up showing up much later in time so patience might be a solution.  I don't like it though.

Thanks for the clarification.  My initial assumption of RAID1 (mirror) was that of simultaneous write but the DNS status page shows a "Drives last sync'd" status message so I was wondering if they had some kind of caching going on where the software writes to drive A and then eventually writes to drive B making it one of the worst RAID1 implementations I'd ever seen.

I have noticed that using rsync may have risks on SAMBA mounted windows shares from this site
http://www.mikerubel.org/computers/rsync_snapshots/

"One report came from a user who mounts a windows share via Samba, much as I do, and had files mysteriously being deleted from the backup even when they weren't deleted from the source. Tim Burt also used this technique, and was seeing files copied even when they hadn't changed. He determined that the problem was modification time precision; adding --modify-window=10 caused rsync to behave correctly in both cases. If you are rsync'ing from a SAMBA share, you must add --modify-window=10 or you may get inconsistent results. Update: --modify-window=1 should be sufficient. Yet another update: the problem appears to still be there. Please let me know if you use this method and files which should not be deleted are deleted."

Once I get my mysterious directories sorted out I might look into this further.

Gambit wrote:

fordem -

Thanks for the details.  I think there is a lot of misunderstandings and erroneous expectations going around about what a RAID1 configuration is useful for.  Your points about user error (or, my personal fav example, viruses attacking available mounts) should be taken to heart by everyone.

My question is a little more specific.  I was under the impression that making a change on the drive, and then later adding an out-of-date member of the same RAID1 cluster should cause it to re-sync.  Is that an erroneous expectation on my part?  Or is that a valid use case that the current 1.03 firmware doesn't support (though I'm seeing some hints that 1.04 will)?

Yes, its an error.  None of the RAID levels support this type of synchronization.  Its not a valid use-case for RAID at all.

To do that would require some sort of journal to be maintained or a full comparison of the drives file system to be constantly carried out.

In all cases of drive failure, RAID systems assume a fresh, blank drive is inserted/made available.  There is never synchronisation from an "outdated" version of a disk from a drive set.  That is outside the scope of what RAID provides.

This is the case for all of the RAID configurations (RAID1, RAID5, etc).

Another problem is that the device is using EXT2, which does not use journaling.  This means if the power goes off during a file write, the disk system will be in a bad state, and a tool like e2fsck needs to be run.  Normally Linux will do this automatically, but who knows if Dlink have configured this.  Hopefully it can be run from the Telnet.

The complication with RAID1 in this power failure situation is that the two disks may be slightly out of sync when the power goes.  So that's a bad situation to be in.  Read the Wikipedia article, concentrate on the topic on Atomicity of RAID arrays here: http://en.wikipedia.org/wiki/RAID

Basically what it says is that for a device like the 323 with software RAID, small CPU and likelihood of non-atomic writes, the two disks in a RAID1 array will probably be slightly out of sync with an uncompleted write when the power goes, and that is not great.  ie both disks will be partly through writing a file, and may be at different points in the write.  eg disk1 is 50% finished and disk2 is 51% finished.

Most likely it will be cleaned up by e2fsck, but its worth thinking about that maybe it won't be.  Even if you have EXT3 you will still have this problem, as RAID1 assumes that the disks are perfectly mirrored. 

In this scenario, they are not.

Therefore its essential to use a UPS to minimise this possibility.

fordem wrote:

Welcome aboard markchicobaby - it's nice to not be the only voice crying in the wilderness

You both are on the subject of "what RAID is" not alone. I read almost everything, and I am often amazed on various user posts.