DSM-G600, DNS-3xx and NSA-220 Hack Forum

jayas wrote:

Dear Fordem

fordem wrote:

Do we possibly have a scenario here where the DNS-323 is detecting the data as not being consistent and flagging it as such?

I think that is letting your imagination run a bit too wild.    Do we think it would to this extent when it does not even use S.M.A.R.T?  I don't think so.

Jaya

I'll willingly admit that you may be right about the imagination part - since I have no idea what it does or how it does it - and since mine doesn't do it, it's hard for me to do anything other than theorize - BUT - trust me on this - systems have been doing this for years, my xSeries IBM (Adaptec based RAID) detects the array as out of synch before the OS loads, as do other hardware controllers.

By the way - S.M.A.R.T does not & can not play a part in verifying data integrity at startup- it's simply a threshold based detection system - a number of different variables are monitored - for example the number of remapped sectors and when the count exceeds a predetermined threshold, S.M.A.R.T will flag it.



The consistency detection mechanism needs to be nothing more than a flag being set/reset - set the flag before writing to disk#1, reset the flag upon completion of the duplicate write to disk#2 - a power fail occuring before the completion of the write will leave the flag set, and this can be detected on the next boot up.

Sure it's a theory - but I started my post by making that clear - Feel free to shoot my theories down - but - please, do so constructively - either show me why I'm wrong, or show me a better theory.  Better yet - since you are more conversant with linux - run with my theory and either blow it out of the water, or verify that I'm right - right now you're ridiculing my guess based on your guess.

For what it's worth, I've had my DNS-323 about 3 weeks now. I initially started with a single 500Gb Samsung HD and upgraded from 1.03 to 1.04 at the initial installation.

After a week I upgraded to a second Samsubg 500Gb and switched to RAID1.

Though I have had other excitement on my setup (see here), RAID1 appears to have been fine over this period and I get the normal messages on the Status page. Frm time to time, I notice a few more clunks and clicks from the new setup than I did from the single disk - guess RAID1 is keeping the two disks a little busier as it spreads the data.

Colin

Hi Basher,

Basher wrote:

Upgraded to 1.04  ...
[ first left drive had amber light ...]
[ then right drive had amber light ...]
Will be waiting for a 1.05.  1.04 looks to be suspect.

DLINK asked a question as to what kind of drives people who have this problem are using.  I wonder how many replied and if DLINK has an idea what the problem is.  Otherwise it would not be reasonble to expect this problem to be fixed in 1.05 firmware.

Is there anything you can tell us DLINK?

Jaya

Megistal - that's an interesting observation - are you running a fun_plug and are you turning it off using the button?

My experiences have been different - I have the first hardware revision, with fw 1.04 off of the US support site and 2x250 GB Seagate Barracuda 7200.9 in a RAID1.

It runs not quite 24/7, but as close to that as I can manage given our less than reliable utility power - I can read/write/delete - whatever I want and the unit is almost never properly shut down, and the only time I've seen that pink/amber light is back when I had JBOD and hot unplugged a drive.

I suspect I've never seen the problem because my DNS-323 will always be the last thing to shut down and so it's never reading/writing when the power goes - it's fed from a PowerWare UPS that shuts a Windows server down before turning off - I did have a script so that the server would do a proper shutdown on the DNS-323, but I reinstalled the server (larger drives) and haven't put the script back yet.

jayas wrote:

Hi Basher,

Basher wrote:

Upgraded to 1.04  ...
[ first left drive had amber light ...]
[ then right drive had amber light ...]
Will be waiting for a 1.05.  1.04 looks to be suspect.

DLINK asked a question as to what kind of drives people who have this problem are using.  I wonder how many replied and if DLINK has an idea what the problem is.  Otherwise it would not be reasonble to expect this problem to be fixed in 1.05 firmware.

Is there anything you can tell us DLINK?

Jaya

In short,

In 1.03 drive failure detection was not working properly. It was fixed in 1.04. Alot of the issues we have seen with the Raid degraded status have been related to some firmware drivers on some WD disks causing critical errors which cause our device to flag it as a failed drive. Still a work in progress and hoping to fixed before 1.05 goes public.

It is difficult to pin point these types of issues since we have limited resources to hard drives. We collect as many different sizes/makes/models as possible but there will always be a few that we do not have access to for testing.

Hi DLINK

Dlink wrote:

In 1.03 drive failure detection was not working properly. It was fixed in 1.04. Alot of the issues we have seen with the Raid degraded status have been related to some firmware drivers on some WD disks causing critical errors which cause our device to flag it as a failed drive. Still a work in progress and hoping to fixed before 1.05 goes public.

I am not sure if the problem is restricted to WD disks.  At least I have seen it occur on DNS-323 with Seagate drives although I suspected (don't recall my reasons as it was a while ago) that power interruption may have caused it.

However in investigating at why the hot-plugging fails I found issues with scripts.   Is there a channel for issues and suggestions to be relayed to the development team?

Jaya

Megistal wrote:

Hi DLink

Could you provide which functionality was fixed in 1.04 for the raid 1 setup?

Looking at the release notes from the DLink web site for the DNS-323, I can't see anything related to raid 1.

ftp://ftp.dlink.com/Multimedia/dns323/F … es_104.txt


The last raid 1 fix was made through 1.03 for the e-mail notifications

http://www.dlink.com/products/support.a … 0#firmware


As for the drives, I do not use WD drives but Seagates:
500 gigs Seagate ST3500320AS

if that can help you

Are you having a particular problem with Raid 1 other than the drives becoming degraded due to hard drive errors?

jayas wrote:

Hi DLINK

Dlink wrote:

In 1.03 drive failure detection was not working properly. It was fixed in 1.04. Alot of the issues we have seen with the Raid degraded status have been related to some firmware drivers on some WD disks causing critical errors which cause our device to flag it as a failed drive. Still a work in progress and hoping to fixed before 1.05 goes public.

I am not sure if the problem is restricted to WD disks.  At least I have seen it occur on DNS-323 with Seagate drives although I suspected (don't recall my reasons as it was a while ago) that power interruption may have caused it.

However in investigating at why the hot-plugging fails I found issues with scripts.   Is there a channel for issues and suggestions to be relayed to the development team?

Jaya

Just so theres no missunderstanding I am not saying it is strictly related to only WD disks just that we have seen this occur more on WD disks. The closest thing to "official channel" would be forums.dlink.com under the DNS-323 section.

Hi, I have as well the same problem, the Led of the first HDD is down but the disk is fine and the D-link french support told me first to save the data of both HDD (of course !), then format them again and try if it works.

if not, send it back to them, they will exchange it. I think there is an article on the wiki about this issue

hope this helps

PS: I have 2 Seagate HDD

Last edited by bodbod (2008-04-25 02:22:37)

I have been watching this thread with interest.

I too can confirm that there are serious issues with the 1.04 firmware release.

I am using two matched 500GB Seagate  ST3500641AS drives which are not even one year old. I am using RAID 1 mirroring and this appears to be the route of the problem.

After upgrade my DNS 323 “bricked” as it were. The drives continued to report failures, a bit like a disco really, first one then the other then both and so on after successive reboots.

Upon further investigation it appears the two RAIDed disks seem to get stuck in a synchronisation loop. The “sync time” after upgrade was regularly reporting over 7,000 minutes and sometimes reported over 16,000 minutes! The hard disks just keep thrashing and they never end their sync cycle. They then get hotter and hotter and then simply stop responding.

Once in this cycle it is virtually impossible to stop this because at each successive reboot the drives need to sync before you can access them to format them.

After previous upgrades of firmware when using RAID we had been advised to reformat and re-establish the RAID set up. I therefore managed to break the everlasting sync cycle and I have now tried this approach and can report this also failed.

All looked well for the first 13 Hours as I restored my files but it failed on information restore at about 160GB (I am not exactly sure though), when the system refused to accept information and then went dead (even though the lights were showing OK). The only solution was a hard reboot. After this the DNS 323 again seemed to get stuck in its RAID disk syncing cycle again.

Solution – revert back to 1.03.

If ”Dlink” is monitoring I would recommend that you pull this firmware release until you have solved these issues. As you can see people are expending huge amounts of effort and time on this issue and I am sure less capable users are simply loosing important data and literally ending up with an expensive brick; you are in danger of finding loads of your product in dumpsters and people going out and buying another like device from one of your competitors. People do not take data loss kindly.

Also, your firmware, if my experience is anything to go by, is actually causing damage to disks by just simply thrashing them to death trying to synchronise the RAID mirror.

Moderators – in view of all the posts and the issues here could we have this topic as a sticky until D-Link resolve the issue for the benefit of all concerned.

fordem wrote:

Also - people do not take data loss kindly - agreed - but if they lose data because they have not backed up their DNS-323 (or other device), then that is not the responsibility of D-Link or the manufacturer of whichever device the data was stored on.

Thank you for your learned comments....

But surely that is the point of RAID mirroring in that you have an inbuilt back-up of your system which can be recovered. If the fix trashes both hard drives then this is a serious issue and I still stand by my original request. You are advocating a backup of the backup - how many backups is it reasonable for people to make !!

The average user, who we must also think of (and a few have made the same comment here) do not have the capacity to back up again 500 GB of data when they have invested in two 500 GB drives for precisely that purpose. Your definition of "balance" is in my book equal to unnecessary risk and D-Link are risking others data and hardware on the back of the usual IT community limitations of use clause, i.e. "At your own risk"

I am now in the process of restoring my second set of Backed up data - in all a total of just 30 mins short of 24 hours effort without a break, and that is without the time taken to create the original backup copy (which is incremental).  Reasonable effort for a firmware upgrade and "balance" - I do not think so!

You are also inferring that because the lights change the firmware can now be trusted to identify if your disk is failing - it's currently registering good disks as failed so my confidence in it performing this task adequately is low, you may be living with a false sense of security.

I agree that if D-Link continue to distribute this grossly faulty firmware than it should at least come with a health waning but currently we don't have that either!

Lastly, my point was that it seems to be an issue with large capacity drives using RAID1, i.e. over the 250-300GB mark, as stated my data restoration failed, this now seems to have occured at the 245GB mark.... Just an observation and trying to provide more info to be fed back to D-Link themselves.

Last edited by Slingers (2008-04-28 02:27:43)

Slingers wrote:

Upon further investigation it appears the two RAIDed disks seem to get stuck in a synchronisation loop. The “sync time” after upgrade was regularly reporting over 7,000 minutes and sometimes reported over 16,000 minutes! The hard disks just keep thrashing and they never end their sync cycle. They then get hotter and hotter and then simply stop responding.

Slingers, am I correct in that you did not have any fun_plug running or telnet access to the
DNS-323 while you were testing, and the the "sync time" you are reporting is from
the D-Link web GUI.

Perhaps you could repeat the test, with fun_plug loaded and view the contents of
/proc/mdstat  This might give some more insight to what the underlying RAID
software mdadm is doing?

OK - here's a different theory as to what may cause this repeated random degradation - this theory is born out of reading a post in the D-Link forum (where the user actually linked to this thread).  As before this is a theory that I have not been able to test, since I have not personally experienced the problem - at least not with my DNS-323.

Perhaps someone who has, can test it.

The user has mentioned that "after the firmware upgrade, he formats the drives and everything works normally until he powers the unit down - at reboot it may, or may not come up with a degraded indication - BTW - he is using Seagate 7200.11 Barracudas.

First a bit of background - my first 250GB SATA drives were Maxtors, and they worked fine in my desktop, but not in the server I had originally purchased them for - the server never detected them at boot up, and after some searching I discovered that the problem was related to a "delayed spin up" function - the drives did not spin up immediately on power up, but would do so either on a command from the controller or after a random delay - this is done to reduce the startup load on the power supply, by staggering the drive spinup and spreading the current demand over a longer period - in the case of my server, the drives were never ready when the server checked for them, and the solution was a jumper setting that disabled the "delayed spin up" function.

Here's the theory now - is it possible, that, like my server, the DNS-323 is checking for the drive before the drive is operationally ready?  It sees one drive, but not the other and so reports a degraded status?

And now - how to test - if you have been experiencing this problem, can you check your drive documentation to see if the drive offers a "delayed spin up" function and if it does, can you disable the function and see if the problem continues.