DSM-G600, DNS-3xx and NSA-220 Hack Forum

Unfortunately no one can be told what fun_plug is - you have to see it for yourself.

You are not logged in.

#1 2017-08-12 12:42:45

luusac
Member
Registered: 2008-04-29
Posts: 360

raid failure/bitmap/group desciptors corruption

Hi,
After storing a few gb on my dns-323 last night, waiting until transfers had stopped and shutting down the box via the web interface (as far as I remember it did it via the web and not the button - memory us fuzzy it was rather late) I turn on the box this morning to be prompted to set up raid (wizard).  I can't telnet or ssh into the box with ffp anymore.  I have pulled one of the drives, put it in a usb caddy (using laptop) and tried to connect via ubuntu using:

mount -o ro -t ext2 /dev/md0 /tmp/mountpoint

after that got me nowhere I installed mdadm and tried

sudo mdadm --assemble /dev/md0 /dev/sdc2 --run
mdadm: /dev/md0 has been started with 1 drive (out of 2).

sudo mdadm --detail /dev/md0
/dev/md0:
        Version : 0.90
  Creation Time : Fri Aug 22 01:41:48 2008
     Raid Level : raid1
     Array Size : 974133312 (929.01 GiB 997.51 GB)
  Used Dev Size : 974133312 (929.01 GiB 997.51 GB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sat Aug 12 10:07:55 2017
          State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
  Spare Devices : 0

           UUID : 0c391687:af39f816:d362f0bc:fac566aa
         Events : 0.2756281

    Number   Major   Minor   RaidDevice State
       0       8       34        0      active sync   /dev/sdc2
       2       0        0        2      removed

dmesg:
[ 4999.902192] md: bind<sdc2>
[ 4999.904887] md/raid1:md0: active with 1 out of 2 mirrors
[ 4999.904948] md0: detected capacity change from 0 to 997512511488

[ 5112.362546] EXT4-fs (md0): mounting ext2 file system using the ext4 subsystem
[ 5112.382698] EXT4-fs (md0): ext4_check_descriptors: Block bitmap for group 3968 not in group (block 384067452)!
[ 5112.382700] EXT4-fs (md0): group descriptors corrupted!

When I click on the 998GB volume in nautilus I get:
Error mounting /dev/md0 at /media/houthalen/e175b8b4-c01e-434d-b90e-754a8bfcb549: Command-line `mount -t "ext2" -o "uhelper=udisks2,nodev,nosuid" "/dev/md0" "/media/houthalen/e175b8b4-c01e-434d-b90e-754a8bfcb549"' exited with non-zero exit status 32: mount: wrong fs type, bad option, bad superblock on /dev/md0,
       missing codepage or helper program, or other error

I relise the fs type is ext2, so I don't understand why the "mounting ext2 file system using the ext4 subsystem".

I have pulled both drives from the DNS-323 (one at a time and returned them to their original bays so I don't get mixed up with left/right drives) and get the same results.

I have another drive that I can copy everything to, but want to take the least destructive method of getting access to the raid partition.  Can anybody help me further?
Thanks

Offline

 

#2 2017-08-13 16:23:20

Mijzelf
Member / Developer
Registered: 2008-07-05
Posts: 692

Re: raid failure/bitmap/group desciptors corruption

I relise the fs type is ext2, so I don't understand why the "mounting ext2 file system using the ext4 subsystem".

In newer Linux kernels the ext4 driver is also used for mounting ext2 and -3 filesystems. I guess this is done to ease the code maintenance.

It looks like the filesystem is corrupted. Maybe you had a power failure when shutting down the NAS?

You can try to run fsck on the md volume. (Don't try on both disks. If it fails on one, it won't succeed on the other). *Maybe* it will help to install the ext2 kernel module, and try to mount using that fs driver.

(one at a time and returned them to their original bays so I don't get mixed up with left/right drives)

That shouldn't matter. The raid header contains all information needed to assemble the array. The physical location is not important.

Offline

 

#3 2017-08-22 00:23:04

luusac
Member
Registered: 2008-04-29
Posts: 360

Re: raid failure/bitmap/group desciptors corruption

Mijzelf wrote:

Maybe you had a power failure when shutting down the NAS?

You can try to run fsck on the md volume. (Don't try on both disks. If it fails on one, it won't succeed on the other). *Maybe* it will help to install the ext2 kernel module, and try to mount using that fs driver.

Possibly, but I don't think there was a power failure - I was next to the dns323 when it shutdown.

How would I do that - do you mean using modprobe/insmod and a .ko file?  If so I am not sure where I would locate an ext2 module.  How about going back to a live cd with a much earlier version of ubuntu, e.g. Hardy Heron?

(one at a time and returned them to their original bays so I don't get mixed up with left/right drives)

Mijzelf wrote:

That shouldn't matter. The raid header contains all information needed to assemble the array. The physical location is not important.

But don't I need to check which of the drives has the highest raid event count and work on that one to try and achieve recovery of the most data?

Also, I don't really want to work on the drives directly for fear of making things worse.  Wouldn't it be better to image the drive and try and repair the image - that way I can try different things without writing to the drives themselves. Any clues on how to do this with dd ?  Many pages give examples, but with different syntax (often assuming that if you need to repair a raid array, you will be doing so for work not home, and will have more knowledge/experiance), so it is a bit difficult to know where to start.
Thanks for your earlier reply, sorry for the delayed response - I am away from home.
thanks

Offline

 

#4 2017-08-22 15:11:45

Mijzelf
Member / Developer
Registered: 2008-07-05
Posts: 692

Re: raid failure/bitmap/group desciptors corruption

luusac wrote:

Mijzelf wrote:

*Maybe* it will help to install the ext2 kernel module, and try to mount using that fs driver.

How would I do that - do you mean using modprobe/insmod and a .ko file?  If so I am not sure where I would locate an ext2 module.

You don't know where to locate it, modprobe should. Try a simple

Code:

modprobe ext2

(And maybe unload the ext4 module:

Code:

rmmod ext4

luusac wrote:

(one at a time and returned them to their original bays so I don't get mixed up with left/right drives)

Mijzelf wrote:

That shouldn't matter. The raid header contains all information needed to assemble the array. The physical location is not important.

But don't I need to check which of the drives has the highest raid event count and work on that one to try and achieve recovery of the most data?

Raid event count? Your disks are supposed to be each others mirror. The raid header contains a 'last updated' field, but AFAIK that can only be different when the array was degraded before the filesystem corrupted.

luusac wrote:

Also, I don't really want to work on the drives directly for fear of making things worse.  Wouldn't it be better to image the drive and try and repair the image - that way I can try different things without writing to the drives themselves. Any clues on how to do this with dd ?

Yes. Assuming there is no hardware issue with the disk, the command is

Code:

dd if=/dev/sda of=/dev/sdb bs=64M

Here /dev/sda is the old disk (if is inputfile) and /dev/sdb is the new disk (of is output file). bs is blocksize. If you omit that, it will default to 512 bytes. Which means it will read and write blocks of 512 bytes. 64MB is just much faster.

Double check the in- and outputfile. dd will happily overwrite whatever you offer as target, without asking.

Offline

 

#5 2017-08-28 12:27:44

luusac
Member
Registered: 2008-04-29
Posts: 360

Re: raid failure/bitmap/group desciptors corruption

Mijzelf wrote:

Double check the in- and outputfile. dd will happily overwrite whatever you offer as target, without asking.

Thanks.  I will give this a try.  Can I use dd to make an image on a disk with data already on it?  e.g. the source is a 1tb drive from the nas and the destination for the dd image is a 6tb disk with other data on it.  As long as there is more than 1tb free on the destination drive nothing will be overwritten will it?
And wouldn't it be dd if=/dev/sda of=/dev/sdb/name_of_image_file bs=64M

Offline

 

#6 2017-08-28 15:25:20

Mijzelf
Member / Developer
Registered: 2008-07-05
Posts: 692

Re: raid failure/bitmap/group desciptors corruption

luusac wrote:

an I use dd to make an image on a disk with data already on it?  e.g. the source is a 1tb drive from the nas and the destination for the dd image is a 6tb disk with other data on it.  As long as there is more than 1tb free on the destination drive nothing will be overwritten will it?

Yes, can be done, but is an other approach. The command I gave you will write it to the raw disk, so after copying you'll have a disk having 1T partitions, with 5TB unpartitioned space.

What you want it to write it to a filesystem, not to a disk. In that case the disk has to be mounted, and the command is

Code:

dd if=/dev/sda of=/mountpoint/of/6tb/disk/subdir/name_of_image_file bs=64M

This gives a 1TB file, containing a disk image. Not all recovery tools like that, but you can probably simply create a loop device.

Offline

 

#7 2017-08-29 12:32:03

luusac
Member
Registered: 2008-04-29
Posts: 360

Re: raid failure/bitmap/group desciptors corruption

Just about to start the image but found that there is more than one partition on the dns323 drive - I had not remembered this.  The partition map is:
/dev/sde1 517MiB linux swap 0 used
/dev/sde4 502MiB ext2 18 MiB used
/dev/sde2 929GiB ext2 912GiB used
unallocated 1.51GiB
What would be the better strategy - image /dev/sde2 or /dev/sde (i.e. the whole disk or the main data partition)?
Thanks

Offline

 

#8 2017-08-29 16:26:28

Mijzelf
Member / Developer
Registered: 2008-07-05
Posts: 692

Re: raid failure/bitmap/group desciptors corruption

If you trust the partition table you can decide to only backup the data partition. In this case it's almost impossible to have a defective partition table, as the raid array could be assembled without errors.

But storing only the data partition will only save you 0.25% in storage space.

Offline

 

#9 2017-09-01 22:39:41

luusac
Member
Registered: 2008-04-29
Posts: 360

Re: raid failure/bitmap/group desciptors corruption

Hi,
A big thank you to Mijzelf.  I have managed to recover a large portion of what was stored in the raid array on the dns-323.  I will just make a few notes below in case anybody else encounters similar problems in the future.  I would be interested in any explanations of what the e2fsck corruption descriptors in my case (below) actually mean.

As far as I know I shut the nas down correctly and only when it had powered down did I switch off the power.  I was doing a lot of file transfers (several gb's) and deletes before, maybe the corruption happened then.  I wanted to image the drive/partition so that whatever I did when trying the recovery wouldn't be permanent - so if I messed up I could go back to a 'fresh' corrupt copy - especially as I didn't understand a lot of the messages from e2fsck that I was giving answers to.  Here are the commands I used to access and clean the partition to get at my files. 

Downloaded latest version of Ubuntu live cd image and put it on a memory stick using Yumi then boot off it and 'Try Ubuntu' or whatever other distro you like provided it has the relevant tools.

Image the drive (note Mijzelf's caution about correct use of dd earlier in the thread).  Obviously substitute the paths and drives for those matching your setup:

Code:

dd if=/dev/corrupted_drive of=/mountpoint/of/destination_drive/name_of_image_file bs=64M

If you have space, duplicate the image so if anything goes wrong you don't have to wait hours (5.5 hours in my case) while another image is being made (assuming of course the source drive hasn't had a hardware failure since you made the first image).

Code:

cp /mountpoint/of/destination_drive/name_of_image_file /mountpoint/of/destination_drive/name_of_image_file_2

Then make loop device for the corrupt partition within the drive image.  Note: on my raid setup the main partition was number 2.  I am pretty sure that I let the dns-323 partition the drives and make the raid setup originally, so it may well be the second partition for other people too:

Code:

sudo losetup -Pf /mountpoint/of/destination_drive/name_of_image_file_2

Clean the filesystem (partition 2 as loop device loop1p2):

Code:

sudo e2fsck -v /dev/loop1p2

Make the mount point

Code:

sudo mkdir /mountpoint/of/destination_drive/corrupt_image

Mount the partition file filesystem type ext2 from the second partition in the image via loop device

Code:

sudo mount -o ro -t ext2 /dev/loop1p2 /mountpoint/of/destination_drive/corrupt_image

Hopefully you will then be able to access your files.

To unmount:

Code:

sudo umount /mountpoint/of/destination_drive/corrupt_image

To detach the loop device (all partitions):

Code:

sudo losetup -d /dev/loop1

Here is a short version of my fsck output

e2fsck 1.42.13 (17-May-2015)
ext2fs_check_desc: Corrupt group descriptor: bad block for block bitmap
e2fsck: Group descriptors look bad... trying backup blocks...
Adding dirhash hint to filesystem.

/dev/loop1p2 was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 118390794, i_blocks is 633816, should be 633824.  Fix<y>? yes

Running additional passes to resolve blocks claimed by more than one inode...
Pass 1B: Rescanning for multiply-claimed blocks
Multiply-claimed block(s) in inode 118390794: 32
Pass 1C: Scanning directories for inodes with multiply-claimed blocks
Pass 1D: Reconciling multiply-claimed blocks
(There are 1 inodes containing multiply-claimed blocks.)

File /blah.blah (inode #118390794, mod time Fri Aug 11 23:12:31 2017)
  has 1 multiply-claimed block(s), shared with 1 file(s):
    <filesystem metadata>
Clone multiply-claimed blocks<y>? yes
Pass 2: Checking directory structure
Problem in HTREE directory inode 54886408: block #1 has bad max hash
Invalid HTREE directory inode 54886408 (/old users/me/dt).  Clear HTree index<y>? yes
Problem in HTREE directory inode 61718546: block #1 has bad max hash
Problem in HTREE directory inode 61718546: block #2 has bad min hash

<repeated>

Invalid HTREE directory inode 61718546 (/memstick/Fav).  Clear HTree index<y>? yes
Problem in HTREE directory inode 61735533: block #1 has bad max hash
Problem in HTREE directory inode 61735533: block #2 has bad min hash

<repeated HTREE directory & min/max hash messages>

Filesystem contains large files, but lacks LARGE_FILE flag in superblock.
Fix<y>? yes
Pass 3: Checking directory connectivity
/lost+found not found.  Create<y>? yes
Pass 3A: Optimizing directories
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong for group #0 (32189, counted=0).
Fix<y>? yes

<'Free blocks count wrong for group' xyz repeated hundreds/thousands of times>

I didn't use the -y with fsck so that was a lot of keypresses!

Again a thank you to Mijzelf for taking the time to help.....

Offline

 

Board footer

Powered by PunBB
© Copyright 2002–2010 PunBB