Subject: Re: Advice requested
From: Phil Turmel <[email-masked]>
To: o1bigtenor <[email-masked]>
Cc: Linux-RAID <[email-masked]>
Date: Mon, 2 Nov 2015 14:43:32 -0500

Hi Dee,

On 11/02/2015 01:21 PM, o1bigtenor wrote:
> On Mon, Nov 2, 2015 at 9:41 AM, Phil Turmel <[email-masked]> wrote:

>> That means you didn't tell it what device(s) to examine, so it did
>> nothing.  Based on your data below, I expect you need to examine
>> /dev/sdb1, /dev/sdc1, /dev/sde1, and /dev/sdf1.
> 
> root@debianbase:/# mdadm -E /dev/sdb
> /dev/sdb:
>   MBR Magic : aa55
> Partition[0] :   1953521664 sectors at         2048 (type fd)
> root@debianbase:/# mdadm -E /dev/sdc
> /dev/sdc:
>   MBR Magic : aa55
> Partition[0] :   1953525167 sectors at            1 (type ee)
> root@debianbase:/# mdadm -E /dev/sde
> /dev/sde:
>   MBR Magic : aa55
> Partition[0] :   1953521664 sectors at         2048 (type fd)
> root@debianbase:/# mdadm -E /dev/sdf
> /dev/sdf:
>   MBR Magic : aa55
> Partition[0] :   1953521664 sectors at         2048 (type fd)

:-) You left off the partition numbers...  Please try again.

> (I had some other system issues about 5 weeks ago an damaged the
> superblock from sdc.)

Hmmm.  That makes me wonder if the correct superblock is available, but
only if the partition is relocated back to sector 2048.  We may revisit
this.

>> { A look at the section on --examine in "man mdadm" would have helped
>> here.  Please spend some time reviewing "man mdadm" and "man md". }

> I have found that man pages are great resources when one does know or has
> used the command, when you haven't - - - well there are most often NO examples,
> either of what to do or what happens when something is wrong. They are most
> often written by those intimately familiar with the area (never have a
> problem with
> anything) and so I don't bother looking at them - - - they just aren't useful.
> (The whole RTFM thing is one of the biggest reasons that I find *nix very
> frustrating - - - I have spent hours reading and re-reading a man page
> and sometimes
> four or five pages talking about it and its still not clear exactly
> how to write the
> command or even better if the result is not positive there is absolutely no idea
> given as to how to troubleshoot things - - - sorry its a rant but you
> did tell me to
> read it and - - - - .)

You may have to conquer this to succeed.  While man pages certainly
aren't perfect, they are the most complete documentation available on
any linux system.  Most people in your situation struggle with the
'synopsis' section, and how to decipher the brackets and braces and
such.  Google 'reading man pages' and sample the results for a wide
overview.  Save the deep dive on 'Backus-Naur' for last. :-)

>> It seems /dev/sdd1 is missing, so I guess the device names have changed.
>> Please supply the reports for all four array devices with their current
>> names (as shown in the smartctl reports).
> 
> sda is my current 'operating system' disc - - - or where I want my operating
> system to reside. I wanted all my information to be on the raid array.
> 
> sdd is a ssd drive that is presently not used and was used previously as
> an 'operating system' disc.
> 
> Therefore neither of these really has anything to do with the raid array.

Well, your initial report referred to sdb1 and sdd1, so I clearly had to
ask.

>> Looking for this is one of the reasons I asked for these reports --
>> non-raid rated drives producing timeout mismatch failures is a common
>> problem seen on this list.  I also wanted to know if your drives are
>> generally healthy -- they are -- and how that might have impacted your
>> situation.  { I had some of the same Seagate .12 drives -- they didn't
>> start failing until they had nearly 40k hours. }
> 
> When these drives were bought, late January 2012 there was no differentiation
> between drives based upon intended usage. That happened some time in
> about 2013 IIRC.

Drive manufacturers have always pitched their enterprise drives for raid
duty, but only started sabotaging raid support in desktop drives in the
2011 time frame (as shown in the email archives I referenced).

But even before they started cutting ERC support, you had to activate it
manually to be safe in a raid array.  So no, desktop drives have never
been safe to use in MD raid "out of the box".  And still are not.

>> Another reason I asked for this is that drive names can change on power
>> cycles and unplug/replug cycles.  Knowing what device is what (versus
>> drive serial number) is important.  Please verify drive serial number
>> versus device name after any power cycle or replug event and let us know
>> any changes.
> 
> There have been no real changes in quite a long time.
> The only UUID change was between before Aug 28th of this and after when the
> UUID of the array was changed. It was not finding the array upon a reboot (used
> to improve web speeds changed by Firefox's software) that informed me that I
> had an issue.

The device you originally referred to as /dev/sdd is now apparently
something else.  So yes, you are subject to device name changes.

A UUID change only happens when an array is re-created.  If you've lost
the original superblocks, recovery may not be possible.  Losing access
at boot time is often a problem with updates to an initramfs, not to any
actual problem with an array.

>> This ^^^ isn't really an error.
> If it isn't an error this was the only place where there were any
> listed changes to the array and/or its discs.
> 
> Subsequently I was unable to mount the array nor did it mount automatically as
> it had for the previous time (at least 4 months).
>>
>>> Aug 28 10:39:28 debiantestingbase kernel: [    4.125325] random: nonblocking pool is initialized
>>> Aug 28 10:39:28 debiantestingbase kernel: [    4.125530] md: bind<sde1>
>>> Aug 28 10:39:28 debiantestingbase kernel: [    4.142140] md: bind<sdf1>
>>> Aug 28 10:39:28 debiantestingbase kernel: [    4.144984] md: raid10 personality registered for level 10
>>> Aug 28 10:39:28 debiantestingbase kernel: [    4.145397] md/raid10:md0: active with 4 out of 4 devices
>>
>> If it was, you wouldn't get this ^^^ success.
> May have been a success but the UUID was changed and I could find no way
> to access the array (still can't for that matter!).

Which suggests that the new UUID array is not pointing at the data you
need.  If the other two drives still have the old superblocks, it might
still be possible.

>> This section of your syslog is too far back in time -- before the failure.
> This section is AT the time of failure.
> Md0 has not been accessible subsequently!

Then all four drives have new superblocks due to some form of array
creation that has blown away the others.  Since the array won't mount,
you had either incorrect array creation parameters and/or some operation
stomping on the contents.

>>
>> [trim /]
>>
>>> Was following directions from one person on linux-raid and when he stopped
>>> responding turned to someone who is somewhat connected with redhat and
>>> hard drive stuff. There seemed to be a consensus that I needed to use low
>>> level tools in an attempt to recover files, if I could. As there are about 200k
>>> I wasn't looking forward to repairing things ;-(  !
>>
>> A link to the linux-raid archive of the prior discussion might help.
>> And you might still need low level tools -- we are still trying to get
>> your array started.  Then we can look at the next layer on top of that.
>>  Based on your prior mails, /dev/md0 was formatted as ext4, yes?
> 
> Yes.

Based on your comments below, NO.

> o1bigtenor <[email-masked]>
> Sep 3
> Reply
> to linux-raid
> Greetings
> 
> Had updated a system to Debian 8 which also had a Raid 10 array that
> has been in use for about 3 1/2 years. (Setup raid under Debian 6 then
> ran it mostly under Debian 7 mounting the array each time after
> booting using the command
> 
> #mount /dev/dm-o /home/myspace/RAID

This indicates that you had a device mapper layer on top of the raid
array, with the filesystem in that layer.

I trimmed the rest of this mail in favor of the archive link:
http://marc.info/?l=linux-raid&m=144131963804698&w=2

The following are not in the archive because linux-raid wasn't copied:

> o1bigtenor <[email-masked]>
> Sep 4
> Reply
> to Alexander
> On Fri, Sep 4, 2015 at 1:03 AM, Alexander Afonyashin
> <[email-masked]> wrote:
>>
>> Hi,
>>
>> Please paste the output of mdadm -E /dev/sdb1 (and other 3 partitions:
>> /dev/sdc1, /dev/sde1, /dev/sdf1)
>>
>  # mdadm -E /dev/sdb1
> /dev/sdb1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
>            Name : debianbase:0
>   Creation Time : Mon Mar  5 08:26:28 2012
>      Raid Level : raid10
>    Raid Devices : 4
> 
>  Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
>      Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
>   Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=1968 sectors, after=1024 sectors
>           State : clean
>     Device UUID : a80c76db:eaea61af:bcb9cbbb:ac99e467
> 
>     Update Time : Fri Aug 28 10:38:32 2015
>        Checksum : 29a4fa98 - correct
>          Events : 47341
> 
>          Layout : near=2
>      Chunk Size : 512K
> 
>    Device Role : Active device 3
>    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
> 
> # mdadm -E /dev/sdc1
> /dev/sdc1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
>            Name : debianbase:0
>   Creation Time : Mon Mar  5 08:26:28 2012
>      Raid Level : raid10
>    Raid Devices : 4
> 
>  Avail Dev Size : 1953521039 (931.51 GiB 1000.20 GB)
>      Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
>   Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=1960 sectors, after=2447 sectors
>           State : clean
>     Device UUID : 840fc99d:6f607ea8:9a4a9b0a:a965c85c
> 
>     Update Time : Fri Aug 28 10:38:32 2015
>   Bad Block Log : 512 entries available at offset 72 sectors
>        Checksum : 28cfd5ab - correct
>          Events : 47341
> 
>          Layout : near=2
>      Chunk Size : 512K
> 
>    Device Role : Active device 2
>    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
> 
> # mdadm -E /dev/sde1
> /dev/sde1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
>            Name : debianbase:0
>   Creation Time : Mon Mar  5 08:26:28 2012
>      Raid Level : raid10
>    Raid Devices : 4
> 
>  Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
>      Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
>   Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=1968 sectors, after=1024 sectors
>           State : clean
>     Device UUID : cb37e722:d3a98f88:13439de2:bdad9d2b
> 
>     Update Time : Fri Aug 28 10:38:32 2015
>        Checksum : 34ce8208 - correct
>          Events : 47341
> 
>          Layout : near=2
>      Chunk Size : 512K
> 
>    Device Role : Active device 0
>    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
> 
> # mdadm -E /dev/sdf1
> /dev/sdf1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
>            Name : debianbase:0
>   Creation Time : Mon Mar  5 08:26:28 2012
>      Raid Level : raid10
>    Raid Devices : 4
> 
>  Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
>      Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
>   Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=1968 sectors, after=1024 sectors
>           State : clean
>     Device UUID : 9e749fa9:a0efe791:ea09d2e2:72b99f6c
> 
>     Update Time : Fri Aug 28 10:38:32 2015
>        Checksum : 615d736 - correct
>          Events : 47341
> 
>          Layout : near=2
>      Chunk Size : 512K
> 
>    Device Role : Active device 1
>    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

Note the device order of this report is /dev/sde1, /dev/sdf1, /dev/sdc1,
/dev/sdb1.  Subject to change at the whim of the operating system at
boot (not that the actual drives can change order, but the name used by
the OS can change).  These show a creation date of 2012, so I would
trust those reports.

> o1bigtenor <[email-masked]>
> Sep 4
> Reply
> to Alexander
> On Fri, Sep 4, 2015 at 8:16 AM, Alexander Afonyashin
> <[email-masked]> wrote:
>> Hi,
>>
>> According to output your raid is healthy and running. Please, paste
>> the output of cat /proc/mdstat.
>>
> # cat /proc/mdstat
> Personalities : [raid10]
> unused devices: <none>
> 
> 
> If it is running - - - then how do I access it?
> 
> Previously I would go into my file manager program (PCManFM) on on the
> left side there would be
> an icon for the array. I needed to click on it and then a dialog box
> would appear asking for the root
> password the completion of which would give access to the array. That
> no longer happens, instead
> I get an error dialog box that says :
> 
> Only system administrators have the permission to do this.
> 
> Would you please tell me how I would set up access for myself as user
> to get access to the array?
> 
> Thanking you for your help.
> 
> Dee
> 
> o1bigtenor <[email-masked]>
> Sep 4
> Reply
> to Alexander
> On Fri, Sep 4, 2015 at 8:16 AM, Alexander Afonyashin
> <[email-masked]> wrote:
>> Hi,
>>
>> According to output your raid is healthy and running. Please, paste
>> the output of cat /proc/mdstat.
>>
> 
> Just remembered a few things after looking through  my open terminals
> (I keep terminals open
> when I am working on long term problems until the issue has been
> resolved with 100 000 line
> storage level).
> 
> # /etc/fstab: static file system information.
> #
> # Use 'blkid' to print the universally unique identifier for a
> # device; this may be used with UUID= as a more robust way to name devices
> # that works even if disks are added and removed. See fstab(5).
> #
> # <file system> <mount point>   <type>  <options>       <dump>  <pass>
> # / was on /dev/sda3 during installation
> UUID=a3fbea19-5b7d-47af-b06c-08c49171211b /               ext4
> errors=remount-ro 0       1
> # /boot/efi was on /dev/sda1 during installation
> UUID=8642-E734  /boot/efi       vfat    umask=0077      0       1
> # /home was on /dev/sda5 during installation
> UUID=b36dbfc6-6897-4311-aa87-9a958ccad072 /home           ext4
> defaults        0       2
> # swap was on /dev/sda4 during installation
> UUID=354a6e33-5743-4c97-bfb9-fd6f9d4521e6 none            swap    sw
>            0       0
> # swap was on /dev/sdd10 during installation
> UUID=04daef55-ff1b-4fe5-a897-ebe397a145b6 none            swap    sw
>            0       0
> /dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
> # raid10 array
> PTUUID=1eb47793-0cdf-4e16-ae84-33ec825ac448 /media/darald/Raid10
>    gpt    defaults         0       2
> 
> § blkid grep md0
> 
> o1bigtenor <[email-masked]>
> Sep 4
> Reply
> to Alexander
> 
> Sorry just in a thunder and lightening (electrical) storm so things
> were not responding correctly!!.
> (keyboard shifted to a different layout)
> 
> # blkid | grep md0
> 
> gives no response.
> 
> the fstab information says that the array is formatted as gpt where
> the discs were formatted as ext4.
> 
> Sorry this is a whole bunch of weirdness. Perhaps it was a storm on
> the 28th of August too.
> Checked my journal it was storming on the 27th. Maybe that produced an
> effect, sorry we are
> beyond anything I know much about.
> 
> Thanking you for your help!!
> 
> Dee
> 
> Alexander Afonyashin <[email-masked]>
> Sep 7
> Reply
> to me
> Hi,
> 
> Try to assemble it almost automatically by running:
> 
> mdadm --assemble --scan
> 
> then add --force if asked.
> 
> If above not helps, try to construct it manually:
> 
> mdadm -A /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sde1 /dev/sdf1
> 
> Regards,
> Alexander
> 
> 
> o1bigtenor <[email-masked]>
> Sep 7
> Reply
> to Alexander
> On Mon, Sep 7, 2015 at 1:40 AM, Alexander Afonyashin
> <[email-masked]> wrote:
>> Hi,
>>
>> Try to assemble it almost automatically by running:
>>
>> mdadm --assemble --scan
>>
>> then add --force if asked.
>>
>> If above not helps, try to construct it manually:
>>
>> mdadm -A /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sde1 /dev/sdf1
>>
> 
> I do not have a complete backup of all the files on the array.
> 
> Will any of this compromise the data on the array?
> 
> (Tried to backup sdc and sdf but the md5sum on sdf indicates that the two items
> are not the same.)
> 
> Regards
> 
> Darald
> 
> Alexander Afonyashin <[email-masked]>
> Sep 7
> Reply
> to me
> Hi,
> 
> By running -A == --assemble option you asked mdadm only to start raid
> not create it writing new metadata to disks.
> 
> Regards,
> Alexander
> 
> 
> o1bigtenor <[email-masked]>
> Sep 7
> Reply
> to Alexander
> On Mon, Sep 7, 2015 at 6:04 AM, Alexander Afonyashin
> <[email-masked]> wrote:
>> Hi,
>>
>> By running -A == --assemble option you asked mdadm only to start raid
>> not create it writing new metadata to disks.
>>
> 
> Thank you - - - just a little paranoid about losing the array!
> 
> # mdadm --assemble --scan
> mdadm: /dev/md/0 has been started with 4 drives.
> 
> 
> # blkid | grep md0
> /dev/md0: PTUUID="1eb47793-0cdf-4e16-ae84-33ec825ac448" PTTYPE="gpt"
> 
> 
> Discs were originally set up as ext4.
> 
> How would I now access the array?
> 
> 
> 
> Thanks for the help!!!
> Regards
> 
> Dee
> 
> Alexander Afonyashin <[email-masked]>
> Sep 8
> Reply
> to me
> Hi,
> 
> mount -t ext4 /dev/md0 /mnt - like this
> 
> But check it with fsck /dev/md0 first.
> 
> Regards,
> Alexander
> 
> 
> o1bigtenor <[email-masked]>
> Sep 8
> Reply
> to Alexander
> On Tue, Sep 8, 2015 at 2:39 AM, Alexander Afonyashin
> <[email-masked]> wrote:
>> Hi,
>>
>> mount -t ext4 /dev/md0 /mnt - like this
>>
>> But check it with fsck /dev/md0 first.
> 
> Looks like there are further issues - -
> 
> # cat /etc/fstab
> using fsck suggests that there is/are bad superblocks
> 
> # fsck /dev/md0
> fsck from util-linux 2.26.2
> e2fsck 1.42.13 (17-May-2015)
> ext2fs_open2: Bad magic number in super-block
> fsck.ext2: Superblock invalid, trying backup blocks...
> fsck.ext2: Bad magic number in super-block while trying to open /dev/md0
> 
> The superblock could not be read or does not describe a valid ext2/ext3/ext4
> filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
> filesystem (and not swap or ufs or something else), then the superblock
> is corrupt, and you might try running e2fsck with an alternate superblock:
>     e2fsck -b 8193 <device>
>  or
>     e2fsck -b 32768 <device>

All of these attempts to fsck and mount /dev/md0 fail to account for the
device mapper layer you must have been using.  From the fstab entry,
you (or debian 7's installer) partitioned the array for you, and you
were using your desktop environment's dynamic mount tools to actually
access the inner partitions.

> Is there a way to make a verifiable backup of the array before doing
> something to the array that might totally corrupt it?
> 
> Please - - there are files on the array that are important. I didn't
> realize that an array could corrupt and then cause these issues.
> Previous issues were only those of a defective hard drive which was a
> very small problem compared to this.
> 
> TIA
> 
> Dee
> 
> 
> another group of emails
> 
> o1bigtenor <[email-masked]>
> Aug 29
> Reply
> to TCLUG
> Greetings
> 
> I don't know what's going on but this is weird!
> 
> System has been running Debian 8 with things going good (raid mounts
> so when saving a file after a reboot all I need to do is click on the
> listed array give the root password and bob's your uncle).
> 
> (Its a uefi setup)
> Well this morning - - -
> fdisk -l shows the array
> its not mounted
> 
> the command I used to use for mounting in Debian 7 no longer works
> 
> so - - - - any suggestions?
> 
> TIA
> 
> Dee
> 
> Marc Skinner <[email-masked]>
> Aug 29
> Reply
> to TCLUG
> Is this a hardware raid or software raid?  More details?  Is the raid
> array internal or externally connected?
> 
> 
> Thanks.
> _______________________________________________
> TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
> [email-masked]
> http://mailman.mn-linux.org/mailman/listinfo/tclug-list
> 
> 
> _______________________________________________
> TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
> [email-masked]
> http://mailman.mn-linux.org/mailman/listinfo/tclug-list
> 
> o1bigtenor <[email-masked]>
> Aug 29
> Reply
> to TCLUG
> On Sat, Aug 29, 2015 at 11:04 AM, Marc Skinner <[email-masked]> wrote:
> Is this a hardware raid or software raid?  More details?  Is the raid
> array internal or externally connected?
> 
> software raid
> internal
> 
> # cat /proc/mdstat
> Personalities : [raid10]
> md0 : active (auto-read-only) raid10 sde1[0] sdf1[1] sdc1[4] sdb1[3]
>       1953518592 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
> 
> # mdadm --examine --scan
> ARRAY /dev/md/0  metadata=1.2 UUID=79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
> name=debianbase:0
> 
> from blkid
> 
> /dev/md0: PTUUID="1eb47793-0cdf-4e16-ae84-33ec825ac448" PTTYPE="gpt"
> 
> 
> from fdisk -l
> 
> Disk /dev/md0: 1.8 TiB, 2000403038208 bytes, 3907037184 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
> Disklabel type: gpt
> Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448
> 
> 
> Marc Skinner
> So what is the problem ... it won't mount? Did you do an fsck on it? Assuming...
> Aug 29
> 
> o1bigtenor
> # fsck /dev/md0 fsck from util-linux 2.26.2 e2fsck 1.42.13 (17-May-2015) ext2...
> Aug 30

Again here, fsck'ing the array doesn't fsck the inner partition, which
we haven't identified yet.

> gerry <[email-masked]>
> Aug 30
> Reply
> to me
> A few thoughts:
> 
> Your prior email shows that the md was set to read-only, right?
> md0 : active (auto-read-only) raid10 sde1[0] sdf1[1] sdc1[4] sdb1[3]
> 
> Look in your logs for reasons why it got set to auto-read-only.
> This isn't your issue but it's something to read:
> http://unix.stackexchange.com/questions/101072/new-md-array-is-auto-read-only-and-has-resync-pending
> 
> That email also showed some output from
>     fdisk -l /dev/md0.
> Is there any chance you partitioned /dev/md0?
> Because the output says there is a gpt partition table on the /dev/md0 device.
> 
> Quoting:
> Disk /dev/md0: 1.8 TiB, 2000403038208 bytes, 3907037184 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
> Disklabel type: gpt
> Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448
> 
> Was there anything after that?
> Run
>     parted -l
> 
> 
> You made a statement about booting that was confusing to me.
> Here's the section:
> 
>         System has been running Debian 8 with things going good (raid
>         mounts so
>         when saving a file after a reboot all I need to do is click on the
>         listed array give the root password and bob's your uncle).
> [snip]
> 
>         the command I used to use for mounting in Debian 7 no longer works
> 
> What does that mean "click on the listed array give the root password" ?
> Where were you doing such a thing?  Did you by any chance make
> /dev/md0 into an encrypted device?
> 
> What is the command you "used to use for mounting in Debian 7" ?
> 
> Gerry

I was thinking about encryption too, but we'll see.

> --
> [email-masked]
> 
> 
> o1bigtenor <[email-masked]>
> Aug 30
> Reply
> to gerry
> On Sun, Aug 30, 2015 at 12:34 PM, gerry <[email-masked]> wrote:
> A few thoughts:
> 
> Your prior email shows that the md was set to read-only, right?
> md0 : active (auto-read-only) raid10 sde1[0] sdf1[1] sdc1[4] sdb1[3]
> 
> Look in your logs for reasons why it got set to auto-read-only.
> This isn't your issue but it's something to read:
> http://unix.stackexchange.com/questions/101072/new-md-array-is-auto-read-only-and-has-resync-pending
> 
> Any ideas which log to look into?
> 
> 
> That email also showed some output from
>     fdisk -l /dev/md0.
> Is there any chance you partitioned /dev/md0?
> 
> (Its over 3 years ago.) All four discs were partitioned (sdb1, sdc1 etc).
> 
> Because the output says there is a gpt partition table on the /dev/md0 device.
> 
> Quoting:
> Disk /dev/md0: 1.8 TiB, 2000403038208 bytes, 3907037184 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
> Disklabel type: gpt
> Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448
> 
> Was there anything after that?
> Run
>     parted -l
> 
> Model: ATA ST31000524AS (scsi)
> Disk /dev/sdf: 1000GB
> Sector size (logical/physical): 512B/512B
> Partition Table: msdos
> Disk Flags:
> 
> Number  Start   End     Size    Type     File system  Flags
>  1      1049kB  1000GB  1000GB  primary               raid
> 
> 
> Model: Linux Software RAID Array (md)
> Disk /dev/md0: 2000GB
> Sector size (logical/physical): 512B/4096B
> Partition Table: gpt
> Disk Flags:
> 
> Number  Start  End  Size  File system  Name  Flags

This ^^^^ is bad.  The array is partitioned (expected from the fstab),
but there's no partition listed by parted.  We may have to go spelunking
with hexdumps. :-(

> That's the information for the last disc (in the array) and the array using
> parted -l
> 
> 
> 
> You made a statement about booting that was confusing to me.
> Here's the section:
> 
>         System has been running Debian 8 with things going good (raid
>         mounts so
>         when saving a file after a reboot all I need to do is click on the
>         listed array give the root password and bob's your uncle).
> [snip]
> 
>         the command I used to use for mounting in Debian 7 no longer works
> 
> What does that mean "click on the listed array give the root password" ?
> 
> Am using LXDE and the file manager is PCManFM.
> 
> The file manager would bring up a representation where the array was listed as a
> device. After a boot clicking on the button a dialog box would open which asked
> for the password for root. Upon entry access was open to the array.
> 
> Where were you doing such a thing?  Did you by any chance make
> /dev/md0 into an encrypted device?
> 
> As far as I know I didn't.
> 
> What is the command you "used to use for mounting in Debian 7" ?
> 
> mount /dev/dm-0 /home/myspace/RAID
> 
> My guess is that I managed to find some kind of a bug that descends
> with a combined use
> of suspend (upon night close down on the system), mdadm, lxde and pcmanfm.
> 
> Thanks for the ideas.
> 
> Dee
> 
> gerry <[email-masked]>
> Aug 30
> Reply
> to me
> Any ideas which log to look into?
> 
> Look in all of them.  Everything in /var/log/
> Use commands like
> 
> cd /var/log
> grep read * | grep only
> 
> and for the compressed files
> zgrep read *.gz | grep only
> 
> Then for those files that look interesting, use less or zless to view them.
> (The "q" key will exit less or zless)
> 
> You'll need to be root do all that.
> 
> That's the information for the last disc (in the array) and the array using
> parted -l
> 
> <sigh>
> I can't tell from that whether you sent me the entire output of parted
> -l or you were picking and choosing.
> 
> For all these commands, run them and send the entire output. Never
> skimp on information when debugging.
> 
> mount /dev/dm-0 /home/myspace/RAID
> Hmm. In debian 7 you mounted /dev/dm-0 but that's just a devmapper
> device and they are dynamic.  More about devmapper devices can be
> gotten like this:
> 
> ls -l /dev/mapper
> dmsetup ls
> dmestup table
> 
> also run (as root, again)
> lsblk
> 
> 
> Hopefully that will give more information.
> 
> 
> o1bigtenor <[email-masked]>
> Aug 30
> Reply
> to gerry
> On Sun, Aug 30, 2015 at 9:57 PM, gerry <[email-masked]> wrote:
> Any ideas which log to look into?
> 
> Look in all of them.  Everything in /var/log/
> Use commands like
> 
> cd /var/log
> grep read * | grep only
> 
> and for the compressed files
> zgrep read *.gz | grep only
> 
> Then for those files that look interesting, use less or zless to view them.
> (The "q" key will exit less or zless)
> 
> You'll need to be root do all that.
> 
> That's the information for the last disc (in the array) and the array using
> parted -l
> 
> <sigh>
> I can't tell from that whether you sent me the entire output of parted
> -l or you were picking and choosing.
> 
> I was trying to give you relevant information.
> 
> # parted -l
> Model: ATA ST1000DM003-1ER1 (scsi)
> Disk /dev/sda: 1000GB
> Sector size (logical/physical): 512B/4096B
> Partition Table: gpt
> Disk Flags:
> 
> Number  Start   End     Size    File system     Name   Flags
>  1      1049kB  269MB   268MB   fat32                  boot, esp
>  2      269MB   538MB   268MB                   /boot  msftdata
>  3      538MB   26.8GB  26.2GB  ext4            /      msftdata
>  4      26.8GB  93.9GB  67.1GB  linux-swap(v1)  swap
>  5      93.9GB  494GB   400GB   ext4
> 
> 
> Model: ATA ST31000524AS (scsi)
> Disk /dev/sdb: 1000GB
> Sector size (logical/physical): 512B/512B
> Partition Table: msdos
> Disk Flags:
> 
> Number  Start   End     Size    Type     File system  Flags
>  1      1049kB  1000GB  1000GB  primary               raid
> 
> 
> Model: ATA ST1000DM003-1ER1 (scsi)
> Disk /dev/sdc: 1000GB
> Sector size (logical/physical): 512B/4096B
> Partition Table: gpt
> Disk Flags:
> 
> Number  Start   End     Size    File system  Name              Flags
>  1      1049kB  1000GB  1000GB               Linux filesystem
> 
> 
> Model: ATA Corsair Force 3 (scsi)
> Disk /dev/sdd: 240GB
> Sector size (logical/physical): 512B/512B
> Partition Table: msdos
> Disk Flags:
> 
> Number  Start   End     Size    Type      File system     Flags
>  1      1049kB  256MB   255MB   primary   ext2            boot
>  4      257MB   123GB   123GB   extended
>  5      258MB   21.2GB  21.0GB  logical   ext4
>  6      21.2GB  42.2GB  21.0GB  logical   ext4
>  7      42.2GB  73.7GB  31.5GB  logical   ext4
>  8      73.7GB  77.9GB  4194MB  logical   ext4
>  9      77.9GB  98.8GB  21.0GB  logical   ext4
> 10      98.8GB  123GB   24.5GB  logical   linux-swap(v1)
> 
> 
> Model: ATA ST31000524AS (scsi)
> Disk /dev/sde: 1000GB
> Sector size (logical/physical): 512B/512B
> Partition Table: msdos
> Disk Flags:
> 
> Number  Start   End     Size    Type     File system  Flags
>  1      1049kB  1000GB  1000GB  primary               raid
> 
> 
> Model: ATA ST31000524AS (scsi)
> Disk /dev/sdf: 1000GB
> Sector size (logical/physical): 512B/512B
> Partition Table: msdos
> Disk Flags:
> 
> Number  Start   End     Size    Type     File system  Flags
>  1      1049kB  1000GB  1000GB  primary               raid
> 
> 
> Model: Linux Software RAID Array (md)
> Disk /dev/md0: 2000GB
> Sector size (logical/physical): 512B/4096B
> Partition Table: gpt
> Disk Flags:
> 
> Number  Start  End  Size  File system  Name  Flags
> 
> 
> 
> For all these commands, run them and send the entire output. Never
> skimp on information when debugging.
> 
> mount /dev/dm-0 /home/myspace/RAID
> Hmm. In debian 7 you mounted /dev/dm-0 but that's just a devmapper
> device and they are dynamic.  More about devmapper devices can be
> gotten like this:
> 
> ls -l /dev/mapper
> 
> # ls -l /dev/mapper
> total 0
> crw------- 1 root root 10, 236 Aug 29 04:49 control
> 
> 
> dmsetup ls
> 
> # dmsetup ls
> No devices found
> 
> 
> dmestup table
> 
>  # dmestup table
> bash: dmestup: command not found
> 
> 
> also run (as root, again)
> lsblk
> 
>  # lsblk
> NAME    MAJ:MIN RM   SIZE RO TYPE   MOUNTPOINT
> sda       8:0    0 931.5G  0 disk
> ├─sda1    8:1    0   256M  0 part   /boot/efi
> ├─sda2    8:2    0   256M  0 part
> ├─sda3    8:3    0  24.4G  0 part   /
> ├─sda4    8:4    0  62.5G  0 part   [SWAP]
> └─sda5    8:5    0 372.5G  0 part   /home
> sdb       8:16   0 931.5G  0 disk
> └─sdb1    8:17   0 931.5G  0 part
>   └─md0   9:0    0   1.8T  0 raid10
> sdc       8:32   0 931.5G  0 disk
> └─sdc1    8:33   0 931.5G  0 part
>   └─md0   9:0    0   1.8T  0 raid10
> sdd       8:48   0 223.6G  0 disk
> ├─sdd1    8:49   0   243M  0 part
> ├─sdd4    8:52   0     1K  0 part
> ├─sdd5    8:53   0  19.5G  0 part
> ├─sdd6    8:54   0  19.5G  0 part
> ├─sdd7    8:55   0  29.3G  0 part
> ├─sdd8    8:56   0   3.9G  0 part
> ├─sdd9    8:57   0  19.5G  0 part
> └─sdd10   8:58   0  22.9G  0 part   [SWAP]
> sde       8:64   0 931.5G  0 disk
> └─sde1    8:65   0 931.5G  0 part
>   └─md0   9:0    0   1.8T  0 raid10
> sdf       8:80   0 931.5G  0 disk
> └─sdf1    8:81   0 931.5G  0 part
>   └─md0   9:0    0   1.8T  0 raid10
> sr0      11:0    1  1024M  0 rom
> 
> 
> There you go.
> 
> Thanks for the help!
> 
> gerry <[email-masked]>
> Aug 30
> Reply
> to me
> Cool.  And did you find anything in the logs?
> 
> The dmsetup output is interesting. I would have expected some dm devices.
> Without dm devices, there are no /dev/dm things to mount so that's why
> your old Debian7 command doesn't work.
> 
> Did you expect there to be a partition table on /dev/md0?
> Or did you expect to be able to just mount /dev/md0?
> (was there a filesystem on /dev/md0 or was it a  partitioned logical
> disk mirror?)
> 
> You might be able to figure that out from the logs too.
> 
> I think the thing went to read-only to protect itself.
> 
> If I were there, I'd run these commands and see what they say.
> parted /dev/sdb1 list
> parted /dev/sdc1 list
> parted /dev/sde1 list
> parted /dev/sdf1 list
> 
> Before trying to "fix" the array, you will want to see if you can
> figure out if you still have one good setup.
> 
> That mirror setup likely mirrors sdb1 with sdc1 and sde1 with sdf1.
> And the stripes were acrosss those two mirrors.
> To see if an individual disk got corrupted, try this:
> (This is non-destructive.)
> for a in /dev/sd{b,c,e,f}1
> do
>   echo -n $a "  "
>   dd skip=10 if=$a bs=1M count=10 status=none| md5sum
> done
> 
> 
> I made a dummy setup with loop devices and I see this:
> for a in /dev/loop*p1
> do
>   echo -n $a "  ";dd skip=10 if=$a bs=1M count=10 status=none| md5sum
> done
> /dev/loop1p1   58b27e44c48997bc126eec1eec12ab8b  -
> /dev/loop2p1   58b27e44c48997bc126eec1eec12ab8b  -
> /dev/loop3p1   5aa8e9cfbcc3f6612f839b9a96da41df  -
> /dev/loop4p1   5aa8e9cfbcc3f6612f839b9a96da41df  -
> 
> See how 1 and 2 are identical? And 3 and 4 are identical?
> That's the mirroring.
> Let's see if yours is similar.
> ....
> 
> If you don't see errors in the log, then it's fairly likely that
> something actually wrote a partition table onto /dev/md0. Otherwise
> it's wouldn't see
> Disklabel type: gpt
> Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448
> 
> In which case you're going to have to do some fancy dancing to recover stuff.
> I suspect the stuff is still there though I don't have a way yet to
> prove it.  (I tried writing a partition table onto a disk with a
> filesystem on it and I was able to recover it).
> 
> Do you happen to have a spare 2T drive around? Can you borrow one?
> If it were me and it really was business data, I'd start out by making
> copies of the existing disks using dd.
> 
> Would you know what kind of filesystem was on there?
> You might be able to  find that in some logs, but I certainly can't in
> my ubuntu system.
> That is likely to be a critical piece of information.
> 
> Keep at it.
> Go slow and careful.
> Research before doing.
> 
> 
> o1bigtenor <[email-masked]>
> Aug 31
> Reply
> to gerry
> Greetings
> 
> On Sun, Aug 30, 2015 at 11:42 PM, gerry <[email-masked]> wrote:
> Cool.  And did you find anything in the logs?
> 
> Aug 30 19:51:40 debiantestingbase kernel: [100718.675021] FAT-fs
> (md0): utf8 is not a recommended IO charset for FAT filesystems,
> filesystem will be case sensitive!
> Aug 30 21:35:33 debiantestingbase kernel: [106954.860493] EXT4-fs
> (md0): VFS: Can't find ext4 filesystem
> Aug 30 21:37:05 debiantestingbase kernel: [107046.607187] EXT4-fs
> (md0): VFS: Can't find ext4 filesystem
> Aug 30 22:20:33 debiantestingbase kernel: [109656.411356] PM: Syncing
> filesystems ... done.
> 
> (included another line that likely had nothing to do with md0) above
> from kern.log
> 
> 
> Aug 28 10:39:56 debiantestingbase udisksd[1777]: Cleaning up mount
> point /media/darald/49552036-b46f-4956-ade9-3541a3dd7f0a (device 253:0
> no longer exist)
> 
> from syslog.3
> That UUID was the 'name' of the array
> 
> 
> Aug 28 10:39:28 debiantestingbase kernel: [    3.025162] scsi 3:0:0:0:
> Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
> Aug 28 10:39:28 debiantestingbase kernel: [    3.025768] sd 3:0:0:0:
> [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
> Aug 28 10:39:28 debiantestingbase kernel: [    3.026296] sd 3:0:0:0:
> [sdb] Write Protect is off
> Aug 28 10:39:28 debiantestingbase kernel: [    3.026304] sd 3:0:0:0:
> [sdb] Mode Sense: 00 3a 00 00
> Aug 28 10:39:28 debiantestingbase kernel: [    3.026484] sd 3:0:0:0:
> [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO
> or FUA
> Aug 28 10:39:28 debiantestingbase kernel: [    3.028273]  sdb: sdb1
> Aug 28 10:39:28 debiantestingbase kernel: [    3.028698] sd 3:0:0:0:
> [sdb] Attached SCSI disk
> Aug 28 10:39:28 debiantestingbase kernel: [    3.086532] Switched to
> clocksource tsc
> Aug 28 10:39:28 debiantestingbase kernel: [    3.168919] md: bind<sdb1>
> Aug 28 10:39:28 debiantestingbase kernel: [    3.342285] ata5: SATA
> link up 3.0 Gbps (SStatus 123 SControl 300)
> Aug 28 10:39:28 debiantestingbase kernel: [    3.343153] ata5.00:
> ATA-9: ST1000DM003-1ER162, CC45, max UDMA/133
> Aug 28 10:39:28 debiantestingbase kernel: [    3.343158] ata5.00:
> 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
> Aug 28 10:39:28 debiantestingbase kernel: [    3.344067] ata5.00:
> configured for UDMA/133
> Aug 28 10:39:28 debiantestingbase kernel: [    3.344255] scsi 4:0:0:0:
> Direct-Access     ATA      ST1000DM003-1ER1 CC45 PQ: 0 ANSI: 5
> Aug 28 10:39:28 debiantestingbase kernel: [    3.344627] sd 4:0:0:0:
> [sdc] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
> Aug 28 10:39:28 debiantestingbase kernel: [    3.344631] sd 4:0:0:0:
> [sdc] 4096-byte physical blocks
> Aug 28 10:39:28 debiantestingbase kernel: [    3.344823] sd 4:0:0:0:
> [sdc] Write Protect is off
> Aug 28 10:39:28 debiantestingbase kernel: [    3.344831] sd 4:0:0:0:
> [sdc] Mode Sense: 00 3a 00 00
> Aug 28 10:39:28 debiantestingbase kernel: [    3.344946] sd 4:0:0:0:
> [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO
> or FUA
> Aug 28 10:39:28 debiantestingbase kernel: [    3.411364]  sdc: sdc1
> Aug 28 10:39:28 debiantestingbase kernel: [    3.412317] sd 4:0:0:0:
> [sdc] Attached SCSI disk
> Aug 28 10:39:28 debiantestingbase kernel: [    3.501080] md: bind<sdc1>
> Aug 28 10:39:28 debiantestingbase kernel: [    3.662509] ata6: SATA
> link up 3.0 Gbps (SStatus 123 SControl 300)
> Aug 28 10:39:28 debiantestingbase kernel: [    3.674767] ata6.00:
> ATA-8: Corsair Force 3 SSD, 1.3.3, max UDMA/133
> Aug 28 10:39:28 debiantestingbase kernel: [    3.674772] ata6.00:
> 468862128 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
> Aug 28 10:39:28 debiantestingbase kernel: [    3.684647] ata6.00:
> configured for UDMA/133
> Aug 28 10:39:28 debiantestingbase kernel: [    3.684933] scsi 5:0:0:0:
> Direct-Access     ATA      Corsair Force 3  3    PQ: 0 ANSI: 5
> Aug 28 10:39:28 debiantestingbase kernel: [    3.685504] sd 5:0:0:0:
> [sdd] 468862128 512-byte logical blocks: (240 GB/223 GiB)
> Aug 28 10:39:28 debiantestingbase kernel: [    3.685975] sd 5:0:0:0:
> [sdd] Write Protect is off
> Aug 28 10:39:28 debiantestingbase kernel: [    3.685983] sd 5:0:0:0:
> [sdd] Mode Sense: 00 3a 00 00
> Aug 28 10:39:28 debiantestingbase kernel: [    3.686186] sd 5:0:0:0:
> [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO
> or FUA
> Aug 28 10:39:28 debiantestingbase kernel: [    3.688051]  sdd: sdd1
> sdd4 < sdd5 sdd6 sdd7 sdd8 sdd9 sdd10 >
> Aug 28 10:39:28 debiantestingbase kernel: [    3.689305] sd 5:0:0:0:
> [sdd] Attached SCSI disk
> Aug 28 10:39:28 debiantestingbase kernel: [    4.002712] ata8: SATA
> link down (SStatus 0 SControl 300)
> Aug 28 10:39:28 debiantestingbase kernel: [    4.003107] scsi 8:0:0:0:
> Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
> Aug 28 10:39:28 debiantestingbase kernel: [    4.003597] sd 8:0:0:0:
> [sde] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
> Aug 28 10:39:28 debiantestingbase kernel: [    4.003843] scsi 9:0:0:0:
> Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
> Aug 28 10:39:28 debiantestingbase kernel: [    4.003975] sd 8:0:0:0:
> [sde] Write Protect is off
> Aug 28 10:39:28 debiantestingbase kernel: [    4.003980] sd 8:0:0:0:
> [sde] Mode Sense: 00 3a 00 00
> Aug 28 10:39:28 debiantestingbase kernel: [    4.004090] sd 8:0:0:0:
> [sde] Write cache: enabled, read cache: enabled, doesn't support DPO
> or FUA
> Aug 28 10:39:28 debiantestingbase kernel: [    4.004478] sd 9:0:0:0:
> [sdf] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
> Aug 28 10:39:28 debiantestingbase kernel: [    4.004645] sd 9:0:0:0:
> [sdf] Write Protect is off
> Aug 28 10:39:28 debiantestingbase kernel: [    4.004650] sd 9:0:0:0:
> [sdf] Mode Sense: 00 3a 00 00
> Aug 28 10:39:28 debiantestingbase kernel: [    4.004737] sd 9:0:0:0:
> [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO
> or FUA
> Aug 28 10:39:28 debiantestingbase kernel: [    4.004778] scsi
> 15:0:0:0: Processor         Marvell  91xx Config      1.01 PQ: 0 ANSI:
> 5
> Aug 28 10:39:28 debiantestingbase kernel: [    4.006375]  sdf: sdf1
> Aug 28 10:39:28 debiantestingbase kernel: [    4.006967] sd 9:0:0:0:
> [sdf] Attached SCSI disk
> Aug 28 10:39:28 debiantestingbase kernel: [    4.008855]  sde: sde1
> Aug 28 10:39:28 debiantestingbase kernel: [    4.009704] sd 8:0:0:0:
> [sde] Attached SCSI disk
> Aug 28 10:39:28 debiantestingbase kernel: [    4.018710] ata16.00:
> exception Emask 0x1 SAct 0x0 SErr 0x0 action 0x0
> Aug 28 10:39:28 debiantestingbase kernel: [    4.018753] ata16.00:
> irq_stat 0x40000001
> Aug 28 10:39:28 debiantestingbase kernel: [    4.018783] ata16.00: cmd
> a0/01:00:00:00:01/00:00:00:00:00/a0 tag 1 dma 16640 in
> Aug 28 10:39:28 debiantestingbase kernel: [    4.018783]
> Inquiry 12 01 00 00 ff 00res 50/00:00:af:6d:70/00:00:74:00:00/e0 Emask
> 0x1 (device error)
> Aug 28 10:39:28 debiantestingbase kernel: [    4.018868] ata16.00:
> status: { DRDY }
> Aug 28 10:39:28 debiantestingbase kernel: [    4.125325] random:
> nonblocking pool is initialized
> Aug 28 10:39:28 debiantestingbase kernel: [    4.125530] md: bind<sde1>
> Aug 28 10:39:28 debiantestingbase kernel: [    4.142140] md: bind<sdf1>
> Aug 28 10:39:28 debiantestingbase kernel: [    4.144984] md: raid10
> personality registered for level 10
> Aug 28 10:39:28 debiantestingbase kernel: [    4.145397]
> md/raid10:md0: active with 4 out of 4 devices
> Aug 28 10:39:28 debiantestingbase kernel: [    4.145440] md0: detected
> capacity change from 0 to 2000403038208
> Aug 28 10:39:28 debiantestingbase kernel: [    4.208978]  md0:
> Aug 28 10:39:28 debiantestingbase kernel: [    4.479305]
> device-mapper: uevent: version 1.0.3
> Aug 28 10:39:28 debiantestingbase kernel: [    4.479536]
> device-mapper: ioctl: 4.30.0-ioctl (2014-12-22) initialised:
> [email-masked]
> 
> 
> from syslog3
> 
> 
> 
> The dmsetup output is interesting. I would have expected some dm devices.
> Without dm devices, there are no /dev/dm things to mount so that's why
> your old Debian7 command doesn't work.
> 
> Did you expect there to be a partition table on /dev/md0?
> 
> I know that there were partitions on each of the discs (1 each) but I
> don't remember any partitions on md0.
> 
> Or did you expect to be able to just mount /dev/md0?
> 
> That was how it had been working after installing Debian8.
> 
> (was there a filesystem on /dev/md0 or was it a  partitioned logical
> disk mirror?)
> 
> Not sure. Sorry don't know how to answer that one.
> 
> You might be able to figure that out from the logs too.
> 
> I think the thing went to read-only to protect itself.
> 
> If I were there, I'd run these commands and see what they say.
> parted /dev/sdb1 list
> 
> 
>  # parted /dev/sdb1 list
> Usage: parted [OPTION]... [DEVICE [COMMAND [PARAMETERS]...]...]
> Apply COMMANDs with PARAMETERS to DEVICE.  If no COMMAND(s) are given, run in
> interactive mode.
> 
> 
> I'm quite sure that that information isn't really useful. (Won't run
> command for the other 3 because would likely get the same result.)
> 
> parted /dev/sdc1 list
> parted /dev/sde1 list
> parted /dev/sdf1 list
> 
> Before trying to "fix" the array, you will want to see if you can
> figure out if you still have one good setup.
> 
> That mirror setup likely mirrors sdb1 with sdc1 and sde1 with sdf1.
> And the stripes were acrosss those two mirrors.
> To see if an individual disk got corrupted, try this:
> (This is non-destructive.)
> for a in /dev/sd{b,c,e,f}1
> do
>   echo -n $a "  "
>   dd skip=10 if=$a bs=1M count=10 status=none| md5sum
> done
> 
> 
>  # for a in /dev/sd{b,c,e,f}1
>> do
>>   echo -n $a "  "
>>   dd skip=10 if=$a bs=1M count=10 status=none| md5sum
>> done
> /dev/sdb1   f1c9645dbc14efddc7d8a322685f26eb  -
> /dev/sdc1   f1c9645dbc14efddc7d8a322685f26eb  -
> /dev/sde1   979c1a460f2130d7fd0e9210e2d15e90  -
> /dev/sdf1   979c1a460f2130d7fd0e9210e2d15e90  -
> 
> 
> 
> I made a dummy setup with loop devices and I see this:
> for a in /dev/loop*p1
> do
>   echo -n $a "  ";dd skip=10 if=$a bs=1M count=10 status=none| md5sum
> done
> /dev/loop1p1   58b27e44c48997bc126eec1eec12ab8b  -
> /dev/loop2p1   58b27e44c48997bc126eec1eec12ab8b  -
> /dev/loop3p1   5aa8e9cfbcc3f6612f839b9a96da41df  -
> /dev/loop4p1   5aa8e9cfbcc3f6612f839b9a96da41df  -
> 
> See how 1 and 2 are identical? And 3 and 4 are identical?
> That's the mirroring.
> Let's see if yours is similar.
> 
> Looks very similar to me.
> ....
> 
> If you don't see errors in the log, then it's fairly likely that
> something actually wrote a partition table onto /dev/md0. Otherwise
> it's wouldn't see
> Disklabel type: gpt
> Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448
> 
> In which case you're going to have to do some fancy dancing to recover stuff.
> I suspect the stuff is still there though I don't have a way yet to
> prove it.  (I tried writing a partition table onto a disk with a
> filesystem on it and I was able to recover it).
> 
> Do you happen to have a spare 2T drive around? Can you borrow one?
> If it were me and it really was business data, I'd start out by making
> copies of the existing disks using dd.
> 
> Not really but I have a NAS box that I am working to setting up and
> have 3 TB drives in there.
> 
> I would just pull on of those and then do the copying thing.
> 
> Did a dd transfer of a 1 TB drive and that took 40 hours - - - so its
> going to be a real slow boat to china for that one.
> 
> There only was about 80 or 100 GB of actual data on the array, would
> there be any way of just copying data and not empty space?
> 
> Would you know what kind of filesystem was on there?
> 
> The partitions were setup as ext4 (IIRC).
> 
> Is there any way of checking that?
> 
> You might be able to  find that in some logs, but I certainly can't in
> my ubuntu system.
> That is likely to be a critical piece of information.
> 
> Keep at it.
> Go slow and careful.
> Research before doing.
> 
> This is why I have been very cautious.
> 
> Thank you for the help!
> 
> Dee
> 
> gerry <[email-masked]>
> Aug 31
> Reply
> to me
> Cool.  Good info.
> So yes it was a filesystem and not a partition and it was ext4.
> 
> So it seems pretty likely that sde went bad and took the array with it.
> 
> Aug 28 10:39:28 debiantestingbase kernel: [    4.009704] sd 8:0:0:0:
> [sde] Attached SCSI disk
> Aug 28 10:39:28 debiantestingbase kernel: [    4.018710] ata16.00:
> exception Emask 0x1 SAct 0x0 SErr 0x0 action 0x0
> Aug 28 10:39:28 debiantestingbase kernel: [    4.018753] ata16.00:
> irq_stat 0x40000001
> Aug 28 10:39:28 debiantestingbase kernel: [    4.018783] ata16.00: cmd
> a0/01:00:00:00:01/00:00:00:00:00/a0 tag 1 dma 16640 in
> Aug 28 10:39:28 debiantestingbase kernel: [    4.018783]
> Inquiry 12 01 00 00 ff 00res 50/00:00:af:6d:70/00:00:74:00:00/e0 Emask
> 0x1 (device
> error)
> 
> By that I mean that the array happily synced all the bad data back up.
> You can see that these are nicely synced.  :-(
> /dev/sdb1   f1c9645dbc14efddc7d8a322685f26eb  -
> /dev/sdc1   f1c9645dbc14efddc7d8a322685f26eb  -
> /dev/sde1   979c1a460f2130d7fd0e9210e2d15e90  -
> /dev/sdf1   979c1a460f2130d7fd0e9210e2d15e90  -
> 
> I would expect that you could fsck the device and mostly recover it.
> But I'd still make a backup. You only have to backup two of them.
> sdb1 and sdf1.
> Remove the md device using
> mdadm --stop /dev/md0
> 
> mdadm --remove /dev/md0
> (but that one doesn't seem to do anything for me, so I'm not all that
> clear about it.)
> 
> To backup, I'd try something like this and watch how fast it happens.
> 
> Then read and compress the filesystem onto a different disk.
> Don't do raw copy directly to the hard drive.
> Parition and make a file system on it. If you mount it on /mnt/newbackupdisk
> 
> Then a likely command to extract the data, compress it and write it to
> the backup disk would be:
>    dd if=/dev/sdc1 bs=1M | gzip > /mnt/newbackupdisk/sdc1.img.gz
>    dd if=/dev/sdf1 bs=1M | gzip > /mnt/newbackupdisk/sdc1.img.gz
> 
> While dd is running, you can use ps -ef | grep dd to find it's PID and send
> a signal to it to let it tell you its progress.
> 
> Paraphrasing  from the dd man page:
> 
> Sending a USR1 signal to a running 'dd' process makes it print I/O
> statistics to standard error and then resume copying.
> if pid is your process id, then
>    kill -USR1 $pid
> Returns a line like this:
>    18335302+0 records in 18335302+0 records out 9387674624 bytes (9.4
> GB) copied, 34.6279 seconds, 271 MB/s
> And you can just do that over and over as your copy runs.
> 
> Once you have a backup, put the md0 device back together.
> Either manually or by rebooting your machine.
> You might want to remove the /dev/sde drive entirely from the machine.
> 
> You could use
>    smartctl -a /dev/sde
> To get a serial number of the disk so you know you unplug the right one.
> 
> For the fsck, make sure you are telling it that you are doing ext4.
> fsck.ext4.
> 
> 
> BTW my command
>     parted /dev/sdc1 list
> should have been
>     parted /dev/sdc1 print
> But it's moot now anyway.
> 
> 
> Keep on keepin on.
> Gerry
> 
> 
> --
> [email-masked]
> 
> 
> o1bigtenor <[email-masked]>
> Aug 31
> Reply
> to ghaverla
> Forwarded conversation
> Subject: computer questions
> ------------------------
> 
> From: o1bigtenor <[email-masked]>
> Date: Sat, Aug 29, 2015 at 5:38 AM
> To: TCLUG Mailing List <[email-masked]>
> 
> 
> Greetings
> 
> I don't know what's going on but this is weird!
> 
> System has been running Debian 8 with things going good (raid mounts
> so when saving a file after a reboot all I need to do is click on the
> listed array give the root password and bob's your uncle).
> 
> (Its a uefi setup)
> Well this morning - - -
> fdisk -l shows the array
> its not mounted
> 
> the command I used to use for mounting in Debian 7 no longer works
> 
> so - - - - any suggestions?
> 
> TIA
> 
> Dee
> 
> 
> ----------
> From: Marc Skinner <[email-masked]>
> Date: Sat, Aug 29, 2015 at 11:04 AM
> To: TCLUG Mailing List <[email-masked]>
> 
> 
> Is this a hardware raid or software raid?  More details?  Is the raid
> array internal or externally connected?
> 
> 
> Thanks.
> _______________________________________________
> TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
> [email-masked]
> http://mailman.mn-linux.org/mailman/listinfo/tclug-list
> 
> 
> _______________________________________________
> TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
> [email-masked]
> http://mailman.mn-linux.org/mailman/listinfo/tclug-list
> 
> ----------
> From: o1bigtenor <[email-masked]>
> Date: Sat, Aug 29, 2015 at 11:41 AM
> To: TCLUG Mailing List <[email-masked]>
> 
> 
> 
> 
> On Sat, Aug 29, 2015 at 11:04 AM, Marc Skinner <[email-masked]> wrote:
> Is this a hardware raid or software raid?  More details?  Is the raid
> array internal or externally connected?
> 
> software raid
> internal
> 
> # cat /proc/mdstat
> Personalities : [raid10]
> md0 : active (auto-read-only) raid10 sde1[0] sdf1[1] sdc1[4] sdb1[3]
>       1953518592 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
> 
> # mdadm --examine --scan
> ARRAY /dev/md/0  metadata=1.2 UUID=79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
> name=debianbase:0
> 
> from blkid
> 
> /dev/md0: PTUUID="1eb47793-0cdf-4e16-ae84-33ec825ac448" PTTYPE="gpt"
> 
> 
> from fdisk -l
> 
> Disk /dev/md0: 1.8 TiB, 2000403038208 bytes, 3907037184 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
> Disklabel type: gpt
> Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448
> 
> 
> 
> Thanks.
> 
> 
> 
> 
> On 08/29/2015 05:38 AM, o1bigtenor wrote:
> Greetings
> 
> I don't know what's going on but this is weird!
> 
> System has been running Debian 8 with things going good (raid mounts so
> when saving a file after a reboot all I need to do is click on the
> listed array give the root password and bob's your uncle).
> 
> (Its a uefi setup)
> Well this morning - - -
> fdisk -l shows the array
> its not mounted
> 
> the command I used to use for mounting in Debian 7 no longer works
> 
> so - - - - any suggestions?
> 
> TIA
> 
> Dee
> 
> 
> 
> _______________________________________________
> TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
> [email-masked]
> http://mailman.mn-linux.org/mailman/listinfo/tclug-list
> 
> 
> _______________________________________________
> TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
> [email-masked]
> http://mailman.mn-linux.org/mailman/listinfo/tclug-list
> 
> 
> ----------
> From: Marc Skinner <[email-masked]>
> Date: Sat, Aug 29, 2015 at 11:47 PM
> To: TCLUG Mailing List <[email-masked]>
> 
> 
> So what is the problem ... it won't mount?
> 
> 
> Did you do an fsck on it? Assuming you had a filesystem that worked on
> it prior to your issue.
> 
> 
> 
> 
> 
> 
> 
> 
> On 08/29/2015 11:41 AM, o1bigtenor wrote:
> 
> 
> On Sat, Aug 29, 2015 at 11:04 AM, Marc Skinner <[email-masked]
>         [email-masked] <mailto:[email-masked]>
>         http://mailman.mn-linux.org/mailman/listinfo/tclug-list
> 
> 
>     _______________________________________________
>     TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
>     [email-masked] <mailto:[email-masked]>
>     http://mailman.mn-linux.org/mailman/listinfo/tclug-list
> 
> 
> 
> 
> _______________________________________________
> TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
> [email-masked]
> http://mailman.mn-linux.org/mailman/listinfo/tclug-list
> 
> 
> ----------
> From: o1bigtenor <[email-masked]>
> Date: Sun, Aug 30, 2015 at 6:25 AM
> To: TCLUG Mailing List <[email-masked]>
> 
> 
> 
> 
> On Sat, Aug 29, 2015 at 11:47 PM, Marc Skinner <[email-masked]> wrote:
> So what is the problem ... it won't mount?
> 
> 
> Did you do an fsck on it? Assuming you had a filesystem that worked on
> it prior to your issue.
> 
> 
> 
> # fsck /dev/md0
> fsck from util-linux 2.26.2
> e2fsck 1.42.13 (17-May-2015)
> ext2fs_open2: Bad magic number in super-block
> fsck.ext2: Superblock invalid, trying backup blocks...
> fsck.ext2: Bad magic number in super-block while trying to open /dev/md0
> 
> The superblock could not be read or does not describe a valid ext2/ext3/ext4
> filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
> filesystem (and not swap or ufs or something else), then the superblock
> is corrupt, and you might try running e2fsck with an alternate superblock:
>     e2fsck -b 8193 <device>
>  or
>     e2fsck -b 32768 <device>
> 
> 
> Did some reading on fsck and there are warnings of data loss in using
> it to repair.
> 
> As this is a data raid (with business data on it) data loss and
> corruption is not
> an option I want to pursue.
> 
> Any ideas on how to re-mount the array?
> 
> TIA
> 
> Dee
> 
> ----------
> From: gerry <[email-masked]>
> Date: Sun, Aug 30, 2015 at 12:34 PM
> To: o1bigtenor <[email-masked]>
> 
> 
> A few thoughts:
> 
> Your prior email shows that the md was set to read-only, right?
> md0 : active (auto-read-only) raid10 sde1[0] sdf1[1] sdc1[4] sdb1[3]
> 
> Look in your logs for reasons why it got set to auto-read-only.
> This isn't your issue but it's something to read:
> http://unix.stackexchange.com/questions/101072/new-md-array-is-auto-read-only-and-has-resync-pending
> 
> That email also showed some output from
>     fdisk -l /dev/md0.
> Is there any chance you partitioned /dev/md0?
> Because the output says there is a gpt partition table on the /dev/md0 device.
> 
> Quoting:
> Disk /dev/md0: 1.8 TiB, 2000403038208 bytes, 3907037184 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
> Disklabel type: gpt
> Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448
> 
> Was there anything after that?
> Run
>     parted -l
> 
> 
> You made a statement about booting that was confusing to me.
> Here's the section:
> 
>         System has been running Debian 8 with things going good (raid
>         mounts so
>         when saving a file after a reboot all I need to do is click on the
>         listed array give the root password and bob's your uncle).
> [snip]
> 
>         the command I used to use for mounting in Debian 7 no longer works
> 
> What does that mean "click on the listed array give the root password" ?
> Where were you doing such a thing?  Did you by any chance make
> /dev/md0 into an encrypted device?
> 
> What is the command you "used to use for mounting in Debian 7" ?
> 
> Gerry
> 
> 
> 
> 
> --
> [email-masked]
> 
> ----------
> From: o1bigtenor <[email-masked]>
> Date: Sun, Aug 30, 2015 at 5:38 PM
> To: gerry <[email-masked]>
> 
> 
> 
> 
> On Sun, Aug 30, 2015 at 12:34 PM, gerry <[email-masked]> wrote:
> A few thoughts:
> 
> Your prior email shows that the md was set to read-only, right?
> md0 : active (auto-read-only) raid10 sde1[0] sdf1[1] sdc1[4] sdb1[3]
> 
> Look in your logs for reasons why it got set to auto-read-only.
> This isn't your issue but it's something to read:
> http://unix.stackexchange.com/questions/101072/new-md-array-is-auto-read-only-and-has-resync-pending
> 
> Any ideas which log to look into?
> 
> 
> That email also showed some output from
>     fdisk -l /dev/md0.
> Is there any chance you partitioned /dev/md0?
> 
> (Its over 3 years ago.) All four discs were partitioned (sdb1, sdc1 etc).
> 
> Because the output says there is a gpt partition table on the /dev/md0 device.
> 
> Quoting:
> Disk /dev/md0: 1.8 TiB, 2000403038208 bytes, 3907037184 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
> Disklabel type: gpt
> Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448
> 
> Was there anything after that?
> Run
>     parted -l
> 
> Model: ATA ST31000524AS (scsi)
> Disk /dev/sdf: 1000GB
> Sector size (logical/physical): 512B/512B
> Partition Table: msdos
> Disk Flags:
> 
> Number  Start   End     Size    Type     File system  Flags
>  1      1049kB  1000GB  1000GB  primary               raid
> 
> 
> Model: Linux Software RAID Array (md)
> Disk /dev/md0: 2000GB
> Sector size (logical/physical): 512B/4096B
> Partition Table: gpt
> Disk Flags:
> 
> Number  Start  End  Size  File system  Name  Flags
> 
> 
> That's the information for the last disc (in the array) and the array using
> parted -l
> 
> 
> 
> You made a statement about booting that was confusing to me.
> Here's the section:
> 
>         System has been running Debian 8 with things going good (raid
>         mounts so
>         when saving a file after a reboot all I need to do is click on the
>         listed array give the root password and bob's your uncle).
> [snip]
> 
>         the command I used to use for mounting in Debian 7 no longer works
> 
> What does that mean "click on the listed array give the root password" ?
> 
> Am using LXDE and the file manager is PCManFM.
> 
> The file manager would bring up a representation where the array was listed as a
> device. After a boot clicking on the button a dialog box would open which asked
> for the password for root. Upon entry access was open to the array.
> 
> Where were you doing such a thing?  Did you by any chance make
> /dev/md0 into an encrypted device?
> 
> As far as I know I didn't.
> 
> What is the command you "used to use for mounting in Debian 7" ?
> 
> mount /dev/dm-0 /home/myspace/RAID
> 
> My guess is that I managed to find some kind of a bug that descends
> with a combined use
> of suspend (upon night close down on the system), mdadm, lxde and pcmanfm.
> 
> Thanks for the ideas.
> 
> Dee
> 
> ----------
> From: gerry <[email-masked]>
> Date: Sun, Aug 30, 2015 at 9:57 PM
> To: o1bigtenor <[email-masked]>
> 
> 
> Any ideas which log to look into?
> 
> Look in all of them.  Everything in /var/log/
> Use commands like
> 
> cd /var/log
> grep read * | grep only
> 
> and for the compressed files
> zgrep read *.gz | grep only
> 
> Then for those files that look interesting, use less or zless to view them.
> (The "q" key will exit less or zless)
> 
> You'll need to be root do all that.
> 
> That's the information for the last disc (in the array) and the array using
> parted -l
> 
> <sigh>
> I can't tell from that whether you sent me the entire output of parted
> -l or you were picking and choosing.
> 
> For all these commands, run them and send the entire output. Never
> skimp on information when debugging.
> 
> mount /dev/dm-0 /home/myspace/RAID
> Hmm. In debian 7 you mounted /dev/dm-0 but that's just a devmapper
> device and they are dynamic.  More about devmapper devices can be
> gotten like this:
> 
> ls -l /dev/mapper
> dmsetup ls
> dmestup table
> 
> also run (as root, again)
> lsblk
> 
> 
> Hopefully that will give more information.
> 
> 
> 
> --
> [email-masked]
> 
> On Sun, 30 Aug 2015, o1bigtenor wrote:
> 
> 
> ----------
> From: o1bigtenor <[email-masked]>
> Date: Sun, Aug 30, 2015 at 10:11 PM
> To: gerry <[email-masked]>
> 
> ----------
> From: gerry <[email-masked]>
> Date: Sun, Aug 30, 2015 at 11:42 PM
> To: o1bigtenor <[email-masked]>
> 
> 
> Cool.  And did you find anything in the logs?
> 
> The dmsetup output is interesting. I would have expected some dm devices.
> Without dm devices, there are no /dev/dm things to mount so that's why
> your old Debian7 command doesn't work.
> 
> Did you expect there to be a partition table on /dev/md0?
> Or did you expect to be able to just mount /dev/md0?
> (was there a filesystem on /dev/md0 or was it a  partitioned logical
> disk mirror?)
> 
> You might be able to figure that out from the logs too.
> 
> I think the thing went to read-only to protect itself.
> 
> If I were there, I'd run these commands and see what they say.
> parted /dev/sdb1 list
> parted /dev/sdc1 list
> parted /dev/sde1 list
> parted /dev/sdf1 list
> 
> Before trying to "fix" the array, you will want to see if you can
> figure out if you still have one good setup.
> 
> That mirror setup likely mirrors sdb1 with sdc1 and sde1 with sdf1.
> And the stripes were acrosss those two mirrors.
> To see if an individual disk got corrupted, try this:
> (This is non-destructive.)
> for a in /dev/sd{b,c,e,f}1
> do
>   echo -n $a "  "
>   dd skip=10 if=$a bs=1M count=10 status=none| md5sum
> done
> 
> 
> I made a dummy setup with loop devices and I see this:
> for a in /dev/loop*p1
> do
>   echo -n $a "  ";dd skip=10 if=$a bs=1M count=10 status=none| md5sum
> done
> /dev/loop1p1   58b27e44c48997bc126eec1eec12ab8b  -
> /dev/loop2p1   58b27e44c48997bc126eec1eec12ab8b  -
> /dev/loop3p1   5aa8e9cfbcc3f6612f839b9a96da41df  -
> /dev/loop4p1   5aa8e9cfbcc3f6612f839b9a96da41df  -
> 
> See how 1 and 2 are identical? And 3 and 4 are identical?
> That's the mirroring.
> Let's see if yours is similar.
> ....
> 
> If you don't see errors in the log, then it's fairly likely that
> something actually wrote a partition table onto /dev/md0. Otherwise
> it's wouldn't see
> Disklabel type: gpt
> Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448
> 
> In which case you're going to have to do some fancy dancing to recover stuff.
> I suspect the stuff is still there though I don't have a way yet to
> prove it.  (I tried writing a partition table onto a disk with a
> filesystem on it and I was able to recover it).
> 
> Do you happen to have a spare 2T drive around? Can you borrow one?
> If it were me and it really was business data, I'd start out by making
> copies of the existing disks using dd.
> 
> Would you know what kind of filesystem was on there?
> You might be able to  find that in some logs, but I certainly can't in
> my ubuntu system.
> That is likely to be a critical piece of information.
> 
> Keep at it.
> Go slow and careful.
> Research before doing.
> 
> Gerry
> 
> 
> --
> [email-masked]
> 
> On Sun, 30 Aug 2015, o1bigtenor wrote:
> 
> 
> ----------
> From: o1bigtenor <[email-masked]>
> Date: Mon, Aug 31, 2015 at 7:11 AM
> To: gerry <[email-masked]>
> 
> ----------
> From: gerry <[email-masked]>
> Date: Mon, Aug 31, 2015 at 8:43 AM
> To: o1bigtenor <[email-masked]>
> 
> 
> o1bigtenor <[email-masked]>
> Aug 31
> Reply
> to gerry
> 
> Sorry - - the hard drive I want to use is showing up as sdg1. That
> means that it has a partition on it - - yes?
> 
> I don't know how to make a file system on it?
> 
> Would you explain this - - please?
> 
> Then I will continue the odessey!
> 
> Dee
> 
> gerry <[email-masked]>
> Aug 31
> Reply
> to me
> You bet.
> 
> run this to look at it:
>    parted /dev/sdg list
> 
> and this to make a file system on it:
>    mkfs -t ext4 /dev/sdg1
> 
> And then mount it.
>    mkdir /mnt/backup
>    mount /dev/sdg1 /mnt/backup
> 
> I see there was a typo in my commands to backup.
> So here's a repeat with that fixed.
> 
>    mdadm --stop /dev/md0
>    dd if=/dev/sdc1 bs=1M | gzip > /mnt/backup/sdc1.img.gz
>    dd if=/dev/sdf1 bs=1M | gzip > /mnt/backup/sdf1.img.gz
> 
> While dd is running, in another window you can use
>    ps -ef | grep dd
> to find its PID and send a signal to it to let it tell you its progress.
>    kill -USR1 $pid
> 
> You can also just do ls -l /mnt/backup and see how big things are getting.
> 
> 
> o1bigtenor <[email-masked]>
> Aug 31
> Reply
> to gerry
> On Mon, Aug 31, 2015 at 4:17 PM, gerry <[email-masked]> wrote:
> You bet.
> 
> run this to look at it:
>    parted /dev/sdg list
> 
> and this to make a file system on it:
>    mkfs -t ext4 /dev/sdg1
> 
> And then mount it.
>    mkdir /mnt/backup
>    mount /dev/sdg1 /mnt/backup
> 
> I see there was a typo in my commands to backup.
> So here's a repeat with that fixed.
> 
>    mdadm --stop /dev/md0
>    dd if=/dev/sdc1 bs=1M | gzip > /mnt/backup/sdc1.img.gz
>    dd if=/dev/sdf1 bs=1M | gzip > /mnt/backup/sdf1.img.gz
> 
> While dd is running, in another window you can use
>    ps -ef | grep dd
> to find its PID and send a signal to it to let it tell you its progress.
>    kill -USR1 $pid
> 
> You can also just do ls -l /mnt/backup and see how big things are getting.
> 
> 
> sdc1 has been copied.
> sdf1 is being worked on.
> 
> When this is completed then I go to:
> 
> Once you have a backup, put the md0 device back together.
> Either manually or by rebooting your machine.
> You might want to remove the /dev/sde drive entirely from the machine.
> 
> You could use
>    smartctl -a /dev/sde
> To get a serial number of the disk so you know you unplug the right one.
> 
> For the fsck, make sure you are telling it that you are doing ext4.
> fsck.ext4.
> 
> So - - - you wouldn't use sde any more?
> (I have a replacement drive on hand but not sure the shop will
> warranty something
> that they can't get to give errors on command.)
> 
> So
> 
> 1. determine which is sde
> 2. remove
> what do I do then?
> 
> Is there any way to check the data on my backup to see if its readable?
> (sorry if you include the commands please? (that would be great!)
> 
> Thanking you for your assistance.
> 
> Dee
> 
> gerry <[email-masked]>
> Sep 1
> Reply
> to me
> How goes the battle?
> You decide to take a nap?  <grin>
> 
> 
> --
> [email-masked]
> 
> o1bigtenor <[email-masked]>
> Sep 1
> Reply
> to gerry
> On Tue, Sep 1, 2015 at 12:14 PM, gerry <[email-masked]> wrote:
> How goes the battle?
> You decide to take a nap?  <grin>
> 
> 
> Not getting in a hurry because I've borked up too many things in the
> past getting in a hurry.
> 
> Had a few questions for you in the last email. Was sort of waiting
> until I had some way
> of verifying that I had my information before I started playing with md0.
> 
> Over to you . . .
> 
> gerry <[email-masked]>
> Sep 1
> Reply
> to me
> I answered all the questions in your last email.
> I'll verify that that got sent. If it didn't I'll send it again.
> Or maybe I'll just send it again anyway. :-)
> 
> 
> --
> [email-masked]
> 
> 
> gerry <[email-masked]>
> Sep 1
> Reply
> to me
> Wow. You already finished a backup!?
> That's cool.  And fast.
> How big is it?
> (ls -l /mnt/backup/sdc1.img.gz )
> 
> Figuring out whether it's readable is an interesting question.
> You could double check that they are right with
> gzip -dc /mnt/backup/sdc1.img.gz | md5sum
> and
> cat /dev/sdc1 | md5sum
> But that would take quite a while.
> 
> Either way, part of a mirrored and striped array is not readable anyway.
> The stripes have to be put back together to be readable.
> 
> BTW while the backup is running, do dmesg | tail once in a while to
> see if as it reads it's finding errors.
> 
> 
> I'm not all that certain whether to say you should or should not use sde.
> But if you do pull it out, don't replace it before finishing any recovery.
> I was just suggesting that you pull it because it definitely had some
> kind of error.  I could be off base there though.
> Either way, you should identify all the drives now.
> 
> Make a list of all the UUIDs from
> lsblk -o +UUID
> match them up with the serial numbers from smartctl.
> 
> Maybe just leave sde in there.
> Put the md device back together ( mdadm --examine --scan
> Then go back to your original fsck command
> 
> fsck.ext4 /dev/md0
> 
> And as it asks you if you want to recover -- hopefully it'll ask you
> if you want to use a different superblock -- just say yes to the
> choices.
> 
> With your image backup, if that goes really horribly, you can put the
> data from the gz files back onto the disk.
> 
> 
> gerry <[email-masked]>
> Sep 1
> Reply
> to me
> Turns out I postponed the message because it wasn't quite complete.
> I just hit "send" on it, but it's probably still not complete.
> 
> 
> o1bigtenor <[email-masked]>
> Sep 1
> Reply
> to gerry
> Greetings
> 
> On Tue, Sep 1, 2015 at 3:37 PM, gerry <[email-masked]> wrote:
> Wow. You already finished a backup!?
> That's cool.  And fast.
> 
> Backup finished after I crashed out but it was almost done at that time
> 
> How big is it?
> (ls -l /mnt/backup/sdc1.img.gz )
> 
> drwx------ 2 root root       16384 Aug 31 18:11 lost+found
> -rw-r--r-- 1 root root 94661812644 Aug 31 21:56 sdc1.img.gz
> -rw-r--r-- 1 root root 95414436881 Aug 31 23:48 sdf1.img.gz
> 
> 
> 
> Figuring out whether it's readable is an interesting question.
> You could double check that they are right with
> gzip -dc /mnt/backup/sdc1.img.gz | md5sum
> and
> cat /dev/sdc1 | md5sum
> But that would take quite a while.
> 
> Either way, part of a mirrored and striped array is not readable anyway.
> The stripes have to be put back together to be readable.
> 
> BTW while the backup is running, do dmesg | tail once in a while to
> see if as it reads it's finding errors.
> 
> Back up all done!
> 
> 
> I'm not all that certain whether to say you should or should not use sde.
> But if you do pull it out, don't replace it before finishing any recovery.
> I was just suggesting that you pull it because it definitely had some
> kind of error.  I could be off base there though.
> Either way, you should identify all the drives now.
> 
> Make a list of all the UUIDs from
> lsblk -o +UUID
> match them up with the serial numbers from smartctl.
> 
> Maybe just leave sde in there.
> Put the md device back together ( mdadm --examine --scan
> Then go back to your original fsck command
> 
> fsck.ext4 /dev/md0
> 
> And as it asks you if you want to recover -- hopefully it'll ask you
> if you want to use a different superblock -- just say yes to the
> choices.
> 
> With your image backup, if that goes really horribly, you can put the
> data from the gz files back onto the disk.
> 
> Well - - - I am trying to re-shingle my house, working on books for
> business, and getting ready to pour 2 pads of concrete.
> 
> So between all these things and the weather I am sort of busy.
> 
> This is a priority but even that doesn't mean that I work at it all
> the time. When my brain is on (I'm a morning person) this is something
> I work on very early
> or in between other things.
> 
> I have lots to work on at this point with this and will get back to
> you later to let you know how things went.
> (At least I assume you would like to know!)
> 
> I would like to thank you very very much for your assistance.
> (Am thinking of writing up the steps and findings and notes and would
> then post it where I asked for help.)
> 
> With a great big thank you!!
> 
> Dee
> 
> gerry <[email-masked]>
> Sep 1
> Reply
> to me
> Holy mackerel. You are an ambitious person.
> Me not so much. :-)
> My carport roof  is in terrible shape.  I've been procrastinating
> about it for way too long.
> 
> The backup sizes look sane so you probably have a good backup.
> 
> I'm glad to help you as others have helped me.
> 
> 
> o1bigtenor <[email-masked]>
> Sep 1
> Reply
> to gerry
> Greetings
> 
> On Tue, Sep 1, 2015 at 5:46 PM, gerry <[email-masked]> wrote:
> Holy mackerel. You are an ambitious person.
> 
> Oh the pads are for extensions an my barn.
> 
> I hope to not only be milking sheep in about 6 1/2 months but also
> processing the milk
> into fresh cheeses, yogurt and ice cream.
> 
> Then I will be real busy!
> 
> Me not so much. :-)
> My carport roof  is in terrible shape.  I've been procrastinating
> about it for way too long.
> 
> The backup sizes look sane so you probably have a good backup.
> 
> Do you know any way to check the backup to make sure?
> 
> I'm glad to help you as others have helped me.
> 
> Well - - - thank you very much.
> 
> If you are travelling through se Manitoba in the future send me a note
> and you are
> welcome to stop by for a meal!
> 
> Dee
> 
> gerry <[email-masked]>
> Sep 2
> Reply
> to me
> I did finally send you the email to say how you could check for sure,
> but here it is again:
> 
> Get an md5sum from the image (uncompressing it on the fly)
>    gzip -dc /mnt/backup/sdc1.img.gz | md5sum
> and compare that to the md5sum from the raw disk:
>    md5sum /dev/sdc1
> If they are the same, then your backup is correct.
> 
> For a concrete example, I picked my smallest partition and did it.
> The transcript follow.
> 
> I added some spaces between them for readability, but on my screen
> they are one after another.
> 
> #root@veeta:/tmp>dd if=/dev/sda1 | gzip > sda1.img.gz
> 2048+0 records in
> 2048+0 records out
> 1048576 bytes (1.0 MB) copied, 0.0634722 s, 16.5 MB/s
> 
> #root@veeta:/tmp>ls -l sda1.img.gz
> -rw-r--r-- 1 root root 54372 Sep  2 10:09 sda1.img.gz
> 
> #root@veeta:/tmp>gzip -dc sda1.img.gz | md5sum
> a9144ef256b8038412ea7426e5be198f  -
> 
> #root@veeta:/tmp>md5sum /dev/sda1
> a9144ef256b8038412ea7426e5be198f  /dev/sda1
> 
> And as always there are multiple ways to do things ... this is a little cleaner:
> 
> No dd command -- just gzip the device to stdout and send it to a file.
> #root@veeta:/tmp>gzip -vc /dev/sda1 > sda1.img.gz
> /dev/sda1:       94.8%
> 
> #root@veeta:/tmp>ls -l sda1.img.gz
> -rw-r--r-- 1 root root 54377 Sep  2 10:11 sda1.img.gz
> 
> zcat rather than gzip -dc:
> #root@veeta:/tmp>zcat sda1.img.gz | md5sum
> a9144ef256b8038412ea7426e5be198f  -
> 
> Thanks for the invite!  You're just over border after all. :-)
> 
> Sorta funny that Minnesota brags about being the state with 10,000
> lakes and Manitoba brags about 10 times that number. :-)
> 
> 
> o1bigtenor <[email-masked]>
> Sep 3
> Reply
> to gerry
> On Wed, Sep 2, 2015 at 10:23 AM, gerry <[email-masked]> wrote:
> I did finally send you the email to say how you could check for sure,
> but here it is again:
> 
> Get an md5sum from the image (uncompressing it on the fly)
>    gzip -dc /mnt/backup/sdc1.img.gz | md5sum
> and compare that to the md5sum from the raw disk:
>    md5sum /dev/sdc1
> If they are the same, then your backup is correct.
> 
> Hoo boy - - - A lot of time and the md5 sums are - - - - not the same!
> 
> Big time bummer.
> 
> Any suggestions?
> 
> gerry <[email-masked]>
> Sep 3
> Reply
> to me
> I would wonder if it's possible that the disk got written to in between time.
> ??
> 
> I'm glad you checked, though.
> Does dmesg | tail show any disk errors?
> 
> Did you do mdadm --stop /dev/md0 before making the backup?
> 
> I don't know how you would make sure the hard drive is readonly before
> doing the backup....
> Hmmmm.
> 
> 
> --
> [email-masked]
> 
> 
> o1bigtenor <[email-masked]>
> Sep 3
> Reply
> to gerry
> On Thu, Sep 3, 2015 at 9:23 PM, gerry <[email-masked]> wrote:
> I would wonder if it's possible that the disk got written to in between time.
> ??
> 
> No idea. I couldn't access it myself - - - I do know that!
> 
> I'm glad you checked, though.
> Does dmesg | tail show any disk errors?
> 
> Will do this one in the morning - - - sorry.
> 
> Did you do mdadm --stop /dev/md0 before making the backup?
> 
> Yes.
> 
> I don't know how you would make sure the hard drive is readonly before
> doing the backup....
> Hmmmm.
> 
> 
> This is being rather interesting isn't it!
> 
> Thanks for the help!
> 
> Dee
> 
> gerry <[email-masked]>
> Sep 4
> Reply
> to me
> It's always interesting.
> 
> I'm on a road trip to Colorado to visit family and go to a wedding, so
> it's a diveresion for me now.
> 
> And, of course, while on the trip, my mom wants me to get her laptop
> updated to window 10. <eyeroll>
> 
> But I do enjoy messing with  computers, so what can I say.
> I think I'd rather be in Manitoba, though. :-)
> 
> Ger
> 
> 
> o1bigtenor <[email-masked]>
> Sep 4
> Reply
> to gerry
> On Fri, Sep 4, 2015 at 10:35 AM, gerry <[email-masked]> wrote:
>> It's always interesting.
>>
>> I'm on a road trip to Colorado to visit family and go to a wedding, so it's
>> a diveresion for me now.
>>
>> And, of course, while on the trip, my mom wants me to get her laptop updated
>> to window 10. <eyeroll>
>>
>> But I do enjoy messing with  computers, so what can I say.
>> I think I'd rather be in Manitoba, though. :-)
>>
> Maybe not today.
> 
> Its very humid (94%) without raining and its supposed to rain heavily
> this evening
> at times. If you are the twin cities its beautiful down there. We'll
> be in that area the
> first weekend in Oct, I hope that you might be fortunate enough to
> live in one of
> the newer subdivisions where you have access to that multi-gigabit ISP services
> (I'm on what I call peasant internet here.)
> 
> Have a safe trip and lots of fun at the wedding.
> 
> Have you thought of setting her up in Zorin - - I'm told its quite win like.
> 
> I've also heard that win 10 is quite a bit of a dog at this time. Not
> sure because
> I'm not using it - - - still stuck at win 7 (have a few VMs with that).
> 
> TTYL
> 
> Dee
> 
> 
> o1bigtenor <[email-masked]>
> Sep 8
> Reply
> to gerry
> Greetings
> 
> How went the trip?
> 
> 
> 
> On Thu, Sep 3, 2015 at 9:23 PM, gerry <[email-masked]> wrote:
>> I would wonder if it's possible that the disk got written to in between
>> time.
>> ??
>>
>> I'm glad you checked, though.
>> Does dmesg | tail show any disk errors?
> 
> # cat /etc/fstab
> # /etc/fstab: static file system information.
> #
> # Use 'blkid' to print the universally unique identifier for a
> # device; this may be used with UUID= as a more robust way to name devices
> # that works even if disks are added and removed. See fstab(5).
> #
> # <file system> <mount point>   <type>  <options>       <dump>  <pass>
> # / was on /dev/sda3 during installation
> UUID=a3fbea19-5b7d-47af-b06c-08c49171211b /               ext4
> errors=remount-ro 0       1
> # /boot/efi was on /dev/sda1 during installation
> UUID=8642-E734  /boot/efi       vfat    umask=0077      0       1
> # /home was on /dev/sda5 during installation
> UUID=b36dbfc6-6897-4311-aa87-9a958ccad072 /home           ext4
> defaults        0       2
> # swap was on /dev/sda4 during installation
> UUID=354a6e33-5743-4c97-bfb9-fd6f9d4521e6 none            swap    sw
>            0       0
> # swap was on /dev/sdd10 during installation
> UUID=04daef55-ff1b-4fe5-a897-ebe397a145b6 none            swap    sw
>            0       0
> /dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
> # raid10 array
> PTUUID=1eb47793-0cdf-4e16-ae84-33ec825ac448 /media/darald/Raid10
>    gpt    defaults         0       2
> root@debiantestingbase:/media/root# fsck -N /dev/md/0
> fsck from util-linux 2.26.2
> [/sbin/fsck.ext2 (1) -- /dev/md0] fsck.ext2 /dev/md0
> root@debiantestingbase:/media/root# fsck /dev/md0
> fsck from util-linux 2.26.2
> e2fsck 1.42.13 (17-May-2015)
> ext2fs_open2: Bad magic number in super-block
> fsck.ext2: Superblock invalid, trying backup blocks...
> fsck.ext2: Bad magic number in super-block while trying to open /dev/md0
> 
> The superblock could not be read or does not describe a valid ext2/ext3/ext4
> filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
> filesystem (and not swap or ufs or something else), then the superblock
> is corrupt, and you might try running e2fsck with an alternate superblock:
>     e2fsck -b 8193 <device>
>  or
>     e2fsck -b 32768 <device>
> 
> root@debiantestingbase:/media/root# dmesg | tail
> [644039.250935] r8169 0000:0b:00.0 enp11s0: link up
> [644039.250943] IPv6: ADDRCONF(NETDEV_CHANGE): enp11s0: link becomes ready
> [644042.651578] IPv6: ADDRCONF(NETDEV_UP): wlp9s0: link is not ready
> [644042.816581] r8169 0000:0b:00.0 enp11s0: link down
> [644042.816598] r8169 0000:0b:00.0 enp11s0: link down
> [644042.816631] IPv6: ADDRCONF(NETDEV_UP): enp11s0: link is not ready
> [644043.008747] r8169 0000:0b:00.0 enp11s0: link down
> [644043.008795] IPv6: ADDRCONF(NETDEV_UP): enp11s0: link is not ready
> [644046.068706] r8169 0000:0b:00.0 enp11s0: link up
> [644046.068722] IPv6: ADDRCONF(NETDEV_CHANGE): enp11s0: link becomes ready
> 
> 
> dmesg doesn't seem to show any disk problems
> 
> I am hoping that you have some more ideas up your sleeve.
> 
> Need to start working on some financial files. A bunch of information
> I use to process those kind of files is on the array.
> 
> Please?
> 
> TIA
> 
> 
> gerry <[email-masked]>
> Sep 9
> Reply
> to me
> It was a grand time!
> Lovely drive. No breakdowns.
> Took two days out and one day back.
> Two weddings - one Christiand and one Hindu (sort of :-)  )
> 4 big meals.
> 
> Here are some pics.
> http://skerbitz.org/photos/2015Co.small/
> All the women got to wear Saaris and get Henna'd.
> 
> Back to real life again now. :-)
> 
> Gerry
> 
> 
> 
> --
> [email-masked]
> 
> 
> o1bigtenor <[email-masked]>
> Sep 9
> Reply
> to gerry
> On Wed, Sep 9, 2015 at 4:47 PM, gerry <[email-masked]> wrote:
>> It was a grand time!
>> Lovely drive. No breakdowns.
>> Took two days out and one day back.
>> Two weddings - one Christiand and one Hindu (sort of :-)  )
>> 4 big meals.
>>
>> Here are some pics.
>> http://skerbitz.org/photos/2015Co.small/
>> All the women got to wear Saaris and get Henna'd.
>>
>> Back to real life again now. :-)
> 
> Glad you had a great time!!
> 
> You bet - - - as much fun as weddings are - - - they end and its back to
> all that other stuff.
> 
> Do you know anything about file system repair?
> 
> Dee
> 
> gerry <[email-masked]>
> Sep 9
> Reply
> to me
> :-)
> Well, I didn't fry all my braincells I don't think. :-)
> 
> How is your filesystem now?
> 
> 
> You should look into a package called "testdisk"
> 
> gsker@veeta:~> dpkg -S =photorec
> testdisk: /usr/bin/photorec
> 
> gsker@veeta:~> aptitude show testdisk
> Package: testdisk
> State: installed
> Automatically installed: no
> Version: 6.14-3build2
> Priority: optional
> Section: universe/admin
> Maintainer: Ubuntu Developers <[email-masked]>
> Architecture: amd64
> Uncompressed Size: 1,238 k
> Depends: e2fslibs (>= 1.41.0), libc6 (>= 2.14), libjpeg8 (>= 8c),
> libncursesw5 (>= 5.6+20070908), libntfs-3g853, libtinfo5, libuuid1 (>=
> 2.16), zlib1g (>=
>          1:1.1.4), ntfs-3g
> Conflicts: testdisk
> Description: Partition scanner and disk recovery tool, and PhotoRec
> file recovery tool
>  TestDisk checks the partition and boot sectors of your disks. It is
> very useful in forensics, recovering lost partitions. It works with :
>  * DOS/Windows FAT12, FAT16 and FAT32
>  * NTFS ( Windows NT/2K/XP )
>  * Linux Ext2 and Ext3
>  * BeFS ( BeOS )
>  * BSD disklabel ( FreeBSD/OpenBSD/NetBSD )
>  * CramFS (Compressed File System)
>  * HFS and HFS+, Hierarchical File System
>  * JFS, IBM's Journaled File System
>  * Linux Raid
>  * Linux Swap (versions 1 and 2)
>  * LVM and LVM2, Linux Logical Volume Manager
>  * Netware NSS
>  * ReiserFS 3.5 and 3.6
>  * Sun Solaris i386 disklabel
>  * UFS and UFS2 (Sun/BSD/...)
>  * XFS, SGI's Journaled File System
> 
>  PhotoRec is file data recovery software designed to recover lost
> pictures from digital camera memory or even Hard Disks. It has been
> extended to search
>  also for non audio/video headers. It searches for following files and
> is able to undelete them:
>  * Sun/NeXT audio data (.au)
>  * RIFF audio/video (.avi/.wav)
>  * BMP bitmap (.bmp)
>  * bzip2 compressed data (.bz2)
>  * Source code written in C (.c)
>  * Canon Raw picture (.crw)
>  * Canon catalog (.ctg)
>  * FAT subdirectory
>  * Microsoft Office Document (.doc)
>  * Nikon dsc (.dsc)
>  * HTML page (.html)
>  * JPEG picture (.jpg)
>  * MOV video (.mov)
>  * MP3 audio (MPEG ADTS, layer III, v1) (.mp3)
>  * Moving Picture Experts Group video (.mpg)
>  * Minolta Raw picture (.mrw)
>  * Olympus Raw Format picture (.orf)
>  * Portable Document Format (.pdf)
>  * Perl script (.pl)
>  * Portable Network Graphics (.png)
>  * Raw Fujifilm picture (.raf)
>  * Contax picture (.raw)
>  * Rollei picture (.rdc)
>  * Rich Text Format (.rtf)
>  * Shell script (.sh)
>  * Tar archive (.tar )
>  * Tag Image File Format (.tiff)
>  * Microsoft ASF (.wma)
>  * Sigma/Foveon X3 raw picture (.x3f)
>  * zip archive (.zip)
> 
> Also debugfs
> 
> --
> [email-masked]
> 
> 
> o1bigtenor <[email-masked]>
> Sep 9
> Reply
> to gerry
> On Wed, Sep 9, 2015 at 6:05 PM, gerry <[email-masked]> wrote:
>> :-)
>> Well, I didn't fry all my braincells I don't think. :-)
>>
>> How is your filesystem now?
> 
> On the raid array - - - problems!!
> 
> TestDisk 6.14, Data Recovery Utility, July 2013
> Christophe GRENIER <[email-masked]>
> http://www.cgsecurity.org
> 
> Disk /dev/md0 - 2000 GB / 1863 GiB - CHS 488379648 2 4
> Current partition structure:
>      Partition                  Start        End    Size in sectors
> 
>  1 P EFI GPT                  0   0  2 488379647   1  4 3907037183
> 
> Warning: Bad ending sector (CHS and LBA don't match)
> No partition is bootable
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> *=Primary bootable  P=Primary  L=Logical  E=Extended  D=Deleted
> 
> 
> use of testdisk generated the empty space included above.
> 
> I'm thinking the bad ending sector is possibly important.
> 
> Does this above mean anything to you?
> 
> Dee
> 
> gerry <[email-masked]>
> Sep 9
> Reply
> to me
> none of what is in that email points to problems.
> "warning:" -- is not necessarily a  problem.
> 
> When you send me output, make sure and include the command you ran to
> get that output.
> 
> Did you do
> testdisk /list
> testdisk /debug
>  ?
> 
> And none of that has anything to do with filesystems.
> The partition is not a filesystem.
> 
> With that said, I don't think it should show up as an EFI GPT
> partition so it MIGHT be a problem.
> 
> Have you done the
>    fsck.ext4 -f -n  /dev/md0
> yet?
> 
> 
> o1bigtenor <[email-masked]>
> Sep 10
> Reply
> to gerry
> On Wed, Sep 9, 2015 at 9:48 PM, gerry <[email-masked]> wrote:
>> none of what is in that email points to problems.
>> "warning:" -- is not necessarily a  problem.
>>
>> When you send me output, make sure and include the command you ran to get
>> that output.
> 
> Could not include the command because testdisk runs itself covering
> the command line.
> I set all of the output from running # testdisk
>>
>> Did you do
>> testdisk /list
> 
> # testdisk /list
> TestDisk 6.14, Data Recovery Utility, July 2013
> Christophe GRENIER <[email-masked]>
> http://www.cgsecurity.org
> Please wait...
> Disk /dev/sda - 1000 GB / 931 GiB - CHS 121601 255 63
> Sector size:512
> Model: ST1000DM003-1ER162, S/N:Z4Y4SPM8, FW:CC45
> 
> Disk /dev/sdb - 1000 GB / 931 GiB - CHS 121601 255 63
> Sector size:512
> Model: ST31000524AS, S/N:9VPE3VX1, FW:JC4B
> 
> Disk /dev/sdc - 1000 GB / 931 GiB - CHS 121601 255 63
> Sector size:512
> Model: ST1000DM003-1ER162, S/N:Z4Y27SPH, FW:CC45
> 
> Disk /dev/sdd - 240 GB / 223 GiB - CHS 29185 255 63
> Sector size:512
> Model: Corsair Force 3 SSD, S/N:11486509000011540460, FW:1.3.3
> 
> Disk /dev/sde - 1000 GB / 931 GiB - CHS 121601 255 63
> Sector size:512
> Model: ST31000524AS, S/N:9VPE44LL, FW:JC4B
> 
> Disk /dev/sdf - 1000 GB / 931 GiB - CHS 121601 255 63
> Sector size:512
> Model: ST31000524AS, S/N:9VPE31PN, FW:JC4B
> 
> Disk /dev/md0 - 2000 GB / 1863 GiB - CHS 488379648 2 4
> Sector size:512
> 
> 
> Disk /dev/sda - 1000 GB / 931 GiB - CHS 121601 255 63
>      Partition            Start        End    Size in sectors
>  1 P EFI System                  2048     526335     524288
>  2 P MS Data                   526336    1050623     524288 [/boot]
>  3 P MS Data                  1050624   52250623   51200000 [/]
>      ext4 blocksize=4096 Large file Sparse superblock Recover
>  4 P Linux Swap              52250624  183322623  131072000 [swap]
>  5 P Unknown                183322624  964573183  781250560
> 
> Disk /dev/sdb - 1000 GB / 931 GiB - CHS 121601 255 63
>      Partition            Start        End    Size in sectors
>  1 P Linux RAID               0  32 33 121601  57 56 1953521664 [debianbase:0]
>      md 1.x L.Endian Raid 10 - Array Slot : 3 (0, 1, failed, 3, 2,
> failed, failed)
> 
> Disk /dev/sdc - 1000 GB / 931 GiB - CHS 121601 255 63
>      Partition            Start        End    Size in sectors
>  1 P Unknown                     2048 1953525134 1953523087 [Linux filesystem]
> 
> Disk /dev/sdd - 240 GB / 223 GiB - CHS 29185 255 63
>      Partition            Start        End    Size in sectors
>  1 * Linux                    0  32 33    31  26 59     497664
>      ext2 blocksize=1024 Sparse superblock
>  4 E extended                31  59 29 14996 156  8  240418816
>  5 L Linux                   31  91 61  2581   0 43   40960000
>      ext4 blocksize=4096 Large file Sparse superblock Recover
>    X extended              2581   0 44  5130 196 58   40962048
>  6 L Linux                 2581  33 13  5130 196 58   40960000
>      ext4 blocksize=4096 Large file Sparse superblock Recover
>    X extended              5130 196 59  8955  92 33   61442048
>  7 L Linux                 5130 229 28  8955  92 33   61440000
>      ext4 blocksize=4096 Large file Sparse superblock Recover
>    X extended              8955  92 34  9465 106 49    8194048
>  8 L Linux                 8955 125  3  9465 106 49    8192000
>      ext4 blocksize=4096 Large file Sparse superblock Recover
>    X extended              9465 106 50 12015  48  1   40962048
>  9 L Linux                 9465 139 19 12015  48  1   40960000
>      ext4 blocksize=4096 Large file Sparse superblock
>    X extended             12015  48  2 14996 156  8   47896576
> 10 L Linux Swap           12015  80 34 14996 156  8   47894528
>      SWAP2 version 1, pagesize=4096
> 
> Disk /dev/sde - 1000 GB / 931 GiB - CHS 121601 255 63
>      Partition            Start        End    Size in sectors
>  1 P Linux RAID               0  32 33 121601  57 56 1953521664 [debianbase:0]
>      md 1.x L.Endian Raid 10 - Array Slot : 0 (0, 1, failed, 3, 2,
> failed, failed)
> 
> Disk /dev/sdf - 1000 GB / 931 GiB - CHS 121601 255 63
>      Partition            Start        End    Size in sectors
>  1 P Linux RAID               0  32 33 121601  57 56 1953521664 [debianbase:0]
>      md 1.x L.Endian Raid 10 - Array Slot : 1 (0, 1, failed, 3, 2,
> failed, failed)
> 
> Disk /dev/md0 - 2000 GB / 1863 GiB - CHS 488379648 2 4
>      Partition            Start        End    Size in sectors
>  1 P EFI GPT                  0   0  2 488379647   1  4 3907037183
> 
> 
> 
> 
>> testdisk /debug
> 
> This is after choosing md0 and a couple other screens.
> 
> TestDisk 6.14, Data Recovery Utility, July 2013
> Christophe GRENIER <[email-masked]>
> http://www.cgsecurity.org
> 
> Disk /dev/md0 - 2000 GB / 1863 GiB - CHS 488379648 2 4
> Current partition structure:
>      Partition                  Start        End    Size in sectors
> 
>  1 P EFI GPT                  0   0  2 488379647   1  4 3907037183
> 
> Warning: Bad ending sector (CHS and LBA don't match)
> No partition is bootable
> 
> 
> 
>>  ?
>>
>> And none of that has anything to do with filesystems.
>> The partition is not a filesystem.
>>
>> With that said, I don't think it should show up as an EFI GPT partition so
>> it MIGHT be a problem.
>>
>> Have you done the
>>    fsck.ext4 -f -n  /dev/md0
>> yet?
>>
> no
> Does this command alter anything but the filesystem info?
> 
> Thanks for the ideas!
> 
> Dee
> 
> gerry <[email-masked]>
> Sep 10
> Reply
> to me
> It botheres me that you saw scsi errors on sde, but the partition
> table for sdc is different.  hmmmm.
> 
> It's like something put a file system on it or at least partitioned it.
> (And it's always been like that -- an earlier email looked the same)
> 
> Getting caught back up here I have some questions.
> 1. Have you been rebooting this machine?
> 2. Have you figured out which physical disk is associated with each devices?
> 3. What device did you put that backup on?
> 
> repeating -- smartctl can show you the serial numbers.
> 
> As far as recovery goes, I'm unsure.
> 
> My approach would be to fail two of the disks and then do a filesystem
> recovery on the two that are left.
> Earlier you sent me this info:
>             /dev/sdb1   f1c9645dbc14efddc7d8a322685f26eb  -
>             /dev/sdc1   f1c9645dbc14efddc7d8a322685f26eb  -
>             /dev/sde1   979c1a460f2130d7fd0e9210e2d15e90  -
>             /dev/sdf1   979c1a460f2130d7fd0e9210e2d15e90  -
> 
> So if you were to "fail" sdc1 and sde1, then you should end up with a
> simple stripe with no mirroring.
> 
> Then if you ran fsck.ext4 -f -n /dev/md0
> it should tell you what it thinks is wrong with the filesystem.
> 
> You asked:
> Does this command alter anything but the filesystem info?
> 
> That command (with the -n) should not alter the filesystem info.
> -n is a read-only filesystem check.
> 
> Gerry
> 
> 
> --
> [email-masked]
> 
> 
> o1bigtenor <[email-masked]>
> Sep 10
> Reply
> to gerry
> Greetings
> 
> On Thu, Sep 10, 2015 at 12:06 PM, gerry <[email-masked]> wrote:
>> It botheres me that you saw scsi errors on sde, but the partition table for
>> sdc is different.  hmmmm.
>>
>> It's like something put a file system on it or at least partitioned it.
>> (And it's always been like that -- an earlier email looked the same)
>>
>> Getting caught back up here I have some questions.
>> 1. Have you been rebooting this machine?
> 
> No - - - I suspend to ram (sleep) most evenings and then just start
> (for the lack
> of a better word) it up in the mornings.
> 
>> 2. Have you figured out which physical disk is associated with each devices?
> 
> Not yet but that would be just a matter of a few minutes with serial numbers in
> hand.
> 
>> 3. What device did you put that backup on?
> 
> The backup was put onto a hdd that I removed from a NAS box that I had just
> setup (its a 3 TB drive) so its connected via USB to the main system and is
> most often not turned on.
> 
>>
>> repeating -- smartctl can show you the serial numbers.
>>
>> As far as recovery goes, I'm unsure.
>>
>> My approach would be to fail two of the disks and then do a filesystem
>> recovery on the two that are left.
>> Earlier you sent me this info:
>>             /dev/sdb1   f1c9645dbc14efddc7d8a322685f26eb  -
>>             /dev/sdc1   f1c9645dbc14efddc7d8a322685f26eb  -
>>             /dev/sde1   979c1a460f2130d7fd0e9210e2d15e90  -
>>             /dev/sdf1   979c1a460f2130d7fd0e9210e2d15e90  -
>>
>> So if you were to "fail" sdc1 and sde1, then you should end up with a simple
>> stripe with no mirroring.
>>
>> Then if you ran fsck.ext4 -f -n /dev/md0
>> it should tell you what it thinks is wrong with the filesystem.
>>
>> You asked:
>>>
>>> Does this command alter anything but the filesystem info?
>>
>>
>> That command (with the -n) should not alter the filesystem info.
>> -n is a read-only filesystem check.
>>
> 
> Just so I understand you I will reiterate:
> 
> 1. determine hdb, hdc, hde, hdf
> 2. shut off machine
> 3. remove hdc and hde (by unplugging the drives)
> 4. restart machine
> 5. hopefully file system is now repairable.
> 
> 
> I have been able to recover files off of the array so it seems that the
> files have not been damaged (as yet).
> 
> If I have not understood correctly - - - please correct the
> incorrect items!!
> 
> Thanking you for your assistance!!!
> 
> Dee
> 
> o1bigtenor <[email-masked]>
> Sep 10
> Reply
> to gerry
> On Thu, Sep 10, 2015 at 12:06 PM, gerry <[email-masked]> wrote:
> # fsck.ext4 -f -n /dev/md0
> e2fsck 1.42.13 (17-May-2015)
> ext2fs_open2: Bad magic number in super-block
> fsck.ext4: Superblock invalid, trying backup blocks...
> fsck.ext4: Bad magic number in super-block while trying to open /dev/md0
> 
> The superblock could not be read or does not describe a valid ext2/ext3/ext4
> filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
> filesystem (and not swap or ufs or something else), then the superblock
> is corrupt, and you might try running e2fsck with an alternate superblock:
>     e2fsck -b 8193 <device>
>  or
>     e2fsck -b 32768 <device>
> 
> 
> Trying both (done previously)
> 
> #e2fsck -b 8193 /dev/md0
> 
> and
> 
> #e2fsck -b 32768 /dev/md0
> 
> gives the same information as did the fsck.ext4 -f -n /dev/md0
> 
> 
> it seems that the array is fine but the file system (table) or some such
> has been corrupted.
> 
> This has been a very non-fun event over the last number of days.
> 
> Thanking you for your assistance
> 
> Dee
> 
> gerry <[email-masked]>
> Sep 10
> Reply
> to me
> Okay you have me totally confused now.
> What did you mean by this:
> 
> I have been able to recover files off of the array so it seems that the
> files have not been damaged (as yet).
> 
> What array?  Not /dev/md0 certainly.... ????
> is /dev/md0 mounted?
> 
> 
> o1bigtenor <[email-masked]>
> Sep 10
> Reply
> to gerry
> On Thu, Sep 10, 2015 at 4:51 PM, gerry <[email-masked]> wrote:
>> Okay you have me totally confused now.
>> What did you mean by this:
>>
>>> I have been able to recover files off of the array so it seems that the
>>> files have not been damaged (as yet).
>>
>>
>> What array?  Not /dev/md0 certainly.... ????
>> is /dev/md0 mounted?
>>
>>
> The command
> #vi /etc/fstab produces this:
> even though I have not mounted the array (at least I don't think I have).
> 
> I get it - - - there is weird stuff going on - - - just wish I
> understood this stuff a little better!!!
> 
> Dee
> 
> gerry <[email-masked]>
> Sep 10
> Reply
> to me
> The /etc/fstab file is just a list of things that can be mounted.
> What do you mean by "recover filess off of the array"
> ?
> Do you mean you get something at /media/darald/Raid10 ?
> 
> PTUUID=1eb47793-0cdf-4e16-ae84-33ec825ac448 /media/darald/Raid10
> 
> You can do an ls of that location?
> 
> What do you get when you type 'mount' all by itself?
>    (It should list everything mounted.)
> 
> Dang. I should have noticed that that was the raid10.
> But maybe this is the first time I've seen the fstab entry.
> 
> How would it get mounted?
> Does that mountpoint exist?
> 
> 
> o1bigtenor <[email-masked]>
> Sep 10
> Reply
> to gerry
> On Thu, Sep 10, 2015 at 5:23 PM, gerry <[email-masked]> wrote:
>> The /etc/fstab file is just a list of things that can be mounted.
>> What do you mean by "recover filess off of the array"
> 
> program called photorec (its with testdisk) . It was finding all kinds
> of files but nothing is organized it all just raw files.
>> ?
>> Do you mean you get something at /media/darald/Raid10 ?
>>
>>> PTUUID=1eb47793-0cdf-4e16-ae84-33ec825ac448 /media/darald/Raid10
>>
>>
>> You can do an ls of that location?
> 
> # ls /media/darald/Raid10/
> #
> 
> 
>>
>> What do you get when you type 'mount' all by itself?
>>    (It should list everything mounted.)
> 
> # mount
> sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
> proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
> udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=8242416,mode=755)
> devpts on /dev/pts type devpts
> (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
> tmpfs on /run type tmpfs (rw,nosuid,relatime,size=13191704k,mode=755)
> /dev/sda3 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
> securityfs on /sys/kernel/security type securityfs
> (rw,nosuid,nodev,noexec,relatime)
> tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
> tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
> tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
> cgroup on /sys/fs/cgroup/systemd type cgroup
> (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
> pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
> efivarfs on /sys/firmware/efi/efivars type efivarfs
> (rw,nosuid,nodev,noexec,relatime)
> cgroup on /sys/fs/cgroup/blkio type cgroup
> (rw,nosuid,nodev,noexec,relatime,blkio)
> cgroup on /sys/fs/cgroup/freezer type cgroup
> (rw,nosuid,nodev,noexec,relatime,freezer)
> cgroup on /sys/fs/cgroup/cpuset type cgroup
> (rw,nosuid,nodev,noexec,relatime,cpuset)
> cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup
> (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
> cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup
> (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
> cgroup on /sys/fs/cgroup/devices type cgroup
> (rw,nosuid,nodev,noexec,relatime,devices)
> cgroup on /sys/fs/cgroup/perf_event type cgroup
> (rw,nosuid,nodev,noexec,relatime,perf_event)
> systemd-1 on /proc/sys/fs/binfmt_misc type autofs
> (rw,relatime,fd=22,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
> mqueue on /dev/mqueue type mqueue (rw,relatime)
> debugfs on /sys/kernel/debug type debugfs (rw,relatime)
> hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
> /dev/sda5 on /home type ext4 (rw,relatime,data=ordered)
> /dev/sda1 on /boot/efi type vfat
> (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=utf8,shortname=mixed,errors=remount-ro)
> rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
> tmpfs on /run/user/117 type tmpfs
> (rw,nosuid,nodev,relatime,size=6595852k,mode=700,uid=117,gid=124)
> tmpfs on /run/user/1000 type tmpfs
> (rw,nosuid,nodev,relatime,size=6595852k,mode=700,uid=1000,gid=1000)
> fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
> gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse
> (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
> binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
> 
>>
>> Dang. I should have noticed that that was the raid10.
>> But maybe this is the first time I've seen the fstab entry.
>>
>> How would it get mounted?
> 
> No idea!
> 
>> Does that mountpoint exist?
> 
> Yes. (folder exists but there is nothing there!!)
> 
> 
> I'm not trying to drive you crazy but this kind of here and not here
> stuff is what I really don't get!!
> 
> Rund (sic) to you.
> 
> Dee
> 
> gerry <[email-masked]>
> Sep 10
> Reply
> to me
> Ahhh.
> Now I understand.
> It isn't mounted. But now I wonder why.
> There are probably some errors in the log about it.
> 
> Yeah. photorec will do that -- just make raw files. It's pretty easy
> to figure out what kind of file they are but not a lot of other stuff
> about them. :-(
> 
> So. The question is -- do you run photorec all the way through, or do
> you try forcing fsck to do something.
> 
> I'd definitely do more with fsck.ext4 because that won't affect photorec at all.
> 
> 
> o1bigtenor <[email-masked]>
> Sep 10
> Reply
> to gerry
> On Thu, Sep 10, 2015 at 6:41 PM, gerry <[email-masked]> wrote:
>> Ahhh.
>> Now I understand.
>> It isn't mounted. But now I wonder why.
>> There are probably some errors in the log about it.
>>
>> Yeah. photorec will do that -- just make raw files. It's pretty easy to
>> figure out what kind of file they are but not a lot of other stuff about
>> them. :-(
>>
>> So. The question is -- do you run photorec all the way through, or do you
>> try forcing fsck to do something.
> 
> Photorec ran until I ran out of room on the drive - - - my bad - - -
> thought I would have enough room but not the case.
> 
> If I'm going to do that I need to get where the backup is on the 3 TB
> drive - - - there's lots of room there! (grin!)
>>
>> I'd definitely do more with fsck.ext4 because that won't affect photorec at
>> all.
> 
> Waiting for your commands, kind sir!
> 
> Dee
> 
> gerry <[email-masked]>
> Sep 10
> Reply
> to me
> hahaha.
> My commands.
> 
> Are there particular files that you need?  Perhaps you already have them?
> 
> 
> You tried a couple of alternate superblocks.
> Perhaps you should try more.
> But to figure out what they are you'll have to pretend to format the disk again.
> But NOT write to the disk.
> 
> Reading the mke2fs man page:
> 
>  -n  Causes mke2fs to not actually create a filesystem, but display
> what it would do if
>      it were to create a filesystem.  This can be used to determine
> the location of the
>      backup superblocks for a particular filesystem, so long as the
> mke2fs  parameters
>      that were passed when the filesystem was originally created are
> used again.  (With
>      the -n option added, of course!)
> 
> so try
> mke2fs -t ext4 -n /dev/md0
> 
> And then try those superblock numbers while running
> fsck.ext4 -b XXXXXXX  /dev/md0
> (replacing XXXXX with those potential superblocks)
> 
> Also, could you send the output of cat /etc/mdstat again?
> I just want to know if it's changed since the reboot.
> 
> Gerry
> 
> 
> Reading material:
> http://askubuntu.com/questions/240372/how-to-recover-data-from-ext4-partitions
> http://www.cyberciti.biz/faq/linux-find-alternative-superblocks/
> http://askubuntu.com/questions/41601/is-there-any-recovery-software-available-for-ext4
> http://ext4magic.sourceforge.net/ext4magic_en.html
> 
> 
> o1bigtenor <[email-masked]>
> Sep 11
> Reply
> to gerry
> On Thu, Sep 10, 2015 at 11:07 PM, gerry <[email-masked]> wrote:
>> hahaha.
>> My commands.
>>
>> Are there particular files that you need?  Perhaps you already have them?
> 
> It would particular folders. There is at least one folder - - my
> business files!!
>>
>>
>> You tried a couple of alternate superblocks.
>> Perhaps you should try more.
>> But to figure out what they are you'll have to pretend to format the disk
>> again.
>> But NOT write to the disk.
>>
>> Reading the mke2fs man page:
>>
>>  -n  Causes mke2fs to not actually create a filesystem, but display what it
>> would do if
>>      it were to create a filesystem.  This can be used to determine the
>> location of the
>>      backup superblocks for a particular filesystem, so long as the  mke2fs
>> parameters
>>      that were passed when the filesystem was originally created are used
>> again.  (With
>>      the -n option added, of course!)
>>
>> so try
>> mke2fs -t ext4 -n /dev/md0
> 
> /# mke2fs -t ext4 -n /dev/md0
> mke2fs 1.42.13 (17-May-2015)
> Found a gpt partition table in /dev/md0
> Proceed anyway? (y,n) y
> Creating filesystem with 488379648 4k blocks and 122101760 inodes
> Filesystem UUID: 33081d9b-8ec3-4382-9922-8ca7d6a7050b
> Superblock backups stored on blocks:
>     32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
>     4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
>     102400000, 214990848
> 
> 
>>
>> And then try those superblock numbers while running
>> fsck.ext4 -b XXXXXXX  /dev/md0
>> (replacing XXXXX with those potential superblocks)
> 
> Tried them ALL - - - got
> 
> # fsck.ext4 -b 214990848 /dev/md0
> e2fsck 1.42.13 (17-May-2015)
> fsck.ext4: Invalid argument while trying to open /dev/md0
> 
> The superblock could not be read or does not describe a valid ext2/ext3/ext4
> filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
> filesystem (and not swap or ufs or something else), then the superblock
> is corrupt, and you might try running e2fsck with an alternate superblock:
>     e2fsck -b 8193 <device>
>  or
>     e2fsck -b 32768 <device>
> 
> 
>>
>> Also, could you send the output of cat /etc/mdstat again?
> 
> # cat /etc/mdstat
> cat: /etc/mdstat: No such file or directory
> 
> 
>> I just want to know if it's changed since the reboot.
> 
> I have not yet gone through the steps that you had asked in the previous email.
> 
> Were those to have been done before I went through this excercise?
> 
> Over to you.
> 
> Dee
> 
> gerry <[email-masked]>
> Sep 11
> Reply
> to me
> Since
> 1. it thinks there's a partition table on /dev/md0
> 2. it can't find useable superblocks ?
> 
> I'm starting to think that the only possible recovery is
> the photorec path. :-(
> 
> Ooooops. I meant
>   cat /proc/mdstat
> Not /etc/mdstat.
> 
> Ger
> 
> --
> [email-masked]
> 
> 
> o1bigtenor <[email-masked]>
> Sep 11
> Reply
> to gerry
> On Fri, Sep 11, 2015 at 9:28 AM, gerry <[email-masked]> wrote:
>> Since
>> 1. it thinks there's a partition table on /dev/md0
>> 2. it can't find useable superblocks ?
>>
>> I'm starting to think that the only possible recovery is
>> the photorec path. :-(
> 
> When I looked into one of the folders that photrec provided there was
> piles of useless stuff in there.
> 
> So you don't believe that there is a way to pull folders rather than just files?
>>
>> Ooooops. I meant
>>   cat /proc/mdstat
>> Not /etc/mdstat.
> 
> # cat /proc/mdstat
> Personalities : [raid10]
> md0 : active (auto-read-only) raid10 sde1[0] sdb1[3] sdc1[4] sdf1[1]
>       1953518592 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
> 
> With a heavy heart
> 
> Dee
> 
> gerry <[email-masked]>
> Sep 11
> Reply
> to me
> I don't believe there is.
> What do you mean by "useless stuff"
> Just numbered files, eh?
> What do you get when you run the
> file *
> command in the directory.
> 
> I see 3 disks. So which one did you pull out?
> And how did you figure out which one to pull out?
> And what does parted -l say now?
> 
> 
> o1bigtenor <[email-masked]>
> Sep 11
> Reply
> to gerry
> On Fri, Sep 11, 2015 at 10:43 AM, gerry <[email-masked]> wrote:
>> I don't believe there is.
>> What do you mean by "useless stuff"
>> Just numbered files, eh?
>> What do you get when you run the
>> file *
>> command in the directory.
>>
>> I see 3 disks. So which one did you pull out?
> 
> Sorry - - - I still se four disks listed.
> 
> 
> I was having internet issues and so I did a system reboot.
> 
> Well - -- I can go in under system repair and use startx to get to a
> working system but cannot log in as user.
> 
> Checking in /etc/fstab
> 
> and I have to type this cause I'm using a different computer
> 
> # / was on /dev/sda3 during installation
> UUID=a3fbea19-5b7d-47af-b06c-08c49171211b /      ext4
> errors=remount-ro  0     1
> 
> so now I have even more issues to play with.
> 
> Any suggestions on how to proceed?
> 
> dee
> 
> gerry <[email-masked]>
> Sep 11
> Reply
> to me
> Yeah. 4. disks.
> So you never excluded any of them and then you started trying to do
> destructive changes.
> 
> I don't know why you sent me the fstab file entry.
> 
> If you are using startx, then you logged in as a user.
> 
> You have lost me -- I don't know what the email below means.
> 
> If you want your data back you'll have to use the photorec tools and
> get what you can but there will be no recovery of directories.
> 
> And you'll have to figure out what each of the  numbered files from a
> photorec recovery are.  If you are "business" files are of a
> particular type, you'll want to make sure that photorec knows about
> that type.
> 
> 
> o1bigtenor <[email-masked]>
> Sep 11
> Reply
> to gerry
> On Fri, Sep 11, 2015 at 11:47 AM, gerry <[email-masked]> wrote:
>> Yeah. 4. disks.
>> So you never excluded any of them and then you started trying to do
>> destructive changes.
> 
> I understood that the
> 
> fsck ext4 ...
> 
> was non descructive only looking for system file sectors
> 
> Sorry
> 
> Was that incorrect.
> 
>>
>> I don't know why you sent me the fstab file entry.
> 
> I sent you that fstab file because it says that
> sda3 /
> 
> is not mounted.
> 
> How do I repair that?
> 
>>
>> If you are using startx, then you logged in as a user.
>>
> System repair says
> 
> give root password for log in or to continue
> 
> give password
> 
> then do
> # startx
> 
> and I have Xwindow session up but I'm working as root - - -
> 
> cannot get to where I would log in as a user.
> 
> I am starting to wonder about Debian - - - the reliability is
> driving me totally crazy!!!
> 
> gerry <[email-masked]>
> Sep 11
> Reply
> to me
> the fstab file has nothing to do with what is actually mounted.
> Use the mount command for that.
> 
> fsck is destructive.
> Unless there's a "-n" option to make it not so.
> 
> 
> And a third group (two sets here):
> 
> o1bigtenor <[email-masked]>
> Sep 8
> Reply
> to Maurice
> Greetings
> 
> Just in case my email address isn't handy.
> 
> Regards
> 
> 
> maurice <[email-masked]>
> Sep 8
> Reply
> to me
> Hi D
> 
> We need the output from:
> 
> cat /proc/mdstat
> 
> Post your array details (mdadm -D), member
> details (mdadm -E) and a summary of what happened.
> Include any excerpts from your dmesg and/or syslogs that look like
> they might be relevant.
> 
> 
> mdadm --detail /dev/mdX
>  - Where "X" is the RAID md device #
> 
> mdadm --examine /dev/sdXy
>  - Where "X" is the RAID md device # and
>  - Where "y" is each partition being used in that RAID on device "X"
> 
> Also, I suggest using lsdrv for listing the  working drive layouts.
> Output of that is also useful.
> You may find the lsdrv script here:
> 
> https://github.com/pturmel/lsdrv
> 
> 
> 
> 
> o1bigtenor <[email-masked]>
> Sep 8
> Reply
> to maurice
> On Tue, Sep 8, 2015 at 6:10 PM, maurice <[email-masked]> wrote:
>> Hi Darald
>>
>> We need the output from:
>>
>> cat /proc/mdstat
> 
> # cat /proc/mdstat
> Personalities : [raid10]
> md0 : active (auto-read-only) raid10 sde1[0] sdb1[3] sdc1[4] sdf1[1]
>       1953518592 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
> 
> unused devices: <none>
> 
> 
>>
>> Post your array details (mdadm -D),
> 
> # mdadm -D
> mdadm: No devices given.
> 
> 
> 
>> member
>> details (mdadm -E)
> 
> # mdadm -E
> mdadm: No devices to examine
> 
> 
>> and a summary of what happened.
>> Include any excerpts from your dmesg and/or syslogs that look like they
>> might be relevant.
> 
> Aug 28 10:39:28 debiantestingbase kernel: [    3.025162] scsi 3:0:0:0:
> Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
> Aug 28 10:39:28 debiantestingbase kernel: [    3.025768] sd 3:0:0:0:
> [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
> Aug 28 10:39:28 debiantestingbase kernel: [    3.026296] sd 3:0:0:0:
> [sdb] Write Protect is off
> Aug 28 10:39:28 debiantestingbase kernel: [    3.026304] sd 3:0:0:0:
> [sdb] Mode Sense: 00 3a 00 00
> Aug 28 10:39:28 debiantestingbase kernel: [    3.026484] sd 3:0:0:0:
> [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO
> or FUA
> Aug 28 10:39:28 debiantestingbase kernel: [    3.028273]  sdb: sdb1
> Aug 28 10:39:28 debiantestingbase kernel: [    3.028698] sd 3:0:0:0:
> [sdb] Attached SCSI disk
> Aug 28 10:39:28 debiantestingbase kernel: [    3.086532] Switched to
> clocksource tsc
> Aug 28 10:39:28 debiantestingbase kernel: [    3.168919] md: bind<sdb1>
> Aug 28 10:39:28 debiantestingbase kernel: [    3.342285] ata5: SATA
> link up 3.0 Gbps (SStatus 123 SControl 300)
> Aug 28 10:39:28 debiantestingbase kernel: [    3.343153] ata5.00:
> ATA-9: ST1000DM003-1ER162, CC45, max UDMA/133
> Aug 28 10:39:28 debiantestingbase kernel: [    3.343158] ata5.00:
> 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
> Aug 28 10:39:28 debiantestingbase kernel: [    3.344067] ata5.00:
> configured for UDMA/133
> Aug 28 10:39:28 debiantestingbase kernel: [    3.344255] scsi 4:0:0:0:
> Direct-Access     ATA      ST1000DM003-1ER1 CC45 PQ: 0 ANSI: 5
> Aug 28 10:39:28 debiantestingbase kernel: [    3.344627] sd 4:0:0:0:
> [sdc] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
> Aug 28 10:39:28 debiantestingbase kernel: [    3.344631] sd 4:0:0:0:
> [sdc] 4096-byte physical blocks
> Aug 28 10:39:28 debiantestingbase kernel: [    3.344823] sd 4:0:0:0:
> [sdc] Write Protect is off
> Aug 28 10:39:28 debiantestingbase kernel: [    3.344831] sd 4:0:0:0:
> [sdc] Mode Sense: 00 3a 00 00
> Aug 28 10:39:28 debiantestingbase kernel: [    3.344946] sd 4:0:0:0:
> [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO
> or FUA
> Aug 28 10:39:28 debiantestingbase kernel: [    3.411364]  sdc: sdc1
> Aug 28 10:39:28 debiantestingbase kernel: [    3.412317] sd 4:0:0:0:
> [sdc] Attached SCSI disk
> Aug 28 10:39:28 debiantestingbase kernel: [    3.501080] md: bind<sdc1>
> Aug 28 10:39:28 debiantestingbase kernel: [    3.662509] ata6: SATA
> link up 3.0 Gbps (SStatus 123 SControl 300)
> Aug 28 10:39:28 debiantestingbase kernel: [    3.674767] ata6.00:
> ATA-8: Corsair Force 3 SSD, 1.3.3, max UDMA/133
> Aug 28 10:39:28 debiantestingbase kernel: [    3.674772] ata6.00:
> 468862128 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
> Aug 28 10:39:28 debiantestingbase kernel: [    3.684647] ata6.00:
> configured for UDMA/133
> Aug 28 10:39:28 debiantestingbase kernel: [    3.684933] scsi 5:0:0:0:
> Direct-Access     ATA      Corsair Force 3  3    PQ: 0 ANSI: 5
> Aug 28 10:39:28 debiantestingbase kernel: [    3.685504] sd 5:0:0:0:
> [sdd] 468862128 512-byte logical blocks: (240 GB/223 GiB)
> Aug 28 10:39:28 debiantestingbase kernel: [    3.685975] sd 5:0:0:0:
> [sdd] Write Protect is off
> Aug 28 10:39:28 debiantestingbase kernel: [    3.685983] sd 5:0:0:0:
> [sdd] Mode Sense: 00 3a 00 00
> Aug 28 10:39:28 debiantestingbase kernel: [    3.686186] sd 5:0:0:0:
> [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO
> or FUA
> Aug 28 10:39:28 debiantestingbase kernel: [    3.688051]  sdd: sdd1
> sdd4 < sdd5 sdd6 sdd7 sdd8 sdd9 sdd10 >
> Aug 28 10:39:28 debiantestingbase kernel: [    3.689305] sd 5:0:0:0:
> [sdd] Attached SCSI disk
> Aug 28 10:39:28 debiantestingbase kernel: [    4.002712] ata8: SATA
> link down (SStatus 0 SControl 300)
> Aug 28 10:39:28 debiantestingbase kernel: [    4.003107] scsi 8:0:0:0:
> Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
> Aug 28 10:39:28 debiantestingbase kernel: [    4.003597] sd 8:0:0:0:
> [sde] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
> Aug 28 10:39:28 debiantestingbase kernel: [    4.003843] scsi 9:0:0:0:
> Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
> Aug 28 10:39:28 debiantestingbase kernel: [    4.003975] sd 8:0:0:0:
> [sde] Write Protect is off
> Aug 28 10:39:28 debiantestingbase kernel: [    4.003980] sd 8:0:0:0:
> [sde] Mode Sense: 00 3a 00 00
> Aug 28 10:39:28 debiantestingbase kernel: [    4.004090] sd 8:0:0:0:
> [sde] Write cache: enabled, read cache: enabled, doesn't support DPO
> or FUA
> Aug 28 10:39:28 debiantestingbase kernel: [    4.004478] sd 9:0:0:0:
> [sdf] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
> Aug 28 10:39:28 debiantestingbase kernel: [    4.004645] sd 9:0:0:0:
> [sdf] Write Protect is off
> Aug 28 10:39:28 debiantestingbase kernel: [    4.004650] sd 9:0:0:0:
> [sdf] Mode Sense: 00 3a 00 00
> Aug 28 10:39:28 debiantestingbase kernel: [    4.004737] sd 9:0:0:0:
> [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO
> or FUA
> Aug 28 10:39:28 debiantestingbase kernel: [    4.004778] scsi
> 15:0:0:0: Processor         Marvell  91xx Config      1.01 PQ: 0 ANSI:
> 5
> Aug 28 10:39:28 debiantestingbase kernel: [    4.006375]  sdf: sdf1
> Aug 28 10:39:28 debiantestingbase kernel: [    4.006967] sd 9:0:0:0:
> [sdf] Attached SCSI disk
> Aug 28 10:39:28 debiantestingbase kernel: [    4.008855]  sde: sde1
> Aug 28 10:39:28 debiantestingbase kernel: [    4.009704] sd 8:0:0:0:
> [sde] Attached SCSI disk
> Aug 28 10:39:28 debiantestingbase kernel: [    4.018710] ata16.00:
> exception Emask 0x1 SAct 0x0 SErr 0x0 action 0x0
> Aug 28 10:39:28 debiantestingbase kernel: [    4.018753] ata16.00:
> irq_stat 0x40000001
> Aug 28 10:39:28 debiantestingbase kernel: [    4.018783] ata16.00: cmd
> a0/01:00:00:00:01/00:00:00:00:00/a0 tag 1 dma 16640 in
> Aug 28 10:39:28 debiantestingbase kernel: [    4.018783]
> Inquiry 12 01 00 00 ff 00res 50/00:00:af:6d:70/00:00:74:00:00/e0 Emask
> 0x1 (device error)
> Aug 28 10:39:28 debiantestingbase kernel: [    4.018868] ata16.00:
> status: { DRDY }
> Aug 28 10:39:28 debiantestingbase kernel: [    4.125325] random:
> nonblocking pool is initialized
> Aug 28 10:39:28 debiantestingbase kernel: [    4.125530] md: bind<sde1>
> Aug 28 10:39:28 debiantestingbase kernel: [    4.142140] md: bind<sdf1>
> Aug 28 10:39:28 debiantestingbase kernel: [    4.144984] md: raid10
> personality registered for level 10
> Aug 28 10:39:28 debiantestingbase kernel: [    4.145397]
> md/raid10:md0: active with 4 out of 4 devices
> Aug 28 10:39:28 debiantestingbase kernel: [    4.145440] md0: detected
> capacity change from 0 to 2000403038208
> Aug 28 10:39:28 debiantestingbase kernel: [    4.208978]  md0:
> Aug 28 10:39:28 debiantestingbase kernel: [    4.479305]
> device-mapper: uevent: version 1.0.3
> Aug 28 10:39:28 debiantestingbase kernel: [    4.479536]
> device-mapper: ioctl: 4.30.0-ioctl (2014-12-22) initialised:
> [email-masked]
> 
>>
>>
>> mdadm --detail /dev/mdX
>>  - Where "X" is the RAID md device #
> 
> # mdadm --detail /dev/md0
> /dev/md0:
>         Version : 1.2
>   Creation Time : Mon Mar  5 08:26:28 2012
>      Raid Level : raid10
>      Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
>   Used Dev Size : 976759296 (931.51 GiB 1000.20 GB)
>    Raid Devices : 4
>   Total Devices : 4
>     Persistence : Superblock is persistent
> 
>     Update Time : Fri Aug 28 10:38:32 2015
>           State : clean
>  Active Devices : 4
> Working Devices : 4
>  Failed Devices : 0
>   Spare Devices : 0
> 
>          Layout : near=2
>      Chunk Size : 512K
> 
>            Name : debianbase:0
>            UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
>          Events : 47341
> 
>     Number   Major   Minor   RaidDevice State
>        0       8       65        0      active sync set-A   /dev/sde1
>        1       8       81        1      active sync set-B   /dev/sdf1
>        4       8       33        2      active sync set-A   /dev/sdc1
>        3       8       17        3      active sync set-B   /dev/sdb1
> 
> 
>>
>> mdadm --examine /dev/sdXy
>>  - Where "X" is the RAID md device # and
>>  - Where "y" is each partition being used in that RAID on device "X"
> 
> # mdadm --examine /dev/sdb1
> /dev/sdb1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
>            Name : debianbase:0
>   Creation Time : Mon Mar  5 08:26:28 2012
>      Raid Level : raid10
>    Raid Devices : 4
> 
>  Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
>      Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
>   Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=1968 sectors, after=1024 sectors
>           State : clean
>     Device UUID : a80c76db:eaea61af:bcb9cbbb:ac99e467
> 
>     Update Time : Fri Aug 28 10:38:32 2015
>        Checksum : 29a4fa98 - correct
>          Events : 47341
> 
>          Layout : near=2
>      Chunk Size : 512K
> 
>    Device Role : Active device 3
>    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
> 
> # mdadm --examine /dev/sdc1
> /dev/sdc1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
>            Name : debianbase:0
>   Creation Time : Mon Mar  5 08:26:28 2012
>      Raid Level : raid10
>    Raid Devices : 4
> 
>  Avail Dev Size : 1953521039 (931.51 GiB 1000.20 GB)
>      Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
>   Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=1960 sectors, after=2447 sectors
>           State : clean
>     Device UUID : 840fc99d:6f607ea8:9a4a9b0a:a965c85c
> 
>     Update Time : Fri Aug 28 10:38:32 2015
>   Bad Block Log : 512 entries available at offset 72 sectors
>        Checksum : 28cfd5ab - correct
>          Events : 47341
> 
>          Layout : near=2
>      Chunk Size : 512K
> 
>    Device Role : Active device 2
>    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
> root@debiantestingbase:/etc# mdadm --examine /dev/sde1
> /dev/sde1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
>            Name : debianbase:0
>   Creation Time : Mon Mar  5 08:26:28 2012
>      Raid Level : raid10
>    Raid Devices : 4
> 
>  Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
>      Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
>   Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=1968 sectors, after=1024 sectors
>           State : clean
>     Device UUID : cb37e722:d3a98f88:13439de2:bdad9d2b
> 
>     Update Time : Fri Aug 28 10:38:32 2015
>        Checksum : 34ce8208 - correct
>          Events : 47341
> 
>          Layout : near=2
>      Chunk Size : 512K
> 
>    Device Role : Active device 0
>    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
> root@debiantestingbase:/etc# mdadm --examine /dev/sdf1
> /dev/sdf1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
>            Name : debianbase:0
>   Creation Time : Mon Mar  5 08:26:28 2012
>      Raid Level : raid10
>    Raid Devices : 4
> 
>  Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
>      Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
>   Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=1968 sectors, after=1024 sectors
>           State : clean
>     Device UUID : 9e749fa9:a0efe791:ea09d2e2:72b99f6c
> 
>     Update Time : Fri Aug 28 10:38:32 2015
>        Checksum : 615d736 - correct
>          Events : 47341
> 
>          Layout : near=2
>      Chunk Size : 512K
> 
>    Device Role : Active device 1
>    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
> 
> 
> 
> 
>>
>> Also, I suggest using lsdrv for listing the  working drive layouts.
>> Output of that is also useful.
>> You may find the lsdrv script here:
>>
>> https://github.com/pturmel/lsdrv
>>
> 
> 
> # ./lsdrv
> PCI [ahci] 00:1f.2 SATA controller: Intel Corporation C600/X79 series
> chipset 6-Port SATA AHCI Controller (rev 05)
> ├scsi 0:0:0:0 HL-DT-ST BD-RE  BH12LS38  {K9HBA7I5354}
> │└sr0 1.00g [11:0] Empty/Unknown
> ├scsi 1:x:x:x [Empty]
> ├scsi 2:0:0:0 ATA      ST1000DM003-1ER1 {Z4Y4SPM8}
> │└sda 931.51g [8:0] Partitioned (gpt)
> │ ├sda1 256.00m [8:1] vfat {8642-E734}
> │ │└Mounted as /dev/sda1 @ /boot/efi
> │ ├sda2 256.00m [8:2] Empty/Unknown
> │ ├sda3 24.41g [8:3] ext4 {a3fbea19-5b7d-47af-b06c-08c49171211b}
> │ │└Mounted as /dev/sda3 @ /
> │ ├sda4 62.50g [8:4] swap {354a6e33-5743-4c97-bfb9-fd6f9d4521e6}
> │ └sda5 372.53g [8:5] ext4 {b36dbfc6-6897-4311-aa87-9a958ccad072}
> │  └Mounted as /dev/sda5 @ /home
> ├scsi 3:0:0:0 ATA      ST31000524AS     {9VPE3VX1}
> │└sdb 931.51g [8:16] Partitioned (dos)
> │ └sdb1 931.51g [8:17] MD raid10,near2 (3/4) (w/ sdc1,sde1,sdf1)
> in_sync 'debianbase:0' {79baaa2f-0aa2-b9fa-18e2-ea6b6e2846b3}
> │  └md0 1.82t [9:0] MD v1.2 raid10,near2 (4) read-auto, 512k Chunk
> {79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3}
> │                   Partitioned (gpt)
> ├scsi 4:0:0:0 ATA      ST1000DM003-1ER1 {Z4Y27SPH}
> │└sdc 931.51g [8:32] Partitioned (gpt)
> │ └sdc1 931.51g [8:33] MD raid10,near2 (2/4) (w/ sdb1,sde1,sdf1)
> in_sync 'debianbase:0' {79baaa2f-0aa2-b9fa-18e2-ea6b6e2846b3}
> │  └md0 1.82t [9:0] MD v1.2 raid10,near2 (4) read-auto, 512k Chunk
> {79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3}
> │                   Partitioned (gpt)
> └scsi 5:0:0:0 ATA      Corsair Force 3  {11486509000011540460}
>  └sdd 223.57g [8:48] Partitioned (dos)
>   ├sdd1 243.00m [8:49] ext2 {a0ce336b-35cd-4413-a1c7-e77a1a4e91a6}
>   ├sdd4 0.00k [8:52] Empty/Unknown
>   ├sdd5 19.53g [8:53] ext4 {bbe77e50-0eaa-4048-934e-9f1b72b157ff}
>   ├sdd6 19.53g [8:54] ext4 {533f504b-30a6-4cf6-86e9-56772c87bae3}
>   ├sdd7 29.30g [8:55] ext4 {ff6bcbe9-f25f-4738-ba6b-5ea3c2fe0b70}
>   ├sdd8 3.91g [8:56] ext4 {2d0d1953-dae6-4e33-8d04-cc685e2e36fb}
>   ├sdd9 19.53g [8:57] ext4 {df754d9a-3dcf-470f-8e26-d38ef6f82df0}
>   └sdd10 22.84g [8:58] swap {04daef55-ff1b-4fe5-a897-ebe397a145b6}
> PCI [ahci] 0a:00.0 SATA controller: ASMedia Technology Inc. ASM1062
> Serial ATA Controller (rev 01)
> ├scsi 6:x:x:x [Empty]
> └scsi 7:x:x:x [Empty]
> PCI [ahci] 0c:00.0 SATA controller: Marvell Technology Group Ltd.
> 88SE9128 PCIe SATA 6 Gb/s RAID controller with HyperDuo (rev 11)
> ├scsi 8:0:0:0 ATA      ST31000524AS     {9VPE44LL}
> │└sde 931.51g [8:64] Partitioned (dos)
> │ └sde1 931.51g [8:65] MD raid10,near2 (0/4) (w/ sdb1,sdc1,sdf1)
> in_sync 'debianbase:0' {79baaa2f-0aa2-b9fa-18e2-ea6b6e2846b3}
> │  └md0 1.82t [9:0] MD v1.2 raid10,near2 (4) read-auto, 512k Chunk
> {79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3}
> │                   Partitioned (gpt)
> ├scsi 9:0:0:0 ATA      ST31000524AS     {9VPE31PN}
> │└sdf 931.51g [8:80] Partitioned (dos)
> │ └sdf1 931.51g [8:81] MD raid10,near2 (1/4) (w/ sdb1,sdc1,sde1)
> in_sync 'debianbase:0' {79baaa2f-0aa2-b9fa-18e2-ea6b6e2846b3}
> │  └md0 1.82t [9:0] MD v1.2 raid10,near2 (4) read-auto, 512k Chunk
> {79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3}
> │                   Partitioned (gpt)
> ├scsi 10:x:x:x [Empty]
> ├scsi 11:x:x:x [Empty]
> ├scsi 12:x:x:x [Empty]
> ├scsi 13:x:x:x [Empty]
> └scsi 14:x:x:x [Empty]
> 
> 
> First time I've ever used git.
> 
> I think that this is everything you've asked for.
> 
> Thanks for the help!
> 
> Darald
> 
> maurice <[email-masked]>
> Sep 9
> Reply
> to me
> On 9/8/2015 8:37 PM, o1bigtenor wrote:
> On Tue, Sep 8, 2015 at 6:10 PM, maurice <[email-masked]> wrote:
> Hi Darald
> 
> We need the output from:
> 
> cat /proc/mdstat
> # cat /proc/mdstat
> Personalities : [raid10]
> md0 : active (auto-read-only) raid10 sde1[0] sdb1[3] sdc1[4] sdf1[1]
>       1953518592 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
> 
> Good!
> We see that you have one md device: md0
> And that it has 4 members.
> All are in use and are active.
> Looks good.
> 
> 
> I still need you to run these two more commands, using the correct md
> device, like this:
> # mdadm -D /dev/md0
> 
> # mdadm -E /dev/md0
> 
> 
> 
> Here we have your drive details, from lsdrv.
> I see that 3 drives were originally prepared with fdisk, and have
> dos/FAT partition tables.
> And that 2 were prepared with something else, and have GPT partition tables.
> I do not think that is bad, just shows me that they were not all
> prepped at the same time.
> 
> # ./lsdrv
> ..      ST1000DM003-1ER1 {Z4Y4SPM8}
> │└sda 931.51g [8:0] Partitioned (gpt)
> │ ├sda1 256.00m [8:1] vfat {8642-E734}
> │ │└Mounted as /dev/sda1 @ /boot/efi
> │ ├sda2 256.00m [8:2] Empty/Unknown
> │ ├sda3 24.41g [8:3] ext4 {a3fbea19-5b7d-47af-b06c-08c49171211b}
> │ │└Mounted as /dev/sda3 @ /
> │ ├sda4 62.50g [8:4] swap {354a6e33-5743-4c97-bfb9-fd6f9d4521e6}
> │ └sda5 372.53g [8:5] ext4 {b36dbfc6-6897-4311-aa87-9a958ccad072}
> │  └Mounted as /dev/sda5 @ /home
> 
> ├scsi 3:0:0:0 ATA      ST31000524AS     {9VPE3VX1}
> │└sdb 931.51g [8:16] Partitioned (dos)
> │ └sdb1 931.51g [8:17] MD raid10,near2 (3/4) (w/ sdc1,sde1,sdf1)
> 
> ├scsi 4:0:0:0 ATA      ST1000DM003-1ER1 {Z4Y27SPH}
> │└sdc 931.51g [8:32] Partitioned (gpt)
> │ └sdc1 931.51g [8:33] MD raid10,near2 (2/4) (w/ sdb1,sde1,sdf1)
> 
> ├scsi 8:0:0:0 ATA      ST31000524AS     {9VPE44LL}
> │└sde 931.51g [8:64] Partitioned (dos)
> │ └sde1 931.51g [8:65] MD raid10,near2 (0/4) (w/ sdb1,sdc1,sdf1)
> 
> ├scsi 9:0:0:0 ATA      ST31000524AS     {9VPE31PN}
> │└sdf 931.51g [8:80] Partitioned (dos)
> │ └sdf1 931.51g [8:81] MD raid10,near2 (1/4) (w/ sdb1,sdc1,sde1)
> 
> 
> If you don't mind, please run those two more commands.
> 
> Then tell me:
> 
> WHAT is WRONG here?
> 
> 
> o1bigtenor <[email-masked]>
> Sep 9
> Reply
> to maurice
> On Wed, Sep 9, 2015 at 7:23 AM, maurice <[email-masked]> wrote:
>> On 9/8/2015 8:37 PM, o1bigtenor wrote:
>>
>> On Tue, Sep 8, 2015 at 6:10 PM, maurice <[email-masked]> wrote:
>>
>> Hi Darald
>>
>> We need the output from:
>>
>> cat /proc/mdstat
>>
>> # cat /proc/mdstat
>> Personalities : [raid10]
>> md0 : active (auto-read-only) raid10 sde1[0] sdb1[3] sdc1[4] sdf1[1]
>>       1953518592 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
>>
>> Good!
>> We see that you have one md device: md0
>> And that it has 4 members.
>> All are in use and are active.
>> Looks good.
>>
>>
>> I still need you to run these two more commands, using the correct md
>> device, like this:
>>
>> # mdadm -D /dev/md0
> 
> # mdadm -D /dev/md0
>> # mdadm -E /dev/md0
> 
> # mdadm -E /dev/md0
> /dev/md0:
>    MBR Magic : aa55
> Partition[0] :   3907037183 sectors at            1 (type ee)
> Previous to Aug 28th I would (after a reboot) use my file manager (PCManFM)
> to navigate to the volume. Upon clicking on its representation a dialog box
> would appear asking for the root password. Upon successful completion
> there would be access to the array.
> 
> At present I can find no way to access the array.
> 
> mounting doesn't seem to work:
> 
> # mount -text4 /dev/md/0 /media/darald/Raid10
> mount: wrong fs type, bad option, bad superblock on /dev/md0,
>        missing codepage or helper program, or other error
> 
>        In some cases useful info is found in syslog - try
>        dmesg | tail or so.
> root@debiantestingbase:/media/darald# mount -text2 /dev/md/0
> /media/darald/Raid10
> mount: wrong fs type, bad option, bad superblock on /dev/md0,
>        missing codepage or helper program, or other error
> 
>        In some cases useful info is found in syslog - try
>        dmesg | tail or so.
> root@debiantestingbase:/media/darald# mount -gpt /dev/md/0
> /media/darald/Raid10
> mount: invalid option -- 'g'
> 
> a different attempt to mount the array
> 
> # blkid | grep md0
> /dev/md0: PTUUID="1eb47793-0cdf-4e16-ae84-33ec825ac448" PTTYPE="gpt"
> root@debiantestingbase:/media/darald# vi /etc/fstab
> root@debiantestingbase:/media/darald# mount -a
> mount: can't find UUID=1eb47793-0cdf-4e16-ae84-33ec825ac448
> 
> 
> I have found this but am concerned that I could also possibly destroy the array
> not knowing if THIS is the solution
> 
> 
> # fsck /dev/md0
> fsck from util-linux 2.26.2
> e2fsck 1.42.13 (17-May-2015)
> ext2fs_open2: Bad magic number in super-block
> fsck.ext2: Superblock invalid, trying backup blocks...
> fsck.ext2: Bad magic number in super-block while trying to open /dev/md0
> 
> The superblock could not be read or does not describe a valid ext2/ext3/ext4
> filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
> filesystem (and not swap or ufs or something else), then the superblock
> is corrupt, and you might try running e2fsck with an alternate superblock:
>     e2fsck -b 8193 <device>
>  or
>     e2fsck -b 32768 <device>
> 
> o1bigtenor <[email-masked]>
> Sep 11
> Reply
> to Maurice
> Greetings
> 
> Were you able to put together anything to facilitate my filesystem repair?
> 
> Thanking you for your assistance.
> 
> 
> maurice <[email-masked]>
> Sep 11
> Reply
> to me
> On 9/11/2015 5:11 AM, o1bigtenor wrote:
> Greetings
> 
> Were you able to put together anything to facilitate my filesystem repair?
> 
> 
> Sorry, last night got really busy.
> Will look at this some more tonight.
> 
> 
> o1bigtenor <[email-masked]>
> Sep 11
> Reply
> to maurice
> On Fri, Sep 11, 2015 at 7:21 AM, maurice <[email-masked]> wrote:
>> On 9/11/2015 5:11 AM, o1bigtenor wrote:
>>
>> Greetings
>>
>> Were you able to put together anything to facilitate my filesystem repair?
>>
>>
>> Sorry, last night got really busy.
>> Will look at this some more tonight.
>>
> 
> Thanks - - - looking forward to seeing the stuff.
> 
> 
> Maurice Hilarius <[email-masked]>
> Sep 11
> Reply
> to me
> Quick question..  Is this an ext4 filesystem?
> 
> 
> 
> o1bigtenor <[email-masked]>
> Sep 11
> Reply
> to Maurice
> Yes
> 
> 
> maurice <[email-masked]>
> Sep 11
> Reply
> to me
> OK, this article provides a pretty good summary of methods to recover
> a damaged ext4 filesystem.
> 
> http://www.vmexplore.com/check-repair-ext3-ext4-filesystem-oracle-linux/
> 
> 
> maurice <[email-masked]>
> Sep 11
> Reply
> to me
> Also useful:
> 
> http://www.cyberciti.biz/faq/recover-bad-superblock-from-corrupted-partition/
> 
> 
> 
> 
> 
> 
> You DID ask to have everything as one huge file - - - so here you have it.
> 
> I would find this email confusing but you did ask for everything to be dumped
> into one big file - - - .
> Any further questions - - - I'll do what I can.
> 
> TIA
> 
> Dee
> 

Lots of useful information, but no-one tried to drill deeper in /dev/dm0.

Everything points to there being another missing layer between the raid
device (md0) and your ext4 filesystem, that previously available as
/dev/dm-0.  We need to decipher the initial contents of /dev/md0 to find
a lost signature.

Please supply the updated mdadm -E reports as I asked.  Trim all of the
old mails from your reply, too.  Makes for very difficult reading.

When we get your array assembled, we'll use hexdump to poke around.

Phil