Subject: Re: Advice requested
From: o1bigtenor <[email-masked]>
To: Phil Turmel <[email-masked]>
Cc: Linux-RAID <[email-masked]>
Date: Mon, 2 Nov 2015 12:21:22 -0600

On Mon, Nov 2, 2015 at 9:41 AM, Phil Turmel <[email-masked]> wrote:
> Good morning Dee,
>
> { Added linux-raid back -- convention on kernel.org is reply-to-all }

I am finding that every linux group has their own way of doing this - - - makes
it difficult to comply with expectations!

>>> What happened to the other two drives?  If you still have them, please
>>> supply "mdadm -E" report for them.
>>
>> root@debianbase:/# mdadm -E
>> mdadm: No devices to examine
>
> That means you didn't tell it what device(s) to examine, so it did
> nothing.  Based on your data below, I expect you need to examine
> /dev/sdb1, /dev/sdc1, /dev/sde1, and /dev/sdf1.

root@debianbase:/# mdadm -E /dev/sdb
/dev/sdb:
  MBR Magic : aa55
Partition[0] :   1953521664 sectors at         2048 (type fd)
root@debianbase:/# mdadm -E /dev/sdc
/dev/sdc:
  MBR Magic : aa55
Partition[0] :   1953525167 sectors at            1 (type ee)
root@debianbase:/# mdadm -E /dev/sde
/dev/sde:
  MBR Magic : aa55
Partition[0] :   1953521664 sectors at         2048 (type fd)
root@debianbase:/# mdadm -E /dev/sdf
/dev/sdf:
  MBR Magic : aa55
Partition[0] :   1953521664 sectors at         2048 (type fd)

(I had some other system issues about 5 weeks ago an damaged the
superblock from sdc.)

>
> { A look at the section on --examine in "man mdadm" would have helped
> here.  Please spend some time reviewing "man mdadm" and "man md". }

I have found that man pages are great resources when one does know or has
used the command, when you haven't - - - well there are most often NO examples,
either of what to do or what happens when something is wrong. They are most
often written by those intimately familiar with the area (never have a
problem with
anything) and so I don't bother looking at them - - - they just aren't useful.
(The whole RTFM thing is one of the biggest reasons that I find *nix very
frustrating - - - I have spent hours reading and re-reading a man page
and sometimes
four or five pages talking about it and its still not clear exactly
how to write the
command or even better if the result is not positive there is absolutely no idea
given as to how to troubleshoot things - - - sorry its a rant but you
did tell me to
read it and - - - - .)

>
> It seems /dev/sdd1 is missing, so I guess the device names have changed.
> Please supply the reports for all four array devices with their current
> names (as shown in the smartctl reports).

sda is my current 'operating system' disc - - - or where I want my operating
system to reside. I wanted all my information to be on the raid array.

sdd is a ssd drive that is presently not used and was used previously as
an 'operating system' disc.

Therefore neither of these really has anything to do with the raid array.
>
> Paste the result inline in your reply.
>
>>> Also supply "smartctl -i -A -l scterc /dev/sdX" for each of your drives.
>>
>> please see attachment
>
> I've converted your libreoffice file to text -- please just post text
> in-line in the future, without wrapping.  No attachments, no pastebin.
>
> Some comments inline:
>
>> root@debianbase:/# smartctl -i -A -l scterc /dev/sdb
>> smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.1.0-2-amd64] (local build)
>> Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
>>
>> === START OF INFORMATION SECTION ===
>> Model Family:     Seagate Barracuda 7200.12
>> Device Model:     ST31000524AS
>> Serial Number:    9VPE3VX1
>> LU WWN Device Id: 5 000c50 03ee8ad75
>> Firmware Version: JC4B
>> User Capacity:    1,000,204,886,016 bytes [1.00 TB]
>> Sector Size:      512 bytes logical/physical
>> Rotation Rate:    7200 rpm
>> Device is:        In smartctl database [for details use: -P show]
>> ATA Version is:   ATA8-ACS T13/1699-D revision 4
>> SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
>> Local Time is:    Mon Nov  2 07:16:28 2015 CST
>> SMART support is: Available - device has SMART capability.
>> SMART support is: Enabled
>>
>> === START OF READ SMART DATA SECTION ===
>> SMART Attributes Data Structure revision number: 10
>> Vendor Specific SMART Attributes with Thresholds:
>> ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
>>  1 Raw_Read_Error_Rate     0x000f   117   100   006    Pre-fail  Always       -       150964952
>>  3 Spin_Up_Time            0x0003   100   100   000    Pre-fail  Always       -       0
>>  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       664
>>  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
>
> This ^^^ is good.
>
>>  7 Seek_Error_Rate         0x000f   066   060   030    Pre-fail  Always       -       3829738
>>  9 Power_On_Hours          0x0032   073   073   000    Old_age   Always       -       24498
>> 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
>> 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       646
>> 183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
>> 184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
>> 187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
>> 188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
>> 189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
>> 190 Airflow_Temperature_Cel 0x0022   069   052   045    Old_age   Always       -       31 (Min/Max 29/31)
>> 194 Temperature_Celsius     0x0022   031   048   000    Old_age   Always       -       31 (0 17 0 0 0)
>> 195 Hardware_ECC_Recovered  0x001a   035   009   000    Old_age   Always       -       150964952
>> 197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
>
> As is this ^^^
>
>> 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
>> 199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
>> 240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       26106 (151 195 0)
>> 241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       3874945427
>> 242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       3008180623
>>
>> SCT Error Recovery Control:
>>           Read: Disabled
>>          Write: Disabled
>
> This ^^^ is *bad*.  You have an older desktop drive that supports ERC,
> but since it is a desktop drive, it's turned off by default.  You *need*
> a boot script that will turn it on every power cycle.  Some references
> on "timeout mismatch" are in the post-script.
>
>> root@debianbase:/# smartctl -i -A -l scterc /dev/sdc
>> smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.1.0-2-amd64] (local build)
>> Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
>>
>> === START OF INFORMATION SECTION ===
>> Model Family:     Seagate Barracuda 7200.14 (AF)
>> Device Model:     ST1000DM003-1ER162
>> Serial Number:    Z4Y27SPH
>> LU WWN Device Id: 5 000c50 079702992
>> Firmware Version: CC45
>> User Capacity:    1,000,204,886,016 bytes [1.00 TB]
>> Sector Sizes:     512 bytes logical, 4096 bytes physical
>> Rotation Rate:    7200 rpm
>> Form Factor:      3.5 inches
>> Device is:        In smartctl database [for details use: -P show]
>> ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
>> SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
>> Local Time is:    Mon Nov  2 07:16:38 2015 CST
>> SMART support is: Available - device has SMART capability.
>> SMART support is: Enabled
>>
>> === START OF READ SMART DATA SECTION ===
>> SMART Attributes Data Structure revision number: 10
>> Vendor Specific SMART Attributes with Thresholds:
>> ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
>>  1 Raw_Read_Error_Rate     0x000f   115   099   006    Pre-fail  Always       -       98239568
>>  3 Spin_Up_Time            0x0003   097   097   000    Pre-fail  Always       -       0
>>  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       89
>>  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
>>  7 Seek_Error_Rate         0x000f   100   253   030    Pre-fail  Always       -       308478
>>  9 Power_On_Hours          0x0032   096   096   000    Old_age   Always       -       4258
>> 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
>> 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       89
>> 183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
>> 184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
>> 187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
>> 188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0 0 0
>> 189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
>> 190 Airflow_Temperature_Cel 0x0022   073   053   045    Old_age   Always       -       27 (Min/Max 24/27)
>> 191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
>> 192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       20
>> 193 Load_Cycle_Count        0x0032   099   099   000    Old_age   Always       -       3807
>> 194 Temperature_Celsius     0x0022   027   047   000    Old_age   Always       -       27 (0 15 0 0 0)
>> 197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
>> 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
>> 199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
>> 240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       3830h+12m+39.793s
>> 241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       2220627944
>> 242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       24802170836
>>
>> SCT Error Recovery Control command not supported
>
> This ^^^ is *really bad*.  You have a modern desktop drive that doesn't
> support ERC at all.  You *must* use a boot script to override linux's
> low level driver timeout for this device.  Please see the references.
>
>> root@debianbase:/# smartctl -i -A -l scterc /dev/sde
>> smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.1.0-2-amd64] (local build)
>> Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
>>
>> === START OF INFORMATION SECTION ===
>> Model Family:     Seagate Barracuda 7200.12
>> Device Model:     ST31000524AS
>> Serial Number:    9VPE44LL
>> LU WWN Device Id: 5 000c50 03eed16df
>> Firmware Version: JC4B
>> User Capacity:    1,000,204,886,016 bytes [1.00 TB]
>> Sector Size:      512 bytes logical/physical
>> Rotation Rate:    7200 rpm
>> Device is:        In smartctl database [for details use: -P show]
>> ATA Version is:   ATA8-ACS T13/1699-D revision 4
>> SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
>> Local Time is:    Mon Nov  2 07:16:45 2015 CST
>> SMART support is: Available - device has SMART capability.
>> SMART support is: Enabled
>>
>> === START OF READ SMART DATA SECTION ===
>> SMART Attributes Data Structure revision number: 10
>> Vendor Specific SMART Attributes with Thresholds:
>> ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
>>  1 Raw_Read_Error_Rate     0x000f   118   100   006    Pre-fail  Always       -       184109834
>>  3 Spin_Up_Time            0x0003   100   100   000    Pre-fail  Always       -       0
>>  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       663
>>  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
>>  7 Seek_Error_Rate         0x000f   065   060   030    Pre-fail  Always       -       3341907
>>  9 Power_On_Hours          0x0032   073   073   000    Old_age   Always       -       24462
>> 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
>> 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       644
>> 183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
>> 184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
>> 187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
>> 188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
>> 189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
>> 190 Airflow_Temperature_Cel 0x0022   069   047   045    Old_age   Always       -       31 (Min/Max 25/31)
>> 194 Temperature_Celsius     0x0022   031   053   000    Old_age   Always       -       31 (0 18 0 0 0)
>> 195 Hardware_ECC_Recovered  0x001a   026   014   000    Old_age   Always       -       184109834
>> 197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
>> 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
>> 199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
>> 240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       26005 (154 33 0)
>> 241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       1062575833
>> 242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       3869056902
>>
>> SCT Error Recovery Control:
>>           Read: Disabled
>>          Write: Disabled
>
> Again.
>
>> root@debianbase:/# smartctl -i -A -l scterc /dev/sdf
>> smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.1.0-2-amd64] (local build)
>> Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
>>
>> === START OF INFORMATION SECTION ===
>> Model Family:     Seagate Barracuda 7200.12
>> Device Model:     ST31000524AS
>> Serial Number:    9VPE31PN
>> LU WWN Device Id: 5 000c50 03eeadf56
>> Firmware Version: JC4B
>> User Capacity:    1,000,204,886,016 bytes [1.00 TB]
>> Sector Size:      512 bytes logical/physical
>> Rotation Rate:    7200 rpm
>> Device is:        In smartctl database [for details use: -P show]
>> ATA Version is:   ATA8-ACS T13/1699-D revision 4
>> SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
>> Local Time is:    Mon Nov  2 07:17:53 2015 CST
>> SMART support is: Available - device has SMART capability.
>> SMART support is: Enabled
>>
>> === START OF READ SMART DATA SECTION ===
>> SMART Attributes Data Structure revision number: 10
>> Vendor Specific SMART Attributes with Thresholds:
>> ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
>>  1 Raw_Read_Error_Rate     0x000f   116   099   006    Pre-fail  Always       -       104938837
>>  3 Spin_Up_Time            0x0003   100   100   000    Pre-fail  Always       -       0
>>  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       594
>>  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
>>  7 Seek_Error_Rate         0x000f   066   060   030    Pre-fail  Always       -       3679723
>>  9 Power_On_Hours          0x0032   073   073   000    Old_age   Always       -       24417
>> 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
>> 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       579
>> 183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
>> 184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
>> 187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
>> 188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
>> 189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
>> 190 Airflow_Temperature_Cel 0x0022   070   040   045    Old_age   Always   In_the_past 30 (19 143 30 29 0)
>> 194 Temperature_Celsius     0x0022   030   060   000    Old_age   Always       -       30 (0 18 0 0 0)
>> 195 Hardware_ECC_Recovered  0x001a   036   013   000    Old_age   Always       -       104938837
>> 197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
>> 198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
>> 199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
>> 240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       25870 (135 125 0)
>> 241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       366206138
>> 242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       37669615
>>
>> SCT Error Recovery Control:
>>           Read: Disabled
>>          Write: Disabled
>
> And again.
>
> Looking for this is one of the reasons I asked for these reports --
> non-raid rated drives producing timeout mismatch failures is a common
> problem seen on this list.  I also wanted to know if your drives are
> generally healthy -- they are -- and how that might have impacted your
> situation.  { I had some of the same Seagate .12 drives -- they didn't
> start failing until they had nearly 40k hours. }

When these drives were bought, late January 2012 there was no differentiation
between drives based upon intended usage. That happened some time in
about 2013 IIRC.
>
> Another reason I asked for this is that drive names can change on power
> cycles and unplug/replug cycles.  Knowing what device is what (versus
> drive serial number) is important.  Please verify drive serial number
> versus device name after any power cycle or replug event and let us know
> any changes.

There have been no real changes in quite a long time.
The only UUID change was between before Aug 28th of this and after when the
UUID of the array was changed. It was not finding the array upon a reboot (used
to improve web speeds changed by Firefox's software) that informed me that I
had an issue.
>
>> enclosed is also a copy of the kernel log which indicates that a drive error
>> had occurred. My thinking is that my ups box (too small but all I could find
>> when I bought the system) didn't provide enough power in a severe brown-
>> out incident.
>
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.025162] scsi 3:0:0:0: Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.025768] sd 3:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.026296] sd 3:0:0:0: [sdb] Write Protect is off
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.026304] sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.026484] sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.028273]  sdb: sdb1
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.028698] sd 3:0:0:0: [sdb] Attached SCSI disk
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.086532] Switched to clocksource tsc
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.168919] md: bind<sdb1>
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.342285] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.343153] ata5.00: ATA-9: ST1000DM003-1ER162, CC45, max UDMA/133
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.343158] ata5.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.344067] ata5.00: configured for UDMA/133
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.344255] scsi 4:0:0:0: Direct-Access     ATA      ST1000DM003-1ER1 CC45 PQ: 0 ANSI: 5
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.344627] sd 4:0:0:0: [sdc] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.344631] sd 4:0:0:0: [sdc] 4096-byte physical blocks
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.344823] sd 4:0:0:0: [sdc] Write Protect is off
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.344831] sd 4:0:0:0: [sdc] Mode Sense: 00 3a 00 00
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.344946] sd 4:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.411364]  sdc: sdc1
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.412317] sd 4:0:0:0: [sdc] Attached SCSI disk
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.501080] md: bind<sdc1>
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.662509] ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.674767] ata6.00: ATA-8: Corsair Force 3 SSD, 1.3.3, max UDMA/133
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.674772] ata6.00: 468862128 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.684647] ata6.00: configured for UDMA/133
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.684933] scsi 5:0:0:0: Direct-Access     ATA      Corsair Force 3  3    PQ: 0 ANSI: 5
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.685504] sd 5:0:0:0: [sdd] 468862128 512-byte logical blocks: (240 GB/223 GiB)
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.685975] sd 5:0:0:0: [sdd] Write Protect is off
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.685983] sd 5:0:0:0: [sdd] Mode Sense: 00 3a 00 00
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.686186] sd 5:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.688051]  sdd: sdd1 sdd4 < sdd5 sdd6 sdd7 sdd8 sdd9 sdd10 >
>> Aug 28 10:39:28 debiantestingbase kernel: [    3.689305] sd 5:0:0:0: [sdd] Attached SCSI disk
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.002712] ata8: SATA link down (SStatus 0 SControl 300)
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.003107] scsi 8:0:0:0: Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.003597] sd 8:0:0:0: [sde] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.003843] scsi 9:0:0:0: Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.003975] sd 8:0:0:0: [sde] Write Protect is off
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.003980] sd 8:0:0:0: [sde] Mode Sense: 00 3a 00 00
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.004090] sd 8:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.004478] sd 9:0:0:0: [sdf] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.004645] sd 9:0:0:0: [sdf] Write Protect is off
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.004650] sd 9:0:0:0: [sdf] Mode Sense: 00 3a 00 00
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.004737] sd 9:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.004778] scsi 15:0:0:0: Processor         Marvell  91xx Config      1.01 PQ: 0 ANSI: 5
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.006375]  sdf: sdf1
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.006967] sd 9:0:0:0: [sdf] Attached SCSI disk
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.008855]  sde: sde1
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.009704] sd 8:0:0:0: [sde] Attached SCSI disk
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.018710] ata16.00: exception Emask 0x1 SAct 0x0 SErr 0x0 action 0x0
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.018753] ata16.00: irq_stat 0x40000001
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.018783] ata16.00: cmd a0/01:00:00:00:01/00:00:00:00:
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.018783]          Inquiry 12 01 00 00 ff 00res 50/00:00:af:6d:70/00:00:74:00:
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.018868] ata16.00: status: { DRDY }
>
> This ^^^ isn't really an error.
If it isn't an error this was the only place where there were any
listed changes to the array and/or its discs.

Subsequently I was unable to mount the array nor did it mount automatically as
it had for the previous time (at least 4 months).
>
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.125325] random: nonblocking pool is initialized
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.125530] md: bind<sde1>
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.142140] md: bind<sdf1>
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.144984] md: raid10 personality registered for level 10
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.145397] md/raid10:md0: active with 4 out of 4 devices
>
> If it was, you wouldn't get this ^^^ success.
May have been a success but the UUID was changed and I could find no way
to access the array (still can't for that matter!).
>
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.145440] md0: detected capacity change from 0 to 2000403038208
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.208978]  md0:
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.479305] device-mapper: uevent: version 1.0.3
>> Aug 28 10:39:28 debiantestingbase kernel: [    4.479536] device-mapper: ioctl: 4.30.0-ioctl (2014-12-22) initialised: [email-masked]
>
> This section of your syslog is too far back in time -- before the failure.
This section is AT the time of failure.
Md0 has not been accessible subsequently!
>
> [trim /]
>
>> Was following directions from one person on linux-raid and when he stopped
>> responding turned to someone who is somewhat connected with redhat and
>> hard drive stuff. There seemed to be a consensus that I needed to use low
>> level tools in an attempt to recover files, if I could. As there are about 200k
>> I wasn't looking forward to repairing things ;-(  !
>
> A link to the linux-raid archive of the prior discussion might help.
> And you might still need low level tools -- we are still trying to get
> your array started.  Then we can look at the next layer on top of that.
>  Based on your prior mails, /dev/md0 was formatted as ext4, yes?

Yes.

o1bigtenor <[email-masked]>
Sep 3
Reply
to linux-raid
Greetings

Had updated a system to Debian 8 which also had a Raid 10 array that
has been in use for about 3 1/2 years. (Setup raid under Debian 6 then
ran it mostly under Debian 7 mounting the array each time after
booting using the command

#mount /dev/dm-o /home/myspace/RAID

and didn't have any major problems with it (besides a failed drive
which was replaced and things went OK).)

Really enjoyed Debian 8's feature where after a boot (done seldom as I
was running the machine 24/7) where in the file manager clicking on
the icon for the 1.8TB drive a dialog box would popup requesting the
root password and upon correct completion the array was accessible.

Found with Debian 8 that I had the option of using a sleep state (shut
down to ram) and liked the idea of reducing power consumption (my box
with 3 gpus and 6 hdds does like to pull power) for over night.

On Aug 28th evening I noticed that I didn't have my raid array accessible.

Went to bed (have learned not to change things when I am very tired as
I can bork things up too fast.

Next morning tried a reboot to see if things would clear themselves -
-  they didn't.

After some running in circles I found this in a log:

Aug 30 19:51:40 debiantestingbase kernel: [100718.675021] FAT-fs
(md0): utf8 is not a recommended IO charset for FAT filesystems,
filesystem will be case sensitive!
Aug 30 21:35:33 debiantestingbase kernel: [106954.860493] EXT4-fs
(md0): VFS: Can't find ext4 filesystem
Aug 30 21:37:05 debiantestingbase kernel: [107046.607187] EXT4-fs
(md0): VFS: Can't find ext4 filesystem
Aug 30 22:20:33 debiantestingbase kernel: [109656.411356] PM: Syncing
filesystems ... done.

Aug 28 10:39:56 debiantestingbase udisksd[1777]: Cleaning up mount
point /media/darald/49552036-b46f-4956-ade9-3541a3dd7f0a (device 253:0
no longer exist)

from syslog.3
That UUID was the 'name' of the array previously

Aug 28 10:39:28 debiantestingbase kernel: [    3.025162] scsi 3:0:0:0:
Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
Aug 28 10:39:28 debiantestingbase kernel: [    3.025768] sd 3:0:0:0:
[sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
Aug 28 10:39:28 debiantestingbase kernel: [    3.026296] sd 3:0:0:0:
[sdb] Write Protect is off
Aug 28 10:39:28 debiantestingbase kernel: [    3.026304] sd 3:0:0:0:
[sdb] Mode Sense: 00 3a 00 00
Aug 28 10:39:28 debiantestingbase kernel: [    3.026484] sd 3:0:0:0:
[sdb] Write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
Aug 28 10:39:28 debiantestingbase kernel: [    3.028273]  sdb: sdb1
Aug 28 10:39:28 debiantestingbase kernel: [    3.028698] sd 3:0:0:0:
[sdb] Attached SCSI disk
Aug 28 10:39:28 debiantestingbase kernel: [    3.086532] Switched to
clocksource tsc
Aug 28 10:39:28 debiantestingbase kernel: [    3.168919] md: bind<sdb1>
Aug 28 10:39:28 debiantestingbase kernel: [    3.342285] ata5: SATA
link up 3.0 Gbps (SStatus 123 SControl 300)
Aug 28 10:39:28 debiantestingbase kernel: [    3.343153] ata5.00:
ATA-9: ST1000DM003-1ER162, CC45, max UDMA/133
Aug 28 10:39:28 debiantestingbase kernel: [    3.343158] ata5.00:
1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
Aug 28 10:39:28 debiantestingbase kernel: [    3.344067] ata5.00:
configured for UDMA/133
Aug 28 10:39:28 debiantestingbase kernel: [    3.344255] scsi 4:0:0:0:
Direct-Access     ATA      ST1000DM003-1ER1 CC45 PQ: 0 ANSI: 5
Aug 28 10:39:28 debiantestingbase kernel: [    3.344627] sd 4:0:0:0:
[sdc] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
Aug 28 10:39:28 debiantestingbase kernel: [    3.344631] sd 4:0:0:0:
[sdc] 4096-byte physical blocks
Aug 28 10:39:28 debiantestingbase kernel: [    3.344823] sd 4:0:0:0:
[sdc] Write Protect is off
Aug 28 10:39:28 debiantestingbase kernel: [    3.344831] sd 4:0:0:0:
[sdc] Mode Sense: 00 3a 00 00
Aug 28 10:39:28 debiantestingbase kernel: [    3.344946] sd 4:0:0:0:
[sdc] Write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
Aug 28 10:39:28 debiantestingbase kernel: [    3.411364]  sdc: sdc1
Aug 28 10:39:28 debiantestingbase kernel: [    3.412317] sd 4:0:0:0:
[sdc] Attached SCSI disk
Aug 28 10:39:28 debiantestingbase kernel: [    3.501080] md: bind<sdc1>
Aug 28 10:39:28 debiantestingbase kernel: [    3.662509] ata6: SATA
link up 3.0 Gbps (SStatus 123 SControl 300)
Aug 28 10:39:28 debiantestingbase kernel: [    3.674767] ata6.00:
ATA-8: Corsair Force 3 SSD, 1.3.3, max UDMA/133
Aug 28 10:39:28 debiantestingbase kernel: [    3.674772] ata6.00:
468862128 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
Aug 28 10:39:28 debiantestingbase kernel: [    3.684647] ata6.00:
configured for UDMA/133
Aug 28 10:39:28 debiantestingbase kernel: [    3.684933] scsi 5:0:0:0:
Direct-Access     ATA      Corsair Force 3  3    PQ: 0 ANSI: 5
Aug 28 10:39:28 debiantestingbase kernel: [    3.685504] sd 5:0:0:0:
[sdd] 468862128 512-byte logical blocks: (240 GB/223 GiB)
Aug 28 10:39:28 debiantestingbase kernel: [    3.685975] sd 5:0:0:0:
[sdd] Write Protect is off
Aug 28 10:39:28 debiantestingbase kernel: [    3.685983] sd 5:0:0:0:
[sdd] Mode Sense: 00 3a 00 00
Aug 28 10:39:28 debiantestingbase kernel: [    3.686186] sd 5:0:0:0:
[sdd] Write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
Aug 28 10:39:28 debiantestingbase kernel: [    3.688051]  sdd: sdd1
sdd4 < sdd5 sdd6 sdd7 sdd8 sdd9 sdd10 >
Aug 28 10:39:28 debiantestingbase kernel: [    3.689305] sd 5:0:0:0:
[sdd] Attached SCSI disk
Aug 28 10:39:28 debiantestingbase kernel: [    4.002712] ata8: SATA
link down (SStatus 0 SControl 300)
Aug 28 10:39:28 debiantestingbase kernel: [    4.003107] scsi 8:0:0:0:
Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
Aug 28 10:39:28 debiantestingbase kernel: [    4.003597] sd 8:0:0:0:
[sde] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
Aug 28 10:39:28 debiantestingbase kernel: [    4.003843] scsi 9:0:0:0:
Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
Aug 28 10:39:28 debiantestingbase kernel: [    4.003975] sd 8:0:0:0:
[sde] Write Protect is off
Aug 28 10:39:28 debiantestingbase kernel: [    4.003980] sd 8:0:0:0:
[sde] Mode Sense: 00 3a 00 00
Aug 28 10:39:28 debiantestingbase kernel: [    4.004090] sd 8:0:0:0:
[sde] Write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
Aug 28 10:39:28 debiantestingbase kernel: [    4.004478] sd 9:0:0:0:
[sdf] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
Aug 28 10:39:28 debiantestingbase kernel: [    4.004645] sd 9:0:0:0:
[sdf] Write Protect is off
Aug 28 10:39:28 debiantestingbase kernel: [    4.004650] sd 9:0:0:0:
[sdf] Mode Sense: 00 3a 00 00
Aug 28 10:39:28 debiantestingbase kernel: [    4.004737] sd 9:0:0:0:
[sdf] Write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
Aug 28 10:39:28 debiantestingbase kernel: [    4.004778] scsi
15:0:0:0: Processor         Marvell  91xx Config      1.01 PQ: 0 ANSI:
5
Aug 28 10:39:28 debiantestingbase kernel: [    4.006375]  sdf: sdf1
Aug 28 10:39:28 debiantestingbase kernel: [    4.006967] sd 9:0:0:0:
[sdf] Attached SCSI disk
Aug 28 10:39:28 debiantestingbase kernel: [    4.008855]  sde: sde1
Aug 28 10:39:28 debiantestingbase kernel: [    4.009704] sd 8:0:0:0:
[sde] Attached SCSI disk
Aug 28 10:39:28 debiantestingbase kernel: [    4.018710] ata16.00:
exception Emask 0x1 SAct 0x0 SErr 0x0 action 0x0
Aug 28 10:39:28 debiantestingbase kernel: [    4.018753] ata16.00:
irq_stat 0x40000001
Aug 28 10:39:28 debiantestingbase kernel: [    4.018783] ata16.00: cmd
a0/01:00:00:00:01/00:00:00:00:00/a0 tag 1 dma 16640 in
Aug 28 10:39:28 debiantestingbase kernel: [    4.018783]
Inquiry 12 01 00 00 ff 00res 50/00:00:af:6d:70/00:00:74:00:00/e0 Emask
0x1 (device error)
Aug 28 10:39:28 debiantestingbase kernel: [    4.018868] ata16.00:
status: { DRDY }
Aug 28 10:39:28 debiantestingbase kernel: [    4.125325] random:
nonblocking pool is initialized
Aug 28 10:39:28 debiantestingbase kernel: [    4.125530] md: bind<sde1>
Aug 28 10:39:28 debiantestingbase kernel: [    4.142140] md: bind<sdf1>
Aug 28 10:39:28 debiantestingbase kernel: [    4.144984] md: raid10
personality registered for level 10
Aug 28 10:39:28 debiantestingbase kernel: [    4.145397]
md/raid10:md0: active with 4 out of 4 devices
Aug 28 10:39:28 debiantestingbase kernel: [    4.145440] md0: detected
capacity change from 0 to 2000403038208
Aug 28 10:39:28 debiantestingbase kernel: [    4.208978]  md0:
Aug 28 10:39:28 debiantestingbase kernel: [    4.479305]
device-mapper: uevent: version 1.0.3
Aug 28 10:39:28 debiantestingbase kernel: [    4.479536]
device-mapper: ioctl: 4.30.0-ioctl (2014-12-22) initialised:
[email-masked]

some further information

>> Disk /dev/md0: 1.8 TiB, 2000403038208 bytes, 3907037184 sectors
>> Units: sectors of 1 * 512 = 512 bytes
>> Sector size (logical/physical): 512 bytes / 4096 bytes
>> I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
>> Disklabel type: gpt
>> Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448
>
>
>
>     parted -l

Model: ATA ST31000524AS (scsi)
Disk /dev/sdf: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1000GB  1000GB  primary               raid

Model: Linux Software RAID Array (md)
Disk /dev/md0: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start  End  Size  File system  Name  Flags

# parted -l
Model: ATA ST1000DM003-1ER1 (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system     Name   Flags
 1      1049kB  269MB   268MB   fat32                  boot, esp
 2      269MB   538MB   268MB                   /boot  msftdata
 3      538MB   26.8GB  26.2GB  ext4            /      msftdata
 4      26.8GB  93.9GB  67.1GB  linux-swap(v1)  swap
 5      93.9GB  494GB   400GB   ext4

Model: ATA ST31000524AS (scsi)
Disk /dev/sdb: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1000GB  1000GB  primary               raid

Model: ATA ST1000DM003-1ER1 (scsi)
Disk /dev/sdc: 1000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name              Flags
 1      1049kB  1000GB  1000GB               Linux filesystem

Model: ATA Corsair Force 3 (scsi)
Disk /dev/sdd: 240GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type      File system     Flags
 1      1049kB  256MB   255MB   primary   ext2            boot
 4      257MB   123GB   123GB   extended
 5      258MB   21.2GB  21.0GB  logical   ext4
 6      21.2GB  42.2GB  21.0GB  logical   ext4
 7      42.2GB  73.7GB  31.5GB  logical   ext4
 8      73.7GB  77.9GB  4194MB  logical   ext4
 9      77.9GB  98.8GB  21.0GB  logical   ext4
10      98.8GB  123GB   24.5GB  logical   linux-swap(v1)

Model: ATA ST31000524AS (scsi)
Disk /dev/sde: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1000GB  1000GB  primary               raid

Model: ATA ST31000524AS (scsi)
Disk /dev/sdf: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start  End  Size  File system  Name  Flags

The array was formed from sdb1, sdc1, sde1 and sdf1.

The array had been setup as an ext4 device.

I halted the array using:

mdadm --stop /dev/md0

I then setup a copy from the original drives (sdc1 and sdf1) to a
drive pulled from a project that I have just started on.

 dd if=/dev/sdc1 bs=1M | gzip > /mnt/newbackupdisk/sdc1.img.gz
  dd if=/dev/sdf1 bs=1M | gzip > /mnt/newbackupdisk/sdf1.img.gz

When I checked using md5sum the copy of sdf1 and the original do NOT match.

Please?

How can I get a copy of the information off of the raid 10 array?
Is there a way to verify this copy?
How to I re-setup the raid 10 array?
Should I be replacing sde (drive with a momentary failure)?

(Hopefully not too many questions nor too much initial information!!)

Any advice will be greatly appreciated!!

TIA

Dee

Alexander Afonyashin <[email-masked]>
Sep 4
Reply
to me, linux-raid
Hi,

Please paste the output of mdadm -E /dev/sdb1 (and other 3 partitions:
/dev/sdc1, /dev/sde1, /dev/sdf1)

Regards,
Alexander

o1bigtenor <[email-masked]>
Sep 4
Reply
to Alexander
On Fri, Sep 4, 2015 at 1:03 AM, Alexander Afonyashin
<[email-masked]> wrote:
>
> Hi,
>
> Please paste the output of mdadm -E /dev/sdb1 (and other 3 partitions:
> /dev/sdc1, /dev/sde1, /dev/sdf1)
>
 # mdadm -E /dev/sdb1
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
           Name : debianbase:0
  Creation Time : Mon Mar  5 08:26:28 2012
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
     Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
  Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=1024 sectors
          State : clean
    Device UUID : a80c76db:eaea61af:bcb9cbbb:ac99e467

    Update Time : Fri Aug 28 10:38:32 2015
       Checksum : 29a4fa98 - correct
         Events : 47341

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

# mdadm -E /dev/sdc1
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
           Name : debianbase:0
  Creation Time : Mon Mar  5 08:26:28 2012
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 1953521039 (931.51 GiB 1000.20 GB)
     Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
  Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1960 sectors, after=2447 sectors
          State : clean
    Device UUID : 840fc99d:6f607ea8:9a4a9b0a:a965c85c

    Update Time : Fri Aug 28 10:38:32 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 28cfd5ab - correct
         Events : 47341

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

# mdadm -E /dev/sde1
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
           Name : debianbase:0
  Creation Time : Mon Mar  5 08:26:28 2012
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
     Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
  Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=1024 sectors
          State : clean
    Device UUID : cb37e722:d3a98f88:13439de2:bdad9d2b

    Update Time : Fri Aug 28 10:38:32 2015
       Checksum : 34ce8208 - correct
         Events : 47341

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

# mdadm -E /dev/sdf1
/dev/sdf1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
           Name : debianbase:0
  Creation Time : Mon Mar  5 08:26:28 2012
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
     Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
  Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=1024 sectors
          State : clean
    Device UUID : 9e749fa9:a0efe791:ea09d2e2:72b99f6c

    Update Time : Fri Aug 28 10:38:32 2015
       Checksum : 615d736 - correct
         Events : 47341

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

Alexander Afonyashin <[email-masked]>
Sep 4
Reply
to me
Hi,

According to output your raid is healthy and running. Please, paste
the output of cat /proc/mdstat.

Regards,
Alexander

o1bigtenor <[email-masked]>
Sep 4
Reply
to Alexander
On Fri, Sep 4, 2015 at 8:16 AM, Alexander Afonyashin
<[email-masked]> wrote:
> Hi,
>
> According to output your raid is healthy and running. Please, paste
> the output of cat /proc/mdstat.
>
# cat /proc/mdstat
Personalities : [raid10]
unused devices: <none>

If it is running - - - then how do I access it?

Previously I would go into my file manager program (PCManFM) on on the
left side there would be
an icon for the array. I needed to click on it and then a dialog box
would appear asking for the root
password the completion of which would give access to the array. That
no longer happens, instead
I get an error dialog box that says :

Only system administrators have the permission to do this.

Would you please tell me how I would set up access for myself as user
to get access to the array?

Thanking you for your help.

Dee

o1bigtenor <[email-masked]>
Sep 4
Reply
to Alexander
On Fri, Sep 4, 2015 at 8:16 AM, Alexander Afonyashin
<[email-masked]> wrote:
> Hi,
>
> According to output your raid is healthy and running. Please, paste
> the output of cat /proc/mdstat.
>

Just remembered a few things after looking through  my open terminals
(I keep terminals open
when I am working on long term problems until the issue has been
resolved with 100 000 line
storage level).

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda3 during installation
UUID=a3fbea19-5b7d-47af-b06c-08c49171211b /               ext4
errors=remount-ro 0       1
# /boot/efi was on /dev/sda1 during installation
UUID=8642-E734  /boot/efi       vfat    umask=0077      0       1
# /home was on /dev/sda5 during installation
UUID=b36dbfc6-6897-4311-aa87-9a958ccad072 /home           ext4
defaults        0       2
# swap was on /dev/sda4 during installation
UUID=354a6e33-5743-4c97-bfb9-fd6f9d4521e6 none            swap    sw
           0       0
# swap was on /dev/sdd10 during installation
UUID=04daef55-ff1b-4fe5-a897-ebe397a145b6 none            swap    sw
           0       0
/dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
# raid10 array
PTUUID=1eb47793-0cdf-4e16-ae84-33ec825ac448 /media/darald/Raid10
   gpt    defaults         0       2

§ blkid grep md0

o1bigtenor <[email-masked]>
Sep 4
Reply
to Alexander

Sorry just in a thunder and lightening (electrical) storm so things
were not responding correctly!!.
(keyboard shifted to a different layout)

# blkid | grep md0

gives no response.

the fstab information says that the array is formatted as gpt where
the discs were formatted as ext4.

Sorry this is a whole bunch of weirdness. Perhaps it was a storm on
the 28th of August too.
Checked my journal it was storming on the 27th. Maybe that produced an
effect, sorry we are
beyond anything I know much about.

Thanking you for your help!!

Dee

Alexander Afonyashin <[email-masked]>
Sep 7
Reply
to me
Hi,

Try to assemble it almost automatically by running:

mdadm --assemble --scan

then add --force if asked.

If above not helps, try to construct it manually:

mdadm -A /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sde1 /dev/sdf1

Regards,
Alexander

o1bigtenor <[email-masked]>
Sep 7
Reply
to Alexander
On Mon, Sep 7, 2015 at 1:40 AM, Alexander Afonyashin
<[email-masked]> wrote:
> Hi,
>
> Try to assemble it almost automatically by running:
>
> mdadm --assemble --scan
>
> then add --force if asked.
>
> If above not helps, try to construct it manually:
>
> mdadm -A /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sde1 /dev/sdf1
>

I do not have a complete backup of all the files on the array.

Will any of this compromise the data on the array?

(Tried to backup sdc and sdf but the md5sum on sdf indicates that the two items
are not the same.)

Regards

Darald

Alexander Afonyashin <[email-masked]>
Sep 7
Reply
to me
Hi,

By running -A == --assemble option you asked mdadm only to start raid
not create it writing new metadata to disks.

Regards,
Alexander

o1bigtenor <[email-masked]>
Sep 7
Reply
to Alexander
On Mon, Sep 7, 2015 at 6:04 AM, Alexander Afonyashin
<[email-masked]> wrote:
> Hi,
>
> By running -A == --assemble option you asked mdadm only to start raid
> not create it writing new metadata to disks.
>

Thank you - - - just a little paranoid about losing the array!

# mdadm --assemble --scan
mdadm: /dev/md/0 has been started with 4 drives.

# blkid | grep md0
/dev/md0: PTUUID="1eb47793-0cdf-4e16-ae84-33ec825ac448" PTTYPE="gpt"

Discs were originally set up as ext4.

How would I now access the array?

Thanks for the help!!!
Regards

Dee

Alexander Afonyashin <[email-masked]>
Sep 8
Reply
to me
Hi,

mount -t ext4 /dev/md0 /mnt - like this

But check it with fsck /dev/md0 first.

Regards,
Alexander

o1bigtenor <[email-masked]>
Sep 8
Reply
to Alexander
On Tue, Sep 8, 2015 at 2:39 AM, Alexander Afonyashin
<[email-masked]> wrote:
> Hi,
>
> mount -t ext4 /dev/md0 /mnt - like this
>
> But check it with fsck /dev/md0 first.

Looks like there are further issues - -

# cat /etc/fstab
using fsck suggests that there is/are bad superblocks

# fsck /dev/md0
fsck from util-linux 2.26.2
e2fsck 1.42.13 (17-May-2015)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/md0

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

Is there a way to make a verifiable backup of the array before doing
something to the array that might totally corrupt it?

Please - - there are files on the array that are important. I didn't
realize that an array could corrupt and then cause these issues.
Previous issues were only those of a defective hard drive which was a
very small problem compared to this.

TIA

Dee

another group of emails

o1bigtenor <[email-masked]>
Aug 29
Reply
to TCLUG
Greetings

I don't know what's going on but this is weird!

System has been running Debian 8 with things going good (raid mounts
so when saving a file after a reboot all I need to do is click on the
listed array give the root password and bob's your uncle).

(Its a uefi setup)
Well this morning - - -
fdisk -l shows the array
its not mounted

the command I used to use for mounting in Debian 7 no longer works

so - - - - any suggestions?

TIA

Dee

Marc Skinner <[email-masked]>
Aug 29
Reply
to TCLUG
Is this a hardware raid or software raid?  More details?  Is the raid
array internal or externally connected?

Thanks.
_______________________________________________
TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
[email-masked]
http://mailman.mn-linux.org/mailman/listinfo/tclug-list

_______________________________________________
TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
[email-masked]
http://mailman.mn-linux.org/mailman/listinfo/tclug-list

o1bigtenor <[email-masked]>
Aug 29
Reply
to TCLUG
On Sat, Aug 29, 2015 at 11:04 AM, Marc Skinner <[email-masked]> wrote:
Is this a hardware raid or software raid?  More details?  Is the raid
array internal or externally connected?

software raid
internal

# cat /proc/mdstat
Personalities : [raid10]
md0 : active (auto-read-only) raid10 sde1[0] sdf1[1] sdc1[4] sdb1[3]
      1953518592 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]

# mdadm --examine --scan
ARRAY /dev/md/0  metadata=1.2 UUID=79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
name=debianbase:0

from blkid

/dev/md0: PTUUID="1eb47793-0cdf-4e16-ae84-33ec825ac448" PTTYPE="gpt"

from fdisk -l

Disk /dev/md0: 1.8 TiB, 2000403038208 bytes, 3907037184 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
Disklabel type: gpt
Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448

Marc Skinner
So what is the problem ... it won't mount? Did you do an fsck on it? Assuming...
Aug 29

o1bigtenor
# fsck /dev/md0 fsck from util-linux 2.26.2 e2fsck 1.42.13 (17-May-2015) ext2...
Aug 30

gerry <[email-masked]>
Aug 30
Reply
to me
A few thoughts:

Your prior email shows that the md was set to read-only, right?
md0 : active (auto-read-only) raid10 sde1[0] sdf1[1] sdc1[4] sdb1[3]

Look in your logs for reasons why it got set to auto-read-only.
This isn't your issue but it's something to read:
http://unix.stackexchange.com/questions/101072/new-md-array-is-auto-read-only-and-has-resync-pending

That email also showed some output from
    fdisk -l /dev/md0.
Is there any chance you partitioned /dev/md0?
Because the output says there is a gpt partition table on the /dev/md0 device.

Quoting:
Disk /dev/md0: 1.8 TiB, 2000403038208 bytes, 3907037184 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
Disklabel type: gpt
Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448

Was there anything after that?
Run
    parted -l

You made a statement about booting that was confusing to me.
Here's the section:

        System has been running Debian 8 with things going good (raid
        mounts so
        when saving a file after a reboot all I need to do is click on the
        listed array give the root password and bob's your uncle).
[snip]

        the command I used to use for mounting in Debian 7 no longer works

What does that mean "click on the listed array give the root password" ?
Where were you doing such a thing?  Did you by any chance make
/dev/md0 into an encrypted device?

What is the command you "used to use for mounting in Debian 7" ?

Gerry

--
[email-masked]

o1bigtenor <[email-masked]>
Aug 30
Reply
to gerry
On Sun, Aug 30, 2015 at 12:34 PM, gerry <[email-masked]> wrote:
A few thoughts:

Your prior email shows that the md was set to read-only, right?
md0 : active (auto-read-only) raid10 sde1[0] sdf1[1] sdc1[4] sdb1[3]

Look in your logs for reasons why it got set to auto-read-only.
This isn't your issue but it's something to read:
http://unix.stackexchange.com/questions/101072/new-md-array-is-auto-read-only-and-has-resync-pending

Any ideas which log to look into?

That email also showed some output from
    fdisk -l /dev/md0.
Is there any chance you partitioned /dev/md0?

(Its over 3 years ago.) All four discs were partitioned (sdb1, sdc1 etc).

Because the output says there is a gpt partition table on the /dev/md0 device.

Quoting:
Disk /dev/md0: 1.8 TiB, 2000403038208 bytes, 3907037184 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
Disklabel type: gpt
Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448

Was there anything after that?
Run
    parted -l

Model: ATA ST31000524AS (scsi)
Disk /dev/sdf: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1000GB  1000GB  primary               raid

Model: Linux Software RAID Array (md)
Disk /dev/md0: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start  End  Size  File system  Name  Flags

That's the information for the last disc (in the array) and the array using
parted -l

You made a statement about booting that was confusing to me.
Here's the section:

        System has been running Debian 8 with things going good (raid
        mounts so
        when saving a file after a reboot all I need to do is click on the
        listed array give the root password and bob's your uncle).
[snip]

        the command I used to use for mounting in Debian 7 no longer works

What does that mean "click on the listed array give the root password" ?

Am using LXDE and the file manager is PCManFM.

The file manager would bring up a representation where the array was listed as a
device. After a boot clicking on the button a dialog box would open which asked
for the password for root. Upon entry access was open to the array.

Where were you doing such a thing?  Did you by any chance make
/dev/md0 into an encrypted device?

As far as I know I didn't.

What is the command you "used to use for mounting in Debian 7" ?

mount /dev/dm-0 /home/myspace/RAID

My guess is that I managed to find some kind of a bug that descends
with a combined use
of suspend (upon night close down on the system), mdadm, lxde and pcmanfm.

Thanks for the ideas.

Dee

gerry <[email-masked]>
Aug 30
Reply
to me
Any ideas which log to look into?

Look in all of them.  Everything in /var/log/
Use commands like

cd /var/log
grep read * | grep only

and for the compressed files
zgrep read *.gz | grep only

Then for those files that look interesting, use less or zless to view them.
(The "q" key will exit less or zless)

You'll need to be root do all that.

That's the information for the last disc (in the array) and the array using
parted -l

<sigh>
I can't tell from that whether you sent me the entire output of parted
-l or you were picking and choosing.

For all these commands, run them and send the entire output. Never
skimp on information when debugging.

mount /dev/dm-0 /home/myspace/RAID
Hmm. In debian 7 you mounted /dev/dm-0 but that's just a devmapper
device and they are dynamic.  More about devmapper devices can be
gotten like this:

ls -l /dev/mapper
dmsetup ls
dmestup table

also run (as root, again)
lsblk

Hopefully that will give more information.

o1bigtenor <[email-masked]>
Aug 30
Reply
to gerry
On Sun, Aug 30, 2015 at 9:57 PM, gerry <[email-masked]> wrote:
Any ideas which log to look into?

Look in all of them.  Everything in /var/log/
Use commands like

cd /var/log
grep read * | grep only

and for the compressed files
zgrep read *.gz | grep only

Then for those files that look interesting, use less or zless to view them.
(The "q" key will exit less or zless)

You'll need to be root do all that.

That's the information for the last disc (in the array) and the array using
parted -l

<sigh>
I can't tell from that whether you sent me the entire output of parted
-l or you were picking and choosing.

I was trying to give you relevant information.

# parted -l
Model: ATA ST1000DM003-1ER1 (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system     Name   Flags
 1      1049kB  269MB   268MB   fat32                  boot, esp
 2      269MB   538MB   268MB                   /boot  msftdata
 3      538MB   26.8GB  26.2GB  ext4            /      msftdata
 4      26.8GB  93.9GB  67.1GB  linux-swap(v1)  swap
 5      93.9GB  494GB   400GB   ext4

Model: ATA ST31000524AS (scsi)
Disk /dev/sdb: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1000GB  1000GB  primary               raid

Model: ATA ST1000DM003-1ER1 (scsi)
Disk /dev/sdc: 1000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name              Flags
 1      1049kB  1000GB  1000GB               Linux filesystem

Model: ATA Corsair Force 3 (scsi)
Disk /dev/sdd: 240GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type      File system     Flags
 1      1049kB  256MB   255MB   primary   ext2            boot
 4      257MB   123GB   123GB   extended
 5      258MB   21.2GB  21.0GB  logical   ext4
 6      21.2GB  42.2GB  21.0GB  logical   ext4
 7      42.2GB  73.7GB  31.5GB  logical   ext4
 8      73.7GB  77.9GB  4194MB  logical   ext4
 9      77.9GB  98.8GB  21.0GB  logical   ext4
10      98.8GB  123GB   24.5GB  logical   linux-swap(v1)

Model: ATA ST31000524AS (scsi)
Disk /dev/sde: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1000GB  1000GB  primary               raid

Model: ATA ST31000524AS (scsi)
Disk /dev/sdf: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1000GB  1000GB  primary               raid

Model: Linux Software RAID Array (md)
Disk /dev/md0: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start  End  Size  File system  Name  Flags

For all these commands, run them and send the entire output. Never
skimp on information when debugging.

mount /dev/dm-0 /home/myspace/RAID
Hmm. In debian 7 you mounted /dev/dm-0 but that's just a devmapper
device and they are dynamic.  More about devmapper devices can be
gotten like this:

ls -l /dev/mapper

# ls -l /dev/mapper
total 0
crw------- 1 root root 10, 236 Aug 29 04:49 control

dmsetup ls

# dmsetup ls
No devices found

dmestup table

 # dmestup table
bash: dmestup: command not found

also run (as root, again)
lsblk

 # lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE   MOUNTPOINT
sda       8:0    0 931.5G  0 disk
├─sda1    8:1    0   256M  0 part   /boot/efi
├─sda2    8:2    0   256M  0 part
├─sda3    8:3    0  24.4G  0 part   /
├─sda4    8:4    0  62.5G  0 part   [SWAP]
└─sda5    8:5    0 372.5G  0 part   /home
sdb       8:16   0 931.5G  0 disk
└─sdb1    8:17   0 931.5G  0 part
  └─md0   9:0    0   1.8T  0 raid10
sdc       8:32   0 931.5G  0 disk
└─sdc1    8:33   0 931.5G  0 part
  └─md0   9:0    0   1.8T  0 raid10
sdd       8:48   0 223.6G  0 disk
├─sdd1    8:49   0   243M  0 part
├─sdd4    8:52   0     1K  0 part
├─sdd5    8:53   0  19.5G  0 part
├─sdd6    8:54   0  19.5G  0 part
├─sdd7    8:55   0  29.3G  0 part
├─sdd8    8:56   0   3.9G  0 part
├─sdd9    8:57   0  19.5G  0 part
└─sdd10   8:58   0  22.9G  0 part   [SWAP]
sde       8:64   0 931.5G  0 disk
└─sde1    8:65   0 931.5G  0 part
  └─md0   9:0    0   1.8T  0 raid10
sdf       8:80   0 931.5G  0 disk
└─sdf1    8:81   0 931.5G  0 part
  └─md0   9:0    0   1.8T  0 raid10
sr0      11:0    1  1024M  0 rom

There you go.

Thanks for the help!

gerry <[email-masked]>
Aug 30
Reply
to me
Cool.  And did you find anything in the logs?

The dmsetup output is interesting. I would have expected some dm devices.
Without dm devices, there are no /dev/dm things to mount so that's why
your old Debian7 command doesn't work.

Did you expect there to be a partition table on /dev/md0?
Or did you expect to be able to just mount /dev/md0?
(was there a filesystem on /dev/md0 or was it a  partitioned logical
disk mirror?)

You might be able to figure that out from the logs too.

I think the thing went to read-only to protect itself.

If I were there, I'd run these commands and see what they say.
parted /dev/sdb1 list
parted /dev/sdc1 list
parted /dev/sde1 list
parted /dev/sdf1 list

Before trying to "fix" the array, you will want to see if you can
figure out if you still have one good setup.

That mirror setup likely mirrors sdb1 with sdc1 and sde1 with sdf1.
And the stripes were acrosss those two mirrors.
To see if an individual disk got corrupted, try this:
(This is non-destructive.)
for a in /dev/sd{b,c,e,f}1
do
  echo -n $a "  "
  dd skip=10 if=$a bs=1M count=10 status=none| md5sum
done

I made a dummy setup with loop devices and I see this:
for a in /dev/loop*p1
do
  echo -n $a "  ";dd skip=10 if=$a bs=1M count=10 status=none| md5sum
done
/dev/loop1p1   58b27e44c48997bc126eec1eec12ab8b  -
/dev/loop2p1   58b27e44c48997bc126eec1eec12ab8b  -
/dev/loop3p1   5aa8e9cfbcc3f6612f839b9a96da41df  -
/dev/loop4p1   5aa8e9cfbcc3f6612f839b9a96da41df  -

See how 1 and 2 are identical? And 3 and 4 are identical?
That's the mirroring.
Let's see if yours is similar.
....

If you don't see errors in the log, then it's fairly likely that
something actually wrote a partition table onto /dev/md0. Otherwise
it's wouldn't see
Disklabel type: gpt
Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448

In which case you're going to have to do some fancy dancing to recover stuff.
I suspect the stuff is still there though I don't have a way yet to
prove it.  (I tried writing a partition table onto a disk with a
filesystem on it and I was able to recover it).

Do you happen to have a spare 2T drive around? Can you borrow one?
If it were me and it really was business data, I'd start out by making
copies of the existing disks using dd.

Would you know what kind of filesystem was on there?
You might be able to  find that in some logs, but I certainly can't in
my ubuntu system.
That is likely to be a critical piece of information.

Keep at it.
Go slow and careful.
Research before doing.

o1bigtenor <[email-masked]>
Aug 31
Reply
to gerry
Greetings

On Sun, Aug 30, 2015 at 11:42 PM, gerry <[email-masked]> wrote:
Cool.  And did you find anything in the logs?

Aug 30 19:51:40 debiantestingbase kernel: [100718.675021] FAT-fs
(md0): utf8 is not a recommended IO charset for FAT filesystems,
filesystem will be case sensitive!
Aug 30 21:35:33 debiantestingbase kernel: [106954.860493] EXT4-fs
(md0): VFS: Can't find ext4 filesystem
Aug 30 21:37:05 debiantestingbase kernel: [107046.607187] EXT4-fs
(md0): VFS: Can't find ext4 filesystem
Aug 30 22:20:33 debiantestingbase kernel: [109656.411356] PM: Syncing
filesystems ... done.

(included another line that likely had nothing to do with md0) above
from kern.log

Aug 28 10:39:56 debiantestingbase udisksd[1777]: Cleaning up mount
point /media/darald/49552036-b46f-4956-ade9-3541a3dd7f0a (device 253:0
no longer exist)

from syslog.3
That UUID was the 'name' of the array

Aug 28 10:39:28 debiantestingbase kernel: [    3.025162] scsi 3:0:0:0:
Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
Aug 28 10:39:28 debiantestingbase kernel: [    3.025768] sd 3:0:0:0:
[sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
Aug 28 10:39:28 debiantestingbase kernel: [    3.026296] sd 3:0:0:0:
[sdb] Write Protect is off
Aug 28 10:39:28 debiantestingbase kernel: [    3.026304] sd 3:0:0:0:
[sdb] Mode Sense: 00 3a 00 00
Aug 28 10:39:28 debiantestingbase kernel: [    3.026484] sd 3:0:0:0:
[sdb] Write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
Aug 28 10:39:28 debiantestingbase kernel: [    3.028273]  sdb: sdb1
Aug 28 10:39:28 debiantestingbase kernel: [    3.028698] sd 3:0:0:0:
[sdb] Attached SCSI disk
Aug 28 10:39:28 debiantestingbase kernel: [    3.086532] Switched to
clocksource tsc
Aug 28 10:39:28 debiantestingbase kernel: [    3.168919] md: bind<sdb1>
Aug 28 10:39:28 debiantestingbase kernel: [    3.342285] ata5: SATA
link up 3.0 Gbps (SStatus 123 SControl 300)
Aug 28 10:39:28 debiantestingbase kernel: [    3.343153] ata5.00:
ATA-9: ST1000DM003-1ER162, CC45, max UDMA/133
Aug 28 10:39:28 debiantestingbase kernel: [    3.343158] ata5.00:
1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
Aug 28 10:39:28 debiantestingbase kernel: [    3.344067] ata5.00:
configured for UDMA/133
Aug 28 10:39:28 debiantestingbase kernel: [    3.344255] scsi 4:0:0:0:
Direct-Access     ATA      ST1000DM003-1ER1 CC45 PQ: 0 ANSI: 5
Aug 28 10:39:28 debiantestingbase kernel: [    3.344627] sd 4:0:0:0:
[sdc] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
Aug 28 10:39:28 debiantestingbase kernel: [    3.344631] sd 4:0:0:0:
[sdc] 4096-byte physical blocks
Aug 28 10:39:28 debiantestingbase kernel: [    3.344823] sd 4:0:0:0:
[sdc] Write Protect is off
Aug 28 10:39:28 debiantestingbase kernel: [    3.344831] sd 4:0:0:0:
[sdc] Mode Sense: 00 3a 00 00
Aug 28 10:39:28 debiantestingbase kernel: [    3.344946] sd 4:0:0:0:
[sdc] Write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
Aug 28 10:39:28 debiantestingbase kernel: [    3.411364]  sdc: sdc1
Aug 28 10:39:28 debiantestingbase kernel: [    3.412317] sd 4:0:0:0:
[sdc] Attached SCSI disk
Aug 28 10:39:28 debiantestingbase kernel: [    3.501080] md: bind<sdc1>
Aug 28 10:39:28 debiantestingbase kernel: [    3.662509] ata6: SATA
link up 3.0 Gbps (SStatus 123 SControl 300)
Aug 28 10:39:28 debiantestingbase kernel: [    3.674767] ata6.00:
ATA-8: Corsair Force 3 SSD, 1.3.3, max UDMA/133
Aug 28 10:39:28 debiantestingbase kernel: [    3.674772] ata6.00:
468862128 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
Aug 28 10:39:28 debiantestingbase kernel: [    3.684647] ata6.00:
configured for UDMA/133
Aug 28 10:39:28 debiantestingbase kernel: [    3.684933] scsi 5:0:0:0:
Direct-Access     ATA      Corsair Force 3  3    PQ: 0 ANSI: 5
Aug 28 10:39:28 debiantestingbase kernel: [    3.685504] sd 5:0:0:0:
[sdd] 468862128 512-byte logical blocks: (240 GB/223 GiB)
Aug 28 10:39:28 debiantestingbase kernel: [    3.685975] sd 5:0:0:0:
[sdd] Write Protect is off
Aug 28 10:39:28 debiantestingbase kernel: [    3.685983] sd 5:0:0:0:
[sdd] Mode Sense: 00 3a 00 00
Aug 28 10:39:28 debiantestingbase kernel: [    3.686186] sd 5:0:0:0:
[sdd] Write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
Aug 28 10:39:28 debiantestingbase kernel: [    3.688051]  sdd: sdd1
sdd4 < sdd5 sdd6 sdd7 sdd8 sdd9 sdd10 >
Aug 28 10:39:28 debiantestingbase kernel: [    3.689305] sd 5:0:0:0:
[sdd] Attached SCSI disk
Aug 28 10:39:28 debiantestingbase kernel: [    4.002712] ata8: SATA
link down (SStatus 0 SControl 300)
Aug 28 10:39:28 debiantestingbase kernel: [    4.003107] scsi 8:0:0:0:
Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
Aug 28 10:39:28 debiantestingbase kernel: [    4.003597] sd 8:0:0:0:
[sde] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
Aug 28 10:39:28 debiantestingbase kernel: [    4.003843] scsi 9:0:0:0:
Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
Aug 28 10:39:28 debiantestingbase kernel: [    4.003975] sd 8:0:0:0:
[sde] Write Protect is off
Aug 28 10:39:28 debiantestingbase kernel: [    4.003980] sd 8:0:0:0:
[sde] Mode Sense: 00 3a 00 00
Aug 28 10:39:28 debiantestingbase kernel: [    4.004090] sd 8:0:0:0:
[sde] Write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
Aug 28 10:39:28 debiantestingbase kernel: [    4.004478] sd 9:0:0:0:
[sdf] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
Aug 28 10:39:28 debiantestingbase kernel: [    4.004645] sd 9:0:0:0:
[sdf] Write Protect is off
Aug 28 10:39:28 debiantestingbase kernel: [    4.004650] sd 9:0:0:0:
[sdf] Mode Sense: 00 3a 00 00
Aug 28 10:39:28 debiantestingbase kernel: [    4.004737] sd 9:0:0:0:
[sdf] Write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
Aug 28 10:39:28 debiantestingbase kernel: [    4.004778] scsi
15:0:0:0: Processor         Marvell  91xx Config      1.01 PQ: 0 ANSI:
5
Aug 28 10:39:28 debiantestingbase kernel: [    4.006375]  sdf: sdf1
Aug 28 10:39:28 debiantestingbase kernel: [    4.006967] sd 9:0:0:0:
[sdf] Attached SCSI disk
Aug 28 10:39:28 debiantestingbase kernel: [    4.008855]  sde: sde1
Aug 28 10:39:28 debiantestingbase kernel: [    4.009704] sd 8:0:0:0:
[sde] Attached SCSI disk
Aug 28 10:39:28 debiantestingbase kernel: [    4.018710] ata16.00:
exception Emask 0x1 SAct 0x0 SErr 0x0 action 0x0
Aug 28 10:39:28 debiantestingbase kernel: [    4.018753] ata16.00:
irq_stat 0x40000001
Aug 28 10:39:28 debiantestingbase kernel: [    4.018783] ata16.00: cmd
a0/01:00:00:00:01/00:00:00:00:00/a0 tag 1 dma 16640 in
Aug 28 10:39:28 debiantestingbase kernel: [    4.018783]
Inquiry 12 01 00 00 ff 00res 50/00:00:af:6d:70/00:00:74:00:00/e0 Emask
0x1 (device error)
Aug 28 10:39:28 debiantestingbase kernel: [    4.018868] ata16.00:
status: { DRDY }
Aug 28 10:39:28 debiantestingbase kernel: [    4.125325] random:
nonblocking pool is initialized
Aug 28 10:39:28 debiantestingbase kernel: [    4.125530] md: bind<sde1>
Aug 28 10:39:28 debiantestingbase kernel: [    4.142140] md: bind<sdf1>
Aug 28 10:39:28 debiantestingbase kernel: [    4.144984] md: raid10
personality registered for level 10
Aug 28 10:39:28 debiantestingbase kernel: [    4.145397]
md/raid10:md0: active with 4 out of 4 devices
Aug 28 10:39:28 debiantestingbase kernel: [    4.145440] md0: detected
capacity change from 0 to 2000403038208
Aug 28 10:39:28 debiantestingbase kernel: [    4.208978]  md0:
Aug 28 10:39:28 debiantestingbase kernel: [    4.479305]
device-mapper: uevent: version 1.0.3
Aug 28 10:39:28 debiantestingbase kernel: [    4.479536]
device-mapper: ioctl: 4.30.0-ioctl (2014-12-22) initialised:
[email-masked]

from syslog3

The dmsetup output is interesting. I would have expected some dm devices.
Without dm devices, there are no /dev/dm things to mount so that's why
your old Debian7 command doesn't work.

Did you expect there to be a partition table on /dev/md0?

I know that there were partitions on each of the discs (1 each) but I
don't remember any partitions on md0.

Or did you expect to be able to just mount /dev/md0?

That was how it had been working after installing Debian8.

(was there a filesystem on /dev/md0 or was it a  partitioned logical
disk mirror?)

Not sure. Sorry don't know how to answer that one.

You might be able to figure that out from the logs too.

I think the thing went to read-only to protect itself.

If I were there, I'd run these commands and see what they say.
parted /dev/sdb1 list

 # parted /dev/sdb1 list
Usage: parted [OPTION]... [DEVICE [COMMAND [PARAMETERS]...]...]
Apply COMMANDs with PARAMETERS to DEVICE.  If no COMMAND(s) are given, run in
interactive mode.

I'm quite sure that that information isn't really useful. (Won't run
command for the other 3 because would likely get the same result.)

parted /dev/sdc1 list
parted /dev/sde1 list
parted /dev/sdf1 list

Before trying to "fix" the array, you will want to see if you can
figure out if you still have one good setup.

That mirror setup likely mirrors sdb1 with sdc1 and sde1 with sdf1.
And the stripes were acrosss those two mirrors.
To see if an individual disk got corrupted, try this:
(This is non-destructive.)
for a in /dev/sd{b,c,e,f}1
do
  echo -n $a "  "
  dd skip=10 if=$a bs=1M count=10 status=none| md5sum
done

 # for a in /dev/sd{b,c,e,f}1
> do
>   echo -n $a "  "
>   dd skip=10 if=$a bs=1M count=10 status=none| md5sum
> done
/dev/sdb1   f1c9645dbc14efddc7d8a322685f26eb  -
/dev/sdc1   f1c9645dbc14efddc7d8a322685f26eb  -
/dev/sde1   979c1a460f2130d7fd0e9210e2d15e90  -
/dev/sdf1   979c1a460f2130d7fd0e9210e2d15e90  -

I made a dummy setup with loop devices and I see this:
for a in /dev/loop*p1
do
  echo -n $a "  ";dd skip=10 if=$a bs=1M count=10 status=none| md5sum
done
/dev/loop1p1   58b27e44c48997bc126eec1eec12ab8b  -
/dev/loop2p1   58b27e44c48997bc126eec1eec12ab8b  -
/dev/loop3p1   5aa8e9cfbcc3f6612f839b9a96da41df  -
/dev/loop4p1   5aa8e9cfbcc3f6612f839b9a96da41df  -

See how 1 and 2 are identical? And 3 and 4 are identical?
That's the mirroring.
Let's see if yours is similar.

Looks very similar to me.
....

If you don't see errors in the log, then it's fairly likely that
something actually wrote a partition table onto /dev/md0. Otherwise
it's wouldn't see
Disklabel type: gpt
Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448

In which case you're going to have to do some fancy dancing to recover stuff.
I suspect the stuff is still there though I don't have a way yet to
prove it.  (I tried writing a partition table onto a disk with a
filesystem on it and I was able to recover it).

Do you happen to have a spare 2T drive around? Can you borrow one?
If it were me and it really was business data, I'd start out by making
copies of the existing disks using dd.

Not really but I have a NAS box that I am working to setting up and
have 3 TB drives in there.

I would just pull on of those and then do the copying thing.

Did a dd transfer of a 1 TB drive and that took 40 hours - - - so its
going to be a real slow boat to china for that one.

There only was about 80 or 100 GB of actual data on the array, would
there be any way of just copying data and not empty space?

Would you know what kind of filesystem was on there?

The partitions were setup as ext4 (IIRC).

Is there any way of checking that?

You might be able to  find that in some logs, but I certainly can't in
my ubuntu system.
That is likely to be a critical piece of information.

Keep at it.
Go slow and careful.
Research before doing.

This is why I have been very cautious.

Thank you for the help!

Dee

gerry <[email-masked]>
Aug 31
Reply
to me
Cool.  Good info.
So yes it was a filesystem and not a partition and it was ext4.

So it seems pretty likely that sde went bad and took the array with it.

Aug 28 10:39:28 debiantestingbase kernel: [    4.009704] sd 8:0:0:0:
[sde] Attached SCSI disk
Aug 28 10:39:28 debiantestingbase kernel: [    4.018710] ata16.00:
exception Emask 0x1 SAct 0x0 SErr 0x0 action 0x0
Aug 28 10:39:28 debiantestingbase kernel: [    4.018753] ata16.00:
irq_stat 0x40000001
Aug 28 10:39:28 debiantestingbase kernel: [    4.018783] ata16.00: cmd
a0/01:00:00:00:01/00:00:00:00:00/a0 tag 1 dma 16640 in
Aug 28 10:39:28 debiantestingbase kernel: [    4.018783]
Inquiry 12 01 00 00 ff 00res 50/00:00:af:6d:70/00:00:74:00:00/e0 Emask
0x1 (device
error)

By that I mean that the array happily synced all the bad data back up.
You can see that these are nicely synced.  :-(
/dev/sdb1   f1c9645dbc14efddc7d8a322685f26eb  -
/dev/sdc1   f1c9645dbc14efddc7d8a322685f26eb  -
/dev/sde1   979c1a460f2130d7fd0e9210e2d15e90  -
/dev/sdf1   979c1a460f2130d7fd0e9210e2d15e90  -

I would expect that you could fsck the device and mostly recover it.
But I'd still make a backup. You only have to backup two of them.
sdb1 and sdf1.
Remove the md device using
mdadm --stop /dev/md0

mdadm --remove /dev/md0
(but that one doesn't seem to do anything for me, so I'm not all that
clear about it.)

To backup, I'd try something like this and watch how fast it happens.

Then read and compress the filesystem onto a different disk.
Don't do raw copy directly to the hard drive.
Parition and make a file system on it. If you mount it on /mnt/newbackupdisk

Then a likely command to extract the data, compress it and write it to
the backup disk would be:
   dd if=/dev/sdc1 bs=1M | gzip > /mnt/newbackupdisk/sdc1.img.gz
   dd if=/dev/sdf1 bs=1M | gzip > /mnt/newbackupdisk/sdc1.img.gz

While dd is running, you can use ps -ef | grep dd to find it's PID and send
a signal to it to let it tell you its progress.

Paraphrasing  from the dd man page:

Sending a USR1 signal to a running 'dd' process makes it print I/O
statistics to standard error and then resume copying.
if pid is your process id, then
   kill -USR1 $pid
Returns a line like this:
   18335302+0 records in 18335302+0 records out 9387674624 bytes (9.4
GB) copied, 34.6279 seconds, 271 MB/s
And you can just do that over and over as your copy runs.

Once you have a backup, put the md0 device back together.
Either manually or by rebooting your machine.
You might want to remove the /dev/sde drive entirely from the machine.

You could use
   smartctl -a /dev/sde
To get a serial number of the disk so you know you unplug the right one.

For the fsck, make sure you are telling it that you are doing ext4.
fsck.ext4.

BTW my command
    parted /dev/sdc1 list
should have been
    parted /dev/sdc1 print
But it's moot now anyway.

Keep on keepin on.
Gerry

--
[email-masked]

o1bigtenor <[email-masked]>
Aug 31
Reply
to ghaverla
Forwarded conversation
Subject: computer questions
------------------------

From: o1bigtenor <[email-masked]>
Date: Sat, Aug 29, 2015 at 5:38 AM
To: TCLUG Mailing List <[email-masked]>

Greetings

I don't know what's going on but this is weird!

System has been running Debian 8 with things going good (raid mounts
so when saving a file after a reboot all I need to do is click on the
listed array give the root password and bob's your uncle).

(Its a uefi setup)
Well this morning - - -
fdisk -l shows the array
its not mounted

the command I used to use for mounting in Debian 7 no longer works

so - - - - any suggestions?

TIA

Dee

----------
From: Marc Skinner <[email-masked]>
Date: Sat, Aug 29, 2015 at 11:04 AM
To: TCLUG Mailing List <[email-masked]>

Is this a hardware raid or software raid?  More details?  Is the raid
array internal or externally connected?

Thanks.
_______________________________________________
TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
[email-masked]
http://mailman.mn-linux.org/mailman/listinfo/tclug-list

_______________________________________________
TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
[email-masked]
http://mailman.mn-linux.org/mailman/listinfo/tclug-list

----------
From: o1bigtenor <[email-masked]>
Date: Sat, Aug 29, 2015 at 11:41 AM
To: TCLUG Mailing List <[email-masked]>

On Sat, Aug 29, 2015 at 11:04 AM, Marc Skinner <[email-masked]> wrote:
Is this a hardware raid or software raid?  More details?  Is the raid
array internal or externally connected?

software raid
internal

# cat /proc/mdstat
Personalities : [raid10]
md0 : active (auto-read-only) raid10 sde1[0] sdf1[1] sdc1[4] sdb1[3]
      1953518592 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]

# mdadm --examine --scan
ARRAY /dev/md/0  metadata=1.2 UUID=79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
name=debianbase:0

from blkid

/dev/md0: PTUUID="1eb47793-0cdf-4e16-ae84-33ec825ac448" PTTYPE="gpt"

from fdisk -l

Disk /dev/md0: 1.8 TiB, 2000403038208 bytes, 3907037184 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
Disklabel type: gpt
Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448

Thanks.

On 08/29/2015 05:38 AM, o1bigtenor wrote:
Greetings

I don't know what's going on but this is weird!

System has been running Debian 8 with things going good (raid mounts so
when saving a file after a reboot all I need to do is click on the
listed array give the root password and bob's your uncle).

(Its a uefi setup)
Well this morning - - -
fdisk -l shows the array
its not mounted

the command I used to use for mounting in Debian 7 no longer works

so - - - - any suggestions?

TIA

Dee

_______________________________________________
TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
[email-masked]
http://mailman.mn-linux.org/mailman/listinfo/tclug-list

_______________________________________________
TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
[email-masked]
http://mailman.mn-linux.org/mailman/listinfo/tclug-list

----------
From: Marc Skinner <[email-masked]>
Date: Sat, Aug 29, 2015 at 11:47 PM
To: TCLUG Mailing List <[email-masked]>

So what is the problem ... it won't mount?

Did you do an fsck on it? Assuming you had a filesystem that worked on
it prior to your issue.

On 08/29/2015 11:41 AM, o1bigtenor wrote:

On Sat, Aug 29, 2015 at 11:04 AM, Marc Skinner <[email-masked]
        [email-masked] <mailto:[email-masked]>
        http://mailman.mn-linux.org/mailman/listinfo/tclug-list

    _______________________________________________
    TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
    [email-masked] <mailto:[email-masked]>
    http://mailman.mn-linux.org/mailman/listinfo/tclug-list

_______________________________________________
TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
[email-masked]
http://mailman.mn-linux.org/mailman/listinfo/tclug-list

----------
From: o1bigtenor <[email-masked]>
Date: Sun, Aug 30, 2015 at 6:25 AM
To: TCLUG Mailing List <[email-masked]>

On Sat, Aug 29, 2015 at 11:47 PM, Marc Skinner <[email-masked]> wrote:
So what is the problem ... it won't mount?

Did you do an fsck on it? Assuming you had a filesystem that worked on
it prior to your issue.

# fsck /dev/md0
fsck from util-linux 2.26.2
e2fsck 1.42.13 (17-May-2015)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/md0

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

Did some reading on fsck and there are warnings of data loss in using
it to repair.

As this is a data raid (with business data on it) data loss and
corruption is not
an option I want to pursue.

Any ideas on how to re-mount the array?

TIA

Dee

----------
From: gerry <[email-masked]>
Date: Sun, Aug 30, 2015 at 12:34 PM
To: o1bigtenor <[email-masked]>

A few thoughts:

Your prior email shows that the md was set to read-only, right?
md0 : active (auto-read-only) raid10 sde1[0] sdf1[1] sdc1[4] sdb1[3]

Look in your logs for reasons why it got set to auto-read-only.
This isn't your issue but it's something to read:
http://unix.stackexchange.com/questions/101072/new-md-array-is-auto-read-only-and-has-resync-pending

That email also showed some output from
    fdisk -l /dev/md0.
Is there any chance you partitioned /dev/md0?
Because the output says there is a gpt partition table on the /dev/md0 device.

Quoting:
Disk /dev/md0: 1.8 TiB, 2000403038208 bytes, 3907037184 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
Disklabel type: gpt
Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448

Was there anything after that?
Run
    parted -l

You made a statement about booting that was confusing to me.
Here's the section:

        System has been running Debian 8 with things going good (raid
        mounts so
        when saving a file after a reboot all I need to do is click on the
        listed array give the root password and bob's your uncle).
[snip]

        the command I used to use for mounting in Debian 7 no longer works

What does that mean "click on the listed array give the root password" ?
Where were you doing such a thing?  Did you by any chance make
/dev/md0 into an encrypted device?

What is the command you "used to use for mounting in Debian 7" ?

Gerry

--
[email-masked]

----------
From: o1bigtenor <[email-masked]>
Date: Sun, Aug 30, 2015 at 5:38 PM
To: gerry <[email-masked]>

On Sun, Aug 30, 2015 at 12:34 PM, gerry <[email-masked]> wrote:
A few thoughts:

Your prior email shows that the md was set to read-only, right?
md0 : active (auto-read-only) raid10 sde1[0] sdf1[1] sdc1[4] sdb1[3]

Look in your logs for reasons why it got set to auto-read-only.
This isn't your issue but it's something to read:
http://unix.stackexchange.com/questions/101072/new-md-array-is-auto-read-only-and-has-resync-pending

Any ideas which log to look into?

That email also showed some output from
    fdisk -l /dev/md0.
Is there any chance you partitioned /dev/md0?

(Its over 3 years ago.) All four discs were partitioned (sdb1, sdc1 etc).

Because the output says there is a gpt partition table on the /dev/md0 device.

Quoting:
Disk /dev/md0: 1.8 TiB, 2000403038208 bytes, 3907037184 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
Disklabel type: gpt
Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448

Was there anything after that?
Run
    parted -l

Model: ATA ST31000524AS (scsi)
Disk /dev/sdf: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1000GB  1000GB  primary               raid

Model: Linux Software RAID Array (md)
Disk /dev/md0: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start  End  Size  File system  Name  Flags

That's the information for the last disc (in the array) and the array using
parted -l

You made a statement about booting that was confusing to me.
Here's the section:

        System has been running Debian 8 with things going good (raid
        mounts so
        when saving a file after a reboot all I need to do is click on the
        listed array give the root password and bob's your uncle).
[snip]

        the command I used to use for mounting in Debian 7 no longer works

What does that mean "click on the listed array give the root password" ?

Am using LXDE and the file manager is PCManFM.

The file manager would bring up a representation where the array was listed as a
device. After a boot clicking on the button a dialog box would open which asked
for the password for root. Upon entry access was open to the array.

Where were you doing such a thing?  Did you by any chance make
/dev/md0 into an encrypted device?

As far as I know I didn't.

What is the command you "used to use for mounting in Debian 7" ?

mount /dev/dm-0 /home/myspace/RAID

My guess is that I managed to find some kind of a bug that descends
with a combined use
of suspend (upon night close down on the system), mdadm, lxde and pcmanfm.

Thanks for the ideas.

Dee

----------
From: gerry <[email-masked]>
Date: Sun, Aug 30, 2015 at 9:57 PM
To: o1bigtenor <[email-masked]>

Any ideas which log to look into?

Look in all of them.  Everything in /var/log/
Use commands like

cd /var/log
grep read * | grep only

and for the compressed files
zgrep read *.gz | grep only

Then for those files that look interesting, use less or zless to view them.
(The "q" key will exit less or zless)

You'll need to be root do all that.

That's the information for the last disc (in the array) and the array using
parted -l

<sigh>
I can't tell from that whether you sent me the entire output of parted
-l or you were picking and choosing.

For all these commands, run them and send the entire output. Never
skimp on information when debugging.

mount /dev/dm-0 /home/myspace/RAID
Hmm. In debian 7 you mounted /dev/dm-0 but that's just a devmapper
device and they are dynamic.  More about devmapper devices can be
gotten like this:

ls -l /dev/mapper
dmsetup ls
dmestup table

also run (as root, again)
lsblk

Hopefully that will give more information.

--
[email-masked]

On Sun, 30 Aug 2015, o1bigtenor wrote:

----------
From: o1bigtenor <[email-masked]>
Date: Sun, Aug 30, 2015 at 10:11 PM
To: gerry <[email-masked]>

----------
From: gerry <[email-masked]>
Date: Sun, Aug 30, 2015 at 11:42 PM
To: o1bigtenor <[email-masked]>

Cool.  And did you find anything in the logs?

The dmsetup output is interesting. I would have expected some dm devices.
Without dm devices, there are no /dev/dm things to mount so that's why
your old Debian7 command doesn't work.

Did you expect there to be a partition table on /dev/md0?
Or did you expect to be able to just mount /dev/md0?
(was there a filesystem on /dev/md0 or was it a  partitioned logical
disk mirror?)

You might be able to figure that out from the logs too.

I think the thing went to read-only to protect itself.

If I were there, I'd run these commands and see what they say.
parted /dev/sdb1 list
parted /dev/sdc1 list
parted /dev/sde1 list
parted /dev/sdf1 list

Before trying to "fix" the array, you will want to see if you can
figure out if you still have one good setup.

That mirror setup likely mirrors sdb1 with sdc1 and sde1 with sdf1.
And the stripes were acrosss those two mirrors.
To see if an individual disk got corrupted, try this:
(This is non-destructive.)
for a in /dev/sd{b,c,e,f}1
do
  echo -n $a "  "
  dd skip=10 if=$a bs=1M count=10 status=none| md5sum
done

I made a dummy setup with loop devices and I see this:
for a in /dev/loop*p1
do
  echo -n $a "  ";dd skip=10 if=$a bs=1M count=10 status=none| md5sum
done
/dev/loop1p1   58b27e44c48997bc126eec1eec12ab8b  -
/dev/loop2p1   58b27e44c48997bc126eec1eec12ab8b  -
/dev/loop3p1   5aa8e9cfbcc3f6612f839b9a96da41df  -
/dev/loop4p1   5aa8e9cfbcc3f6612f839b9a96da41df  -

See how 1 and 2 are identical? And 3 and 4 are identical?
That's the mirroring.
Let's see if yours is similar.
....

If you don't see errors in the log, then it's fairly likely that
something actually wrote a partition table onto /dev/md0. Otherwise
it's wouldn't see
Disklabel type: gpt
Disk identifier: 1EB47793-0CDF-4E16-AE84-33EC825AC448

In which case you're going to have to do some fancy dancing to recover stuff.
I suspect the stuff is still there though I don't have a way yet to
prove it.  (I tried writing a partition table onto a disk with a
filesystem on it and I was able to recover it).

Do you happen to have a spare 2T drive around? Can you borrow one?
If it were me and it really was business data, I'd start out by making
copies of the existing disks using dd.

Would you know what kind of filesystem was on there?
You might be able to  find that in some logs, but I certainly can't in
my ubuntu system.
That is likely to be a critical piece of information.

Keep at it.
Go slow and careful.
Research before doing.

Gerry

--
[email-masked]

On Sun, 30 Aug 2015, o1bigtenor wrote:

----------
From: o1bigtenor <[email-masked]>
Date: Mon, Aug 31, 2015 at 7:11 AM
To: gerry <[email-masked]>

----------
From: gerry <[email-masked]>
Date: Mon, Aug 31, 2015 at 8:43 AM
To: o1bigtenor <[email-masked]>

o1bigtenor <[email-masked]>
Aug 31
Reply
to gerry

Sorry - - the hard drive I want to use is showing up as sdg1. That
means that it has a partition on it - - yes?

I don't know how to make a file system on it?

Would you explain this - - please?

Then I will continue the odessey!

Dee

gerry <[email-masked]>
Aug 31
Reply
to me
You bet.

run this to look at it:
   parted /dev/sdg list

and this to make a file system on it:
   mkfs -t ext4 /dev/sdg1

And then mount it.
   mkdir /mnt/backup
   mount /dev/sdg1 /mnt/backup

I see there was a typo in my commands to backup.
So here's a repeat with that fixed.

   mdadm --stop /dev/md0
   dd if=/dev/sdc1 bs=1M | gzip > /mnt/backup/sdc1.img.gz
   dd if=/dev/sdf1 bs=1M | gzip > /mnt/backup/sdf1.img.gz

While dd is running, in another window you can use
   ps -ef | grep dd
to find its PID and send a signal to it to let it tell you its progress.
   kill -USR1 $pid

You can also just do ls -l /mnt/backup and see how big things are getting.

o1bigtenor <[email-masked]>
Aug 31
Reply
to gerry
On Mon, Aug 31, 2015 at 4:17 PM, gerry <[email-masked]> wrote:
You bet.

run this to look at it:
   parted /dev/sdg list

and this to make a file system on it:
   mkfs -t ext4 /dev/sdg1

And then mount it.
   mkdir /mnt/backup
   mount /dev/sdg1 /mnt/backup

I see there was a typo in my commands to backup.
So here's a repeat with that fixed.

   mdadm --stop /dev/md0
   dd if=/dev/sdc1 bs=1M | gzip > /mnt/backup/sdc1.img.gz
   dd if=/dev/sdf1 bs=1M | gzip > /mnt/backup/sdf1.img.gz

While dd is running, in another window you can use
   ps -ef | grep dd
to find its PID and send a signal to it to let it tell you its progress.
   kill -USR1 $pid

You can also just do ls -l /mnt/backup and see how big things are getting.

sdc1 has been copied.
sdf1 is being worked on.

When this is completed then I go to:

Once you have a backup, put the md0 device back together.
Either manually or by rebooting your machine.
You might want to remove the /dev/sde drive entirely from the machine.

You could use
   smartctl -a /dev/sde
To get a serial number of the disk so you know you unplug the right one.

For the fsck, make sure you are telling it that you are doing ext4.
fsck.ext4.

So - - - you wouldn't use sde any more?
(I have a replacement drive on hand but not sure the shop will
warranty something
that they can't get to give errors on command.)

So

1. determine which is sde
2. remove
what do I do then?

Is there any way to check the data on my backup to see if its readable?
(sorry if you include the commands please? (that would be great!)

Thanking you for your assistance.

Dee

gerry <[email-masked]>
Sep 1
Reply
to me
How goes the battle?
You decide to take a nap?  <grin>

--
[email-masked]

o1bigtenor <[email-masked]>
Sep 1
Reply
to gerry
On Tue, Sep 1, 2015 at 12:14 PM, gerry <[email-masked]> wrote:
How goes the battle?
You decide to take a nap?  <grin>

Not getting in a hurry because I've borked up too many things in the
past getting in a hurry.

Had a few questions for you in the last email. Was sort of waiting
until I had some way
of verifying that I had my information before I started playing with md0.

Over to you . . .

gerry <[email-masked]>
Sep 1
Reply
to me
I answered all the questions in your last email.
I'll verify that that got sent. If it didn't I'll send it again.
Or maybe I'll just send it again anyway. :-)

--
[email-masked]

gerry <[email-masked]>
Sep 1
Reply
to me
Wow. You already finished a backup!?
That's cool.  And fast.
How big is it?
(ls -l /mnt/backup/sdc1.img.gz )

Figuring out whether it's readable is an interesting question.
You could double check that they are right with
gzip -dc /mnt/backup/sdc1.img.gz | md5sum
and
cat /dev/sdc1 | md5sum
But that would take quite a while.

Either way, part of a mirrored and striped array is not readable anyway.
The stripes have to be put back together to be readable.

BTW while the backup is running, do dmesg | tail once in a while to
see if as it reads it's finding errors.

I'm not all that certain whether to say you should or should not use sde.
But if you do pull it out, don't replace it before finishing any recovery.
I was just suggesting that you pull it because it definitely had some
kind of error.  I could be off base there though.
Either way, you should identify all the drives now.

Make a list of all the UUIDs from
lsblk -o +UUID
match them up with the serial numbers from smartctl.

Maybe just leave sde in there.
Put the md device back together ( mdadm --examine --scan
Then go back to your original fsck command

fsck.ext4 /dev/md0

And as it asks you if you want to recover -- hopefully it'll ask you
if you want to use a different superblock -- just say yes to the
choices.

With your image backup, if that goes really horribly, you can put the
data from the gz files back onto the disk.

gerry <[email-masked]>
Sep 1
Reply
to me
Turns out I postponed the message because it wasn't quite complete.
I just hit "send" on it, but it's probably still not complete.

o1bigtenor <[email-masked]>
Sep 1
Reply
to gerry
Greetings

On Tue, Sep 1, 2015 at 3:37 PM, gerry <[email-masked]> wrote:
Wow. You already finished a backup!?
That's cool.  And fast.

Backup finished after I crashed out but it was almost done at that time

How big is it?
(ls -l /mnt/backup/sdc1.img.gz )

drwx------ 2 root root       16384 Aug 31 18:11 lost+found
-rw-r--r-- 1 root root 94661812644 Aug 31 21:56 sdc1.img.gz
-rw-r--r-- 1 root root 95414436881 Aug 31 23:48 sdf1.img.gz

Figuring out whether it's readable is an interesting question.
You could double check that they are right with
gzip -dc /mnt/backup/sdc1.img.gz | md5sum
and
cat /dev/sdc1 | md5sum
But that would take quite a while.

Either way, part of a mirrored and striped array is not readable anyway.
The stripes have to be put back together to be readable.

BTW while the backup is running, do dmesg | tail once in a while to
see if as it reads it's finding errors.

Back up all done!

I'm not all that certain whether to say you should or should not use sde.
But if you do pull it out, don't replace it before finishing any recovery.
I was just suggesting that you pull it because it definitely had some
kind of error.  I could be off base there though.
Either way, you should identify all the drives now.

Make a list of all the UUIDs from
lsblk -o +UUID
match them up with the serial numbers from smartctl.

Maybe just leave sde in there.
Put the md device back together ( mdadm --examine --scan
Then go back to your original fsck command

fsck.ext4 /dev/md0

And as it asks you if you want to recover -- hopefully it'll ask you
if you want to use a different superblock -- just say yes to the
choices.

With your image backup, if that goes really horribly, you can put the
data from the gz files back onto the disk.

Well - - - I am trying to re-shingle my house, working on books for
business, and getting ready to pour 2 pads of concrete.

So between all these things and the weather I am sort of busy.

This is a priority but even that doesn't mean that I work at it all
the time. When my brain is on (I'm a morning person) this is something
I work on very early
or in between other things.

I have lots to work on at this point with this and will get back to
you later to let you know how things went.
(At least I assume you would like to know!)

I would like to thank you very very much for your assistance.
(Am thinking of writing up the steps and findings and notes and would
then post it where I asked for help.)

With a great big thank you!!

Dee

gerry <[email-masked]>
Sep 1
Reply
to me
Holy mackerel. You are an ambitious person.
Me not so much. :-)
My carport roof  is in terrible shape.  I've been procrastinating
about it for way too long.

The backup sizes look sane so you probably have a good backup.

I'm glad to help you as others have helped me.

o1bigtenor <[email-masked]>
Sep 1
Reply
to gerry
Greetings

On Tue, Sep 1, 2015 at 5:46 PM, gerry <[email-masked]> wrote:
Holy mackerel. You are an ambitious person.

Oh the pads are for extensions an my barn.

I hope to not only be milking sheep in about 6 1/2 months but also
processing the milk
into fresh cheeses, yogurt and ice cream.

Then I will be real busy!

Me not so much. :-)
My carport roof  is in terrible shape.  I've been procrastinating
about it for way too long.

The backup sizes look sane so you probably have a good backup.

Do you know any way to check the backup to make sure?

I'm glad to help you as others have helped me.

Well - - - thank you very much.

If you are travelling through se Manitoba in the future send me a note
and you are
welcome to stop by for a meal!

Dee

gerry <[email-masked]>
Sep 2
Reply
to me
I did finally send you the email to say how you could check for sure,
but here it is again:

Get an md5sum from the image (uncompressing it on the fly)
   gzip -dc /mnt/backup/sdc1.img.gz | md5sum
and compare that to the md5sum from the raw disk:
   md5sum /dev/sdc1
If they are the same, then your backup is correct.

For a concrete example, I picked my smallest partition and did it.
The transcript follow.

I added some spaces between them for readability, but on my screen
they are one after another.

#root@veeta:/tmp>dd if=/dev/sda1 | gzip > sda1.img.gz
2048+0 records in
2048+0 records out
1048576 bytes (1.0 MB) copied, 0.0634722 s, 16.5 MB/s

#root@veeta:/tmp>ls -l sda1.img.gz
-rw-r--r-- 1 root root 54372 Sep  2 10:09 sda1.img.gz

#root@veeta:/tmp>gzip -dc sda1.img.gz | md5sum
a9144ef256b8038412ea7426e5be198f  -

#root@veeta:/tmp>md5sum /dev/sda1
a9144ef256b8038412ea7426e5be198f  /dev/sda1

And as always there are multiple ways to do things ... this is a little cleaner:

No dd command -- just gzip the device to stdout and send it to a file.
#root@veeta:/tmp>gzip -vc /dev/sda1 > sda1.img.gz
/dev/sda1:       94.8%

#root@veeta:/tmp>ls -l sda1.img.gz
-rw-r--r-- 1 root root 54377 Sep  2 10:11 sda1.img.gz

zcat rather than gzip -dc:
#root@veeta:/tmp>zcat sda1.img.gz | md5sum
a9144ef256b8038412ea7426e5be198f  -

Thanks for the invite!  You're just over border after all. :-)

Sorta funny that Minnesota brags about being the state with 10,000
lakes and Manitoba brags about 10 times that number. :-)

o1bigtenor <[email-masked]>
Sep 3
Reply
to gerry
On Wed, Sep 2, 2015 at 10:23 AM, gerry <[email-masked]> wrote:
I did finally send you the email to say how you could check for sure,
but here it is again:

Get an md5sum from the image (uncompressing it on the fly)
   gzip -dc /mnt/backup/sdc1.img.gz | md5sum
and compare that to the md5sum from the raw disk:
   md5sum /dev/sdc1
If they are the same, then your backup is correct.

Hoo boy - - - A lot of time and the md5 sums are - - - - not the same!

Big time bummer.

Any suggestions?

gerry <[email-masked]>
Sep 3
Reply
to me
I would wonder if it's possible that the disk got written to in between time.
??

I'm glad you checked, though.
Does dmesg | tail show any disk errors?

Did you do mdadm --stop /dev/md0 before making the backup?

I don't know how you would make sure the hard drive is readonly before
doing the backup....
Hmmmm.

--
[email-masked]

o1bigtenor <[email-masked]>
Sep 3
Reply
to gerry
On Thu, Sep 3, 2015 at 9:23 PM, gerry <[email-masked]> wrote:
I would wonder if it's possible that the disk got written to in between time.
??

No idea. I couldn't access it myself - - - I do know that!

I'm glad you checked, though.
Does dmesg | tail show any disk errors?

Will do this one in the morning - - - sorry.

Did you do mdadm --stop /dev/md0 before making the backup?

Yes.

I don't know how you would make sure the hard drive is readonly before
doing the backup....
Hmmmm.

This is being rather interesting isn't it!

Thanks for the help!

Dee

gerry <[email-masked]>
Sep 4
Reply
to me
It's always interesting.

I'm on a road trip to Colorado to visit family and go to a wedding, so
it's a diveresion for me now.

And, of course, while on the trip, my mom wants me to get her laptop
updated to window 10. <eyeroll>

But I do enjoy messing with  computers, so what can I say.
I think I'd rather be in Manitoba, though. :-)

Ger

o1bigtenor <[email-masked]>
Sep 4
Reply
to gerry
On Fri, Sep 4, 2015 at 10:35 AM, gerry <[email-masked]> wrote:
> It's always interesting.
>
> I'm on a road trip to Colorado to visit family and go to a wedding, so it's
> a diveresion for me now.
>
> And, of course, while on the trip, my mom wants me to get her laptop updated
> to window 10. <eyeroll>
>
> But I do enjoy messing with  computers, so what can I say.
> I think I'd rather be in Manitoba, though. :-)
>
Maybe not today.

Its very humid (94%) without raining and its supposed to rain heavily
this evening
at times. If you are the twin cities its beautiful down there. We'll
be in that area the
first weekend in Oct, I hope that you might be fortunate enough to
live in one of
the newer subdivisions where you have access to that multi-gigabit ISP services
(I'm on what I call peasant internet here.)

Have a safe trip and lots of fun at the wedding.

Have you thought of setting her up in Zorin - - I'm told its quite win like.

I've also heard that win 10 is quite a bit of a dog at this time. Not
sure because
I'm not using it - - - still stuck at win 7 (have a few VMs with that).

TTYL

Dee

o1bigtenor <[email-masked]>
Sep 8
Reply
to gerry
Greetings

How went the trip?

On Thu, Sep 3, 2015 at 9:23 PM, gerry <[email-masked]> wrote:
> I would wonder if it's possible that the disk got written to in between
> time.
> ??
>
> I'm glad you checked, though.
> Does dmesg | tail show any disk errors?

# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda3 during installation
UUID=a3fbea19-5b7d-47af-b06c-08c49171211b /               ext4
errors=remount-ro 0       1
# /boot/efi was on /dev/sda1 during installation
UUID=8642-E734  /boot/efi       vfat    umask=0077      0       1
# /home was on /dev/sda5 during installation
UUID=b36dbfc6-6897-4311-aa87-9a958ccad072 /home           ext4
defaults        0       2
# swap was on /dev/sda4 during installation
UUID=354a6e33-5743-4c97-bfb9-fd6f9d4521e6 none            swap    sw
           0       0
# swap was on /dev/sdd10 during installation
UUID=04daef55-ff1b-4fe5-a897-ebe397a145b6 none            swap    sw
           0       0
/dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
# raid10 array
PTUUID=1eb47793-0cdf-4e16-ae84-33ec825ac448 /media/darald/Raid10
   gpt    defaults         0       2
root@debiantestingbase:/media/root# fsck -N /dev/md/0
fsck from util-linux 2.26.2
[/sbin/fsck.ext2 (1) -- /dev/md0] fsck.ext2 /dev/md0
root@debiantestingbase:/media/root# fsck /dev/md0
fsck from util-linux 2.26.2
e2fsck 1.42.13 (17-May-2015)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/md0

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

root@debiantestingbase:/media/root# dmesg | tail
[644039.250935] r8169 0000:0b:00.0 enp11s0: link up
[644039.250943] IPv6: ADDRCONF(NETDEV_CHANGE): enp11s0: link becomes ready
[644042.651578] IPv6: ADDRCONF(NETDEV_UP): wlp9s0: link is not ready
[644042.816581] r8169 0000:0b:00.0 enp11s0: link down
[644042.816598] r8169 0000:0b:00.0 enp11s0: link down
[644042.816631] IPv6: ADDRCONF(NETDEV_UP): enp11s0: link is not ready
[644043.008747] r8169 0000:0b:00.0 enp11s0: link down
[644043.008795] IPv6: ADDRCONF(NETDEV_UP): enp11s0: link is not ready
[644046.068706] r8169 0000:0b:00.0 enp11s0: link up
[644046.068722] IPv6: ADDRCONF(NETDEV_CHANGE): enp11s0: link becomes ready

dmesg doesn't seem to show any disk problems

I am hoping that you have some more ideas up your sleeve.

Need to start working on some financial files. A bunch of information
I use to process those kind of files is on the array.

Please?

TIA

gerry <[email-masked]>
Sep 9
Reply
to me
It was a grand time!
Lovely drive. No breakdowns.
Took two days out and one day back.
Two weddings - one Christiand and one Hindu (sort of :-)  )
4 big meals.

Here are some pics.
http://skerbitz.org/photos/2015Co.small/
All the women got to wear Saaris and get Henna'd.

Back to real life again now. :-)

Gerry

--
[email-masked]

o1bigtenor <[email-masked]>
Sep 9
Reply
to gerry
On Wed, Sep 9, 2015 at 4:47 PM, gerry <[email-masked]> wrote:
> It was a grand time!
> Lovely drive. No breakdowns.
> Took two days out and one day back.
> Two weddings - one Christiand and one Hindu (sort of :-)  )
> 4 big meals.
>
> Here are some pics.
> http://skerbitz.org/photos/2015Co.small/
> All the women got to wear Saaris and get Henna'd.
>
> Back to real life again now. :-)

Glad you had a great time!!

You bet - - - as much fun as weddings are - - - they end and its back to
all that other stuff.

Do you know anything about file system repair?

Dee

gerry <[email-masked]>
Sep 9
Reply
to me
:-)
Well, I didn't fry all my braincells I don't think. :-)

How is your filesystem now?

You should look into a package called "testdisk"

gsker@veeta:~> dpkg -S =photorec
testdisk: /usr/bin/photorec

gsker@veeta:~> aptitude show testdisk
Package: testdisk
State: installed
Automatically installed: no
Version: 6.14-3build2
Priority: optional
Section: universe/admin
Maintainer: Ubuntu Developers <[email-masked]>
Architecture: amd64
Uncompressed Size: 1,238 k
Depends: e2fslibs (>= 1.41.0), libc6 (>= 2.14), libjpeg8 (>= 8c),
libncursesw5 (>= 5.6+20070908), libntfs-3g853, libtinfo5, libuuid1 (>=
2.16), zlib1g (>=
         1:1.1.4), ntfs-3g
Conflicts: testdisk
Description: Partition scanner and disk recovery tool, and PhotoRec
file recovery tool
 TestDisk checks the partition and boot sectors of your disks. It is
very useful in forensics, recovering lost partitions. It works with :
 * DOS/Windows FAT12, FAT16 and FAT32
 * NTFS ( Windows NT/2K/XP )
 * Linux Ext2 and Ext3
 * BeFS ( BeOS )
 * BSD disklabel ( FreeBSD/OpenBSD/NetBSD )
 * CramFS (Compressed File System)
 * HFS and HFS+, Hierarchical File System
 * JFS, IBM's Journaled File System
 * Linux Raid
 * Linux Swap (versions 1 and 2)
 * LVM and LVM2, Linux Logical Volume Manager
 * Netware NSS
 * ReiserFS 3.5 and 3.6
 * Sun Solaris i386 disklabel
 * UFS and UFS2 (Sun/BSD/...)
 * XFS, SGI's Journaled File System

 PhotoRec is file data recovery software designed to recover lost
pictures from digital camera memory or even Hard Disks. It has been
extended to search
 also for non audio/video headers. It searches for following files and
is able to undelete them:
 * Sun/NeXT audio data (.au)
 * RIFF audio/video (.avi/.wav)
 * BMP bitmap (.bmp)
 * bzip2 compressed data (.bz2)
 * Source code written in C (.c)
 * Canon Raw picture (.crw)
 * Canon catalog (.ctg)
 * FAT subdirectory
 * Microsoft Office Document (.doc)
 * Nikon dsc (.dsc)
 * HTML page (.html)
 * JPEG picture (.jpg)
 * MOV video (.mov)
 * MP3 audio (MPEG ADTS, layer III, v1) (.mp3)
 * Moving Picture Experts Group video (.mpg)
 * Minolta Raw picture (.mrw)
 * Olympus Raw Format picture (.orf)
 * Portable Document Format (.pdf)
 * Perl script (.pl)
 * Portable Network Graphics (.png)
 * Raw Fujifilm picture (.raf)
 * Contax picture (.raw)
 * Rollei picture (.rdc)
 * Rich Text Format (.rtf)
 * Shell script (.sh)
 * Tar archive (.tar )
 * Tag Image File Format (.tiff)
 * Microsoft ASF (.wma)
 * Sigma/Foveon X3 raw picture (.x3f)
 * zip archive (.zip)

Also debugfs

--
[email-masked]

o1bigtenor <[email-masked]>
Sep 9
Reply
to gerry
On Wed, Sep 9, 2015 at 6:05 PM, gerry <[email-masked]> wrote:
> :-)
> Well, I didn't fry all my braincells I don't think. :-)
>
> How is your filesystem now?

On the raid array - - - problems!!

TestDisk 6.14, Data Recovery Utility, July 2013
Christophe GRENIER <[email-masked]>
http://www.cgsecurity.org

Disk /dev/md0 - 2000 GB / 1863 GiB - CHS 488379648 2 4
Current partition structure:
     Partition                  Start        End    Size in sectors

 1 P EFI GPT                  0   0  2 488379647   1  4 3907037183

Warning: Bad ending sector (CHS and LBA don't match)
No partition is bootable

*=Primary bootable  P=Primary  L=Logical  E=Extended  D=Deleted

use of testdisk generated the empty space included above.

I'm thinking the bad ending sector is possibly important.

Does this above mean anything to you?

Dee

gerry <[email-masked]>
Sep 9
Reply
to me
none of what is in that email points to problems.
"warning:" -- is not necessarily a  problem.

When you send me output, make sure and include the command you ran to
get that output.

Did you do
testdisk /list
testdisk /debug
 ?

And none of that has anything to do with filesystems.
The partition is not a filesystem.

With that said, I don't think it should show up as an EFI GPT
partition so it MIGHT be a problem.

Have you done the
   fsck.ext4 -f -n  /dev/md0
yet?

o1bigtenor <[email-masked]>
Sep 10
Reply
to gerry
On Wed, Sep 9, 2015 at 9:48 PM, gerry <[email-masked]> wrote:
> none of what is in that email points to problems.
> "warning:" -- is not necessarily a  problem.
>
> When you send me output, make sure and include the command you ran to get
> that output.

Could not include the command because testdisk runs itself covering
the command line.
I set all of the output from running # testdisk
>
> Did you do
> testdisk /list

# testdisk /list
TestDisk 6.14, Data Recovery Utility, July 2013
Christophe GRENIER <[email-masked]>
http://www.cgsecurity.org
Please wait...
Disk /dev/sda - 1000 GB / 931 GiB - CHS 121601 255 63
Sector size:512
Model: ST1000DM003-1ER162, S/N:Z4Y4SPM8, FW:CC45

Disk /dev/sdb - 1000 GB / 931 GiB - CHS 121601 255 63
Sector size:512
Model: ST31000524AS, S/N:9VPE3VX1, FW:JC4B

Disk /dev/sdc - 1000 GB / 931 GiB - CHS 121601 255 63
Sector size:512
Model: ST1000DM003-1ER162, S/N:Z4Y27SPH, FW:CC45

Disk /dev/sdd - 240 GB / 223 GiB - CHS 29185 255 63
Sector size:512
Model: Corsair Force 3 SSD, S/N:11486509000011540460, FW:1.3.3

Disk /dev/sde - 1000 GB / 931 GiB - CHS 121601 255 63
Sector size:512
Model: ST31000524AS, S/N:9VPE44LL, FW:JC4B

Disk /dev/sdf - 1000 GB / 931 GiB - CHS 121601 255 63
Sector size:512
Model: ST31000524AS, S/N:9VPE31PN, FW:JC4B

Disk /dev/md0 - 2000 GB / 1863 GiB - CHS 488379648 2 4
Sector size:512

Disk /dev/sda - 1000 GB / 931 GiB - CHS 121601 255 63
     Partition            Start        End    Size in sectors
 1 P EFI System                  2048     526335     524288
 2 P MS Data                   526336    1050623     524288 [/boot]
 3 P MS Data                  1050624   52250623   51200000 [/]
     ext4 blocksize=4096 Large file Sparse superblock Recover
 4 P Linux Swap              52250624  183322623  131072000 [swap]
 5 P Unknown                183322624  964573183  781250560

Disk /dev/sdb - 1000 GB / 931 GiB - CHS 121601 255 63
     Partition            Start        End    Size in sectors
 1 P Linux RAID               0  32 33 121601  57 56 1953521664 [debianbase:0]
     md 1.x L.Endian Raid 10 - Array Slot : 3 (0, 1, failed, 3, 2,
failed, failed)

Disk /dev/sdc - 1000 GB / 931 GiB - CHS 121601 255 63
     Partition            Start        End    Size in sectors
 1 P Unknown                     2048 1953525134 1953523087 [Linux filesystem]

Disk /dev/sdd - 240 GB / 223 GiB - CHS 29185 255 63
     Partition            Start        End    Size in sectors
 1 * Linux                    0  32 33    31  26 59     497664
     ext2 blocksize=1024 Sparse superblock
 4 E extended                31  59 29 14996 156  8  240418816
 5 L Linux                   31  91 61  2581   0 43   40960000
     ext4 blocksize=4096 Large file Sparse superblock Recover
   X extended              2581   0 44  5130 196 58   40962048
 6 L Linux                 2581  33 13  5130 196 58   40960000
     ext4 blocksize=4096 Large file Sparse superblock Recover
   X extended              5130 196 59  8955  92 33   61442048
 7 L Linux                 5130 229 28  8955  92 33   61440000
     ext4 blocksize=4096 Large file Sparse superblock Recover
   X extended              8955  92 34  9465 106 49    8194048
 8 L Linux                 8955 125  3  9465 106 49    8192000
     ext4 blocksize=4096 Large file Sparse superblock Recover
   X extended              9465 106 50 12015  48  1   40962048
 9 L Linux                 9465 139 19 12015  48  1   40960000
     ext4 blocksize=4096 Large file Sparse superblock
   X extended             12015  48  2 14996 156  8   47896576
10 L Linux Swap           12015  80 34 14996 156  8   47894528
     SWAP2 version 1, pagesize=4096

Disk /dev/sde - 1000 GB / 931 GiB - CHS 121601 255 63
     Partition            Start        End    Size in sectors
 1 P Linux RAID               0  32 33 121601  57 56 1953521664 [debianbase:0]
     md 1.x L.Endian Raid 10 - Array Slot : 0 (0, 1, failed, 3, 2,
failed, failed)

Disk /dev/sdf - 1000 GB / 931 GiB - CHS 121601 255 63
     Partition            Start        End    Size in sectors
 1 P Linux RAID               0  32 33 121601  57 56 1953521664 [debianbase:0]
     md 1.x L.Endian Raid 10 - Array Slot : 1 (0, 1, failed, 3, 2,
failed, failed)

Disk /dev/md0 - 2000 GB / 1863 GiB - CHS 488379648 2 4
     Partition            Start        End    Size in sectors
 1 P EFI GPT                  0   0  2 488379647   1  4 3907037183

> testdisk /debug

This is after choosing md0 and a couple other screens.

TestDisk 6.14, Data Recovery Utility, July 2013
Christophe GRENIER <[email-masked]>
http://www.cgsecurity.org

Disk /dev/md0 - 2000 GB / 1863 GiB - CHS 488379648 2 4
Current partition structure:
     Partition                  Start        End    Size in sectors

 1 P EFI GPT                  0   0  2 488379647   1  4 3907037183

Warning: Bad ending sector (CHS and LBA don't match)
No partition is bootable

>  ?
>
> And none of that has anything to do with filesystems.
> The partition is not a filesystem.
>
> With that said, I don't think it should show up as an EFI GPT partition so
> it MIGHT be a problem.
>
> Have you done the
>    fsck.ext4 -f -n  /dev/md0
> yet?
>
no
Does this command alter anything but the filesystem info?

Thanks for the ideas!

Dee

gerry <[email-masked]>
Sep 10
Reply
to me
It botheres me that you saw scsi errors on sde, but the partition
table for sdc is different.  hmmmm.

It's like something put a file system on it or at least partitioned it.
(And it's always been like that -- an earlier email looked the same)

Getting caught back up here I have some questions.
1. Have you been rebooting this machine?
2. Have you figured out which physical disk is associated with each devices?
3. What device did you put that backup on?

repeating -- smartctl can show you the serial numbers.

As far as recovery goes, I'm unsure.

My approach would be to fail two of the disks and then do a filesystem
recovery on the two that are left.
Earlier you sent me this info:
            /dev/sdb1   f1c9645dbc14efddc7d8a322685f26eb  -
            /dev/sdc1   f1c9645dbc14efddc7d8a322685f26eb  -
            /dev/sde1   979c1a460f2130d7fd0e9210e2d15e90  -
            /dev/sdf1   979c1a460f2130d7fd0e9210e2d15e90  -

So if you were to "fail" sdc1 and sde1, then you should end up with a
simple stripe with no mirroring.

Then if you ran fsck.ext4 -f -n /dev/md0
it should tell you what it thinks is wrong with the filesystem.

You asked:
Does this command alter anything but the filesystem info?

That command (with the -n) should not alter the filesystem info.
-n is a read-only filesystem check.

Gerry

--
[email-masked]

o1bigtenor <[email-masked]>
Sep 10
Reply
to gerry
Greetings

On Thu, Sep 10, 2015 at 12:06 PM, gerry <[email-masked]> wrote:
> It botheres me that you saw scsi errors on sde, but the partition table for
> sdc is different.  hmmmm.
>
> It's like something put a file system on it or at least partitioned it.
> (And it's always been like that -- an earlier email looked the same)
>
> Getting caught back up here I have some questions.
> 1. Have you been rebooting this machine?

No - - - I suspend to ram (sleep) most evenings and then just start
(for the lack
of a better word) it up in the mornings.

> 2. Have you figured out which physical disk is associated with each devices?

Not yet but that would be just a matter of a few minutes with serial numbers in
hand.

> 3. What device did you put that backup on?

The backup was put onto a hdd that I removed from a NAS box that I had just
setup (its a 3 TB drive) so its connected via USB to the main system and is
most often not turned on.

>
> repeating -- smartctl can show you the serial numbers.
>
> As far as recovery goes, I'm unsure.
>
> My approach would be to fail two of the disks and then do a filesystem
> recovery on the two that are left.
> Earlier you sent me this info:
>             /dev/sdb1   f1c9645dbc14efddc7d8a322685f26eb  -
>             /dev/sdc1   f1c9645dbc14efddc7d8a322685f26eb  -
>             /dev/sde1   979c1a460f2130d7fd0e9210e2d15e90  -
>             /dev/sdf1   979c1a460f2130d7fd0e9210e2d15e90  -
>
> So if you were to "fail" sdc1 and sde1, then you should end up with a simple
> stripe with no mirroring.
>
> Then if you ran fsck.ext4 -f -n /dev/md0
> it should tell you what it thinks is wrong with the filesystem.
>
> You asked:
>>
>> Does this command alter anything but the filesystem info?
>
>
> That command (with the -n) should not alter the filesystem info.
> -n is a read-only filesystem check.
>

Just so I understand you I will reiterate:

1. determine hdb, hdc, hde, hdf
2. shut off machine
3. remove hdc and hde (by unplugging the drives)
4. restart machine
5. hopefully file system is now repairable.

I have been able to recover files off of the array so it seems that the
files have not been damaged (as yet).

If I have not understood correctly - - - please correct the
incorrect items!!

Thanking you for your assistance!!!

Dee

o1bigtenor <[email-masked]>
Sep 10
Reply
to gerry
On Thu, Sep 10, 2015 at 12:06 PM, gerry <[email-masked]> wrote:
# fsck.ext4 -f -n /dev/md0
e2fsck 1.42.13 (17-May-2015)
ext2fs_open2: Bad magic number in super-block
fsck.ext4: Superblock invalid, trying backup blocks...
fsck.ext4: Bad magic number in super-block while trying to open /dev/md0

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

Trying both (done previously)

#e2fsck -b 8193 /dev/md0

and

#e2fsck -b 32768 /dev/md0

gives the same information as did the fsck.ext4 -f -n /dev/md0

it seems that the array is fine but the file system (table) or some such
has been corrupted.

This has been a very non-fun event over the last number of days.

Thanking you for your assistance

Dee

gerry <[email-masked]>
Sep 10
Reply
to me
Okay you have me totally confused now.
What did you mean by this:

I have been able to recover files off of the array so it seems that the
files have not been damaged (as yet).

What array?  Not /dev/md0 certainly.... ????
is /dev/md0 mounted?

o1bigtenor <[email-masked]>
Sep 10
Reply
to gerry
On Thu, Sep 10, 2015 at 4:51 PM, gerry <[email-masked]> wrote:
> Okay you have me totally confused now.
> What did you mean by this:
>
>> I have been able to recover files off of the array so it seems that the
>> files have not been damaged (as yet).
>
>
> What array?  Not /dev/md0 certainly.... ????
> is /dev/md0 mounted?
>
>
The command
#vi /etc/fstab produces this:
even though I have not mounted the array (at least I don't think I have).

I get it - - - there is weird stuff going on - - - just wish I
understood this stuff a little better!!!

Dee

gerry <[email-masked]>
Sep 10
Reply
to me
The /etc/fstab file is just a list of things that can be mounted.
What do you mean by "recover filess off of the array"
?
Do you mean you get something at /media/darald/Raid10 ?

PTUUID=1eb47793-0cdf-4e16-ae84-33ec825ac448 /media/darald/Raid10

You can do an ls of that location?

What do you get when you type 'mount' all by itself?
   (It should list everything mounted.)

Dang. I should have noticed that that was the raid10.
But maybe this is the first time I've seen the fstab entry.

How would it get mounted?
Does that mountpoint exist?

o1bigtenor <[email-masked]>
Sep 10
Reply
to gerry
On Thu, Sep 10, 2015 at 5:23 PM, gerry <[email-masked]> wrote:
> The /etc/fstab file is just a list of things that can be mounted.
> What do you mean by "recover filess off of the array"

program called photorec (its with testdisk) . It was finding all kinds
of files but nothing is organized it all just raw files.
> ?
> Do you mean you get something at /media/darald/Raid10 ?
>
>> PTUUID=1eb47793-0cdf-4e16-ae84-33ec825ac448 /media/darald/Raid10
>
>
> You can do an ls of that location?

# ls /media/darald/Raid10/
#

>
> What do you get when you type 'mount' all by itself?
>    (It should list everything mounted.)

# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=8242416,mode=755)
devpts on /dev/pts type devpts
(rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,relatime,size=13191704k,mode=755)
/dev/sda3 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
securityfs on /sys/kernel/security type securityfs
(rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup
(rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs
(rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/blkio type cgroup
(rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/freezer type cgroup
(rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpuset type cgroup
(rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup
(rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup
(rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/devices type cgroup
(rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/perf_event type cgroup
(rw,nosuid,nodev,noexec,relatime,perf_event)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs
(rw,relatime,fd=22,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
/dev/sda5 on /home type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /boot/efi type vfat
(rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=utf8,shortname=mixed,errors=remount-ro)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /run/user/117 type tmpfs
(rw,nosuid,nodev,relatime,size=6595852k,mode=700,uid=117,gid=124)
tmpfs on /run/user/1000 type tmpfs
(rw,nosuid,nodev,relatime,size=6595852k,mode=700,uid=1000,gid=1000)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse
(rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)

>
> Dang. I should have noticed that that was the raid10.
> But maybe this is the first time I've seen the fstab entry.
>
> How would it get mounted?

No idea!

> Does that mountpoint exist?

Yes. (folder exists but there is nothing there!!)

I'm not trying to drive you crazy but this kind of here and not here
stuff is what I really don't get!!

Rund (sic) to you.

Dee

gerry <[email-masked]>
Sep 10
Reply
to me
Ahhh.
Now I understand.
It isn't mounted. But now I wonder why.
There are probably some errors in the log about it.

Yeah. photorec will do that -- just make raw files. It's pretty easy
to figure out what kind of file they are but not a lot of other stuff
about them. :-(

So. The question is -- do you run photorec all the way through, or do
you try forcing fsck to do something.

I'd definitely do more with fsck.ext4 because that won't affect photorec at all.

o1bigtenor <[email-masked]>
Sep 10
Reply
to gerry
On Thu, Sep 10, 2015 at 6:41 PM, gerry <[email-masked]> wrote:
> Ahhh.
> Now I understand.
> It isn't mounted. But now I wonder why.
> There are probably some errors in the log about it.
>
> Yeah. photorec will do that -- just make raw files. It's pretty easy to
> figure out what kind of file they are but not a lot of other stuff about
> them. :-(
>
> So. The question is -- do you run photorec all the way through, or do you
> try forcing fsck to do something.

Photorec ran until I ran out of room on the drive - - - my bad - - -
thought I would have enough room but not the case.

If I'm going to do that I need to get where the backup is on the 3 TB
drive - - - there's lots of room there! (grin!)
>
> I'd definitely do more with fsck.ext4 because that won't affect photorec at
> all.

Waiting for your commands, kind sir!

Dee

gerry <[email-masked]>
Sep 10
Reply
to me
hahaha.
My commands.

Are there particular files that you need?  Perhaps you already have them?

You tried a couple of alternate superblocks.
Perhaps you should try more.
But to figure out what they are you'll have to pretend to format the disk again.
But NOT write to the disk.

Reading the mke2fs man page:

 -n  Causes mke2fs to not actually create a filesystem, but display
what it would do if
     it were to create a filesystem.  This can be used to determine
the location of the
     backup superblocks for a particular filesystem, so long as the
mke2fs  parameters
     that were passed when the filesystem was originally created are
used again.  (With
     the -n option added, of course!)

so try
mke2fs -t ext4 -n /dev/md0

And then try those superblock numbers while running
fsck.ext4 -b XXXXXXX  /dev/md0
(replacing XXXXX with those potential superblocks)

Also, could you send the output of cat /etc/mdstat again?
I just want to know if it's changed since the reboot.

Gerry

Reading material:
http://askubuntu.com/questions/240372/how-to-recover-data-from-ext4-partitions
http://www.cyberciti.biz/faq/linux-find-alternative-superblocks/
http://askubuntu.com/questions/41601/is-there-any-recovery-software-available-for-ext4
http://ext4magic.sourceforge.net/ext4magic_en.html

o1bigtenor <[email-masked]>
Sep 11
Reply
to gerry
On Thu, Sep 10, 2015 at 11:07 PM, gerry <[email-masked]> wrote:
> hahaha.
> My commands.
>
> Are there particular files that you need?  Perhaps you already have them?

It would particular folders. There is at least one folder - - my
business files!!
>
>
> You tried a couple of alternate superblocks.
> Perhaps you should try more.
> But to figure out what they are you'll have to pretend to format the disk
> again.
> But NOT write to the disk.
>
> Reading the mke2fs man page:
>
>  -n  Causes mke2fs to not actually create a filesystem, but display what it
> would do if
>      it were to create a filesystem.  This can be used to determine the
> location of the
>      backup superblocks for a particular filesystem, so long as the  mke2fs
> parameters
>      that were passed when the filesystem was originally created are used
> again.  (With
>      the -n option added, of course!)
>
> so try
> mke2fs -t ext4 -n /dev/md0

/# mke2fs -t ext4 -n /dev/md0
mke2fs 1.42.13 (17-May-2015)
Found a gpt partition table in /dev/md0
Proceed anyway? (y,n) y
Creating filesystem with 488379648 4k blocks and 122101760 inodes
Filesystem UUID: 33081d9b-8ec3-4382-9922-8ca7d6a7050b
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
    102400000, 214990848

>
> And then try those superblock numbers while running
> fsck.ext4 -b XXXXXXX  /dev/md0
> (replacing XXXXX with those potential superblocks)

Tried them ALL - - - got

# fsck.ext4 -b 214990848 /dev/md0
e2fsck 1.42.13 (17-May-2015)
fsck.ext4: Invalid argument while trying to open /dev/md0

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

>
> Also, could you send the output of cat /etc/mdstat again?

# cat /etc/mdstat
cat: /etc/mdstat: No such file or directory

> I just want to know if it's changed since the reboot.

I have not yet gone through the steps that you had asked in the previous email.

Were those to have been done before I went through this excercise?

Over to you.

Dee

gerry <[email-masked]>
Sep 11
Reply
to me
Since
1. it thinks there's a partition table on /dev/md0
2. it can't find useable superblocks ?

I'm starting to think that the only possible recovery is
the photorec path. :-(

Ooooops. I meant
  cat /proc/mdstat
Not /etc/mdstat.

Ger

--
[email-masked]

o1bigtenor <[email-masked]>
Sep 11
Reply
to gerry
On Fri, Sep 11, 2015 at 9:28 AM, gerry <[email-masked]> wrote:
> Since
> 1. it thinks there's a partition table on /dev/md0
> 2. it can't find useable superblocks ?
>
> I'm starting to think that the only possible recovery is
> the photorec path. :-(

When I looked into one of the folders that photrec provided there was
piles of useless stuff in there.

So you don't believe that there is a way to pull folders rather than just files?
>
> Ooooops. I meant
>   cat /proc/mdstat
> Not /etc/mdstat.

# cat /proc/mdstat
Personalities : [raid10]
md0 : active (auto-read-only) raid10 sde1[0] sdb1[3] sdc1[4] sdf1[1]
      1953518592 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]

With a heavy heart

Dee

gerry <[email-masked]>
Sep 11
Reply
to me
I don't believe there is.
What do you mean by "useless stuff"
Just numbered files, eh?
What do you get when you run the
file *
command in the directory.

I see 3 disks. So which one did you pull out?
And how did you figure out which one to pull out?
And what does parted -l say now?

o1bigtenor <[email-masked]>
Sep 11
Reply
to gerry
On Fri, Sep 11, 2015 at 10:43 AM, gerry <[email-masked]> wrote:
> I don't believe there is.
> What do you mean by "useless stuff"
> Just numbered files, eh?
> What do you get when you run the
> file *
> command in the directory.
>
> I see 3 disks. So which one did you pull out?

Sorry - - - I still se four disks listed.

I was having internet issues and so I did a system reboot.

Well - -- I can go in under system repair and use startx to get to a
working system but cannot log in as user.

Checking in /etc/fstab

and I have to type this cause I'm using a different computer

# / was on /dev/sda3 during installation
UUID=a3fbea19-5b7d-47af-b06c-08c49171211b /      ext4
errors=remount-ro  0     1

so now I have even more issues to play with.

Any suggestions on how to proceed?

dee

gerry <[email-masked]>
Sep 11
Reply
to me
Yeah. 4. disks.
So you never excluded any of them and then you started trying to do
destructive changes.

I don't know why you sent me the fstab file entry.

If you are using startx, then you logged in as a user.

You have lost me -- I don't know what the email below means.

If you want your data back you'll have to use the photorec tools and
get what you can but there will be no recovery of directories.

And you'll have to figure out what each of the  numbered files from a
photorec recovery are.  If you are "business" files are of a
particular type, you'll want to make sure that photorec knows about
that type.

o1bigtenor <[email-masked]>
Sep 11
Reply
to gerry
On Fri, Sep 11, 2015 at 11:47 AM, gerry <[email-masked]> wrote:
> Yeah. 4. disks.
> So you never excluded any of them and then you started trying to do
> destructive changes.

I understood that the

fsck ext4 ...

was non descructive only looking for system file sectors

Sorry

Was that incorrect.

>
> I don't know why you sent me the fstab file entry.

I sent you that fstab file because it says that
sda3 /

is not mounted.

How do I repair that?

>
> If you are using startx, then you logged in as a user.
>
System repair says

give root password for log in or to continue

give password

then do
# startx

and I have Xwindow session up but I'm working as root - - -

cannot get to where I would log in as a user.

I am starting to wonder about Debian - - - the reliability is
driving me totally crazy!!!

gerry <[email-masked]>
Sep 11
Reply
to me
the fstab file has nothing to do with what is actually mounted.
Use the mount command for that.

fsck is destructive.
Unless there's a "-n" option to make it not so.

And a third group (two sets here):

o1bigtenor <[email-masked]>
Sep 8
Reply
to Maurice
Greetings

Just in case my email address isn't handy.

Regards

maurice <[email-masked]>
Sep 8
Reply
to me
Hi D

We need the output from:

cat /proc/mdstat

Post your array details (mdadm -D), member
details (mdadm -E) and a summary of what happened.
Include any excerpts from your dmesg and/or syslogs that look like
they might be relevant.

mdadm --detail /dev/mdX
 - Where "X" is the RAID md device #

mdadm --examine /dev/sdXy
 - Where "X" is the RAID md device # and
 - Where "y" is each partition being used in that RAID on device "X"

Also, I suggest using lsdrv for listing the  working drive layouts.
Output of that is also useful.
You may find the lsdrv script here:

https://github.com/pturmel/lsdrv

o1bigtenor <[email-masked]>
Sep 8
Reply
to maurice
On Tue, Sep 8, 2015 at 6:10 PM, maurice <[email-masked]> wrote:
> Hi Darald
>
> We need the output from:
>
> cat /proc/mdstat

# cat /proc/mdstat
Personalities : [raid10]
md0 : active (auto-read-only) raid10 sde1[0] sdb1[3] sdc1[4] sdf1[1]
      1953518592 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]

unused devices: <none>

>
> Post your array details (mdadm -D),

# mdadm -D
mdadm: No devices given.

> member
> details (mdadm -E)

# mdadm -E
mdadm: No devices to examine

> and a summary of what happened.
> Include any excerpts from your dmesg and/or syslogs that look like they
> might be relevant.

Aug 28 10:39:28 debiantestingbase kernel: [    3.025162] scsi 3:0:0:0:
Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
Aug 28 10:39:28 debiantestingbase kernel: [    3.025768] sd 3:0:0:0:
[sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
Aug 28 10:39:28 debiantestingbase kernel: [    3.026296] sd 3:0:0:0:
[sdb] Write Protect is off
Aug 28 10:39:28 debiantestingbase kernel: [    3.026304] sd 3:0:0:0:
[sdb] Mode Sense: 00 3a 00 00
Aug 28 10:39:28 debiantestingbase kernel: [    3.026484] sd 3:0:0:0:
[sdb] Write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
Aug 28 10:39:28 debiantestingbase kernel: [    3.028273]  sdb: sdb1
Aug 28 10:39:28 debiantestingbase kernel: [    3.028698] sd 3:0:0:0:
[sdb] Attached SCSI disk
Aug 28 10:39:28 debiantestingbase kernel: [    3.086532] Switched to
clocksource tsc
Aug 28 10:39:28 debiantestingbase kernel: [    3.168919] md: bind<sdb1>
Aug 28 10:39:28 debiantestingbase kernel: [    3.342285] ata5: SATA
link up 3.0 Gbps (SStatus 123 SControl 300)
Aug 28 10:39:28 debiantestingbase kernel: [    3.343153] ata5.00:
ATA-9: ST1000DM003-1ER162, CC45, max UDMA/133
Aug 28 10:39:28 debiantestingbase kernel: [    3.343158] ata5.00:
1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
Aug 28 10:39:28 debiantestingbase kernel: [    3.344067] ata5.00:
configured for UDMA/133
Aug 28 10:39:28 debiantestingbase kernel: [    3.344255] scsi 4:0:0:0:
Direct-Access     ATA      ST1000DM003-1ER1 CC45 PQ: 0 ANSI: 5
Aug 28 10:39:28 debiantestingbase kernel: [    3.344627] sd 4:0:0:0:
[sdc] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
Aug 28 10:39:28 debiantestingbase kernel: [    3.344631] sd 4:0:0:0:
[sdc] 4096-byte physical blocks
Aug 28 10:39:28 debiantestingbase kernel: [    3.344823] sd 4:0:0:0:
[sdc] Write Protect is off
Aug 28 10:39:28 debiantestingbase kernel: [    3.344831] sd 4:0:0:0:
[sdc] Mode Sense: 00 3a 00 00
Aug 28 10:39:28 debiantestingbase kernel: [    3.344946] sd 4:0:0:0:
[sdc] Write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
Aug 28 10:39:28 debiantestingbase kernel: [    3.411364]  sdc: sdc1
Aug 28 10:39:28 debiantestingbase kernel: [    3.412317] sd 4:0:0:0:
[sdc] Attached SCSI disk
Aug 28 10:39:28 debiantestingbase kernel: [    3.501080] md: bind<sdc1>
Aug 28 10:39:28 debiantestingbase kernel: [    3.662509] ata6: SATA
link up 3.0 Gbps (SStatus 123 SControl 300)
Aug 28 10:39:28 debiantestingbase kernel: [    3.674767] ata6.00:
ATA-8: Corsair Force 3 SSD, 1.3.3, max UDMA/133
Aug 28 10:39:28 debiantestingbase kernel: [    3.674772] ata6.00:
468862128 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
Aug 28 10:39:28 debiantestingbase kernel: [    3.684647] ata6.00:
configured for UDMA/133
Aug 28 10:39:28 debiantestingbase kernel: [    3.684933] scsi 5:0:0:0:
Direct-Access     ATA      Corsair Force 3  3    PQ: 0 ANSI: 5
Aug 28 10:39:28 debiantestingbase kernel: [    3.685504] sd 5:0:0:0:
[sdd] 468862128 512-byte logical blocks: (240 GB/223 GiB)
Aug 28 10:39:28 debiantestingbase kernel: [    3.685975] sd 5:0:0:0:
[sdd] Write Protect is off
Aug 28 10:39:28 debiantestingbase kernel: [    3.685983] sd 5:0:0:0:
[sdd] Mode Sense: 00 3a 00 00
Aug 28 10:39:28 debiantestingbase kernel: [    3.686186] sd 5:0:0:0:
[sdd] Write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
Aug 28 10:39:28 debiantestingbase kernel: [    3.688051]  sdd: sdd1
sdd4 < sdd5 sdd6 sdd7 sdd8 sdd9 sdd10 >
Aug 28 10:39:28 debiantestingbase kernel: [    3.689305] sd 5:0:0:0:
[sdd] Attached SCSI disk
Aug 28 10:39:28 debiantestingbase kernel: [    4.002712] ata8: SATA
link down (SStatus 0 SControl 300)
Aug 28 10:39:28 debiantestingbase kernel: [    4.003107] scsi 8:0:0:0:
Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
Aug 28 10:39:28 debiantestingbase kernel: [    4.003597] sd 8:0:0:0:
[sde] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
Aug 28 10:39:28 debiantestingbase kernel: [    4.003843] scsi 9:0:0:0:
Direct-Access     ATA      ST31000524AS     JC4B PQ: 0 ANSI: 5
Aug 28 10:39:28 debiantestingbase kernel: [    4.003975] sd 8:0:0:0:
[sde] Write Protect is off
Aug 28 10:39:28 debiantestingbase kernel: [    4.003980] sd 8:0:0:0:
[sde] Mode Sense: 00 3a 00 00
Aug 28 10:39:28 debiantestingbase kernel: [    4.004090] sd 8:0:0:0:
[sde] Write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
Aug 28 10:39:28 debiantestingbase kernel: [    4.004478] sd 9:0:0:0:
[sdf] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
Aug 28 10:39:28 debiantestingbase kernel: [    4.004645] sd 9:0:0:0:
[sdf] Write Protect is off
Aug 28 10:39:28 debiantestingbase kernel: [    4.004650] sd 9:0:0:0:
[sdf] Mode Sense: 00 3a 00 00
Aug 28 10:39:28 debiantestingbase kernel: [    4.004737] sd 9:0:0:0:
[sdf] Write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
Aug 28 10:39:28 debiantestingbase kernel: [    4.004778] scsi
15:0:0:0: Processor         Marvell  91xx Config      1.01 PQ: 0 ANSI:
5
Aug 28 10:39:28 debiantestingbase kernel: [    4.006375]  sdf: sdf1
Aug 28 10:39:28 debiantestingbase kernel: [    4.006967] sd 9:0:0:0:
[sdf] Attached SCSI disk
Aug 28 10:39:28 debiantestingbase kernel: [    4.008855]  sde: sde1
Aug 28 10:39:28 debiantestingbase kernel: [    4.009704] sd 8:0:0:0:
[sde] Attached SCSI disk
Aug 28 10:39:28 debiantestingbase kernel: [    4.018710] ata16.00:
exception Emask 0x1 SAct 0x0 SErr 0x0 action 0x0
Aug 28 10:39:28 debiantestingbase kernel: [    4.018753] ata16.00:
irq_stat 0x40000001
Aug 28 10:39:28 debiantestingbase kernel: [    4.018783] ata16.00: cmd
a0/01:00:00:00:01/00:00:00:00:00/a0 tag 1 dma 16640 in
Aug 28 10:39:28 debiantestingbase kernel: [    4.018783]
Inquiry 12 01 00 00 ff 00res 50/00:00:af:6d:70/00:00:74:00:00/e0 Emask
0x1 (device error)
Aug 28 10:39:28 debiantestingbase kernel: [    4.018868] ata16.00:
status: { DRDY }
Aug 28 10:39:28 debiantestingbase kernel: [    4.125325] random:
nonblocking pool is initialized
Aug 28 10:39:28 debiantestingbase kernel: [    4.125530] md: bind<sde1>
Aug 28 10:39:28 debiantestingbase kernel: [    4.142140] md: bind<sdf1>
Aug 28 10:39:28 debiantestingbase kernel: [    4.144984] md: raid10
personality registered for level 10
Aug 28 10:39:28 debiantestingbase kernel: [    4.145397]
md/raid10:md0: active with 4 out of 4 devices
Aug 28 10:39:28 debiantestingbase kernel: [    4.145440] md0: detected
capacity change from 0 to 2000403038208
Aug 28 10:39:28 debiantestingbase kernel: [    4.208978]  md0:
Aug 28 10:39:28 debiantestingbase kernel: [    4.479305]
device-mapper: uevent: version 1.0.3
Aug 28 10:39:28 debiantestingbase kernel: [    4.479536]
device-mapper: ioctl: 4.30.0-ioctl (2014-12-22) initialised:
[email-masked]

>
>
> mdadm --detail /dev/mdX
>  - Where "X" is the RAID md device #

# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Mon Mar  5 08:26:28 2012
     Raid Level : raid10
     Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
  Used Dev Size : 976759296 (931.51 GiB 1000.20 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Fri Aug 28 10:38:32 2015
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 512K

           Name : debianbase:0
           UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
         Events : 47341

    Number   Major   Minor   RaidDevice State
       0       8       65        0      active sync set-A   /dev/sde1
       1       8       81        1      active sync set-B   /dev/sdf1
       4       8       33        2      active sync set-A   /dev/sdc1
       3       8       17        3      active sync set-B   /dev/sdb1

>
> mdadm --examine /dev/sdXy
>  - Where "X" is the RAID md device # and
>  - Where "y" is each partition being used in that RAID on device "X"

# mdadm --examine /dev/sdb1
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
           Name : debianbase:0
  Creation Time : Mon Mar  5 08:26:28 2012
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
     Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
  Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=1024 sectors
          State : clean
    Device UUID : a80c76db:eaea61af:bcb9cbbb:ac99e467

    Update Time : Fri Aug 28 10:38:32 2015
       Checksum : 29a4fa98 - correct
         Events : 47341

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

# mdadm --examine /dev/sdc1
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
           Name : debianbase:0
  Creation Time : Mon Mar  5 08:26:28 2012
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 1953521039 (931.51 GiB 1000.20 GB)
     Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
  Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1960 sectors, after=2447 sectors
          State : clean
    Device UUID : 840fc99d:6f607ea8:9a4a9b0a:a965c85c

    Update Time : Fri Aug 28 10:38:32 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 28cfd5ab - correct
         Events : 47341

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
root@debiantestingbase:/etc# mdadm --examine /dev/sde1
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
           Name : debianbase:0
  Creation Time : Mon Mar  5 08:26:28 2012
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
     Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
  Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=1024 sectors
          State : clean
    Device UUID : cb37e722:d3a98f88:13439de2:bdad9d2b

    Update Time : Fri Aug 28 10:38:32 2015
       Checksum : 34ce8208 - correct
         Events : 47341

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
root@debiantestingbase:/etc# mdadm --examine /dev/sdf1
/dev/sdf1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
           Name : debianbase:0
  Creation Time : Mon Mar  5 08:26:28 2012
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
     Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
  Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=1024 sectors
          State : clean
    Device UUID : 9e749fa9:a0efe791:ea09d2e2:72b99f6c

    Update Time : Fri Aug 28 10:38:32 2015
       Checksum : 615d736 - correct
         Events : 47341

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

>
> Also, I suggest using lsdrv for listing the  working drive layouts.
> Output of that is also useful.
> You may find the lsdrv script here:
>
> https://github.com/pturmel/lsdrv
>

# ./lsdrv
PCI [ahci] 00:1f.2 SATA controller: Intel Corporation C600/X79 series
chipset 6-Port SATA AHCI Controller (rev 05)
├scsi 0:0:0:0 HL-DT-ST BD-RE  BH12LS38  {K9HBA7I5354}
│└sr0 1.00g [11:0] Empty/Unknown
├scsi 1:x:x:x [Empty]
├scsi 2:0:0:0 ATA      ST1000DM003-1ER1 {Z4Y4SPM8}
│└sda 931.51g [8:0] Partitioned (gpt)
│ ├sda1 256.00m [8:1] vfat {8642-E734}
│ │└Mounted as /dev/sda1 @ /boot/efi
│ ├sda2 256.00m [8:2] Empty/Unknown
│ ├sda3 24.41g [8:3] ext4 {a3fbea19-5b7d-47af-b06c-08c49171211b}
│ │└Mounted as /dev/sda3 @ /
│ ├sda4 62.50g [8:4] swap {354a6e33-5743-4c97-bfb9-fd6f9d4521e6}
│ └sda5 372.53g [8:5] ext4 {b36dbfc6-6897-4311-aa87-9a958ccad072}
│  └Mounted as /dev/sda5 @ /home
├scsi 3:0:0:0 ATA      ST31000524AS     {9VPE3VX1}
│└sdb 931.51g [8:16] Partitioned (dos)
│ └sdb1 931.51g [8:17] MD raid10,near2 (3/4) (w/ sdc1,sde1,sdf1)
in_sync 'debianbase:0' {79baaa2f-0aa2-b9fa-18e2-ea6b6e2846b3}
│  └md0 1.82t [9:0] MD v1.2 raid10,near2 (4) read-auto, 512k Chunk
{79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3}
│                   Partitioned (gpt)
├scsi 4:0:0:0 ATA      ST1000DM003-1ER1 {Z4Y27SPH}
│└sdc 931.51g [8:32] Partitioned (gpt)
│ └sdc1 931.51g [8:33] MD raid10,near2 (2/4) (w/ sdb1,sde1,sdf1)
in_sync 'debianbase:0' {79baaa2f-0aa2-b9fa-18e2-ea6b6e2846b3}
│  └md0 1.82t [9:0] MD v1.2 raid10,near2 (4) read-auto, 512k Chunk
{79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3}
│                   Partitioned (gpt)
└scsi 5:0:0:0 ATA      Corsair Force 3  {11486509000011540460}
 └sdd 223.57g [8:48] Partitioned (dos)
  ├sdd1 243.00m [8:49] ext2 {a0ce336b-35cd-4413-a1c7-e77a1a4e91a6}
  ├sdd4 0.00k [8:52] Empty/Unknown
  ├sdd5 19.53g [8:53] ext4 {bbe77e50-0eaa-4048-934e-9f1b72b157ff}
  ├sdd6 19.53g [8:54] ext4 {533f504b-30a6-4cf6-86e9-56772c87bae3}
  ├sdd7 29.30g [8:55] ext4 {ff6bcbe9-f25f-4738-ba6b-5ea3c2fe0b70}
  ├sdd8 3.91g [8:56] ext4 {2d0d1953-dae6-4e33-8d04-cc685e2e36fb}
  ├sdd9 19.53g [8:57] ext4 {df754d9a-3dcf-470f-8e26-d38ef6f82df0}
  └sdd10 22.84g [8:58] swap {04daef55-ff1b-4fe5-a897-ebe397a145b6}
PCI [ahci] 0a:00.0 SATA controller: ASMedia Technology Inc. ASM1062
Serial ATA Controller (rev 01)
├scsi 6:x:x:x [Empty]
└scsi 7:x:x:x [Empty]
PCI [ahci] 0c:00.0 SATA controller: Marvell Technology Group Ltd.
88SE9128 PCIe SATA 6 Gb/s RAID controller with HyperDuo (rev 11)
├scsi 8:0:0:0 ATA      ST31000524AS     {9VPE44LL}
│└sde 931.51g [8:64] Partitioned (dos)
│ └sde1 931.51g [8:65] MD raid10,near2 (0/4) (w/ sdb1,sdc1,sdf1)
in_sync 'debianbase:0' {79baaa2f-0aa2-b9fa-18e2-ea6b6e2846b3}
│  └md0 1.82t [9:0] MD v1.2 raid10,near2 (4) read-auto, 512k Chunk
{79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3}
│                   Partitioned (gpt)
├scsi 9:0:0:0 ATA      ST31000524AS     {9VPE31PN}
│└sdf 931.51g [8:80] Partitioned (dos)
│ └sdf1 931.51g [8:81] MD raid10,near2 (1/4) (w/ sdb1,sdc1,sde1)
in_sync 'debianbase:0' {79baaa2f-0aa2-b9fa-18e2-ea6b6e2846b3}
│  └md0 1.82t [9:0] MD v1.2 raid10,near2 (4) read-auto, 512k Chunk
{79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3}
│                   Partitioned (gpt)
├scsi 10:x:x:x [Empty]
├scsi 11:x:x:x [Empty]
├scsi 12:x:x:x [Empty]
├scsi 13:x:x:x [Empty]
└scsi 14:x:x:x [Empty]

First time I've ever used git.

I think that this is everything you've asked for.

Thanks for the help!

Darald

maurice <[email-masked]>
Sep 9
Reply
to me
On 9/8/2015 8:37 PM, o1bigtenor wrote:
On Tue, Sep 8, 2015 at 6:10 PM, maurice <[email-masked]> wrote:
Hi Darald

We need the output from:

cat /proc/mdstat
# cat /proc/mdstat
Personalities : [raid10]
md0 : active (auto-read-only) raid10 sde1[0] sdb1[3] sdc1[4] sdf1[1]
      1953518592 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]

Good!
We see that you have one md device: md0
And that it has 4 members.
All are in use and are active.
Looks good.

I still need you to run these two more commands, using the correct md
device, like this:
# mdadm -D /dev/md0

# mdadm -E /dev/md0

Here we have your drive details, from lsdrv.
I see that 3 drives were originally prepared with fdisk, and have
dos/FAT partition tables.
And that 2 were prepared with something else, and have GPT partition tables.
I do not think that is bad, just shows me that they were not all
prepped at the same time.

# ./lsdrv
..      ST1000DM003-1ER1 {Z4Y4SPM8}
│└sda 931.51g [8:0] Partitioned (gpt)
│ ├sda1 256.00m [8:1] vfat {8642-E734}
│ │└Mounted as /dev/sda1 @ /boot/efi
│ ├sda2 256.00m [8:2] Empty/Unknown
│ ├sda3 24.41g [8:3] ext4 {a3fbea19-5b7d-47af-b06c-08c49171211b}
│ │└Mounted as /dev/sda3 @ /
│ ├sda4 62.50g [8:4] swap {354a6e33-5743-4c97-bfb9-fd6f9d4521e6}
│ └sda5 372.53g [8:5] ext4 {b36dbfc6-6897-4311-aa87-9a958ccad072}
│  └Mounted as /dev/sda5 @ /home

├scsi 3:0:0:0 ATA      ST31000524AS     {9VPE3VX1}
│└sdb 931.51g [8:16] Partitioned (dos)
│ └sdb1 931.51g [8:17] MD raid10,near2 (3/4) (w/ sdc1,sde1,sdf1)

├scsi 4:0:0:0 ATA      ST1000DM003-1ER1 {Z4Y27SPH}
│└sdc 931.51g [8:32] Partitioned (gpt)
│ └sdc1 931.51g [8:33] MD raid10,near2 (2/4) (w/ sdb1,sde1,sdf1)

├scsi 8:0:0:0 ATA      ST31000524AS     {9VPE44LL}
│└sde 931.51g [8:64] Partitioned (dos)
│ └sde1 931.51g [8:65] MD raid10,near2 (0/4) (w/ sdb1,sdc1,sdf1)

├scsi 9:0:0:0 ATA      ST31000524AS     {9VPE31PN}
│└sdf 931.51g [8:80] Partitioned (dos)
│ └sdf1 931.51g [8:81] MD raid10,near2 (1/4) (w/ sdb1,sdc1,sde1)

If you don't mind, please run those two more commands.

Then tell me:

WHAT is WRONG here?

o1bigtenor <[email-masked]>
Sep 9
Reply
to maurice
On Wed, Sep 9, 2015 at 7:23 AM, maurice <[email-masked]> wrote:
> On 9/8/2015 8:37 PM, o1bigtenor wrote:
>
> On Tue, Sep 8, 2015 at 6:10 PM, maurice <[email-masked]> wrote:
>
> Hi Darald
>
> We need the output from:
>
> cat /proc/mdstat
>
> # cat /proc/mdstat
> Personalities : [raid10]
> md0 : active (auto-read-only) raid10 sde1[0] sdb1[3] sdc1[4] sdf1[1]
>       1953518592 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
>
> Good!
> We see that you have one md device: md0
> And that it has 4 members.
> All are in use and are active.
> Looks good.
>
>
> I still need you to run these two more commands, using the correct md
> device, like this:
>
> # mdadm -D /dev/md0

# mdadm -D /dev/md0
> # mdadm -E /dev/md0

# mdadm -E /dev/md0
/dev/md0:
   MBR Magic : aa55
Partition[0] :   3907037183 sectors at            1 (type ee)
Previous to Aug 28th I would (after a reboot) use my file manager (PCManFM)
to navigate to the volume. Upon clicking on its representation a dialog box
would appear asking for the root password. Upon successful completion
there would be access to the array.

At present I can find no way to access the array.

mounting doesn't seem to work:

# mount -text4 /dev/md/0 /media/darald/Raid10
mount: wrong fs type, bad option, bad superblock on /dev/md0,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
root@debiantestingbase:/media/darald# mount -text2 /dev/md/0
/media/darald/Raid10
mount: wrong fs type, bad option, bad superblock on /dev/md0,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
root@debiantestingbase:/media/darald# mount -gpt /dev/md/0
/media/darald/Raid10
mount: invalid option -- 'g'

a different attempt to mount the array

# blkid | grep md0
/dev/md0: PTUUID="1eb47793-0cdf-4e16-ae84-33ec825ac448" PTTYPE="gpt"
root@debiantestingbase:/media/darald# vi /etc/fstab
root@debiantestingbase:/media/darald# mount -a
mount: can't find UUID=1eb47793-0cdf-4e16-ae84-33ec825ac448

I have found this but am concerned that I could also possibly destroy the array
not knowing if THIS is the solution

# fsck /dev/md0
fsck from util-linux 2.26.2
e2fsck 1.42.13 (17-May-2015)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/md0

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

o1bigtenor <[email-masked]>
Sep 11
Reply
to Maurice
Greetings

Were you able to put together anything to facilitate my filesystem repair?

Thanking you for your assistance.

maurice <[email-masked]>
Sep 11
Reply
to me
On 9/11/2015 5:11 AM, o1bigtenor wrote:
Greetings

Were you able to put together anything to facilitate my filesystem repair?

Sorry, last night got really busy.
Will look at this some more tonight.

o1bigtenor <[email-masked]>
Sep 11
Reply
to maurice
On Fri, Sep 11, 2015 at 7:21 AM, maurice <[email-masked]> wrote:
> On 9/11/2015 5:11 AM, o1bigtenor wrote:
>
> Greetings
>
> Were you able to put together anything to facilitate my filesystem repair?
>
>
> Sorry, last night got really busy.
> Will look at this some more tonight.
>

Thanks - - - looking forward to seeing the stuff.

Maurice Hilarius <[email-masked]>
Sep 11
Reply
to me
Quick question..  Is this an ext4 filesystem?

o1bigtenor <[email-masked]>
Sep 11
Reply
to Maurice
Yes

maurice <[email-masked]>
Sep 11
Reply
to me
OK, this article provides a pretty good summary of methods to recover
a damaged ext4 filesystem.

http://www.vmexplore.com/check-repair-ext3-ext4-filesystem-oracle-linux/

maurice <[email-masked]>
Sep 11
Reply
to me
Also useful:

http://www.cyberciti.biz/faq/recover-bad-superblock-from-corrupted-partition/

You DID ask to have everything as one huge file - - - so here you have it.

I would find this email confusing but you did ask for everything to be dumped
into one big file - - - .
Any further questions - - - I'll do what I can.

TIA

Dee