[makrospex]

No sleep yet  

Good points/ideas! Will be interesting to see the output of sgdisk and if it confirms my current theory.

I think we are on a similar pathway.

Never seen that md127 before, would check /etc/mdadm/mdadm.conf, but that's only a wild guess.

Seems to appear in some recovery situations automatically, according to some outbound forum threads.
(possibly also created during reorg/rebuilds?)

After revisiting the below, I noticed that md0p1 has no recognized filesystem. So trying to rebuild the journal with tune2fs is probably of no help. But it indeed hints to a broken super block me thinks.

...

It definitely should read TYPE="ext4" at the md0p1 line, but there is no partition type detected at all.

Based on this the best recommendation I can come up with at this point is trying to restore a backup superblock on md0p1.

If dumpe2fs finds no superblock backups, there is the risky way of using mke2fs in simulation mode, by basically 'dry'-reformatting your partition the computer will come up with the same superblock addresses as the first time it formatted it.

RL calls, will check back tomorrow.

Maybe try

Code:

fsck -t ext4 /dev/md0p1

first?
With a forced repair afterwards, following a backup, if not already created.

Hurr... fsck on mountpoint... Durr... I'm special.

you and me both, guess who had to learn that you can't mount multiple file systems on a single mount point...

That's the way it goes.
Like Zen, only linux-wise 

[1714825074]

1714825074

[1714825074]

1714825074

[1714825074]

1714825074

[makrospex]

Good morning

So you have a drive sector size of 4096 bytes = 512 bytes * 8 and we know the starting sector number.

Number  Start (sector)    End (sector)  Size       Code  Name
   1            5118     15611104253   7.3 TiB     FD00  

Psycodad's opinion on recreating the filesystem with mke2fs seems a valid option to me there.
But first, i want you to try to mount partition1 directly, but without trying to read the bad superblock,which (iirc) should work this way:

Code:

mount -t ext4 -o ro,offset=((5118*4096)) /dev/md0 /path/to/mountpoint 

or

Code:

mount -t ext4 -o ro,offset=((5118*512)) /dev/md0 /path/to/mountpoint 

I don't know exactly which size is the right one to calc the offset, but imho it's the physical sector size (=command 1 should work, imo).
If either mount command succeeds, check if your files are there in /path/to/mountpoint.
Partition 1 (which is /dev/md0p1 in linux) will be mounted in read-only mode.

BUT: The filesystem could alsohave errors, that's why it is mounted in read-only mode. At least you are able to access the data for backup, but the final goal should be recreation of a healthy superblock, so fsck can do it's work before you remount in read-write mode.

If none of the above commands work, i'd try mke2fs to create a new superblock and handle the drive conventionally.
Wait, peeking for the first superblock backup location by multiples of the sector size for

Code:

fsck -B ... 

would be a valid option before that, but i want to read Psycodad's opinion on this first.

For future filesystem creation(s):
Optimally, logical sector size would match physical sector size and the start of any partition should be a multiple of that, unless you have to store many very small files (below physical sector size).

[makrospex]

Code:

mount -t ext4 -o ro,offset=((5118*4096)) /dev/md0 /path/to/mountpoint 

or

Code:

mount -t ext4 -o ro,offset=((5118*512)) /dev/md0 /path/to/mountpoint 

Code:

root@bitcorn:/dev# mount -t ext4 -o ro,offset=((5118*4096)) /dev/md0 /mnt/md0
bash: syntax error near unexpected token `('
root@bitcorn:/dev# mount -t ext4 -o ro,offset=((5118*512)) /dev/md0 /mnt/md0
bash: syntax error near unexpected token `('

Poop.

Arrr :/
Please calculate the value manually and omit the quotes.

mount -t ext4 -o ro,offset=20963328 /dev/md0 /mnt/md0

or

mount -t ext4 -o ro,offset=2620416 /dev/md0 /mnt/md0

[makrospex]

Please calculate the value manually and omit the quotes.

Yeah, figured I'd just enter the calculated values. No freaking idea what /dev/loop0 has anything to do with anything.

Linux really makes me sad sometimes

EDIT: Sleep time. Back in a few hours. Thanks again !

you're welcome.
The "loop block device" lets you use a file as if it were a block device (a virtual hard drive, for example).

https://en.wikipedia.org/wiki/Loop_device

Windows made me even more sad, tbh.
But that's a longer story, and way off topic 

[psycodad]

Moinmoin,

Thanks makrospex for picking up where I left :-)

Anyway, there is one thing that strikes me as very odd, but it may be too early or I may be simply too stupid (or both), but...:

sgdisk -p /dev/md0

Code:

root@bitcorn:/dev# sgdisk -p /dev/md0
Disk /dev/md0: 15627544576 sectors, 7.3 TiB
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 00000000-0000-0000-0000-000000000000
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 15627544542
Partitions will be aligned on 8-sector boundaries
Total free space is 8141 sectors (4.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            5118     15611104253   7.3 TiB     FD00  
   2     15611104256     15627541487   7.8 GiB     8200  

While p2 is correctly showing as type "linux swap" (0x82), p1 shows as "Linux raid autodetect" (0xfd).
IMHO it should show as 0x83 for an ext4 partition. Looking for some confirmation or shown to be wrong from makrospex here.

@Bob: Can you show us /etc/mdadm/mdadm.conf and when it was last modified (so we know it was not recreated recently):

Code:

cat /etc/mdadm/mdadm.conf; stat /etc/mdadm/mdadm.conf

It might gives us a new idea..

* psycodad needs much more coffee..

[makrospex]

Moinmoin,

Thanks makrospex for picking up where I left :-)

Gern geschehen  

Anyway, there is one thing that strikes me as very odd, but it may be too early or I may be simply too stupid (or both), but...:

sgdisk -p /dev/md0

Code:

root@bitcorn:/dev# sgdisk -p /dev/md0
Disk /dev/md0: 15627544576 sectors, 7.3 TiB
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 00000000-0000-0000-0000-000000000000
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 15627544542
Partitions will be aligned on 8-sector boundaries
Total free space is 8141 sectors (4.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            5118     15611104253   7.3 TiB     FD00  
   2     15611104256     15627541487   7.8 GiB     8200  

While p2 is correctly showing as type "linux swap" (0x82), p1 shows as "Linux raid autodetect" (0xfd).
IMHO it should show as 0x83 for an ext4 partition. Looking for some confirmation or shown to be wrong from makrospex here.

@Bob: Can you show us /etc/mdadm/mdadm.conf and when it was last modified (so we know it was not recreated recently):

Code:

cat /etc/mdadm/mdadm.conf; stat /etc/mdadm/mdadm.conf

It might gives us a new idea..

* psycodad needs much more coffee..

There can't be enough coffee, ever.
I realize that i need one too  

I was also wondering, looks like *cough* linuxnoob partitioning to me.

Now i'll paste what i was about to post before reading your reply:

I read into the mke2fs man page and found the -n switch:

-n
    Causes mke2fs to not actually create a filesystem, but display what it would do if it were to create a filesystem. This can be used to determine the location of the backup superblocks for a particular filesystem, so long as the mke2fs parameters that were passed when the filesystem was originally created are used again. (With the -n option added, of course!)

I suggest the default values ("auto" at setup) were used on filesystem creation, so

Code:

mke2fs -n /dev/md0p1

should output the same backup block numbers for use with

Code:

fsck -B ... 

Waddaya think?

EDIT: knowing mdadm.conf contents would surely help, but BobLawBlaw has to wake up first. Damned timeshifts.

[psycodad]

<snip>

-n
    Causes mke2fs to not actually create a filesystem, but display what it would do if it were to create a filesystem. This can be used to determine the location of the backup superblocks for a particular filesystem, so long as the mke2fs parameters that were passed when the filesystem was originally created are used again. (With the -n option added, of course!)

I suggest the default values ("auto" at setup) were used on filesystem creation, so

Code:

mke2fs -n /dev/md0p1

should output the same backup block numbers for use with

Code:

fsck -B ... 

Waddaya think?

Not trying to be a smartass at all (ok, maybe a tiny lil bit  ), that was what I meant by:

<snip>

If dumpe2fs finds no superblock backups, there is the risky way of using mke2fs in simulation mode, by basically 'dry'-reformatting your partition the computer will come up with the same superblock addresses as the first time it formatted it.

RL calls, will check back tomorrow.

It seems dangerous, it will ask you to answer y to reformat your partition (even though with -n it doesn't). Though I tried it on USB stick and it does indeed not reformat (even the wording and confirmation is confusion as the author of mke2fs once admitted).

But yes, that would be one of the next steps I'd also propose.

After that I only have testdisk in my mind, but that's rather dangerous too easy to do something wrong.

[makrospex]

Not trying to be a smartass at all (ok, maybe a tiny lil bit  ), that was what I meant by:

<snip>

If dumpe2fs finds no superblock backups, there is the risky way of using mke2fs in simulation mode, by basically 'dry'-reformatting your partition the computer will come up with the same superblock addresses as the first time it formatted it.

RL calls, will check back tomorrow.

It seems dangerous, it will ask you to answer y to reformat your partition (even though with -n it doesn't). Though I tried it on USB stick and it does indeed not reformat (even the wording and confirmation is confusion as the author of mke2fs once admitted).

But yes, that would be one of the next steps I'd also propose.

After that I only have testdisk in my mind, but that's rather dangerous too easy to do something wrong.

Right, testdisk would also not be suitable for guided use, i would want my own hands on that.
So the next steps are clear now. I'd like to know if the offset mount of /dev/md0 was successful, but i'll have to wait for the TC to show up again.
Also mdadm.conf contents...

Since the source of the errors was a power outage and the filesystem wasn't in sync, i'm positive that the problem will finally be solved.
bad blocks would be worse.
Also, i seem to be in luck this week. Two days ago i shadow-copied an old laptop's hdd, which was making occasional noises of horror, into a 18gig .vhd file for use in Virtualbox for almost six hours. Roughly fifteen minutes after completion, the laptop rebooted and got stuck at the bios splash screen. Then i removed the drive, connected it via usb-ide cable and copied over the .vhd file to the virtualbox host. The file was vital, but the drive died just before i wanted to disconnected it later (i was lazy).
With so much luck, i'm confident  

[jojo69]

Windows made me even more sad, tbh.

precisely

[psycodad]

@Bob: Can you show us /etc/mdadm/mdadm.conf and when it was last modified (so we know it was not recreated recently):

Code:

cat /etc/mdadm/mdadm.conf; stat /etc/mdadm/mdadm.conf

It might gives us a new idea..

* psycodad needs much more coffee..

I commented out the mdadm.conf ARRAY stuff. I figured it looked fucked up enough that it was effectively useless. I have no idea what I'm doing.

Code:

#ARRAY /dev/md0 metadata=1.2 spares=1 name=xserver:0 UUID=e0104263:8aef7ef8:2d66045f:26a69f31
#ARRAY /dev/md0 metadata=1.2 spares=1 name=xserver:0 UUID=6314fb8d:1466fbbd:5aa88188:793c3019
#ARRAY /dev/md0 metadata=1.2 spares=1 name=bitcorn:0 UUID=4b7f5741:476a4550:ba1184b2:76efd22e
#ARRAY /dev/md/0  metadata=1.2 UUID=4b7f5741:476a4550:ba1184b2:76efd22e name=bitcorn:0
#ARRAY /dev/md/bitcorn:0 metadata=1.2 name=bitcorn:0 UUID=4196079a:155fab25:eb7fa044:37adb2ce

Okay, that looks pretty weird, I guess that system has seen quite a few changes already.

I am still circling around this partition type 0xfd on m0p1. Just for fun and giggles could you try to see if there is another raid device on top of m0p1 when you scan now?

Code:

mdadm --assemble --scan

[makrospex]

@Bob: Can you show us /etc/mdadm/mdadm.conf and when it was last modified (so we know it was not recreated recently):

Code:

cat /etc/mdadm/mdadm.conf; stat /etc/mdadm/mdadm.conf

It might gives us a new idea..

* psycodad needs much more coffee..

I commented out the mdadm.conf ARRAY stuff. I figured it looked fucked up enough that it was effectively useless. I have no idea what I'm doing.

Code:

#ARRAY /dev/md0 metadata=1.2 spares=1 name=xserver:0 UUID=e0104263:8aef7ef8:2d66045f:26a69f31
#ARRAY /dev/md0 metadata=1.2 spares=1 name=xserver:0 UUID=6314fb8d:1466fbbd:5aa88188:793c3019
#ARRAY /dev/md0 metadata=1.2 spares=1 name=bitcorn:0 UUID=4b7f5741:476a4550:ba1184b2:76efd22e
#ARRAY /dev/md/0  metadata=1.2 UUID=4b7f5741:476a4550:ba1184b2:76efd22e name=bitcorn:0
#ARRAY /dev/md/bitcorn:0 metadata=1.2 name=bitcorn:0 UUID=4196079a:155fab25:eb7fa044:37adb2ce

Hmm, that was unexpected  
I hope you made a backup of the file, which you can move back in place?
Or roll back any changes?

Let's continue with the "script":

First a question: Did one of the

Code:

mount -t ext4 -o=ro,offset xxx /dev/md0 /mnt/md0

succeed?
So

Code:

ls -la /mnt/md0

is showing the contents of the raid?

If yes, copy away what you need and then:

Code:

umount /dev/md0
mke2fs -t ext4 -n /dev/md0p1

and paste the output into the reply.

Don't give it up yet, you seem frustrated, it's too early, man  

EDIT: First please follow Psycodad's guidance, i crossposted but was definitely later than him.

[makrospex]

First a question: Did one of the

Code:

mount -t ext4 -o=ro,offset xxx /dev/md0 /mnt/md0

succeed?
So

Code:

ls -la /mnt/md0

is showing the contents of the raid?

Was not able to get it to mount at all using the offset calculations.

doh
then it's making out the backup superblock adresses via

Code:

mke2fs -n

and using these to find a vital copy to use.
My fucking spacebar is making me mad, something sticksunder it, so i have to punch it after every word to get a space in the text 

[psycodad]

doh
then it's making out the backup superblock adresses via

Code:

mke2fs -n

and using these to find a vital copy to use.
My fucking spacebar is making me mad, something sticksunder it, so i have to punch it after every word to get a space in the text 

Agree with makrospex, next try would be actually finding and restoring a good backup of the superblock of m0p1, I think he meant to say:

Code:

mke2fs -n /dev/md0p1

to find the superblocks.
Triplecheck it has the -n before you answer 'y' to the following prompt!


Regarding the mounts to try from makrospex, please try these again with /dev/md0p1 instead of /dev/md0, i.e.:

Code:

mount -t ext4 -o=ro,offset xxx /dev/md0p1 /mnt/md0

Maybe also check that /mnt/md0 is empty, otherwise the mount might fail with a slightly different error message that you could overlook easily.

[psycodad]

At least there are valid superblock backups, that's a start I guess.
^The software just simply tells us where these backup superblocks would be stored on a partition of given size.

Please try the mounts with the device I suggested above and not /dev/md0:

i.e.

Code:

mount -t ext4 -o ro,offset=32768 /dev/md0p1 /mnt/md0

[psycodad]

At least there are valid superblock backups, that's a start I guess.
Please try the mounts with the device I suggested above and not /dev/md0:
i.e.

Code:

mount -t ext4 -o ro,offset=32768 /dev/md0p1 /mnt/md0

Code:

root@bitcorn:/mnt/md0# mount -t ext4 -o ro,offset=32768 /dev/md0p1 /mnt/md0
mount: /mnt/md0: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error.

from mke2fs: /dev/md0p1 alignment is offset by 1024 bytes.
This may result in very poor performance, (re)-partitioning suggested.

Might that have something to do with it ? Add 1024 to whatever the offsets say ?

Hmm, not sure about the offset, you could try with 1024 and by adding 1024, you could also try to fsck it with the backup superblock added:

Code:

fsck.ext4 -b 32768 /dev/md0p1

Otherwise I am bit confused about the loop part still, did you by chance have for example an encrypted filesystem on top of that raid device?

[psycodad]

You sure there is an ext4 filesystem to be expected? (Just asking, no offence meant)

Otherwise I'd be keen to hear of any ideas from makrospex, I am getting a bit lost, the loop part still doesn't let go on me.

Just as some therapeutic exercise, could you post the output of

Code:

dumpe2fs -h /dev/md0p1

(I am really just fishing for new ideas here)

At this point I need to ask: Did this contain a wallet that you didn't have backuped otherwise or any other unique and valuable data or just a synched btc node that you want to get back online without synching from genesis?

[GoMaD]

can you please paste the output of

 

Code:

parted --list

[GoMaD]

please try

Code:

fsck -fCV /dev/md0p1

and paste the output

EDIT: and the output from

Code:

dmesg | grep md0

please

[makrospex]

Sorry, had to put the kids to sleep.
will backread now and answer asap.

[makrospex]

This took a little longer, since i had to back/crossread and type a reply in a text editor, not to lose overview (cause of my impaired working memory ya'll might know already about). Here we go:

/dev/md0p1 alignment is offset by 1024 bytes.

This means the start of md0p1 is not at the start of a physical sector (multiple of 4096 bytes).
It's not a cause for failure.

compare to this:

root@bitcorn:/dev# sgdisk -p /dev/md0
Disk /dev/md0: 15627544576 sectors, 7.3 TiB
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 00000000-0000-0000-0000-000000000000
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 15627544542
Partitions will be aligned on 8-sector boundaries
Total free space is 8141 sectors (4.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            5118     15611104253   7.3 TiB     FD00  
   2     15611104256     15627541487   7.8 GiB     8200  

The partition should start at sector mulitple of 8, but starts at 34, because the partition table takes up 33 sectors.
So it's 2 sectors (each 512 bytes) = 1024 bytes "behind" of the optimum (or 6 sectors in front). Would have been better to start at sector 40, forexample.
There are unused sectors at the end of the disk, meaning this is a GPT partition (in contrast to classic MBR).
Strangely, the GUID of the drive is zero (00000000-0000-0000-0000-000000000000), but that might not be important, too.
Also strange is the fact that the table lists 5118 as starting sector of md0p1, instead of 34.
Partition 2 (md0p2) starts at 15611104256, which is a multiple of 8, thus PERFECT.
When calculating the length of md0p1, its 2 sectors short of a multiple of 8.
Conclusion: Like

Code:

parted --list

suggests,

Error: The primary GPT table is corrupt, but the backup appears OK, so that will
be used.

the Partition table is corrupted. I'll read up how to handle this after this post.

The superblock backup positions from

Code:

mke2fs -n ...

are for

Code:

fsck -t ext4 -b 

, not for mount.
Since the master partition table seems to be invalid, we can't calculate a valid offset for mount, because we don't know the correct starting sector yet.
The solution to these might be found in the backup partition table too.

An ext4 superblock has a size of 1024 bytes. coincidence?

Please try mount -t ext4 -o ro,offset=... /dev/md0p1 /mnt/md0 with offset values of 34*512 (and 35*512).
Make sure you are outside /mnt/md0 when mounting. Check contents of /mnt/md0 after mounting trials.

Then issue a

e2fsck -b some_backup_block_numbers_from_mke2fs /dev/md0p1

preferably using the higher backup superblock numbers.
The blocksize would be good to know exactly, while it defaults to 4096.

Next shot would be using "testdrive" in non-destructive mode, to find out more details about the partitions of md0 and values like real start, end, blocksize of md0p1
Even to restore it all completely by testdisk logs manually.

If you want to put up a thread in an expert forum (the community support forum of your linux distribution, i'd suggest) please link it here, so we can follow and give further assistance, raise our cups when solved, or whatever

EDIT:

Advice: I was avoiding raid arrays most of the time, because if you got trouble with them beyond anything a rebuild/resync can handle, you're going to face problems in the higher PITA levels. Either the disks/partitions are not readable after the raid controller died and you have no backup hardware at hand, or in case of software raid, you can't just mount the raid sub-partitions and read them easily. I went to single drives (ssd for performance) and frequent backups just NOT to rely on raid whenever possible.
Backup is uncomfortable, takes a lot of storage but also somehow inevitable. Sorry for the pain. I hope you have a key/seed backup for the wallet and the blockchain sync does not take forever if you set up a new node. If you didn't do already, DISABLE the f**ing green power saving functions of any hard drive you use, these lead to parking the heads much more often, spinning down drives too soon, which puts stress on the hardware and kills drives sooner (which adds to the electronic waste problem, thus anything but "green"). I disabled this on all my "green" WDC 2TB drives in my NAS, after i saw one failing after half a year because of that).