Bitcoind 0.12 pruned locks file system

[rupy]

I'm trying to run bitcoind pruned, the blockchain sync worked fine, and also once the sync was done bitcoind was running smoothly.

But then after a reboot, when I start bitcoind it boots up ok and does it's thing verifying some stuff but then it blocks 3 cores at 100% io-wait and the disk is frozen (if I try to cat any file bitcoind is holding the cat process freezes) and I can't kill bitcoind?!

Is anyone actually using pruned?

How should pruned be used? What is prune=<n>? MB of what, how, where?

I'm trying to reindex and run unpruned to see if pruned is the problem. Just have to wait 4 days!

So bitcoind goes from taking 3 cores to 100% io-wait and locking these files:

Code:

COMMAND           PID  TYPE SIZE MODE  M START END PATH
cron              653 FLOCK   0B WRITE 0     0   0 /run
whoopsie          640 FLOCK   0B WRITE 0     0   0 /run/lock
master            864 FLOCK   0B WRITE 0     0   0 /
master            864 FLOCK   0B WRITE 0     0   0 /
bitcoind         2014 POSIX   0B WRITE 0     0   0 /mnt/ssd/bitcoin/.lock
bitcoind         2014 POSIX   0B WRITE 0     0   0 /mnt/ssd/bitcoin/blocks/index/LOCK
bitcoind         2014 POSIX   0B WRITE 0     0   0 /mnt/ssd/bitcoin/chainstate/LOCK

to having only 1-2 cores 100% io-wait and locking this when I try kill -9:

Code:

COMMAND           PID  TYPE SIZE MODE  M START END PATH
cron              653 FLOCK   0B WRITE 0     0   0 /run
whoopsie          640 FLOCK   0B WRITE 0     0   0 /run/lock
master            864 FLOCK   0B WRITE 0     0   0 /
master            864 FLOCK   0B WRITE 0     0   0 /
bitcoind         2014 POSIX   0B WRITE 0     0   0 /mnt/ssd
bitcoind         2014 POSIX   0B WRITE 0     0   0 /mnt/ssd
bitcoind         2014 POSIX   0B WRITE 0     0   0 /mnt/ssd

[1715032603]

1715032603

[1715032603]

1715032603

[1715032603]

1715032603

[rupy]

Ok, pruned is broken on 0.12. Nobody cares?... well when the blockchain reaches 100GB and can't fit on my SSD I'll throw a hissyfit like Mike Hern!

To reproduce launch a new bitcoind with -prune=600 and let it sync, then reboot the machine and the bitcoind will lockup disk and the bitcoind process while verifying the block headers.

It will take you 4 days to reproduce every time, enjoy!

I'll copy the blockchain and prune it from this state to see if it works then... maybe this bug is due to the "prune from scratch" premise?

How do you debug a process that freezes linux like that?!

[achow101]

Can you post the debug log?

[cr1776]

...

How do you debug a process that freezes linux like that?!

What distro? Version? Locally hosted? Etc

[rupy]

@knightdk There was nothing in the debug.log at the failure time I checked.
@cr1776 Ubuntu 15.10, locally hosted? I guess yes!

[HackerBOSS]

@knightdk There was nothing in the debug.log at the failure time I checked.
@cr1776 Ubuntu 15.10, locally hosted? I guess yes!

use ubuntu 14.04 because 15 not recommend

[achow101]

@knightdk There was nothing in the debug.log at the failure time I checked.
@cr1776 Ubuntu 15.10, locally hosted? I guess yes!

use ubuntu 14.04 because 15 not recommend

Says who?

@knightdk There was nothing in the debug.log at the failure time I checked.
@cr1776 Ubuntu 15.10, locally hosted? I guess yes!

I'm using 15.10 as well and I cannot reproduce this. Can you provide specific steps for getting this problem? Also what are the specs of your system?

[-ck]

Just a thought; you're using an SSD. Are you using discard on the mount options, if supported by your distribution? If not, are you using trim on the file system at regular intervals?

[rupy]

Ok, seems it works fine if you start the bitcoind normally (= it downloads the whole chain on disk) and then enable pruning.

Might be the amount of pruning that solved it, I used -prune=650 first time around, now I used -prune=10000.

Seems to run fine... better hurry and boot/prune those bitcoinds before the chain grows beyond cheap SSD size!

Just a thought; you're using an SSD. Are you using discard on the mount options, if supported by your distribution? If not, are you using trim on the file system at regular intervals?

Ok, yes it's a SSD, but it wasn't slow; it was frozen!

And if linux doesn't handle SSDs properly by default on recent versions I don't know what to say...

It is connected via a USB3 -> SATA adapter though, maybe that's the problem?

Code:

/dev/sda on /mnt/ssd type ext4 (rw,relatime,data=ordered)

What is the syntax for discard?

Code:

mount -o discard /dev/sda /mnt/ssd

[rupy]

So this might be related to the USB3 - SATA - SSD dongle I'm using...

When I reboot the computer any command that tries to access the SSD freezes the SSH terminal.

But if I pull the USB3 plug and push it back in everything works fine!?!

Any clues?!

Is there a command to reset the USB3 ports without manually reconnecting them?

echo 0 > /sys/bus/usb/drivers/usb/X-X.X/authorized also freezes the prompt!

Ok, seems to be something bad with the USB3 - SATA stuff:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=a9c54caa456dccba938005f6479892b589975e6a

But the most recent kernel is 3.10 for my hardware... how could I apply a fix without recompiling the kernel?

Found this fix, but I'm missing mkinitcpio!?

https://bbs.archlinux.org/viewtopic.php?pid=1428782#p1428782