Raspberry Pi - corrupted blocks

[Rath_]

Hey,

does anyone know if there are any Raspibolt/Raspiblitz alternatives with c-lightning support instead of LND? I had no luck so far in finding any. I will end up installing everything manually if I can't find any alternatives by tomorrow.

[DaveF]

I have looked too but not found any. But, it was a while ago that I looked.
mynode is LND too
Any reason why you want c-lightning in a device instead of LND?
I wanted to learn c-lightning a bit more and figured a pre-done environment would be better to start to play with.

Wound up just doing it myself anyway to learn.

-Dave

[BitMaxz]

Take a look at this one alternative just found when searching on Google.

- https://github.com/mynodebtc/mynode


Or you might end up with manual installation below.
- https://github.com/bitembassy/home-node

I don't know this one will work in raspberry but since it's related to ubuntu it might be work with a normal OS(Ubuntu.)

[Rath_]

Any reason why you want c-lightning in a device instead of LND?

I have already played around with LND when I was getting started with the Lightning Network, so I want to try something new. Also, I heard a lot of people complaining about LND being a mess now. The only thing I don't like about c-lightning is that it lacks decent clients. The only one I can recall is Spark while LND can be connected to Zap or even BlueWallet (if configured properly), and some other clients.

-snip

Thanks for your input. The first link is not what I am looking for - that project supports only LND. As for the other link, it might actually come in useful.

[Carlton Banks]

The only thing I don't like about c-lightning is that it lacks decent clients. The only one I can recall is Spark while LND can be connected to Zap or even BlueWallet (if configured properly), and some other clients.

there's also Lightning-qt

and I think there was a Lightning client on Android called "Zeus" that could operate the c-lightning API as a client, but I didn't try it to see how well it worked.

[DaveF]

The only thing I don't like about c-lightning is that it lacks decent clients. The only one I can recall is Spark while LND can be connected to Zap or even BlueWallet (if configured properly), and some other clients.

there's also Lightning-qt

and I think there was a Lightning client on Android called "Zeus" that could operate the c-lightning API as a client, but I didn't try it to see how well it worked.

Watch your quoting I did not say that :-)

Just did a c-lightning install, on a machine that had LND running.
Yeah, don't do that. No clue what I was thinking. But obviously didn't work till I changed the port.
Shutdown LND to play with c-lightning

Once you have a wallet synced, it's all kind of the same. But LND does seem to have more support.

-Dave

[Carlton Banks]

Watch your quoting I did not say that :-)

oops, sorry Dave. edited

Once you have a wallet synced, it's all kind of the same. But LND does seem to have more support.

LND uses the same/similar resources? I can see that being the case, golang is essentially just c underneath, so it should perform pretty well if you know how to write it well (I imagine it's more forgiving than c)

[Rath_]

I might end up going for RaspiBolt since I am having some trouble with setting up everything correctly. My chainstate folder got corrupted twice for no reason when launching bitcoind. I am not sure if it's because my external SSD was formatted to exFat or because I am also running a samba server pointing to that drive. I am running Ubuntu Server 20.04.1 and I followed this guide.

Is there any particular reason why you guys chose c-lightning over LND or vice-versa?

[DaveF]

LND uses the same/similar resources? I can see that being the case, golang is essentially just c underneath, so it should perform pretty well if you know how to write it well (I imagine it's more forgiving than c)

Yes, both seem to use the same RAM / CPU.
It's constrained since I was testing on a RPi but neither one seems to have more bloat.

Is there any particular reason why you guys chose c-lightning over LND or vice-versa?

I originally started to play with lightning with one of the node in a box setups. It used LND and I just ran with it.
I keep wanting to spend more time with c-lightning and now that I have it running again in my office I probably will, but the main reason I have LND everyplace is that it was what I learned 1st...

-Dave

[Carlton Banks]

My chainstate folder got corrupted twice for no reason when launching bitcoind. I am not sure if it's because my external SSD was formatted to exFat or because I am also running a samba server pointing to that drive.[/url].

I wouldn't choose exFat, but not sure if it's actually susceptible to data corruption. It's just that you can likely get something more reliable (ZFS or btrfs), or simply with more features/security (ext4)

Is there any particular reason why you guys chose c-lightning over LND or vice-versa?

I think people initially chose LND because it was pre-compiled, and it had more features. The features gap has largely closed, and c-lightning is now available pre-compiled too

I chose c-lightning because they went for a lightweight modular design (and because they had a veteran Linux dev running the project). The LND team seemed to work very fast (easier when writing software with golang, or so I hear), but make alot of mistakes. Don't know to what extent they're still like that, not been keeping up with it

[Rath_]

I wouldn't choose exFat, but not sure if it's actually susceptible to data corruption. It's just that you can likely get something more reliable (ZFS or btrfs), or simply with more features/security (ext4)

I ended up splitting my SSD into two separate partitions (exFat and ext4). I need to keep the exFat one because I want to be able to occasionally use that SSD with my Windows laptop on the go. Now, I am trying to figure out how to force RaspiBlitz to mount and use only the ext4 partition. The automatic installer insists on making one large partition, so I can't proceed with the installation.

[DaveF]

I wouldn't choose exFat, but not sure if it's actually susceptible to data corruption. It's just that you can likely get something more reliable (ZFS or btrfs), or simply with more features/security (ext4)

I ended up splitting my SSD into two separate partitions (exFat and ext4). I need to keep the exFat one because I want to be able to occasionally use that SSD with my Windows laptop on the go. Now, I am trying to figure out how to force RaspiBlitz to mount and use only the ext4 partition. The automatic installer insists on making one large partition, so I can't proceed with the installation.

How large is your SSD? How much space do you need for the exFat? Do you need anything that is on it now?
If it's a 1 TB and you don't need anything that's on it just let installer do its thing and then do a shutdown and re-size / shrink the partition and create a ex-Fat one.
Keep in mind between the blockchain and the lightning DB and everything else I would no go below 650 or so GB otherwise you are going to be doing this again soon.
Yes you can do it in 500GB, I am doing that now, but why create more work for yourself later?

-Dave

[Rath_]

How large is your SSD? How much space do you need for the exFat? Do you need anything that is on it now?

1 TB (931.50 GB available). I would like to keep at least 400 GB for the exFat partition. I don't have any personal files on that drive, so I can safely keep playing with it. I am currently uploading the blockchain data through SCP to the ext4 partition, though.

If it's a 1 TB and you don't need anything that's on it just let installer do its thing and then do a shutdown and re-size.

Sounds good, but I might need to modify some auto-mount setting later.

Keep in mind between the blockchain and the lightning DB and everything else I would no go below 650 or so GB otherwise you are going to be doing this again soon.

My blockchain data takes about 350 GB right now. I assume that it will grow at a rate of about 50 GB per year. How much does the lightning database weight? 450-500 GB for the ext4 partition sounds fine to me at the moment.

[NotATether]

My chainstate folder got corrupted twice for no reason when launching bitcoind. I am not sure if it's because my external SSD was formatted to exFat or because I am also running a samba server pointing to that drive. I am running Ubuntu Server 20.04.1 and I followed this guide.

bitcoind should not care which filesystem is underlying the folder. If you pointed bitcoin's folder to some other partition, then there is either a problem with the way the filesystem was made, or, like this guide is doing, you're symlinking the folder from filesystem to another and it's giving the read/write system calls hiccups.

If it's the latter, and it probably is if other folders on that filesystem are getting corrupted too, then the ext4 filesystem was made on the wrong device. You should be making it on one like /dev/sda1 or sda2, not /dev/sda (which is reserved for the partition table on-disk).

For reference, this is the correct way to format a disk and create an ext4 filesystem on the first partition of a disk.

Code:

$ sudo fdisk /dev/sda
Command (m for help): n

You’ll be prompted to enter the partition number. Hit “Enter” to use the default value (1):

Code:

Partition number (1-128, default 1):

Next, the command will ask you to specify the first sector. Generally it is always recommended to use the default values for the first value. Hit “Enter” to use the default value (2048):

Code:

First sector (2048-500118158, default 2048):

On the next prompt, you’ll need to enter the last sector. You can use an absolute value for the last sector or relative value to the start sector, using the + symbol following by the partition size. The size can be specified in kibibytes (K), mebibytes (M), gibibytes (G), tebibytes (T), or pebibytes (P).

Enter +100G to set the partition size to 100 GiB:

Code:

Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-500118158, default 500118158): +100G
Created a new partition 1 of type 'Linux filesystem' and of size 100 GiB.

...

Save the changes by running the w command:

Code:

Command (m for help): w

The command will write the table to disk and exit the fdisk menu.

Code:

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

The kernel will read the device partition table without the need to reboot the system.

...

Code:

$ sudo mkfs.ext4 -F /dev/sda1

Alternatively, you may use some other partitioning utility besides fdisk such as gparted. Adjust the instructions accordingly.

Your samba share is for all purposes unknown to bitcoind and so should not be related to this problem.

I ended up splitting my SSD into two separate partitions (exFat and ext4). I need to keep the exFat one because I want to be able to occasionally use that SSD with my Windows laptop on the go. Now, I am trying to figure out how to force RaspiBlitz to mount and use only the ext4 partition. The automatic installer insists on making one large partition, so I can't proceed with the installation.

Perhaps you can create a docker container and pass the mount point of your ext4 partition as a volume and install raspiblitz inside that? Then it will only be able to see the ext4 partition.

[DaveF]

For an synced install with nothing else installed / configured:

Code:

admin@raspberrypi:~ $ df
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/root       14989948   7946132   6384848  56% /
devtmpfs         1966332         0   1966332   0% /dev
tmpfs            2000124         0   2000124   0% /dev/shm
tmpfs            2000124    205300   1794824  11% /run
tmpfs               5120         4      5116   1% /run/lock
tmpfs            2000124         0   2000124   0% /sys/fs/cgroup
tmpfs              32768         8     32760   1% /var/cache/raspiblitz
/dev/mmcblk0p1    258095     52237    205858  21% /boot
/dev/sda1      479620500 367258156  87929260  81% /mnt/hdd
tmpfs             400024         0    400024   0% /run/user/1000
tmpfs             400024         0    400024   0% /run/user/1001

So yes, 500GB is more then enough.
If you go through and install all the optional items like ElectRS / BTCPay / mempool.space explorer / etc you can chew though some more storage.
When I installed all of it, I was well over 400GB and the RPi was about to melt it was trying to so much.

-Dave

[darkv0rt3x]

I'm using symlinks just to point ~/.bitcoin folder to my actual data folder and despite the fact that I also have corrupted data once or twice, I'm not sure I can say it's due to symlinks. But that's a good point.

I am using a 1 TB drive in ext4 and I think better than this, only btrfs or ZFS. But I'm not running my nodes in a Raspbi as the OP. Anyways, I only wanted to point out that I'm using symlinks too with not big issues.

[Rath_]

If it's the latter, and it probably is if other folders on that filesystem are getting corrupted too, then the ext4 filesystem was made on the wrong device. You should be making it on one like /dev/sda1 or sda2, not /dev/sda (which is reserved for the partition table on-disk).

The corruption occurred when my whole drive was formatted to exFat. Thanks for your instructions, though.

Perhaps you can create a docker container and pass the mount point of your ext4 partition as a volume and install raspiblitz inside that? Then it will only be able to see the ext4 partition.

That's a good idea. However, RaspiBlitz is only available as an OS image. I have decided to make a clean setup and install everything manually again.

[Rath_]

Both Bitcoin Core compilation and SCP upload finished over night, so I decided to give it another try. All the blockchain data is now stored on the ext4 partition. Unfortunately, something is still wrong. Block #667874 was the latest one at that moment.

Code:

2021-01-27T09:54:38Z init message: Verifying blocks...
2021-01-27T09:54:38Z Verifying last 6 blocks at level 3
2021-01-27T09:54:38Z [0%]...ERROR: ReadBlockFromDisk: Deserialize or I/O error - CAutoFile::read: end of file: iostream error at FlatFilePos(nFile=2425, nPos=42621844)
2021-01-27T09:54:38Z ERROR: VerifyDB(): *** ReadBlockFromDisk failed at 667874, hash=00000000000000000008e979aaca803bc56599f1bc80aa3196c1d9089958034f
2021-01-27T09:54:38Z : Corrupted block database detected.
Please restart with -reindex or -reindex-chainstate to recover.
2021-01-27T09:54:38Z Aborted block database rebuild. Exiting.

Here's my complete debug.log.

[NotATether]

Both Bitcoin Core compilation and SCP upload finished over night, so I decided to give it another try. All the blockchain data is now stored on the ext4 partition. Unfortunately, something is still wrong. Block #667874 was the latest one at that moment.

Do you think that the SCP upload got disrupted mid-copy and abruptly left some blocks partially-copied? I don't think SCP attempts to finish copying interrupted transfers.

It's ssh so even if you have a fast network, it's still encrypting all the files it sends and maybe the speed was so slow that it stopped transferring the files. Given that the scp transfer took a whole night the speed probably wasn't that great. Possibly several dozen Kbytes/s?

Check your block files with md5sum and make sure they're all the same as the ones you have locally.

Try using rsync or some other copy tool that doesn't make swathes of encryption.

[DaveF]

Weird, since 667874 was just mined and causing the issue I wonder if something else was "touching" that information / block when you were moving the data.

Other than that the only thing I can think of is there is some hardware issue that is only showing itself when the system is under load, like when it's verifying the blockchain or a block. But, that is really a stretch.

Wish I could help more, but I got nothing....

-Dave