Can't dl blockchain - 'fatal internal error' then segfault

[Nancarrow]

Hi all. Though I use electrum for all my wallet needs, I use bitcoin-qt to have a local copy of the blockchain. A few days ago it told me something about a need to reindex, or a corrupt database, or something... can't remember.

Anyway since then I have repeatedly tried to download the blockchain from scratch and it cannot proceed past four years ago. I have now completely removed and reinstalled bitcoin-qt running under Ubuntu-12.04, and I still get the same problem. I try downloading from scratch, it goes okay until I get to about 4 years out-of-sync. Then bitcoin-qt tells me there was a fatal internal error. Trying to run it subsequently it keeps mentioning a segmentation fault. All I can do is delete the whole of ~/.bitcoin and... round we go again.

Any ideas? There's still 60GB of space on the hd partition.

ETA: in case it's important, the software tells me it's "Bitcoin Core version v0.10.1.0-gd8ac901 (32-bit)" and I installed it using the standard Ubuntu apt-get way of doing things.

[achow101]

What version of Bitcoin Core are you running? Are you compiling from source? Also, what are the specs of your machine? Can you post the debug.log here? All of these will help us diagnose the issue.

[Nancarrow]

What version of Bitcoin Core are you running? Are you compiling from source? Also, what are the specs of your machine? Can you post the debug.log here? All of these will help us diagnose the issue.

Machine is a 5-yr old laptop with 3GB of memory. Bitcoin version and installation as above.

I'm now getting different behaviour that may be more acceptable. In my most recent run, it crashed out thus:

The last few lines of debug.log are all of the form:

2015-06-03 20:50:19 UpdateTip: new best=0000000000000778dfa81c3a653acf1ef06ce13da4747cde8529cc4fe86a81a6  height=138839  log2_work=65.787511  tx=1169347  date=2011-07-30 19:22:59 progress=0.007195  cache=171211
2015-06-03 20:50:19 UpdateTip: new best=00000000000007fadd5d4e5e0210a8bbfd4eca94af0bb2b63de6898e56c7f983  height=138840  log2_work=65.787675  tx=1169378  date=2011-07-30 19:37:30 progress=0.007195  cache=171228

... for hundreds of lines

This time Bitcoin-qt never even showed me a dialog box telling me it encountered an error. It simply crashed out. I'd invoked it from a terminal window and on crashing I see the following in the terminal:

*** glibc detected *** bitcoin-qt: double free or corruption (out): 0x2b308d40 ***
======= Backtrace: =========
/lib/i386-linux-gnu/libc.so.6(+0x74f82)[0x2a61f82]
/usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x1f)[0x96fc51f]
bitcoin-qt(+0x291f0f)[0x5c8f0f]
bitcoin-qt(+0x28eff9)[0x5c5ff9]

... followed by several similar lines and then a 'memory map', however that helps.

HOWEVER, I have now restarted bitcoin-qt, and after a while it just resumes the synchronisation. So with a bit of luck it will simply carry on with no further crashing.
Do you reckon it's dodgy memory? It looks like dodgy memory to me.

[achow101]

Those lines of the debug.log are for the synchronization of the blockchain.

I'm not sure what is happening, it could be bad memory, maybe run a memory check on the laptop?

[cakir]

It's probably because of the predefined settings.
Can you try to change datadir of blockchain?

something like that; "./bitcoin -datadir=/somewhereElse/ "

[Nancarrow]

Well, just to give an update... bitcoin-qt is still running, it hasn't crashed... but I don't like what it IS doing. Basically block downloading stalled 40 min ago, as shown by the timestamps for blk00006.dat and rev00006.dat. It has been stuck at block #156748 all this time. What IS being continually updated is debug.log, which is now almost 900MB!!

The last few lines of debug.log now are:

2015-06-03 21:50:46 ERROR: CheckBlock() : hashMerkleRoot mismatch
2015-06-03 21:50:46 Misbehaving: 83.85.132.142:8333 (329801600 -> 329801700)
2015-06-03 21:50:46 ERROR: ConnectTip() : ConnectBlock 0000000000000088dc6197ddd62a76882d4d6ec4ea619e79d51fdecab423c71d failed
2015-06-03 21:50:46 ERROR: CheckBlock() : hashMerkleRoot mismatch
2015-06-03 21:50:46 Misbehaving: 83.85.132.142:8333 (329801700 -> 329801800)
2015-06-03 21:50:46 ERROR: ConnectTip() : ConnectBlock 0000000000000088dc6197ddd62a76882d4d6ec4ea619e79d51fdecab423c71d failed


I presume that most of the 900MB is like that. Core is also telling me that there are 10 active connections to the network which sounds like *that* isn't the problem.

GRR!   If it doesn't like the last few blocks for some reason, why the hell can't it just go back to whenever it DID like them and restart the download from there? And how about trimming debug.log while you're at it?

ETA: hmm wait a minute... are those lines telling me that it's being fed bad data by some peer? But if so, why doesn't it just try someone else?

[grue]

ETA: hmm wait a minute... are those lines telling me that it's being fed bad data by some peer? But if so, why doesn't it just try someone else?

you can try -connect=191.236.50.217 to connect only to my node and download from there. You might want to delete existing block data prior to doing this to clear any bad block data from before.

[Nancarrow]

Okay trying that. It's only early days yet (~block #46000) but it hasn't crashed yet.

A more general question: if I do somehow end up with a big blockchain that's mostly correct until the last little bit gets corrupted, what exactly would I have to delete to 'roll it back'? I get that I'd have to delete the last blk00xxx.dat, but what other files? I'd hate to have 19GB of perfect blockchain data and have it redownload every damn thing from the beginning.

[achow101]

I think that you need to delete the latest blk....dat files, latest rev.....dat files, and the latest .ldb files in blocks/index. However, I'm not very sure about this. You might want to check this with someone else.

[Nancarrow]

Hi all, just to keep you updated... I still haven't managed to sync up the blockchain completely. The client does seem to keep crashing out sometimes. Half the time it says 'a fatal internal error occurred' and shuts down nicely, the other half it just quits and I see a core dump.

Removing the last, incomplete blk00xxx.dat at least allows it to restart, but I notice it always insists on reindexing from the beginning, even when I leave all the rev00xxx.dats alone. But at least I don't have to keep redownloading the blockchain-so-far so that's something.

Anyway I seem to be okay up to blk00133.dat, apparently that's up to "2 years and 24 weeks behind" so we shall see. But it's still in the process of reindexing all that.

[Nancarrow]

Okay I'm at my wit's end here.

Please, if somebody could talk me through this I'd be grateful. Apologies for the vitriol in this post. Please be assured it is directed at the general nature of computers, not at you.

Let's be clear, I'm NOT expecting anybody to get me to a fully synced up chain here. My goals are far, FAR more modest.

First off: suppose I have deleted EVERYTHING in the .bitcoin folder with the SOLE exception of the blocks/blk00xxx.dat files, of which I have the first 146. That's some 16GB of data that I'm really, really hoping bitcoin-qt isn't TOO STUPID TO SEE. No, Mr. Bitcoin, you do NOT need to start downloading the blocks all over again, they are RIGHT FUCKING HERE.
How do I translate that previous sentence into something Mr. Bitcoin can understand?

The problem is I started it again and thought well I'll let it dl them for a bit, I now have blk00000 and 00001.dat that are timestamped AFTER the blk00002-00146.dat.
Surely that can't be a problem? Surely Mr. Bitcoin can't decide he's going to ignore the subsequent block files because of their timestamp? Please tell me the code can't be designed THAT badly?

[achow101]

First off: suppose I have deleted EVERYTHING in the .bitcoin folder with the SOLE exception of the blocks/blk00xxx.dat files, of which I have the first 146.

Therein lies the problem. the blk files are just the raw data from the network dumped to the disk. What Bitcoin Core actually uses for everything are database files. These are found in blocks/index which contains data on where to find blocks, and the chainstate directory in the datadir. The chainstate directory is the most important regarding the blockchain. From the Bitcoin Wiki:

chainstate subdirectory
[v0.8 and above] A LevelDB database with a compact representation of all currently unspent transaction outputs and some metadata about the transactions they are from. The data here is necessary for validating new incoming blocks and transactions. It can theoretically be rebuilt from the block data (see the -reindex command line option), but this takes a rather long time. Without it, you could still theoretically do validation indeed, but it would mean a full scan through the blocks (7 GB as of may 2013) for every output being spent.

By deleting this directory, Bitcoin Core must rescan the entire blockchain from the blk files in order to rebuild these databases. It is not redownloading them either. The different timestamps that you see is because Bitcoin Core is reading through every single blk file and rescanning them to rebuild the database.

[Nancarrow]

Okay thanks that makes it somewhat clearer. In that case I shall just let it rescan to its heart's content.

ETA: actually I'll ask while I'm here: what's the difference, if any, between the process you just described, and the process of 'reindexing' that you can force it to do with the '--reindex' option? I ask because I've just started it up again and the status bar at the bottom says "Synchronizing with network..." which is what made me worry it's redownloading blocks.

I shall scoot along to the wiki to see if I can get to grips with what all the different files are trying to do... I understand the blk00xxx.dat files themselves, but I don't yet understand the rev00xxx.dat files, the index/000xxx.ldb files (and a .log file in with them too), and the stuff in the chainstate dir. Anyway ta for the pointers.

[xmaxbit]

I have no idea about it . I am using blockchain.info as my wallet and now its saying "Chain Head Not Found" . Can anyone tell me about it please ?

[shorena]

I have no idea about it . I am using blockchain.info as my wallet and now its saying "Chain Head Not Found" . Can anyone tell me about it please ?

Different issue, bc.i is a service that currently has problems, OP is talking about THE blockchain.

[achow101]

Okay thanks that makes it somewhat clearer. In that case I shall just let it rescan to its heart's content.

ETA: actually I'll ask while I'm here: what's the difference, if any, between the process you just described, and the process of 'reindexing' that you can force it to do with the '--reindex' option? I ask because I've just started it up again and the status bar at the bottom says "Synchronizing with network..." which is what made me worry it's redownloading blocks.

It should say "reindexing blocks on disk..." when it is reindexing. However, yours doesn't. Perhaps it is because you haven't downloaded to entire blockchain yet. Since it indexes the blocks as it downloads, I think that since you don't have the full blockchain on the disk yet, it will both reindex and download the blocks, and say "Synchronizing with network..."

I shall scoot along to the wiki to see if I can get to grips with what all the different files are trying to do... I understand the blk00xxx.dat files themselves, but I don't yet understand the rev00xxx.dat files, the index/000xxx.ldb files (and a .log file in with them too), and the stuff in the chainstate dir. Anyway ta for the pointers.

You can read some more about the data directory on the Bitcoin Wiki here: https://en.bitcoin.it/wiki/Data_directory

[Nancarrow]

I already read that wiki entry and didn't find it too enlightening. But I shall keep looking for the grittier technical details.

Anyway on to the next question in my series, "How to Eventually Reverse Engineer the Complete Bitcoin-Qt Code by Asking About Every Damn Thing that is Bugging Me"...

Up until blk00070.dat, it has been rebuilding the database, and timestamping the blockfiles, at a rate of about one blockfile every two or three minutes. BUT, for the last 40 minutes or so, nothing. Why not? The blockfiles up to 00145 are all sitting there, waiting hungrily to be reindexed. What's the holdup? The "Synchronising with the network" progress bar has also halted.

I'll give it another half hour and then close and restart bitcoin-qt.

ETA: EEEEEK! I've just looked at debug.log to see what it's doing. So... 70MB of cruft in there. Now some of it is, I understand, perfectly OK. Fr'instance stuff like this:

2015-06-06 21:06:52 UpdateTip: new best=0000000000000103b325ddcc16512a6754e710c387d069a17e217169f293447c  height=233167  log2_work=69.868797  tx=16765534  date=2013-04-26 01:02:19 progress=0.102582  cache=109732

And this:

2015-06-06 21:06:52 Pre-allocating up to position 0xa00000 in rev00056.dat

... though even then I must question why it's storing so damn MUCH of it. From block 226924 to block 246158! I know debug.log is supposed to be 'flushed' at intervals but couldn't we make those intervals a little more... timely?

Anyway I don't mind, but THEN I see this:

2015-06-06 21:38:50 ERROR: CheckBlock() : hashMerkleRoot mismatch
2015-06-06 21:38:50 Misbehaving: 71.93.44.219:8333 (0 -> 100) BAN THRESHOLD EXCEEDED
2015-06-06 21:38:50 ERROR: ConnectTip() : ConnectBlock 000000000000004bbd569ecddf773e8f2d92345fb7027ead5c89221196cf9ec3 failed
2015-06-06 21:38:50 ERROR: CheckBlock() : hashMerkleRoot mismatch
2015-06-06 21:38:50 Misbehaving: 71.93.44.219:8333 (100 -> 200)
2015-06-06 21:38:50 ERROR: ConnectTip() : ConnectBlock 000000000000004bbd569ecddf773e8f2d92345fb7027ead5c89221196cf9ec3 failed
2015-06-06 21:38:50 ERROR: CheckBlock() : hashMerkleRoot mismatch
2015-06-06 21:38:50 Misbehaving: 71.93.44.219:8333 (200 -> 300)
2015-06-06 21:38:50 ERROR: ConnectTip() : ConnectBlock 000000000000004bbd569ecddf773e8f2d92345fb7027ead5c89221196cf9ec3 failed
2015-06-06 21:38:50 ERROR: CheckBlock() : hashMerkleRoot mismatch

... and so on, for the vast bulk of the file, with the timestamp up front increasing by the seconds... for about 40 minutes until I stop it!

Okay... WHY???

1) Why is there a hashMerkleRoot mismatch?
2) Why is 71.93.44.219 misbehaving?
3) Why should bitcoin-qt CARE about some node misbehaving when it is allegedly NOT downloading blocks, but simply rebuilding the database on the blocks it has?
4) Obvious follow-on... is it, in fact, despite what you said, in *fact* redownloading those blocks after all?
5) After the first 10000 times* of printing this little set of error messages, should not the client somehow grasp that what it is doing is NOT WORKING and, just perhaps, try another node? Or even give up and say "There is something wrong, I'm stopping now."?

*obvious snark - I know what computers can and cannot do. I suppose my snark is really directed at all these coders who must somehow have never seen this behaviour before and never thought to put a little 'break' in that loop.

[achow101]

I already read that wiki entry and didn't find it too enlightening. But I shall keep looking for the grittier technical details.

Anyway on to the next question in my series, "How to Eventually Reverse Engineer the Complete Bitcoin-Qt Code by Asking About Every Damn Thing that is Bugging Me"...

I suppose if you really want to get technical and if you can read code, you could check out the developer docs here: https://dev.visucore.com/bitcoin/doxygen/. These are useful as you can see paths of which method calls what. Then, if you can read code, you can look at the source code provided on the site and try to find how everything works. There are some explanations as these are developer docs, but not very many.

Up until blk00070.dat, it has been rebuilding the database, and timestamping the blockfiles, at a rate of about one blockfile every two or three minutes. BUT, for the last 40 minutes or so, nothing. Why not? The blockfiles up to 00145 are all sitting there, waiting hungrily to be reindexed. What's the holdup? The "Synchronising with the network" progress bar has also halted.

I'll give it another half hour and then close and restart bitcoin-qt.

I'm not sure why this would happen. Try looking in the debug.log and see if it is giving you any error messages. Also, if you hover the mouse over the progress bar, it should say on what block it is on. If you do it again a minute later and the block number hasn't changed, it likely has hung on something, and you should restart it.

[Nancarrow]

Thanks for your reply, obviously I've just been spending a bit too much time seething and updating my previous post! I will check out the developer-docs link you gave, and in the meantime delete debug.log, and restart bitcoin-qt, probably I'll point it at grue's node.

[2112]

*obvious snark - I know what computers can and cannot do. I suppose my snark is really directed at all these coders who must somehow have never seen this behaviour before.

Your hardware is broken. Get another, correctly working, machine and all your problems will disappear. Why you keep torturing yourself and us if you know how the computers work?