Bitcoin Forum

Hi all. Though I use electrum for all my wallet needs, I use bitcoin-qt to have a local copy of the blockchain. A few days ago it told me something about a need to reindex, or a corrupt database, or something... can't remember.

Anyway since then I have repeatedly tried to download the blockchain from scratch and it cannot proceed past four years ago. I have now completely removed and reinstalled bitcoin-qt running under Ubuntu-12.04, and I still get the same problem. I try downloading from scratch, it goes okay until I get to about 4 years out-of-sync. Then bitcoin-qt tells me there was a fatal internal error. Trying to run it subsequently it keeps mentioning a segmentation fault. All I can do is delete the whole of ~/.bitcoin and... round we go again.

Any ideas? There's still 60GB of space on the hd partition.

ETA: in case it's important, the software tells me it's "Bitcoin Core version v0.10.1.0-gd8ac901 (32-bit)" and I installed it using the standard Ubuntu apt-get way of doing things.

What version of Bitcoin Core are you running? Are you compiling from source? Also, what are the specs of your machine? Can you post the debug.log here? All of these will help us diagnose the issue.

Machine is a 5-yr old laptop with 3GB of memory. Bitcoin version and installation as above.

I'm now getting different behaviour that may be more acceptable. In my most recent run, it crashed out thus:

The last few lines of debug.log are all of the form:

2015-06-03 20:50:19 UpdateTip: new best=0000000000000778dfa81c3a653acf1ef06ce13da4747cde8529cc4fe86a81a6  height=138839  log2_work=65.787511  tx=1169347  date=2011-07-30 19:22:59 progress=0.007195  cache=171211
2015-06-03 20:50:19 UpdateTip: new best=00000000000007fadd5d4e5e0210a8bbfd4eca94af0bb2b63de6898e56c7f983  height=138840  log2_work=65.787675  tx=1169378  date=2011-07-30 19:37:30 progress=0.007195  cache=171228

... for hundreds of lines

This time Bitcoin-qt never even showed me a dialog box telling me it encountered an error. It simply crashed out. I'd invoked it from a terminal window and on crashing I see the following in the terminal:

*** glibc detected *** bitcoin-qt: double free or corruption (out): 0x2b308d40 ***
======= Backtrace: =========
/lib/i386-linux-gnu/libc.so.6(+0x74f82)[0x2a61f82]
/usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x1f)[0x96fc51f]
bitcoin-qt(+0x291f0f)[0x5c8f0f]
bitcoin-qt(+0x28eff9)[0x5c5ff9]

... followed by several similar lines and then a 'memory map', however that helps.

HOWEVER, I have now restarted bitcoin-qt, and after a while it just resumes the synchronisation. So with a bit of luck it will simply carry on with no further crashing.
Do you reckon it's dodgy memory? It looks like dodgy memory to me.

Those lines of the debug.log are for the synchronization of the blockchain.

I'm not sure what is happening, it could be bad memory, maybe run a memory check on the laptop?

It's probably because of the predefined settings.
Can you try to change datadir of blockchain?

something like that; "./bitcoin -datadir=/somewhereElse/ "

Well, just to give an update... bitcoin-qt is still running, it hasn't crashed... but I don't like what it IS doing. Basically block downloading stalled 40 min ago, as shown by the timestamps for blk00006.dat and rev00006.dat. It has been stuck at block #156748 all this time. What IS being continually updated is debug.log, which is now almost 900MB!!

The last few lines of debug.log now are:

2015-06-03 21:50:46 ERROR: CheckBlock() : hashMerkleRoot mismatch
2015-06-03 21:50:46 Misbehaving: 83.85.132.142:8333 (329801600 -> 329801700)
2015-06-03 21:50:46 ERROR: ConnectTip() : ConnectBlock 0000000000000088dc6197ddd62a76882d4d6ec4ea619e79d51fdecab423c71d failed
2015-06-03 21:50:46 ERROR: CheckBlock() : hashMerkleRoot mismatch
2015-06-03 21:50:46 Misbehaving: 83.85.132.142:8333 (329801700 -> 329801800)
2015-06-03 21:50:46 ERROR: ConnectTip() : ConnectBlock 0000000000000088dc6197ddd62a76882d4d6ec4ea619e79d51fdecab423c71d failed


I presume that most of the 900MB is like that. Core is also telling me that there are 10 active connections to the network which sounds like *that* isn't the problem.

GRR!  >:( If it doesn't like the last few blocks for some reason, why the hell can't it just go back to whenever it DID like them and restart the download from there? And how about trimming debug.log while you're at it?

ETA: hmm wait a minute... are those lines telling me that it's being fed bad data by some peer? But if so, why doesn't it just try someone else?

you can try -connect=191.236.50.217 to connect only to my node and download from there. You might want to delete existing block data prior to doing this to clear any bad block data from before.

Okay trying that. It's only early days yet (~block #46000) but it hasn't crashed yet.

A more general question: if I do somehow end up with a big blockchain that's mostly correct until the last little bit gets corrupted, what exactly would I have to delete to 'roll it back'? I get that I'd have to delete the last blk00xxx.dat, but what other files? I'd hate to have 19GB of perfect blockchain data and have it redownload every damn thing from the beginning.

I think that you need to delete the latest blk....dat files, latest rev.....dat files, and the latest .ldb files in blocks/index. However, I'm not very sure about this. You might want to check this with someone else.

Hi all, just to keep you updated... I still haven't managed to sync up the blockchain completely. The client does seem to keep crashing out sometimes. Half the time it says 'a fatal internal error occurred' and shuts down nicely, the other half it just quits and I see a core dump.

Removing the last, incomplete blk00xxx.dat at least allows it to restart, but I notice it always insists on reindexing from the beginning, even when I leave all the rev00xxx.dats alone. But at least I don't have to keep redownloading the blockchain-so-far so that's something.

Anyway I seem to be okay up to blk00133.dat, apparently that's up to "2 years and 24 weeks behind" so we shall see. But it's still in the process of reindexing all that.

Okay I'm at my wit's end here.

Please, if somebody could talk me through this I'd be grateful. Apologies for the vitriol in this post. Please be assured it is directed at the general nature of computers, not at you.

Let's be clear, I'm NOT expecting anybody to get me to a fully synced up chain here. My goals are far, FAR more modest.

First off: suppose I have deleted EVERYTHING in the .bitcoin folder with the SOLE exception of the blocks/blk00xxx.dat files, of which I have the first 146. That's some 16GB of data that I'm really, really hoping bitcoin-qt isn't TOO STUPID TO SEE. No, Mr. Bitcoin, you do NOT need to start downloading the blocks all over again, they are RIGHT FUCKING HERE.
How do I translate that previous sentence into something Mr. Bitcoin can understand?

The problem is I started it again and thought well I'll let it dl them for a bit, I now have blk00000 and 00001.dat that are timestamped AFTER the blk00002-00146.dat.
Surely that can't be a problem? Surely Mr. Bitcoin can't decide he's going to ignore the subsequent block files because of their timestamp? Please tell me the code can't be designed THAT badly?

Therein lies the problem. the blk files are just the raw data from the network dumped to the disk. What Bitcoin Core actually uses for everything are database files. These are found in blocks/index which contains data on where to find blocks, and the chainstate directory in the datadir. The chainstate directory is the most important regarding the blockchain. From the Bitcoin Wiki:
By deleting this directory, Bitcoin Core must rescan the entire blockchain from the blk files in order to rebuild these databases. It is not redownloading them either. The different timestamps that you see is because Bitcoin Core is reading through every single blk file and rescanning them to rebuild the database.

Okay thanks that makes it somewhat clearer. In that case I shall just let it rescan to its heart's content.

ETA: actually I'll ask while I'm here: what's the difference, if any, between the process you just described, and the process of 'reindexing' that you can force it to do with the '--reindex' option? I ask because I've just started it up again and the status bar at the bottom says "Synchronizing with network..." which is what made me worry it's redownloading blocks.

I shall scoot along to the wiki to see if I can get to grips with what all the different files are trying to do... I understand the blk00xxx.dat files themselves, but I don't yet understand the rev00xxx.dat files, the index/000xxx.ldb files (and a .log file in with them too), and the stuff in the chainstate dir. Anyway ta for the pointers.

I have no idea about it . I am using blockchain.info as my wallet and now its saying "Chain Head Not Found" . Can anyone tell me about it please ?

Different issue, bc.i is a service that currently has problems, OP is talking about THE blockchain.

It should say "reindexing blocks on disk..." when it is reindexing. However, yours doesn't. Perhaps it is because you haven't downloaded to entire blockchain yet. Since it indexes the blocks as it downloads, I think that since you don't have the full blockchain on the disk yet, it will both reindex and download the blocks, and say "Synchronizing with network..."

You can read some more about the data directory on the Bitcoin Wiki here: https://en.bitcoin.it/wiki/Data_directory

I already read that wiki entry and didn't find it too enlightening. But I shall keep looking for the grittier technical details.

Anyway on to the next question in my series, "How to Eventually Reverse Engineer the Complete Bitcoin-Qt Code by Asking About Every Damn Thing that is Bugging Me"...

Up until blk00070.dat, it has been rebuilding the database, and timestamping the blockfiles, at a rate of about one blockfile every two or three minutes. BUT, for the last 40 minutes or so, nothing. Why not? The blockfiles up to 00145 are all sitting there, waiting hungrily to be reindexed. What's the holdup? The "Synchronising with the network" progress bar has also halted.

I'll give it another half hour and then close and restart bitcoin-qt.

ETA: EEEEEK! I've just looked at debug.log to see what it's doing. So... 70MB of cruft in there. Now some of it is, I understand, perfectly OK. Fr'instance stuff like this:

2015-06-06 21:06:52 UpdateTip: new best=0000000000000103b325ddcc16512a6754e710c387d069a17e217169f293447c  height=233167  log2_work=69.868797  tx=16765534  date=2013-04-26 01:02:19 progress=0.102582  cache=109732

And this:

2015-06-06 21:06:52 Pre-allocating up to position 0xa00000 in rev00056.dat

... though even then I must question why it's storing so damn MUCH of it. From block 226924 to block 246158! I know debug.log is supposed to be 'flushed' at intervals but couldn't we make those intervals a little more... timely?

Anyway I don't mind, but THEN I see this:

2015-06-06 21:38:50 ERROR: CheckBlock() : hashMerkleRoot mismatch
2015-06-06 21:38:50 Misbehaving: 71.93.44.219:8333 (0 -> 100) BAN THRESHOLD EXCEEDED
2015-06-06 21:38:50 ERROR: ConnectTip() : ConnectBlock 000000000000004bbd569ecddf773e8f2d92345fb7027ead5c89221196cf9ec3 failed
2015-06-06 21:38:50 ERROR: CheckBlock() : hashMerkleRoot mismatch
2015-06-06 21:38:50 Misbehaving: 71.93.44.219:8333 (100 -> 200)
2015-06-06 21:38:50 ERROR: ConnectTip() : ConnectBlock 000000000000004bbd569ecddf773e8f2d92345fb7027ead5c89221196cf9ec3 failed
2015-06-06 21:38:50 ERROR: CheckBlock() : hashMerkleRoot mismatch
2015-06-06 21:38:50 Misbehaving: 71.93.44.219:8333 (200 -> 300)
2015-06-06 21:38:50 ERROR: ConnectTip() : ConnectBlock 000000000000004bbd569ecddf773e8f2d92345fb7027ead5c89221196cf9ec3 failed
2015-06-06 21:38:50 ERROR: CheckBlock() : hashMerkleRoot mismatch

... and so on, for the vast bulk of the file, with the timestamp up front increasing by the seconds... for about 40 minutes until I stop it!

Okay... WHY???

1) Why is there a hashMerkleRoot mismatch?
2) Why is 71.93.44.219 misbehaving?
3) Why should bitcoin-qt CARE about some node misbehaving when it is allegedly NOT downloading blocks, but simply rebuilding the database on the blocks it has?
4) Obvious follow-on... is it, in fact, despite what you said, in *fact* redownloading those blocks after all?
5) After the first 10000 times* of printing this little set of error messages, should not the client somehow grasp that what it is doing is NOT WORKING and, just perhaps, try another node? Or even give up and say "There is something wrong, I'm stopping now."?

*obvious snark - I know what computers can and cannot do. I suppose my snark is really directed at all these coders who must somehow have never seen this behaviour before and never thought to put a little 'break' in that loop.

I suppose if you really want to get technical and if you can read code, you could check out the developer docs here: https://dev.visucore.com/bitcoin/doxygen/. These are useful as you can see paths of which method calls what. Then, if you can read code, you can look at the source code provided on the site and try to find how everything works. There are some explanations as these are developer docs, but not very many.

I'm not sure why this would happen. Try looking in the debug.log and see if it is giving you any error messages. Also, if you hover the mouse over the progress bar, it should say on what block it is on. If you do it again a minute later and the block number hasn't changed, it likely has hung on something, and you should restart it.

Thanks for your reply, obviously I've just been spending a bit too much time seething and updating my previous post! I will check out the developer-docs link you gave, and in the meantime delete debug.log, and restart bitcoin-qt, probably I'll point it at grue's node.

Your hardware is broken. Get another, correctly working, machine and all your problems will disappear. Why you keep torturing yourself and us if you know how the computers work?

I doubt my hardware is broken if absolutely every program on it apart from bitcoin-qt works fine.

My "I know what computers can and can't do" remark was a flippant way of acknowledging that if I see a computer going through an endless loop it is pointless to expect it to change.

I am torturing myself, but no one else, you don't have to read this! Well, maybe I'm torturing knightdk but it seems to be consensual! (And I must repeat I am grateful to knightdk for their help so far)

knightdk is just working on his required activity for the signature campaign.

What you are doing is just confirming for everyone that you have no idea what you are doing with your machine and you are the perfect mark for an enterprising scammer.

What I'm doing here is trying to discern if this is a natural, genuine stupidity at work or if there was some better hidden ulterior motive. We'll find out in due time.

What is at work here is an attempt to learn why my bitcoin-qt client appears not to function correctly, and in doing so, learn more about what exactly it is doing under the hood. If I snark a little sometimes it is because I am frustrated. I'm sure you can figure out who in this thread is helping and who is... well, not.

But please *do* tell me more about my "hidden ulterior motive". I just love hearing strangers explaining mine to me. You can also explain how I am "the perfect mark for an enterprising scammer". While you're at it, have another swipe at knightdk for no discernible reason.

For some reason, this block, block 246159 (according to blockchain.info) is failing validation. Can't really say why. My guess would be that somehow it got corrupted in that file and it is missing some part of the block so the validation fails. This is probably where it gets hung.

Bitcoin Core is still connected to its peers. Each peer receives a banscore based on whether the node thinks it is misbehaving. After the score reaches a certain threshold (default 100), the node should disconncect from the node. It cares whether the node is misbehaving since the misbehaviour involves spam and can eat up your bandwidth.

It should still be downloading blocks, just not ones that it already has. The node is constantly receiving and sending data, but it should not be redownloading those blocks, only downloading blocks that it still does not have.

It should, but I don't know if it actually does (probably not).

Your best bet to fix this would be to just let it completely redownload and reindex the entire blockchain. I know that can be frustrating and time consuming, but it should fix the issues. If that doesn't work, try reinstalling Bitcoin Core.

At least I'm being useful to someone, unlike most people in sig campaigns.

Knightdk: my surprise at seeing that it still cares about 'misbehaving nodes' is due to the fact that it ISN'T downloading any new blocks at all, it has only been reindexing the ones it already has. THAT behaviour makes perfect sense to me - presumably it would have no way to validate newly downloaded blocks until it has validated, reindexed and built a chainstate for all the existing blocks. So I can't blame it for not downloading new blocks, I don't see why it *should*. But then, *given* that it's not doing that, I cannot see what use there would be in connecting to peers at all.

However I don't suppose that's all terribly important. The crucial issue is the apparently corrupted block 246159. I'd just delete the last blk00xxx.dat file, but it seems that rather than reindexing from somewhere fairly recent, it always starts reindexing from the very beginning and ignoring all those other files (even when I *don't* delete them - I've tried it that way, too).

I've also tried removing everything and redownloading the blockchain from scratch...

...AND removing and reinstalling bitcoin-qt. I think that, notwithstanding how I stuck up for my computer hardware, I may have to suck it up and get a new hard drive and put it on there.

Look, this situation is really easy to read, it happened multiple times on this forum.

1) If you are really an inexperienced, frustrated computer user: just run the hardware tests for your machine, memory and disk faults are the most common root source of the problems you're seeing.

2) If your are just another gang of scammers trying to build mutual reputation of fixing the tough computer problems so they can then offer remote assistance using Teamviewer to the fresh marks: this isn't new either, it has been done multiple times already.

This is what makes bitcointalk.org such an anthropologically fascinating place: trying to figure out if this is a new scam or new convolution of old scams.

It is a completely independent process to connect to nodes that begins before the whole blockchain verification at startup. This runs in another thread independent of the state of the rest of the program.

Looked at the code. It looks like it is hard coded to start at the beginning whenever it is reindexing other than when it is reindexing while syncing.

Before you do that, as 2112 said, run some diagnostic checks on your memory and hard drive. There should be an option to do this when your computer first starts up.

2112: Now that's a MUCH better response!

Yes, it looks like hardware faults is where it's at. I will try a new hard drive (actually have one lying around) and test the memory.

Re scams: I'm sure you're right about their existence. I don't read that many threads. But their frequency? Surely the ratio of genuine tech problems to mutual-scammer-backscratching must be quite high here? We're not talking about my granny trying to read MS Word documents in Firefox here.

And I may not be that tech savvy, but TeamViewer... come on, dude.  :P

Knightdk: will do.

Thanks all.

Could be hard drive fail. Get a program to test it and also find out waht mode the hard drive is in sounds like read error. D oyou get any clicking from it? Or find that when you loading wallet everything slows down to a point cant use anything else?

2 ideas you could try to resolve this to keep using on current hardware if nothing else is is broke only bitcoin.

1 Delete everything in app data folder but backup and keep your wallet safe.  Now download blockchain using torrent and then once it is downloaded drop te .dat file into bitcoin folder in app data and allow it to index all of it and see how or where it gets to. If you get an error doing this could more likely be part read wright error no the drive that is the cause to it.

2 if you needing access to your wallets you could if it will allow you put wallet.dat back into bitcoin folder and dump prive keys on addresses you need access to and use bither I have started to use this wallet since it is lightweight and fast and simple to import wallets directly in from pri key. Am slowly moving away from qt due to the fact of amount of data it uses and should just be like bither light weight effective and does not kill drive space and takes time to load up. bitcoin wallet need to get their act together if they want to go global. Think of it like this if  it went mainstream and a trillion transactions where done in a day or more. How big would the wallet get to use on system and peoples devices. They need to adapt features like in bither or using blockchain wallets.

Hope you manage to resolve your problem or have some usability in accessing your addresses and wallet :)

Depends on which "ratio" are you asking. You may be right about ratio of people with problems to people scamming. But if you measure ratio by posts not by people, then the scammers are majority, especially in the mining forums.

Some of them possibly even start without evil intentions. But with time they learn that scamming here is easier than shooting fish in the barrel, and it becomes too hard to resist.

I'm kinda disappointed that you've regained your senses and decided to test hardware. Just less than a week ago I had interesting experience of watching similar scam (but one the corporate contract level and not related to Bitcoin) unfold personally in front of my eyes. They key hook was to convince person sitting in front of the computer with 3" floppy drive that there is no working drive there and they need to download the test program instead running one off of the 3" floppy. I learned a lot watching and listening that unfold.