Nancarrow (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 03, 2015, 08:33:17 PM |
|
Hi all. Though I use electrum for all my wallet needs, I use bitcoin-qt to have a local copy of the blockchain. A few days ago it told me something about a need to reindex, or a corrupt database, or something... can't remember.
Anyway since then I have repeatedly tried to download the blockchain from scratch and it cannot proceed past four years ago. I have now completely removed and reinstalled bitcoin-qt running under Ubuntu-12.04, and I still get the same problem. I try downloading from scratch, it goes okay until I get to about 4 years out-of-sync. Then bitcoin-qt tells me there was a fatal internal error. Trying to run it subsequently it keeps mentioning a segmentation fault. All I can do is delete the whole of ~/.bitcoin and... round we go again.
Any ideas? There's still 60GB of space on the hd partition.
ETA: in case it's important, the software tells me it's "Bitcoin Core version v0.10.1.0-gd8ac901 (32-bit)" and I installed it using the standard Ubuntu apt-get way of doing things.
|
If I've said anything amusing and/or informative and you're feeling generous: 1GNJq39NYtf7cn2QFZZuP5vmC1mTs63rEW
|
|
|
achow101
Moderator
Legendary
Offline
Activity: 3486
Merit: 6824
Just writing some code
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 03, 2015, 08:35:25 PM |
|
What version of Bitcoin Core are you running? Are you compiling from source? Also, what are the specs of your machine? Can you post the debug.log here? All of these will help us diagnose the issue.
|
|
|
|
Nancarrow (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 03, 2015, 09:03:07 PM |
|
What version of Bitcoin Core are you running? Are you compiling from source? Also, what are the specs of your machine? Can you post the debug.log here? All of these will help us diagnose the issue.
Machine is a 5-yr old laptop with 3GB of memory. Bitcoin version and installation as above. I'm now getting different behaviour that may be more acceptable. In my most recent run, it crashed out thus: The last few lines of debug.log are all of the form: 2015-06-03 20:50:19 UpdateTip: new best=0000000000000778dfa81c3a653acf1ef06ce13da4747cde8529cc4fe86a81a6 height=138839 log2_work=65.787511 tx=1169347 date=2011-07-30 19:22:59 progress=0.007195 cache=171211 2015-06-03 20:50:19 UpdateTip: new best=00000000000007fadd5d4e5e0210a8bbfd4eca94af0bb2b63de6898e56c7f983 height=138840 log2_work=65.787675 tx=1169378 date=2011-07-30 19:37:30 progress=0.007195 cache=171228 ... for hundreds of lines This time Bitcoin-qt never even showed me a dialog box telling me it encountered an error. It simply crashed out. I'd invoked it from a terminal window and on crashing I see the following in the terminal: *** glibc detected *** bitcoin-qt: double free or corruption (out): 0x2b308d40 *** ======= Backtrace: ========= /lib/i386-linux-gnu/libc.so.6(+0x74f82)[0x2a61f82] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x1f)[0x96fc51f] bitcoin-qt(+0x291f0f)[0x5c8f0f] bitcoin-qt(+0x28eff9)[0x5c5ff9] ... followed by several similar lines and then a 'memory map', however that helps. HOWEVER, I have now restarted bitcoin-qt, and after a while it just resumes the synchronisation. So with a bit of luck it will simply carry on with no further crashing. Do you reckon it's dodgy memory? It looks like dodgy memory to me.
|
If I've said anything amusing and/or informative and you're feeling generous: 1GNJq39NYtf7cn2QFZZuP5vmC1mTs63rEW
|
|
|
achow101
Moderator
Legendary
Offline
Activity: 3486
Merit: 6824
Just writing some code
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 03, 2015, 09:23:11 PM |
|
Those lines of the debug.log are for the synchronization of the blockchain.
I'm not sure what is happening, it could be bad memory, maybe run a memory check on the laptop?
|
|
|
|
cakir
Legendary
Offline
Activity: 1274
Merit: 1000
★ BitClave ICO: 15/09/17 ★
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 03, 2015, 09:25:51 PM |
|
It's probably because of the predefined settings. Can you try to change datadir of blockchain?
something like that; "./bitcoin -datadir=/somewhereElse/ "
|
|
|
|
| ,'#██+: ,█████████████' +██████████████████ ;██████████████████████ ███████: .███████` ██████ ;█████' `█████ #████# ████+ `████+ ████: ████, ████: .# █ ████ ;███+ ██ ███ ████ ████ ███' ███. '███, +███ #████ ,████ ████ ████ █████ .+██████: █████+ `███. ,███ ███████████████████████ ████ ████ ███████████████████████' :███ ███: +████████████████████████ ███` ███ █████████████████████████` ███+ ,███ ██████████████████████████ #███ '███ '██████████████████████████ ;███ #███ ███████████████████████████ ,███ ████ ███████████████████████████. .███ ████ ███████████████████████████' .███ +███ ███████████████████████████+ :███ :███ ███████████████████████████' +███ ███ ███████████████████████████. ███# ███. #██████████████████████████ ███, ████ █████████████████████████+ `███ '███ '████████████████████████ ████ ███; ███████████████████████ ███; ████ #████████████████████ ████ ███# .██████████████████ `███+ ████` ;██████████████ ████ ████ '███████#. ████. .████ █████ '████ █████ #████' █████ +█████` ██████ ,██████: `███████ ████████#;,..:+████████. ,███████████████████+ .███████████████; `+███████#,
| |
|
|
|
Nancarrow (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 03, 2015, 09:54:22 PM |
|
Well, just to give an update... bitcoin-qt is still running, it hasn't crashed... but I don't like what it IS doing. Basically block downloading stalled 40 min ago, as shown by the timestamps for blk00006.dat and rev00006.dat. It has been stuck at block #156748 all this time. What IS being continually updated is debug.log, which is now almost 900MB!! The last few lines of debug.log now are: 2015-06-03 21:50:46 ERROR: CheckBlock() : hashMerkleRoot mismatch 2015-06-03 21:50:46 Misbehaving: 83.85.132.142:8333 (329801600 -> 329801700) 2015-06-03 21:50:46 ERROR: ConnectTip() : ConnectBlock 0000000000000088dc6197ddd62a76882d4d6ec4ea619e79d51fdecab423c71d failed 2015-06-03 21:50:46 ERROR: CheckBlock() : hashMerkleRoot mismatch 2015-06-03 21:50:46 Misbehaving: 83.85.132.142:8333 (329801700 -> 329801800) 2015-06-03 21:50:46 ERROR: ConnectTip() : ConnectBlock 0000000000000088dc6197ddd62a76882d4d6ec4ea619e79d51fdecab423c71d failed I presume that most of the 900MB is like that. Core is also telling me that there are 10 active connections to the network which sounds like *that* isn't the problem. GRR! ![Angry](https://bitcointalk.org/Smileys/default/angry.gif) If it doesn't like the last few blocks for some reason, why the hell can't it just go back to whenever it DID like them and restart the download from there? And how about trimming debug.log while you're at it? ETA: hmm wait a minute... are those lines telling me that it's being fed bad data by some peer? But if so, why doesn't it just try someone else?
|
If I've said anything amusing and/or informative and you're feeling generous: 1GNJq39NYtf7cn2QFZZuP5vmC1mTs63rEW
|
|
|
grue
Legendary
Offline
Activity: 2058
Merit: 1434
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 03, 2015, 10:04:09 PM |
|
ETA: hmm wait a minute... are those lines telling me that it's being fed bad data by some peer? But if so, why doesn't it just try someone else?
you can try -connect=191.236.50.217 to connect only to my node and download from there. You might want to delete existing block data prior to doing this to clear any bad block data from before.
|
|
|
|
Nancarrow (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 03, 2015, 10:39:23 PM |
|
Okay trying that. It's only early days yet (~block #46000) but it hasn't crashed yet.
A more general question: if I do somehow end up with a big blockchain that's mostly correct until the last little bit gets corrupted, what exactly would I have to delete to 'roll it back'? I get that I'd have to delete the last blk00xxx.dat, but what other files? I'd hate to have 19GB of perfect blockchain data and have it redownload every damn thing from the beginning.
|
If I've said anything amusing and/or informative and you're feeling generous: 1GNJq39NYtf7cn2QFZZuP5vmC1mTs63rEW
|
|
|
achow101
Moderator
Legendary
Offline
Activity: 3486
Merit: 6824
Just writing some code
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 03, 2015, 11:57:11 PM |
|
I think that you need to delete the latest blk....dat files, latest rev.....dat files, and the latest .ldb files in blocks/index. However, I'm not very sure about this. You might want to check this with someone else.
|
|
|
|
Nancarrow (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 05, 2015, 04:43:59 PM |
|
Hi all, just to keep you updated... I still haven't managed to sync up the blockchain completely. The client does seem to keep crashing out sometimes. Half the time it says 'a fatal internal error occurred' and shuts down nicely, the other half it just quits and I see a core dump.
Removing the last, incomplete blk00xxx.dat at least allows it to restart, but I notice it always insists on reindexing from the beginning, even when I leave all the rev00xxx.dats alone. But at least I don't have to keep redownloading the blockchain-so-far so that's something.
Anyway I seem to be okay up to blk00133.dat, apparently that's up to "2 years and 24 weeks behind" so we shall see. But it's still in the process of reindexing all that.
|
If I've said anything amusing and/or informative and you're feeling generous: 1GNJq39NYtf7cn2QFZZuP5vmC1mTs63rEW
|
|
|
Nancarrow (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 06, 2015, 11:03:00 AM |
|
Okay I'm at my wit's end here.
Please, if somebody could talk me through this I'd be grateful. Apologies for the vitriol in this post. Please be assured it is directed at the general nature of computers, not at you.
Let's be clear, I'm NOT expecting anybody to get me to a fully synced up chain here. My goals are far, FAR more modest.
First off: suppose I have deleted EVERYTHING in the .bitcoin folder with the SOLE exception of the blocks/blk00xxx.dat files, of which I have the first 146. That's some 16GB of data that I'm really, really hoping bitcoin-qt isn't TOO STUPID TO SEE. No, Mr. Bitcoin, you do NOT need to start downloading the blocks all over again, they are RIGHT FUCKING HERE. How do I translate that previous sentence into something Mr. Bitcoin can understand?
The problem is I started it again and thought well I'll let it dl them for a bit, I now have blk00000 and 00001.dat that are timestamped AFTER the blk00002-00146.dat. Surely that can't be a problem? Surely Mr. Bitcoin can't decide he's going to ignore the subsequent block files because of their timestamp? Please tell me the code can't be designed THAT badly?
|
If I've said anything amusing and/or informative and you're feeling generous: 1GNJq39NYtf7cn2QFZZuP5vmC1mTs63rEW
|
|
|
achow101
Moderator
Legendary
Offline
Activity: 3486
Merit: 6824
Just writing some code
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 06, 2015, 03:48:34 PM |
|
First off: suppose I have deleted EVERYTHING in the .bitcoin folder with the SOLE exception of the blocks/blk00xxx.dat files, of which I have the first 146.
Therein lies the problem. the blk files are just the raw data from the network dumped to the disk. What Bitcoin Core actually uses for everything are database files. These are found in blocks/index which contains data on where to find blocks, and the chainstate directory in the datadir. The chainstate directory is the most important regarding the blockchain. From the Bitcoin Wiki: chainstate subdirectory [v0.8 and above] A LevelDB database with a compact representation of all currently unspent transaction outputs and some metadata about the transactions they are from. The data here is necessary for validating new incoming blocks and transactions. It can theoretically be rebuilt from the block data (see the -reindex command line option), but this takes a rather long time. Without it, you could still theoretically do validation indeed, but it would mean a full scan through the blocks (7 GB as of may 2013) for every output being spent. By deleting this directory, Bitcoin Core must rescan the entire blockchain from the blk files in order to rebuild these databases. It is not redownloading them either. The different timestamps that you see is because Bitcoin Core is reading through every single blk file and rescanning them to rebuild the database.
|
|
|
|
Nancarrow (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 06, 2015, 04:35:52 PM |
|
Okay thanks that makes it somewhat clearer. In that case I shall just let it rescan to its heart's content.
ETA: actually I'll ask while I'm here: what's the difference, if any, between the process you just described, and the process of 'reindexing' that you can force it to do with the '--reindex' option? I ask because I've just started it up again and the status bar at the bottom says "Synchronizing with network..." which is what made me worry it's redownloading blocks.
I shall scoot along to the wiki to see if I can get to grips with what all the different files are trying to do... I understand the blk00xxx.dat files themselves, but I don't yet understand the rev00xxx.dat files, the index/000xxx.ldb files (and a .log file in with them too), and the stuff in the chainstate dir. Anyway ta for the pointers.
|
If I've said anything amusing and/or informative and you're feeling generous: 1GNJq39NYtf7cn2QFZZuP5vmC1mTs63rEW
|
|
|
xmaxbit
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 06, 2015, 05:01:22 PM |
|
I have no idea about it . I am using blockchain.info as my wallet and now its saying "Chain Head Not Found" . Can anyone tell me about it please ?
|
|
|
|
shorena
Copper Member
Legendary
Offline
Activity: 1498
Merit: 1530
No I dont escrow anymore.
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 06, 2015, 05:12:21 PM |
|
I have no idea about it . I am using blockchain.info as my wallet and now its saying "Chain Head Not Found" . Can anyone tell me about it please ?
Different issue, bc.i is a service that currently has problems, OP is talking about THE blockchain.
|
Im not really here, its just your imagination.
|
|
|
achow101
Moderator
Legendary
Offline
Activity: 3486
Merit: 6824
Just writing some code
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 06, 2015, 10:02:55 PM |
|
Okay thanks that makes it somewhat clearer. In that case I shall just let it rescan to its heart's content.
ETA: actually I'll ask while I'm here: what's the difference, if any, between the process you just described, and the process of 'reindexing' that you can force it to do with the '--reindex' option? I ask because I've just started it up again and the status bar at the bottom says "Synchronizing with network..." which is what made me worry it's redownloading blocks.
It should say "reindexing blocks on disk..." when it is reindexing. However, yours doesn't. Perhaps it is because you haven't downloaded to entire blockchain yet. Since it indexes the blocks as it downloads, I think that since you don't have the full blockchain on the disk yet, it will both reindex and download the blocks, and say "Synchronizing with network..." I shall scoot along to the wiki to see if I can get to grips with what all the different files are trying to do... I understand the blk00xxx.dat files themselves, but I don't yet understand the rev00xxx.dat files, the index/000xxx.ldb files (and a .log file in with them too), and the stuff in the chainstate dir. Anyway ta for the pointers.
You can read some more about the data directory on the Bitcoin Wiki here: https://en.bitcoin.it/wiki/Data_directory
|
|
|
|
Nancarrow (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 06, 2015, 10:19:12 PM Last edit: June 06, 2015, 10:46:02 PM by Nancarrow |
|
I already read that wiki entry and didn't find it too enlightening. But I shall keep looking for the grittier technical details.
Anyway on to the next question in my series, "How to Eventually Reverse Engineer the Complete Bitcoin-Qt Code by Asking About Every Damn Thing that is Bugging Me"...
Up until blk00070.dat, it has been rebuilding the database, and timestamping the blockfiles, at a rate of about one blockfile every two or three minutes. BUT, for the last 40 minutes or so, nothing. Why not? The blockfiles up to 00145 are all sitting there, waiting hungrily to be reindexed. What's the holdup? The "Synchronising with the network" progress bar has also halted.
I'll give it another half hour and then close and restart bitcoin-qt.
ETA: EEEEEK! I've just looked at debug.log to see what it's doing. So... 70MB of cruft in there. Now some of it is, I understand, perfectly OK. Fr'instance stuff like this:
2015-06-06 21:06:52 UpdateTip: new best=0000000000000103b325ddcc16512a6754e710c387d069a17e217169f293447c height=233167 log2_work=69.868797 tx=16765534 date=2013-04-26 01:02:19 progress=0.102582 cache=109732
And this:
2015-06-06 21:06:52 Pre-allocating up to position 0xa00000 in rev00056.dat
... though even then I must question why it's storing so damn MUCH of it. From block 226924 to block 246158! I know debug.log is supposed to be 'flushed' at intervals but couldn't we make those intervals a little more... timely?
Anyway I don't mind, but THEN I see this:
2015-06-06 21:38:50 ERROR: CheckBlock() : hashMerkleRoot mismatch 2015-06-06 21:38:50 Misbehaving: 71.93.44.219:8333 (0 -> 100) BAN THRESHOLD EXCEEDED 2015-06-06 21:38:50 ERROR: ConnectTip() : ConnectBlock 000000000000004bbd569ecddf773e8f2d92345fb7027ead5c89221196cf9ec3 failed 2015-06-06 21:38:50 ERROR: CheckBlock() : hashMerkleRoot mismatch 2015-06-06 21:38:50 Misbehaving: 71.93.44.219:8333 (100 -> 200) 2015-06-06 21:38:50 ERROR: ConnectTip() : ConnectBlock 000000000000004bbd569ecddf773e8f2d92345fb7027ead5c89221196cf9ec3 failed 2015-06-06 21:38:50 ERROR: CheckBlock() : hashMerkleRoot mismatch 2015-06-06 21:38:50 Misbehaving: 71.93.44.219:8333 (200 -> 300) 2015-06-06 21:38:50 ERROR: ConnectTip() : ConnectBlock 000000000000004bbd569ecddf773e8f2d92345fb7027ead5c89221196cf9ec3 failed 2015-06-06 21:38:50 ERROR: CheckBlock() : hashMerkleRoot mismatch
... and so on, for the vast bulk of the file, with the timestamp up front increasing by the seconds... for about 40 minutes until I stop it!
Okay... WHY???
1) Why is there a hashMerkleRoot mismatch? 2) Why is 71.93.44.219 misbehaving? 3) Why should bitcoin-qt CARE about some node misbehaving when it is allegedly NOT downloading blocks, but simply rebuilding the database on the blocks it has? 4) Obvious follow-on... is it, in fact, despite what you said, in *fact* redownloading those blocks after all? 5) After the first 10000 times* of printing this little set of error messages, should not the client somehow grasp that what it is doing is NOT WORKING and, just perhaps, try another node? Or even give up and say "There is something wrong, I'm stopping now."?
*obvious snark - I know what computers can and cannot do. I suppose my snark is really directed at all these coders who must somehow have never seen this behaviour before and never thought to put a little 'break' in that loop.
|
If I've said anything amusing and/or informative and you're feeling generous: 1GNJq39NYtf7cn2QFZZuP5vmC1mTs63rEW
|
|
|
achow101
Moderator
Legendary
Offline
Activity: 3486
Merit: 6824
Just writing some code
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 06, 2015, 10:28:04 PM |
|
I already read that wiki entry and didn't find it too enlightening. But I shall keep looking for the grittier technical details.
Anyway on to the next question in my series, "How to Eventually Reverse Engineer the Complete Bitcoin-Qt Code by Asking About Every Damn Thing that is Bugging Me"...
I suppose if you really want to get technical and if you can read code, you could check out the developer docs here: https://dev.visucore.com/bitcoin/doxygen/. These are useful as you can see paths of which method calls what. Then, if you can read code, you can look at the source code provided on the site and try to find how everything works. There are some explanations as these are developer docs, but not very many. Up until blk00070.dat, it has been rebuilding the database, and timestamping the blockfiles, at a rate of about one blockfile every two or three minutes. BUT, for the last 40 minutes or so, nothing. Why not? The blockfiles up to 00145 are all sitting there, waiting hungrily to be reindexed. What's the holdup? The "Synchronising with the network" progress bar has also halted.
I'll give it another half hour and then close and restart bitcoin-qt.
I'm not sure why this would happen. Try looking in the debug.log and see if it is giving you any error messages. Also, if you hover the mouse over the progress bar, it should say on what block it is on. If you do it again a minute later and the block number hasn't changed, it likely has hung on something, and you should restart it.
|
|
|
|
Nancarrow (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 06, 2015, 10:42:05 PM |
|
Thanks for your reply, obviously I've just been spending a bit too much time seething and updating my previous post! I will check out the developer-docs link you gave, and in the meantime delete debug.log, and restart bitcoin-qt, probably I'll point it at grue's node.
|
If I've said anything amusing and/or informative and you're feeling generous: 1GNJq39NYtf7cn2QFZZuP5vmC1mTs63rEW
|
|
|
2112
Legendary
Offline
Activity: 2128
Merit: 1073
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 06, 2015, 10:44:12 PM |
|
*obvious snark - I know what computers can and cannot do. I suppose my snark is really directed at all these coders who must somehow have never seen this behaviour before.
Your hardware is broken. Get another, correctly working, machine and all your problems will disappear. Why you keep torturing yourself and us if you know how the computers work?
|
|
|
|
|