Problems with Bitcoin Core

[chiropteran]

I don't run the bitcoin core program regularly, I normally just run it once a month or so to synchronize the latest transactions.

Anyway, I ran it a couple weeks ago for the first time recently, it started downloading transactions but after some time it failed with an error indicating a corrupted database of some sort. I don't have the exact error message saved, unfortunately.

I figured I could simply remove the existing database and re-download all the transactions to fix this, so I attempted doing that. I deleted my bitcoin directory, downloaded bitcoin-qt 0.10.2, installed and ran. It started downloading transactions, and this process took over 24 hours (as expected) to rebuild the database.  However, it never finished- it hung up on some error message before completing, and never worked. After 0.11 was released a few days ago, I tried the same with that, and a similar result occurred- after over 24 hours of downloading and confirming transactions, it threw an error and stopped working.

Is this indicative of a hardware problem on my computer? It really never crashes, it's been perfectly stable generally, and other programs work fine. I ran some memory tests and disk scans just to be sure, and I tried switching the bitcoin data directory to an alternate drive and nothing has improved.

Is there some part of 'bitcoin core' that is saved somewhere other than the install folder and the bitcoin folder (the one under my profile)? I have been deleting these to try to do a clean install, but maybe there is some other folder I need to be deleting also?

[1715404059]

1715404059

[1715404059]

1715404059

[achow101]

What was the error message? Maybe you are running out of disk space?

[chiropteran]

I should have taken a screen shot, I can't recall the exact phrasing.

Actually, I googled the errors when I saw them, to try to figure out what was going on, so I can check my chrome history and bring up the searches I did.

First error seen was "error reading from database" or some approximation, this was back on 0.10.0

"error reading from database, shutting down" was also occurring after my first attempt to fix the problem by rebuilding.

The last error (on 0.11) -

https://www.google.com/#q=bitcoin+-+error+error:+a+fatal+internal+error+occured%2C+see+debug.log+for+details.

edit: Disk space is fine I think, had more than 20GB free when the error occurred last time.

[Borisz]

However, it never finished- it hung up on some error message before completing, and never worked.

So it downloaded almost the entire chain and got hung at the end? What happened when you restarted the wallet afterwards?

[chiropteran]

However, it never finished- it hung up on some error message before completing, and never worked.

So it downloaded almost the entire chain and got hung at the end? What happened when you restarted the wallet afterwards?

Sometimes it would immediately give another error (not sure which error this was) and close, other times it would hang during the loading process, before it even shows the wallet and block progress, and after hanging for a couple hours I gave up and force quit and tried again from a fresh new start (deleted bitcoin folder etc).

[Borisz]

However, it never finished- it hung up on some error message before completing, and never worked.

So it downloaded almost the entire chain and got hung at the end? What happened when you restarted the wallet afterwards?

Sometimes it would immediately give another error (not sure which error this was) and close, other times it would hang during the loading process, before it even shows the wallet and block progress, and after hanging for a couple hours I gave up and force quit and tried again from a fresh new start (deleted bitcoin folder etc).

Not sure if it has to do anything with this, but when I was experiencing slow starts and hungs for a long time I used

Code:

checkblocks=50

in my bitcoin.conf

This tells the client to only check the last 50 blocks and speed up everything. 50 should also be safe even with the latest blockchain split issue, based on the recommended 30 blocks confirmation waiting time.

[chiropteran]

I just got back to my home PC, after restarting this morning I see the error occurred again. 
Sync was 1 year and 17 weeks behind, error is "Error: A fatal internal error occurred, see debug.log for details."

debug.log is a rather huge file, the end is as follows:

Code:

2015-07-13 13:58:10 UpdateTip: new best=00000000000000000ef5c4a6f6a2559ba5452031ca0b21e6aee9ea75c3228c5c  height=290078  log2_work=77.202684  tx=34638586  date=2014-03-11 19:17:58 progress=0.198562  cache=62.9MiB(27370tx)
2015-07-13 13:58:10 UpdateTip: new best=0000000000000000bb67842942050e16f8be923fa234f185fcb6aa7406790444  height=290079  log2_work=77.20282  tx=34639235  date=2014-03-11 19:29:51 progress=0.198566  cache=63.9MiB(28440tx)
2015-07-13 13:58:10 UpdateTip: new best=0000000000000000769d317df34a88e6cada3d48b44b099415740ab22576dbde  height=290080  log2_work=77.202956  tx=34640372  date=2014-03-11 19:50:40 progress=0.198573  cache=65.4MiB(30093tx)
2015-07-13 13:58:10 Corruption: block checksum mismatch
2015-07-13 13:58:10 *** System error while flushing: Database corrupted
2015-07-13 14:14:57 UPnP Port Mapping successful.
2015-07-13 14:17:05 socket sending timeout: 1201s
2015-07-13 14:17:42 socket sending timeout: 1201s
2015-07-13 14:18:09 socket sending timeout: 1201s
2015-07-13 14:18:09 socket sending timeout: 1201s
2015-07-13 14:18:09 socket sending timeout: 1201s
2015-07-13 14:18:09 socket sending timeout: 1201s
2015-07-13 14:18:09 socket sending timeout: 1201s
2015-07-13 14:18:09 socket sending timeout: 1201s
2015-07-13 14:34:57 UPnP Port Mapping successful.
2015-07-13 14:54:57 UPnP Port Mapping successful.
2015-07-13 15:14:57 UPnP Port Mapping successful.
2015-07-13 15:34:58 UPnP Port Mapping successful.
2015-07-13 15:54:58 UPnP Port Mapping successful.
2015-07-13 16:14:58 UPnP Port Mapping successful.
2015-07-13 16:34:58 UPnP Port Mapping successful.
2015-07-13 16:54:58 UPnP Port Mapping successful.
2015-07-13 17:14:58 UPnP Port Mapping successful.
2015-07-13 17:34:58 UPnP Port Mapping successful.
2015-07-13 17:54:58 UPnP Port Mapping successful.
2015-07-13 18:14:59 UPnP Port Mapping successful.
2015-07-13 18:34:59 UPnP Port Mapping successful.
2015-07-13 18:54:59 UPnP Port Mapping successful.
2015-07-13 19:14:59 UPnP Port Mapping successful.
2015-07-13 19:34:59 UPnP Port Mapping successful.

[chiropteran]

Not sure if it has to do anything with this, but when I was experiencing slow starts and hungs for a long time I used

Code:

checkblocks=50

in my bitcoin.conf

This tells the client to only check the last 50 blocks and speed up everything. 50 should also be safe even with the latest blockchain split issue, based on the recommended 30 blocks confirmation waiting time.

I got the error still with that added to bitcoin.conf, though it might have loaded quicker.

[2112]

Is this indicative of a hardware problem on my computer? It really never crashes, it's been perfectly stable generally, and other programs work fine. I ran some memory tests and disk scans just to be sure, and I tried switching the bitcoin data directory to an alternate drive and nothing has improved.

Exactly, it is a hardware problem. Bitcoin Core is really well optimized and stresses the machine very close to the theoretical limits. In particular "disc scan" isn't a good test, what you need is a "disk read-modify-write" test.

Also, if your home PC is running Windows the unfortunate standard advice does apply: online antivirus programs are frequent culprits of corruption of databases, especially since a while back when some joker pushed a virus signatures into the Bitcoin blockchain.

For your comparison: I have a very old kitchen computer, original Athlon 64 Clawhammer from 2003 running Windows XP SP1. It still runs fine 4 coin wallets {Bitcoin,Litecoin}*{mainnet,testnet} simultaneously.

Here's another thread from another user who had "mostly stable" computer:

https://bitcointalk.org/index.php?topic=1106669.0

so you could commiserate together.

[soy]

What operating system are you running?  If Win7 or Win8 and only 4 GB of ram, try putting Bitcoin Core in compatibility mode.

[chiropteran]

Running win 8.1 64-bit with all updates with 32GB of RAM.

I'm thinking about how to diagnose the possible hardware issue later tonight, thanks for the suggestions.

[soy]

There was an article among those on hacker news the other day that said Win8 and Win7 both write to a 32 bit register by mistake.  (http://blogs.msdn.com/b/oldnewthing/archive/2015/07/09/10624293.aspx).  I was repeatedly getting the BSOD on Win7.  The BSOD would result in a complete reindexing.  Each time it seemed to get slower.  The BSOD would occur after I got to the realm of new blocks getting downloaded.  Putting the program in Win7 compatibility mode has gotten it up to July 4, 2015 at the present and it should be complete tonight.  I only have 4 gb of ram in that system.

[2112]

Running win 8.1 64-bit with all updates with 32GB of RAM.

I'm thinking about how to diagnose the possible hardware issue later tonight, thanks for the suggestions.

Oh, so it is a very modern hardware. In that case check it up for "row hammer" memory failure. I heard that the hardware that is susceptible to row hammering is being sold cheaply all over the world because the old memory tests don't yet have specific diagnostics for that fault.

[soy]

Just a thought, what is the higher probability, the noted error by Win7 and Win8 or a targeted attack on a new Bitcoin Core installation with fresh addresses not having any btc?

[chiropteran]

No hardware changes done yet. Restarted the current 0.11 bitcoin core install in windows 7 compatibility mode, and it didn't immediately error-out.

It seems to be continuing with synchronization, so I'll let it go and see if it finishes successfully.

[soy]

No hardware changes done yet. Restarted the current 0.11 bitcoin core install in windows 7 compatibility mode, and it didn't immediately error-out.

It seems to be continuing with synchronization, so I'll let it go and see if it finishes successfully.

Mine completed successfully.  Win7 compatibility fixed it.

[chiropteran]

It seems to be continuing with synchronization, so I'll let it go and see if it finishes successfully.

It did not. It seemed to successfully download and process more of the blockchain in win 7 compatibility mode, but eventually it crashed in the same ways as before.

HOWEVER,

I have a Windows Server 2012 VM for testing purposes that wasn't needed for anything else, so I installed bitcoin core 0.11 on that, and it seems to be working perfectly. As of this morning it was almost fully synced up- 16 weeks remaining, which is closer than it ever got on my base Win 8.1 install.

Odd thing to me is that this VM is still residing on the same physical computer, so the exact same underlying hardware. I don't think it's a hardware issue, I really suspect something is lingering around on my win 8.1 install even after uninstalling that is screwing up every "fresh" install I try to do.

In any case, assuming the last 16 weeks of blockchain download and process correctly, I'll just run bitcoin core from my VM for now. I'll probably be doing a fresh install of Windows 10 when it becomes available and then I'll see how things go with that.

[soy]

I should also note I researched the Win7 BSOD problem and followed suggestions like getting rid of a font file (who knows, maybe firefox calls a font windows doesn't use) and wireless problems I bypassed by going back to a wheeled wired mouse, so the failure was possibly more than just that Bitcoin Core needed a Win7 compatibility mode setting. 

[chiropteran]

Just to clarify, this PC hasn't crashed (no BSOD or otherwise) in at least the last 6 months, it is extremely stable.

Other than bitcoin core, I don't see software crash much either.