Bitcoin Core sync 'stuck'

[jbreher]

I am on
Bitcoin Core version v0.9.2.1-g354c03f-beta (64-bit)

On
Mac OS X 10.7.5
2.5 G i7, 16 GB mem

Last night, I fired up Bitcoin for the first time in about a week. It synced some, then got stuck at block 322870.

Quit & restart, stuck at block 322870.

Quit, restart with -reindex - churned for a while then got 'database corrupted' message.

Restored Bitcoin-QT/[blocks chainstate] from about a week ago.

Restarted - churned for a while, now stuck on block 322870.

What's so special about block 322870 that I can't seem to sync past it?

tail of log file:

Code:

  forking the chain at height 322870 (00000000000000001f7bbfa5ab7f7883d372065d6f18c9ab91dcfb891394c8c8)
  lasting to height 323012 (000000000000000019b16f4a6cd8717ab8f4836bbc9822894cc49785badfdaa7).
Chain state database corruption likely.
2014-09-29 04:25:05 ProcessBlock: ACCEPTED
2014-09-29 04:29:42 CheckForkWarningConditions: Warning: Large valid fork found
  forking the chain at height 322870 (00000000000000001f7bbfa5ab7f7883d372065d6f18c9ab91dcfb891394c8c8)
  lasting to height 323013 (00000000000000000057d23bab8b8449858c05fcf70b2079ed233ad6380980a7).
Chain state database corruption likely.
2014-09-29 04:29:42 ProcessBlock: ACCEPTED
2014-09-29 04:43:15 CheckForkWarningConditions: Warning: Large valid fork found
  forking the chain at height 322870 (00000000000000001f7bbfa5ab7f7883d372065d6f18c9ab91dcfb891394c8c8)
  lasting to height 323014 (00000000000000001ae5fbe7ef691842dcea1bdfa037eb137755090e490bcafb).
Chain state database corruption likely.
2014-09-29 04:43:15 ProcessBlock: ACCEPTED
2014-09-29 04:54:38 ERROR: CheckInputs() : tried to spend coinbase at depth 26
2014-09-29 04:54:38 ERROR: AcceptToMemoryPool: : ConnectInputs failed bc1a10b54a024e280d7c309edb8c24e932b7243da28644657bd43540dcef2f82
2014-09-29 04:56:38 ERROR: CheckInputs() : tried to spend coinbase at depth 26
2014-09-29 04:56:38 ERROR: AcceptToMemoryPool: : ConnectInputs failed bc1a10b54a024e280d7c309edb8c24e932b7243da28644657bd43540dcef2f82
2014-09-29 04:58:19 ERROR: CheckInputs() : tried to spend coinbase at depth 26
2014-09-29 04:58:19 ERROR: AcceptToMemoryPool: : ConnectInputs failed a8515c17bd06ab0729ab9a803ec7d6dc1916258c7f4928ddf07b37354d52ae43
2014-09-29 04:58:31 CheckForkWarningConditions: Warning: Large valid fork found
  forking the chain at height 322870 (00000000000000001f7bbfa5ab7f7883d372065d6f18c9ab91dcfb891394c8c8)
  lasting to height 323015 (0000000000000000092e056ab9c5a3242bfa911ba9a0a8a3836bcd4b5fcab2f8).
Chain state database corruption likely.
2014-09-29 04:58:31 ProcessBlock: ACCEPTED
2014-09-29 04:58:37 ERROR: CheckInputs() : tried to spend coinbase at depth 26
2014-09-29 04:58:37 ERROR: AcceptToMemoryPool: : ConnectInputs failed bc1a10b54a024e280d7c309edb8c24e932b7243da28644657bd43540dcef2f82
2014-09-29 05:00:19 ERROR: CheckInputs() : tried to spend coinbase at depth 26
2014-09-29 05:00:19 ERROR: AcceptToMemoryPool: : ConnectInputs failed a8515c17bd06ab0729ab9a803ec7d6dc1916258c7f4928ddf07b37354d52ae43
2014-09-29 05:00:38 ERROR: CheckInputs() : tried to spend coinbase at depth 26
2014-09-29 05:00:38 ERROR: AcceptToMemoryPool: : ConnectInputs failed bc1a10b54a024e280d7c309edb8c24e932b7243da28644657bd43540dcef2f82
2014-09-29 05:02:18 ERROR: CheckInputs() : tried to spend coinbase at depth 26
2014-09-29 05:02:18 ERROR: AcceptToMemoryPool: : ConnectInputs failed a8515c17bd06ab0729ab9a803ec7d6dc1916258c7f4928ddf07b37354d52ae43
2014-09-29 05:02:38 ERROR: CheckInputs() : tried to spend coinbase at depth 26
2014-09-29 05:02:38 ERROR: AcceptToMemoryPool: : ConnectInputs failed bc1a10b54a024e280d7c309edb8c24e932b7243da28644657bd43540dcef2f82
2014-09-29 05:04:18 ERROR: CheckInputs() : tried to spend coinbase at depth 26
2014-09-29 05:04:18 ERROR: AcceptToMemoryPool: : ConnectInputs failed a8515c17bd06ab0729ab9a803ec7d6dc1916258c7f4928ddf07b37354d52ae43
2014-09-29 05:04:37 ERROR: CheckInputs() : tried to spend coinbase at depth 26
2014-09-29 05:04:37 ERROR: AcceptToMemoryPool: : ConnectInputs failed bc1a10b54a024e280d7c309edb8c24e932b7243da28644657bd43540dcef2f82
2014-09-29 05:06:19 ERROR: CheckInputs() : tried to spend coinbase at depth 26
2014-09-29 05:06:19 ERROR: AcceptToMemoryPool: : ConnectInputs failed a8515c17bd06ab0729ab9a803ec7d6dc1916258c7f4928ddf07b37354d52ae43
2014-09-29 05:06:38 ERROR: CheckInputs() : tried to spend coinbase at depth 26
2014-09-29 05:06:38 ERROR: AcceptToMemoryPool: : ConnectInputs failed bc1a10b54a024e280d7c309edb8c24e932b7243da28644657bd43540dcef2f82
2014-09-29 05:08:19 ERROR: CheckInputs() : tried to spend coinbase at depth 26
2014-09-29 05:08:19 ERROR: AcceptToMemoryPool: : ConnectInputs failed a8515c17bd06ab0729ab9a803ec7d6dc1916258c7f4928ddf07b37354d52ae43
2014-09-29 05:08:38 ERROR: CheckInputs() : tried to spend coinbase at depth 26
2014-09-29 05:08:38 ERROR: AcceptToMemoryPool: : ConnectInputs failed bc1a10b54a024e280d7c309edb8c24e932b7243da28644657bd43540dcef2f82
2014-09-29 05:10:19 ERROR: CheckInputs() : tried to spend coinbase at depth 26
2014-09-29 05:10:19 ERROR: AcceptToMemoryPool: : ConnectInputs failed a8515c17bd06ab0729ab9a803ec7d6dc1916258c7f4928ddf07b37354d52ae43
2014-09-29 05:12:19 ERROR: CheckInputs() : tried to spend coinbase at depth 26
2014-09-29 05:12:19 ERROR: AcceptToMemoryPool: : ConnectInputs failed a8515c17bd06ab0729ab9a803ec7d6dc1916258c7f4928ddf07b37354d52ae43
2014-09-29 05:14:41 CheckForkWarningConditions: Warning: Large valid fork found
  forking the chain at height 322870 (00000000000000001f7bbfa5ab7f7883d372065d6f18c9ab91dcfb891394c8c8)
  lasting to height 323016 (000000000000000010807c7fcd7d76520353fb79562e155126cdd6e412742c96).
Chain state database corruption likely.
2014-09-29 05:14:41 ProcessBlock: ACCEPTED
2014-09-29 05:15:16 ERROR: CheckInputs() : tried to spend coinbase at depth 67
2014-09-29 05:15:16 ERROR: AcceptToMemoryPool: : ConnectInputs failed 64183b9ea2f93dd8a48ab61a699dd34fb21440badbe885d91d49ead73a9aa46b
2014-09-29 05:17:15 ERROR: CheckInputs() : tried to spend coinbase at depth 67
2014-09-29 05:17:15 ERROR: AcceptToMemoryPool: : ConnectInputs failed 64183b9ea2f93dd8a48ab61a699dd34fb21440badbe885d91d49ead73a9aa46b
2014-09-29 05:19:15 ERROR: CheckInputs() : tried to spend coinbase at depth 67
2014-09-29 05:19:15 ERROR: AcceptToMemoryPool: : ConnectInputs failed 64183b9ea2f93dd8a48ab61a699dd34fb21440badbe885d91d49ead73a9aa46b
2014-09-29 05:21:15 ERROR: CheckInputs() : tried to spend coinbase at depth 67
2014-09-29 05:21:15 ERROR: AcceptToMemoryPool: : ConnectInputs failed 64183b9ea2f93dd8a48ab61a699dd34fb21440badbe885d91d49ead73a9aa46b
2014-09-29 05:22:14 CheckForkWarningConditions: Warning: Large valid fork found
  forking the chain at height 322870 (00000000000000001f7bbfa5ab7f7883d372065d6f18c9ab91dcfb891394c8c8)
  lasting to height 323017 (00000000000000001ab82f6e47b65eaf58f2df3b2bf50663aad70b9619691fd5).
Chain state database corruption likely.
2014-09-29 05:22:14 ProcessBlock: ACCEPTED
2014-09-29 05:22:54 ERROR: CheckInputs() : tried to spend coinbase at depth 45
2014-09-29 05:22:54 ERROR: AcceptToMemoryPool: : ConnectInputs failed 5f4545d58667d0b8a9637698529f3146688a7ef53d62262cf642a5bd3bb0eaa3
2014-09-29 05:23:15 ERROR: CheckInputs() : tried to spend coinbase at depth 67
2014-09-29 05:23:15 ERROR: AcceptToMemoryPool: : ConnectInputs failed 64183b9ea2f93dd8a48ab61a699dd34fb21440badbe885d91d49ead73a9aa46b
2014-09-29 05:24:53 ERROR: CheckInputs() : tried to spend coinbase at depth 45
2014-09-29 05:24:53 ERROR: AcceptToMemoryPool: : ConnectInputs failed 5f4545d58667d0b8a9637698529f3146688a7ef53d62262cf642a5bd3bb0eaa3
2014-09-29 05:25:15 ERROR: CheckInputs() : tried to spend coinbase at depth 67
2014-09-29 05:25:15 ERROR: AcceptToMemoryPool: : ConnectInputs failed 64183b9ea2f93dd8a48ab61a699dd34fb21440badbe885d91d49ead73a9aa46b
2014-09-29 05:25:54 CheckForkWarningConditions: Warning: Large valid fork found
  forking the chain at height 322870 (00000000000000001f7bbfa5ab7f7883d372065d6f18c9ab91dcfb891394c8c8)
  lasting to height 323018 (000000000000000017363b882751f64830786000e1d2ea1b59cce501354074a6).
Chain state database corruption likely.
2014-09-29 05:25:54 ProcessBlock: ACCEPTED
2014-09-29 05:26:53 ERROR: CheckInputs() : tried to spend coinbase at depth 45
2014-09-29 05:26:53 ERROR: AcceptToMemoryPool: : ConnectInputs failed 5f4545d58667d0b8a9637698529f3146688a7ef53d62262cf642a5bd3bb0eaa3
 

[1714609233]

1714609233

[1714609233]

1714609233

[BitcoinPopulation.com]

Perhaps you could try Bitcoin Core v0.9.3

[jbreher]

Another day has passed. Still stuck at the same point.

Perhaps you could try Bitcoin Core v0.9.3

Perhaps I could. Any reason to expect it might solve my issue?

I guess I was hoping there would be some sort of a clue in the log output. Rather than just bulldoze around it, this might be a learning opportunity.

But if nobody finds anything probative in my log output before too long, I guess that's what I'll need to do.

[poewerden]

I got stuck in the past as well. Deleting blockchain so it can be redownloaded from scratch works. Now I making backups of blockchain from time to time so if this happens I dont need to dowload full blockchain again...

[jbreher]

I got stuck in the past as well. Deleting blockchain so it can be redownloaded from scratch works. Now I making backups of blockchain from time to time so if this happens I dont need to dowload full blockchain again...

Perhaps you missed where I noted that I restored the blockchain from an earlier backup. Unless I needed to include something other than the blocks folder and the chainstate folder.

I'll try an earlier backup - though things were working past the day of the selected backup.

[jbreher]

Nope - no good. Restored blockchain from Sep 18 backup chokes at the same block 322870.

WTF is it with block 322870?

Is there nobody that can has the knowledge to help me decipher my log output?

[Newar]

Did you do another -reindex after you restored from backup?

[jbreher]

Did you do another -reindex after you restored from backup?

Nope. I'll give that a try now.

...aaaand es no bueno. Reindex results in seg fault within ten seconds:

Code:

Process:         Bitcoin-Qt [22096]
Path:            /Applications/Bitcoin-Qt.app/Contents/MacOS/./Bitcoin-Qt
Identifier:      org.bitcoinfoundation.Bitcoin-Qt
Version:         0.9.2 (0.9.2)
Code Type:       X86-64 (Native)
Parent Process:  bash [14270]

Date/Time:       2014-09-30 16:29:15.551 -0600
OS Version:      Mac OS X 10.7.5 (11G63)
Report Version:  9

Interval Since Last Report:          509477 sec
Crashes Since Last Report:           5
Per-App Interval Since Last Report:  332521 sec
Per-App Crashes Since Last Report:   2
Anonymous UUID:                      2073DD49-7F33-4177-827A-9A4C34ABF7BA

Crashed Thread:  0  Dispatch queue: com.apple.main-thread

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x00007fd2a87e342b

VM Regions Near 0x7fd2a87e342b:
    MALLOC_TINY            00007fb9d7800000-00007fb9d8a00000 [ 18.0M] rw-/rwx SM=PRV  
--> 
    STACK GUARD            00007fff5e142000-00007fff61942000 [ 56.0M] ---/rwx SM=NUL  stack guard for thread 0

Application Specific Information:
objc[22096]: garbage collection is OFF

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
...

[cr1776]

One thing to also check is to ensure "app nap" is off. That shouldn't cause a crash though.

Did you try running disk util to repair disk and permissions?  (Make sure you have many backups of wallet.dat)  This could be caused by something unrelated to bitcoin.

[jbreher]

News is good but confusing.

Before I saw the suggestion to repair disk & perms, I deleted [blocks chainstate database]

Upon restarting QT, I was met with a dialog saying something akin to 'crash during window creation'. There were two buttons, one of which was similar to 'restore windows', which I clicked.
- I was met with another dialog sating something like 'wallet corrupt'.

I restarted QT, and it rebuilt the blockchain from scratch.
- ~28 hours later, I had caught up to the current blockchain, but I was showing 0 BTC.

Stopped QT, Restored wallet.dat from Sep 25, restarted QT.
- my bitcoin were back.

Whew.

The confusing part: before things went south (i.e. getting stuck @ block 322870), I had previously restored [blocks chainstate] from the very same backup that did not work earlier upthread. Ergo, it would seem that there may be some database corruption errors that are not caught until some number of blocks later, when they fail without any meaningful indication.