bitcoin-qt crashing on startup (ubuntu) [RESOLVED: Faulty RAM]

[Jaymax]

Code:

jaymax@Antec:~/.bitcoin$ bitcoin-qt
bitcoin-qt: table/table_builder.cc:97: void leveldb::TableBuilder::Add(const leveldb::Slice&, const leveldb::Slice&): Assertion `r->options.comparator->Compare(key, Slice(r->last_key)) > 0' failed.
Aborted (core dumped)

Hi.  It's a long time since I've used linux, but decided my newly engaged bitcoin adventure was a sensible time to see how things have moved on.  So far, no so much fun.  Appreciate any help I can get.

So, having established an offline LiveCD environment wallet for safekeeping, it was time to install ubuntu to the HDD and get a sync'ed up bitcoin-qt client running, and try sending over some coins to it...  Ignoring for a moment my puzzlement as to why I can install the client from CLI (sudo apt-get install bitcoin-qt) but not from the Software Centre, I've removed and reinstalled the client against the existing data files, with no effect on the error.

The b-qt was syncing up okay, got to about 300 days to go when I last checked, but then died at some point.  This is what seems to be appended to the debug.log (db.log is zero bytes) with each attempted restart:

Code:

Bitcoin version v0.8.1.0-g34d62a8-beta ()
Using OpenSSL version OpenSSL 1.0.1 14 Mar 2012
Startup time: 2013-04-25 10:10:03
Default data directory /home/jaymax/.bitcoin
Used data directory /home/jaymax/.bitcoin
Using 4 threads for script verification
init message: Verifying wallet integrity...
dbenv.open LogDir=/home/jaymax/.bitcoin/database ErrorFile=/home/jaymax/.bitcoin/db.log
Bound to [::]:8333
Bound to 0.0.0.0:8333
init message: Loading block index...
Opening LevelDB in /home/jaymax/.bitcoin/blocks/index
Opened LevelDB successfully
Opening LevelDB in /home/jaymax/.bitcoin/chainstate
Opened LevelDB successfully
LoadBlockIndex(): last block file = 14
LoadBlockIndex(): last block file: CBlockFileInfo(blocks=296, size=23721895, heights=185551..185846, time=2012-06-21..2012-06-23)
LoadBlockIndex(): transaction index disabled
LoadBlockIndex(): hashBestChain=00000000000003a7b870d4f4f619e2d658081920f88d395b3d4cdb7ea58dfc96  height=185816 date=2012-06-23 04:23:02
init message: Verifying block database integrity...
Verifying last 288 blocks at level 3
No coin database inconsistencies in last 289 blocks (51010 transactions)
 block index            4056ms
init message: Loading wallet...
nFileVersion = 80100
 wallet                  662ms
init message: Importing blocks from block database...
SetBestChain: new best=000000000000009dc848a2e74d431cead8259f54aea54724a9460727a85dab48  height=185817  work=363577274360209577448  tx=4401836  date=2012-06-23 04:29:30
SetBestChain: new best=00000000000002656b7cc7df48a5fd77f8fba8ee664805afe8040ff0237ea119  height=185818  work=363584690020269673308  tx=4401849  date=2012-06-23 04:47:40
SetBestChain: new best=0000000000000286910c277b2e60be58733c06c42108f1b414ec4472a2f6d5e3  height=185819  work=363592105680329769168  tx=4402314  date=2012-06-23 04:54:46
SetBestChain: new best=000000000000080bc951a85b543ef8272f2dc4ac062143f80cd3ff27fbfab3e5  height=185820  work=363599521340389865028  tx=4402332  date=2012-06-23 04:56:11
SetBestChain: new best=00000000000001136cc9e179c6dbf99415298a5c1053fdd0211db369737ade24  height=185821  work=363606937000449960888  tx=4402344  date=2012-06-23 05:00:21
SetBestChain: new best=000000000000094baa405a2b9f68de9939124ff1ab7a84b1b9b4c38d50a9b1e8  height=185822  work=363614352660510056748  tx=4402461  date=2012-06-23 05:09:34
SetBestChain: new best=00000000000009009a48704acb7f70e0cd8a2117e0411bad0ad20074d41a576c  height=185823  work=363621768320570152608  tx=4402620  date=2012-06-23 05:16:59
SetBestChain: new best=00000000000002459afbfea5a84b19d49e14650f73ffb0a284df135c7c766b0c  height=185824  work=363629183980630248468  tx=4402682  date=2012-06-23 05:26:33
SetBestChain: new best=0000000000000943b16012b040da2d4e22e80be814f4b7833001a9570bcbecb8  height=185825  work=363636599640690344328  tx=4402899  date=2012-06-23 06:09:19
SetBestChain: new best=00000000000009ae7f1e6746e30636df17a9e68dea730b7c6dd190e12b2a228e  height=185826  work=363644015300750440188  tx=4403495  date=2012-06-23 06:21:17
SetBestChain: new best=000000000000040dfeecf7ed1685e7076cb9d4abb331a97ec95add8b21aa0629  height=185827  work=363651430960810536048  tx=4403687  date=2012-06-23 06:41:21
SetBestChain: new best=0000000000000531b00efcd4957106d08e6ff88e6e9bbed41af226a0bae6e2d0  height=185828  work=363658846620870631908  tx=4403747  date=2012-06-23 06:45:56
SetBestChain: new best=00000000000002c921c7eefe8dc56c26b290104c8f23f6baf65c1679fb864ca2  height=185829  work=363666262280930727768  tx=4403875  date=2012-06-23 06:53:23
SetBestChain: new best=00000000000006006ab313edb284b005050588be5f9d80e9b04bc07ceeaa3dfc  height=185830  work=363673677940990823628  tx=4403972  date=2012-06-23 06:55:48
SetBestChain: new best=00000000000006d4c039294ce281c8bcd8d3eb9b4df48426a13fa762f07e91ad  height=185831  work=363681093601050919488  tx=4403979  date=2012-06-23 06:56:34
SetBestChain: new best=00000000000006431cde74f76be47d01068a477bec59339640941fa326d02e51  height=185832  work=363688509261111015348  tx=4404242  date=2012-06-23 07:08:42
SetBestChain: new best=0000000000000283ee30a0b6b1e05633795980da805ecb93a9c82b7d50f9c5be  height=185833  work=363695924921171111208  tx=4404263  date=2012-06-23 07:11:34
SetBestChain: new best=000000000000079884965176a8af8f417151f23ddf8a0659b17c7ce936e533b2  height=185834  work=363703340581231207068  tx=4404395  date=2012-06-23 07:44:12
SetBestChain: new best=000000000000007b2fab130730787d1215f24d52a842bf4e445a0a08852f67d4  height=185835  work=363710756241291302928  tx=4404409  date=2012-06-23 07:53:56
SetBestChain: new best=00000000000000b66f8718e946a2d0a2cf8d664f460fc6554cffa24184f5b124  height=185836  work=363718171901351398788  tx=4404421  date=2012-06-23 07:57:18
SetBestChain: new best=000000000000005376be5e75334c47fcef39f2b6aeb3554a4033b716f3b0289e  height=185837  work=363725587561411494648  tx=4404677  date=2012-06-23 07:57:09
SetBestChain: new best=00000000000007fb962bb18c2c2e8dea2ac65c6e6223576008a098fbb89734f9  height=185838  work=363733003221471590508  tx=4404829  date=2012-06-23 08:02:53
SetBestChain: new best=00000000000001ee0b8ccb36ed344235c9482bde4cfce9486df453f8063029fd  height=185839  work=363740418881531686368  tx=4404943  date=2012-06-23 08:09:40
SetBestChain: new best=0000000000000294a6fbd85c6fea3e64d735619b1d6826c10275a184a94afe9e  height=185840  work=363747834541591782228  tx=4405285  date=2012-06-23 08:16:37
SetBestChain: new best=0000000000000405eae449c3c49495402a8778cb153d629417253eb5193d6cad  height=185841  work=363755250201651878088  tx=4405301  date=2012-06-23 08:17:51
SetBestChain: new best=0000000000000043633966bac7d1c83679b53fd2e3ad0310748229ffffbb7f8f  height=185842  work=363762665861711973948  tx=4405314  date=2012-06-23 08:26:05
SetBestChain: new best=000000000000014811f1424a7c753e23de63a68befac91c6fe5a9138c338bee4  height=185843  work=363770081521772069808  tx=4405691  date=2012-06-23 08:40:42
SetBestChain: new best=00000000000006dbe4664646e0c23207dc9913e721365e17ecd223bf8c5b2411  height=185844  work=363777497181832165668  tx=4405969  date=2012-06-23 08:46:33
SetBestChain: new best=0000000000000547265576ccc0a022f9780693a14dbc4f4efa3674d79526dd4c  height=185845  work=363784912841892261528  tx=4406174  date=2012-06-23 08:48:18
SetBestChain: new best=000000000000010c8d578beed666732a56640e2d82fba05d798c7509ff402fc4  height=185846  work=363792328501952357388  tx=4406371  date=2012-06-23 08:53:47
init message: Loading addresses...
Loaded 13935 addresses from peers.dat  23ms
mapBlockIndex.size() = 185847
nBestHeight = 185846
setKeyPool.size() = 100
mapWallet.size() = 0
mapAddressBook.size() = 1
send version message: version 70001, blocks=185846, us=0.0.0.0:0, them=0.0.0.0:0, peer=127.0.0.1:0
ThreadMapPort started
ThreadSocketHandler started
ThreadMessageHandler started
ThreadIRCSeed started
ThreadIRCSeed exited
ThreadDumpAddress started
ThreadDNSAddressSeed started
Loading addresses from DNS seeds (could take a while)
ThreadOpenConnections started
init message: Done loading
ThreadOpenAddedConnections started
Flushed 13935 addresses to peers.dat  102ms

Of course I COULD just delete the whole .Bitcoin directory and start it up again, but for a variety of reasons I've already started from scratch several times, and I'm dreading doing it all again just to have it fail in the same place :-/

[KALRONG]

Hi!

Did you try running bitcoind alone? with the flag -debug to see everything that it does, just in case is not a problem with bitcoin-qt but with the actual client.

[Jaymax]

Thanks for replying.

So I installed bitcoind (via sudo apt-get install bitcoind) and ran it as suggested:

Code:

jaymax@Antec:~$ bitcoind -debug
bitcoind: table/table_builder.cc:97: void leveldb::TableBuilder::Add(const leveldb::Slice&, const leveldb::Slice&): Assertion `r->options.comparator->Compare(key, Slice(r->last_key)) > 0' failed.
Aborted (core dumped)

db.log still zero bytes.  debug.log:

Code:

[Bitcoin version v0.8.1.0-g34d62a8-beta ()
Using OpenSSL version OpenSSL 1.0.1 14 Mar 2012
Startup time: 2013-04-25 12:38:52
Default data directory /home/jaymax/.bitcoin
Used data directory /home/jaymax/.bitcoin
Using 4 threads for script verification
init message: Verifying wallet integrity...
dbenv.open LogDir=/home/jaymax/.bitcoin/database ErrorFile=/home/jaymax/.bitcoin/db.log
Bound to [::]:8333
Bound to 0.0.0.0:8333
init message: Loading block index...
Opening LevelDB in /home/jaymax/.bitcoin/blocks/index
Opened LevelDB successfully
Opening LevelDB in /home/jaymax/.bitcoin/chainstate
Opened LevelDB successfully
LoadBlockIndex(): last block file = 14
LoadBlockIndex(): last block file: CBlockFileInfo(blocks=296, size=23721895, heights=185551..185846, time=2012-06-21..2012-06-23)
LoadBlockIndex(): transaction index disabled
LoadBlockIndex(): hashBestChain=00000000000003a7b870d4f4f619e2d658081920f88d395b3d4cdb7ea58dfc96  height=185816 date=2012-06-23 04:23:02
init message: Verifying block database integrity...
Verifying last 288 blocks at level 3
No coin database inconsistencies in last 289 blocks (51010 transactions)
 block index            4005ms
init message: Loading wallet...
nFileVersion = 80100
 wallet                  676ms
init message: Importing blocks from block database...
SetBestChain: new best=000000000000009dc848a2e74d431cead8259f54aea54724a9460727a85dab48  height=185817  work=363577274360209577448  tx=4401836  date=2012-06-23 04:29:30
SetBestChain: new best=00000000000002656b7cc7df48a5fd77f8fba8ee664805afe8040ff0237ea119  height=185818  work=363584690020269673308  tx=4401849  date=2012-06-23 04:47:40
SetBestChain: new best=0000000000000286910c277b2e60be58733c06c42108f1b414ec4472a2f6d5e3  height=185819  work=363592105680329769168  tx=4402314  date=2012-06-23 04:54:46
SetBestChain: new best=000000000000080bc951a85b543ef8272f2dc4ac062143f80cd3ff27fbfab3e5  height=185820  work=363599521340389865028  tx=4402332  date=2012-06-23 04:56:11
SetBestChain: new best=00000000000001136cc9e179c6dbf99415298a5c1053fdd0211db369737ade24  height=185821  work=363606937000449960888  tx=4402344  date=2012-06-23 05:00:21
SetBestChain: new best=000000000000094baa405a2b9f68de9939124ff1ab7a84b1b9b4c38d50a9b1e8  height=185822  work=363614352660510056748  tx=4402461  date=2012-06-23 05:09:34
SetBestChain: new best=00000000000009009a48704acb7f70e0cd8a2117e0411bad0ad20074d41a576c  height=185823  work=363621768320570152608  tx=4402620  date=2012-06-23 05:16:59
SetBestChain: new best=00000000000002459afbfea5a84b19d49e14650f73ffb0a284df135c7c766b0c  height=185824  work=363629183980630248468  tx=4402682  date=2012-06-23 05:26:33
SetBestChain: new best=0000000000000943b16012b040da2d4e22e80be814f4b7833001a9570bcbecb8  height=185825  work=363636599640690344328  tx=4402899  date=2012-06-23 06:09:19
SetBestChain: new best=00000000000009ae7f1e6746e30636df17a9e68dea730b7c6dd190e12b2a228e  height=185826  work=363644015300750440188  tx=4403495  date=2012-06-23 06:21:17
SetBestChain: new best=000000000000040dfeecf7ed1685e7076cb9d4abb331a97ec95add8b21aa0629  height=185827  work=363651430960810536048  tx=4403687  date=2012-06-23 06:41:21
SetBestChain: new best=0000000000000531b00efcd4957106d08e6ff88e6e9bbed41af226a0bae6e2d0  height=185828  work=363658846620870631908  tx=4403747  date=2012-06-23 06:45:56
SetBestChain: new best=00000000000002c921c7eefe8dc56c26b290104c8f23f6baf65c1679fb864ca2  height=185829  work=363666262280930727768  tx=4403875  date=2012-06-23 06:53:23
SetBestChain: new best=00000000000006006ab313edb284b005050588be5f9d80e9b04bc07ceeaa3dfc  height=185830  work=363673677940990823628  tx=4403972  date=2012-06-23 06:55:48
SetBestChain: new best=00000000000006d4c039294ce281c8bcd8d3eb9b4df48426a13fa762f07e91ad  height=185831  work=363681093601050919488  tx=4403979  date=2012-06-23 06:56:34
SetBestChain: new best=00000000000006431cde74f76be47d01068a477bec59339640941fa326d02e51  height=185832  work=363688509261111015348  tx=4404242  date=2012-06-23 07:08:42
SetBestChain: new best=0000000000000283ee30a0b6b1e05633795980da805ecb93a9c82b7d50f9c5be  height=185833  work=363695924921171111208  tx=4404263  date=2012-06-23 07:11:34
SetBestChain: new best=000000000000079884965176a8af8f417151f23ddf8a0659b17c7ce936e533b2  height=185834  work=363703340581231207068  tx=4404395  date=2012-06-23 07:44:12
SetBestChain: new best=000000000000007b2fab130730787d1215f24d52a842bf4e445a0a08852f67d4  height=185835  work=363710756241291302928  tx=4404409  date=2012-06-23 07:53:56
SetBestChain: new best=00000000000000b66f8718e946a2d0a2cf8d664f460fc6554cffa24184f5b124  height=185836  work=363718171901351398788  tx=4404421  date=2012-06-23 07:57:18
SetBestChain: new best=000000000000005376be5e75334c47fcef39f2b6aeb3554a4033b716f3b0289e  height=185837  work=363725587561411494648  tx=4404677  date=2012-06-23 07:57:09
SetBestChain: new best=00000000000007fb962bb18c2c2e8dea2ac65c6e6223576008a098fbb89734f9  height=185838  work=363733003221471590508  tx=4404829  date=2012-06-23 08:02:53
SetBestChain: new best=00000000000001ee0b8ccb36ed344235c9482bde4cfce9486df453f8063029fd  height=185839  work=363740418881531686368  tx=4404943  date=2012-06-23 08:09:40

should I be perturbed by that 'beta' on the top line?

A second run gave me:

Code:

Bitcoin version v0.8.1.0-g34d62a8-beta ()
Using OpenSSL version OpenSSL 1.0.1 14 Mar 2012
Startup time: 2013-04-25 12:45:04
Default data directory /home/jaymax/.bitcoin
Used data directory /home/jaymax/.bitcoin
Using 4 threads for script verification
init message: Verifying wallet integrity...
dbenv.open LogDir=/home/jaymax/.bitcoin/database ErrorFile=/home/jaymax/.bitcoin/db.log
Bound to [::]:8333
Bound to 0.0.0.0:8333
init message: Loading block index...
Opening LevelDB in /home/jaymax/.bitcoin/blocks/index
Opened LevelDB successfully
Opening LevelDB in /home/jaymax/.bitcoin/chainstate
Opened LevelDB successfully
LoadBlockIndex(): last block file = 14
LoadBlockIndex(): last block file: CBlockFileInfo(blocks=296, size=23721895, heights=185551..185846, time=2012-06-21..2012-06-23)
LoadBlockIndex(): transaction index disabled
LoadBlockIndex(): hashBestChain=00000000000003a7b870d4f4f619e2d658081920f88d395b3d4cdb7ea58dfc96  height=185816 date=2012-06-23 04:23:02
init message: Verifying block database integrity...
Verifying last 288 blocks at level 3
No coin database inconsistencies in last 289 blocks (51010 transactions)
 block index            3979ms
init message: Loading wallet...
nFileVersion = 80100
 wallet                  721ms
init message: Importing blocks from block database...
SetBestChain: new best=000000000000009dc848a2e74d431cead8259f54aea54724a9460727a85dab48  height=185817  work=363577274360209577448  tx=4401836  date=2012-06-23 04:29:30
SetBestChain: new best=00000000000002656b7cc7df48a5fd77f8fba8ee664805afe8040ff0237ea119  height=185818  work=363584690020269673308  tx=4401849  date=2012-06-23 04:47:40
SetBestChain: new best=0000000000000286910c277b2e60be58733c06c42108f1b414ec4472a2f6d5e3  height=185819  work=363592105680329769168  tx=4402314  date=2012-06-23 04:54:46
SetBestChain: new best=000000000000080bc951a85b543ef8272f2dc4ac062143f80cd3ff27fbfab3e5  height=185820  work=363599521340389865028  tx=4402332  date=2012-06-23 04:56:11
SetBestChain: new best=00000000000001136cc9e179c6dbf99415298a5c1053fdd0211db369737ade24  height=185821  work=363606937000449960888  tx=4402344  date=2012-06-23 05:00:21
SetBestChain: new best=000000000000094baa405a2b9f68de9939124ff1ab7a84b1b9b4c38d50a9b1e8  height=185822  work=363614352660510056748  tx=4402461  date=2012-06-23 05:09:34
SetBestChain: new best=00000000000009009a48704acb7f70e0cd8a2117e0411bad0ad20074d41a576c  height=185823  work=363621768320570152608  tx=4402620  date=2012-06-23 05:16:59
SetBestChain: new best=00000000000002459afbfea5a84b19d49e14650f73ffb0a284df135c7c766b0c  height=185824  work=363629183980630248468  tx=4402682  date=2012-06-23 05:26:33
SetBestChain: new best=0000000000000943b16012b040da2d4e22e80be814f4b7833001a9570bcbecb8  height=185825  work=363636599640690344328  tx=4402899  date=2012-06-23 06:09:19
SetBestChain: new best=00000000000009ae7f1e6746e30636df17a9e68dea730b7c6dd190e12b2a228e  height=185826  work=363644015300750440188  tx=4403495  date=2012-06-23 06:21:17
SetBestChain: new best=000000000000040dfeecf7ed1685e7076cb9d4abb331a97ec95add8b21aa0629  height=185827  work=363651430960810536048  tx=4403687  date=2012-06-23 06:41:21
SetBestChain: new best=0000000000000531b00efcd4957106d08e6ff88e6e9bbed41af226a0bae6e2d0  height=185828  work=363658846620870631908  tx=4403747  date=2012-06-23 06:45:56
SetBestChain: new best=00000000000002c921c7eefe8dc56c26b290104c8f23f6baf65c1679fb864ca2  height=185829  work=363666262280930727768  tx=4403875  date=2012-06-23 06:53:23
SetBestChain: new best=00000000000006006ab313edb284b005050588be5f9d80e9b04bc07ceeaa3dfc  height=185830  work=363673677940990823628  tx=4403972  date=2012-06-23 06:55:48
SetBestChain: new best=00000000000006d4c039294ce281c8bcd8d3eb9b4df48426a13fa762f07e91ad  height=185831  work=363681093601050919488  tx=4403979  date=2012-06-23 06:56:34
SetBestChain: new best=00000000000006431cde74f76be47d01068a477bec59339640941fa326d02e51  height=185832  work=363688509261111015348  tx=4404242  date=2012-06-23 07:08:42
SetBestChain: new best=0000000000000283ee30a0b6b1e05633795980da805ecb93a9c82b7d50f9c5be  height=185833  work=363695924921171111208  tx=4404263  date=2012-06-23 07:11:34
SetBestChain: new best=000000000000079884965176a8af8f417151f23ddf8a0659b17c7ce936e533b2  height=185834  work=363703340581231207068  tx=4404395  date=2012-06-23 07:44:12
SetBestChain: new best=000000000000007b2fab130730787d1215f24d52a842bf4e445a0a08852f67d4  height=185835  work=363710756241291302928  tx=4404409  date=2012-06-23 07:53:56
SetBestChain: new best=00000000000000b66f8718e946a2d0a2cf8d664f460fc6554cffa24184f5b124  height=185836  work=363718171901351398788  tx=4404421  date=2012-06-23 07:57:18
SetBestChain: new best=000000000000005376be5e75334c47fcef39f2b6aeb3554a4033b716f3b0289e  height=185837  work=363725587561411494648  tx=4404677  date=2012-06-23 07:57:09
SetBestChain: new best=00000000000007fb962bb18c2c2e8dea2ac65c6e6223576008a098fbb89734f9  height=185838  work=363733003221471590508  tx=4404829  date=2012-06-23 08:02:53
SetBestChain: new best=00000000000001ee0b8ccb36ed344235c9482bde4cfce9486df453f8063029fd  height=185839  work=363740418881531686368  tx=4404943  date=2012-06-23 08:09:40
SetBestChain: new best=0000000000000294a6fbd85c6fea3e64d735619b1d6826c10275a184a94afe9e  height=185840  work=363747834541591782228  tx=4405285  date=2012-06-23 08:16:37
SetBestChain: new best=0000000000000405eae449c3c49495402a8778cb153d629417253eb5193d6cad  height=185841  work=363755250201651878088  tx=4405301  date=2012-06-23 08:17:51
SetBestChain: new best=0000000000000043633966bac7d1c83679b53fd2e3ad0310748229ffffbb7f8f  height=185842  work=363762665861711973948  tx=4405314  date=2012-06-23 08:26:05
SetBestChain: new best=000000000000014811f1424a7c753e23de63a68befac91c6fe5a9138c338bee4  height=185843  work=363770081521772069808  tx=4405691  date=2012-06-23 08:40:42
SetBestChain: new best=00000000000006dbe4664646e0c23207dc9913e721365e17ecd223bf8c5b2411  height=185844  work=363777497181832165668  tx=4405969  date=2012-06-23 08:46:33
SetBestChain: new best=0000000000000547265576ccc0a022f9780693a14dbc4f4efa3674d79526dd4c  height=185845  work=363784912841892261528  tx=4406174  date=2012-06-23 08:48:18
SetBestChain: new best=000000000000010c8d578beed666732a56640e2d82fba05d798c7509ff402fc4  height=185846  work=363792328501952357388  tx=4406371  date=2012-06-23 08:53:47
init message: Loading addresses...
Loaded 13935 addresses from peers.dat  146ms
mapBlockIndex.size() = 185847
nBestHeight = 185846
setKeyPool.size() = 100
mapWallet.size() = 0
mapAddressBook.size() = 1
init message: Done loading
ThreadRPCServer started
send version message: version 70001, blocks=185846, us=0.0.0.0:0, them=0.0.0.0:0, peer=127.0.0.1:0
sending: version (100 bytes)
ThreadDNSAddressSeed started
Loading addresses from DNS seeds (could take a while)
ThreadIRCSeed started
ThreadIRCSeed exited
ThreadSocketHandler started
ThreadOpenConnections started
ThreadOpenAddedConnections started
ThreadMessageHandler started
ThreadDumpAddress started
GetMyExternalIP() received [46.64.72.232] 46.64.72.232:0
GetMyExternalIP() returned 46.64.72.232
AddLocal(46.64.72.232:8333,5)

And a third run had three additional lines at the end - very odd.

Is there anywhere else I should be looking? 

[KALRONG]

Dont worry about the BETA label hehe all crypto-currencies are still in beta


Looks like in the second run it started fine just fine and in the second it crashed while loading the blocks.

What do you have in your bitcoin.conf? I always recommend to install the bitcoin client from the sources but Im also using the bitcoind from debian repositories and didn't have any problems

If you run bitcoind with -daemon (to run it in the background) or in one terminal and in another one doing a bitcoind getinfo do you get something?

In any case Im betting on the chance that the database of blocks has been corrupted for some reason, but lets check the rest first

[Jaymax]

Hey, thanks for your ongoing help...

for some reason

Ran a RAM test overnight, I think I know what the problem is now   :-)   :-(

Kinda explains a few other minor oddball things I shoulda paid more heed to at the time. 

[KALRONG]

Glad to hear that you fixed it hope its fixed after changing the faulty RAM

[cjp]

I'm having the same error right now, with Bitcoin-qt 0.9.3. After downloading and processing several weeks of new blocks, it crashed, and now it doesn't start anymore:

Code:

$ bitcoin-qt -wallet=empty.dat -debug
bitcoin-qt: table/table_builder.cc:97: void leveldb::TableBuilder::Add(const leveldb::Slice&, const leveldb::Slice&): Controletest 'r->options.comparator->Compare(key, Slice(r->last_key)) > 0' faalt.
Afgebroken
$ 

debug.log:

Code:

2015-06-20 11:27:40 Bitcoin version v0.9.3.0-b146f97-dirty-beta (2015-01-11 22:25:06 +0100)
2015-06-20 11:27:40 Using OpenSSL version OpenSSL 1.0.1e 11 Feb 2013
2015-06-20 11:27:40 Using BerkeleyDB version Berkeley DB 5.1.29: (October 25, 2011)
2015-06-20 11:27:40 Default data directory /home/cjp/.bitcoin
2015-06-20 11:27:40 Using data directory /home/cjp/.bitcoin
2015-06-20 11:27:40 Using at most 125 connections (1024 file descriptors available)
2015-06-20 11:27:40 Using 3 threads for script verification
2015-06-20 11:27:40 Using wallet empty.dat
2015-06-20 11:27:40 init message: Portemonnee aan het controleren...
2015-06-20 11:27:40 CDBEnv::Open : LogDir=/home/cjp/.bitcoin/database ErrorFile=/home/cjp/.bitcoin/db.log
2015-06-20 11:27:40 Bound to [::]:8333
2015-06-20 11:27:40 Bound to 0.0.0.0:8333
2015-06-20 11:27:40 init message: Blokindex aan het laden...
2015-06-20 11:27:40 Opening LevelDB in /home/cjp/.bitcoin/blocks/index
2015-06-20 11:27:40 Opened LevelDB successfully
2015-06-20 11:27:40 Opening LevelDB in /home/cjp/.bitcoin/chainstate
2015-06-20 11:27:41 Opened LevelDB successfully

I don't think I have bad RAM, but  guess it doesn't hurt to do a RAM test. Anyway, the error now happens consistently a few seconds after start-up of bitcoin-qt, and I think it's unlikely it would hit the same RAM error each time in the same way. So, even if it was initially caused by a RAM error, right now it seems something is wrong with the state of the (block chain?) data on my disk, which is blocking me from re-starting bitcoin-qt.

Can I draw the conclusion, based on the log lines "Opened LevelDB successfully", that the error is NOT in .bitcoin/blocks/index and NOT in .bitcoin/chainstate? Based on the source file that reported the error, I'd say it has to be in something LevelDB-related. What does Bitcoin do AFTER successfully opening .bitcoin/chainstate?

I'll see if I can make a core dump and examine the function call stack. In the mean time, please reply if you know something that could help. My objective right now is to try to repair the files on my disk (e.g. removing the last ones that might be corrupt), in the hope that I can recover from this situation without re-processing the entire block chain again.

[cjp]

Code:

(gdb) backtrace
#0  0x00007f73a77d5165 in *__GI_raise (sig=<optimized out>)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007f73a77d83e0 in *__GI_abort () at abort.c:92
#2  0x00007f73a77ce311 in *__GI___assert_fail (
    assertion=0x7f73abb88fa8 "r->options.comparator->Compare(key, Slice(r->last_key)) > 0", file=<optimized out>, line=97, 
    function=0x7f73abb89040 "void leveldb::TableBuilder::Add(const leveldb::Slice&, const leveldb::Slice&)") at assert.c:81
#3  0x00007f73ab74ad96 in leveldb::TableBuilder::Add (this=0x7f73acf98200, 
    key=..., value=...) at table/table_builder.cc:97
#4  0x00007f73ab732fca in leveldb::DBImpl::DoCompactionWork (
    this=this@entry=0x7f738091fd10, compact=compact@entry=0x7f73acc3e110)
    at db/db_impl.cc:963
#5  0x00007f73ab7337c3 in leveldb::DBImpl::BackgroundCompaction (
    this=this@entry=0x7f738091fd10) at db/db_impl.cc:706
#6  0x00007f73ab734222 in leveldb::DBImpl::BackgroundCall (this=0x7f738091fd10)
    at db/db_impl.cc:644
#7  0x00007f73ab75078f in BGThread (this=<optimized out>)
    at util/env_posix.cc:571
#8  leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (
    arg=0x7f7380923e80) at util/env_posix.cc:510
#9  0x00007f73a8edab50 in start_thread (arg=<optimized out>)
    at pthread_create.c:304
---Type <return> to continue, or q <return> to quit---
#10 0x00007f73a787e95d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#11 0x0000000000000000 in ?? ()
(gdb)

OK, not much help from the core dump here. Apparently, the failure happens in a background thread that originates from the levelDB code, not from the Bitcoin core code. So it could be related to any already-loaded LevelDB database.

Now, while I'll do some further debugging, does someone know where to find the database lay-out used in Bitcoin's LevelDB databases? I'd like to know how I can recognize inconsistent states, and recover Bitcoin to a consistent state.


EDIT:
Apparently, .bitcoin/blocks/index/LOG gets rewritten after each attempt to start bitcoin-qt. The most recent contents right now is:

Code:

2015/06/20-13:51:31.656165 7f7399b09700 Recovering log #53983
2015/06/20-13:51:31.895293 7f7399b09700 Delete type=2 #53987
2015/06/20-13:51:31.895893 7f7399b09700 Delete type=2 #53989
2015/06/20-13:51:31.896966 7f7399b09700 Delete type=2 #53991
2015/06/20-13:51:31.897791 7f7399b09700 Delete type=2 #53984
2015/06/20-13:51:31.898344 7f7399b09700 Delete type=2 #53992
2015/06/20-13:51:31.898468 7f7399b09700 Delete type=2 #53985
2015/06/20-13:51:31.898998 7f7399b09700 Delete type=2 #53990
2015/06/20-13:51:31.899528 7f7399b09700 Delete type=2 #53986
2015/06/20-13:51:31.900043 7f7399b09700 Delete type=0 #53983
2015/06/20-13:51:31.900074 7f7399b09700 Delete type=3 #53982
2015/06/20-13:51:31.900224 7f7399b09700 Delete type=2 #53988
2015/06/20-13:51:31.901200 7f738654b700 Compacting 5@0 + 6@1 files
2015/06/20-13:51:32.113270 7f738654b700 Generated table #53986: 37886 keys, 2013418 bytes
2015/06/20-13:51:32.324513 7f738654b700 Generated table #53987: 41465 keys, 2169643 bytes
2015/06/20-13:51:32.534105 7f738654b700 Generated table #53988: 41487 keys, 2171790 bytes
2015/06/20-13:51:32.633776 7f738654b700 Generated table #53989: 8389 keys, 439433 bytes
2015/06/20-13:51:32.832583 7f738654b700 Generated table #53990: 41405 keys, 2168859 bytes
2015/06/20-13:51:33.042172 7f738654b700 Generated table #53991: 33492 keys, 1756388 bytes
2015/06/20-13:51:33.251728 7f738654b700 Generated table #53992: 41395 keys, 2172333 bytes
2015/06/20-13:51:33.439531 7f738654b700 Generated table #53993: 31681 keys, 1662583 bytes

Interestingly, there is a 053994.ldb in the directory. So, my current guess is the error is in that file.

EDIT 2: apparently, every time I re-start bitcoin-qt, it creates a couple of new .ldb files in that directory. I don't know yet whether that's a good or a bad thing though.

[cjp]

OK, eventually I ended up removing the blocks and chainstate directories, and recovering from a backup I made in April. Re-syncing from there took a while, but it worked. Apparently, the issue was a rare kind of glitch on my system; I'll be making more frequent block chain back-ups in the future.