Bitcoin-Qt / bitcoind version 0.8.0 release candidate 1

[btcven]

I only connect to 8 nodes, even restarting the client.

Have been hours running, still downloading the blockchain (really slow internet connection).

Problems: closing the app window is not enough, must go to Dock and Quit the app.

MacBook Air, 128GB SSD, 10.8.2

[1714163937]

1714163937

[1714163937]

1714163937

[1714163937]

1714163937

[1714163937]

1714163937

[1714163937]

1714163937

[1714163937]

1714163937

[neuronics]

Everything working like expected on Win7 32 and XP 64 machines / v0.8.0rc1-beta / Armory 0.87 beta, reindexing blocks ~75 minutes.  Very impressing !  Good job, thx alot.

[Ente]

The whole pruning and bloom feature is a bit unclear to me.
So I basically send tx to other nodes when they fit their bloom filter?
But I still have the whole blockchain locally stored?

Where my question aims at:
I run a bitcoind on my desktopmachine, connecting to it via Armory when needed. I wish to have a "full" node, saving and serving every tx in the blockchain.
Also, I wish to connect to that bitcoind via Android Schildbach Client from time to time, for fast catch-up.
And, you know, the network would not work if *every* node was a light client, right?

There must be a place where these kind of questions are already answered, sorry for asking here.

Can't wait to upgrade once I get home!

Ente

[caveden]

The whole pruning and bloom feature is a bit unclear to me.
So I basically send tx to other nodes when they fit their bloom filter?
But I still have the whole blockchain locally stored?

These are two unrelated things. Bloom filter is pretty much what you describe: a protocol for full nodes to serve lightweight nodes only the transactions that concern them, plus some false positives for privacy reasons.

The index pruning changed the main transaction index in bitcoind, now it only stores unspent transactions, what makes it much smaller, and thus synchronizing with the network now requires much less I/O. Keep in mind this is just the index, the full blockchain content is still stored.

[deepceleron]

hi, i did some unethical "stress testing". Deleted all the data, started to re-download everything. after some time, i repeatedly killed it via pkill -9 bitcoin. well, that worked fine.
...
PPS: and after deleting everything, I also had a corrupted wallet. (nothing lost, don't worry, but just let you know that for some reason, it got corrupted during that  treatment! likely (?) due to the forced shutdown while it was still open). … and yes, it was fine before, and it's size was around 100k before and afterwards.

Corrupting a wallet is almost expected eventually if you continue to kill Bitcoin while downloading blocks. The last-seen block hash is written to the wallet every block, which means the wallet is sometimes updated dozens of times a second while downloading blocks that fast.

Did Bitcoin attempt to automatically salvage the wallet, and was this successful? I noted and filed an issue where you continue to get warning messages every launch after a wallet salvage was performed.

[Rygon]

hi, i did some unethical "stress testing". Deleted all the data, started to re-download everything. after some time, i repeatedly killed it via pkill -9 bitcoin. well, that worked fine.
...
PPS: and after deleting everything, I also had a corrupted wallet. (nothing lost, don't worry, but just let you know that for some reason, it got corrupted during that  treatment! likely (?) due to the forced shutdown while it was still open). … and yes, it was fine before, and it's size was around 100k before and afterwards.

Corrupting a wallet is almost expected eventually if you continue to kill Bitcoin while downloading blocks. The last-seen block hash is written to the wallet every block, which means the wallet is sometimes updated dozens of times a second while downloading blocks that fast.

Did Bitcoin attempt to automatically salvage the wallet, and was this successful? I noted and filed an issue where you continue to get warning messages every launch after a wallet salvage was performed.

Is there a way to safely shut down bitcoin-qt while downloading blocks without the risk of corrupting a wallet? Sometimes I just have to shut down the computer while waiting hours to download blocks, and the only way to stop Bitcon is to close the window in Ubuntu. Is that still safe?

Also, I've had a corrupted wallet file that Bitcoin was unable to salvage. No loss, but it was concerning.

[Ente]

The whole pruning and bloom feature is a bit unclear to me.
So I basically send tx to other nodes when they fit their bloom filter?
But I still have the whole blockchain locally stored?

These are two unrelated things. Bloom filter is pretty much what you describe: a protocol for full nodes to serve lightweight nodes only the transactions that concern them, plus some false positives for privacy reasons.

The index pruning changed the main transaction index in bitcoind, now it only stores unspent transactions, what makes it much smaller, and thus synchronizing with the network now requires much less I/O. Keep in mind this is just the index, the full blockchain content is still stored.

Thank you for clarifying.
So with pruning, the whole block[chain] is still downloaded and served. Only a part of it is then used for the "user-side" of bitcoin, where it cares about what your addresses are and that?
Sounds great!

Ente

[Mike Hearn]

I run a bitcoind on my desktopmachine, connecting to it via Armory when needed. I wish to have a "full" node, saving and serving every tx in the blockchain.
Also, I wish to connect to that bitcoind via Android Schildbach Client from time to time, for fast catch-up.
And, you know, the network would not work if *every* node was a light client, right?

Your Android client will use Bloom filtering to reduce its bandwidth usage and speed things up (if you use the latest versions which are not released yet).

[wtfvanity]

Error upgrading on Ubuntu 11.04 server.

************************
EXCEPTION: St9bad_alloc
std::bad_alloc
bitcoin in ProcessMessages()



************************
EXCEPTION: St9bad_alloc
std::bad_alloc
bitcoin in ThreadMessageHandler()

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted

It had the full block chain, was upgrading, not sure where it was in the blocks > 170k I know that for sure...

[Pieter Wuille]

So with pruning, the whole block[chain] is still downloaded and served. Only a part of it is then used for the "user-side" of bitcoin, where it cares about what your addresses are and that?

Not really. None of this has anything to do with the wallet implementation. Wallets always track all transactions relevant to the user. This has not (or hardly) changed between 0.7 and 0.8.

What has changed, is how the block and transaction validation works. Previously, we stored

• the full blocks (blk000?.dat)
• the (byte) position of every block and transaction in it (blkindex.dat)
• for every transaction output, whether and where is was spent (also blkindex.dat)

This required an ever-growing index, and fast access to the full (ever growing) block data. This was slow.

The new system stores:

• the full blocks (blocks/blk000??.dat)
• the (byte) position of every block in it (but not every transaction!) (blocks/index/*)
• a separate database with a copy of the unspent transaction outputs (so not an index with byte positions for into the block chain, just a copy of not even the full transactions, but only the part that may be relevant in the future) (chainstate/*)
• an undo log for the chain state, so we can go back in time for reorganisations (blocks/rev000??.dat).

The big advantage is that we now only need fast access to the chain state (around 150 MB), instead of to the full blocks and the full index (several GB).

Although I initially called this new database/validation system "ultraprune". This is a very confusing name, as there is no actual pruning going on: we still keep all blocks/transactions around. The block data is still necessary for rescanning, reorganising, serving to other nodes, and the getrawtransaction RPC call. The code that resulted in the new database system actually started as an experiment about how small the chain state (aka UTXO set) could be represented, by pruning it as hard as possible. This is where the name comes from.

[kgo]

Upgraded a 64-bit system that was previously running next-test on the latest Ubuntu LTS.  Instructions provided about deleting files worked just fine.

Upgraded a 32-bit system running 0.7.3 on Debian Squeeze.  The conversion of the old blocks seemed to slow down near the end, but left things running overnight and everything worked.

Sent one bitcoin back and forth between systems without incident.

[dooglus]

I built the v0.8.0rc1 tag from git on ubuntu 12.10 64 bit and it works fine so far.

I missed this bit:

If you helped test pre-release versions, there are two changes that you
should be aware of:

and so it started downloading/importing (not sure which) the whole blockchain from scratch.  So I quit, ran the mkdir/mv commands from the OP, and restarted but it was too late.  The debug.log showed all the blocks past the first 500 as being orphaned:

SetBestChain: new best=00000000806df68baab17e49e567d4211177fef4849ffd8242d095c6a1169f45  height=499  work=2147516416500  tx=508  date=2009-01-14 21:14:40
ProcessBlock: ACCEPTED
SetBestChain: new best=000000004ff664bfa7d217f6df64c1627089061429408e1da5ef903b8f3c77db  height=500  work=2151811449333  tx=509  date=2009-01-14 21:27:29
ProcessBlock: ACCEPTED
ProcessBlock: ORPHAN BLOCK, prev=00000000046739c1ea3612322e1769f07783ebb22fb62498b7fd2ff249a52f29
ProcessBlock: ORPHAN BLOCK, prev=000000000bbb0dde89c4ccd3d5faced4cedb506bd8a74a74db09558e7df55959

and then, after complaining about 220k orphaned blocks started downloading from block 501 from peers:

received block 0000000046887292a76cd113a5fd6af38b17c9fb77e5936cd9856694030598f9
SetBestChain: new best=0000000046887292a76cd113a5fd6af38b17c9fb77e5936cd9856694030598f9  height=501  work=2156106482166  tx=510  date=2009-01-14 21:38:31
ProcessBlock: ACCEPTED
received block 000000003727134a4823b6c4c5245e961fff7816fc494742fb0abd353c94757b
SetBestChain: new best=000000003727134a4823b6c4c5245e961fff7816fc494742fb0abd353c94757b  height=502  work=2160401514999  tx=511  date=2009-01-14 21:46:15
ProcessBlock: ACCEPTED

So I quit again, deleted the blocks and chainstate folders and re-ran with the -reindex flag and that worked OK.

[ghostshirt]

I installed the new release on a relatively older Mac yesterday, It took about 8-9 hours of re-indexing blocks. It's been running smoothly for more than 24 hours. I tested a few transactions. Nothing unexpected to report.

[wtfvanity]

I went ahead and upgraded my public node to the .8 RC and in the past 36 hours, I have noticed that I have had an increase of almost 3x in the amount of bandwidth being used. it went from 2-3 GB outgoing per day to almost 10 GB outgoing per day. Is that because of the bloom filtering? Or just more people upgrading and testing out the RC?

[zvs]

I went ahead and upgraded my public node to the .8 RC and in the past 36 hours, I have noticed that I have had an increase of almost 3x in the amount of bandwidth being used. it went from 2-3 GB outgoing per day to almost 10 GB outgoing per day. Is that because of the bloom filtering? Or just more people upgrading and testing out the RC?

it seems to send blocks out to more people.  

when i look at my logs, i receive the same block a half dozen times usually.

using dstat, it will burst up to 100MB upstream sometimes, i guess from myself sending blocks to someone that has already received


like this:

2013-02-12 16:44:49 received block 00000000000004f958de0219d509e8ba156d9893f4903f9fa6c23edfed5ef565
2013-02-12 16:44:55 received block 00000000000004f958de0219d509e8ba156d9893f4903f9fa6c23edfed5ef565
2013-02-12 16:44:57 received block 00000000000004f958de0219d509e8ba156d9893f4903f9fa6c23edfed5ef565
2013-02-12 16:44:59 received block 00000000000004f958de0219d509e8ba156d9893f4903f9fa6c23edfed5ef565
2013-02-12 16:45:12 received block 00000000000004f958de0219d509e8ba156d9893f4903f9fa6c23edfed5ef565
2013-02-12 16:45:29 received block 00000000000000eb3c170ca260e1dffa2b1a91772dcae18a4df3446f377bd1b4
2013-02-12 16:46:09 received block 000000000000040681ba05ed54daa23667b2673d3ef6acbabd2dc36c95d66ab7
2013-02-12 16:48:44 received block 00000000000001f94f1644dfd3d056e5098d325b4b2bf4cb3c2690d7006c1a87
2013-02-12 17:01:41 received block 0000000000000379de774e67d9bdb305af68a956f891c548f0da19c538a79071
2013-02-12 17:01:44 received block 0000000000000379de774e67d9bdb305af68a956f891c548f0da19c538a79071
2013-02-12 17:01:47 received block 0000000000000379de774e67d9bdb305af68a956f891c548f0da19c538a79071
2013-02-12 17:01:56 received block 0000000000000379de774e67d9bdb305af68a956f891c548f0da19c538a79071
2013-02-12 17:02:08 received block 0000000000000379de774e67d9bdb305af68a956f891c548f0da19c538a79071
2013-02-12 17:10:07 received block 00000000000001cfaf40b52daf78d4cf9c0e93e427c242b90386ea6aae9d079b
2013-02-12 17:10:08 received block 00000000000001cfaf40b52daf78d4cf9c0e93e427c242b90386ea6aae9d079b
2013-02-12 17:10:11 received block 00000000000001cfaf40b52daf78d4cf9c0e93e427c242b90386ea6aae9d079b
2013-02-12 17:10:16 received block 00000000000001cfaf40b52daf78d4cf9c0e93e427c242b90386ea6aae9d079b
2013-02-12 17:10:31 received block 00000000000001cfaf40b52daf78d4cf9c0e93e427c242b90386ea6aae9d079b
2013-02-12 17:23:20 received block 0000000000000437a3ec102cf9050b7a26c0d986acf74cf9b3d6eff8ad7959de
2013-02-12 17:42:09 received block 00000000000004af3c30dac15e153941696c078a8e5c968bfe364de9a858e7c0
2013-02-12 17:57:25 received block 00000000000000576ea85bd5de07bdbc81a00549f6d20f8549b04a7597f5d64a
2013-02-12 17:57:26 received block 00000000000000576ea85bd5de07bdbc81a00549f6d20f8549b04a7597f5d64a
2013-02-12 17:57:33 received block 00000000000000576ea85bd5de07bdbc81a00549f6d20f8549b04a7597f5d64a
2013-02-12 17:57:39 received block 00000000000000576ea85bd5de07bdbc81a00549f6d20f8549b04a7597f5d64a
2013-02-12 17:58:02 received block 00000000000000576ea85bd5de07bdbc81a00549f6d20f8549b04a7597f5d64a
2013-02-12 18:00:04 received block 00000000000002bc74b17eb31d45dcac53ec1ba082c0bc7113e73ec32bae952f
2013-02-12 18:09:16 received block 000000000000015bd9e602c027f1eb2367ec7a8c8d4d70baa44cb27e86c6af81
2013-02-12 18:09:21 received block 000000000000015bd9e602c027f1eb2367ec7a8c8d4d70baa44cb27e86c6af81
2013-02-12 18:09:22 received block 000000000000015bd9e602c027f1eb2367ec7a8c8d4d70baa44cb27e86c6af81
2013-02-12 18:09:23 received block 000000000000015bd9e602c027f1eb2367ec7a8c8d4d70baa44cb27e86c6af81
2013-02-12 18:09:43 received block 000000000000015bd9e602c027f1eb2367ec7a8c8d4d70baa44cb27e86c6af81

and here is an example of network activity when a new block is found:

-net/total-
 recv  send
 104k  497k
 230k  708k
 143k  594k
  10k  271k
  10k  312k
7144B  223k
 204k  849k
  36k  470k
  98k  732k
  36k  364k
 115k  575k
 179k  785k
  73k  362k
 129k  570k
  15k  288k
 231k 1108k
 316k 1266k
  71k  435k
 116k  456k
 107k  564k
 149k 1058k
  24k  328k
8382B  202k
 129k  444k
 127k  539k
  15k  410k
  13k  459k
 262k  843k
 104k  451k
 286k  977k
 144k  724k
  21k  264k
  20k  385k
 164k  561k
  86k  524k
 277k 1068k
 229k  791k
 146k  522k
  15k  326k
 117k  439k
  22k  188k
 137k  531k
  45k  320k
 107k  575k
 222k  648k
  50k  283k
8960B  296k
  39k  376k
 725k   49M
 334k   68M
 176k   15M
 236k 3906k
 104k  671k
 151k 1093k
 131k  712k
  22k  549k
  26k  363k
 339k  989k
  47k  430k
 143k  603k
 131k  548k
  41k  281k
  29k  254k
 148k  709k
 182k  835k
 171k  620k
 130k  560k
 107k  529k
 124k  494k
  12k  226k
9887B  239k
6773B  177k
  96k  461k
 142k  529k
  12k  259k
 132k  527k
  10k  260k
 160k  546k
  44k 1148k
 114k  763k
  31k  381k
 147k  699k


this is with ~600 connections

[mb300sd]

I'm noticing the same issue with more bandwidth usage. Used to handle >200 connections fine but now its slowing down my connection.  Thought it was ISP issues until I checked.

[dooglus]

I just started the v0.8.0rc1-beta that I built myself from git and saw this splash screen:



Looks like the font is too big, or the message too long...

[LightRider]

Wow, I just noticed that I'm sending 1 mbit upstream as well. It would be nice if these issues were addressed. Thanks again for all the work so far though.

[dserrano5]

• Installed 0.7.2 on a WinXP box (vmware virtual machine).
• Downloaded some blocks, up to 135k or so.
• Cleanly shut down bitcoin.
• Installed 0.8.0rc1.
• Bitcoin sucessfully reindexed the existing blocks, then started to download from that point.
• Suspended the virtual machine.
• Downloaded some more blocks from home, then suspended the VM again.
• (related?) Installed Ares p2p software. Version 2.2.4 FWIW. (this was yesterday at home)
• Today at work, downloaded more blocks.
• While clicking around on Ares, I got a couple of dialog boxes about Visual C++ runtime <something> and mentioning bitcoin. However, I was able to click on the bitcoin window and bring it to top, giving it focus. I wasn't able to do that with Ares though (when I tried, one of the dialog boxes' title flashed, indicating me that I had to pay attention to it before using Ares again). So I happily disregarded the dialogs, only to see the bitcoin window disappear and the Ares one become unresponsive.

Code:

received block 00000000000000ad0e1ec1f69424abd586077ef7a7b953ecec8ebafa30a61ee2
ProcessBlock: ORPHAN BLOCK, prev=00000000000006c74bc0eaf942f5c93b53afd19c4b7d4b0ed35dcfe549ea824f
received block 00000000000006a58a50091c4feb7d8af30cb316ba6ecae8472efb29f125d9e5
ProcessBlock: ORPHAN BLOCK, prev=00000000000000ad0e1ec1f69424abd586077ef7a7b953ecec8ebafa30a61ee2
received block 000000000000051cfd88579acf5d0a5d20bd0232dc7ca174dd057e35ecf15187
ProcessBlock: ORPHAN BLOCK, prev=00000000000006a58a50091c4feb7d8af30cb316ba6ecae8472efb29f125d9e5
received block 000000000000006e3986d9f8510fc62cf8370678ca501ba24c228184cd268bef
ProcessBlock: ORPHAN BLOCK, prev=000000000000051cfd88579acf5d0a5d20bd0232dc7ca174dd057e35ecf15187


************************
EXCEPTION: St9bad_alloc       
std::bad_alloc       
C:\Archivos de programa\Bitcoin\bitcoin-qt.exe in ThreadMessageHandler()       



************************
EXCEPTION: St9bad_alloc       
std::bad_alloc       
C:\Archivos de programa\Bitcoin\bitcoin-qt.exe in ThreadSocketHandler()       

Flushed 12767 addresses to peers.dat  78ms

Sorry no timestamping (it should be on by default IMHO). Searching for "EXCEPTION" in the debug.log I find two additional bad_alloc messages. One of them appears 400 lines above this, so most probably it happened earlier today. The other one could be from yesterday, can't tell for sure. They could be related to the VM suspends.

[dansmith]

Upgraded a 32-bit system running 0.7.3 on Debian Squeeze.  The conversion of the old blocks seemed to slow down near the end, but left things running overnight and everything worked.

Pretty much a similar story here. On Ubuntu LTS running with -reindex.
A significant slowdown towards the last 4000 blocks. Took almost a day to reindex those 4000.

Another issue:
I had to reset my machine (cold restart) and now I get in terminal upon launching bitcoin-qt:
************************
EXCEPTION: 13leveldb_error       
Database I/O error       
bitcoin in Runaway exception       

I'm on ext4 with journaling.
Geez, I thought LevelDB on a journaling fs can't be corrupted by a reset.