Title: Next Steps and Testers wanted Post by: Andreas Schildbach on February 06, 2013, 12:03:27 PM I thought I'd give you a quick update on the progress lately.
I'm working on the next version of Bitcoin Wallet which will not have a lot of user-facing features but rather focusses on much more efficient network usage. For that to happen, Matt Corallo and Mike Hearn have implemented Bloom Filters into the soon to be released bitcoin-qt 0.8 and bitcoinj (will go into 0.7). With Bloom Filters, only the data relevant to your wallet will be transferred, plus some more for anonymity purposes. This saves network, CPU resources and at the end of the day your battery will last longer. You can help testing this. Get a preview from: http://code.google.com/p/bitcoin-wallet/downloads/list Use the trusted peer prefs for connecting to a Bloom enabled node. Since Bitcoin 0.8 is not released yet you either need to compile and run your own node or you can use Mike's riker.plan99.net. I'd be especially interested in comments from people who suffered from high usage of their network data plan. Title: Re: Next Steps and Testers wanted Post by: DeathAndTaxes on February 06, 2013, 02:22:29 PM Are there any docs on how the bloom filter works?
Title: Re: Next Steps and Testers wanted Post by: Andreas Schildbach on February 06, 2013, 02:32:36 PM Are there any docs on how the bloom filter works? The specification can be found here: https://en.bitcoin.it/wiki/BIP_0037 And there is a general description of bloom filters here: http://en.wikipedia.org/wiki/Bloom_filter Title: Re: Next Steps and Testers wanted Post by: Mike Hearn on February 06, 2013, 04:41:30 PM Here is a 10 second explanation.
Currently SPV (lightweight p2p) clients must download all transactions from the entire Bitcoin system, even though they do not verify them or check signatures. Instead they're downloaded, checked to see if they send money to or from your wallet, and if not the data is just thrown away. This is wasteful, slow and doesn't scale because it means as more people use Bitcoin syncing the chain gets ever slower. Also, it pushes people towards clients like Electrum and BitcoinSpinner which use custom protocols which do all the work on dedicated servers. Bloom filtering solves this. When it connects, the client creates a data structure (the filter) that represents the keys in the users wallet and sends it to the remote peers. They then only send the transactions that match the filter across the network. The system we have chosen has desirable properties:
In this way, we can take a step closer to having Electrum/BCSpinner levels of performance but with strong privacy, no central points of failure and using Satoshis original vision of a purely peer to peer network. Title: Re: Next Steps and Testers wanted Post by: streblo on February 06, 2013, 06:52:34 PM Wow, thank you for the excellent explanation! :o
Title: Re: Next Steps and Testers wanted Post by: Andreas Schildbach on February 06, 2013, 08:56:00 PM Anyone know where there's a copy of bitcoin 0.8? Can't find it on Gavin's page anymore. I believe there is no binary yet. You can clone the bitcoin repository and build your own copy: https://github.com/bitcoin/bitcoin (as long as 0.8 is not tagged, just use the master branch) Update: rc1 has been tagged as v0.8.0rc1 Title: Re: Next Steps and Testers wanted Post by: Andreas Schildbach on February 06, 2013, 10:47:47 PM I believe there is no binary yet. Sometimes things change quicker than expected. Binaries have been uploaded to http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.8.0/test/ Title: Re: Next Steps and Testers wanted Post by: slothbag on February 07, 2013, 05:55:22 AM Just upgraded from one of the 0.8 turbo builds.. Renamed database folders, syncing from very start again.. Argh, last time I went through this it was a week long process. :(
Downloading bootstrap again :( Title: Re: Next Steps and Testers wanted Post by: cardinalG on February 07, 2013, 06:33:55 AM Here is a 10 second explanation. Currently SPV (lightweight p2p) clients must download all transactions from the entire Bitcoin system, even though they do not verify them or check signatures. Instead they're downloaded, checked to see if they send money to or from your wallet, and if not the data is just thrown away. This is wasteful, slow and doesn't scale because it means as more people use Bitcoin syncing the chain gets ever slower. Also, it pushes people towards clients like Electrum and BitcoinSpinner which use custom protocols which do all the work on dedicated servers. Bloom filtering solves this. When it connects, the client creates a data structure (the filter) that represents the keys in the users wallet and sends it to the remote peers. They then only send the transactions that match the filter across the network. The system we have chosen has desirable properties:
In this way, we can take a step closer to having Electrum/BCSpinner levels of performance but with strong privacy, no central points of failure and using Satoshis original vision of a purely peer to peer network. ??? Title: Re: Next Steps and Testers wanted Post by: Andreas Schildbach on February 07, 2013, 09:39:34 AM I get only 1 connected peer with plan99.net as trusted peer on both 3g and wifi, same with my internal desktop ip with wifi. This is expected. If you configure a trusted peer, Bitcoin Wallet only connects to that peer. Just don't forget to remove the trusted peer after finishing your testing. Title: Re: Next Steps and Testers wanted Post by: Andreas Schildbach on February 08, 2013, 08:07:59 PM Have port 8333 open now, 0.8 seems to be causing some network lag in other apps (noticeable in online games) not had a look what's going on yet, maybe its just the extra connections. You can address that to the core bitcoin team. Either open a ticket on their bugtracker or post in the forum (https://bitcointalk.org/index.php?board=6.0). Quote Something strange, when I put my desktop internal ip as a trusted peer I just have that 1 connection. If I shut down bitcoin-qt and re-open it later Bitcoin Wallet doesn't seem to re-connect to it, sync is stalled on 1 hour atm. Don't think its an issue with my wifi, its been weak but constant the whole time. Thanks for catching this. Well, trusted peers are not sticky like any other peer. So if they cannot be connected to or disconnect, they'll be thrown away. You can deliberately disconnect (see action bar overflow menu) and reconnect by opening the app again. I'll open a ticket for bitcoinj so addresses can be marked as sticky. Title: Re: Next Steps and Testers wanted Post by: Mike Hearn on February 09, 2013, 10:17:10 AM ??? OK, let me try again. It makes things fast and cheap (bandwidth wise). Title: Re: Next Steps and Testers wanted Post by: Mike Hearn on February 09, 2013, 10:18:22 AM If your Bitcoin-Qt is still syncing then it's expected to put heavy load on the system. Generally Bitcoin is an intensive app and will only get moreso. Running it along side performance sensitive apps like games is a recipe for problems.
Title: Re: Next Steps and Testers wanted Post by: Andreas Schildbach on February 12, 2013, 07:49:12 AM Mike's test node (running bitcoind 0.8) moved to
riker.plan99.net Title: Re: Next Steps and Testers wanted Post by: secp256k1 on February 12, 2013, 08:16:49 PM Testing just uploaded: bitcoin-wallet-2.39_bitcoinj0.7_spendpolicies-test.apk
'Trusted Peer' set to: riker.plan99.net 'Skip Regular Peer discovery': set The 'Peer Monitor' shows 'No peers connected'. Is riker.plan99.net alive? Will install http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.8.0/test/ and try connect to it. Title: Re: Next Steps and Testers wanted Post by: jim618 on February 12, 2013, 08:50:58 PM Hi Andreas,
I was wondering what sync speeds you were seeing with bloom filters ? If I download 1 months / 3 months of blocks I am getting about 80 blocks/ second on 3G and 130 blocks/ second on WiFi (1MB down, 100kB up). This is from riker.plan99.net Android Wallet and MultiBit should be getting about the same I think but it would be nice to have the numbers to confirm it. Title: Re: Next Steps and Testers wanted Post by: Andreas Schildbach on February 13, 2013, 05:41:42 PM Testing just uploaded: bitcoin-wallet-2.39_bitcoinj0.7_spendpolicies-test.apk 'Trusted Peer' set to: riker.plan99.net 'Skip Regular Peer discovery': set The 'Peer Monitor' shows 'No peers connected'. Is riker.plan99.net alive? Will install http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.8.0/test/ and try connect to it. riker.plan99.net is only serving prodnet, not testnet. Sorry, I forgot to mention this. However, on testnet its quite easy to connect to a 0.8 node. Just have a look at the peer monitor and see which peer its downloading from (printed in bold). Of course, its always good to connect to your own peer, for debugging purposes. Title: Re: Next Steps and Testers wanted Post by: Andreas Schildbach on February 13, 2013, 05:51:16 PM My Nexus Galaxy appears to be I/O-limited, maxing out at about 40 blocks/second even with just getheaders (no bloom filtering involved) and on speedy & stable WLAN. My g-slate can do up to 80 blocks/second.
Write ahead caching might be a solution. This can write blocks in larger chunks and skip most of the seeking to the start and writing the hash of the chain head all the time. However, this would introduce the risk of the chain getting out of sync with the wallet(s). Thus, after getting advice from Mike I postponed these optimization until bitcoinj can recover from drift without replaying the whole blockchain. Hi Andreas, I was wondering what sync speeds you were seeing with bloom filters ? If I download 1 months / 3 months of blocks I am getting about 80 blocks/ second on 3G and 130 blocks/ second on WiFi (1MB down, 100kB up). This is from riker.plan99.net Android Wallet and MultiBit should be getting about the same I think but it would be nice to have the numbers to confirm it. Title: Re: Next Steps and Testers wanted Post by: jim618 on February 13, 2013, 06:00:07 PM These speeds look comparable I think.
This is only the first Bloom filter code so I imagine there will be refinements in the future. For mobile phones the reduction in total bandwidth used will be very handy for people on monthly data caps also. Title: Re: Next Steps and Testers wanted Post by: Andreas Schildbach on February 13, 2013, 06:07:17 PM This is only the first Bloom filter code so I imagine there will be refinements in the future. For mobile phones the reduction in total bandwidth used will be very handy for people on monthly data caps also. Bloom filters will never be faster than just using getheaders. But yes, network efficiency is more important anyway. And who knows, maybe its good that SPV clients are not fully optimized right away. After all, bloom filters puts some burden on the full node. We can optimize when bloom filters have been rolled out to 40.000+ Bitcoin users and understand the implications. Title: Re: Next Steps and Testers wanted Post by: secp256k1 on February 13, 2013, 09:13:00 PM Testing just uploaded: bitcoin-wallet-2.39_bitcoinj0.7_spendpolicies-test.apk 'Trusted Peer' set to: riker.plan99.net 'Skip Regular Peer discovery': set The 'Peer Monitor' shows 'No peers connected'. Is riker.plan99.net alive? Will install http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.8.0/test/ and try connect to it. riker.plan99.net is only serving prodnet, not testnet. Sorry, I forgot to mention this. However, on testnet its quite easy to connect to a 0.8 node. Just have a look at the peer monitor and see which peer its downloading from (printed in bold). Of course, its always good to connect to your own peer, for debugging purposes. Running my own node on testnet. Peer monitor shows it is connected. I always get 'wallet isn't responding' OS message and need to close the app as soon as enter Peer Monitor Activity. LogCat shows: W/System.err(23852): 290979 [New I/O client worker #1-1] INFO com.google.bitcoin .core.AbstractBlockChain - Difficulty transition traversal took 39msec W/System.err(23852): 291369 [Wallet auto save thread] INFO com.google.bitcoin.co re.Wallet - Auto-saving wallet, last seen block is 00000000005675aa4a61b4f66248b 2eed522d697e82eff2e38a90d6434f7e5b0 W/System.err(23852): 291424 [New I/O client worker #1-1] INFO com.google.bitcoin .core.AbstractBlockChain - 50 blocks per second W/System.err(23852): 292395 [Wallet auto save thread] INFO com.google.bitcoin.co re.Wallet - Auto-saving wallet, last seen block is 000000002c5e306733b2a26847693 b100ef375ce47eacd93a6379ea99089ea30 W/System.err(23852): 293438 [New I/O client worker #1-1] INFO com.google.bitcoin .core.AbstractBlockChain - 27 blocks per second W/System.err(23852): 294436 [Wallet auto save thread] INFO com.google.bitcoin.co re.Wallet - Auto-saving wallet, last seen block is 00000000008f45ecaa1806b3ca8ef defff8242ede3a5bfd2fa8656933489f272 W/System.err(23852): 294465 [New I/O client worker #1-1] INFO com.google.bitcoin .core.AbstractBlockChain - 23 blocks per second W/System.err(23852): 295463 [Wallet auto save thread] INFO com.google.bitcoin.co re.Wallet - Auto-saving wallet, last seen block is 000000000004d0a3b85431492bbce f69284d617efb3812fe61ffcca4191c295c W/System.err(23852): 295491 [New I/O client worker #1-1] INFO com.google.bitcoin .core.AbstractBlockChain - 54 blocks per second W/System.err(23852): 296491 [Wallet auto save thread] INFO com.google.bitcoin.co re.Wallet - Auto-saving wallet, last seen block is 0000000000f50172ca238e36bc4d9 2f66d640feaeb0d7f6e5a460896c7ae3900 W/System.err(23852): 296916 [New I/O client worker #1-1] INFO com.google.bitcoin .core.AbstractBlockChain - 3 blocks per second W/System.err(23852): 297789 [Wallet auto save thread] INFO com.google.bitcoin.co re.Wallet - Auto-saving wallet, last seen block is 00000000002704402ff14fb06f338 2a9c5be5a45d42a4f14e5a94a868aa6a4a7 W/System.err(23852): 297978 [New I/O client worker #1-1] INFO com.google.bitcoin .core.AbstractBlockChain - 15 blocks per second W/System.err(23852): 298835 [Wallet auto save thread] INFO com.google.bitcoin.co re.Wallet - Auto-saving wallet, last seen block is 00000000001f6cda2111f0ad8897e 5d52695771fa0458591da077325e20e60c0 W/System.err(23852): 298981 [New I/O client worker #1-1] INFO com.google.bitcoin .core.AbstractBlockChain - 53 blocks per second W/System.err(23852): 299863 [Wallet auto save thread] INFO com.google.bitcoin.co re.Wallet - Auto-saving wallet, last seen block is 00000000007a13c5d39841f3571ef 1bc3bd924557c1994686f34283603feec85 W/System.err(23852): 300614 [New I/O client worker #1-1] INFO com.google.bitcoin .core.AbstractBlockChain - 8 blocks per second Title: Re: Next Steps and Testers wanted Post by: Andreas Schildbach on February 13, 2013, 09:26:18 PM Running my own node on testnet. Peer monitor shows it is connected. I always get 'wallet isn't responding' OS message and need to close the app as soon as enter Peer Monitor Activity. You're probably running into http://code.google.com/p/bitcoin-wallet/issues/detail?id=134 https://code.google.com/p/bitcoinj/issues/detail?id=310 Can you run a adb pull /data/anr/traces.txt just after the ANR message appears and attach it to the Bitcoin Wallet ticket? I will confirm if its the same problem. Title: Re: Next Steps and Testers wanted Post by: secp256k1 on February 14, 2013, 08:41:46 AM Running my own node on testnet. Peer monitor shows it is connected. I always get 'wallet isn't responding' OS message and need to close the app as soon as enter Peer Monitor Activity. You're probably running into http://code.google.com/p/bitcoin-wallet/issues/detail?id=134 https://code.google.com/p/bitcoinj/issues/detail?id=310 Can you run a adb pull /data/anr/traces.txt just after the ANR message appears and attach it to the Bitcoin Wallet ticket? I will confirm if its the same problem. Traces submitted, see Issue 136. Yes, it looks like the problem you mentioned. Title: Re: Next Steps and Testers wanted Post by: Mike Hearn on February 14, 2013, 11:12:43 AM The workaround for now is to not open the peer monitor until block chain sync has finished.
Title: Re: Next Steps and Testers wanted Post by: Andreas Schildbach on February 18, 2013, 07:24:41 PM I have updated the previews with the latest code from bitcoinj and Bitcoin Wallet:
http://code.google.com/p/bitcoin-wallet/downloads/list There is a lot of small changes and optimizations, and the peer monitor lockup should also be fixed. |