Bitcoin Forum
March 29, 2024, 10:09:26 AM *
News: Latest Bitcoin Core release: 26.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1] 2 »  All
  Print  
Author Topic: Next Steps and Testers wanted  (Read 14364 times)
Andreas Schildbach (OP)
Moderator
Hero Member
*
Offline Offline

Activity: 483
Merit: 501


View Profile
February 06, 2013, 12:03:27 PM
Last edit: February 12, 2013, 07:48:19 AM by Goonie
 #1

I thought I'd give you a quick update on the progress lately.

I'm working on the next version of Bitcoin Wallet which will not have a lot of user-facing features but rather focusses on much more efficient network usage.

For that to happen, Matt Corallo and Mike Hearn have implemented Bloom Filters into the soon to be released bitcoin-qt 0.8 and bitcoinj (will go into 0.7).

With Bloom Filters, only the data relevant to your wallet will be transferred, plus some more for anonymity purposes. This saves network, CPU resources and at the end of the day your battery will last longer.

You can help testing this. Get a preview from:

http://code.google.com/p/bitcoin-wallet/downloads/list

Use the trusted peer prefs for connecting to a Bloom enabled node. Since Bitcoin 0.8 is not released yet you either need to compile and run your own node or you can use Mike's riker.plan99.net.

I'd be especially interested in comments from people who suffered from high usage of their network data plan.
1711706966
Hero Member
*
Offline Offline

Posts: 1711706966

View Profile Personal Message (Offline)

Ignore
1711706966
Reply with quote  #2

1711706966
Report to moderator
1711706966
Hero Member
*
Offline Offline

Posts: 1711706966

View Profile Personal Message (Offline)

Ignore
1711706966
Reply with quote  #2

1711706966
Report to moderator
1711706966
Hero Member
*
Offline Offline

Posts: 1711706966

View Profile Personal Message (Offline)

Ignore
1711706966
Reply with quote  #2

1711706966
Report to moderator
TalkImg was created especially for hosting images on bitcointalk.org: try it next time you want to post an image
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
1711706966
Hero Member
*
Offline Offline

Posts: 1711706966

View Profile Personal Message (Offline)

Ignore
1711706966
Reply with quote  #2

1711706966
Report to moderator
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1063


Gerald Davis


View Profile
February 06, 2013, 02:22:29 PM
 #2

Are there any docs on how the bloom filter works?
Andreas Schildbach (OP)
Moderator
Hero Member
*
Offline Offline

Activity: 483
Merit: 501


View Profile
February 06, 2013, 02:32:36 PM
 #3

Are there any docs on how the bloom filter works?

The specification can be found here:
https://en.bitcoin.it/wiki/BIP_0037

And there is a general description of bloom filters here:
http://en.wikipedia.org/wiki/Bloom_filter
Mike Hearn
Legendary
*
Offline Offline

Activity: 1526
Merit: 1128


View Profile
February 06, 2013, 04:41:30 PM
 #4

Here is a 10 second explanation.

Currently SPV (lightweight p2p) clients must download all transactions from the entire Bitcoin system, even though they do not verify them or check signatures. Instead they're downloaded, checked to see if they send money to or from your wallet, and if not the data is just thrown away.

This is wasteful, slow and doesn't scale because it means as more people use Bitcoin syncing the chain gets ever slower. Also, it pushes people towards clients like Electrum and BitcoinSpinner which use custom protocols which do all the work on dedicated servers.

Bloom filtering solves this. When it connects, the client creates a data structure (the filter) that represents the keys in the users wallet and sends it to the remote peers. They then only send the transactions that match the filter across the network.

The system we have chosen has desirable properties:

  • When transactions are sent to the client, they are accompanied by proofs that they were included in the block chain (merkle branches). Remote nodes cannot send you fake money unless they are willing to mine fake chains, which is hard. (note: they can try and omit transactions, but because you can use any Bitcoin node they are unlikely to get away with it)
  • The Bloom filters are probabilistic. They can have false positives, but no false negatives. That means the remote peer receives a noisy view of your wallet - it'll end up sending you transactions that aren't actually involving your keys (which get thrown away of course) and can't tell which are really yours and which are not. You can choose the false positive rate to trade off bandwidth vs privacy.

In this way, we can take a step closer to having Electrum/BCSpinner levels of performance but with strong privacy, no central points of failure and using Satoshis original vision of a purely peer to peer network.
streblo
Full Member
***
Offline Offline

Activity: 165
Merit: 100


View Profile
February 06, 2013, 06:52:34 PM
 #5

Wow, thank you for the excellent explanation!  Shocked
Andreas Schildbach (OP)
Moderator
Hero Member
*
Offline Offline

Activity: 483
Merit: 501


View Profile
February 06, 2013, 08:56:00 PM
Last edit: February 06, 2013, 09:13:01 PM by Goonie
 #6

Anyone know where there's a copy of bitcoin 0.8? Can't find it on Gavin's page anymore.

I believe there is no binary yet.

You can clone the bitcoin repository and build your own copy:

https://github.com/bitcoin/bitcoin

(as long as 0.8 is not tagged, just use the master branch)

Update: rc1 has been tagged as v0.8.0rc1
Andreas Schildbach (OP)
Moderator
Hero Member
*
Offline Offline

Activity: 483
Merit: 501


View Profile
February 06, 2013, 10:47:47 PM
 #7

I believe there is no binary yet.

Sometimes things change quicker than expected. Binaries have been uploaded to

http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.8.0/test/
slothbag
Sr. Member
****
Offline Offline

Activity: 369
Merit: 250



View Profile
February 07, 2013, 05:55:22 AM
 #8

Just upgraded from one of the 0.8 turbo builds.. Renamed database folders, syncing from very start again.. Argh, last time I went through this it was a week long process. Sad

Downloading bootstrap again Sad
cardinalG
Newbie
*
Offline Offline

Activity: 56
Merit: 0


View Profile
February 07, 2013, 06:33:55 AM
 #9

Here is a 10 second explanation.

Currently SPV (lightweight p2p) clients must download all transactions from the entire Bitcoin system, even though they do not verify them or check signatures. Instead they're downloaded, checked to see if they send money to or from your wallet, and if not the data is just thrown away.

This is wasteful, slow and doesn't scale because it means as more people use Bitcoin syncing the chain gets ever slower. Also, it pushes people towards clients like Electrum and BitcoinSpinner which use custom protocols which do all the work on dedicated servers.

Bloom filtering solves this. When it connects, the client creates a data structure (the filter) that represents the keys in the users wallet and sends it to the remote peers. They then only send the transactions that match the filter across the network.

The system we have chosen has desirable properties:

  • When transactions are sent to the client, they are accompanied by proofs that they were included in the block chain (merkle branches). Remote nodes cannot send you fake money unless they are willing to mine fake chains, which is hard. (note: they can try and omit transactions, but because you can use any Bitcoin node they are unlikely to get away with it)
  • The Bloom filters are probabilistic. They can have false positives, but no false negatives. That means the remote peer receives a noisy view of your wallet - it'll end up sending you transactions that aren't actually involving your keys (which get thrown away of course) and can't tell which are really yours and which are not. You can choose the false positive rate to trade off bandwidth vs privacy.

In this way, we can take a step closer to having Electrum/BCSpinner levels of performance but with strong privacy, no central points of failure and using Satoshis original vision of a purely peer to peer network.

Huh
Andreas Schildbach (OP)
Moderator
Hero Member
*
Offline Offline

Activity: 483
Merit: 501


View Profile
February 07, 2013, 09:39:34 AM
 #10

I get only 1 connected peer with plan99.net as trusted peer on both 3g and wifi, same with my internal desktop ip with wifi.

This is expected. If you configure a trusted peer, Bitcoin Wallet only connects to that peer. Just don't forget to remove the trusted peer after finishing your testing.
Andreas Schildbach (OP)
Moderator
Hero Member
*
Offline Offline

Activity: 483
Merit: 501


View Profile
February 08, 2013, 08:07:59 PM
 #11

Have port 8333 open now, 0.8 seems to be causing some network lag in other apps (noticeable in online games) not had a look what's going on yet, maybe its just the extra connections.

You can address that to the core bitcoin team. Either open a ticket on their bugtracker or post in the forum.

Quote
Something strange, when I put my desktop internal ip as a trusted peer I just have that 1 connection. If I shut down bitcoin-qt and re-open it later Bitcoin Wallet doesn't seem to re-connect to it, sync is stalled on 1 hour atm. Don't think its an issue with my wifi, its been weak but constant the whole time.

Thanks for catching this.

Well, trusted peers are not sticky like any other peer. So if they cannot be connected to or disconnect, they'll be thrown away. You can deliberately disconnect (see action bar overflow menu) and reconnect by opening the app again.

I'll open a ticket for bitcoinj so addresses can be marked as sticky.
Mike Hearn
Legendary
*
Offline Offline

Activity: 1526
Merit: 1128


View Profile
February 09, 2013, 10:17:10 AM
 #12

Huh

OK, let me try again. It makes things fast and cheap (bandwidth wise).
Mike Hearn
Legendary
*
Offline Offline

Activity: 1526
Merit: 1128


View Profile
February 09, 2013, 10:18:22 AM
 #13

If your Bitcoin-Qt is still syncing then it's expected to put heavy load on the system. Generally Bitcoin is an intensive app and will only get moreso. Running it along side performance sensitive apps like games is a recipe for problems.
Andreas Schildbach (OP)
Moderator
Hero Member
*
Offline Offline

Activity: 483
Merit: 501


View Profile
February 12, 2013, 07:49:12 AM
 #14

Mike's test node (running bitcoind 0.8) moved to

riker.plan99.net
secp256k1
Newbie
*
Offline Offline

Activity: 9
Merit: 0



View Profile
February 12, 2013, 08:16:49 PM
 #15

Testing just uploaded: bitcoin-wallet-2.39_bitcoinj0.7_spendpolicies-test.apk

'Trusted Peer' set to: riker.plan99.net
'Skip Regular Peer discovery': set

The 'Peer Monitor' shows 'No peers connected'. Is riker.plan99.net alive?

Will install http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.8.0/test/ and try connect to it.
jim618
Legendary
*
Offline Offline

Activity: 1708
Merit: 1066



View Profile WWW
February 12, 2013, 08:50:58 PM
 #16

Hi Andreas,
I was wondering what sync speeds you were seeing with bloom filters ?

If I download 1 months / 3 months of blocks I am getting about 80 blocks/ second on 3G and 130 blocks/ second on WiFi (1MB down, 100kB up). This is from riker.plan99.net

Android Wallet and MultiBit should be getting about the same I think but it would be nice to have the numbers to confirm it.

MultiBit HD   Lightweight desktop client.                    Bitcoin Solutions Ltd   Bespoke software. Consultancy.
Andreas Schildbach (OP)
Moderator
Hero Member
*
Offline Offline

Activity: 483
Merit: 501


View Profile
February 13, 2013, 05:41:42 PM
 #17

Testing just uploaded: bitcoin-wallet-2.39_bitcoinj0.7_spendpolicies-test.apk

'Trusted Peer' set to: riker.plan99.net
'Skip Regular Peer discovery': set

The 'Peer Monitor' shows 'No peers connected'. Is riker.plan99.net alive?

Will install http://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.8.0/test/ and try connect to it.

riker.plan99.net is only serving prodnet, not testnet. Sorry, I forgot to mention this.

However, on testnet its quite easy to connect to a 0.8 node. Just have a look at the peer monitor and see which peer its downloading from (printed in bold). Of course, its always good to connect to your own peer, for debugging purposes.
Andreas Schildbach (OP)
Moderator
Hero Member
*
Offline Offline

Activity: 483
Merit: 501


View Profile
February 13, 2013, 05:51:16 PM
 #18

My Nexus Galaxy appears to be I/O-limited, maxing out at about 40 blocks/second even with just getheaders (no bloom filtering involved) and on speedy & stable WLAN. My g-slate can do up to 80 blocks/second.

Write ahead caching might be a solution. This can write blocks in larger chunks and skip most of the seeking to the start and writing the hash of the chain head all the time.

However, this would introduce the risk of the chain getting out of sync with the wallet(s). Thus, after getting advice from Mike I postponed these optimization until bitcoinj can recover from drift without replaying the whole blockchain.


Hi Andreas,
I was wondering what sync speeds you were seeing with bloom filters ?

If I download 1 months / 3 months of blocks I am getting about 80 blocks/ second on 3G and 130 blocks/ second on WiFi (1MB down, 100kB up). This is from riker.plan99.net

Android Wallet and MultiBit should be getting about the same I think but it would be nice to have the numbers to confirm it.
jim618
Legendary
*
Offline Offline

Activity: 1708
Merit: 1066



View Profile WWW
February 13, 2013, 06:00:07 PM
 #19

These speeds look comparable I think.

This is only the first Bloom filter code so I imagine there will be refinements in the future.
For mobile phones the reduction in total bandwidth used will be very handy for people on monthly data caps also.

MultiBit HD   Lightweight desktop client.                    Bitcoin Solutions Ltd   Bespoke software. Consultancy.
Andreas Schildbach (OP)
Moderator
Hero Member
*
Offline Offline

Activity: 483
Merit: 501


View Profile
February 13, 2013, 06:07:17 PM
 #20

This is only the first Bloom filter code so I imagine there will be refinements in the future.
For mobile phones the reduction in total bandwidth used will be very handy for people on monthly data caps also.

Bloom filters will never be faster than just using getheaders. But yes, network efficiency is more important anyway.

And who knows, maybe its good that SPV clients are not fully optimized right away. After all, bloom filters puts some burden on the full node. We can optimize when bloom filters have been rolled out to 40.000+ Bitcoin users and understand the implications.
Pages: [1] 2 »  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!