A bit of criticism on how the bitcoin client does it

[piotr_n]

As some of you may have noticed, I have worked on my own implementation of a bitcoin node. During this process I found things, which I would like to point out, so maybe they could be improved in a future...
Basically, they are all network bandwidth wasting related.

1. [EDIT: never mind]
Maybe I have not verified it well enough, but I have an impression that the original client, whenever it sends "getblocks", it always asks as deep as possible - why to do this? It just creates unnecessary traffic. You can surely optimize it, without much effort.

2. [EDIT: never mind]
IMO, you should also optimize the way you do "getdata".
Don't just send getdata for all the block that you don't know, to all the peers at the same time - it's crazy.
Instead, try to i.e. ask each node for a different block - one at a time, until you collect them all...

3. [EDIT: please, do mind]
Last, but not least.
The blocks are getting bigger, but there have been no improvements to the protocol, whatsoever.
You cannot i.e. ask a peer for a part of a block - you just need to download the whole 1MB of it from a single IP.
Moreover, each block has an exact hash, so it is just stupid that in times when even MtGox API goes through CloudFlare to save bandwidth, there is no solution that would allow a node to just download a block from an HTTP server, and so it is forced to leech it from the poor, mostly DSL armed, peers.
The way I see it, the solution would be very simple: every mining pool can easily use CloudFlare (or any other cloud service) to serve blocks via HTTP.
So if my node says "getdata ...", I do not necessarily mean that I want to have this megabyte of data from the poor node and its thin DSL connection. I would be more than happy to just get a URL, where I can download the block from - it surely would be faster, and would not drain the peer's uplink bandwidth that much.


That's about it, but if I recall something more, I will surely append it to this topic.

Also, please don't take my criticism personally - it's only meant as a feedback, pointing out areas that I think are important to improve, becasue the official bitcoin client is already eating up my internet connection more efficiently than BitTorrent, and that is very not cool

[1715700052]

1715700052

[1715700052]

1715700052

[1715700052]

1715700052

[1715700052]

1715700052

[Remember remember the 5th of November]

In what language are you implementing the node? I've been having an idea of also writing one, perhaps in C, but who knows, I don't usually finish projects.

[piotr_n]

In what language are you implementing the node? I've been having an idea of also writing one, perhaps in C, but who knows, I don't usually finish projects.

I have implemented it, in Go language: https://bitcointalk.org/index.php?topic=199306.0

[Remember remember the 5th of November]

In what language are you implementing the node? I've been having an idea of also writing one, perhaps in C, but who knows, I don't usually finish projects.

I have implemented it, in Go language: https://bitcointalk.org/index.php?topic=199306.0

I thought the moto of the Go language is simple and efficient, will it even perform well?

[piotr_n]

In what language are you implementing the node? I've been having an idea of also writing one, perhaps in C, but who knows, I don't usually finish projects.

I have implemented it, in Go language: https://bitcointalk.org/index.php?topic=199306.0

I thought the moto of the Go language is simple and efficient, will it even perform well?

It is simple and efficient and it surely performs well.
I would even risk a statement that it performs better than the current satoshi client, though it takes much more RAM, so we are talking about different abstraction layers here.
Crypto operation are likely less efficient in Go, then what openssl gives, but my Go client only does them after checking a new highest block against the expected difficulty - so less often...

But the only way to be sure how fast it actually is, is to check it out by yourself

[kjj]

These are all pretty common topics in IRC.  I recall some of them from the mailing list too, but I don't read that daily any more.  People are working on various parts.  For one example, see here.  The bootstrap torrent is another example.

Pitch in and help if you can.

[piotr_n]

Sorry, but I think I can only help by implementing any of these new ideas in my client.
I just don't like to have a boss and in my code I am the boss - hope you understand

Anyway, if anyone wants to test a solution against a different client - just let me know.
I can implement it, if it's not really crazy - and then we can test each other, benefiting everyone

[Pieter Wuille]

1.
Maybe I have not verified it well enough, but I have an impression that the original client, whenever it sends "getblocks", it always asks as deep as possible - why to do this? It just creates unnecessary traffic. You can surely optimize it, without much effort.

Not sure what you mean by "as deep as possible". We always send getdata starting at whatever block we already know. The reason for starting from early blocks and moving forward is because validation is done is stages, and at each point as much as possible is already validated (as a means to prevent DoS attacks, mostly). As most checks can only be done when you have the entire chain of blocks from genesis to the one being verified, you need them more or less in order.

2.
IMO, you should also optimize the way you do "getdata".
Don't just send getdata for all the block that you don't know, to all the peers at the same time - it's crazy.
Instead, try to i.e. ask each node for a different block - one at a time, until you collect them all...

That's not true, we only ask for each block once (and retry after a timeout), but it is done to a single peer (not to all, and not balanced across nodes). That's a known badness, but changing isn't trivial, because of how validation is done.

There is one strategy however that's pretty much accepted as the way to go, but of course someone still has to implement it, test it, ... and it's a pretty large change. The basic idea is that downloading happens in stages as well, where first only headers are fetched (using getheaders) in a process similar to how getblocks is done now, only much faster of course. However, instead of immediately fetching blocks, wait until a long chain of headers is available and verified. Then you can start fetching individual blocks from individual peers, assemble them, and validate as they are connected to the chain. The advantage is that you already know which chain to fetch blocks from, and don't need to infer that from what others tell you.

3.
Last, but not least.
Forgive me the sarcasm, but I really don't know what all the people in the Bitcoin Foundation have been doing for the past years, besides suing each other and increasing transaction fees

The Bitcoin Foundation has only been around for a year or so, and they don't control development. They pay Gavin's salary, but other developers are volunteers that work on Bitcoin in their free time.

We all know that the current bitcoin protocol does not scale - so what has been done to improve it?
Nothing!
The blocks are getting bigger, but there have been no improvements to the protocol, whatsoever.
You cannot i.e. ask a peer for a part of a block - you just need to download the whole 1MB of it from a single IP.

BIP37 actually introduced a way to fetch parts of blocks, and it can be used to fetch a block with just the transactions you haven't heard about, so it avoids resending those that have already been transferred as separate transactions (though I don't know of any software that uses this mechanism of block fetching yet; once BIP37 is available on more nodes, I expect it will be). Any other system which requires negotiating which transactions to send adds latency to block propagation time, so it's not necessarily the improvement you want.

Moreover, each block has an exact hash, so it is just stupid that in times when even MtGox API goes through CloudFlare to save bandwidth, there is no solution that would allow a node to just download a block from an HTTP server, and so it is forced to leech it from the poor, mostly DSL armed, peers.
The way I see it, the solution would be very simple: every mining pool can easily use CloudFlare (or any other cloud service) to serve blocks via HTTP.
So if my node says "getdata ...", I do not necessarily mean that I want to have this megabyte of data from the poor node and its thin DSL connection. I would be more than happy to just get a URL, where I can download the block from - it surely would be faster, and would not drain the peer's uplink bandwidth that much.

There are many ideas about how to improve historic block download. I've been arguing for a separation between archive storage and fresh block relaying, so nodes could be fully verifying active nodes on the network without being required to provide any ancient block to anyone who asks. Regarding moving to other protocols, there is the bootstrap.dat torrent, and there's recently been talk about other mechanism on the bitcoin-development mailinglist.

[piotr_n]

Not sure what you mean by "as deep as possible". We always send getdata starting at whatever block we already know. The reason for starting from early blocks and moving forward is because validation is done is stages, and at each point as much as possible is already validated (as a means to prevent DoS attacks, mostly). As most checks can only be done when you have the entire chain of blocks from genesis to the one being verified, you need them more or less in order.

What I mean is when the client has already downloaded the full chain - and is basically waiting for a new block.
Why to ask 500 blocks back?

That's not true, we only ask for each block once (and retry after a timeout), but it is done to a single peer (not to all, and not balanced across nodes). That's a known badness, but changing isn't trivial, because of how validation is done.

OK - than I'm sorry.
It only proves how little I know about the bitcoin client, so I should not be changing it

There is one strategy however that's pretty much accepted as the way to go, but of course someone still has to implement it, test it, ... and it's a pretty large change. The basic idea is that downloading happens in stages as well, where first only headers are fetched (using getheaders) in a process similar to how getblocks is done now, only much faster of course. However, instead of immediately fetching blocks, wait until a long chain of headers is available and verified. Then you can start fetching individual blocks from individual peers, assemble them, and validate as they are connected to the chain. The advantage is that you already know which chain to fetch blocks from, and don't need to infer that from what others tell you.

I saw getheaders and I was thinking about using it. But it would basically only help for the initial chain download.
Now I think if you really want to combine the data you got from getheaders, with the parts of blocks acquired from you peers after they have implemented BIP37 (otherwise it won't be much faster) - then good luck with that project, man!
I mean, I would rather prefer baby steps - even extreme, like having a central sever from which you can fetch a block, by its hash. I mean: how expensive would be that? But how much bandwidth would it save for these poor people..

BIP37 actually introduced a way to fetch parts of blocks, and it can be used to fetch a block with just the transactions you haven't heard about, so it avoids resending those that have already been transferred as separate transactions (though I don't know of any software that uses this mechanism of block fetching yet; once BIP37 is available on more nodes, I expect it will be).

Interesting.. thank you, then maybe that should be the next thing that I will add to my client.

There are many ideas about how to improve historic block download. I've been arguing for a separation between archive storage and fresh block relaying, so nodes could be fully verifying active nodes on the network without being required to provide any ancient block to anyone who asks. Regarding moving to other protocols, there is the bootstrap.dat torrent, and there's recently been talk about other mechanism on the bitcoin-development mailinglist.

I was talking more about single blocks being available via HTTP - at the very moment when they have been mined.
I think all you need is an URL - so it should be up to a peer to choose which URL to give you. As long as the hash of the data you download from there matches what you needed, you have no reason to question his method. Otherwise, just ban the bastard

[Pieter Wuille]

Why to ask 500 blocks back?

It doesn't, as far as I know. It asks for "up 500 blocks starting at hash X", where X is the last known block.

There is one strategy however that's pretty much accepted as the way to go, but of course someone still has to implement it, test it, ... and it's a pretty large change. The basic idea is that downloading happens in stages as well, where first only headers are fetched (using getheaders) in a process similar to how getblocks is done now, only much faster of course. However, instead of immediately fetching blocks, wait until a long chain of headers is available and verified. Then you can start fetching individual blocks from individual peers, assemble them, and validate as they are connected to the chain. The advantage is that you already know which chain to fetch blocks from, and don't need to infer that from what others tell you.

I saw getheaders and I was thinking about using it.
Now I think if you really want to combine the data you got from getheaders, with the parts of blocks acquired from you peers after they have implemented BIP37 (otherwise it won't be much faster) - then good luck with that project, man!

Using Bloom filtering may not be entirely viable yet, I'll have to check. The big changes is first downloading and validating headers, and then downloading and validating the blocks itself. IMHO, it's the only way to have a sync mechanism that is both fast, stable and understandable (I have no doubt that there are other emchanisms that share two of those three properties...).

I mean, I would rather prefer baby steps - even extreme, like having a central sever from which you can fetch a block, by its hash. I mean: how expensive would be that? But how much bandwidth would it save for these poor people..

What protocol is used to actually fetch blocks is pretty much orthogonal to the logic of deciding what to fetch, and how to validate it, IMHO.

[piotr_n]

Using Bloom filtering may not be entirely viable yet, I'll have to check. The big changes is first downloading and validating headers, and then downloading and validating the blocks itself. IMHO, it's the only way to have a sync mechanism that is both fast, stable and understandable (I have no doubt that there are other emchanisms that share two of those three properties...).

I still think a simple solution, like "give me this part of this block/transaction", would have a much better chance of success in a short term.
And I also think that it would be nice to have something in a short term

What protocol is used to actually fetch blocks is pretty much orthogonal to the logic of deciding what to fetch, and how to validate it, IMHO.

I disagree. If you are a node behind DSL, and you have a very limited upload bandwidth, you do not want to serve blocks (and maybe even transactions), unless it is really necessary.
There are servers out there, connected to the fastest networks in the world - these you should use, as much as you can. Who is going to stop you?

[Pieter Wuille]

What protocol is used to actually fetch blocks is pretty much orthogonal to the logic of deciding what to fetch, and how to validate it, IMHO.

I disagree. If you are a node behind DSL, and you have a very limited upload bandwidth, you do not want to serve blocks, unless it is really necessary.
There are servers out there, connected to the fastest networks in the world - these you should use, as much as you can. Who is going to stop you?

I agree completely. But it still has nothing to do with your logic of deciding what to fetch and how to validate it. It's just using a different protocol to do it.

[piotr_n]

I agree.
But in reality, the logic of what to fetch is only important during the initial chain download.
Later you just fetch whatever new there is...

So it is not really so important, is it?

[piotr_n]

Maybe we should not be focusing so much on the initial blockchain download, but rather on limiting the bandwidth usage of a completely synchronized node.
As for relaying transactions, I would even go crazy enough to implement a web of trust - where you don't verify every transaction, but only random ones and then build a trust to the node that sends them to you - then you check them randomly, but less frequent.

But also for transactions - the data should be kept on WWW servers. There is no economic reason to fetch them from China

[Mike Hearn]

I still think a simple solution, like "give me this part of this block/transaction", would have a much better chance of success in a short term.

I don't understand your point - that is exactly what Bloom filtering provides. It is deployed and working for SPV clients for some months already. There have been no issues with it. You can't use it as a full node because a full node, by definition, must download full blocks as it must know about all transactions.

Incidentally, if you're going to make sarcastic comments implying Bitcoin hasn't improved, you should actually know what you're talking about. Bloom filtering launched at the start of this year, it's not something that was originally a part of the protocol - so there have been big improvements quite recently.

For distributing the block chain you can as well use Bittorrent or some other large file distribution mechanism rather than HTTP serving, it's already possible and there are already torrents distributing the chain in this way. They aren't designed for end users because end users should eventually all end up on SPV wallets which already only download partial blocks.

[grau]

Yes, Bloom filtering is a significant improvement to the core protocol.

In addition to serving SPV clients it is used to optimize the BOP server's communication to lightweight clients connected to its message bus.

The BOP message bus also offers an API to get blocks.

[piotr_n]

I still think a simple solution, like "give me this part of this block/transaction", would have a much better chance of success in a short term.

I don't understand your point - that is exactly what Bloom filtering provides. It is deployed and working for SPV clients for some months already. There have been no issues with it. You can't use it as a full node because a full node, by definition, must download full blocks as it must know about all transactions.

Well, you have just said it yourself: this Bloom filtering does not help at all, if you want to have a full node.
And I do want to have a full node - probably as well as most of you guys out there.
So how does it help us?

Incidentally, if you're going to make sarcastic comments implying Bitcoin hasn't improved, you should actually know what you're talking about. Bloom filtering launched at the start of this year, it's not something that was originally a part of the protocol - so there have been big improvements quite recently.

I think you were the one who did not know what I was talking about.
Improvements are worthless if there is no actual software, which people want to use, that gets advantage of them.

What I meant by "give me this part of this block/transaction", is literally  "give me X bytes of block Y, starting at offset Z".
So, when a new block appears in the network and I need to download it, while being connected to a number of peers, I don't ask each one of them for the same megabyte of data - instead I can just split the work into, let's say 32KB chunks, and this way fetch the entire new block from my peers much quicker.

But that would only be useful before making the protocol to support fetching blocks from HTTP servers - which is the ultimate solution, which IMO should be implemented ASAP, if you guys really care about all these small bitcoin users and their internet connections. The mining pools should help here, because it is in their very best interest to propagate the blocks they have mined as quickly as possible across the network - and what could be quicker than a static file, served via http, from the pool's domain, through a clodflare-like infrastructure?

For distributing the block chain you can as well use Bittorrent or some other large file distribution mechanism rather than HTTP serving, it's already possible and there are already torrents distributing the chain in this way. They aren't designed for end users because end users should eventually all end up on SPV wallets which already only download partial blocks.

As I said before: I would prefer to focus on improving a behavior of a node that already is synchronized, rather than focusing on making it faster to setup a new one from a scratch.
Initial chain download, and the fact that it takes so long, is inconvenient, but it is not really such a big issue for the actual network.
Besides when you setup a node from scratch, and so need to re-parse the 236+k blocks, the network communication does not seem to me as much of an issue, as all the hashing and elliptic math that your PC needs to go through.


There is another thing that came to my mind, so I will just add it here, to this post.
I believe nodes should not relay transactions that use any inputs which exist only in the memory pool. It should only relay transactions that use inputs from an actually mined blocks.
This IMHO would improve the network's traffic pretty much. A regular user never needs (usually is not never able) to spend an output that has not been mined yet - while, at the other hand, relaying of such transactions takes a huge part of his network connection.

[Mike Hearn]

You're trying to solve a non-existant problem: block propagation is not upload bandwidth limited today so why would anyone add such a protocol feature? That's why I'm confused. You're asking for something that just wouldn't speed anything up.

[piotr_n]

You're trying to solve a non-existant problem: block propagation is not upload bandwidth limited today so why would anyone add such a protocol feature? That's why I'm confused. You're asking for something that just wouldn't speed anything up.

Well man, if that problem is non-existant to you, then I can only envy you the internet connection that you have at home

But even having such a great connection - if you want to create a software that would be able to import the entire block chain, from a scratch, within a few minutes - then I would rather suggest you looking into developing a hardware that supports elliptic curve math, because IMO that seems to be the weakest link in this process - not the network protocol.

And the "block propagation" is eating up a hell lot of the poor bitcoin users' bandwidth - it might not be a problem for you, but it is a problem.

[Peter Todd]

And the "block propagation" is eating up a hell lot of the poor's bitcoins users bandwidth - it might not be a problem for you, but it is a problem.

If you don't have enough bandwidth to be CPU limited, stop trying to run a node. SPV clients are just fine for any users needs unless you want to run a mining pool or maybe operate a big business. If you really want, go get a VPS server; $20-$100/month should buy a fast enough one, at least for another year or two.

[piotr_n]

And the "block propagation" is eating up a hell lot of the poor's bitcoins users bandwidth - it might not be a problem for you, but it is a problem.

If you don't have enough bandwidth to be CPU limited, stop trying to run a node. SPV clients are just fine for any users needs unless you want to run a mining pool or maybe operate a big business. If you really want, go get a VPS server; $20-$100/month should buy a fast enough one, at least for another year or two.

so your advise is: don't run a bitcoin node.
?

because, you know, I would actually like to run a bitcoin node, just to support this fine network, but the current protocol is wasting my bandwidth - and that is my issue.
though, I understand that an unnecessary bandwidth usage is not something that is easy to be noticed, so I am not even surprised that nobody gives a shit about it

[Mike Hearn]

You're just confused, sorry. You have to download all data in every block to run a full node. This is fundamental. You can't reduce bandwidth usage by downloading parts of each block from different peers. This might reduce the upload bandwidth on their side, but it doesn't reduce the download bandwidth on your side. If you're talking about hosting blocks on Cloudfare then you're talking about download bandwidth. So your proposed change wouldn't impact anything.

[2112]

You're just confused, sorry. You have to download all data in every block to run a full node. This is fundamental. You can't reduce bandwidth usage by downloading parts of each block from different peers. This might reduce the upload bandwidth on their side, but it doesn't reduce the download bandwidth on your side. If you're talking about hosting blocks on Cloudfare then you're talking about download bandwidth. So your proposed change wouldn't impact anything.

The above is just another example how Mike Hearn spreads misinformation about Bitcoin protocol.

For normal operation of the Bitcoin network the majority of the "block" has already been previously transferred as a separate "transactions".

The obvious optimization of the bandwidth usage is for the clients to inspect the Merkle tree and ask the peer only for the transactions that weren't previously broadcast.

This gives close to 50% bandwidth savings for free.

I'm writing this to underline the fact that whenver you read something written by Mike Hearn you have to double check and verify for yourself the 3 broad possibilities:

1) he was correct;
2) he was incorrect because he didn't understand the question or doesn't understand the underlying tehnology;
3) he was incorrect intentionally to disparage other developers and spread misinformation.

Thankfully for now he still doesn't belong to the core development team.

[jgarzik]

You're just confused, sorry. You have to download all data in every block to run a full node. This is fundamental. You can't reduce bandwidth usage by downloading parts of each block from different peers. This might reduce the upload bandwidth on their side, but it doesn't reduce the download bandwidth on your side. If you're talking about hosting blocks on Cloudfare then you're talking about download bandwidth. So your proposed change wouldn't impact anything.

The above is just another example how Mike Hearn spreads misinformation about Bitcoin protocol.

For normal operation of the Bitcoin network the majority of the "block" has already been previously transferred as a separate "transactions".

This is true.  The current bitcoind client uses this knowledge in a signature cache, to avoid validating signatures twice (once upon TX reception, once upon block reception).

The obvious optimization of the bandwidth usage is for the clients to inspect the Merkle tree and ask the peer only for the transactions that weren't previously broadcast.

This gives close to 50% bandwidth savings for free.

heh, nothing is free.  This proposal would add additional round-trips with associated latency, slowing block validation and block propagation.

As such, miners could lose their 25 BTC due to orphaning, if their block is slowed.

[2112]

heh, nothing is free.  This proposal would add additional round-trips with associated latency, slowing block validation and block propagation.

As such, miners could lose their 25 BTC due to orphaning, if their block is slowed.

Actually you are wrong. The average block propagation latency would improve, both in case of honest miners and secretive miners.

Firstly, let me explain honest/secretive miner distinction. Honest miner broadcasts all the transactions that he's going to put in the block ahead of time. Then when he wins the block he sends all the transactions once again just to comply with the classic/inefficient block propagation protocol, wasting up to 50% of overall bandwidth.

On the other hand secretive miner tries to play games to disadvantage other miners and raise their orphan rate. He omits broadcasting private transactions separately and when he wins the block he sends those private transactions for the first time. It saves his bandwidth, but disadvantages the competing miners because they see the transactions late and have to spend time to verify them.

So for the network of mostly honest miners there simply isn't any additional round trips and the win is 100% clear. Sending just the block header and the Merkle tree is a no-brainer.

To understand why even the secretive miners case is a net win for the global network is somewhat more complicated. Firstly, the "round-trips" should be singular: only one additional round trip is required to ask for the private transactions of the uncooperative miner.

Secondly, you'll have to understand how the bandwidth is getting sold nowadays. It is very rare to actually linearly limit the bandwith sold to eg. 1ms per bit for 1 Megabit per second. Nearly every modern IP transport uses some sort of statistical limiter/multiplexer. When you buy e.g. 10Mbits per second your effective bandwidth may be 1Gigabit per second for the first, say 4kilobytes and the statistical multiplexer/limiter will throttle your remaining kilobytes to maintain the long-term average. When asked some ISPs will plainly state the setting for e.g. "fair-queue" command in Cisco IOS. Most ISPs will however consider those setting a trade secret; and frequently they change them according to the time-of-day, day-of-the-week or even to quell the packet storms after "network events".

The interested reader can either read up about the above: e.g. it is called PowerBoost with the DOCSIS cable modems. Anyone with two machines, two GPS receivers to NTP-sync the time on them and a copy of Wireshark can verify what I wrote above running their own experiments and as a side effect reverse-engineer the setting used by their ISP.

To quickly summarize the above two paragraphs: the latency to send e.g. first 4kB of 1MB block is much less than 4/1000 of the latency of the whole block.

I understand that implementing the changes suggested by piotr_n isn't trivial. But if implemented they will give at least two benefits to the global Bitcoin network:

1) information theoretic benefit of saving the bandwith, reducing the block propagation latency and thus reducing the orphan rate

2) motivate more miners to be honest about broadcasting transactions. This will probably have further domino-effect of improving overall game-theorethic strength of the Bitcoin network and may allow in the future to detect some attacks which we even haven't thought through.

In summary I wanted to stress that I enjoyed this brief, but honest exchange of technical arguments with you. Thanks.

[kjj]

You're just confused, sorry. You have to download all data in every block to run a full node. This is fundamental. You can't reduce bandwidth usage by downloading parts of each block from different peers. This might reduce the upload bandwidth on their side, but it doesn't reduce the download bandwidth on your side. If you're talking about hosting blocks on Cloudfare then you're talking about download bandwidth. So your proposed change wouldn't impact anything.

The above is just another example how Mike Hearn spreads misinformation about Bitcoin protocol.

For normal operation of the Bitcoin network the majority of the "block" has already been previously transferred as a separate "transactions".

The obvious optimization of the bandwidth usage is for the clients to inspect the Merkle tree and ask the peer only for the transactions that weren't previously broadcast.

This gives close to 50% bandwidth savings for free.

I'm writing this to underline the fact that whenver you read something written by Mike Hearn you have to double check and verify for yourself the 3 broad possibilities:

1) he was correct;
2) he was incorrect because he didn't understand the question or doesn't understand the underlying tehnology;
3) he was incorrect intentionally to disparage other developers and spread misinformation.

Thankfully for now he still doesn't belong to the core development team.

You forgot #4, technically correct, about something pointless and intentionally ignoring the distinction to give the illusion of superiority.  Odd that you'd forget it, since your post was a perfect example.

Your memory pool won't help you during the initial download, which is the time when people care about traffic.  No one gives a shit about 8 gigs over 4 years.  Everyone cares about 8 gigs today.  Using a piecewise downloading system will save some traffic* for people that don't give a shit about traffic, and save not even a single byte for people that do.

*  "close to 50%" savings needs a lot of assumptions to be true at the same time, when some of them are never true at all.  A partial list:  bitcoin traffic is evenly divided between block bodies and transactions, the list of transactions in a block can be transmitted for free, every node knows about every transaction.

[piotr_n]

I have been looking at BIT37 and it seems that "merkleblock" is exactly what I need in order to divide a new block's download into small chunks and then distribute the block's download among different peers, using a bunch of "getdata 1" instead of one "getdata 2".

The only problem I see is that:

If no filter has been set on the connection, a request for filtered blocks is ignored

So I guess I will need to setup some dummy filter first, just to be able to receive this nice and useful messages.
But... now I wonder: if I setup such a filter, will I still be receiving invs, for freshly mined blocks...? Or what will be other side effects...

I will appreciate any advise here - I really want to implement it. Preferably today

[Pieter Wuille]

I have been looking at BIT37 and it seems that "merkleblock" is exactly what I need in order to divide a new block's download into small chunks and then distribute the block's download among different peers, using a bunch of "getdata 1" instead of one "getdata 2".

I don't think you can perfectly partition blocks using filtered blocks, as there are several matching criteria, and one match is enough to include a transaction. So if you send disjunct Bloom filters (nHashes=1, and disjunct bitsets to each peer), you'll still get doubles. It is perhaps an interesting extension to BIP37 to support such partitioning, especially for larger blocks.

However, one case that is explicitly (and intentionally) supported by BIP37 is requesting blocks (full-node wise) with the transactions already known to the peer filtered out. So contrary to what Mike says, I'm pretty sure you can have a full node that uses BIP37 to fetch blocks, and save download bandwidth using that.

[piotr_n]

I see what you mean now.
I can trick the node into sending me "merkleblock" messages, but they are always followed by a bunch of "tx" messages, which completely ruins my concept of distributing requests for these "tx" among all the connected peers.

But if I make a long bloom filter, then it should never match and so I should be only getting "merkleblock" not followed by any "tx"...
Haven't given up yet

[piotr_n]

Oh, I'm stupid. I forgot that "tx" only work for a not yet mined transactions, so how did I want to acquire a block's payload with them?

Now I have given up

[Pieter Wuille]

But if I make a long bloom filter, then it should never match and so I should be only getting "merkleblock" not followed by any "tx"...
Haven't given up yet

If you don't want any transactions at all, just use getheaders.

[piotr_n]

But if I make a long bloom filter, then it should never match and so I should be only getting "merkleblock" not followed by any "tx"...
Haven't given up yet

If you don't want any transactions at all, just use getheaders.

Yes, this I know, thanks.
But what I wanted to achieve was downloading a block's payload in fragments (from different peers in parallel), using the current protocol; "merkleblock" followed by a bunch of "getdata 1"
And for a moment (or rather for the entire morning) I though that it would be possible...

[Pieter Wuille]

But what I wanted to achieve was downloading a block's payload in fragments (from different peers in parallel), using the current protocol; "merkleblock" followed by a bunch of "getdata 1"
And for a moment (or rather for the entire morning) I though that it would be possible...

I explained in #28 why that is not possible: even with disjunct filters, you'll get transactions matched by both.

[Pieter Wuille]

Oh, I think I see your confusion.

The only way to request blocks is using getdata block or merkleblock, you can't just fetch individual transactions from it, as that would require the peer to have full transaction index. So what you hoped to do was send a merkleblock request to one peer, but without transactions, and then fetch the transactions themself from separate peers. That won't work.

Anyway, as said, in the future this may become a useful idea for an extension to the filtering protocol: adding a "only match the 3rd 1/5 of all transactions" to the filter specification.

[piotr_n]

But what I wanted to achieve was downloading a block's payload in fragments (from different peers in parallel), using the current protocol; "merkleblock" followed by a bunch of "getdata 1"
And for a moment (or rather for the entire morning) I though that it would be possible...

I explained in #28 why that is not possible: even with disjunct filters, you'll get transactions matched by both.

I was assuming that all I needed would be a list of hashes returned by "merkleblock" - because this command seems to be returning all the transaction hashes for a requested block, without any filtering.

BTW, the spec on the wiki is different from the actual format of this message.
There is an extra var_length field between "total_transactions" and "hashes", carrying the same value as "total_transactions".

[piotr_n]

you can't just fetch individual transactions from it, as that would require the peer to have full transaction index.

Exactly - and not that I did not know that...  it just somehow slipped my mind

[Mike Hearn]

What I wrote was correct - you have to download all data in every block. Yes, you don't have to download it twice in the steady state if you set a full match Bloom filter (which Matt short circuited already), but as kjj notes that isn't solving a problem that's important today so it was never finished. It might be useful to optimise block propagation amongst miners, but if you aren't mining it wouldn't have much impact.

I still don't understand what piotr_n is trying to do. Downloading block contents in parallel only helps if remote peers are upload constrained and you are network constrained. That isn't the case for any real nodes today, so why are you trying to optimise it? It won't reduce bandwidth usage one bit.

Edit: OK, I reread the thread and now I see what Piotr is getting at. You want to minimize upload bandwidth for yourself running a full node, not minimize download bandwidth. The references to requesting blocks from different peers made me think you wanted to optimise download. May I ask if this is a real problem for you today or just theoretical? The node I run uploaded around 2.7 gigabytes yesterday, spread out over the full 24 hours. I certainly wouldn't want to run this off my mobile connection but it's not a bank breaker for a node run in a datacenter. Given that there are 144 blocks per day and each one yesterday was less than half a megabyte, even downloaded twice for each node that's downloading from me that's only 144 megabytes of transaction data. If we suppose most bandwidth usage is tx data then perhaps I distributed data to 19 nodes for the day - not bad, I could have supported much more even on a budget VPS.

The problem is just sharding block download doesn't change the aggregate steady state bandwidth usage for the network. Finishing the full match Bloom filter work would, but Piotr already said he isn't going to do anything with bitcoind, just on his own Go node. Well, as most nodes today are Satoshi nodes and I doubt that will change, just implementing full-match filters in the Go node won't change upload bandwidth because remote peers will still request full blocks with redundant tx data in them. To reduce upload bandwidth you have to optimise the other end.

[piotr_n]

May I ask if this is a real problem for you today or just theoretical? The node I run uploaded around 2.7 gigabytes yesterday, spread out over the full 24 hours. I certainly wouldn't want to run this off my mobile connection but it's not a bank breaker for a node run in a datacenter.

The problem is real - trust me. Not everyone lives in a datacenter, some people just have homes, you know.

As for my measurements 2.7 GB per day seems fairly low (its 32KB/s in average), so I tend to disbelieve that you actually run your node from a datacenter.
Unless you only have 8 outgoing connections, in which case that number would make more sense to me.

But anyway: lets say that it is 2.7GB per day now - while the blocks are still far below the 1MB limit. But they will obviously be growing, like they have in the past. So expect 10 GB /day pretty soon. Still makes no impression?

[piotr_n]

Why to ask 500 blocks back?

It doesn't, as far as I know. It asks for "up 500 blocks starting at hash X", where X is the last known block.

I have just checked it again, putting a debug into my getblocks handler.
Each time my node receives "getblocks", there are like tens of different locator hashes, followed by a zero-filled stop.
This (from what I see) always forces my node to return the maximum of 500 blocks.
For instance:

Code:

getblocks  hashes_inside=29  => returned 500 invs / 18003 bytes

and these are the locator hashes that were inside (note the genesis hash at the end):

Code:

00000000000000b1ffa04c08a1372309ee0f38d99a6ea7ac3d98c6bc76747b4d
00000000000000ebddd1d499c21ffe41a99c2fbefe809aecc1e31b7f5eb2a39d
000000000000003372bd8cb3d9cf78b86a0f64b2c30f8c0ce1f4e46f983d091c
000000000000017c700b2179c55343830a5ff07041d2155654c8ba77ef00677f
00000000000001365066267c0084347b180df7070ed112d2cac316c6565ccd2b
0000000000000013b9efa409b459baf11b4ccb43a60f853bdd8d25fb9b867ec3
00000000000000b8e0d6f2f5437cb2e03e973eac6653b1e2ee7c4bfd822ae1f5
000000000000002031feb915bfb85bfd138a0c9abaa53d3d577e022ca01ddacf
00000000000000a15f169467fbbab6bcf9ce7359fce414215e353b31aa7f17bc
00000000000000b705393b66eb3adf92e69cbbb04753357955b5b8dbaf9d273e
0000000000000127933cc0ad1e6a23a852bd391daa843e457ab40d5c226fed38
000000000000009d39a568e0bf5984fe5f3e9882813b686651b691bdfa07ea39
0000000000000142ad32a203b1627bee8126fa4bcd940b0da3f32bf1b5b07a24
000000000000001aef6e9c1ff8e5833c5f38bb1b660d3a3060169f10ba1a293f
00000000000000f5710fe34e22a0791afe7a817044f9c7da7a139295da995b06
0000000000000168cb9711469ccfa6704398c82fca5dd42bb700f11970946be1
00000000000000d9955f9fe2161524121f0e21f3e2b1a57f97578c98dc31d636
000000000000011f76c57b0f941acf03da10cce12cb989ed5c4b7b6722b89cc4
00000000000000d8c733f7bc4295a8af6e0b95f2bafc901076444505572353c1
000000000000009be2ca7e5f878c2235ac7b2ebf653d77c9d4d57b359e071ca2
000000000000014158e599c40a2e3fef433cf70cc1caedd9099c21028e4d97f7
00000000000000f8cc33dbc8c3c37f88598f33373fbb03b96a1eaf6426da7898
0000000000000008c3e7b684e2aa56bebba3beb267a8d932fb84217becc0406b
0000000000000057c46a1306febe518cdb6c1fc882d856bea07e5927a5f15dc3
0000000000000381683c0b95b17233f8683fb89060d6e4eb0d70f96bd51a539f
0000000000000201892dd6ed76236eb8f42e93bf0dad2606cc5aaa246f253bf1
00000000000003d8e22ddc9a6b3b2226dfe57830d793be012720e74db28bc6be
000000000001dbc957fbe83e9d38aed40c9e083b830c36890d538726b24ae1d0
000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f

Actually, if you look at this article it even clearly advises:

To create the block locator hashes, keep pushing hashes until you go back to the genesis block. After pushing 10 hashes back, the step backwards doubles every loop

Yeah, you are only at block 236k+, so just keep pushing all the hashes, starting from the top, until you reach the genesis block - a brilliant idea

[Zeilap]

Actually, if you look at this article it even clearly advises:

To create the block locator hashes, keep pushing hashes until you go back to the genesis block. After pushing 10 hashes back, the step backwards doubles every loop

Yeah, you are only at block 236k+, so just keep pushing all the hashes, starting from the top, until you reach the genesis block - a brilliant idea

What's wrong with this? It sounds to me like you don't understand the purpose of the locators.

[piotr_n]

Actually, if you look at this article it even clearly advises:

To create the block locator hashes, keep pushing hashes until you go back to the genesis block. After pushing 10 hashes back, the step backwards doubles every loop

Yeah, you are only at block 236k+, so just keep pushing all the hashes, starting from the top, until you reach the genesis block - a brilliant idea

What's wrong with this? It sounds to me like you don't understand the purpose of the locators.

Again: this wastes a hell lot of bandwidth, generating lots of traffic in the network each time a new block is mined.

I believe that I understand the purpose of the locators very well - and that is why my implementation always sends only one locator... which surely does not point to a genesis block, while I am already at #236475. The only reason to send a locator pointing to a genesis block, in such a case, would be to "recover" from 4+ years long lasting fork... so whoever does that, I think he doesn't understand the purpose of the locators.

[Mike Hearn]

You don't send a full locator every time a new block is mined. Nodes are supposed to send a getdata and only do a getblocks if what they get back is an orphan.

No, the increase in bandwidth wouldn't faze me. 10GB per day would be 300GB per month and I get 500 on the cheapest plan available via BitVPS, which costs hardly anything. But regardless, in future I anticipate lower bandwidth usage because nodes will become more stable. Right now a lot of regular users end up installing Bitcoin-Qt and sucking down the chain without really intending to run a full node, because they don't know about SPV clients. After we get a new MultiBit release out, I'm hoping we can point people towards that instead and the amount of chain that is served to new users will go down.

[Zeilap]

Again: this wastes a hell lot of bandwidth, generating lots of traffic in the network each time a new block is mined.

No. That's not how it works.

I believe that I understand the purpose of the locators very well - and that is why my implementation always sends only one locator

If you do this after the last checkpoint, you're in for a surprise. Good luck.

[piotr_n]

You don't send a full locator every time a new block is mined. Nodes are supposed to send a getdata and only do a getblocks if what they get back is an orphan.

Of course I don't.
But I am saying that this is what I get from a bunch of nodes running the official client, often after a new block has been mined.
I have no idea why they do it, but feel free to explain it to me..

Again: this wastes a hell lot of bandwidth, generating lots of traffic in the network each time a new block is mined.

No. That's not how it works.

Are you saying that I am sending all these screwed up "getblocks" to myself, from all the different IPs - or are you saying that I am lying about what I see?
Or maybe I'm just crazy and what I see is not real...

If you do this after the last checkpoint, you're in for a surprise. Good luck.

Thanks, but I don't use any checkpoints, so I don't think I'm going to need any luck here.

It's very simple:
1) measure a time difference between now and when you received your last block - divide it by whatever period you like, to get a number (I use minutes)
2) then go back the chain, starting from the head, as many blocks up, as the number from point 1 was
3) pass the hash of the block you have reached in point 2 to "getblocks" - and voila.
Using such a simple way, you can recover from basically any fork, as long as it isn't longer than a couple of hundreds blocks. And if a fork would happen o be longer, I will surely have enough time to develop an improvement before it's actually needed

[Mike Hearn]

They were probably on a fork or had missed a block. I don't know why else it would happen - there might be a bug, if you spot such a thing let us know.

[piotr_n]

They were probably on a fork or had missed a block. I don't know why else it would happen - there might be a bug, if you spot such a thing let us know.

You mean: let you know, even more than I have been trying to let you know so far?
Sorry, no - I think pushing it even more would be just rude and completely unproductive.

[Mike Hearn]

What I meant is, if you believe it's a bug and not intended behaviour, then feel free to investigate and figure out the cause. It's expected that remote nodes will sometimes send you block locators when new blocks are mined (if it's an orphan). I didn't see any evidence that isn't the cause here.

As we've already pointed out, it doesn't seem like many people are complaining about bandwidth usage at the moment. That's why it's not been worked on. If they were, then the best improvement available right now is full-match Bloom filters. Perhaps optimising addr/inv broadcasts might improve things too, it's a bit hard to know exactly where all the bandwidth goes right now as the software lacks instrumentation.

[piotr_n]

Oh, I just figured out what I was doing wrong handling these "getblocks" requests with 29 different locator hashes up to the genesis block.

Originally I had thought that the only limit, for building the "inv" reply, should be either reaching the hash_stop or the max 500 records.
But now I figured that I should have ignored the remaining locators after matching on a first one from the list - which almost always points to the end of my tree, so in most cases the reply should be just empty, and not 500 hashes long.
At least that's how satoshi does it..
So after fixing this in my code, the upload bandwidth usage looks indeed much better - and the whole idea of 29 locator hashes makes much more sense, as well.

Anyway, sorry - my bad.
Mike's node might be indeed in a data center