A bit of criticism on how the bitcoin client does it

[piotr_n]

And the "block propagation" is eating up a hell lot of the poor's bitcoins users bandwidth - it might not be a problem for you, but it is a problem.

If you don't have enough bandwidth to be CPU limited, stop trying to run a node. SPV clients are just fine for any users needs unless you want to run a mining pool or maybe operate a big business. If you really want, go get a VPS server; $20-$100/month should buy a fast enough one, at least for another year or two.

so your advise is: don't run a bitcoin node.
?

because, you know, I would actually like to run a bitcoin node, just to support this fine network, but the current protocol is wasting my bandwidth - and that is my issue.
though, I understand that an unnecessary bandwidth usage is not something that is easy to be noticed, so I am not even surprised that nobody gives a shit about it

[1715193002]

1715193002

[1715193002]

1715193002

[Mike Hearn]

You're just confused, sorry. You have to download all data in every block to run a full node. This is fundamental. You can't reduce bandwidth usage by downloading parts of each block from different peers. This might reduce the upload bandwidth on their side, but it doesn't reduce the download bandwidth on your side. If you're talking about hosting blocks on Cloudfare then you're talking about download bandwidth. So your proposed change wouldn't impact anything.

[2112]

You're just confused, sorry. You have to download all data in every block to run a full node. This is fundamental. You can't reduce bandwidth usage by downloading parts of each block from different peers. This might reduce the upload bandwidth on their side, but it doesn't reduce the download bandwidth on your side. If you're talking about hosting blocks on Cloudfare then you're talking about download bandwidth. So your proposed change wouldn't impact anything.

The above is just another example how Mike Hearn spreads misinformation about Bitcoin protocol.

For normal operation of the Bitcoin network the majority of the "block" has already been previously transferred as a separate "transactions".

The obvious optimization of the bandwidth usage is for the clients to inspect the Merkle tree and ask the peer only for the transactions that weren't previously broadcast.

This gives close to 50% bandwidth savings for free.

I'm writing this to underline the fact that whenver you read something written by Mike Hearn you have to double check and verify for yourself the 3 broad possibilities:

1) he was correct;
2) he was incorrect because he didn't understand the question or doesn't understand the underlying tehnology;
3) he was incorrect intentionally to disparage other developers and spread misinformation.

Thankfully for now he still doesn't belong to the core development team.

[jgarzik]

You're just confused, sorry. You have to download all data in every block to run a full node. This is fundamental. You can't reduce bandwidth usage by downloading parts of each block from different peers. This might reduce the upload bandwidth on their side, but it doesn't reduce the download bandwidth on your side. If you're talking about hosting blocks on Cloudfare then you're talking about download bandwidth. So your proposed change wouldn't impact anything.

The above is just another example how Mike Hearn spreads misinformation about Bitcoin protocol.

For normal operation of the Bitcoin network the majority of the "block" has already been previously transferred as a separate "transactions".

This is true.  The current bitcoind client uses this knowledge in a signature cache, to avoid validating signatures twice (once upon TX reception, once upon block reception).

The obvious optimization of the bandwidth usage is for the clients to inspect the Merkle tree and ask the peer only for the transactions that weren't previously broadcast.

This gives close to 50% bandwidth savings for free.

heh, nothing is free.  This proposal would add additional round-trips with associated latency, slowing block validation and block propagation.

As such, miners could lose their 25 BTC due to orphaning, if their block is slowed.

[2112]

heh, nothing is free.  This proposal would add additional round-trips with associated latency, slowing block validation and block propagation.

As such, miners could lose their 25 BTC due to orphaning, if their block is slowed.

Actually you are wrong. The average block propagation latency would improve, both in case of honest miners and secretive miners.

Firstly, let me explain honest/secretive miner distinction. Honest miner broadcasts all the transactions that he's going to put in the block ahead of time. Then when he wins the block he sends all the transactions once again just to comply with the classic/inefficient block propagation protocol, wasting up to 50% of overall bandwidth.

On the other hand secretive miner tries to play games to disadvantage other miners and raise their orphan rate. He omits broadcasting private transactions separately and when he wins the block he sends those private transactions for the first time. It saves his bandwidth, but disadvantages the competing miners because they see the transactions late and have to spend time to verify them.

So for the network of mostly honest miners there simply isn't any additional round trips and the win is 100% clear. Sending just the block header and the Merkle tree is a no-brainer.

To understand why even the secretive miners case is a net win for the global network is somewhat more complicated. Firstly, the "round-trips" should be singular: only one additional round trip is required to ask for the private transactions of the uncooperative miner.

Secondly, you'll have to understand how the bandwidth is getting sold nowadays. It is very rare to actually linearly limit the bandwith sold to eg. 1ms per bit for 1 Megabit per second. Nearly every modern IP transport uses some sort of statistical limiter/multiplexer. When you buy e.g. 10Mbits per second your effective bandwidth may be 1Gigabit per second for the first, say 4kilobytes and the statistical multiplexer/limiter will throttle your remaining kilobytes to maintain the long-term average. When asked some ISPs will plainly state the setting for e.g. "fair-queue" command in Cisco IOS. Most ISPs will however consider those setting a trade secret; and frequently they change them according to the time-of-day, day-of-the-week or even to quell the packet storms after "network events".

The interested reader can either read up about the above: e.g. it is called PowerBoost with the DOCSIS cable modems. Anyone with two machines, two GPS receivers to NTP-sync the time on them and a copy of Wireshark can verify what I wrote above running their own experiments and as a side effect reverse-engineer the setting used by their ISP.

To quickly summarize the above two paragraphs: the latency to send e.g. first 4kB of 1MB block is much less than 4/1000 of the latency of the whole block.

I understand that implementing the changes suggested by piotr_n isn't trivial. But if implemented they will give at least two benefits to the global Bitcoin network:

1) information theoretic benefit of saving the bandwith, reducing the block propagation latency and thus reducing the orphan rate

2) motivate more miners to be honest about broadcasting transactions. This will probably have further domino-effect of improving overall game-theorethic strength of the Bitcoin network and may allow in the future to detect some attacks which we even haven't thought through.

In summary I wanted to stress that I enjoyed this brief, but honest exchange of technical arguments with you. Thanks.

[kjj]

You're just confused, sorry. You have to download all data in every block to run a full node. This is fundamental. You can't reduce bandwidth usage by downloading parts of each block from different peers. This might reduce the upload bandwidth on their side, but it doesn't reduce the download bandwidth on your side. If you're talking about hosting blocks on Cloudfare then you're talking about download bandwidth. So your proposed change wouldn't impact anything.

The above is just another example how Mike Hearn spreads misinformation about Bitcoin protocol.

For normal operation of the Bitcoin network the majority of the "block" has already been previously transferred as a separate "transactions".

The obvious optimization of the bandwidth usage is for the clients to inspect the Merkle tree and ask the peer only for the transactions that weren't previously broadcast.

This gives close to 50% bandwidth savings for free.

I'm writing this to underline the fact that whenver you read something written by Mike Hearn you have to double check and verify for yourself the 3 broad possibilities:

1) he was correct;
2) he was incorrect because he didn't understand the question or doesn't understand the underlying tehnology;
3) he was incorrect intentionally to disparage other developers and spread misinformation.

Thankfully for now he still doesn't belong to the core development team.

You forgot #4, technically correct, about something pointless and intentionally ignoring the distinction to give the illusion of superiority.  Odd that you'd forget it, since your post was a perfect example.

Your memory pool won't help you during the initial download, which is the time when people care about traffic.  No one gives a shit about 8 gigs over 4 years.  Everyone cares about 8 gigs today.  Using a piecewise downloading system will save some traffic* for people that don't give a shit about traffic, and save not even a single byte for people that do.

*  "close to 50%" savings needs a lot of assumptions to be true at the same time, when some of them are never true at all.  A partial list:  bitcoin traffic is evenly divided between block bodies and transactions, the list of transactions in a block can be transmitted for free, every node knows about every transaction.

[piotr_n]

I have been looking at BIT37 and it seems that "merkleblock" is exactly what I need in order to divide a new block's download into small chunks and then distribute the block's download among different peers, using a bunch of "getdata 1" instead of one "getdata 2".

The only problem I see is that:

If no filter has been set on the connection, a request for filtered blocks is ignored

So I guess I will need to setup some dummy filter first, just to be able to receive this nice and useful messages.
But... now I wonder: if I setup such a filter, will I still be receiving invs, for freshly mined blocks...? Or what will be other side effects...

I will appreciate any advise here - I really want to implement it. Preferably today

[Pieter Wuille]

I have been looking at BIT37 and it seems that "merkleblock" is exactly what I need in order to divide a new block's download into small chunks and then distribute the block's download among different peers, using a bunch of "getdata 1" instead of one "getdata 2".

I don't think you can perfectly partition blocks using filtered blocks, as there are several matching criteria, and one match is enough to include a transaction. So if you send disjunct Bloom filters (nHashes=1, and disjunct bitsets to each peer), you'll still get doubles. It is perhaps an interesting extension to BIP37 to support such partitioning, especially for larger blocks.

However, one case that is explicitly (and intentionally) supported by BIP37 is requesting blocks (full-node wise) with the transactions already known to the peer filtered out. So contrary to what Mike says, I'm pretty sure you can have a full node that uses BIP37 to fetch blocks, and save download bandwidth using that.

[piotr_n]

I see what you mean now.
I can trick the node into sending me "merkleblock" messages, but they are always followed by a bunch of "tx" messages, which completely ruins my concept of distributing requests for these "tx" among all the connected peers.

But if I make a long bloom filter, then it should never match and so I should be only getting "merkleblock" not followed by any "tx"...
Haven't given up yet

[piotr_n]

Oh, I'm stupid. I forgot that "tx" only work for a not yet mined transactions, so how did I want to acquire a block's payload with them?

Now I have given up

[Pieter Wuille]

But if I make a long bloom filter, then it should never match and so I should be only getting "merkleblock" not followed by any "tx"...
Haven't given up yet

If you don't want any transactions at all, just use getheaders.

[piotr_n]

But if I make a long bloom filter, then it should never match and so I should be only getting "merkleblock" not followed by any "tx"...
Haven't given up yet

If you don't want any transactions at all, just use getheaders.

Yes, this I know, thanks.
But what I wanted to achieve was downloading a block's payload in fragments (from different peers in parallel), using the current protocol; "merkleblock" followed by a bunch of "getdata 1"
And for a moment (or rather for the entire morning) I though that it would be possible...

[Pieter Wuille]

But what I wanted to achieve was downloading a block's payload in fragments (from different peers in parallel), using the current protocol; "merkleblock" followed by a bunch of "getdata 1"
And for a moment (or rather for the entire morning) I though that it would be possible...

I explained in #28 why that is not possible: even with disjunct filters, you'll get transactions matched by both.

[Pieter Wuille]

Oh, I think I see your confusion.

The only way to request blocks is using getdata block or merkleblock, you can't just fetch individual transactions from it, as that would require the peer to have full transaction index. So what you hoped to do was send a merkleblock request to one peer, but without transactions, and then fetch the transactions themself from separate peers. That won't work.

Anyway, as said, in the future this may become a useful idea for an extension to the filtering protocol: adding a "only match the 3rd 1/5 of all transactions" to the filter specification.

[piotr_n]

But what I wanted to achieve was downloading a block's payload in fragments (from different peers in parallel), using the current protocol; "merkleblock" followed by a bunch of "getdata 1"
And for a moment (or rather for the entire morning) I though that it would be possible...

I explained in #28 why that is not possible: even with disjunct filters, you'll get transactions matched by both.

I was assuming that all I needed would be a list of hashes returned by "merkleblock" - because this command seems to be returning all the transaction hashes for a requested block, without any filtering.

BTW, the spec on the wiki is different from the actual format of this message.
There is an extra var_length field between "total_transactions" and "hashes", carrying the same value as "total_transactions".

[piotr_n]

you can't just fetch individual transactions from it, as that would require the peer to have full transaction index.

Exactly - and not that I did not know that...  it just somehow slipped my mind

[Mike Hearn]

What I wrote was correct - you have to download all data in every block. Yes, you don't have to download it twice in the steady state if you set a full match Bloom filter (which Matt short circuited already), but as kjj notes that isn't solving a problem that's important today so it was never finished. It might be useful to optimise block propagation amongst miners, but if you aren't mining it wouldn't have much impact.

I still don't understand what piotr_n is trying to do. Downloading block contents in parallel only helps if remote peers are upload constrained and you are network constrained. That isn't the case for any real nodes today, so why are you trying to optimise it? It won't reduce bandwidth usage one bit.

Edit: OK, I reread the thread and now I see what Piotr is getting at. You want to minimize upload bandwidth for yourself running a full node, not minimize download bandwidth. The references to requesting blocks from different peers made me think you wanted to optimise download. May I ask if this is a real problem for you today or just theoretical? The node I run uploaded around 2.7 gigabytes yesterday, spread out over the full 24 hours. I certainly wouldn't want to run this off my mobile connection but it's not a bank breaker for a node run in a datacenter. Given that there are 144 blocks per day and each one yesterday was less than half a megabyte, even downloaded twice for each node that's downloading from me that's only 144 megabytes of transaction data. If we suppose most bandwidth usage is tx data then perhaps I distributed data to 19 nodes for the day - not bad, I could have supported much more even on a budget VPS.

The problem is just sharding block download doesn't change the aggregate steady state bandwidth usage for the network. Finishing the full match Bloom filter work would, but Piotr already said he isn't going to do anything with bitcoind, just on his own Go node. Well, as most nodes today are Satoshi nodes and I doubt that will change, just implementing full-match filters in the Go node won't change upload bandwidth because remote peers will still request full blocks with redundant tx data in them. To reduce upload bandwidth you have to optimise the other end.

[piotr_n]

May I ask if this is a real problem for you today or just theoretical? The node I run uploaded around 2.7 gigabytes yesterday, spread out over the full 24 hours. I certainly wouldn't want to run this off my mobile connection but it's not a bank breaker for a node run in a datacenter.

The problem is real - trust me. Not everyone lives in a datacenter, some people just have homes, you know.

As for my measurements 2.7 GB per day seems fairly low (its 32KB/s in average), so I tend to disbelieve that you actually run your node from a datacenter.
Unless you only have 8 outgoing connections, in which case that number would make more sense to me.

But anyway: lets say that it is 2.7GB per day now - while the blocks are still far below the 1MB limit. But they will obviously be growing, like they have in the past. So expect 10 GB /day pretty soon. Still makes no impression?

[piotr_n]

Why to ask 500 blocks back?

It doesn't, as far as I know. It asks for "up 500 blocks starting at hash X", where X is the last known block.

I have just checked it again, putting a debug into my getblocks handler.
Each time my node receives "getblocks", there are like tens of different locator hashes, followed by a zero-filled stop.
This (from what I see) always forces my node to return the maximum of 500 blocks.
For instance:

Code:

getblocks  hashes_inside=29  => returned 500 invs / 18003 bytes

and these are the locator hashes that were inside (note the genesis hash at the end):

Code:

00000000000000b1ffa04c08a1372309ee0f38d99a6ea7ac3d98c6bc76747b4d
00000000000000ebddd1d499c21ffe41a99c2fbefe809aecc1e31b7f5eb2a39d
000000000000003372bd8cb3d9cf78b86a0f64b2c30f8c0ce1f4e46f983d091c
000000000000017c700b2179c55343830a5ff07041d2155654c8ba77ef00677f
00000000000001365066267c0084347b180df7070ed112d2cac316c6565ccd2b
0000000000000013b9efa409b459baf11b4ccb43a60f853bdd8d25fb9b867ec3
00000000000000b8e0d6f2f5437cb2e03e973eac6653b1e2ee7c4bfd822ae1f5
000000000000002031feb915bfb85bfd138a0c9abaa53d3d577e022ca01ddacf
00000000000000a15f169467fbbab6bcf9ce7359fce414215e353b31aa7f17bc
00000000000000b705393b66eb3adf92e69cbbb04753357955b5b8dbaf9d273e
0000000000000127933cc0ad1e6a23a852bd391daa843e457ab40d5c226fed38
000000000000009d39a568e0bf5984fe5f3e9882813b686651b691bdfa07ea39
0000000000000142ad32a203b1627bee8126fa4bcd940b0da3f32bf1b5b07a24
000000000000001aef6e9c1ff8e5833c5f38bb1b660d3a3060169f10ba1a293f
00000000000000f5710fe34e22a0791afe7a817044f9c7da7a139295da995b06
0000000000000168cb9711469ccfa6704398c82fca5dd42bb700f11970946be1
00000000000000d9955f9fe2161524121f0e21f3e2b1a57f97578c98dc31d636
000000000000011f76c57b0f941acf03da10cce12cb989ed5c4b7b6722b89cc4
00000000000000d8c733f7bc4295a8af6e0b95f2bafc901076444505572353c1
000000000000009be2ca7e5f878c2235ac7b2ebf653d77c9d4d57b359e071ca2
000000000000014158e599c40a2e3fef433cf70cc1caedd9099c21028e4d97f7
00000000000000f8cc33dbc8c3c37f88598f33373fbb03b96a1eaf6426da7898
0000000000000008c3e7b684e2aa56bebba3beb267a8d932fb84217becc0406b
0000000000000057c46a1306febe518cdb6c1fc882d856bea07e5927a5f15dc3
0000000000000381683c0b95b17233f8683fb89060d6e4eb0d70f96bd51a539f
0000000000000201892dd6ed76236eb8f42e93bf0dad2606cc5aaa246f253bf1
00000000000003d8e22ddc9a6b3b2226dfe57830d793be012720e74db28bc6be
000000000001dbc957fbe83e9d38aed40c9e083b830c36890d538726b24ae1d0
000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f

Actually, if you look at this article it even clearly advises:

To create the block locator hashes, keep pushing hashes until you go back to the genesis block. After pushing 10 hashes back, the step backwards doubles every loop

Yeah, you are only at block 236k+, so just keep pushing all the hashes, starting from the top, until you reach the genesis block - a brilliant idea

[Zeilap]

Actually, if you look at this article it even clearly advises:

To create the block locator hashes, keep pushing hashes until you go back to the genesis block. After pushing 10 hashes back, the step backwards doubles every loop

Yeah, you are only at block 236k+, so just keep pushing all the hashes, starting from the top, until you reach the genesis block - a brilliant idea

What's wrong with this? It sounds to me like you don't understand the purpose of the locators.