do miners zip blocks when trying to propagate them?

[adamstgBit]

anyone know?

what kind of compression do they get on a 1MB block?

[1714242618]

1714242618

[monthlyemployee1]

I don't think this would be very effective. It would only cause extra steps to need to be taken to verify them

[DannyHamilton]

Seems like a bad idea.

Some other miner could finish solving and broadcast a block during the time a miner wastes compressing the block.  Then the miner is sitting on a worthless compressed orphan block.

Seems it would be better to have a protocol that allows the miner to just broadcast the 80 byte header, then the merkle tree.

Since most peers will already have most transactions, there's no need to re-broadcast the entire megabyte worth of them (compressed or otherwise).  Then peers can just request any transactions they are missing from their immediate peers.

[adamstgBit]

Seems like a bad idea.

Some other miner could finish solving and broadcast a block during the time a miner wastes compressing the block.  Then the miner is sitting on a worthless compressed orphan block.

Seems it would be better to have a protocol that allows the miner to just broadcast the 80 byte header, then the merkle tree.

Since most peers will already have most transactions, there's no need to re-broadcast the entire megabyte worth of them (compressed or otherwise).  Then peers can just request any transactions they are missing from their immediate peers.

zipping a few MBs has got to be a less then 1 second operation no?

but just sending the header, and merkle tree, would require even less bandwidth than sending the the hole block zipped.

so there you go, bitcoin can have GB blocks without requiring miners to use a huge amount of bandwidth propagate these huge blocks

miners would simply be required to keep up with the TPS.

[Peter R]

The p2p protocol presently only supports propagation of solved blocks in full; i.e., blocks are not compressed.  

However, the Corallo Relay Network does support a sort of compression.  Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block). Greg Maxwell claims that the Corallo Relay Network attains a coding gain of about 250 (1 MB is compressed to about 4 kilobytes); however, I believe it is less in practice.  

Techniques like invertible bloom lookup tables (IBLTs) could also be used to compress solved blocks in the future; Rusty Russell is presently researching this possibility.    

Compression of solved blocks has the effect of reducing the network propagation impedance.  It results in lower fees to users (positive) but also lower fees to spammers (negative).  This chart shows how fees decline with improvements to network propagation impedance (x-axis in the chart):



Source: https://dl.dropboxusercontent.com/u/43331625/feemarket.pdf

[achow101]

I don't think blocks are easily compressed. The data is rather random since hashes are random and in order to preserve all of the data so it can be verified, it must be losslessly compressed. However, since there isn't that much of a pattern to the data, it becomes rather difficult to losslessly compress blocks with a good compression ratio.

[adamstgBit]

I don't think blocks are easily compressed. The data is rather random since hashes are random and in order to preserve all of the data so it can be verified, it must be losslessly compressed. However, since there isn't that much of a pattern to the data, it becomes rather difficult to losslessly compress blocks with a good compression ratio.

your probably right.

The p2p protocol presently only supports propagation of solved blocks in full; i.e., blocks are not compressed. 

However, the Corallo Relay Network does support a sort of compression.  Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block). Greg Maxwell claims that the Corallo Relay Network attains a coding gain of about 250 (1 MB is compressed to about 4 kilobytes); however, I believe it is less in practice. 

Techniques like invertible bloom lookup tables (IBLTs) could also be used to compress solved blocks in the future; Rusty Russell is presently researching this possibility.   

Compression of solved blocks has the effect of reducing the network propagation impedance.  It results in lower fees to users (positive) but also lower fees to spammers (negative).  This chart shows how fees decline with improvements to network propagation impedance (x-axis in the chart):



Source: https://dl.dropboxusercontent.com/u/43331625/feemarket.pdf

this "Corallo Relay Network" sounds like what bitcoin needs to scale properly. 250 coding gain, that's great!

[Peter R]

I don't think blocks are easily compressed. The data is rather random since hashes are random and in order to preserve all of the data so it can be verified, it must be losslessly compressed. However, since there isn't that much of a pattern to the data, it becomes rather difficult to losslessly compress blocks with a good compression ratio.

This would be true if the receiver (another node or miner) didn't already know a lot about the contents of the block already.  However, since most nodes have fairly similar mempools, and since block are built from transactions held in a miners mempool, this redundancy can be exploited to achieve compression.  Examples of this type of compression include the Relay Network, the IBLT proposal, or Danny's Merkle Tree idea (which I like, BTW).

[brg444]

I mean.. we're really trying to have rational discussions about scaling and security yet some of you are ignorant of some of the most basic innerworkings of Bitcoin.

Then you wonder why no one cares or value your opinion...

[adamstgBit]

I mean.. we're really trying to have rational discussions about scaling and security yet some of you are ignorant of some of the most basic innerworkings of Bitcoin.

Then you wonder why no one cares or value your opinion...

bitcoin does not currently compress blocks for propagation in anyway.

do you think we should explore this idea, or should we stick to 1MB limit because fees?


"most basic innerworkings"

is that an oxymoron?

[Peter R]

I mean.. we're really trying to have rational discussions about scaling and security yet some of you are ignorant of some of the most basic innerworkings of Bitcoin.

Then you wonder why no one cares or value your opinion...

What is wrong with people asking questions and trying to learn?

[Soros Shorts]

However, the Corallo Relay Network does support a sort of compression.  Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block).

I think a more appropriate term for this would be encoding - using codes to represent larger blocks of information that is already known by all parties. This is usually way more effective than blind data compression.

[Klestin]

I don't think blocks are easily compressed. The data is rather random since hashes are random and in order to preserve all of the data so it can be verified, it must be losslessly compressed. However, since there isn't that much of a pattern to the data, it becomes rather difficult to losslessly compress blocks with a good compression ratio.

Speaking generally, random data reliably compresses to 50% of original size.

[brg444]

I mean.. we're really trying to have rational discussions about scaling and security yet some of you are ignorant of some of the most basic innerworkings of Bitcoin.

Then you wonder why no one cares or value your opinion...

bitcoin does not currently compress blocks for propagation in anyway.

do you think we should explore this idea, or should we stick to 1MB limit because fees?


"most basic innerworkings"

is that an oxymoron?

Bitcoin does not currently propagate blocks in their full size, only the tx hashes so your idea is not in touch with how Bitcoin network currently works.

[adamstgBit]

I mean.. we're really trying to have rational discussions about scaling and security yet some of you are ignorant of some of the most basic innerworkings of Bitcoin.

Then you wonder why no one cares or value your opinion...

bitcoin does not currently compress blocks for propagation in anyway.

do you think we should explore this idea, or should we stick to 1MB limit because fees?


"most basic innerworkings"

is that an oxymoron?

Bitcoin does not currently propagate blocks in their full size, only the tx hashes so your idea is not in touch with how Bitcoin network currently works.

The p2p protocol presently only supports propagation of solved blocks in full; i.e., blocks are not compressed.  

so which is it?

[OnkelPaul]

Speaking generally, random data reliably compresses to 50% of original size.

Definitely not.
Random data does not compress. Any compressibility is a sure sign of non-randomness.

Onkel Paul

[adamstgBit]

However, the Corallo Relay Network does support a sort of compression.  Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block).

I think a more appropriate term for this would be encoding - using codes to represent larger blocks of information that is already known by all parties. This is usually way more effective than blind data compression.

if minner could communicate 100MB blocks with 250KB of encoded data, this is what will allow bitcoin to scale don't you think.

my guess is they wouldn't even need to send the full 64byte TX IDs only a 4 byte hash of the TX ID should be enoght for other minner to identify the TX's included in the new blocks.

using this method a miner could communicate a block with 250000 TX's!!! ( >400TPS ) with only 1MB of data

[brg444]

However, the Corallo Relay Network does support a sort of compression.  Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block).

I think a more appropriate term for this would be encoding - using codes to represent larger blocks of information that is already known by all parties. This is usually way more effective than blind data compression.

if minner could communicate 100MB blocks with 250KB of encoded data, this is what will allow bitcoin to scale don't you think.

my guess is they wouldn't even need to send the full 64byte TX IDs only a 4 byte hash of the TX ID should be enoght for other minner to identify the TX's included in the new blocks.

using this method a miner could communicate a block with 250000 TX's!!! ( >400TPS ) with only 1MB of data

 

What novel idea, maybe you should apply to become a core developer?

Here:

The relay network includes an optimized transmission protocol which enables sending the "entire" block typically in just a smal number of bytes (much smaller than the summaries you suggest, which still leave the participants needing to send the block).

E.g. block 000ce90846 was 999950 bytes and the relay network protocol sent it using at most 4906 bytes.

http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-August/009942.html

[adamstgBit]

However, the Corallo Relay Network does support a sort of compression.  Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block).

I think a more appropriate term for this would be encoding - using codes to represent larger blocks of information that is already known by all parties. This is usually way more effective than blind data compression.

if minner could communicate 100MB blocks with 250KB of encoded data, this is what will allow bitcoin to scale don't you think.

my guess is they wouldn't even need to send the full 64byte TX IDs only a 4 byte hash of the TX ID should be enoght for other minner to identify the TX's included in the new blocks.

using this method a miner could communicate a block with 250000 TX's!!! ( >400TPS ) with only 1MB of data

 

What novel idea, maybe you should apply to become a core developer?

Here:

The relay network includes an optimized transmission protocol which enables sending the "entire" block typically in just a smal number of bytes (much smaller than the summaries you suggest, which still leave the participants needing to send the block).

E.g. block 000ce90846 was 999950 bytes and the relay network protocol sent it using at most 4906 bytes.

http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-August/009942.html

right this method is already being used by some miners, but this isn't standard way to propagate blocks.
should it be?
is this the key to scaling bitcoin?
I think yes on both counts.
right now this method is netting miners >200X coding gain!

[brg444]

However, the Corallo Relay Network does support a sort of compression.  Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block).

I think a more appropriate term for this would be encoding - using codes to represent larger blocks of information that is already known by all parties. This is usually way more effective than blind data compression.

if minner could communicate 100MB blocks with 250KB of encoded data, this is what will allow bitcoin to scale don't you think.

my guess is they wouldn't even need to send the full 64byte TX IDs only a 4 byte hash of the TX ID should be enoght for other minner to identify the TX's included in the new blocks.

using this method a miner could communicate a block with 250000 TX's!!! ( >400TPS ) with only 1MB of data

 

What novel idea, maybe you should apply to become a core developer?

Here:

The relay network includes an optimized transmission protocol which enables sending the "entire" block typically in just a smal number of bytes (much smaller than the summaries you suggest, which still leave the participants needing to send the block).

E.g. block 000ce90846 was 999950 bytes and the relay network protocol sent it using at most 4906 bytes.

http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-August/009942.html

right this method is already being used by some most miners, but this isn't standard way to propagate blocks.
should it be?
is this the key to scaling bitcoin?
I think yes on both counts.
right now this method is netting miners >200X coding gain!

Corrected.

This doesn't solve any of the centralization or attack vectors concerns btw