do miners zip blocks when trying to propagate them?

[adamstgBit]

anyone know?

what kind of compression do they get on a 1MB block?

[1715315109]

1715315109

[1715315109]

1715315109

[monthlyemployee1]

I don't think this would be very effective. It would only cause extra steps to need to be taken to verify them

[DannyHamilton]

Seems like a bad idea.

Some other miner could finish solving and broadcast a block during the time a miner wastes compressing the block.  Then the miner is sitting on a worthless compressed orphan block.

Seems it would be better to have a protocol that allows the miner to just broadcast the 80 byte header, then the merkle tree.

Since most peers will already have most transactions, there's no need to re-broadcast the entire megabyte worth of them (compressed or otherwise).  Then peers can just request any transactions they are missing from their immediate peers.

[adamstgBit]

Seems like a bad idea.

Some other miner could finish solving and broadcast a block during the time a miner wastes compressing the block.  Then the miner is sitting on a worthless compressed orphan block.

Seems it would be better to have a protocol that allows the miner to just broadcast the 80 byte header, then the merkle tree.

Since most peers will already have most transactions, there's no need to re-broadcast the entire megabyte worth of them (compressed or otherwise).  Then peers can just request any transactions they are missing from their immediate peers.

zipping a few MBs has got to be a less then 1 second operation no?

but just sending the header, and merkle tree, would require even less bandwidth than sending the the hole block zipped.

so there you go, bitcoin can have GB blocks without requiring miners to use a huge amount of bandwidth propagate these huge blocks

miners would simply be required to keep up with the TPS.

[Peter R]

The p2p protocol presently only supports propagation of solved blocks in full; i.e., blocks are not compressed.  

However, the Corallo Relay Network does support a sort of compression.  Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block). Greg Maxwell claims that the Corallo Relay Network attains a coding gain of about 250 (1 MB is compressed to about 4 kilobytes); however, I believe it is less in practice.  

Techniques like invertible bloom lookup tables (IBLTs) could also be used to compress solved blocks in the future; Rusty Russell is presently researching this possibility.    

Compression of solved blocks has the effect of reducing the network propagation impedance.  It results in lower fees to users (positive) but also lower fees to spammers (negative).  This chart shows how fees decline with improvements to network propagation impedance (x-axis in the chart):



Source: https://dl.dropboxusercontent.com/u/43331625/feemarket.pdf

[achow101]

I don't think blocks are easily compressed. The data is rather random since hashes are random and in order to preserve all of the data so it can be verified, it must be losslessly compressed. However, since there isn't that much of a pattern to the data, it becomes rather difficult to losslessly compress blocks with a good compression ratio.

[adamstgBit]

I don't think blocks are easily compressed. The data is rather random since hashes are random and in order to preserve all of the data so it can be verified, it must be losslessly compressed. However, since there isn't that much of a pattern to the data, it becomes rather difficult to losslessly compress blocks with a good compression ratio.

your probably right.

The p2p protocol presently only supports propagation of solved blocks in full; i.e., blocks are not compressed. 

However, the Corallo Relay Network does support a sort of compression.  Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block). Greg Maxwell claims that the Corallo Relay Network attains a coding gain of about 250 (1 MB is compressed to about 4 kilobytes); however, I believe it is less in practice. 

Techniques like invertible bloom lookup tables (IBLTs) could also be used to compress solved blocks in the future; Rusty Russell is presently researching this possibility.   

Compression of solved blocks has the effect of reducing the network propagation impedance.  It results in lower fees to users (positive) but also lower fees to spammers (negative).  This chart shows how fees decline with improvements to network propagation impedance (x-axis in the chart):



Source: https://dl.dropboxusercontent.com/u/43331625/feemarket.pdf

this "Corallo Relay Network" sounds like what bitcoin needs to scale properly. 250 coding gain, that's great!

[Peter R]

I don't think blocks are easily compressed. The data is rather random since hashes are random and in order to preserve all of the data so it can be verified, it must be losslessly compressed. However, since there isn't that much of a pattern to the data, it becomes rather difficult to losslessly compress blocks with a good compression ratio.

This would be true if the receiver (another node or miner) didn't already know a lot about the contents of the block already.  However, since most nodes have fairly similar mempools, and since block are built from transactions held in a miners mempool, this redundancy can be exploited to achieve compression.  Examples of this type of compression include the Relay Network, the IBLT proposal, or Danny's Merkle Tree idea (which I like, BTW).

[brg444]

I mean.. we're really trying to have rational discussions about scaling and security yet some of you are ignorant of some of the most basic innerworkings of Bitcoin.

Then you wonder why no one cares or value your opinion...

[adamstgBit]

I mean.. we're really trying to have rational discussions about scaling and security yet some of you are ignorant of some of the most basic innerworkings of Bitcoin.

Then you wonder why no one cares or value your opinion...

bitcoin does not currently compress blocks for propagation in anyway.

do you think we should explore this idea, or should we stick to 1MB limit because fees?


"most basic innerworkings"

is that an oxymoron?

[Peter R]

I mean.. we're really trying to have rational discussions about scaling and security yet some of you are ignorant of some of the most basic innerworkings of Bitcoin.

Then you wonder why no one cares or value your opinion...

What is wrong with people asking questions and trying to learn?

[Soros Shorts]

However, the Corallo Relay Network does support a sort of compression.  Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block).

I think a more appropriate term for this would be encoding - using codes to represent larger blocks of information that is already known by all parties. This is usually way more effective than blind data compression.

[Klestin]

I don't think blocks are easily compressed. The data is rather random since hashes are random and in order to preserve all of the data so it can be verified, it must be losslessly compressed. However, since there isn't that much of a pattern to the data, it becomes rather difficult to losslessly compress blocks with a good compression ratio.

Speaking generally, random data reliably compresses to 50% of original size.

[brg444]

I mean.. we're really trying to have rational discussions about scaling and security yet some of you are ignorant of some of the most basic innerworkings of Bitcoin.

Then you wonder why no one cares or value your opinion...

bitcoin does not currently compress blocks for propagation in anyway.

do you think we should explore this idea, or should we stick to 1MB limit because fees?


"most basic innerworkings"

is that an oxymoron?

Bitcoin does not currently propagate blocks in their full size, only the tx hashes so your idea is not in touch with how Bitcoin network currently works.

[adamstgBit]

I mean.. we're really trying to have rational discussions about scaling and security yet some of you are ignorant of some of the most basic innerworkings of Bitcoin.

Then you wonder why no one cares or value your opinion...

bitcoin does not currently compress blocks for propagation in anyway.

do you think we should explore this idea, or should we stick to 1MB limit because fees?


"most basic innerworkings"

is that an oxymoron?

Bitcoin does not currently propagate blocks in their full size, only the tx hashes so your idea is not in touch with how Bitcoin network currently works.

The p2p protocol presently only supports propagation of solved blocks in full; i.e., blocks are not compressed.  

so which is it?

[OnkelPaul]

Speaking generally, random data reliably compresses to 50% of original size.

Definitely not.
Random data does not compress. Any compressibility is a sure sign of non-randomness.

Onkel Paul

[adamstgBit]

However, the Corallo Relay Network does support a sort of compression.  Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block).

I think a more appropriate term for this would be encoding - using codes to represent larger blocks of information that is already known by all parties. This is usually way more effective than blind data compression.

if minner could communicate 100MB blocks with 250KB of encoded data, this is what will allow bitcoin to scale don't you think.

my guess is they wouldn't even need to send the full 64byte TX IDs only a 4 byte hash of the TX ID should be enoght for other minner to identify the TX's included in the new blocks.

using this method a miner could communicate a block with 250000 TX's!!! ( >400TPS ) with only 1MB of data

[brg444]

However, the Corallo Relay Network does support a sort of compression.  Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block).

I think a more appropriate term for this would be encoding - using codes to represent larger blocks of information that is already known by all parties. This is usually way more effective than blind data compression.

if minner could communicate 100MB blocks with 250KB of encoded data, this is what will allow bitcoin to scale don't you think.

my guess is they wouldn't even need to send the full 64byte TX IDs only a 4 byte hash of the TX ID should be enoght for other minner to identify the TX's included in the new blocks.

using this method a miner could communicate a block with 250000 TX's!!! ( >400TPS ) with only 1MB of data

 

What novel idea, maybe you should apply to become a core developer?

Here:

The relay network includes an optimized transmission protocol which enables sending the "entire" block typically in just a smal number of bytes (much smaller than the summaries you suggest, which still leave the participants needing to send the block).

E.g. block 000ce90846 was 999950 bytes and the relay network protocol sent it using at most 4906 bytes.

http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-August/009942.html

[adamstgBit]

However, the Corallo Relay Network does support a sort of compression.  Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block).

I think a more appropriate term for this would be encoding - using codes to represent larger blocks of information that is already known by all parties. This is usually way more effective than blind data compression.

if minner could communicate 100MB blocks with 250KB of encoded data, this is what will allow bitcoin to scale don't you think.

my guess is they wouldn't even need to send the full 64byte TX IDs only a 4 byte hash of the TX ID should be enoght for other minner to identify the TX's included in the new blocks.

using this method a miner could communicate a block with 250000 TX's!!! ( >400TPS ) with only 1MB of data

 

What novel idea, maybe you should apply to become a core developer?

Here:

The relay network includes an optimized transmission protocol which enables sending the "entire" block typically in just a smal number of bytes (much smaller than the summaries you suggest, which still leave the participants needing to send the block).

E.g. block 000ce90846 was 999950 bytes and the relay network protocol sent it using at most 4906 bytes.

http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-August/009942.html

right this method is already being used by some miners, but this isn't standard way to propagate blocks.
should it be?
is this the key to scaling bitcoin?
I think yes on both counts.
right now this method is netting miners >200X coding gain!

[brg444]

However, the Corallo Relay Network does support a sort of compression.  Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block).

I think a more appropriate term for this would be encoding - using codes to represent larger blocks of information that is already known by all parties. This is usually way more effective than blind data compression.

if minner could communicate 100MB blocks with 250KB of encoded data, this is what will allow bitcoin to scale don't you think.

my guess is they wouldn't even need to send the full 64byte TX IDs only a 4 byte hash of the TX ID should be enoght for other minner to identify the TX's included in the new blocks.

using this method a miner could communicate a block with 250000 TX's!!! ( >400TPS ) with only 1MB of data

 

What novel idea, maybe you should apply to become a core developer?

Here:

The relay network includes an optimized transmission protocol which enables sending the "entire" block typically in just a smal number of bytes (much smaller than the summaries you suggest, which still leave the participants needing to send the block).

E.g. block 000ce90846 was 999950 bytes and the relay network protocol sent it using at most 4906 bytes.

http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-August/009942.html

right this method is already being used by some most miners, but this isn't standard way to propagate blocks.
should it be?
is this the key to scaling bitcoin?
I think yes on both counts.
right now this method is netting miners >200X coding gain!

Corrected.

This doesn't solve any of the centralization or attack vectors concerns btw

[adamstgBit]

However, the Corallo Relay Network does support a sort of compression.  Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block).

I think a more appropriate term for this would be encoding - using codes to represent larger blocks of information that is already known by all parties. This is usually way more effective than blind data compression.

if miner could communicate 100MB blocks with 250KB of encoded data, this is what will allow bitcoin to scale don't you think.

my guess is they wouldn't even need to send the full 64byte TX IDs only a 4 byte hash of the TX ID should be enoght for other minner to identify the TX's included in the new blocks.

using this method a miner could communicate a block with 250000 TX's!!! ( >400TPS ) with only 1MB of data

 

What novel idea, maybe you should apply to become a core developer?

Here:

The relay network includes an optimized transmission protocol which enables sending the "entire" block typically in just a smal number of bytes (much smaller than the summaries you suggest, which still leave the participants needing to send the block).

E.g. block 000ce90846 was 999950 bytes and the relay network protocol sent it using at most 4906 bytes.

http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-August/009942.html

right this method is already being used by some most miners, but this isn't standard way to propagate blocks.
should it be?
is this the key to scaling bitcoin?
I think yes on both counts.
right now this method is netting miners >200X coding gain!

Corrected.

This doesn't solve any of the centralization or attack vectors concerns btw

why?
bigger blocks will no longer mean centralization, if a 1GB block and can be sent using 4MBs!

[brg444]

However, the Corallo Relay Network does support a sort of compression.  Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block).

I think a more appropriate term for this would be encoding - using codes to represent larger blocks of information that is already known by all parties. This is usually way more effective than blind data compression.

if minner could communicate 100MB blocks with 250KB of encoded data, this is what will allow bitcoin to scale don't you think.

my guess is they wouldn't even need to send the full 64byte TX IDs only a 4 byte hash of the TX ID should be enoght for other minner to identify the TX's included in the new blocks.

using this method a miner could communicate a block with 250000 TX's!!! ( >400TPS ) with only 1MB of data

 

What novel idea, maybe you should apply to become a core developer?

Here:

The relay network includes an optimized transmission protocol which enables sending the "entire" block typically in just a smal number of bytes (much smaller than the summaries you suggest, which still leave the participants needing to send the block).

E.g. block 000ce90846 was 999950 bytes and the relay network protocol sent it using at most 4906 bytes.

http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-August/009942.html

right this method is already being used by some most miners, but this isn't standard way to propagate blocks.
should it be?
is this the key to scaling bitcoin?
I think yes on both counts.
right now this method is netting miners >200X coding gain!

Corrected.

This doesn't solve any of the centralization or attack vectors concerns btw

why?
bigger blocks will no longer mean centralization, if a 1GB block and can be sent using 4MBs!

 

Did you forget about the nodes? 1 GB blocks ? 

Seriously, this stuff is just beyond your understanding, just stop it. 

[adamstgBit]

However, the Corallo Relay Network does support a sort of compression.  Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block).

I think a more appropriate term for this would be encoding - using codes to represent larger blocks of information that is already known by all parties. This is usually way more effective than blind data compression.

if minner could communicate 100MB blocks with 250KB of encoded data, this is what will allow bitcoin to scale don't you think.

my guess is they wouldn't even need to send the full 64byte TX IDs only a 4 byte hash of the TX ID should be enoght for other minner to identify the TX's included in the new blocks.

using this method a miner could communicate a block with 250000 TX's!!! ( >400TPS ) with only 1MB of data

 

What novel idea, maybe you should apply to become a core developer?

Here:

The relay network includes an optimized transmission protocol which enables sending the "entire" block typically in just a smal number of bytes (much smaller than the summaries you suggest, which still leave the participants needing to send the block).

E.g. block 000ce90846 was 999950 bytes and the relay network protocol sent it using at most 4906 bytes.

http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-August/009942.html

right this method is already being used by some most miners, but this isn't standard way to propagate blocks.
should it be?
is this the key to scaling bitcoin?
I think yes on both counts.
right now this method is netting miners >200X coding gain!

Corrected.

This doesn't solve any of the centralization or attack vectors concerns btw

why?
bigger blocks will no longer mean centralization, if a 1GB block and can be sent using 4MBs!

 

Did you forget about the nodes? 1 GB blocks ?  

Seriously, this stuff is just beyond your understanding, just stop it.  

node can do the same damn thing....

1GB is pushing the limits i agree

but its totally reasonable to expect a typical home computer to be able to stream 100MB of TX within 10mins.

at 1MBps you can stream 600MB of data every 10mins

miners/full nodes would simply be expected to be able to keep up with the TPS happening on the network.

and another optimization would be that miners don't include TX's that are likely not to have fully propagate through the network yet.

wtf is wrong with that?

[Peter R]

Did you forget about the nodes? 1 GB blocks ?  

Seriously, this stuff is just beyond your understanding, just stop it.  

He seems to understand it better than you, and is more humble with what he claims to know too.  For example, you earlier claimed that the

"Bitcoin does not currently propagate blocks in their full size, only the tx hashes so your idea is not in touch with how Bitcoin network currently works."

which is clearly wrong; the Bitcoin P2P protocol supports the propagation of complete (non-encoded) blocks between nodes/miners.  

Some miners use a centralized service called the Relay Network to improve their propagation rate and reduce their orphaning risk. However, this is not Bitcoin; the members of the Relay Network can be thought of as an Industrial Trade Group less centralized than a mining pool but more centralized than an independent miner operating on the P2P network.

Furthermore, the validity of your claim that most miners use the Relay Network depends on the definition of "most."  The empirical network propagation impedance is closer to what you would expect if no miners used the Relay Network than if all miners used the relay network.

[CIYAM]

Firstly zipping is unlikely to reduce the size much at all (as was pointed out) due to most of it being random bytes.

Next I would like to point out that unfortunately, when you use the internet in China, speeds for transmitting (or receiving) data outside of China is actually often much like that in the early 1990's (with home-grade internet at least and even with business-grade it is very slow compared to data transmission between western countries) and the majority of hash-power is currently located in China.

This bandwidth issue is why so-called SPV mining has already been happening in China (which I understand was what had led to the most recent fork that occurred). So even if the block size can be reduced by only including tx hashes the problem will be that some (or maybe many) of the miners in China might still have a problem in keeping up with the TPS (which will likely result in blocks with few txs or perhaps only the coinbase tx).

There is no rule in Bitcoin that says you need to include any tx beyond the "coinbase" (or block reward) and we have seen at times new blocks appearing with no other txs (wasting the potential TPS dramatically). So if the network bandwidth is being pushed too far don't be surprised to start seeing blocks with only one tx in them which would actually mean supporting much larger blocks might simply fail to get txs even included at all.

[brg444]

Did you forget about the nodes? 1 GB blocks ?  

Seriously, this stuff is just beyond your understanding, just stop it.  

He seems to understand it better than you, and is more humble with what he claims to know too.  For example, you earlier claimed that the

"Bitcoin does not currently propagate blocks in their full size, only the tx hashes so your idea is not in touch with how Bitcoin network currently works."

which is clearly wrong; the Bitcoin P2P protocol supports the propagation of complete (non-encoded) blocks between nodes/miners.  

Some miners use a centralized service called the Relay Network to improve their propagation rate and reduce their orphaning risk. However, this is not Bitcoin; the members of the Relay Network can be thought of as an Industrial Trade Group less centralized than a mining pool but more centralized than an independent miner operating on the P2P network.

Furthermore, the validity of your claim that most miners use the Relay Network depends on the definition of "most."  The empirical network propagation impedance is closer to what you would expect if no miners used the Relay Network than if all miners used the relay network.

Sure, let's use empirical notions and ignore reality 

[adamstgBit]

Firstly zipping is unlikely to reduce the size much at all (as was pointed out) due to most of it being random bytes.

Next I would like to point out that unfortunately when you use the internet in China speeds for transmitting (or receiving) data outside of China is actually often much like that in the early 1990's (with home-grade internet at least and even with business-grade it is very slow compared to data transmission between western countries) and the majority of hash-power is currently located in China.

This bandwidth issue is why so-called SPV mining has already been happening in China (which I understand was what had led to the most recent fork that occurred). So even if the block size can be reduced by only including tx hashes the problem will be that some (or maybe many) of the miners in China might still have a problem in keeping up with the TPS (which will likely result in blocks with few txs or perhaps only the coinbase tx).

chinese miners should setup a node somewhere with a better internet connection that will handle the validation while they hash away in chain. how hard is it for a chinese solo miner to run some software in the cloud? does his full node need to be next to him for it to be considered under his control?

"SPV mining has already been happening in China "

Good! problem solved.

[CIYAM]

chinese miners should setup a node somewhere with a better internet connection that will handle the validation while they hash away in chain. how hard is it for a chinese solo miner to run some software in the cloud? does his full node need to be next to him for it to be considered under his control?

Except that it is simply not possible to do this (unless perhaps you own your own internet infrastructure that has fibre cables between China and the outside world and which is not subject to government control and I've certainly not heard of such a thing).

Setting up a VPN or whatever method you want to use to get around the Great Chinese Firewall will stop things like website access restrictions but you can't do anything about the bandwidth restrictions (i.e. you are slowed down to the 1990s).

So it is quite easy to get around blocks to YouTube but forget about trying to watch a HD video.

[brg444]

Did you forget about the nodes? 1 GB blocks ?  

Seriously, this stuff is just beyond your understanding, just stop it.  

He seems to understand it better than you, and is more humble with what he claims to know too.  For example, you earlier claimed that the

"Bitcoin does not currently propagate blocks in their full size, only the tx hashes so your idea is not in touch with how Bitcoin network currently works."

which is clearly wrong; the Bitcoin P2P protocol supports the propagation of complete (non-encoded) blocks between nodes/miners.  

Let me rephrase it then:

Bitcoin, as it is implemented right now by most if not all miners, does not currently propagate blocks in their full size, only the tx hashes so your idea is not in touch with how Bitcoin network currently works.

[adamstgBit]

chinese miners should setup a node somewhere with a better internet connection that will handle the validation while they hash away in chain. how hard is it for a chinese solo miner to run some software in the cloud? does his full node need to be next to him for it to be considered under his control?

Except that it is simply not possible to do this (unless perhaps you own your own internet infrastructure that has fibre cables between China and the outside world and is not subject to government control).

Setting up a VPN or whatever method you want to use to get around the Great Chinese Firewall will stop things like website access restrictions but you can't do anything about the bandwidth restrictions (i.e. you are slowed down to the 1990s).

ya but they don't need any bandwidth  if their full node is running on a US server.

[CIYAM]

ya but they don't need any bandwidth  if their full node is running on a US server.

The point is that their mining operations are in China so having a server running in the US is of zero use.

You have to get the txs sent to China (it doesn't matter whether you are using a VPN or whatever as I pointed out).

Again - to make the point clear - it is easy to get around the blocks to YouTube but you can't watch HD videos because the bandwidth to the "outside world" is just slowed down too much (on purpose).

The US is just too expensive in terms of electricity costs so the Chinese mining operations are not likely to want to set up there.

[adamstgBit]

ya but they don't need any bandwidth  if their full node is running on a US server.

The point is that their mining operations are in China so having a server running in the US is of zero use.

You have to get the txs sent to China (it doesn't matter whether you are using a VPN or whatever as I pointed out).

the use would be that there node in US does all the validation work and there mining farm dose only hashing and sends the answer to the full node when they solve the problem so that the node in US can make a block.

you said it yourself there already doing this,  "SPV mining".

also is chain planning on having the worlds shityest interent forever?

[adamstgBit]

Did you forget about the nodes? 1 GB blocks ?  

Seriously, this stuff is just beyond your understanding, just stop it.  

He seems to understand it better than you, and is more humble with what he claims to know too.  For example, you earlier claimed that the

"Bitcoin does not currently propagate blocks in their full size, only the tx hashes so your idea is not in touch with how Bitcoin network currently works."

which is clearly wrong; the Bitcoin P2P protocol supports the propagation of complete (non-encoded) blocks between nodes/miners.  

Let me rephrase it then:

Bitcoin, as it is implemented right now by most if not all miners, does not currently propagate blocks in their full size, only the tx hashes so your idea is not in touch with how Bitcoin network currently works.

the way blocks are propagated wouldn't need a hardfork then?

GOOD.

[CIYAM]

you said it yourself there already doing this,  "SPV mining".

Okay - I get what you mean now and that has some merit (although I guess it is an extra hassle for them to set that up so there is the question of whether they would do so but perhaps enough fees earned per block might make that worthwhile).

I don't think that SPV mining (at least how it is being done currently) works that way though as my understanding was that there was no proper block validation being done at all (which is why the fork happened).

And yes how miners propagate blocks (well block headers actually) to their hashers is not really relevant to the main protocol at all (which is why things like SPV mining already occurs).

The internet within China itself is actually very fast as my wife and I can both watch two different HD movies (that are sourced from within China) at the same time from our fairly standard home connection (hint: the issue with communications outside of China is not really a technical one).

[brg444]

What you are attempting to describe is the relay network:

http://bitcoinrelaynetwork.org/

[Peter R]

What you are attempting to describe is the relay network:

http://bitcoinrelaynetwork.org/

One problem with the Relay Network is that it is a centralized service.  I would like to see coding gain for block solutions implemented at the P2P level.  For example, sometimes the Relay Network stops working; if the encoding is done at the P2P level, this wouldn't happen. 

[CIYAM]

What you are attempting to describe is the relay network:

http://bitcoinrelaynetwork.org/

What I had read in terms of the recent fork had mentioned SPV mining so if that *is* the relay network then yes the same thing (and yes reading that link does look to explain why the fork happened in terms of minimal validation being performed).

@Peter R - the link points out that it is not centralised so what makes you say that it is (or do you mean just that the number of nodes is very limited)?

[adamstgBit]

What you are attempting to describe is the relay network:

http://bitcoinrelaynetwork.org/

What I had read in terms of the recent fork had mentioned SPV mining so if that *is* the relay network then yes the same thing.

@Peter R - the link points out that it is not centralised so what makes you say that?

it's sorta centralized. a few nodes all under 1 dudes control....  it could be decentralized, simply by making this the way peers communicate new blocks.
SPV mining and this  relay network  are different things.
this is just a more efficient way of communicating the contents of a new block.
SPV mining is a way of mining without handling block validation so you don't have to download any blocks at all.

[Peter R]

@Peter R - the link points out that it is not centralised so what makes you say that it is (or do you mean just that the number of nodes is very limited)?

Anyone can presently join the Relay Network; however, because it is not implemented as a P2P protocol in the same way that Bitcoin is, sometimes it "breaks."  If you read through the Relay Network thread, you can confirm for yourself that this is empirically true: https://bitcointalk.org/index.php?topic=766190.0

[CIYAM]

Okay - thanks for the info - so SPV mining is not the same as the Relay Network?

So do we know exactly what software was being run by the pools that caused the fork?

[Kprawn]

Is the internet data exiting and entering China being filtered via Firewalls at data centers, where the cables join with the mainland? Is this why the internet speed within the

country is so low? Like working through a online proxy or VPN service? I saw a video once, where a Chinese guy were playing Starcraft and his response time was pretty fast.

His opponent was from the same country, but from a different location within China. I guess the Chinese have a point, when they complain about the West having the advantage

of unrestricted fast internet access, when they have to compete from their country, through these firewalls.      

[adamstgBit]

Okay - thanks for the info - so SPV mining is not the same as the Relay Network?

So do we know exactly what software was being run by the pools that caused the fork?

IDK, a faulty SPV written by the chinese miners?

I once had to reverse engineer a piece of code written by chinese poeple, their code was unbelievable, pure spaghetti! goto's and all.

[CIYAM]

Is the internet data exiting and entering China being filtered via Firewalls at data centers, where the cables join with the mainland?

Slowed down rather than filtered at the lower levels (so even if you get past the firewall you are back in the 1990s).

Is this why the internet speed within the country is so low?

Within China itself the speed is very fast (as I mentioned my wife and I can both watch a different HD movie at the same time with no problem here).

[brg444]

Okay - thanks for the info - so SPV mining is not the same as the Relay Network?

So do we know exactly what software was being run by the pools that caused the fork?

Not at all.

Here:
http://bitcoin.stackexchange.com/questions/38437/what-is-spv-mining-and-how-did-it-inadvertently-cause-the-fork-after-bip66-wa
https://www.reddit.com/r/Bitcoin/comments/3c305f/if_you_are_using_any_wallet_other_than_bitcoin/csrsrf9

[CIYAM]

I once had to reverse engineer a piece of code written by chinese poeple, their code was unbelievable, pure spaghetti! goto's and all.

There are goto's in Bitcoin's code written by some Americans - so go figure - people all around the world can write spaghetti and use goto statements.

Also reverse engineering C or C++ won't produce anything readable at all (as it is compiled).

[adamstgBit]

I once had to reverse engineer a piece of code written by chinese poeple, their code was unbelievable, pure spaghetti! goto's and all.

There are goto's in Bitcoin's code written by some Americans - so go figure - people all around the world can write spaghetti and use goto statements.

Also reverse engineering C or C++ won't produce anything readable at all (as it is compiled).

they gave me their source, we work together. ( maybe reverse engineer isn't the right thing to say, i needed to see how they handled certain things so my code could adhere to the same rules, which their code defined. )
well i had seen spaghetti code before, but this took it to a new level.

[adamstgBit]

so there you have it we can easily scale bitcoin to 4,000TPS while making sure typical home internet can handle mining operations.

no side chains.
no 3rd party payment processors.

call the devs, tell them can bump block limit to 1GB  

[uxgpf]

I mean.. we're really trying to have rational discussions about scaling and security yet some of you are ignorant of some of the most basic innerworkings of Bitcoin.

Then you wonder why no one cares or value your opinion...

What is wrong with people asking questions and trying to learn?

They might form informed opinions of their own, which in turn would reduce effectiveness of trolling.

[btc_enigma]

Really interesting ... !  There is some redundancy in a block , you easily see multiple txs referring to same txid as input ... saw https://gappleto97.github.io/visualizer/ today and looks like there are lots of cluster.  Not sure how much that would help to reduce size

[johnyj]

That invertible bloom look-up table sounds promising, but would take some time to implement, especially need to have a consensus first