Bitcoin Forum
February 06, 2023, 04:00:58 PM *
News: Latest Bitcoin Core release: 24.0.1 [Torrent]
 
   Home   Help Search Login Register More  
Pages: « 1 [2] 3 »  All
  Print  
Author Topic: BlockReduce: Scaling Blockchain to human commerce  (Read 1173 times)
mechanikalk (OP)
Member
**
Offline Offline

Activity: 91
Merit: 72


View Profile WWW
November 23, 2018, 07:16:20 PM
Merited by Welsh (2)
 #21

Looking closer to your calculations it is about comparing bitcoin transaction propagation with your schema's block reconciliation. These are two basically different things (and yet you are exaggerating about the situation with bitcoin).

It is not comparing propagation to block reconciliation.  I understand how this could be confused. The  blocks found at the lowest level are the mechanism for sharing groups of transactions.  That is why I am comparing Bitcoins transaction propagation to BlockReduce Zone block propagation.  Propogating transactions in groups in a whisper protocol is much more bandwidth efficient.

BlockReduce will be as inefficient as Bitcoin at the lowest level zone groups.  The efficiency is gained by the aggregation of transactions into groups via "Zone blocks" and whispering them further in the network as blocks of transactions.

As I've argued before, both bitcoin p2p and your schema need the same amount of effort and resources to be consumed for raw transaction propagation because they are full nodes and you just can't dictate a predefined topology like a central authority or something.

The structure for grouping and propagating transactions would not be dictated, but rather would be incentivized.  Miners would be incentivized to find lower level blocks and the impact of network latency on effectively using their hash power would incentivize them to find low latency zones to mine in.  This would cause each zone group to have lower latency then the total network and be able to process a higher TPS.  For example, if there were 4 zones available to mine in, miners would roughly divide the world into 4 geographic zones.  They wouldn't be well defined and would overlap from a geography standpoint, but having 4 networks at the zone level would be much more performant then having a single network.  The single network would still exist at the PRIME level.  However, by the time the transactions make it to PRIME they would be whispered in region blocks of 1000 transactions.

Well, I'm ok with collateral work and most of the ideas above, but not with dedicated nodes it would be hard to implement an incentive mechanism.

The incentive mechanism would be giving some reward for the collateral work via merge mining of zone, and region blocks with PRIME.

Right now, Greg Maxwell and others are working on a privacy focused improvement to transaction relay protocol in bitcoin, Dandelion.  The good thing about this BIP is their strategy for delaying transaction relay procedure a while (to make it very hard if not impossible for surveillance services to track sender's IP) and it looks to be committed to the release code in near future! It will be a great opportunity  to do more processing (classification, aggregation, ... ) while the transaction is queued waiting for relay.

I have reviewed this proposal by Greg Maxwell.  It is interesting from an anonymization standpoint.  However, I did not see a mechanism by which the transactions would be delayed and aggregated but rather a routing schema that obfuscates the origin of a transaction.
1675699258
Hero Member
*
Offline Offline

Posts: 1675699258

View Profile Personal Message (Offline)

Ignore
1675699258
Reply with quote  #2

1675699258
Report to moderator
1675699258
Hero Member
*
Offline Offline

Posts: 1675699258

View Profile Personal Message (Offline)

Ignore
1675699258
Reply with quote  #2

1675699258
Report to moderator
1675699258
Hero Member
*
Offline Offline

Posts: 1675699258

View Profile Personal Message (Offline)

Ignore
1675699258
Reply with quote  #2

1675699258
Report to moderator
"Governments are good at cutting off the heads of a centrally controlled networks like Napster, but pure P2P networks like Gnutella and Tor seem to be holding their own." -- Satoshi
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
1675699258
Hero Member
*
Offline Offline

Posts: 1675699258

View Profile Personal Message (Offline)

Ignore
1675699258
Reply with quote  #2

1675699258
Report to moderator
coopex
Copper Member
Newbie
*
Offline Offline

Activity: 9
Merit: 3


View Profile
November 13, 2019, 10:51:51 PM
Merited by mechanikalk (3)
 #22

Hi there! I was looking through some old research papers about merge-mining and came upon this thread. I'm very interested in your proposal as it seems like a great way to shard state without losing security via merge mining! I have a question for you though: If miners have to verify all the state that passes up to Prime, they have to run a full node so that they have the state of all the blockchains to properly verify everything. They are incentivized to do this so that they don't mine invalid blocks, but in doing so they might put a strain on the network because their zone and region nodes are not necessarily in the same geographic region as the rest of the zone and region nodes. (Of course, the zone that the miner is located in will be optimal, but I am talking about the rest of the zones and regions necessary for running a full state node).
For example, for n zones, n/2 regions, and m miners running a full state node, we have m - 1 latent nodes in each zone (or n*(m-1) latent zone nodes total) and m - 1 latent nodes in each region (or (n/2)*(m-1) latent region nodes total). Do you consider this an issue for network latency? Is there perhaps some way or incentive for a miner to run a full node and also run each sub-node (zone and region) in the proper geographic location? This might be physically difficult without the use of some cloud provider like AWS.

Looking forward to hearing more! Thanks.
Wind_FURY
Legendary
*
Offline Offline

Activity: 2450
Merit: 1490



View Profile
November 17, 2019, 07:49:18 AM
 #23

OP, I read the white paper. I'm not a technical person, but are you actually serious about your proposal? For Bitcoin as a hard fork?

Wind_FURY
Legendary
*
Offline Offline

Activity: 2450
Merit: 1490



View Profile
November 19, 2019, 06:57:29 AM
 #24

OP, I read the white paper. I'm not a technical person, but are you actually serious about your proposal? For Bitcoin as a hard fork?

With lots of breaking changes, i doubt soft-fork is possible.

P.S. with amount of effort OP has done, it's very obvious that he's serious.


You gave the OP merits for the topic. What's your opinion on this?

https://github.com/mechanikalk/bips/blob/master/bip-%3F%3F%3F%3F.mediawiki

Quote

Merge-Mining and Difficulty

A key aspect of BlockReduce is all nodes will mine at the zone, region, and PRIME level at the same time. The simultaneous mining of PRIME, regions, and zones allows BlockReduce to keep the entire network's proof-of-work on PRIME.


I already said I'm not a technical person, I'm actually the stupid one, but would the technical people/coders take this proposal seriously?

Wind_FURY
Legendary
*
Offline Offline

Activity: 2450
Merit: 1490



View Profile
November 20, 2019, 05:01:49 AM
 #25

You gave the OP merits for the topic. What's your opinion on this?

See #2 and #4

I already said I'm not a technical person, I'm actually the stupid one, but would the technical people/coders take this proposal seriously?

It depends on their ideology

The ideology?

Quote

and understanding about this proposal.


What about your understanding about the proposal, you believe it's viable?

I'm trying to learn.

Quote

But it's easy to imagine due to hard fork and development complexity (mostly communicate or data sharing between different kind of nodes)


It's "easy to imagine"? I'm confused.

R.I.U. iol
Newbie
*
Offline Offline

Activity: 15
Merit: 2


View Profile
November 21, 2019, 06:40:36 AM
 #26

is there any update on this? Is this still in developement?
Wind_FURY
Legendary
*
Offline Offline

Activity: 2450
Merit: 1490



View Profile
November 22, 2019, 10:31:53 AM
 #27

The ideology?

1. Left wing vs Right wing
2. Small block vs big block
3. On-chain vs off-chain
4. etc.


The debate should be technical, not political.

Quote

What about your understanding about the proposal, you believe it's viable?

I'm trying to learn.

As far as i understand, the proposal is viable. But personally i don't like the idea of full node only store specific set of information.


Really? For Bitcoin?

Quote

It's "easy to imagine"? I'm confused.

Many would reject due to the complexity, at least that's what i think if we were to ask the experts.


Wouldn't more complexity widen the attack-vector on the network?

Wind_FURY
Legendary
*
Offline Offline

Activity: 2450
Merit: 1490



View Profile
November 28, 2019, 11:48:15 AM
 #28

The debate should be technical, not political.

History prove it's impossible to avoid political completely on debate


I'm lost. Include every quote for context please.

But for you, you said that you would like OP's idea depending on the "ideology". What about technically?

Wind_FURY
Legendary
*
Offline Offline

Activity: 2450
Merit: 1490



View Profile
November 29, 2019, 05:37:45 AM
 #29

I'm lost. Include every quote for context please.

My bad, i did that because many users often complain about "pyramid quote"

I'm lost. Include every quote for context please.

But for you, you said that you would like OP's idea depending on the "ideology". What about technically?

It's vague question. Technically it's possible, but personally i wouldn't support the idea for Bitcoin.

I'm too stupid to know which ideas are viable or not. But if you ask me, the risks taken might not be worth the outcome.

gmaxwell, achow, comments?

But just for clarification, what is "3+ orders of magnitude"? 3 times more?

https://github.com/mechanikalk/bips/blob/master/bip-%3F%3F%3F%3F.mediawiki

Quote

BlockReduce presents a new blockchain topology that offers 3+ orders of magnitude improvement in transaction throughput while avoiding the introduction of hierarchical power structures and centralization.


tromp
Hero Member
*****
Offline Offline

Activity: 861
Merit: 807


View Profile
November 29, 2019, 10:30:05 AM
 #30

But just for clarification, what is "3+ orders of magnitude"? 3 times more?

No, 10^3=1000 time more. One order of magnitude is 10x.
coopex
Copper Member
Newbie
*
Offline Offline

Activity: 9
Merit: 3


View Profile
December 01, 2019, 06:14:54 AM
 #31

I'm lost. Include every quote for context please.

My bad, i did that because many users often complain about "pyramid quote"

I'm lost. Include every quote for context please.

But for you, you said that you would like OP's idea depending on the "ideology". What about technically?

It's vague question. Technically it's possible, but personally i wouldn't support the idea for Bitcoin.

I'm too stupid to know which ideas are viable or not. But if you ask me, the risks taken might not be worth the outcome.

gmaxwell, achow, comments?

But just for clarification, what is "3+ orders of magnitude"? 3 times more?

https://github.com/mechanikalk/bips/blob/master/bip-%3F%3F%3F%3F.mediawiki

Quote

BlockReduce presents a new blockchain topology that offers 3+ orders of magnitude improvement in transaction throughput while avoiding the introduction of hierarchical power structures and centralization.


I've read the paper and it certainly seems feasible. You can compare it to using mapreduce in a blockchain context. Of course the attack vector would increase but that's just how it is with complex cryptographic protocols - it's not something a few security audits and a team of smart engineers can't fix (and a testnet obviously). I don't think Bitcoin would take it on though because the changes would be too dramatic for the developers and the miners to accept - this probably needs to be a separate project unfortunately.
Wind_FURY
Legendary
*
Offline Offline

Activity: 2450
Merit: 1490



View Profile
December 01, 2019, 07:43:56 AM
 #32

But just for clarification, what is "3+ orders of magnitude"? 3 times more?

No, 10^3=1000 time more. One order of magnitude is 10x.


From your post-history, I see that you're a developer. What are your initial thoughts on OP's proposal?

tromp
Hero Member
*****
Offline Offline

Activity: 861
Merit: 807


View Profile
December 01, 2019, 06:05:29 PM
Merited by Wind_FURY (1)
 #33

From your post-history, I see that you're a developer. What are your initial thoughts on OP's proposal?

The author appears knowledgeable and this may well be a sensible approach to sharding.
But personally I'm not a fan of sharding and its associated complexity increase.
It should appeal to Ethereum more than Bitcoin...
Wind_FURY
Legendary
*
Offline Offline

Activity: 2450
Merit: 1490



View Profile
December 02, 2019, 06:35:17 AM
 #34

From your post-history, I see that you're a developer. What are your initial thoughts on OP's proposal?

The author appears knowledgeable and this may well be a sensible approach to sharding.
But personally I'm not a fan of sharding and its associated complexity increase.
It should appeal to Ethereum more than Bitcoin...


This doesn't sound like "blockchain woo-woo" for you?  

https://github.com/mechanikalk/bips/blob/master/bip-%3F%3F%3F%3F.mediawiki

Quote

BlockReduce is a new blockchain structure which only segments consistency, allowing it to scale to handle tens of thousands of transactions per second without impacting fault tolerance or decentralization.


I'm not trying to troll/criticize, I'm trying to debate/learn. Because from what I have been told, "sharding" doesn't scale the network out, but only gives the impression that it's scaling out.

tromp
Hero Member
*****
Offline Offline

Activity: 861
Merit: 807


View Profile
December 02, 2019, 09:25:01 AM
Merited by Wind_FURY (1), ETFbitcoin (1), mechanikalk (1)
 #35

This doesn't sound like "blockchain woo-woo" for you?  

https://github.com/mechanikalk/bips/blob/master/bip-%3F%3F%3F%3F.mediawiki

I'm not trying to troll/criticize, I'm trying to debate/learn. Because from what I have been told, "sharding" doesn't scale the network out, but only gives the impression that it's scaling out.

I had to look up woo woo on wikipedia where it's said to be "a term used by magician and skeptic James Randi to denote paranormal, supernatural and occult claims". I see no such claims in BlockReduce :-)

The proposal basically deals with the bandwidth problem of on-chain scaling, trading it off against trust that miners are doing the proper cross-shard checks that they're supposed and incentivized to. What it fails to do making the whole chain fully verifiable by a typical desktop computer, as should be apparent from "the total chain will require around 8 Tb/year of storage".

I don't see these tradeoffs as being acceptable to the Bitcoin community, but they might appeal to the Ethereum community.
mechanikalk (OP)
Member
**
Offline Offline

Activity: 91
Merit: 72


View Profile WWW
December 02, 2019, 02:37:13 PM
 #36


The proposal basically deals with the bandwidth problem of on-chain scaling, trading it off against trust that miners are doing the proper cross-shard checks that they're supposed and incentivized to. What it fails to do making the whole chain fully verifiable by a typical desktop computer, as should be apparent from "the total chain will require around 8 Tb/year of storage".


Tromp, I appreciate the time that you have taken to look at BlockReduce.  One thing that I would debate is the use of the word sharding.  Although, a miner can depend upon a zone blocks work as an attestation to the correctness of the included transactions, they are not required to.  Much like an SPV node doesn't have to keep the entire chainstate but rather just looks at a block header.  This is not sharding per say, but rather a mode of operation that a node can work within to use less resources.  I would anticipate that serious miners or pools will run and validate full state because they have an economic incentive to do so, while merchants will likely run partial state much like SPV. 

Another way to think about BlockReduce is as a form of multi-level erlay where "sketches" are sent when a zone, or region block is found rather than an arbitrary delay.  The obvious difference being that actual sub-blocks are found which are rewarded to incentivize miners to self organize in a network optimal way.
mechanikalk (OP)
Member
**
Offline Offline

Activity: 91
Merit: 72


View Profile WWW
December 02, 2019, 02:47:48 PM
Merited by coopex (1)
 #37

Hi there! I was looking through some old research papers about merge-mining and came upon this thread. I'm very interested in your proposal as it seems like a great way to shard state without losing security via merge mining! I have a question for you though: If miners have to verify all the state that passes up to Prime, they have to run a full node so that they have the state of all the blockchains to properly verify everything. They are incentivized to do this so that they don't mine invalid blocks, but in doing so they might put a strain on the network because their zone and region nodes are not necessarily in the same geographic region as the rest of the zone and region nodes. (Of course, the zone that the miner is located in will be optimal, but I am talking about the rest of the zones and regions necessary for running a full state node).
For example, for n zones, n/2 regions, and m miners running a full state node, we have m - 1 latent nodes in each zone (or n*(m-1) latent zone nodes total) and m - 1 latent nodes in each region (or (n/2)*(m-1) latent region nodes total). Do you consider this an issue for network latency? Is there perhaps some way or incentive for a miner to run a full node and also run each sub-node (zone and region) in the proper geographic location? This might be physically difficult without the use of some cloud provider like AWS.

Looking forward to hearing more! Thanks.

Coopex, great question!  Sorry, it has taken me a bit to get back to you. The miner is incentivized to hold zone state which they are not mining because it reduces the risk they will include a zone block in a region block which eventually gets rolled back in the zone.  If they were to wait or delay including zone blocks in the region blocks, they could also achieve greater certainty, however they would get lower rewards.  Running the alternate zone state allows them to have greater certainty about a zone block faster.  Doing so with a node which is appropriately placed in the network topology will decrease that nodes latency and further decrease risk.  Therefore, miners will be incentivized to keep state and do so in a network optimal way.  I would absolutely expect that a person running full state would do so using something like AWS to allow optimization of the geographic placement of nodes.
coopex
Copper Member
Newbie
*
Offline Offline

Activity: 9
Merit: 3


View Profile
December 02, 2019, 08:27:01 PM
 #38

Hi there! I was looking through some old research papers about merge-mining and came upon this thread. I'm very interested in your proposal as it seems like a great way to shard state without losing security via merge mining! I have a question for you though: If miners have to verify all the state that passes up to Prime, they have to run a full node so that they have the state of all the blockchains to properly verify everything. They are incentivized to do this so that they don't mine invalid blocks, but in doing so they might put a strain on the network because their zone and region nodes are not necessarily in the same geographic region as the rest of the zone and region nodes. (Of course, the zone that the miner is located in will be optimal, but I am talking about the rest of the zones and regions necessary for running a full state node).
For example, for n zones, n/2 regions, and m miners running a full state node, we have m - 1 latent nodes in each zone (or n*(m-1) latent zone nodes total) and m - 1 latent nodes in each region (or (n/2)*(m-1) latent region nodes total). Do you consider this an issue for network latency? Is there perhaps some way or incentive for a miner to run a full node and also run each sub-node (zone and region) in the proper geographic location? This might be physically difficult without the use of some cloud provider like AWS.

Looking forward to hearing more! Thanks.

Coopex, great question!  Sorry, it has taken me a bit to get back to you. The miner is incentivized to hold zone state which they are not mining because it reduces the risk they will include a zone block in a region block which eventually gets rolled back in the zone.  If they were to wait or delay including zone blocks in the region blocks, they could also achieve greater certainty, however they would get lower rewards.  Running the alternate zone state allows them to have greater certainty about a zone block faster.  Doing so with a node which is appropriately placed in the network topology will decrease that nodes latency and further decrease risk.  Therefore, miners will be incentivized to keep state and do so in a network optimal way.  I would absolutely expect that a person running full state would do so using something like AWS to allow optimization of the geographic placement of nodes.

Thanks for your response! I see now that miners are incentivized to run all of their nodes in the least latent way possible. However, miners might not physically be able to do so without moving their mining operation outside of the zone that they operate in, unless they want to pay a cloud provider to host it for them - which may not work if the cloud provider does not offer server hosting close enough or with proper precision to the geographic location of the zone. Perhaps a business could evolve to host servers in close proximity to every zone and move them around when necessary, kind of like high frequency trading does with the stock market, but even then you'd have the business be a centralizing factor.

In any case, my question is more general. Do you consider it an issue if some of the nodes in a zone are more latent than others? Are there bandwidth concerns with users or miners who run latent nodes? What if I just have a really shitty internet connection - could I be causing bandwidth issues for the network, or am I just causing issues for myself?

Thank you for your responses!
Wind_FURY
Legendary
*
Offline Offline

Activity: 2450
Merit: 1490



View Profile
December 03, 2019, 04:38:53 AM
 #39

This doesn't sound like "blockchain woo-woo" for you?  

https://github.com/mechanikalk/bips/blob/master/bip-%3F%3F%3F%3F.mediawiki

I'm not trying to troll/criticize, I'm trying to debate/learn. Because from what I have been told, "sharding" doesn't scale the network out, but only gives the impression that it's scaling out.

I had to look up woo woo on wikipedia where it's said to be "a term used by magician and skeptic James Randi to denote paranormal, supernatural and occult claims". I see no such claims in BlockReduce :-)

The proposal basically deals with the bandwidth problem of on-chain scaling, trading it off against trust that miners are doing the proper cross-shard checks that they're supposed and incentivized to. What it fails to do making the whole chain fully verifiable by a typical desktop computer, as should be apparent from "the total chain will require around 8 Tb/year of storage".


Then I would debate that the statement, "allowing it to scale to handle tens of thousands of transactions per second without impacting fault tolerance or decentralization", is u true.


The proposal basically deals with the bandwidth problem of on-chain scaling, trading it off against trust that miners are doing the proper cross-shard checks that they're supposed and incentivized to. What it fails to do making the whole chain fully verifiable by a typical desktop computer, as should be apparent from "the total chain will require around 8 Tb/year of storage".


Tromp, I appreciate the time that you have taken to look at BlockReduce.  One thing that I would debate is the use of the word sharding.  Although, a miner can depend upon a zone blocks work as an attestation to the correctness of the included transactions, they are not required to.  2Much like an SPV node doesn't have to keep the entire chainstate but rather just looks at a block header.  This is not sharding per say, but rather a mode of operation that a node can work within to use less resources.  I would anticipate that serious miners or pools will run and validate full state because they have an economic incentive to do so, while merchants will likely run partial state much like SPV


You don't believe that that will centralize Bitcoin toward the miners? Or you don't believe that users/economic majority should have the ability to run their own full nodes?

mechanikalk (OP)
Member
**
Offline Offline

Activity: 91
Merit: 72


View Profile WWW
December 06, 2019, 11:49:25 PM
 #40

Hi there! I was looking through some old research papers about merge-mining and came upon this thread. I'm very interested in your proposal as it seems like a great way to shard state without losing security via merge mining! I have a question for you though: If miners have to verify all the state that passes up to Prime, they have to run a full node so that they have the state of all the blockchains to properly verify everything. They are incentivized to do this so that they don't mine invalid blocks, but in doing so they might put a strain on the network because their zone and region nodes are not necessarily in the same geographic region as the rest of the zone and region nodes. (Of course, the zone that the miner is located in will be optimal, but I am talking about the rest of the zones and regions necessary for running a full state node).
For example, for n zones, n/2 regions, and m miners running a full state node, we have m - 1 latent nodes in each zone (or n*(m-1) latent zone nodes total) and m - 1 latent nodes in each region (or (n/2)*(m-1) latent region nodes total). Do you consider this an issue for network latency? Is there perhaps some way or incentive for a miner to run a full node and also run each sub-node (zone and region) in the proper geographic location? This might be physically difficult without the use of some cloud provider like AWS.

Looking forward to hearing more! Thanks.

Coopex, great question!  Sorry, it has taken me a bit to get back to you. The miner is incentivized to hold zone state which they are not mining because it reduces the risk they will include a zone block in a region block which eventually gets rolled back in the zone.  If they were to wait or delay including zone blocks in the region blocks, they could also achieve greater certainty, however they would get lower rewards.  Running the alternate zone state allows them to have greater certainty about a zone block faster.  Doing so with a node which is appropriately placed in the network topology will decrease that nodes latency and further decrease risk.  Therefore, miners will be incentivized to keep state and do so in a network optimal way.  I would absolutely expect that a person running full state would do so using something like AWS to allow optimization of the geographic placement of nodes.

Thanks for your response! I see now that miners are incentivized to run all of their nodes in the least latent way possible. However, miners might not physically be able to do so without moving their mining operation outside of the zone that they operate in, unless they want to pay a cloud provider to host it for them - which may not work if the cloud provider does not offer server hosting close enough or with proper precision to the geographic location of the zone. Perhaps a business could evolve to host servers in close proximity to every zone and move them around when necessary, kind of like high frequency trading does with the stock market, but even then you'd have the business be a centralizing factor.

In any case, my question is more general. Do you consider it an issue if some of the nodes in a zone are more latent than others? Are there bandwidth concerns with users or miners who run latent nodes? What if I just have a really shitty internet connection - could I be causing bandwidth issues for the network, or am I just causing issues for myself?

Thank you for your responses!

That is a pretty insightful question.  However, the answer is pretty simple.  With all distributed networks from things like Napster to Bitcoin you always have a seed and a leech problem.  In the example of Napster it is driven by storage space more than bandwidth.  In the context of Bitcoin it is driven by bandwidth more than storage.  Therefore, if you are a Bitcoin node that has low bandwidth you are slowing the overall network down (leech), whereas if you have high bandwidth you are speeding it up (seed).  BlockReduce is not different from Bitcoin in this regard. However, BlockReduce rewards mining in the zone chains creating an economic incentive for participants who are mining to optimize for latency.
Pages: « 1 [2] 3 »  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!