Bitcoin Scaling Solution Without Lightning Network...

[bones261]

You can't really know if "the vast majority of the network" is on the same page or not until the D day actually comes. We have already seen miners voting supposed "intention" to support something with their hashrate, then when the day come some of then backpeddle. You would also need all exchanges on board. And ultimately you would need all whales on board, and many of them may not bother to say their opinion at all, then the day of the fork comes and you see an huge dump on your forked coin.

You basically need 100% consensus for a hardfork to be a success and not end up with 2 coins, and I don't see how this is even possible ever when a project gets as big as Bitcoin is (I mean it's still small in the grand scheme of things, but open source software development/network effect wise, it's big enough to not be able to ever hardfork seamlessly again. Maybe im wrong and there is a consensus in the future for a hardfork, but again I don't see how.

What's this fear of having 2 coins? We already have hundreds of coins, most of them being more or less hard forks off of BTC. The free market will decide which coins persists and which coins go by the wayside. BTC has already demonstrated over and over again that it is the honey badger. If we honestly have faith that BTC is anti-fragile, pesky minority coins are nothing but a mere nuisance.

[BitHodler]

What's this fear of having 2 coins? We already have hundreds of coins, most of them being more or less hard forks off of BTC. The free market will decide which coins persists and which coins go by the wayside. BTC has already demonstrated over and over again that it is the honey badger. If we honestly have faith that BTC is anti-fragile, pesky minority coins are nothing but a mere nuisance.

Absolutely nothing is wrong with that. I would even like to say that it improves scalability potential with how you no longer have to care about another side thinking its own roadmap and implementation is the one to follow.

Remember the drama we went through before the Bcash split? They are gone and we no longer have to care about what their plans are, which is the best thing that could happen to Bitcoin.

I get it that people want to protect Bitcoin, or that they feel they have to, but if it is the unbreakable powerhouse people say it is, then why worry about what's going to happen? Let the economy do its work.

Most forks will fail anyway because they won't be backed by any noteworthy players. There won't be a second Bcash. We've seen the worst.

[cfbtcman]

The main problem is that most of these scaling solutions that are being proposed will first require a hardfork. This means we'll have the drama of 2 competing bitcoins trying to claim that they are the real one (see the BCash ABC vs BCash SV ongoing war right now). This will not end well. Without consensus we will just end up with 2 bitcoins which are in sum of lesser value than before the hardfork happened.

Most bitcoin whales don't support any of the proposed scaling solutions so far so your scaling fork will end up dumped by tons of coins.

Hi, i think we dont need to hard-fork for implement something like the Type of nodes A,B,C,D... they could exist at same time as full-nodes, we only need to hard fork after if we want to upgrade block-size or not, because i read something other day about some solution to change block without hard forking.

But even Satoshi left the block size limit to can be changed in future and we will really need to change it, even with LN working 100% we will need to change blocksize or else 2MB will not support all transactions ONCHAIN->OFFCHAIN and vice-versa.

If we dont do that, that would be dangerous for Layer 1, who would support layer 1 nodes and mining if all the transactions would be done almost 100% in layer 2?

Satoshi said that after mining the 21 millions the miners would be supported only by transactions, what transactions if everyone only uses LN?
When you want to put money ONCHAIN the miners would ask you maybe $5000 because would be very few transactions and that transactions would need to support layer1 expenses.

[cellard]

You can't really know if "the vast majority of the network" is on the same page or not until the D day actually comes. We have already seen miners voting supposed "intention" to support something with their hashrate, then when the day come some of then backpeddle. You would also need all exchanges on board. And ultimately you would need all whales on board, and many of them may not bother to say their opinion at all, then the day of the fork comes and you see an huge dump on your forked coin.

You basically need 100% consensus for a hardfork to be a success and not end up with 2 coins, and I don't see how this is even possible ever when a project gets as big as Bitcoin is (I mean it's still small in the grand scheme of things, but open source software development/network effect wise, it's big enough to not be able to ever hardfork seamlessly again. Maybe im wrong and there is a consensus in the future for a hardfork, but again I don't see how.

What's this fear of having 2 coins? We already have hundreds of coins, most of them being more or less hard forks off of BTC. The free market will decide which coins persists and which coins go by the wayside. BTC has already demonstrated over and over again that it is the honey badger. If we honestly have faith that BTC is anti-fragile, pesky minority coins are nothing but a mere nuisance.

Well, isn't it obvious? Ask to someone with a decent amount at stake in Bitcoin if they want to see say, their 100 BTC, crash in value because someone decided to hardfork with the same hashing algorithm, which means that you will see miners speculating with the hashrate while they can. See the recent dip in hashrate, which is the biggest loss of hashrate on adjustment time of the year:



I guarantee you that Bitcoin holders don't appreciate this bullshit. Not that they pose a systemic risk for Bitcoin, but they are annoying and slowing down the rocket.

Anyhow, the real question should be: what's the fear of starting your own altcoin if it's such a good idea, instead of constantly trying to milk 5 minutes of fame for your altcoin by forking it off Bitcoin? If your idea is so good, start it as a an actual altcoin and compete.

[DooMAD]

What's this fear of having 2 coins? We already have hundreds of coins, most of them being more or less hard forks off of BTC. The free market will decide which coins persists and which coins go by the wayside. BTC has already demonstrated over and over again that it is the honey badger. If we honestly have faith that BTC is anti-fragile, pesky minority coins are nothing but a mere nuisance.

Well, isn't it obvious? Ask to someone with a decent amount at stake in Bitcoin if they want to see say, their 100 BTC, crash in value because someone decided to hardfork with the same hashing algorithm, which means that you will see miners speculating with the hashrate while they can. See the recent dip in hashrate, which is the biggest loss of hashrate on adjustment time of the year:

I guarantee you that Bitcoin holders don't appreciate this bullshit. Not that they pose a systemic risk for Bitcoin, but they are annoying and slowing down the rocket.

This sounds like an argument that the feelings of those who hold bitcoin because they're speculating on the price should somehow outweigh the feelings of those who hold Bitcoin because they appreciate the fundamental principles of freedom and permissionlessness.  It's an argument for the ages, certainly, but one that neither side will ever back down from.

Ideological differences are naturally going to occur and forks are going to happen.  Your desire to see higher prices isn't going to prevent that.  These are exactly the risks people should consider when they invest if all they care about is the potential profit.

Bitcoin is still very much the "Wild West of money".  Don't decide to play cowboy if you can't handle some bandits every now and then.  

[aliashraf]

I suppose this thread is getting out the rails by falling into the old "To fork or not to fork? This is the question!" story, which cellard is an expert in it  
The fact is no matter how and what bitcoin whales want it to be, forks happen and sooner or later we need to have current bitcoin retired as the former legitimate chain and the community should converge on an improved version. You just can't stop evolution from happening, ok?

For now, I'm trying to save this thread, rolling back to the original technical discussion and sharing an interesting idea with you guys meanwhile, please focus:

so lets say there is a transaction Tx1 and it is in the other half of the "data" in node Type B. and say i run node Type A. if someone pays me by spending Tx1, i do not have it in my database (blockchain) so how can i know it is a valid translation? am i supposed to assume that it is valid on faith?! or am i supposed to connect to another node and ask if the transaction is valid and then trust that other node to tell me the truth? how would you prevent fraud?

What op proposes, as correctly have been categorized by other posters in the thread is a special version of sharding.
Although it is an open research field, it is a MUST for bitcoin. Projecting scaling problem to second layer protocols (like LN) is the worst idea because, you can't simulate bitcoin on top of bitcoin as a #2 layer, it is absurd. Going to second layer won't happen unless by giving up about some essential feature of bitcoin or at least being tolerant about centralization and censorship threats, compromising the cause.

So, this is it, our destiny, we need an scalable blockchain solution and as of now, we got just sharding.

Back to Pooya objection, it occurs when a transaction that is supposed to be processed in a partition/shard is trying to access an unspent output from another shard. I think there may be a workaround for this:

Suppose in a sharding based on transaction partitioning that uses a simple mod operation where txid mod N determines the transaction's partition number in an N shards network, we put a constraint on transactions such that wallets are highly de-incentivized/not-allowed to make a transaction from heterogeneous outputs, i.e. outputs from transactions in multiple shards.
 
Now we have this transaction tx1 with its outputs belonging to transactions that are maintained on a same shard, the problem would be to which shard the transaction itself belongs? The trick is adding a nonce field to the transaction format and make the wallet client software to perform like N/2 hashes (a very small amount of work) to find a proper nonce that makes txid mod N such that it fits to the same shard as its output. For coinbase transaction, the same measure should be taken by miners.

It looks somewhat scary, being too partitioned but I'm working on it as it looks very promising to me.

[cfbtcman]

Then how about backward compability with older node/client?
1. They wouldn't know that they need to connect to different type of nodes to get and verify all transaction
2. There's no guarantee they will get all transaction/block as there's no guarantee they connect to all different type of nodes

I think its possible to create new bitcoin core version supporting type nodes and emulating a normal node communication with old ones and new TYPE Node communication with new ones, at least with the actual block size, but that would be only experience, because without bigger blocks that is not usefull.

[cfbtcman]

I suppose this thread is getting out the rails by falling into the old "To fork or not to fork? This is the question!" story, which cellard is an expert in it  
The fact is no matter how and what bitcoin whales want it to be, forks happen and sooner or later we need to have current bitcoin retired as the former legitimate chain and the community should converge on an improved version. You just can't stop evolution from happening, ok?

For now, I'm trying to save this thread, rolling back to the original technical discussion and sharing an interesting idea with you guys meanwhile, please focus:

so lets say there is a transaction Tx1 and it is in the other half of the "data" in node Type B. and say i run node Type A. if someone pays me by spending Tx1, i do not have it in my database (blockchain) so how can i know it is a valid translation? am i supposed to assume that it is valid on faith?! or am i supposed to connect to another node and ask if the transaction is valid and then trust that other node to tell me the truth? how would you prevent fraud?

What op proposes, as correctly have been categorized by other posters in the thread is a special version of sharding.
Although it is an open research field, it is a MUST for bitcoin. Projecting scaling problem to second layer protocols (like LN) is the worst idea because, you can't simulate bitcoin on top of bitcoin as a #2 layer, it is absurd. Going to second layer won't happen unless by giving up about some essential feature of bitcoin or at least being tolerant about centralization and censorship threats, compromising the cause.

So, this is it, our destiny, we need an scalable blockchain solution and as of now, we got just sharding.

Back to Pooya objection, it occurs when a transaction that is supposed to be processed in a partition/shard is trying to access an unspent output from another shard. I think there may be a workaround for this:

Suppose in a sharding based on transaction partitioning that uses a simple mod operation where txid mod N determines the transaction's partition number in an N shards network, we put a constraint on transactions such that wallets are highly de-incentivized/not-allowed to make a transaction from heterogeneous outputs, i.e. outputs from transactions in multiple shards.
 
Now we have this transaction tx1 with its outputs belonging to transactions that are maintained on a same shard, the problem would be to which shard the transaction itself belongs? The trick is adding a nonce field to the transaction format and make the wallet client software to perform like N/2 hashes (a very small amount of work) to find a proper nonce that makes txid mod N such that it fits to the same shard as its output. For coinbase transaction, the same measure should be taken by miners.

It looks somewhat scary, being too partitioned but I'm working on it as it looks very promising to me.

Woow, you are the man 

[mechanikalk]

This has been discussed many times and unfortunately, majority of Bitcoiner would disagree since increasing block size would increase the cost of running full nodes.
Split block data to many different nodes type is called Sharding and already proposed many times such as BlockReduce: Scaling Blockchain to human commerce
Besides, IMO sharding open lots of attack vector, increase development complexity and requiring more trust.

Additionally, LN help bitcoin scaling a lot, even though it's not perfect solution. Those who said that clearly don't understand how LN works and it's potential.
Lots of cryptocurrency including Ethereum are preparing 2nd-layer/off-chain as scaling solution because they know it's good scaling solution.

If anyone is interested in a simpler understanding of BlockReduce, they can checkout this 30 minute intro that I presented at University of Texas Blockchain Conference.  I ultimately don't think it is all that complicated, it is really just multithreading Bitcoin and tying it back together with merge mining.

BlockReduce Presentation

In terms of issues with lightning, I think that the biggest problem is the cost of capital.  Transaction fees are necessarily going to be non-zero in economic equilibrium because node operators will need to pay for the cost of money, or opportunity cost.  Therefore, you would expect to pay ~1-4% annually of a transaction that takes place in a channel that is open for any period of time.  It will also need to account for the costs of opening and closing the channel.  The only reason that LN kinda works is because people are not fully accounting for the cost of capital needed to run the nodes. 

[cfbtcman]

People,

What about something like this:

https://ibb.co/c249h0

Vertical blockchain block scaling.
Each block would have the same limit size as now.
Possibility to create many "vertical" blocks in each 10 minutes interval when Mempool size is overcharged.
Each new layer blockchain would be part of a new layer node so we could have infinite Layer Type Nodes

That Layer Type nodes will include less repeated data, so we save disk space, if we have 1 million nodes we dont need to save all information in the same nodes, thats a waste, so when someone wants to install a node, the node install program will sugest the best Layer Type node to install, the one the system needs more.
The Layer Type nodes exchange information between them, something like pruned nodes.

This way we prevent download and upload of big quantities of data like it happens with BCH with huge block sizes.

All the Layer Type Nodes share information they need that can be tested with the information that Layer Type Node already have, everything is hashed with everything.

[mechanikalk]

People,

What about something like this:

https://ibb.co/c249h0

Vertical blockchain block scaling.
Each block would have the same limit size as now.
Possibility to create many "vertical" blocks in each 10 minutes interval when Mempool size is overcharged.
Each new layer blockchain would be part of a new layer node so we could have infinite Layer Type Nodes

That Layer Type nodes will include less repeated data, so we save disk space, if we have 1 million nodes we dont need to save all information in the same nodes, thats a waste, so when someone wants to install a node, the node install program will sugest the best Layer Type node to install, the one the system needs more and the Layer Type nodes exchange information, something like pruned nodes.

This way we prevent download and upload of big quantities of data like it happens with BCH with huge block sizes by block.

All the Layer Type Nodes share information they need that can be tested with the information that Layer Type Node already have, everything is hashed with everything.

The issue with your proposal above is that you do not know which transactions to include in which vertical block.  You will not know which transactions take precedence and will introduce a mechanism by which a double spend can occur because a conflicting transaction could be put into two blocks at once.  However, I do think you are thinking in the right direction.

You should check out BlockReduce, which is similar to this idea, but moves transactions in a PoW managed hierarchy to insure consistency of state.  If the manuscript is a bit long, please also check out a presentation I did at University of Texas Blockchain conference.

[Anti-Cen]

"Lightning network" == Mini banks

I did warn you all and no the so called new "Off-Block" hubs did not save BTC as we can see from the price.

CPU-Wars, mere 9 transactions per second from 20,000 miners and fees hitting $55 per transaction is what
the BTC code will be remembered for as it enters our history books just like Tulip Mania did in the 1700's

Casino managers are not the best people in the world to take financial advise from and the same goes for the
dis-information moderator here that keeps pressing the delete button here because he hates the truth being exposed.

[HeRetiK]

"Lightning network" == Mini banks

They're not.

I did warn you all and no the so called new "Off-Block" hubs did not save BTC as we can see from the price

Caring about short-term fluctuations are usually a sign of a lack of long-term thinking.

CPU-Wars, mere 9 transactions per second from 20,000 miners and fees hitting $55 per transaction is what
the BTC code will be remembered for as it enters our history books just like Tulip Mania did in the 1700's

Irrelevant to the discussion.

Casino managers are not the best people in the world to take financial advise from and the same goes for the
dis-information moderator here that keeps pressing the delete button here because he hates the truth being exposed.

See above, which may also be the reason why some of these posts got deleted, rather than a hidden conspiracy by big crypto.

[DooMAD]

<snip>

Off topic, but I thought RNC and all related accounts were banned?  I did have a link, but copying it on a phone is a ballache.

//EDIT:   https://bitcointalk.org/index.php?topic=2617240.msg31377296#msg31377296
RNC admitted to ban evasion and being Anti-Cen there.

[cfbtcman]

"Lightning network" == Mini banks

I did warn you all and no the so called new "Off-Block" hubs did not save BTC as we can see from the price.

CPU-Wars, mere 9 transactions per second from 20,000 miners and fees hitting $55 per transaction is what
the BTC code will be remembered for as it enters our history books just like Tulip Mania did in the 1700's

Casino managers are not the best people in the world to take financial advise from and the same goes for the
dis-information moderator here that keeps pressing the delete button here because he hates the truth being exposed.

The exchangers are the mini banks yet, maybe worst than Lightning Network, LN is like you put your salary of 1 month and after that you will pay you expenses and you receive back and waste and earn, always the same money, if you loose money you will lose not so much, the actual exchanges are worst, MtGox, Btc-e, Wex, etc they are stealing millions and nobody cares.

Maybe BTC its a little complicated for common citizen and we need that "mini or big banks" to work together.

[Wind_FURY]

I would be very curious what Bitcoin's network topology will look like if by some miraculous event, Bitcoin hard forks to bigger blocks and sharding, with consensus. Haha.

Plus big blocks are inherently centralizing the bigger they go. Wouldn't sharding only prolong the issue, and not solve it?

[aliashraf]

I would be very curious what Bitcoin's network topology will look like if by some miraculous event, Bitcoin hard forks to bigger blocks and sharding, with consensus. Haha.

Plus big blocks are inherently centralizing the bigger they go. Wouldn't sharding only prolong the issue, and not solve it?

Op's proposal is a multilevel hierarchical sharding schema in which bigger blocks are handled in the top level. As I have debated it extensively above thread there are a lot of issues remained unsolved but I think we need to take every sharding idea as a serious one and figure out a solution for scaling problems eventually, hence hierarchical schemas are promising enough to be discussed and improved, imo.

[greg458]

The main problem is that most of these scaling solutions that are being proposed will first require a hardfork. This means we'll have the drama of 2 competing bitcoins trying to claim that they are the real one (see the BCash ABC vs BCash SV ongoing war right now). This will not end well. Without consensus we will just end up with 2 bitcoins which are in sum of lesser value than before the hardfork happened.

Most bitcoin whales don't support any of the proposed scaling solutions so far so your scaling fork will end up dumped by tons of coins.

[Wind_FURY]

I would be very curious what Bitcoin's network topology will look like if by some miraculous event, Bitcoin hard forks to bigger blocks and sharding, with consensus. Haha.

Plus big blocks are inherently centralizing the bigger they go. Wouldn't sharding only prolong the issue, and not solve it?

Op's proposal is a multilevel hierarchical sharding schema in which bigger blocks are handled in the top level.

Ok, but big blocks are inherently centralizing the bigger they go, are they not? Sharding would only prolong the issue on the network, any blockchain network, of scaling in.

As I have debated it extensively above thread there are a lot of issues remained unsolved but I think we need to take every sharding idea as a serious one and figure out a solution for scaling problems eventually, hence hierarchical schemas are promising enough to be discussed and improved, imo.

For Bitcoin? I believe it would be better proposed in a network that has big blocks, and does not have firm restrictions on hard forks. Bitcoin Cash ABC.

[franky1]

its sufficient that in a group of 1000 nodes, 500 save backup of 50% of the data (Type A node) and the other 50% can save backup of the other 50% of the data (Type B node), its a waste of space all the nodes in the network save all information repeated thousands of times.

Then each node Type A can "ask" to other node Type B the information is trying to find and vice-versa and gets the information anyway

Bitcoin was designed to be trustless.  The idea of running a node is that you can validate and verify every single transaction yourself.  If you run a Type A node, you would have to trust the Type B nodes to do half of the validation for you.  If you're going to do that, why not just trust Visa and forget all about Bitcoin?

finally doomad sees the light about "compatibility not = full node".. and how "compatible" is not good for the network..
one merit earned... may he accept the merit and drop that social drama debate now he seen the light.

onto the topic
the hard fork of removing full nodes that can only accept 1mb blocks has been done already, in mid 2017. reference the "compatible" nodes still on the network are not full nodes no more.

all is required is to remove the "witness scale factor" and the full 4mb can be utilised by legacy transactions AND segwit transactions.
this too can have positives by removing alot of wishy washy lines of code too and bring things back inline with a code base that resembled pre segwit block structuring to rsemble a single block structure where everything is together that doesnt need to be stripped/"compatible".
yes the "compatible" nodes would stall out and not add blocks to their hard drive. but these nodes are not full nodes anyway so people using them might aswell just use litewallets and bloom filter transaction information they NEED for personal use.

we will then have the network able to actually handle more tx/s at a 4x level rather than a 1.3-2.5x level(which current segwit blockstructure LIMITS (yep even with 4mb suggested weight. actual calculations limit it to 2.5x compared to legacy 1mb))

...
as for how to scale onchain.
please do not throw out "LN is the solution" or "servers will be needed" or "you cant buy coffee"
1. instead of needing LN for coffee by channelling to starbucks. just onchain use a tx to btc buy a $40 starbucks giftcard once a fortnight.

after all from a non technical real life utility perception of average joe. if your LN locking funds with starbucks for a fortnight anyway. its the same 'feel' as just pre-buying a fortnights worth of coffee via a giftcard.
(it also solves the routing, multiple channel requirement for good connection, spendability, also the other problems LN has)

2. onchain scaling is not about just raising the blocksize. its also about reducing the sigops limit so that someone cant bloat/maliciously fill a block with just 5 transactions to use up the limits. EG blocksigops=80k and txsigops=16k meaning 5 tx's can fill a block should they wish. this is by far a bad thing to let continue to be allowed as a network rule.*

3. point 2 had been allowed to let exchanges batch transactions into single transactions of more in/outputs so that exchanges could get cheaper fee's. yet if an exchange is being allowed to bloat a block alone. then that exchange should be paying more for that privelige, not less.
(this stubbornly opens up the debate of should bitcoins blockchain be only used by reserve hoarders of multiple users in permissioned services(exchanges/ln factories).. or should the network be allowing individuals wanting permissionless transacting).. in my view permissioned services should be charged more than an individual

4. and as we move away from centralised exchanges that hoard coins we will have less need for xchanges to batch such huge transactions and so there will be less need of such bloated transactions

5. scaling onchain is not just about raising the blocksize. its about making it more expensive for users who transact more often than those who transact less frequently.
EG imagine a person spend funds to himself every block. and was doing it via 2000 separate transactions a block (spam attack)
he is punishing EVERYONE else. as others that only spends once a month are finding that the fee is higher, even though they have not done nothing wrong.
the blocks are still only collating the same 2000tx average. so from a technical prospective are not causing any more 'processing cost' to mining pool nodes tx's into block collation mechanism. (they still only collate ~2000tx so no cost difference)
so why is the whole network being punished. due to one persons spam.

the person spending every block should pay more for spending funds that have less confirms than others. in short the more confirms your UTXO has the cheaper the transactions get. that way spammers are punished more.
this can go a stage further that the child fee also increases not just on how young the parent is but also the grandparent

in short bring back a fee priority mechanism. but one that concentrates on age of utxo rather than value of utxo(which old one was)