An idea to significantly improve transaction throughput in bitcoin

[draad]

The Problem

Currently the bitcoin blocksize debate is causing concern about the future scalability of bitcoin. Hardforking bitcoin to have larger bocksize does not have the support needed by the community. The recent ETH/ETC hardfork debacle has shown that hardforks are problematic and that an increase in transactions should not be done by hard forking bitcoin core if at all possible.

Which raises the Questions:
1. What can we do to increase the amount of transactions that bitcoin can support without hardforking?
2. Can we increase the number of transactions per second supported without increasing the size of the blocks in the blockchain?

I think the key lies in re-inventing the mempool.

A Solution

A major bitcoin core component is the mempool. All transactions that potentially could be in a future block are broadcast to the network and are initially placed in the mempool of each node. At the moment this mempool is pretty simple, often bloated and chaotic. I see a way of changing the mempool in stages so that bitcoin can support many more transactions per second with only soft forks required.

Double spends occur when someone attempts to spend the same bitcoin twice. At present the first transaction that gets into the blockchain wins the double spend race – this is usually the transaction with the biggest mining fee. It is in the mempool that double spends are first detected. By re-working how the mempool manages double spends, & allowing exceptions to double spends we shall be able to significantly improve the transaction rate of bitcoin and support new features such as large scale micro-payments and more anonymous/private payments.

Step 1 : Turn the mempool into a second ‘fast’ blockchain with a fixed length.

At the moment when a new transaction is broadcast, it gets held in the mempool of each bitcoin node: each transaction that has not found it’s way into a block gets held in memory by every bitcoin node in the network – this is the mempool. When a miner mines a block she selects the best paying transactions from the mempool and places them in her new block. Often many transactions are stuck in the mempool because the fee on the transaction is too low for a miner to want to place it in a block. This has become much worse as the blocks have become full.   

The collection of transactions in the mempool is currently unordered in time and not collectively structured. The first stage would be for each miner, when a block is mined to also mine mempool-blocks. These mempool-blocks would contain any new transactions that has not been included in the core blockchain. The ‘mempool-blockchain’ can be secured against the current core blockchain and signed by the current miner – this can be done in a number of ways and the semantics are pretty simple (I am not going to go into it here). There would be no PoW needed as the PoW required to mine a block is sufficient.

Mempool-blocks should also be done at a faster rate then core blocks. The current active miner could produce blocks ‘on demand’ as transactions turn up. This is to allow quick confirmation of tiny bitcoin transactions where individuals do not need to wait for confirmations (i.e. buying a coffee with bitcoin). The miner that mined the last block would be responsible for creating these small rapid mempool-blocks on demand until the next core block is mined. The intention would be to implement the mempool-blockchain in a similar way to the design of the ‘Bitcoin-NG’ protocol.

The mempool-blockchain should be short lived. Surviving about 6 months in total. The mempool-blockchain would be 6x24x30x6=25920 ‘core’ blocks in length. Any mempool-block older then 25920 core blocks could be safely discarded.

Step 2 : Re-work how double spends work.

The enhanced mempool-blockchain now provides a time locked chain of transactions that have not yet been set into the core blockchain.

The New Rules Are:
1. A valid transaction in the mempool-blockchain wins any double spend race. So when buying a coffee with bitcoin, you would only need to wait a few seconds for the for the confirmation in the mempool-blockchain.
2. Valid double spends can (and should) occur so long as they are equivalent to the original transaction or set of transactions and reduce the size or complexity of the core blockchain. 
3. Any conflicting double spend is rejected from future mempool-blocks by the miner and the network.

Step 3 : Make the core/miners enforce the new double spend rules.

This is the soft fork.

By causing miners to take into account the new rules will cause a soft fork of bitcoin core.

Community and mining consensus would need to occur before this happens. However as it is a soft fork, only a majority of mining power would need to support the change for it to succeed.

Step 4 :  Provide the incentive for bitcoin users to Crush transactions in the mempool

A simple example:
1. Adam has 100000 Satoshi, Bob has no Satoshi, Carrol has no Satoshi.
2. Adam sends 99990 satoshi to Bob (and 10 to the miner as fees). The transaction goes into the mempool blockchain – but not the core blockchain as the fee is too low.
3. Bob then on-sends 99980 satoshi to Carrol (and 10 to the miner). The transaction goes into the mempool blockchain – but not the core blockchain because the fee is too low.
4. The net result would be 2 transactions, 20 satoshi for the miner and 99980 for Carrol. Adam has no Satoshi, Bob has no Satoshi. This is all recorded in the mempool blockchain - but not the core blockchain.
5. Adam on seeing this transaction could now create a valid double spend transaction. Adam creates a new transaction that sends 99985 satoshi to Carrol, 15 Satoshi to the miner and keeps 5 satoshi for himself. This again gets recorded into the mempool blockchain. 
6. The miners can see that Adam’s new single transaction is consistent with the previous 2 transactions while offering a higher mining fee. The miner accepts this transaction in her new core block.
7. Two transactions have been successfully crushed into one transaction with the following benefits to all:
++++ The number of transactions in the core blockchain is reduced for the same net result.
+ Everyone gets an increased level of anonymity.
++ The miner gets an increased fee for the single transaction.
++ Adam gets to save on fees.
–+ Some information is lost after time.
     
A lightning network example:
1. Darren buys subscription content on a pay per view basis using bitcoin micro-payments for each view.
2. Over several weeks Darren sends many separate small transactions to his content provider & pays a small fee to the miner each time. Each of Darren’s transactions is stored in the mempool-blockchain – but not in the core blockchain as his fees on each transaction is too low.
3. Each time Darren sends a new micropayment, he also creates a crush transaction that replaces all his transactions to date with a single larger transaction.
4. After 42 micro-payments the amount of mining fee created by the single crush payment is acceptable to a miner who places the single crush transaction into a core block.
5. 42 transactions have been successfully crushed into one transaction with the following benefits to all:
++++ The number of transactions in the core blockchain is reduced by 41 for the same net result. (97% compression of bitcoin data).
++ The miner gets an increased fee for the single transaction.
++ Darren gets to save on fees.
–+ Some information is lost after time.
+++ Networks / Side-chains such as the lightning network become largely obsolete.

More complex examples:
   More complex trees of transactions can be ‘crushed down’ into fewer simpler transactions. Computers are pretty good at that sort of thing and it is beyond this document to show all the permutations and combinations of that. Significant compression of the amount of data needed to be stored in the core blockchain can be achieved via this design. 

Giving Bitcoin users choice:
1. If you are making a large transaction (buying a Tesla) with bitcoin then you would put a large enough fee on the transaction such that it moved into the core blockchain on the next block.
2. If you are making small transactions (micro-payments) then you may place the transaction into the mempool-blockchain for future compaction and crushing.
3. In time the mempool-blockchain would become more trusted providing new use cases for micro-payments and other secondary uses (i.e. short term smart-contracts).   
4. The design is backwards compatible with current wallets/nodes, to use the new features wallets/nodes would need to be enhanced – but old wallets would still function.

 
This is a work in progress, but I would like to here what others think. Am I full of crap or am I onto something here?

[1715232398]

1715232398

[1715232398]

1715232398

[1715232398]

1715232398

[1715232398]

1715232398

[1715232398]

1715232398

[achow101]

The mempool is not uniform across all nodes. What my node says is its mempool is not the same as what your node says is its mempool. This is all dependent upon the settings of the node. There can be transactions that are in the mempool of one node, but not the rest.

[draad]

True, that's why you get the current miner to order them in a blockchain. So then they become ordered.

[achow101]

True, that's why you get the current miner to order them in a blockchain. So then they become ordered.

What is "the current miner"? There is no one miner for each block. All miners are trying to solve a block for the same height, it's not like one is assigned to each block.

What prevents a malicious miner from making ridiculously large mempool-blocks? What if a malicious miner creates thousands of tiny transaction, say 1 satoshi each, and include 0 fee. Then he includes those transactions in a mempool block. How do nodes handle that?

[draad]

True, that's why you get the current miner to order them in a blockchain. So then they become ordered.

What is "the current miner"? There is no one miner for each block. All miners are trying to solve a block for the same height, it's not like one is assigned to each block.

What prevents a malicious miner from making ridiculously large mempool-blocks? What if a malicious miner creates thousands of tiny transaction, say 1 satoshi each, and include 0 fee. Then he includes those transactions in a mempool block. How do nodes handle that?

The last miner to mine a block can generate mempool blocks until a new miner comes along == the current miner.

As for the malicuos miner, there could be block size limits placed on the mempool blockchain just as is done on the core blockchain. But we would probably build the flexibility in from the start so that we do not repeat the current issue with the block limit. Yes, malicous intent needs to be addressed, but I think we can work the details out for that issue prior to implementation. In general it i not in a miners best interests to mess up bitcoin...

Thanks for the comments :-)

[achow101]

The last miner to mine a block can generate mempool blocks until a new miner comes along == the current miner.

So what about a race condition, stale blocks? What happens if two miners find two blocks of the same height and broadcast them almost at the same time? One of those blocks will become stale, but before that happens, who is making those mempool blocks?

As for the malicuos miner, there could be block size limits placed on the mempool blockchain just as is done on the core blockchain. But we would probably build the flexibility in from the start so that we do not repeat the current issue with the block limit. Yes, malicous intent needs to be addressed, but I think we can work the details out for that issue prior to implementation. In general it i not in a miners best interests to mess up bitcoin...

It isn't but there are always people who want to see the world burn... Or just mess up Bitcoin because they can gain from Bitcoin crashing.

Thanks for the comments :-)

No problem.

[2c0de]

Satoshi walks into a bar and asks for a beer. Dorian walks into a Walmart and asks for a TV. The barman asks for 0.654 BTC The cashier asks for a 7.46 BTC. Satoshi activates the magic last miner key, a green light appears at the bar, Satoshi walks out with a beer. Dorian activates the magic last miner key, a green light appears at the cashier's screen, Dorian walks out with a TV.

[2c0de]

Problem #1: Why did the confirmation (green light) disappear?
Problem #2: Why do the real miners suddenly collect more fees? Where does the money originate from?

[draad]

Problem #1: Why did the confirmation (green light) disappear?
Problem #2: Why do the real miners suddenly collect more fees? Where does the money originate from?

#1. Green light disappears when a new block is generated by a new miner. If you take a look at the bitcoin-NG whitepaper you can see how this would be done with coreblocks only occurring about 10 minutes apart and small blocks generated between them. (I am not involved with Bitcoin-NG at all I am just partially stealing there design). 

#2. Real miners only get fees when they include a transaction into a block. If the block is almost full then the number of transactions they can include is fixed. So they need to maximize the fee per transaction. Thus in the example, because the number of transactions being placed in the core is reduced, the the actual fee/per transaction increases; and thus miners make more money and thus have incentive to do this.   

[draad]

So what about a race condition, stale blocks? What happens if two miners find two blocks of the same height and broadcast them almost at the same time? One of those blocks will become stale, but before that happens, who is making those mempool blocks?

Same as always, the longest chain wins.

[achow101]

So what about a race condition, stale blocks? What happens if two miners find two blocks of the same height and broadcast them almost at the same time? One of those blocks will become stale, but before that happens, who is making those mempool blocks?

Same as always, the longest chain wins.

That's not what I'm talking about. I'm talking about the ~10 minutes in which there are two longest chains because a block has not been found yet that builds on either of the two competing blocks.

[draad]

So what about a race condition, stale blocks? What happens if two miners find two blocks of the same height and broadcast them almost at the same time? One of those blocks will become stale, but before that happens, who is making those mempool blocks?

Same as always, the longest chain wins.

That's not what I'm talking about. I'm talking about the ~10 minutes in which there are two longest chains because a block has not been found yet that builds on either of the two competing blocks.

Both of the competing miners would be creating their own mempool blocks. Yes at that point no-one really knows which chain will win. However if you can identify the transaction that you are interested in exists in both active chains then you know that the transaction has been confirmed in the mempool blockchain. Which one wins is unimportant to you at that stage. Does this answer your question?

[achow101]

So what about a race condition, stale blocks? What happens if two miners find two blocks of the same height and broadcast them almost at the same time? One of those blocks will become stale, but before that happens, who is making those mempool blocks?

Same as always, the longest chain wins.

That's not what I'm talking about. I'm talking about the ~10 minutes in which there are two longest chains because a block has not been found yet that builds on either of the two competing blocks.

Both of the competing miners would be creating their own mempool blocks. Yes at that point no-one really knows which chain will win. However if you can identify the transaction that you are interested in exists in both active chains then you know that the transaction has been confirmed in the mempool blockchain. Which one wins is unimportant to you at that stage. Does this answer your question?

Yes.

[draad]

Problem #1: Why did the confirmation (green light) disappear?
Problem #2: Why do the real miners suddenly collect more fees? Where does the money originate from?

Hi 2c0de,

I was wondering if I completely answered your question? I do not see a flaw in my position, do you still see a magic money flaw here?