Why does the Bitcoin protocol send transactions twice?

[fevirfevir]

TL;DR: Why are transactions (240 bytes) added in the block body, instead of the SHA256 hash (256 bits) of the transaction?

From what I understand, a transaction (approx 240 bytes) is 'broadcasted' to the everyone in the network through inv-messages. A miner collects transactions, and adds 1 or more in a block(body). After finding the hash, a miner 'broadcasts the block (header and body). In essence, transactions are send twice. First the actual transaction, second, a cumulated set of transactions.

If a transactions is broadcasted and everyone receives the transaction, why not add a hash (256 bits) of the transaction in a block body? Propagation time of a block would be similar, the number of transactions send in a block is increased by a factor 8-ish.

One argument could be that not all mempools are synced. However, assuming that a transactions are broadcasted much faster than a block, part of the ruleset could be that a miner only can create a block from  transactions received. In practise, if a miner sends a block to its direct peers (in the P2P network) and the transaction is not known by these nodes, than the block is not valid. Since a block creation time is (on average) 10 minutes, transactions added to the block have plenty of time to propagate.

[shorena]

You probably want to read this -> https://gist.github.com/gavinandresen/e20c3b5a1d4b97f79ac2

[CIYAM]

The main reason why this isn't very feasable at the moment is due to tx id malleability (as the ECDSA signature is included in the tx id and the same tx can be signed with a different valid signature because ECDSA approach used by pretty much all Bitcoin wallets includes a "random" number).

The introduction of SegWit gets rid of this problem by removing the signature information from the tx (it is provided separately in a "witness block") so once this is in place it will be possible to make the sort of bandwidth size reductions you are suggesting (and that is planned).

That's not to say that such an approach couldn't be used currently - but it could be susceptible to attacks whilst the tx id malleability issue remains (there are other advantages to removing the malleability as well which is why SegWit is higher priority at this stage).

[fbueller]

Because your node requests them twice.

A newly broadcast transaction has to be downloaded & checked. Same with new blocks. You can't know what transactions are inside a block without downloading the block. Ergo, you request it twice.

One argument could be that not all mempools are synced. However, assuming that a transactions are broadcasted much faster than a block, part of the ruleset could be that a miner only can create a block from  transactions received. In practise, if a miner sends a block to its direct peers (in the P2P network) and the transaction is not known by these nodes, than the block is not valid.

The answer *is* that not all mempools are synced. Mempools are not consistent, and a miner could have special arrangements to mine transactions which aren't published over the p2p network (hence the special arrangement).

The fact that the contents of a block is unpredictable is an important property for anti-censorship by the way; If a cartel of miners tries to censor low-fee-paying transactions, another miner can still include those transactions. That much is a feature, not a bug.

[elitecstrike]

Maybe for checkout.

[fevirfevir]

You probably want to read this -> https://gist.github.com/gavinandresen/e20c3b5a1d4b97f79ac2

Thanks, an interesting read.

The main reason why this isn't very feasable at the moment is due to tx id malleability (as the ECDSA signature is included in the tx id and the same tx can be signed with a different valid signature because ECDSA approach used by pretty much all Bitcoin wallets includes a "random" number).

I see. So if I get it right, this could lead to a double spend, by creating two similar (inputs, outputs # of coins) transactions with separate txids?

[piotr_n]

If a transactions is broadcasted and everyone receives the transaction, why not add a hash (256 bits) of the transaction in a block body? Propagation time of a block would be similar, the number of transactions send in a block is increased by a factor 8-ish.

The problem is that not everyone receives the transaction.
And even if they do, they may not store it in their memory pool.

Having learning about the new block (that these days usually contains over thousand txs), you will most likely always be missing some of them - even if your node has been online for weeks.
You need each single transaction to verify the new block and that is why the protocol sends you all the txs along with each block.

There is obviously a space for some optimization here, but the problem is actually quite complex and I think that is why nobody has implemented any solution for it yet.

Just think of a specific scenario:
Your node receives a notification announcing a new block. You know it is a new block, but you still don't know what transactions are inside it. So you need to ask the peer who knows that block already for the list of the tx_ids inside it - best case scenario, it will come back after the ping time, plus the time needed to transfer the data (which often would exceed 100KB).
Now, having the list of tx_ids already, the odds are that you will miss some of the txs and you will need to ask the peer for them. This requires another ping time plus the time to transfer back the data - and that is the problem.

What I am saying is that in this scenario you need two request to acquire the content of the block: first for txs list, second for txs you didn't know.
You cannot do these two in parallel - need to proocess the answer for the first request, to issue the second one.

And currently you need only one request: asking for the entire block, with all the transactions.

It obviously depends on the specifics of the connection to the peer (ping vs bandwidth), but in general it is questionable whether the scenario with two requests would give you the entire content of the block faster (aka block propagation speed). It would most likely require to send less data, but causing an additional ping-long delay for the additional request.

[kn_b_y]

The main reason why this isn't very feasable at the moment is due to tx id malleability (as the ECDSA signature is included in the tx id and the same tx can be signed with a different valid signature because ECDSA approach used by pretty much all Bitcoin wallets includes a "random" number).

I see. So if I get it right, this could lead to a double spend, by creating two similar (inputs, outputs # of coins) transactions with separate txids?

No. That can't happen. Only one of the transactions can be added to the blockchain.

The danger is that a maliciously malleated transaction (and not the one you created) will be confirmed, meaning that the txid won't be as you expected.

In more esoteric uses of bitcoin it is sometimes necessary to create and presign transactions which spend from transactions that have not yet been broadcast. This is known as chaining.

In such cases the txid of transactions appearing earlier in the chain must be guaranteed (ie unalterable). If a txid is altered all successive transactions in the chain will fail (because they will refer to outputs from transactions with txids that do not exist).

[fevirfevir]

No. That can't happen. Only one of the transactions can be added to the blockchain.

The danger is that a maliciously malleated transaction (and not the one you created) will be confirmed, meaning that the txid won't be as you expected.

In more esoteric uses of bitcoin it is sometimes necessary to create and presign transactions which spend from transactions that have not yet been broadcast. This is known as chaining.

In such cases the txid of transactions appearing earlier in the chain must be guaranteed (ie unalterable). If a txid is altered all successive transactions in the chain will fail (because they will refer to outputs from transactions with txids that do not exist).

Thanks! That's helpful, appreciate the explanation.