Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: TierNolan on March 19, 2015, 03:55:53 PM



Title: Transactions which spend coinbase outputs early
Post by: TierNolan on March 19, 2015, 03:55:53 PM
What happens if a transaction is submitted that spends a coinbase output early?

Is the transaction just placed in the orphan pool and potentially lost by the time the timeout ends?  In theory, they should be higher priority than normal orphans, since they are guaranteed to be valid eventually.


Title: Re: Transactions which spend coinbase outputs early
Post by: DeathAndTaxes on March 20, 2015, 03:05:04 AM
It is an invalid txn.  It won't be relayed or included in a block by default.  If a miner did include it in a block then the block is invalid.  Similar to a txn with a future nlocktime.


Title: Re: Transactions which spend coinbase outputs early
Post by: TierNolan on March 20, 2015, 06:32:07 PM
It is an invalid txn.  It won't be relayed or included in a block by default.  If a miner did include it in a block then the block is invalid.  Similar to a txn with a future nlocktime.

Does the node just drop it though?  It could be placed in the orphan pool and then added to the memory pool once the timeout has expired (and thus relayed).


Title: Re: Transactions which spend coinbase outputs early
Post by: CIYAM on March 20, 2015, 06:35:30 PM
Does the node just drop it though?  It could be placed in the orphan pool and then added to the memory pool once the timeout has expired (and thus relayed).

Interesting - as a potential attack vector it wouldn't be so easy but I'd guess it still could be such a potential if the txs were relayed wouldn't it?


Title: Re: Transactions which spend coinbase outputs early
Post by: TierNolan on March 20, 2015, 06:49:31 PM
Interesting - as a potential attack vector it wouldn't be so easy but I'd guess it still could be such a potential if the txs were relayed wouldn't it?

I don't think it is any easier than other transactions.  In effect, it would be a part of the memory pool with a height restriction.  If you spend the coinbase output twice, then the 2nd one wouldn't be relayed, as it is a doublespend.


Title: Re: Transactions which spend coinbase outputs early
Post by: CIYAM on March 20, 2015, 07:10:19 PM
If you spend the coinbase output twice, then the 2nd one wouldn't be relayed, as it is a doublespend.

Yes - good point - so nice to have a topic with smart people rather than just ad sig posts (but I guess they might be about to follow).


Title: Re: Transactions which spend coinbase outputs early
Post by: StephenMorse on March 24, 2015, 04:02:22 AM
What happens if a transaction is submitted that spends a coinbase output early?

Is the transaction just placed in the orphan pool and potentially lost by the time the timeout ends?  In theory, they should be higher priority than normal orphans, since they are guaranteed to be valid eventually.

I don't think they are guaranteed to be valid eventually. For example, it could be the case that the first block in which the coinbase becomes spendable (so 100 blocks after the block it is mined in) sends the funds it to a different address than the one you saved in your mempool.

AFAIK, the current invariant of the mempool is that all the transactions could be gathered up into one big block (assuming the 1 MB restriction doesn't apply). This would have to change to allow relaying of prematures coinbase spends. The block building code would have to take this into account as well, making sure it doesn't build a block that will be invalid.

I like the idea of being able to use this for things like award pool payouts immediately after a block is solved, though.


Title: Re: Transactions which spend coinbase outputs early
Post by: TierNolan on March 24, 2015, 12:03:19 PM
I don't think they are guaranteed to be valid eventually. For example, it could be the case that the first block in which the coinbase becomes spendable (so 100 blocks after the block it is mined in) sends the funds it to a different address than the one you saved in your mempool.

That can only happen with a double spend though.


Title: Re: Transactions which spend coinbase outputs early
Post by: Cryddit on March 26, 2015, 06:00:47 PM

AFAIK, the current invariant of the mempool is that all the transactions could be gathered up into one big block (assuming the 1 MB restriction doesn't apply). This would have to change to allow relaying of prematures coinbase spends. The block building code would have to take this into account as well, making sure it doesn't build a block that will be invalid.


I believe the restriction is more than that; the mempool transactions could be gathered up into one big block (if they fit under the blocksize restriction) *AND* that hypothetical block would be valid as the very next block in the chain. 




Title: Re: Transactions which spend coinbase outputs early
Post by: DeathAndTaxes on March 26, 2015, 07:50:00 PM
Well I think that was his point right.   Valid is sort of implied (one big block = one big valid block).  The txns in the memory pool could all be included in the next block (outside of size limits).  In fact the core client assumes that by using the memorypool as the as the source for Getblocktemplate.   Partially signed txns and txns with future nlocktimes are not stored in the memory pool (and thus not relayed) for the same reasons.  They aren't valid for inclusion in the next block.


Title: Re: Transactions which spend coinbase outputs early
Post by: Cryddit on March 26, 2015, 08:32:51 PM

I believe the restriction is more than that; the mempool transactions could be gathered up into one big block (if they fit under the blocksize restriction) *AND* that hypothetical block would be valid as the very next block in the chain.  


I have reviewed the source code.  A transaction which contains a spend of a coinbase, when that coinbase's spendable height has not yet arrived, is not accepted into the mempool.  It is not relayed, and the peer reporting it is marked as misbehaving.

Other tx which appear to be valid (and do not actively conflict with tx in the mempool or in the chain as accepted so far),  are retained (as "orphans") even if the transactions that create the txOuts they spend have not yet been received.   "Orphans" would not be valid appearing without their parent transactions in the next block, and they appear to be the only kind of transaction that is retained in the memory pool even though there is any question as to whether it will eventually become valid.  

I don't see a compelling reason why a premature spend of a coinbase should not be treated the same way as an orphan tx.  Both are essentially tx that could (or will) become valid if (or when) some event occurs.  The amount of memory reserved for orphans is limited to prevent transmitting lots of orphans from becoming a memory-exhaustion attack.  

One relevant difference is that it is at least possible that an orphan tx will be in the very next block, whereas a tx that prematurely spends a coinbase would under no circumstances be valid in the very next block.



Title: Re: Transactions which spend coinbase outputs early
Post by: Cryddit on March 26, 2015, 08:59:05 PM
Well I think that was his point right.   Valid is sort of implied (one big block = one big valid block). 

The thing I was talking about was not validity, it was about it being valid AS THE VERY NEXT BLOCK.  A coinbase spend is a valid transaction and does not conflict with any other tx in the mempool, and could eventually appear in the block chain.  But if it can't appear IN THE VERY NEXT BLOCK the implementation just ignores it except to start doubting the sanity of the node reporting it. 

Everything else that could become valid and is not conflicting with anything already in the memory pool gets accepted, even if the conditions for it to be valid have not yet occurred (ie orphan tx which spend outputs not yet visible in any other mempool tx nor in the block chain).

The mempool then is tx that COULD be valid in the very next block - but at any given moment, depending on which txOuts appear in other tx the mempool, you may have tx in there that couldn't appear in a block made exclusively out of your own mempool. 




Title: Re: Transactions which spend coinbase outputs early
Post by: DeathAndTaxes on March 26, 2015, 09:51:46 PM
Well I think that was his point right.   Valid is sort of implied (one big block = one big valid block).  

The thing I was talking about was not validity, it was about it being valid AS THE VERY NEXT BLOCK.  

Yes I understand and it was implied in the post you replied to as well.  That post can't be read any other way.  Anyways moving on.

Quote
Other tx which appear to be valid (and do not actively conflict with tx in the mempool or in the chain as accepted so far),  are retained (as "orphans") even if the transactions that create the txOuts they spend have not yet been received.

I do not believe that is correct. To be included in the memory pool the inputs of a txn must reference an output in the utxo.  The sending peer is not penalized as there are legitimate reasons but the txn is not added to the memory pool.

https://github.com/bitcoin/bitcoin/blob/687f10d9ec3548f13f929ca14cd813a0919639ec/src/main.cpp#L1021
https://github.com/bitcoin/bitcoin/blob/687f10d9ec3548f13f929ca14cd813a0919639ec/src/main.cpp#L1380


Title: Re: Transactions which spend coinbase outputs early
Post by: Cryddit on March 26, 2015, 10:47:44 PM
Well I think that was his point right.   Valid is sort of implied (one big block = one big valid block).  

The thing I was talking about was not validity, it was about it being valid AS THE VERY NEXT BLOCK.  

Yes that was understood and the person and it was understood from the post you replied to as well.  That post can't be read any other way.

Quote
Other tx which appear to be valid (and do not actively conflict with tx in the mempool or in the chain as accepted so far),  are retained (as "orphans") even if the transactions that create the txOuts they spend have not yet been received.

I do not believe that is correct. To be included in the memory pool the inputs of a txn must reference an output in the utxo.  The sending peer is not penalized as there are legitimate reasons but the txn is not added to the memory pool.

https://github.com/bitcoin/bitcoin/blob/687f10d9ec3548f13f929ca14cd813a0919639ec/src/main.cpp#L1021
https://github.com/bitcoin/bitcoin/blob/687f10d9ec3548f13f929ca14cd813a0919639ec/src/main.cpp#L1380


Hm.  Looks like you're right.  But it does keep orphans.  

https://github.com/bitcoin/bitcoin/blob/687f10d9ec3548f13f929ca14cd813a0919639ec/src/main.cpp#L495

I guess they aren't part of "Memory pool?"  

Edit:  I found it.

https://github.com/bitcoin/bitcoin/blob/687f10d9ec3548f13f929ca14cd813a0919639ec/src/main.cpp#L3869

Is where transactions get moved from the orphan map to the memory pool when the tx that complete their inputs are received. 

I had been under the mistaken impression that the orphan map referred to tx which were already stored in the mempool; this makes it clear that they are stored in a separate structure.


Title: Re: Transactions which spend coinbase outputs early
Post by: StephenMorse on March 27, 2015, 05:08:32 AM
If this feature were implemented, the pre-mature spends of coinbase outputs would probably have to be stored in an entirely different structure (similar to the COrphanTx and mapOrphanTransactions), in order to preserve the mempool for block building. Or, you could probably also just make a CMempoolTx struct that stores the transaction and any meta data about how/when it should be included in a block.

The thing I'm not sure about is transaction relaying. Have any versions of the software penalized nodes for forwarding pre-mature spends of coinbase transactions? Because, if so, this feature could end up getting nodes banned. I've looked through a few versions, can't seem to find any misbehavior points being applied for doing this, but I certainly didn't check every version.

Coinbase transactions can have many outputs, and each one of those outputs can be spent by a very large transaction, so the same (or similar) sorts of rate limiting that go into what can be accepted to the mempool has to be applied to these.

I'd be interested to see what some core devs think about this, I'll see if anyone in IRC can weigh in.


Title: Re: Transactions which spend coinbase outputs early
Post by: StephenMorse on March 27, 2015, 05:48:47 AM
Consensus on the bitcoin-dev IRC chat is that no other versions have penalized forwarding of pre-mature spends of coinbase transactions, but that there doesn't seem to be any real use-case for such a feature.


Title: Re: Transactions which spend coinbase outputs early
Post by: TierNolan on March 27, 2015, 04:41:27 PM
Consensus on the bitcoin-dev IRC chat is that no other versions have penalized forwarding of pre-mature spends of coinbase transactions, but that there doesn't seem to be any real use-case for such a feature.

I was thinking that it could be used to make the coinbase smaller by paying to a transaction rather than a list of addresses.

This would make automatic mining software easier.  You wouldn't need to hash a 10-20kB coinbase to update the extraNonce.

It requires a way to "pay to transaction", which isn't supported. 

Miners could wait the 100 blocks and as long as at least one of them remembers the transactions, then it doesn't matter about the delay.

A hard-fork rule would be that transactions are allowed to spend the coinbase in the current block.  The outputs of those transactions are considered coinbase outputs for the purposes of the 100 block rule.

The could add an opcode OP_HASH_THIS_TRANSACTION. 

It hashes the current transaction

    TXID for this input is set to 32 zeros
    scriptSig is set to zero length

The scriptPubKey would be

Code:
<hash of tx> OP_HASH_THIS_TRANSACTION OP_EQUALVERIFY

The scriptSig would be empty.  The scriptPubKey does the verification on its own.

Since the new opcode ignores the TXID it can be set after the coinbase is setup.


Title: Re: Transactions which spend coinbase outputs early
Post by: Cryddit on March 27, 2015, 04:55:09 PM
If you'd like to hard fork to deal with the problem of updating the extranonce, the "correct" solution would be to add it to the header.  But that'll never happen.

That would be the correct solution if CPU or GPU mined.  There is no way in hell that there'd ever be miner support for something that would destroy the value of their investment in ASICs.



Title: Re: Transactions which spend coinbase outputs early
Post by: TierNolan on March 27, 2015, 05:58:52 PM
If you'd like to hard fork to deal with the problem of updating the extranonce, the "correct" solution would be to add it to the header.  But that'll never happen.

Agreed.  I asked the question before before making sure everything was actually possible.

I think a hard fork changing the Merkle tree would be better.  It would allow a second header at the top level of the Merkle tree.

Quote
That would be the correct solution if CPU or GPU mined.  There is no way in hell that there'd ever be miner support for something that would destroy the value of their investment in ASICs.

A second header would mean that ASICs are not affected.

The merkle root field would be filled with double_hash(old_merkle_root | aux_header).


Title: Re: Transactions which spend coinbase outputs early
Post by: DeathAndTaxes on March 27, 2015, 07:15:00 PM
There is no way in hell that there'd ever be miner support for something that would destroy the value of their investment in ASICs.

Luckily that is not correct.  A while back I proposed a way to expand the header while still maintaining compatibility with existing ASICs (but would require a hard fork and updated mining software). Maybe I can find a link.  The header will eventually need to be updated when the Unix time value overflows the 32 bit integer but we have a 'little' while before that happens.


Title: Re: Transactions which spend coinbase outputs early
Post by: DannyHamilton on March 27, 2015, 08:22:10 PM
- snip -
A while back I proposed a way to expand the header while still maintaining compatibility with existing ASICs (but would require a hard fork and updated mining software). Maybe I can find a link.
- snip -

This one?

https://bitcointalk.org/index.php?topic=626377.msg6963192#msg6963192

SHA-256 handles messages larger than 512 bits by breaking the message into blocks that are 512 bits long.  The first block input is fed into the hashing function and that output (h0) is combined with the second input block and that output (h1) is combined with the third input, etc, etc, until all inputs have been hashed. This is how a message of arbitrary size (even say a 4GB ISO) can be reduced to a single 256 bit value.

The "zeroes" are simply spacing bits.  They could be anything as long as the format of the final block (partial hash not bitcoin blocks) remains the same length and ends with a 32 bit nonce.

The current bitcoin block header is 640 bits long
This means the first 512 bits is hashed and the output of the function is stored h0
The next 128 bits is then hashed with the output of h0 to produce h1.  This is then hashed again because Bitcoin uses SHA256d.

Now ASICS are "dumb" they really have no idea what they are hashing.  Since h0 remains relatively static (the nonce is in the second block), the mining software (local miner) will compute the h0 partial hash and feed that plus the 96 bits in the second block to the ASIC.  The ASIC appends the 32 bit counter (nonce) and computes the hash incrementing internally as needed and returning results.

So to remain compatible with existing ASICS requires two things:
a) a 256 bit h(x) partial hash value
b) the last block (partial hash input) must consist of exactly 96 bits + a 32 bit nonce.

Anything different would result in a format that existing ASICS don't understand.   

The comment about "must be zero do not use" just means it is padding and must be consistent in the protocol it could be anything all zeroes, all ones, ASCII code for "Satoshi Rules 123...." as long as it is 48 bytes long.  There is no actual requirement that the protocol/standard must only use zeroes.  If a different format was used the padding would be smaller or longer to to ensure that 96 bits + a 32 bit nonce are the only elements in the final partial hash.  This way no matter how large you want the header to be it will still hash down to a h(x) partial hash plus 96 bits + a 32 bit nonce and be compatible with existing hardware.

Personally I have considered a change like this but I would propose a modified structure.  I would use a 64 bit nonce in the last segment (replacing the nonce, nonce2, and nonce3).  This would support for miners as powerful as 18,446 PH/s without overflowing the nonce.   The larger nonce field could be supported by existing hardware in a backwards compatible mode in a similar manner to how 32 bit values are stored in 64 bit CPU registers.   The upper 32 bits would simply be zeroed.   The existing miners (who only understand the concept of a 32 bit nonce) would simply hash a value which ends with 00000000 plus a 32 bit nonce.  The local miner (cgminer) could increment the upper 32 bits to provide support for higher hashrate hardware.  In time ASIC developers would release "enhanced" hardware which uses a larger nonce space internally to handle higher throughput.


Title: Re: Transactions which spend coinbase outputs early
Post by: TierNolan on March 27, 2015, 09:18:47 PM
Luckily that is not correct.  A while back I proposed a way to expand the header while still maintaining compatibility with existing ASICs (but would require a hard fork and updated mining software). Maybe I can find a link.  

Changing the merkle tree is easiest if you are going to hard-fork anyway.

Quote
The header will eventually need to be updated when the Unix time value overflows the 32 bit integer but we have a 'little' while before that happens.

That is pretty easy to fix without changing the size of the field.  Define the timestamp field as the 4 least significant bytes of the timestamp.  The MSBs can be worked out by selecting the value so that the time step between blocks is smallest.

This will work unless the time between 2 blocks is more than ~70 years.

It needs a protocol fork, but no change to mining equipment.


Title: Re: Transactions which spend coinbase outputs early
Post by: DeathAndTaxes on March 27, 2015, 10:05:16 PM
This one?

https://bitcointalk.org/index.php?topic=626377.msg6963192#msg6963192

Thanks for finding it.  That was the thread.  I think this post is clearer:
https://bitcointalk.org/index.php?topic=626377.msg6975690#msg6975690


Title: Re: Transactions which spend coinbase outputs early
Post by: StephenMorse on March 30, 2015, 04:07:53 AM
If the point is to make getting more work easier by not having to hash 10-20 Kb of coinbase data, then why not just use transaction #2 in the block as the place where the extraNonce is stored rather than transaction #1 (the coinbase tx)? Transaction #2 can just be a really simple transaction with an OP_RETURN output for including extranonce data. Any input signed would have to be signed with SIGHASH_NONE, so that the scriptPubKey with the OP_RETURN output can be modified by the miner.

This essentially makes use of the transaction's malleability to change the merkle root and get more nonce space. Once it's included in the block, though, it's no longer malleable.

In the stratum protocol, this would just mean that the list of hashes that create the merkle branch would just have to be used slightly differently, with the first element being hashed on the left and the rest of the elements being hashed on the right. A new parameter could even be introduced that describes how the merkle branch should be used (the specific left-right pattern), and the default could be all on the right (like for the coinbase).

One downside might be that it means the miners can't look at the coinbase to see that they are getting paid, because they don't even have access to it. But I don't know of any mining programs that actually check the data like that before doing work. Another would be the slight waste of space, but I think this is pretty negligible. The coinbase's input could be seen as a slight waste of space as well, since it could be as small as a few bytes. The extraNonce data has to be put somewhere, may as well put it somewhere that's not connected to a massive transaction.


Title: Re: Transactions which spend coinbase outputs early
Post by: DeathAndTaxes on March 30, 2015, 05:10:11 PM
If the point is to make getting more work easier by not having to hash 10-20 Kb of coinbase data, then why not ...

The 'work' is recreating the merkle tree not the single hash of a single tx.  A change to the tx set will require recomputing the merkle tree.   It really isn't that much of a problem.  If the block header is updated it would be very smart to put a extanonce in the header itself to prevent needing to recompute the merkle tree but hashing even the full merkle tree is not that much of a load compared to the cost of ECDSA validation.  So I would chalk it up as a "it would be nice to do if we are updating the header format anyways".


Title: Re: Transactions which spend coinbase outputs early
Post by: StephenMorse on March 30, 2015, 06:32:51 PM
You don't need to recompute the whole merkle tree, though, just one branch of it (the branch that connects the transaction with the extra nonce in it). Recalculating this requires hashing the transaction with extraNonce updated and 64*(height of merkle tree) bytes. So, updating the merkle root by modifying the 20 kb coinbase transaction in a block with 2000 transactions would require hashing

20,000 + 64*log2(2000) = 20.7 kb

But modifying the extranonce in a transaction of size 300 bytes only requires hashing

300 + 64*log2(2000) = 1 kb

Either way, I haven't seen any numbers that show that a CPU can't recalculate merkle roots as fast as an ASIC can exhaust a whole 2^32 nonce range. Is that actually an issue?

I agree that it would be better to have the extra nonce right in the header to avoid recalculating the merkle root entirely, but while I find the extended header idea interesting, I'm not sure it's in sufficient need that it would be worth all the extra work and support.

Related: http://bitcoin.stackexchange.com/questions/32603/why-didnt-satoshi-make-the-nonce-space-larger


Title: Re: Transactions which spend coinbase outputs early
Post by: DeathAndTaxes on March 30, 2015, 07:16:06 PM
Either way, I haven't seen any numbers that show that a CPU can't recalculate merkle roots as fast as an ASIC can exhaust a whole 2^32 nonce range. Is that actually an issue?

No and probably will not be for sometime if ever.

Quote
I agree that it would be better to have the extra nonce right in the header to avoid recalculating the merkle root entirely, but while I find the extended header idea interesting, I'm not sure it's in sufficient need that it would be worth all the extra work and support.

A hard fork and new block header format just to move the extranonce to the header?  No it probably isn't worth it.  My post was more to illustrate that expanding the header is possible while still being backwards compatible with existing ASIC hardware.  Eventually the header will probably need to be updated for one reason or another.