Consensus supported sequence numbers

[TierNolan]

Sequence numbers are currently not enforceable by Bitcoin.  If two transactions spend the same outputs, then the miner is supposed to pick the transaction (input) with the higher sequence number.   This cannot be enforced and so, the miner would probably pick the one with the highest fees.

They could be enforced, if transaction replacement was possible in blocks.  With a hard fork, the rule could change so that transactions with locktimes in the future are allowed into the blocks.  The locktime would prevent the outputs from being spent.  Afterwards, if someone broadcasts a transaction which double spends some of the inputs and where all those inputs have a higher sequence number, then it would effectively cancel the original transaction.  Once the locktime is reached, then the outputs of the highest sequence transaction could be spent.

It is possible to do this (less efficiently) with a soft fork.

The anchor/multisig transaction would have N outputs of the following form:

Code:

IF
    <now + 40 days> CLTV DROP <refund public key> OP_CHECKSIG
ELSE
    <N> <N public keys> <N> CHECKMULTISIGVERIFY 
END

The state update transactions would be N input and N output transactions with the following outputs.

Code:

IF
    <now + 30 days> CLTV DROP <new owner's public key> OP_CHECKSIG
ELSE
    <sequence_number> OP_CHECKSEQUENCEVERIFY
END

They would spends the first N outputs of the anchor transaction and spend them to N new outputs.  Each of the state update transactions can act as a link.  As long as it is spending a link with a lower sequence number, then it is a valid transaction.  Once the 30 day CLTV passed, the result output can be spent.

There needs to be an allowance for at some extra inputs into the transaction, in order to pay fees.  This could work similar to SIGHASH_ANYONE_CAN_PAY.

OP_CHECKSEQUENCEVERIFY means that you can effectively use the parent transaction's scriptPubKeys instead of this transaction's.  Each lower sequence number transaction uses the public key from its parent in a chain back to the root/anchor transaction.

In psuedo-code, it does the following

Note: max_sequence starts at 0xFFFFFFFF

if (sequence_number >= max_sequence)
  return FAIL;

If (txid of first N inputs aren't equal)
  return FAIL;

parent = getTransaction(TxIn[n].getPrevTransaction())

if (txid of first N inputs of parent aren't equal)
  return FAIL;

stack.pop(); // remove <sequence number>
stack.pop(); // remove <1> for IF

subScript.max_sequence = sequence_number
subScript.scriptSig = stack.copy()
subScript.scriptPubKey =parent.getgetPubKey()

transactionCopy = transaction with first N txids replaced with grandparents txid

if (subScript.execute(transactionCopy) == FAIL)  // Use transactionCopy for all scriptSig operations
  return FAIL;

proceed as if it was a NOP

[jl777]

Sequence numbers are currently not enforceable by Bitcoin.  If two transactions spend the same outputs, then the miner is supposed to pick the transaction (input) with the higher sequence number.   This cannot be enforced and so, the miner would probably pick the one with the highest fees.

This does not sound good...

How can micropayment channels be reliable if sequence numbers are ignored? As a workaround, maybe adding an extra satoshi for each new payment would be a practical workaround.

With micropayments, the same inputs are "used" over and over, with just the payment amount, signature and sequenceid changing. Without sequenceid enforcement to ensure the prior payments are invalid, then if an earlier one is broadcast, it seems to mean the later wont cant be counted on to be valid.

Does this mean that there no point to using sequenceid's at all? It seems that all protocols need to be secure without any reliance on sequenceids and the assumption that any offchain transfer of a signed tx could be broadcast.

James

[TierNolan]

This does not sound good...

How can micropayment channels be reliable if sequence numbers are ignored? As a workaround, maybe adding an extra satoshi for each new payment would be a practical workaround.

None of the latest channel proposals actually use sequence numbers, exactly for this reason.  The only thing sequence numbers are used for is to turn on and off locktime (and also to opt-in to replace by fee).

The original channels were one way.  Bob would have lots of transactions, and each one would pay him more than the last.  He could broadcast and earlier one, but that would mean he gets less money.

The Lightning network uses revoke codes.  You can broadcast a revoked transactions, but if you do, the other person gets 100% of the money in the channel.

I was trying to come up with a way to get sequence numbers to work.  For it to be enforceable, transactions have to frozen for a while to give later blocks a chance to revoke them.

[jl777]

This does not sound good...

How can micropayment channels be reliable if sequence numbers are ignored? As a workaround, maybe adding an extra satoshi for each new payment would be a practical workaround.

None of the latest channel proposals actually use sequence numbers, exactly for this reason.  The only thing sequence numbers are used for is to turn on and off locktime (and also to opt-in to replace by fee).

The original channels were one way.  Bob would have lots of transactions, and each one would pay him more than the last.  He could broadcast and earlier one, but that would mean he gets less money.

The Lightning network uses revoke codes.  You can broadcast a revoked transactions, but if you do, the other person gets 100% of the money in the channel.

I was trying to come up with a way to get sequence numbers to work.  For it to be enforceable, transactions have to frozen for a while to give later blocks a chance to revoke them.

So sequence numbers are really just a binary flag? 0xffffffff means locktime is ignored, anything else means locktime is honored.

I guess once sequence numbers invalidate other tx with earlier ones, it opens up many edge cases about when they are broadcast vs confirmed

James

[TierNolan]

So sequence numbers are really just a binary flag? 0xffffffff means locktime is ignored, anything else means locktime is honored.

Yes, but there is a plan to turn them into a system to support relative locktimes instead.

If the sequence is less than 0x80000000 (and some other conditions), then it will be interpreted as a relative block height or relative timestamp. 

It is like CLTV, but it is relative.  You can pay to

Code:

IF
    <144 blocks> CSV DROP <alice pub key> CHECKSIG
ELSE
    HASH160 <hash(revoke code)> EQUALVERIFY <bob pub key> CHECKSIG

That means that alice must wait at least 144 blocks to spend that output.  If Bob has the revoke code, he will always have at least 1 day to claim his output.

He doesn't have to use CLTV and lock it for 30 days but he still is guaranteed a 144 block window.

This is the pull request:

https://github.com/bitcoin/bitcoin/pull/7184

and BIP

https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki

I guess once sequence numbers invalidate other tx with earlier ones, it opens up many edge cases about when they are broadcast vs confirmed

The original proposal only affected memory pools, so it isn't secured by POW.  If locked transactions could be included in blocks, then the rule could be a consensus rule.

[jl777]

So sequence numbers are really just a binary flag? 0xffffffff means locktime is ignored, anything else means locktime is honored.

Yes, but there is a plan to turn them into a system to support relative locktimes instead.

If the sequence is less than 0x80000000 (and some other conditions), then it will be interpreted as a relative block height or relative timestamp. 

It is like CLTV, but it is relative.  You can pay to

Code:

IF
    <144 blocks> CSV DROP <alice pub key> CHECKSIG
ELSE
    HASH160 <hash(revoke code)> EQUALVERIFY <bob pub key> CHECKSIG

That means that alice must wait at least 144 blocks to spend that output.  If Bob has the revoke code, he will always have at least 1 day to claim his output.

He doesn't have to use CLTV and lock it for 30 days but he still is guaranteed a 144 block window.

This is the pull request:

https://github.com/bitcoin/bitcoin/pull/7184

and BIP

https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki

I guess once sequence numbers invalidate other tx with earlier ones, it opens up many edge cases about when they are broadcast vs confirmed

The original proposal only affected memory pools, so it isn't secured by POW.  If locked transactions could be included in blocks, then the rule could be a consensus rule.

It is surprising that sequence numbers dont work as it is documented... I had thought CHECKSEQUENCEVERIFY was already on mainnet, but if sequenceid's arent PoW enforced, it probably is still ok, but need to be very careful with assumptions as to who gets fully signed tx with sequenceids in them. without any enforcement, then currently CSV just allows the signer of a CSV output to do an invalid spend?

I want to make sure my current usage is safe. What I do is set sequenceid to 0xffffffff for all tx unless it uses CLTV, then I set it based on the current unixtime. I assume the same +/-2 hours variance is allowed? So I think I need to pad all the times by 2 hours to prevent mismatched timestamps from timing someone out inadvertently

James

[TierNolan]

I want to make sure my current usage is safe. What I do is set sequenceid to 0xffffffff for all tx unless it uses CLTV, then I set it based on the current unixtime. I assume the same +/-2 hours variance is allowed? So I think I need to pad all the times by 2 hours to prevent mismatched timestamps from timing someone out inadvertently

As far as I know, locktime works by simply ignoring transactions that have a locktime in the future, i.e. the transactions aren't forwarded or added to the memory pool.

If the sequence on all inputs is 0xFFFFFFFF, then it counts as final and will be forwarded (and can be included in blocks).

You could set the sequence to 0xFFFFFFFE.  That is opt out of replace by fee.  The protocols don't care about transaction replacement anyway (or at least they shouldn't).

The new plan is to change the sequence field as follows:

All inputs are 0xFFFFFFFF: Final

This transaction can be included in any block as it is final.  These transactions do not allow replace by fee.

All inputs are 0xFFFFFFFE or higher: Disable replace by fee

This transaction is marked as opt-out of replace by fee.  However, it still has a locktime.

All inputs are >= 0x80000000: Opt-in replace by fee

These transactions have opted in to replace by fee.  The inputs with sequence numbers below 0xFFFFFFFE can be double spent and the miner can replace the transaction.

Note: the miner could replace the transaction anyway.

Any inputs below 0x80000000: Check sequence verify

The transactions have inputs with relative check locktime and cannot be spent for a delay after the UTXO being spent is included in the blockchain.

[gmaxwell]

The original proposal only affected memory pools, so it isn't secured by POW.  If locked transactions could be included in blocks, then the rule could be a consensus rule.

This is confused.

See even the see even the title of BIP68: "Relative lock-time using consensus-enforced sequence numbers."

The idea is that this would be consensus enforced (via a soft-fork).

What you saw with "mempool only" is due to process now used for soft forks in Core: Where possible core prefers to first implement the functionality as policy in order to gain experience with the design and implementation and to allow not-very-secure experimentation with client software in order to facilitate development and gain operating experience.  (this was done for CLTV, it's also currently being done for median-time-past (since 0.11.2)... and will be done with sequence enforcement).

No replacement of transactions in the blockchain is required, nor do I think it would provides any value-- certainly not enough to justify the huge layering violation; e.g. it would totally break SPV assumptions: "Here is proof you were paid", "Uh, how do I know that some later transaction didn't replace it?" "Uhh, let me send you the whole blockchain for the timeout window". It would also be less efficient than BIP58, since you'd end up with perpetually carrying around transactions that were ultimately replaced.

Both schemes require that there is at least one block by an 'honest' miner (meaning accepting the superior sequence spend, even if bribed to take an inferior one) before the inferior sequenced transaction timeout expires.

Can you compare what you're thinking of to BIP68 and highlight any advantages?

It is surprising that sequence numbers dont work as it is documented... I had thought CHECKSEQUENCEVERIFY was already on mainnet,

Huh? Sequence numbers work precisely as documented (they currently don't do much of anything; though BIP68 seeks to change that...).  CSV is a very new proposal, why did you think it was active on mainnet?

[jl777]

Huh? Sequence numbers work precisely as documented (they currently don't do much of anything; though BIP68 seeks to change that...).  CSV is a very new proposal, why did you think it was active on mainnet?

I probably looked at the wrong documentation?

The official definition:
"A number intended to allow unconfirmed time-locked transactions to be updated before being finalized; not currently used except to disable locktime in a transaction"

If you read the above definition and dont realize that "intended" means that the entire part about unconfirmed time-locked transactions is irrelevant, it is quite easy to get confused. Also, bitcoinj documents micropayment channels and it uses increasing sequenceids.

Assuming the OP is correct, it appears to contradict the bitcoinj documented behavior.

If two transactions spend the same outputs, then the miner is supposed to pick the transaction (input) with the higher sequence number.   This cannot be enforced and so, the miner would probably pick the one with the highest fees

Additionally, each input has a sequenceid field, its only current function is to toggle locktime on/off, but there is only one locktime per tx. The many toggles all operating on the same locktime is another confusing part. I assume that for SIGHASH_ALL, if any sequenceids are not -1 locktime is on, or is it off, or do all have to have the same. and for SIGHASH_SINGLE I have no idea how it is supposed to work.

I doubt I am the only one confused by all this and the only reason I need to know the details is because I am writing a fullnode from scratch so I need to be able to handle how it has been done historically.

James

[TierNolan]

See even the see even the title of BIP68: "Relative lock-time using consensus-enforced sequence numbers."

I am aware of CSV and that wasn't what I was thinking.  In retrospect, I should have used different terms.

What was supposed to happen was that sequence numbers would allow replacement, but it was never implemented and isn't secure anyway.

Can you compare what you're thinking of to BIP68 and highlight any advantages?

It would give much more finely grained sequence numbering and closer to the original intention.  I don't see it as a replacement for CSV.  In fact, CSV or locktime could be used to make sure later versions of the transaction have time to be broadcast.

The desired operation is that each fully signed transaction could act as a link in a chain.  It would count as a valid transaction conditional on it being built on a link with a lower sequence number (or the anchor/root of the chain).  This means that there is no point in broadcasting anything but the last version, since if you broadcast an earlier version and it gets into a block, one of the other parties will just broadcast a later version.

I was thinking about multi-party versions of payment channels.  Lightning is inherently based on linking up lots of 2 party channels.  This means that a hub must spend capital on each channel being maintained.

In principle, the hub could create an N party channel with a single capital deposit and share it between the customers.  Even in the simple case, where all the customers have 100% uptime and always sign state transitions that don't reduce their allocation, there isn't an easy way to do it.

Penalties based on revoke codes don't work as well with a multi-party channel.  In a 2 party channel, you simply send 100% of the channel's funds to the compliant party.  With a multi-party channel, that doesn't work.

The original two-way channel would work.  You could start with a locktime 60 days in the future and then step it back 2 hours for each transition.  All parties would have to sign the new transaction though.  It would be better if the signature from parties that aren't having their allocation reduced wasn't needed.