TX replacement and nLockTime

[kjj]

Over time hopefully p2pool will reduce the power of individual pools to impact rule changes, as most miners will just trust the de-facto judgement of the project maintainers in such matters (ie, you). And the biggest pools are all run by friendly, involved operators who aren't driven purely by profit maximization.

The proposed change would make it impossible for users on lightweight clients to do anything with a zero-confirm transaction, even for small amounts. That would make something as trivial as paying for a beer for Bitcoin largely impossible, which seems like a pretty basic litmus test of "does this currency actually work?".

What's more, allowing such double spends would mean that logically it should be extended to already confirmed transactions too. Once merchants all start waiting for 1 block, or 2 blocks, or whatever, large numbers of users would simply start broadcasting double spends after 3 blocks. Once you have enough replacements for transactions 3 blocks back, if all miners use the "profit by default" rule, everyone would start mining on a 3-block back fork automatically in order to claim the higher fees.

The system would become largely useless.

It is one thing to entice miners to prefer one unconfirmed transaction over another unconfirmed transaction.  Getting them to revert even one block is an entirely different sort of thing, even for a large reward.

P.S.  I don't think that the "convert to fee" idea would work anyway.

[Mike Hearn]

It is one thing to entice miners to prefer one unconfirmed transaction over another unconfirmed transaction.  Getting them to revert even one block is an entirely different sort of thing, even for a large reward.

Why?

The proposal made here was for a specific technical thing, replacing of memory pool transactions with higher fee double spends, but the rationale given is a dramatically stronger set of design constraints that Bitcoin has not previously operated under. That would be a major direction change for the project!

The design of Bitcoin is already bound by pretty severe design constraints - constraints like there must be no central authorities for anything, including such basic things as announcing news to network participants (hence broadcast alerts). That fits with Satoshis vision of a pure p2p system but obviously complicates things a lot.

The constraint that the system must be built on the assumption that miners will do anything for profit, and thus we might as well help them, makes it radically harder still. I'm not even sure it's possible to do such a cryptocurrency well, at least, it would not be Bitcoin.

Consider the case of setting the inflation schedule. In a hypothetical world where Bitcoin is very successful, we assume it has lots of users and because of the block sizes involved most of those users are on lightweight clients. Lightweight clients cannot verify the size of a coinbase transaction, so they can't check the inflation schedule is being followed.

In this world miners are financially incentivized to change the inflation schedule and start making more Bitcoins than Satoshi intended. Inflation is a tax on all other system participants that accrues to miners, so it makes sense for them to do this. But with Bitcoin as it is today there's a very practical starting problem - you have to find a majority of mining power and convince them to switch to your new set of rules. Firstly, you can't find most miners, because mining happens anonymously. Secondly, of those you do find, some of them will reject your proposal for ideological reasons despite that it could lead to more profit for them. Thirdly, the act of trying to set up such a large conspiracy would certainly be detected and could be addressed via social means, like (e.g.) laws, or other forms of social pressure.

However if you assume that purely profit driven miners are inevitable and so we might as well just help them get on with it, you'd design the software to make resetting the inflation schedule automatic, like based on votes encoded into the coinbase. With this new design constraint you eliminate all 3 reasons above from your consideration, because in your world miners are driven purely by profit and have no other limits on their behavior, so there's no reason not to do this.

A Bitcoin in which miners can and do vote on how much inflation they'd like to award themselves is not a Bitcoin I'd feel like participating in. But it's the logical outcome of assuming miners are a perfectly informed conspiracy of entirely selfish actors.

[blueadept]

Apologies for locking the topic. I've been reading/replying on my phone and must have accidentally fat fingered it. I'm still thinking through the implications of Mike's last post.

[flower1024]

Apologies for locking the topic. I've been reading/replying on my phone and must have accidentally fat fingered it. I'm still thinking through the implications of Mike's last post.

me thinks the problem is that there is no way to stop a miner which replaces a memory pool transaction with another one which offers a higher fee.

afaik only transactions in a block (older is better) are safed. all other ones are just maybe's.

but if only a few miners do this (transaction replacement) there is no use at all. you can't be sure that a zero-conf transaction don't get replaced AND you dont have the benefit of nlocktime and transaction replacements.

so i vote for "allow replacing of zero-conf transactions" so it would be possible to build escrow services around it.

[Gavin Andresen]

A Bitcoin in which miners can and do vote on how much inflation they'd like to award themselves is not a Bitcoin I'd feel like participating in. But it's the logical outcome of assuming miners are a perfectly informed conspiracy of entirely selfish actors.

The Bitcoin ecosystem consists of more that just miners, and even if miners decided to try to form a cabal to increase inflation merchants, users, and exchanges could all veto their block-chain by simply refusing to recognize it.

I think the policy of which transactions to keep in the memory pool is fundamentally different, because miners can do whatever they like and there's not a whole lot merchants/users/exchanges or other miners can do about it. There's no way to enforce a "first broadcast version of a transaction must be mined" rule, if there was then we wouldn't need the block chain at all.

As for re-writing the blockchain if an after-the-fact, large-fee double-spend is broadcast:  I think that's covered by this case:

Should a miner put Transaction 1 into the block they're mining and take the smaller fee now, or not include it, hoping that nobody else mines Transaction 1 in the next 3 days so maybe they can mine Transaction 2 and get the bigger fee?

I'm not an expert in game theory, but I believe the winning strategy in the above situation, assuming everybody knows about both transactions, is to mine Transaction 1 right away (any economists reading who know a lot more about game theory than I do?).

Or, in other words, you'd need to be pretty sure that you've got a majority of miners who will cooperate with you to rewrite the block chain. The longer the chain, the harder that will be (because it becomes increasingly likely that one of your co-conspirators mined one of the blocks you want to overwrite).

[jevon]

Or, in other words, you'd need to be pretty sure that you've got a majority of miners who will cooperate with you to rewrite the block chain. The longer the chain, the harder that will be (because it becomes increasingly likely that one of your co-conspirators mined one of the blocks you want to overwrite).

Suppose there's a transaction for 100,000BTC and several blocks later a double spend with 50,000BTC fee... and by then the 100,000BTC was already spent out to others in smaller amounts.

It only takes 1 or 2 pool miners to have 51%.

(Maybe there should be a maximum fee per transaction--edit: nevermind, not possible)

[blueadept]

With a maximum fee per transaction, you throw out the double spend defense you proposed of a recipient countering a double spend by spending the output of a previous transaction completely to fee. There are other ways to get around it too and they involve spamming the blockchain.

[Gavin Andresen]

Suppose there's a transaction for 100,000BTC and several blocks later a double spend with 50,000BTC fee...

Maybe there should be a maximum fee per transaction.

Or maybe you should wait 60 blocks before considering a half-a-million-dollar transaction final if you think there is any possibility of a mining cabal trying to do this.

All of this reminds me, I need to clean up my user-defined-checkpoints code for the 0.7 release, so a cabal of merchants and exchanges can get together and decide they will "lock in" an agreed-upon blockchain after 6 (or 60 or whatever) confirmations. I think that would go a long way towards infrastructure for injecting real-world knowledge about who is trustworthy into the block-chain.

[jevon]

Suppose there's a transaction for 100,000BTC and several blocks later a double spend with 50,000BTC fee... and by then the 100,000BTC was already spent out to others in smaller amounts.

Or maybe you should wait 60 blocks before considering a half-a-million-dollar transaction final if you think there is any possibility of a mining cabal trying to do this.

He should wait before delivering the goods, but there's no incentive for him to wait to spend the BTC. Actually, he's better off if he does spend it and someone else down the line takes the loss.

What if they break it into 1000 payments of 100BTC, and then 1000 double spends with 50BTC fee?

[jevon]

I see three options for who should get the double spent amount:

1. The first transaction - Currently working OK, although imperfect incentive for miners. Double spend race attack is manageable.

2. The highest fee double spender - Aligned with miner incentive, but rewards double-spending even more.

3. Neither, the miner gets it automatically - Aligned with miner incentive, trumps fee bribe, and removes all incentive to double spend. See https://bitcointalk.org/index.php?topic=76276.0

[etotheipi]

I see three options for who should get the double spent amount:

1. The first transaction - Currently working OK, although imperfect incentive for miners, and double spending still possible with race attack.

2. The highest fee double spender - Aligned with miner incentive, but rewards double-spending even more.

3. Neither, the miner gets it automatically - Aligned with miner incentive, trumps fee bribe, and removes all incentive to double spend. See https://bitcointalk.org/index.php?topic=76276.0

Can you remind me how "the miner gets it automatically"?  I think it would be a totally unacceptable change to allow coins to be transferred to a party/script that was not signed by the originator.  i.e. AddrA signed a tx to go to AddrB, but instead the network allows it to be awarded to AddrC.   I don't see how there's even a hint of feasibility in this idea, unless I'm misunderstanding the proposal...

There's also some legit reasons a tx might end up appearing to be a double spend, even though it's not.  Programmatically, it would be impossible to distinguish those cases, and miners would have free reign to eat up any such transactions anyway.

[jevon]

Can you remind me how "the miner gets it automatically"? 

It's in "Defense against double spending" thread https://bitcointalk.org/index.php?topic=76276.0

I think it would be a totally unacceptable change to allow coins to be transferred to a party/script that was not signed by the originator.  i.e. AddrA signed a tx to go to AddrB, but instead the network allows it to be awarded to AddrC.   I don't see how there's even a hint of feasibility in this idea, unless I'm misunderstanding the proposal...

Signing a second spend authorizes a miner to take it. So don't double spend!

There's also some legit reasons a tx might end up appearing to be a double spend, even though it's not.  Programmatically, it would be impossible to distinguish those cases, and miners would have free reign to eat up any such transactions anyway.

The legit cases are where the outputs are the same, or any non-final. Don't count those as double spends.

Double spend = same input, different outputs.

[Mike Hearn]

The Bitcoin ecosystem consists of more that just miners, and even if miners decided to try to form a cabal to increase inflation merchants, users, and exchanges could all veto their block-chain by simply refusing to recognize it.

The problem is that there are classes of changes that miners would want to make which cannot be recognized by a majority of users who will be on lightweight clients (assuming the system continues to grow, of course).

Lightweight clients can't efficiently calculate the size of the coinbase for a block without downloading the whole block and then downloading the dependencies of every transaction in that block, along with the Merkle branches linking them to the relevant block headers (which may also need to be fetched because I think in future lightweight clients will throw away very old headers).

In todays world it's not a big deal because most miners cash out immediately via a small number of exchanges, and if they chose not to recognize a rule change it wouldn't matter how many miners adopted it. But in future if there are lots more merchants and more competition in the exchange space (or even p2p exchanges), then the chances of miners being able to enforce major rule changes like that becomes a lot larger, because lots more users will just follow their new rules - either because they can't efficiently detect the change or because they don't want to deal with a big slowdown of the system.

I think there's enough resistance from other aspects of the system (can't find miners, people mine for ideological reasons, conspiracy would be detected) to ensure this won't happen with the way things are currently set up. But if we switch to a "miners profit is primary no matter what" assumption, the risk becomes much greater.

I think the policy of which transactions to keep in the memory pool is fundamentally different, because miners can do whatever they like and there's not a whole lot merchants/users/exchanges or other miners can do about it. There's no way to enforce a "first broadcast version of a transaction must be mined" rule, if there was then we wouldn't need the block chain at all.

As long as most people are using the standard rules, actually performing such a transaction-switch isn't so easy. You have to find a miner who is using different rules, and you have to hope that this miner gets the next block. If they don't, you're out of luck. That's a high enough barrier that Finney attacks are likely to be rare in practice.

If you change the standard rules so everyone offers such a service by default, on the assumption that everyone would end up doing it anyway, then it's a self fulfilling prophecy. The attack would become a lot more reliable, so people would do it a lot more, so there's now profit to be had by using the new software, so people would upgrade, and it'd become basically impossible to do any kind of low-latency trade with lightweight clients.

Or, in other words, you'd need to be pretty sure that you've got a majority of miners who will cooperate with you to rewrite the block chain. The longer the chain, the harder that will be (because it becomes increasingly likely that one of your co-conspirators mined one of the blocks you want to overwrite).

Let's assume a world where mining is incented primarily by fees rather than inflation (as is, people don't usually attach fees anyway). If a transaction that was mined 3 blocks ago is re-broadcast with significantly higher fees, and every other transaction remains the same, everyone will be incented to reset the chain to that point so they have a chance of claiming the higher fee. All the other transactions can just be included in the new blocks being mined so it's no big deal. The co-conspirators who mined the blocks being replaced would lose their inflation, but presumably that's quite low at this point anyway and they can reclaim the fees on any other transactions that they re-include anyway. Once everyone is simultaneously incented to start re-mining from a few blocks back, the fees in all those transactions become up for grabs again so why not do it?

[blueadept]

If we assume a low margin mining industry (not necessarily the case yet but it will be eventually), it will cost almost the same amount to mine a block as the block reward (even if the reward is mostly fees).

To mine the last three blocks, the fee increase would have to be at least as much as the total fees in those three blocks (as a proxy for the cost to mine those blocks again) to make it worthwhile. That means that if a transaction is sufficiently large, the receiver will need to wait until the fees/inflation paid since the transaction was confirmed are greater than the amount of the transaction or until the block chain is locked in as Gavin suggested, whichever is shorter.

Even with a conspiracy of 51% of mining power, it would be cheaper to keep mining the existing block chain unless the fee increase is large enough to pay for the mining power to remine previous blocks. Without a 51% conspiracy, you won't be able to remine those old blocks anyway.

ETA: I could be wrong though. Assuming a 51% cabal, there could be profit in taking the fees of the other 49% too. The recipient may want to wait until the fees and inflation paid to miners since her transaction is at least twice the value of all the inputs, in case the transaction involves other recipients and the value to th sender of a double spend is greater than the value oof no double spend to the recipient.

This opens up more cans of worms maybe. If double spenders collude to broadcast multiple double spends to incent miners to remine after delivery of goods because recipients only waited to be safe based on their own transactions, my assumption goes out the window.

[Serith]

I think I have an idea how to make accepting 0-confirmation transactions as safe as it is now and allow transaction replacement mechanics.

Preliminary requirement:

• if a miner has two conflicting transaction where one of them is a normal transaction and the other is multisig then multisig gets priority and goes into the next block.

Regarding the issue whether or not miner would act to maximize his own profit vs benefiting network as a whole, I think there is not enough information yet and we should wait and see what miners will do.

To accept 0-confirmation transaction, multisig transaction should be created, signed by both parties and broadcasted before releasing the purchase. An attacker won't be able to replace first transaction with another that conflicting it, because merchant won't sign second transaction.

EDIT: I am wondering if it's possible to create clearing house using multisig transactions to reduce the risk of accepting 0-confirmation transaction, e.g. a multisig transaction that has to be signed by several major pools

[Andrew Vorobyov]

So what do we agree on?

[Serith]

So what do we agree on?

I found this from etotheipi in mailing:

On 04/26/2012 01:30 PM, Peter Todd wrote:
>
>> More difficulty shortens the safe time we can transact large volumes in,
>> which is good for the network.
>>
>> I'm not sure of the current implementation of replacement transactions, can
>> anyone on the core team speak to this? Can I replace transactions, or is
>> that part of the spec unimplemented or deprecated right now?
> My understanding is it's completely disabled.

Went on a scavenger hunt with Gavin a couple weeks concerning tx
replacement.  The conclusion was that if,
(1) Transaction has lock-time in the future  AND
(2) Transaction has non-maximum sequence number

Then the transaction will both propagate and be accepted into nodes'
memory pools, but will not go into any block until locktime expires.  If
the lock-time is in the past OR sequence number on all TxIns is
0xffffffff, then it will be immediately valid and included in the
blockchain.

But the actual "replacement" mechanism is disabled.  Therefore, the
nodes accept the tx as if it's replaceable, but don't allow it to be
replaced.  This means that it is effectively replaceable *once*, but
only if you inject a final transaction into the blockchain.   You can't
broadcast a final version of the same tx, because it will conflict with
the non-final one sitting in all the other nodes' memory pools.  You
need a miner to agree to remove the non-final tx from their memory pool
and specifically include your replacement.

-Alan

Why can't we have transaction replacement tied with multisignature, that way to replace transaction everyone involved has to explicitly agree on that? To elaborate:

1. Miner accepts multisignature transaction with nLockTime in future into memory pool.

2. Before nLockTime expires a new multisignature transaction comes in, and it has been signed by every private key that was used to sign previous transaction and maybe some more.
The new transaction also has greater sequence num, which is just an arbitrary number put by the one who created transaction (second party won't sign unfavourable transaction with greater sequence num).

3. Miner checks the new transaction and if it satisfies  those two conditions, then the old transaction gets replaced.


From wiki - Contracts:

Sequence numbers can be used to issue new versions of a transaction without invalidating other inputs signatures, e.g., in the case where each input on a transaction comes from a different party, each input may start with a sequence number of zero, and those numbers can be incremented independently.

What is the rational behind implementing independent sequence number change? Why it is important for any participant to be able to issue a valid new version of a transaction independently?

[Andrew Vorobyov]

I think "sequence number" must be extra fee for this transaction..

For example, transaction fee is 0.01 BTC + every next transaction will add 0.00001 BTC to it..

This way we can seduce miners to have oldest transaction in pool.

What do you think?

[Andrew Vorobyov]

Yeah! it's brilliant idea, isn't it?? We can't avoid total protocol change though, because nLockTime still must be somewhere in transaction, but still...

PS. I assume nLockTime is some block number greater than current one, correct?

[Mike Hearn]

Re-activating nLockTime does not require any protocol changes exactly, just a reversion to the original protocol we had before the security scare. I don't agree with any of the arguments for trying to "fix" it at the same, the current system works fine.

There are examples of why you need per-input lock times in the contracts wiki.