Initial replace-by-fee implementation is now available on testnet

[jdillon]

(reposting the bitcoin-development email for visibility)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

After some consultation with affected sites by myself and Peter we have decided
to release an initial replace-by-fee implementation and setup a server using
those rules on testnet. This implementation does not include recursive fee
evaluation, and is therefore vulnerable to DoS attack, so hopefully that will
continue to allow adoption to proceed gradually. We can-not recommend mining on
mainnet with it. It does not include an "undo" RPC command or an adjust fees,
and Peter says he has not implemented one yet.  Patches are welcome.

Specifically there were requests from vulnerable parties, which interestingly
included a site that knew they had bugs related to replacement but not
financial vulnerabilities, to put up a server on testnet to check wallet code.
The vulnerable requested to remain undisclosed. An additional consideration was
the upcoming anti-dust rules which are yet another example of why zero-conf is
so much more dangerous to accept than single-conf. Two of the people contacting
us brought up that issue in fact.

The code is on github:

    https://github.com/petertodd/bitcoin/tree/replace-by-fee

and a replace-by-fee server operating on testnet is available at
testnet-replace-by-fee.bitcoin.petertodd.org To test you will need to use the
raw transaction API and manually create the replacement transaction. Do note
that your wallet will retain the existing one and no mechanism yet exists to
delete the old transaction from your wallet. Again, a certain amount of
"cludgyness" to this is intentional to discourage premature non-testing use.


Regarding the reward, I've decided Peter will collect the full amount even
though the work is not %100 complete (the mempool aspect) due to his concern
about staging an implementation properly, working with vulnerable sites, and
overall genuine interest in the actual issues at hand rather than the reward.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQEcBAEBCAAGBQJRi3LeAAoJEEWCsU4mNhiPwscH/2CI0d3h/3jix3iyz2I9I8Sz
6nbP8eA01l9kzG37cH1rFAbt7C+fL/nardV4U1qmiwC0MN7NPpX6BFn5eQ2PUKbu
41+AnjgWicB2tnCC07ngboQ1JCeZ+RTfATepuMxEdWFBsc8ZQXs0apWS01FT+TDq
J/a7QkhNfTaAQzXyqmLp0TQO7/Z7ysmCftOhtGbfvfhF2o23BuphQiRVA9IOoUuj
Fgb5wrfQqJ8TjvXRXAUQ7SUlzfN9BlPxMkTc6NhbcgIpuq1Kb43mLoDl3s2irH4A
GtjRtobV5Cfozm1r+8KPtIYEoQoj0PqTjO5YMwD/vTaRfNzdS4Tse5LQLGT6Jug=
=M1mj
-----END PGP SIGNATURE-----

[smoothie]

Peter who?

[ripper234]

http://bitcoin.stackexchange.com/questions/10733/what-is-replace-by-fee

[Peter Todd]

Peter who?

Peter Todd

[Stephen Gornick]

Regarding the reward,

For context on that:

Someone by the name of John Dillon (john.dillon892@googlemail.com) emailed the bitcoin-development email list earlier this morning offering a $500USD reward to anyone who implements a transaction replacement-by-fee patch.

That thread is:

Reminder: zero-conf is not safe; $1000USD reward posted for replace-by-fee patch
 - http://bitcointalk.org/index.php?topic=179612.0

[Peter Todd]

Regarding the reward,

For context on that:

Someone by the name of John Dillon (john.dillon892@googlemail.com) emailed the bitcoin-development email list earlier this morning offering a $500USD reward to anyone who implements a transaction replacement-by-fee patch.

That thread is:

Reminder: zero-conf is not safe; $1000USD reward posted for replace-by-fee patch
 - http://bitcointalk.org/index.php?topic=179612.0

Yup. The reward was initially $500 and increased to $1000 after I pointed out to jdillon that replace-by-fee needed recursive fee evaluation, IE Luke's child-pays-for-parent concept, or it would be susceptible to DoS attacks. That code is easily $500 of engineering time for a prototype.

To be clear, replace by fee was initially something I proposed, and jdillon decided to offer a reward to actually implement it a few days later. Like myself he has strong feelings about ensuring mining remains decentralized and free from regulations.

[jonytk]

To be sure: this only adds priority to a transaction right?
it doesn't modify the outputs?

because if not, someone could just fork the client to make an alt-coin that allows you to pay a fee to replace a transaction.
hence why they are paying for this...

[Peter Todd]

To be sure: this only adds priority to a transaction right?
it doesn't modify the outputs?

because if not, someone could just fork the client to make an alt-coin that allows you to pay a fee to replace a transaction.
hence why they are paying for this...

Nope. This is pure replace-by-fee, under any circumstance where two transactions would spend the same inputs. For instance I could have a transaction with a whole bunch of inputs and outputs, and another one that spends only one of those inputs and has one output. If the fee-per-KB of the second was higher than the first, and the total fee paid by the second was also higher, then the first transaction and all dependents would be removed from the mempool and the second added in it's place.

Among other things, I intend to implement an "undo" button that simply creates a transaction spending one of the inputs of an unconfirmed transaction in your wallet with a higher fee, sending the remainder back to your wallet.

[jonytk]

To be sure: this only adds priority to a transaction right?
it doesn't modify the outputs?

because if not, someone could just fork the client to make an alt-coin that allows you to pay a fee to replace a transaction.
hence why they are paying for this...

Nope. This is pure replace-by-fee, under any circumstance where two transactions would spend the same inputs. For instance I could have a transaction with a whole bunch of inputs and outputs, and another one that spends only one of those inputs and has one output. If the fee-per-KB of the second was higher than the first, and the total fee paid by the second was also higher, then the first transaction and all dependents would be removed from the mempool and the second added in it's place.

Among other things, I intend to implement an "undo" button that simply creates a transaction spending one of the inputs of an unconfirmed transaction in your wallet with a higher fee, sending the remainder back to your wallet.

But then anyone can send a transaction to satoshidice and then, if he loses, send another transaction with more fees sending the coins to himself???

Can't wait to see the hundreds of noobs complaining, "i sent xcoin to a trader when i show the btc coming at blockchain but they never arrived!!"

Or the merchants saying "this destroy instant payments, unless you reduce your 10 min confirmation to 2 we're moving to ltc..."

etc... or am i wrong?

[Gavin Andresen]

I think this is a terrible idea.

If it was "higher fee with superset of outputs of first spend" then that'd be fine.

Zero-confirmation transactions will never be safe because of potential Finney attacks. But we don't need (and, in my opinion, shouldn't) make them less safe by encouraging anti-social behavior via default mining policy.

[Peter Todd]

But then anyone can send a transaction to satoshidice and then if he loses another transaction with more fees sending the coins to himself???

Yes, evoorhees is well aware of that and has been contacted.

They will be introducing off-chain betting in the near future so it won't be an issue for them soon. I and John are discouraging mainnet miners from making use of this patch immediately through a variety of methods to give people time to respond.

[jonytk]

I think this is a terrible idea.

If it was "higher fee with superset of outputs of first spend" then that'd be fine.

Zero-confirmation transactions will never be safe because of potential Finney attacks. But we don't need (and, in my opinion, shouldn't) make them less safe by encouraging anti-social behavior via default mining policy.

yes, what is the idea of the patch?, i thought it was to make transactions stuck on limbo ( because of lack of insufficient fees) to go thru.
the fee , and the change, should be the only thing changed here.

Otherwise you are implementing charge-back, and for that, wouldn't it be easy-er to just make the bitcoin-qt wait, by default, internally 1 minute before really sending the transaction to the network?  

Let the problem be solved client-sided? there you have your cancel button!


Who is this johndillion? Could it be the Chinese-man moving around 500 bitcoins all the time against SD ?
 :S

[kjj]

I think this is a terrible idea.

If it was "higher fee with superset of outputs of first spend" then that'd be fine.

Zero-confirmation transactions will never be safe because of potential Finney attacks. But we don't need (and, in my opinion, shouldn't) make them less safe by encouraging anti-social behavior via default mining policy.

I see this by analogy to security research vulnerability disclosure.

Right now, tons of people* are acting as if they were safe, when they really are not.  They've been warned, over and over again.  Until the tool gets posted to metasploit for one-click use, they will not really accept that they are in danger.

I'm pretty sure there have been other bitcoin patches accepted or rejected on the basis of avoiding lulling users into a false sense of security.  In the end, it doesn't matter.  Eventually, the network will operate on the principle embodied in this patch.  The miners that aren't already using their own tools for block generation can get this if they want it.

That said, I'm not sure that I'd want to sign off on this either.  Inevitable or not, it does kinda feel icky.

*  I don't believe that tons of people are actually accepting zero confirmation transactions.  I suspect that there are a dozen people on these forums (incorrectly) whining about this patch making 0-conf unsafe for every person actually accepting them.

[Peter Todd]

I'm pretty sure there have been other bitcoin patches accepted or rejected on the basis of avoiding lulling users into a false sense of security.  In the end, it doesn't matter.  Eventually, the network will operate on the principle embodied in this patch.  The miners that aren't already using their own tools for block generation can get this if they want it.

Yup.

Also, if the network doesn't operate on this principle, it will be because we've implemented ways to punish those who do otherwise, and all the ways to do that involve ad-hoc secondary consensus mechanisms that make the network more fragile, and make it harder to run a profitable mining pool on a limited budget. Remember that there is no magic way to know if a transaction is a double-spend - you're limited network bandwidth can easily mean you just don't see the original transaction where a bigger, more centralized mining pool with a fast network connection will maintain a more view of the mempool that is more consistent with other large pools.

That said, I'm not sure that I'd want to sign off on this either.  Inevitable or not, it does kinda feel icky.

Indeed. Part of the reason why I decided I'd write the patch and collect the reward was because I knew if I did so I would have some control over how it was "released into the wild" - someone else might just want to for the money and not bother to do responsible disclosure. Heck, I was thinking of turning down the reward, but with John also donating to my blocksize video... well, Bitcoins are fungible. In any case I'm pretty sure his intentions are honorable after the discussions I've had with him.

[jonytk]

yes, i understand zero-confirmations it's unsafe, the problem is the users, does everyone understand? they don't want to wait 10 minutes for a confirmation

of their online pizza purchase... they want it now.

I will support this and to keep the block-size the same if the bitcoin speed was increased to 2 minutes for example.
Then no problem waiting for confirmations.

[jgarzik]

yes, i understand zero-confirmations it's unsafe, the problem is the users, does everyone understand? they don't want to wait 10 minutes for a confirmation

of their online pizza purchase... they want it now.

I will support this and to keep the block-size the same if the bitcoin speed was increased to 2 minutes for example.
Then no problem waiting for confirmations.

Use a service layered on top of bitcoin, that provides the instant+secure guarantees you want.

[Zeilap]

Is there a subtle issue with recursive fee calculation that's prevented it being implemented?

[d'aniel]

Businesses that accept zero-confirmation transactions are surely quite aware that they are not perfectly safe, but some (e.g. bitpay IIRC) still find it in their interest to accept them anyway, and incorporate the risk of loss into their prices.  Simply assuming they're stupid to do so in order to justify unintentionally harming their business model seems a bit... insensitive.

Use a service layered on top of bitcoin, that provides the instant+secure guarantees you want.

Wouldn't bitpay, for example, lose the universality of the Bitcoin payment network by switching to something like this?  Part of what makes them so convenient is that customers don't have to sign up for special accounts.

[Peter Todd]

Use a service layered on top of bitcoin, that provides the instant+secure guarantees you want.

That service may not be at all what you'd expect either: the bitcoin-otc.com P2P exchange provides a P2P reputations system. You could combine that with a simple "payment protocol" of just PGP signing your trades, and a mechanism to retrieve double-spent transactions from your wallet to make it easy to prove to others that you were defrauded. The initial implementation doesn't have to be fancy, just something an expert could use to verify someone else's claims, but it'd still be significantly better than what bitcoin-otc has right now.

Is there a subtle issue with recursive fee calculation that's prevented it being implemented?

In fact I've already got most of a recursive fee calculation implementation written, including unit tests. But because recursive fee calc is required to make the patch robust against certain types of DoS attacks, I want to hold off on making that code available to give people time to experiment with tx replacement, in particular how they're merchant code handles it, while still discouraging miners from using it on mainnet. Similarly that vector-vs-set bug you found was a very nice catch, but again, I'm going to hold off fixing it for another two or three weeks.

Use a service layered on top of bitcoin, that provides the instant+secure guarantees you want.

Wouldn't bitpay, for example, lose the universality of the Bitcoin payment network by switching to something like this?  Part of what makes them so convenient is that customers don't have to sign up for special accounts.

The "service" layered on top doesn't need to be an account tied to one vendor. I mentioned bitcoin-otc above; another approach is to purchase a fidelity bond tied to a pseudonym, and use that bond to vouch that you will not double-spend some given payment. Break that promise and the recipient can publish the fraud, invalidating the bond.

[Herbert]

How does this patch behave with regards to chained transactions?
Say the output of TX1 is spent in TX2 (both unconfirmed).
Now TX1 gets replaced - From my understanding of the code this means that TX2 can never confirm, because it's inpoints are not existing anymore. Or am i missing something?

Sidenote:
Although i run myself a site which heavily relies on unconfirmed transactions I fully support implementing this. I agree with the opinion that this behaviour is inevitable in the future, so we should better prepare for it sooner than later. And if this means I have to close down or rewrite my site - so be it. Was fun while it lasted