sendtoaddress sendmany api change proposal.

[btc4ever]

Having worked on a few sites using bitcoind, a recurring pain point for me (and apparently many others) has been the inability to know the fee prior to sending.  This is particulary troublesome because the fee can vary wildly as a percentage of the send total, and business rules may preclude sending when a fee is higher than a given percentage.

It's sort of like if you walk into Western Union or Money Gram, and ask how much it will cost to send money.  They say, don't worry about it.... it's kinda complicated to figure out so just leave us your credit card and you can look at your statement later to find out how much we charged you.  They don't do that because people would not accept it.  We like to know up front, or at least be able to set limits.

Often when sending bitcoin it may be desirable to wait until the fee is less -- either because the coins have aged, or we have restructured our inputs, or we are sending a larger total resulting in a smaller percentage fee.

In other words, the fee itself can be an important factor in decision making whether to send a transaction.

I understand it may be difficult or problematic to implement a getsendfee() type of API because the fee could change, etc.

However I know that for my own use cases, it would be very helpful to be able to set a fee limit.  So that sendtoaddress or sendmany would succeed if the fee is below the limit and fail if over.

So the APIs could be:

sendtoaddress "bitcoinaddress" amount ( "comment" "comment-to" "max-fee" )
sendmany "fromaccount" {"address":amount,...} ( minconf "comment" "max-fee" )

where max-fee can be either:
a) a fixed amount, eg: 0.001
b) a percentage amount, eg: 1%

If the transaction would exceed max-fee then an error is thrown.  Applications can catch that error and optionally try again with a different combination of amount and max-fee.  At least the app knows the fee was insufficient and can act accordingly.

Use cases for this functionality

1. A merchant or other bitcoin receiving website maintains a hot wallet and regularly sweeps to cold-storage.  

   Owner wishes to wait until either:
        a) enough funds have accumulated and coins aged sufficiently to send with 0 fee.
        b) funds total passes a given threshold, and will then accept up to a 1% fee.
        c) funds total passes a higher threshold and will then accept up to a 3% fee.
        etc.

2. A mining pool or other agent needs to send to many recipients.  

   Agent wishes to wait until enough funds have accumulated and coins aged sufficiently to send with 0 fee.
   Agent will send to individuals sooner if requested, but will deduct max_fee amount from their payment to cover the send fee.


But what about floating fees and txconfirmtarget?

At first I was hoping that floating fees would remove this pain point, but it doesn't really seem like it does.  According to Gavin:

There is a new option that lets you control how quickly you’d like your transactions to confirm: txconfirmtarget. The default value is 1, meaning “I’d like my transactions to be sent with enough fee or priority so they are very likely to be included in the next block.” Set it to 6 and it will take on average six blocks for your transactions to get their first confirmation.

Now that sounds pretty fuzzy to me.  I don't see how I can use that capability to enforce a business rule that we don't send unless fees are below X percentage.  If it is possible, please enlighten me.



So... what do you think?    good idea or bad?    and how easy or hard to implement correctly?


I'd be glad to a crack at this and submit a patch, but only if there is some amount of consensus that it is desirable functionality and reasonably straight-forward to implement.

[gmaxwell]

We don't generally use BIPs for random UI things.

As far as the idea goes— perhaps. Though I'd recommend adjusting your business plans such that the average of the fees in total is what you worry about; rather than the fees on any particular transaction. Otherwise you're just going to get jammed up and unable to transact when someone has sent you a number of dust inputs and the only way to eliminate them is to pay a somewhat high fee... all your low value transactions will end up failing until you up that limit.

[DannyHamilton]

We don't generally use BIPs for random UI things.

Ah, ok.

Removed my comment.

Thanks for explaining.

I've been considering getting involved and making contributions in the development of Bitcoin Core.  I'm still figuring out that process (I've never been involved in programming or designing for any open source projects before).

Seems like some sort of mentoring system to make the transition easier would be useful for talented programmers that want to get involved in the effort but aren't sure where or how to get started.  Are there any good tutorials or primers that I should take a look at?

Do I just install git on my MacBook, clone the repository, throw a dart at a BIP list, and start from scratch on my own implementation of whatever proposal my dart lands on?

[btchris]

Are there any good tutorials or primers that I should take a look at?

I don't know the full answer, but I can give you a handful of pointers.

• The mailing list is a good place to discuss Bitcoin Core implementation specifics, especially non-trivial changes. (It's not a good place for arbitrary questions / thoughts about Bitcoin Core....)
• The coding guide is an essential read.
• There's also a lot of discussion that goes on in the GitHub repo, especially in pull requests -- if you're thinking of implementing something, I'd check here to see if someone might already be working on it, or if the idea has already been tried but had some issues.

[btc4ever]

As far as the idea goes— perhaps. Though I'd recommend adjusting your business plans such that the average of the fees in total is what you worry about; rather than the fees on any particular transaction. Otherwise you're just going to get jammed up and unable to transact when someone has sent you a number of dust inputs and the only way to eliminate them is to pay a somewhat high fee... all your low value transactions will end up failing until you up that limit.

I'm not entirely sure what good worrying about the average of fees in total is when I have zero control over it using send* apis.  In fact I am here making a proposal because I do worry about it.

The case of small amounts may actually be where this is most useful because the fees can presently eat up 50% or even 100+% of the send amount.  But its hard to KNOW that until you actually try to send.  And then you get the surprise fee amount like an unwanted toy out of a cracker jack box.   So it is often preferable to wait until those inputs have aged sufficiently and/or enough other larger inputs have come in that the overall fee percentage goes down to something like 1%. Or maybe just wait till btc price rises and those dust are worth something.  whatever.... it should be my application (or its user's) choice, not some algo within bitcoind.

In the case you mention where many dust inputs are received and it jams things up, it is still useful for the app to be able to make informed decisions.  Maybe the decision is to call lock_unspent for some utxo, or log a warning or email the site administrator.  Or maybe it is just to send again with a higher fee limit or no fee limit at all.

Right now the app developer's options are to either blindly call sendto* and take the fee it arbitrarily comes up with or else write your own coin selection and fee calculation routines and generate a raw transaction.  I've been down that route....  it is painful and error-prone, and only a few people will get it right.


I'm not saying the proposed API change is perfect... actually perfection to me would be if I could write app code something like:

// business application pseudo-code

while( num_tries++ < max_tries ) {

   fee = rpc.calcsendfee( address, amount )

   if( not fee_is_ok( fee ) )
       break;

   code = rpc.sendtoaddress( address, amount, .., fee )

   if( code != insufficient_fee ) {
      break;
   }
}

There would be a corresponding calcsendmanyfee also.  sendtoaddress might (rarely) return insufficient_fee error code if conditions have changed since calcsendfee() returned.  In that case, the application should retry, possibly multiple times.

I'm guessing this approach is a bit more involved to implement than my first proposal, but maybe not too bad since it is already calc'ing the fee internally anyway.

thoughts?

[2112]

I'm not saying the proposed API change is perfect... actually perfection to me would be if I could write app code something like:

// business application pseudo-code

while( num_tries++ < max_tries ) {

   fee = rpc.calcsendfee( address, amount )

   if( not fee_is_ok( fee ) )
       break;

   code = rpc.sendtoaddress( address, amount, .., fee )

   if( code != insufficient_fee ) {
      break;
   }
}

You seem to be missing the crucial fact: the coin selector in the "send" is a stochastic knapsack solver. For any non-trivial wallet it pseudo-randomly selects different coins resulting in a different size of the transaction and thus requiring a different minimal fee. This has been discussed many times on this forum, please use search.

[btc4ever]

You seem to be missing the point that this is a sharp pain point for developers and businesses.  And just plain bizarre that one can't know a fee until after the fact.   too bad if the fee turns out to be 20%, 50%, 100% or even more of your send amount!

Also, if you look at my original proposal in this thread, you will see I am well aware that the fee could change between API calls.  That's why I proposed an alternative with a fee_limit param for sendtoaddress.  That would work fine with the existing pseudo-random sendtoaddress implementation.

Even still, I can wish for something better.  As I said, "perfection to me would be...".

Bitcoin-qt allows the user to view the fee and either send or not.  So what technical reason prevents that capability from being exposed at the RPC API level?   Do we need to add some locking/state around calcsendfee/sendtoaddress?  fine, let's do that.

Or there could be a deterministic alternative to sendtoaddress that RPC users could use when they want a fee calculated.

I believe this is not a technical problem so much as a "la la la" we don't want to fix it cultural problem.

My concern is that if I (or anyone) go to the work to fix it and submit a patch, it won't get accepted, due to core team resistance.

Am I wrong?

My hope with this thread is that we can get some amount of concensus on an approach, so that when it gets implemented, it will be likely to be accepted.

You seem to be missing the crucial fact: the coin selector in the "send" is a stochastic knapsack solver. For any non-trivial wallet it pseudo-randomly selects different coins resulting in a different size of the transaction and thus requiring a different minimal fee. This has been discussed many times on this forum, please use search.

[btc4ever]

oh and also, if you look at this pseudo code again:

// business application pseudo-code

while( num_tries++ < max_tries ) {

   fee = rpc.calcsendfee( address, amount )

   if( not fee_is_ok( fee ) )
       break;

   code = rpc.sendtoaddress( address, amount, .., fee )

   if( code != insufficient_fee ) {
      break;
   }
}

Notice that there is a new fee parameter to sendtoaddress(), which in this case is the value returned by calcsendfee().

That fee would act as a limit for sendtoaddress.    So if the fee that sendtoaddress generates pseudo-randomly is higher than the limit, then it would not send and would return an "insufficient fee" code.   If it is less or equal, then it would send.

So I believe that handles your objection.

[2112]

My concern is that if I (or anyone) go to the work to fix it and submit a patch, it won't get accepted, due to core team resistance.

Am I wrong?

Well, do something different than all the previous "enterprise" programmers. There was a common thread in all the "enterprise" patches submitted to the core team: they interfered with or completely destroyed the blockchain reorganization behavior. I'm on the record disagreeing with various technical issues regarding Bitcoin Core client, but I fully support the Bitcoin Core developers in their insistence on preserving the fundamental properties of the Bitcoin protocol: the global consensus is achieved through a somewhat random convergence that could involve orphaning some blocks.

The list of things you seem to be missing is actually longer:

1) The coin-selection is a non-trivial problem that is NP-complete in a general case (I've already mentioned that the current solution is a stochastic approximation, which is non-monotonic and therefore breaks the code like yours)

2) Bitcoin wallet isn't static while your code snippets are running, new blocks can appear randomly and change the coin selection both by adding and removing coins from it. (This is the most common mistake in the various "enterprise" patches that I've seen: mishandling of orphan blocks)

3) After you correctly solve the first two problems there is an issue of repeated greedy local optimization leaving wallet full of near-dust coins, so the optimization needs to be upgraded to some non-local, near-global or at least allow for optimizing coin selection for a set of transactions.

4) After you correctly solve problem (3) you need to provide a solution for the common operational requirement of keeping 2 wallets: hot and cold, and manage the coin transfers between the two of them that fulfill the non-scalar optimization goal of maximizing security and minimizing fees.

I hope that the above list helps you understand the true "enterprise" requirements and why your proposal is nothing more than a short-sighted temporary hack.

From myself I'll add that any true "enterprise" solution would obey http://en.wikipedia.org/wiki/Two-phase_commit_protocol and will easily integrate with any of the available http://en.wikipedia.org/wiki/Transaction_processing_system . From your insistence on using unstructured RPC calls I surmise that you have neither "enterprise" coding experience nor even plain multi-threaded development.

So what technical reason prevents that capability from being exposed at the RPC API level?

The technical reason is called "inversion of control" (i.e. server calling client to make a decision). Again this was discussed on this forum many times, at least since the wx->Qt UI transition. Use the search function.

You seem to be missing the point that this is a sharp pain point for developers and businesses.

If it is really such a "sharp pain point" then you are better off basing your work on a different code that Bitcoin Core and submit raw transactions to be reference implementation.

[btc4ever]

Hi 2112.  I appreciate your taking the time to reply in depth.

That said, it seems that whenever the subject of fee calculation via API comes up, there is a lot of negativity shooting the messenger and very little contructive effort to actually come up with a solution.   I'm hoping to see more of the latter this time around.

anyway, I will address your points.

1) The coin-selection is a non-trivial problem that is NP-complete in a general case (I've already mentioned that the current solution is a stochastic approximation, which is non-monotonic and therefore breaks the code like yours)

a) How coin-selection works is an implementation detail of bitcoin core.  It is not something that API clients should need to know or care about.  Getting hit with a surprise fee that is 20%, 50%, or larger of the send amount IS something that API clients should and DO care about.

b) I do not see how this "stochastic approximation" breaks the sample application code I pasted.  You see, with my proposed modification, we can ignore what goes on under the hood with coin-selection.  When sendtoaddress is called, it works its' coin-selection black magic and determines a fee.  If that fee is higher than the max_fee it received from the API caller, then sendtoaddress returns with an "insufficient fee" error code. Otherwise it sends.  So that is simple, right?

Now, in addition to that, I proposed as a nice-to-have, an additional API calcsendfee that would take the same address and amount args as sendtoaddress.  It would call the same black magic code to select inputs and return a fee amount to the caller.  The caller could then decide whether to accept or reject that fee amount.  If it accepts it, it could then pass that fee amount as a LIMIT to sendtoaddress.  So sendtoaddress may come up with a new fee based on pseudo-randomness or input changes, and if the new fee is higher than the limit, then the send will fail.  If it is lower, then the lower fee will be used.  Everybody is happy.

Probably calcsendfee should be called estimatesendfee instead, since that is technically more accurate.

The main purpose of estimatesendfee is to provide an "order of magnitude" type of estimate to the caller, so it doesn't have to waste cycles starting with a very low limit (eg 0) and working upwards calling sendtoaddress.  This API could also be useful to determine how much fee variation is taking place from the pseudo-random behavior.

2) Bitcoin wallet isn't static while your code snippets are running, new blocks can appear randomly and change the coin selection both by adding and removing coins from it. (This is the most common mistake in the various "enterprise" patches that I've seen: mishandling of orphan blocks)

Yes indeed.  I'm well aware and my proposed API change accounts for that.  See 1b above.

Nevertheless I would not be at all surprised if in the majority of cases, the fee calculated by sendtoaddress would be the same as the one returned by estimatesendfee.  Anyway, that would be interesting to watch.

3) After you correctly solve the first two problems there is an issue of repeated greedy local optimization leaving wallet full of near-dust coins, so the optimization needs to be upgraded to some non-local, near-global or at least allow for optimizing coin selection for a set of transactions.

I'm not sure what you mean by "solve the first two problems".  Rather than "solve" them, I just observe that those are implementation details the caller doesn't need to know about, and can be worked around with the proposed fee limit.

4) After you correctly solve problem (3) you need to provide a solution for the common operational requirement of keeping 2 wallets: hot and cold, and manage the coin transfers between the two of them that fulfill the non-scalar optimization goal of maximizing security and minimizing fees.

yeah, that's one of the use cases this proposal is aimed at.  See original description.   If one is able to define a max fee, then one can enforce a business rule that we do not pay above 1% fees (for example) when transferring to cold-storage.   If the fee would be higher, one can then choose to wait until a later time, or retry and maybe get lucky with a new coin selection, or create a raw transaction.

I hope that the above list helps you understand the true "enterprise" requirements and why your proposal is nothing more than a short-sighted temporary hack.

I don't see that this list added anything to the discussion that I had not already considered and/or discussed already in this thread. Nevertheless I thank you for the opportunity to more fully elaborate.  I also don't think that calling it a "short-sighted temporary hack" is justified or polite.

If you have a far-sighted, non-hackish proposal in mind, please do share.

From myself I'll add that any true "enterprise" solution would obey http://en.wikipedia.org/wiki/Two-phase_commit_protocol and will easily integrate with any of the available http://en.wikipedia.org/wiki/Transaction_processing_system . From your insistence on using unstructured RPC calls I surmise that you have neither "enterprise" coding experience nor even plain multi-threaded development.

surmise whatever you wish.  I'm not here out of ego.  I just want something implemented that is useful.

I have no objection to a 2 phase commit approach.  That's what I meant by locking/state in an earlier comment.  However I think that perfection is often the enemy of the good, and right now we have neither.   I believe my proposal is at least workable and should be simple enough to implement that even someone with relatively little experience with bitcoind code like myself could do it.  While I would not trust myself to implement 2 phase commit correctly.

Again:  if you have something better in mind, please post it here.  I would very much like to see the actual API calls you propose and pseudo-code for using them. That can lead to more productive discussion.   Otherwise, it seems you prefer to just cast stones...


I'm still hoping we can get some type of consensus that either:

a) that my proposed API changes make sense and will be accepted if implemented well.   - OR -
b) a core dev will implement some type of 2 phase commit for a true calcsendfee API

So what technical reason prevents that capability from being exposed at the RPC API level?

The technical reason is called "inversion of control" (i.e. server calling client to make a decision). Again this was discussed on this forum many times, at least since the wx->Qt UI transition. Use the search function.

This is getting off into the woods a bit.  I agree it is messy for the server to call the client.  Still I don't see what prevents something like:  lockallunspent( true );  fee = calcsendfee();  sendtoaddress( ..., fee );  localallunspent( false );    Anyway, I don't wish to debate about this, because my proposal handles it in another way without requiring locking.

If it is really such a "sharp pain point" then you are better off basing your work on a different code that Bitcoin Core and submit raw transactions to be reference implementation.

I went down that route.  Unfortunately the required fees keep changing from release to release, so properly estimating them is tedious and error-prone.   Another factor for me personally (and for some others) is that my code is designed to work with altcoins as well as bitcoin, so it is much cleaner if the wallet software itself lets me know what an appropriate fee is and/or enforces a limit.

[2112]

I had to read your reply twice before I understood your key argument.

I also don't think that calling it a "short-sighted temporary hack" is justified or polite.

Perhaps you'll find the original H.L.Mencken complete quote polite enough:

Explanations exist; they have existed for all time; there is always a well-known solution to every human problem — neat, plausible, and wrong.

In other words, what I see here is another shtylman/Bitfloor or davout&Boussac/Instawallet situation brewing: substandard and/or questionable implementation; then a "hack" and loss of customer's funds. Or maybe just another "accidental double payment" like the ones that happened to GLBSE and ASICMINER.

I tried, but my work here is done. Now the emptor will have to caveat.

Edit: I'll save the btc4ever's signature block for the future reference.

Psst!!  Wanna make bitcoin unstoppable? Why the Only Real Way to Buy Bitcoins Is on the Streets. Avoid banks and centralized exchanges.   Buy/Sell coins locally.  Meet other bitcoiners and develop your network.   Try localbitcoins.com or find or start a buttonwood / satoshi square in your area.  Pass it on!

[btc4ever]

Hi 2112.  You haven't yet said a single thing that demonstrates a technical flaw in my proposal.  You also have not proposed something better or indeed anything at all.

If my proposal is "wrong" as you say, please support that statement by providing a scenario in which the outcome would not be as desired/expected by the API caller or otherwise somehow wrong/broken behavior. I doubt you can because the approach is very simple and straight-forward.  But I'm willing to consider anything concrete.  If you can point an actual flaw, then either I need to find a way to fix it, or the proposal is unworkable.  But you have not done this.

Also if you'd like to explain how the status quo is somehow better than my proposal, I'd also be mildly amused to read that justification, even though I'm sure I wouldn't agree.  Because the status quo is pretty broken with regards to fees.  It's like walking in to MoneyGram, handing them your credit card, asking them to send 10 cents for you, and please surprise me with the send fee.... the sky is the limit.  ( moneygram would probably say no to sending 10 cents, but bitcoin doesn't, so the analogy is apropriate. )

While I appreciate your contributing to the thread, I sense only negativity from you.   If you are done, that is probably just as well.

Does anyone else have comments?


Tomorrow I will consolidate the estimatesendfee addition into the original proposal for easier reading.