[BOUNTY] A patch for bitcoind to modify tx list in "getmemorypool"

[DeathAndTaxes]

Proposal:
A patch to bitcoind that modifies tx list returned by getmemorypool by excluding tx below a user defined fee.

Background:
The getmemorypool RPC call returns the components necessary to build a block header.

version" : block version
"previousblockhash" : hash of current highest block
"transactions" : contents of non-coinbase transactions that should be included in the next block
"coinbasevalue" : maximum allowable input to coinbase transaction, including the generation award and transaction fees
"time" : timestamp appropriate for next block
"bits" : compressed target of next block

https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_calls_list

The getmemorypool RPC call is used by p2pool to generate blockheaders for getwork requests by miners.  The default action of "getmemorypool" is to return all valid tx not yet known by the node to be in a block. I am seeking development of a patch which will exclude transactions with a transaction fee below the min fee set by the operator.  The list of tx returned should simply be the tx with an appropriate fee.

Implementation:
I imagine the easiest way to accomplish this is to have a new RPC call "setminfee".  By default "setminfee" is set to 0 BTC.  The modified "getmemorypool" would check the min-fee value (default 0) and exclude any tx with fee < min fee.  I am open to alternative implementation however a reason should be provided.

Bounty:
Make me an offer.  Honestly I have no idea the amount of work this will involve.  I don't want to vastly overpay or insult anyone with a low ball offer.

[1714040466]

1714040466

[1714040466]

1714040466

[1714040466]

1714040466

[1714040466]

1714040466

[1714040466]

1714040466

[Luke-Jr]

Actually, bitcoind already excludes transactions that don't meet the hard-coded fee rules. I'd be glad to write the simple JSON-RPC method to configure it for only 8 BTC/hr*. I expect it would take at most 4 hours, probably less.

* 8 BTC/hr rate assumes I publish it under the MIT license. I don't care who holds the copyright itself.

[Polvos]

I'm absolutely agree with your statements D&T so I will contribute here. I lack the programming skills and don't understand deeply the BTC protocol but I can donate some coins to the bounty.

[DeathAndTaxes]

Well Luke I waited to see if anyone else was interested and I guess not so the job is yours.

MIT license is fine.  You can retain the copyright (unless you don't want you name associated with the code in which case it can be copyrighted under my name).

8 BTC/hr * likely <4 hrs = 32 BTC.

20 BTC advanced and balance due upon completion sound ok?

Just to clarify this min fee would be in addition to any spam rules.  So bitcoind should do its normal spam checks and then exclude any tx (i.e setting min fee to 0.001 should still result in spam tx without min spam fee from being excluded).   I could see some people setting the min fee to 1E-8 to exclude only free tx and I don't want the min fee rule to accidentally compromise any spam prevention.

One thing I did think of is making bitcoind smarter to detect dependencies and including tx chains but that likely will need to wait for a future bounty.  What I mean is I send 20 BTC from A to B (you) but I include no fee.  You get tired of waiting so you make a tx based on the unconfirmed A->B as an input sending it to C with a fee.  If the bitcoind was "smart" enough someday it could detect that B-> C depends on A->B and include both even if A->B is under the min fee (as long as B->C is over min fee).

Let me know if the terms are ok and provide me an address to deposit the 20 BTC advance.

[Luke-Jr]

20 BTC advanced and balance due upon completion sound ok?

Sure. 1FXGUecLqVPdqGUR5qEpLt9QUjGg5P1k2K

Just to clarify this min fee would be in addition to any spam rules.  So bitcoind should do its normal spam checks and then exclude any tx (i.e setting min fee to 0.001 should still result in spam tx without min spam fee from being excluded).   I could see some people setting the min fee to 1E-8 to exclude only free tx and I don't want the min fee rule to accidentally compromise any spam prevention.

So if vanilla bitcoind would charge 0.0005 BTC for a fee, you want to charge 0.0005 BTC + "min fee" for the same? Or just the greater of the two?

One thing I did think of is making bitcoind smarter to detect dependencies and including tx chains but that likely will need to wait for a future bounty.  What I mean is I send 20 BTC from A to B (you) but I include no fee.  You get tired of waiting so you make a tx based on the unconfirmed A->B as an input sending it to C with a fee.  If the bitcoind was "smart" enough someday it could detect that B-> C depends on A->B and include both even if A->B is under the min fee (as long as B->C is over min fee).

Yeah, I've wanted this kind of functionality for a while, but I haven't been able to justify spending the time to figure out the algorithms it would need.

[DeathAndTaxes]

So if vanilla bitcoind would charge 0.0005 BTC for a fee, you want to charge 0.0005 BTC + "min fee" for the same? Or just the greater of the two?

Although it is an edge case I think greater of the two would work the best.

One thing I thought of is it would be useful to have a getminfee to complement setminfee.

getminfee - returns minfee value
setminefee -sets minfee value (default 0)
getmemorypool - modified to exclude tx which have a fee < minfee.

20 BTC sent
tx id: 76c9a9cda85e34560c1c0f2ce827f71e0e92bcb2a42ec1e4bdd6dd8174e738e4

Let me know if you have any questions.

[Luke-Jr]

One thing I thought of is it would be useful to have a getminfee to complement setminfee.

Probably sufficient to just add it to the getmininginfo results?

[DeathAndTaxes]

One thing I thought of is it would be useful to have a getminfee to complement setminfee.

Probably sufficient to just add it to the getmininginfo results?

Sure that works too. 

[Haplo]

Just sort of a relevant comment while you're on about this, but what about making fees proportional to the amount being sent? ie charging 0.1% as a fee rather than a nominal BTC amount. You could just use a higher % for extra small tx.

The main reason I would suggest doing it that way is since the trade value of BTC fluctuates so much it may be impractical on the user end to deal with fees set in nominal terms, whereas %s scale automatically and are simple to understand.

[Luke-Jr]

Just sort of a relevant comment while you're on about this, but what about making fees proportional to the amount being sent? ie charging 0.1% as a fee rather than a nominal BTC amount. You could just use a higher % for extra small tx.

The main reason I would suggest doing it that way is since the trade value of BTC fluctuates so much it may be impractical on the user end to deal with fees set in nominal terms, whereas %s scale automatically and are simple to understand.

It's not possible to determine how much is being sent, and most of the transfers you would want to "tax" more are inversely proportional to that amount anyway. I intend to make it such that you can set it per-N-bytes, though.

[DeathAndTaxes]

A couple reasons why % based fee systems are not compatible with Bitcoin:

1) The bitcoin network has no knowledge of the payment amount only the total inputs and total outputs.  I paid 20 BTC with an input worth 180 BTC.  If the miner charged 1% I would need to pay 1.8 BTC in fees.  If I from inputs worth 20 BTC I would only need to pay 0.2 BTC.  Given users have no direct control over which inputs are used it would only lead to confusion.

2) As a miner my costs is not dependent on the tx value.  Processing a 1,000,000 BTC tx takes no more work than processing a 1 BTC one.  Charging a % simply takes more value from higher valued txs for no valid reason other than you can.  The analogy I would use is that Bitcoin cost structure is more like a wire transfer or check than a credit card.  

3) While BTC changes the daily fluctuations aren't particularly meaningful.  Over the last 90 days 0.01 BTC has had a value in USD of 4.2 to 8.1 cents.  I doubt many people would say "oh for 4.2 cents I would send that but not for 8.1 cents".

4) The critical resource is space in the block chain which doesn't depend on tx value.  A 1,000,000 BTC tx involving a few inputs and a single output may only require 1 KB of space.  On the other hand a 1 BTC tx made up of a hundreds of junk (0.00001, 0.00038, etc.) inputs could take 20KB+.  Charging more for the low cost tx and less for the high cost tx is illogical.

A miner with a fixed min fee will need to periodically adjust that fee.  Obviously 0.01 BTC @ ~$ USD is 5 cents if BTC:USD rises by a magnitude (say to 50 USD) then revising the fee schedule is likely required.  Still if one kept a fee of 0.01 BTC it isn't materially more or less expensive as long as BTC trades in a range of 2 USD to 9 USD which means miner wouldn't need to adjust very often.

[DeathAndTaxes]

I intend to make it such that you can set it per-N-bytes, though.

Good point.  In hindsight it should be coupled to the critical resource which is blockchain space.

[Luke-Jr]

Here's your first draft: https://github.com/luke-jr/bitcoin/compare/customfee

Time billed: 30 mins

Awaiting input/comments...

[Matt Corallo]

Out of curiosity, can I ask why you want this?

[Polvos]

Out of curiosity, can I ask why you want this?

I think he wants some FREEDOM choosing what transactions to add in a block he's mining. Those above 0,01 BTC fee specifically.

[DeathAndTaxes]

Here's your first draft: https://github.com/luke-jr/bitcoin/compare/customfee

Time billed: 30 mins

Awaiting input/comments...

Looks good so far.  I have to compliment you.  We don't always agree on everything but you write some solid code.   Very clean and well documented.

Only have three minor comments ...

on line 64 (main.cpp)

Code:

int64 nMinFeePer = 1000;

I would change this to 1024.  It will be much easier to understand if miners announce pricing either as flat value or per KB  (i.e. "0.01 BTC per KB").  

on line 5222 (init.cpp)

Code:

if (nMinFeeBase / nMinFeePer > 0.00025 * COIN

Since this is for warning on excessively high fees I would increase the threshold.  The current threshold ~0.1 US cents even if BTC value increased significantly it would still be sub 1 US cent.  I would increase the warning point to 1 bitcent.  (> 0.01).  While 1 bitcent may be "high" it is likely an intentional value.  Someone misunderstanding the concept or units (thinking they are satoshis, fractional BTC or %) would likely put a much higher value and still be warned.

one line 592-602 of main.h
Not sure what is happening here (this bounty gives me more excuse to look into bitcoind sourcecode).   My first thought is we are comparing the bitcoind "spam" min fee to the user defined fee?  The spam fee depends on coinage though and the processing fee shouldn't.  Likely I am misunderstanding so feel free to school me.

I like it so far.

To anyone reading.  If you have any comments please provide them.  Although I am paying the bounty I intend for this to be open source (MIT licensed) and hopefully someday be integrated into the mainline.  The quicker we start the "open" discuss the more relevent this module will become to mining community.

[DeathAndTaxes]

Out of curiosity, can I ask why you want this?

I think he wants some FREEDOM choosing what transactions to add in a block he's mining. Those above 0,01 BTC fee specifically.

Thats it.  Although 0.01 is somewhat arbitrarily chosen by me.  We need to start somewhere. I feel the service provided is worth AT LEAST 1 bitcent and it is a nice round value.   The larger goal is to allow ALL miners (not just me) to see fees at a level they feel aproporiate.  

If anyone has an issue with 0.01 BTC as a fee amount it is better discussed in another thread on appropriate fee pricing.  The code would allow enforcing fees on as low as flat 1 satoshi per tx (of unlimited size - although soon you would hit the spam limit).

I don't want to clutter this thread up too much but from this humble begining it would be my hope to see fee handling become more robust over time.   Bitcoin will eventually require fees to support at least part of the network cost.  Miners will need robust tools to price their service appropriately:

Some ideas for enhancement (future versions outside the scope of this bounty):
* Allow "charity" windows.  Miner could set a certain # or % of tx to allow below the fee threshold.
* Provide protocol to allow miners to announce their hashing power (verified by PofW) and price.
* Update client side processing to take miner announcements and provide users simple and easy pricing
" for this transaction current mining power indicates a fee of 0.01 has 95% confidence of being in the next block,  a fee of 0.0005 has a 95% confidence of being in the next 20 blocks, no fee has only a 95% confidence of being included in the next 288 blocks"
* Allow re-fees.  Receiver adding fees to a tx chain to speed up confirmation (sender issues w/ no fee and receiver adds a fee on dependent tx to speed up confirmation).

[Luke-Jr]

on line 64 (main.cpp)

Code:

int64 nMinFeePer = 1000;

I would change this to 1024.  It will be much easier to understand if miners announce pricing either as flat value or per KB  (i.e. "0.01 BTC per KB").

bitcoind has always priced transactions per 1000 bytes (SI kB). I can certainly change it, but I don't think upstream will accept it that way.

on line 5222 (init.cpp)

Code:

if (nMinFeeBase / nMinFeePer > 0.00025 * COIN

Since this is for warning on excessively high fees I would increase the threshold.  The current threshold ~0.1 US cents even if BTC value increased significantly it would still be sub 1 US cent.  I would increase the warning point to 1 bitcent.  (> 0.01).  While 1 bitcent may be "high" it is likely an intentional value.  Someone misunderstanding the concept or units (thinking they are satoshis, fractional BTC or %) would likely put a much higher value and still be warned.

The warning limit here is 0.00025 BTC per byte, or 0.25 BTC per 1000 bytes. Surely that is sufficiently high?

one line 592-602 of main.h
Not sure what is happening here (this bounty gives me more excuse to look into bitcoind sourcecode).   My first thought is we are comparing the bitcoind "spam" min fee to the user defined fee?  The spam fee depends on coinage though and the processing fee shouldn't.  Likely I am misunderstanding so feel free to school me.

If the "spam fee" is higher than the user-defined fee, the spam fee is left alone. Otherwise, the user fee overrides it.

[DeathAndTaxes]

bitcoind has always priced transactions per 1000 bytes (SI kB). I can certainly change it, but I don't think upstream will accept it that way.

No reason to change it then.

The warning limit here is 0.00025 BTC per byte, or 0.25 BTC per 1000 bytes. Surely that is sufficiently high?

Oops. No that is more than sufficiently high.

If the "spam fee" is higher than the user-defined fee, the spam fee is left alone. Otherwise, the user fee overrides it.

Gotcha.  I see now that prior to this segment the min "spam fee" is already computed based on coinage of the inputs.

Looks good to me.  Thanks for the prompt response.

[Luke-Jr]

Looks good to me.  Thanks for the prompt response.

If this is complete, where would you like me to send a refund (if you want one) of your advance? Let me know if you would prefer to wait for testing and merging to mainline (which might require more time, depending on the reactions of other developers).

Polvos, you also offered to contribute toward the bounty; if you feel the desire to tip me, you can use 1DGzpZzce1c7nsg1SN7exV6bsaju1Mcrc6 - it's fine with me if you choose not to, though.

I've submitted an upstream pull request. Miners who want this feature should express their support on this thread, and developers with comments can do so on the pull request itself.