CreateTransaction: suggest/enforce fee for big low-priority transactions

[jgarzik]

Having a memory pool size limit where transactions are removed based on priority would solve that, I think. Enabling the -limitfreerelay switch by default might also help (with lower minimum fees).

Agreed x 2

Lack of that size limit is really an unfixed DoS issue.

[jgarzik]

This is pretty simple:

Code:

diff --git a/main.cpp b/main.cpp
index a47f3a9..0e56b59 100644
--- a/main.cpp
+++ b/main.cpp
@@ -739,7 +739,8 @@ bool CTransaction::AcceptToMemoryPool(CTxDB& txdb, bool fChe
             return error("AcceptToMemoryPool() : not enough fees");
 
         // Limit free transactions per 10 minutes
-        if (nFees < CENT && GetBoolArg("-limitfreerelay"))
+       bool fLimitFreeRelay = GetBoolArg("-nolimitfreerelay") ? false : true;
+        if (nFees < CENT && fLimitFreeRelay)
         {
             static int64 nNextReset;
             static int64 nFreeCount;

[caveden]

No, what I suggest is making such limit adjustable just like the difficulty factor, as a "network rule", not something a user could change. There in the topic I explain more: http://bitcointalk.org/index.php?topic=1865.0

I'm still against that. Until there is some out-of-band method for lightweight clients to be notified of their own transactions, the max block size must be kept low enough for everyone to keep up with blocks. It's not reasonable to expect every client to download 100MB per hour (or whatever).

The block size wouldn't reach 100MB unless that was really needed. And if it's needed, clients would need to adapt anyway (ligthtweight clients or banks, whatever).
And such increase should be gradual, somebody there in the topic suggested making the maximum limit something like 110% of the average size from the last N blocks. It might be a good approach, in order not to increase too fast.

Anyway, I understand your concern.
Any chances of having the blockexplorer data provided via JSON and/or RPC/RMI/etc? Having an interface like that could make it easier for the development of such lightweight client.

[caveden]

Currently clients will not even relay transactions if their fees are "too low". This needs to be changed for the fee market to work.

I disagree.  You cannot change that without vastly increasing the ability to spam the network with useless transactions.

It's not up to the client - specially, not up to the default implementation of it - to decide what are "useless" transactions or not. It's up to the miners. The clients should reject only illegal transactions, like double-spending, wrong signatures etc.

[Gavin Andresen]

RE: limitfreerelay:

Agreed, that's better.  I think we should start soft-coding the amount that is considered "free" instead of hard-coding CENT, and make the default less than 0.0001 BTC.

Separate subjects:

The entire transaction memory pool should be size-limited, with lower-priority transactions dropped.

I agree with Steve-- free transactions are a HUGE selling point, and I think rational miners should realize that their bitcoins will be more valuable if there are more users.  And there will be more users if "free transactions" is a feature.  I think we can come up with a solution where "normal" transactions are free, but spam or abnormally complicated transactions (yes, like those bitpenny rewards pooled mining participants are currently getting once a day) "require" a fee (where "require" is really "if you don't include a fee your transaction will probably never be confirmed.")

And finally... we've got a problem right now; lets think about what fix(es) move us in the right direction quickly, or think about fixes that will solve the current problem (that we may need to undo/rework later).

[lfm]

Can we make it easier to create multi-output txn for peole like slush to distribute to large lists?

 That could save a lot of bytes both network and disk in the long run. pudinpop had custom ways of making them in his miner.

[caveden]

And finally... we've got a problem right now; lets think about what fix(es) move us in the right direction quickly, or think about fixes that will solve the current problem (that we may need to undo/rework later).

Well, the quickest solution to the current problem would be raising the free space, I guess. Say, 3 times the current values... ?

Also, stop picking which transactions to relay based on fees... the transaction memory pool size is another matter*, but at least relaying all valid transactions should be the default behavior. But this I don't know if it would be quick to develop...

*By the way, why should a client which is not mining keep a transaction pool? Shouldn't we just leave it to the interested parties (sender and receiver) to rebroadcast the transactions once in a while if they look like "forgotten"?

[StevenC]

Hi, I made some further observations regarding the stalled transactions that started off this discussion (so hopefully this is on-topic) :

Code:

ERROR: AcceptToMemoryPool() : ConnectInputs failed 8074fb67ce

ERROR: ConnectInputs() : 8074fb67ce mapTransactions prev not found 864bef9df1
ERROR: AcceptToMemoryPool() : ConnectInputs failed 8074fb67ce

ERROR: ConnectInputs() : 8ec51bbaf5 mapTransactions prev not found 8074fb67ce
ERROR: AcceptToMemoryPool() : ConnectInputs failed 8ec51bbaf5

ERROR: ConnectInputs() : f9ccaa5e8b mapTransactions prev not found 8ec51bbaf5
ERROR: AcceptToMemoryPool() : ConnectInputs failed f9ccaa5e8b

ERROR: ConnectInputs() : 7dddd55332 mapTransactions prev not found f9ccaa5e8b
ERROR: AcceptToMemoryPool() : ConnectInputs failed 7dddd55332

ERROR: ConnectInputs() : dafce7d622 mapTransactions prev not found 7dddd55332

I then realised these transactions are being processed sequentially, only one per block:

• tx 8074fb67ce... was in block http://blockexplorer.com/b/111257
• tx 8ec51bbaf5... was in block http://blockexplorer.com/b/111258
• tx f9ccaa5e8b... was in block http://blockexplorer.com/b/111259
• tx 7dddd55332... was in block http://blockexplorer.com/b/111260

Any guesses tx dafce7d622... makes block #111261?

I find it very odd that exactly one transaction from this 'dependency chain' is being included into block.  Is this really the expected behaviour?  A mere coincidence?  Or something to be worried about?

My graph of 'dependency chains' I've observed in my debug.log is here:  http://pyro.eu.org/stuff/bitcoin-txqueue.pdf

Update: I think it's okay... in #111261 block, a huge chunk of items from that 'chain' I was looking at, made it into the block.  I guess there's nothing to be worried about.

[Hal]

When investigating this I noticed lots of errors in ~/.bitcoin/debug.log like this:

Code:

ERROR: ConnectInputs() : 94ba8e6447 mapTransactions prev not found d884bcbf17
ERROR: AcceptToMemoryPool() : ConnectInputs failed 94ba8e6447

Some of these partial txid's seem to 'cascade' on from one another, so I figured there was a dependency chain forming here.  I've managed to render this in the PDF below;  maybe that helps people to visualise what's going on.  If anyone else has transactions not completing, they may try searching for the transaction ID in the PDF document.  I found my transaction in there with at least 5 preceding it.  I count about 275 transactions in total at the moment, but I don't currently have the means to check if any of the displayed transactions eventually got processed.

• http://pyro.eu.org/stuff/bitcoin-txqueue.pdf

I'd suggest people look at this PDF, make sure we're fixing the right problem. This is a graph of orphan transactions showing dependencies. Note that there's a chain of 100+ transactions all dependent on the one before. Probablythis is the Faucet.

It goes without saying that a tx can't go in a block if it's predecessors are not in blocks. I wonder if a single faucet txn getting dropped could lead to a whole chain of them being blocked?

[StevenC]

It goes without saying that a tx can't go in a block if it's predecessors are not in blocks. I wonder if a single faucet txn getting dropped could lead to a whole chain of them being blocked?

Transactions seem to be being processed 'up' the chain (the opposite direction of the arrows), albeit very slowly.  Usually only 1 in a block, but http://blockexplorer.com/b/111261 contained about 10 in a row (from the same chain).

ps. the green was my attempt to highlight transactions that have been confirmed, but my data for that was unreliable/wrong.

[theymos]

Probablythis is the Faucet.

It could also be the pool, which sends many little transactions that are dependent on each other. Ideally it would send them all in one transaction.

It goes without saying that a tx can't go in a block if it's predecessors are not in blocks.

The protocol allows for a transaction and its predecessor(s) to be in the same block. The priority mechanism just prevents it from ever happening (for miners running mainline Bitcoin).

[slush]

Can we make it easier to create multi-output txn for peole like slush to distribute to large lists?

+1

[jgarzik]

Can we make it easier to create multi-output txn for peole like slush to distribute to large lists?

+1

I can see a possible use case for a "mass pay" feature, but that does not really solve the problem here, where slush is generating a bunch of free transactions.  The use of a mass-pay feature without paying a TX fee might result in creating a single transaction that is larger than the entire free-transaction area (27k).

You wind up with the same end result: stalled transactions.  The root cause of the problem is that the pool is generating a lot of work for the network, without paying for it.  This is called Tragedy Of The Commons, or less graciously, free-loading.

So, I would support the following tweaks,

• Convert 27k free-TX area to purely score-based, eliminating 4000-byte limit
• Make -limitfreerelay the default
• Add a mass-pay JSON-RPC method, provided that the user interface requires a TX fee parameter (NOTE: zero is a valid TX fee)
• Reduce dust-spam/fee triggers from 0.01 to 0.001

[Raulo]

You wind up with the same end result: stalled transactions.  The root cause of the problem is that the pool is generating a lot of work for the network, without paying for it.  

The pools contribute to the problem but currently the largest volume of transactions comes from the faucet.

[slush]

I can see a possible use case for a "mass pay" feature, but that does not really solve the problem here, where slush is generating a bunch of free transactions.

This is not about pool itself; pool is just one of the first case for large amount of tiny transactions. In my specific case, I can (in the future) include my own transactions into the next mined block for free (or with fees paid to myself .

You wind up with the same end result: stalled transactions.  The root cause of the problem is that the pool is generating a lot of work for the network, without paying for it.  This is called Tragedy Of The Commons, or less graciously, free-loading.

I'll be happy with not spamming the network, but this is not the real problem; I solve my case somehow (rising sending threshold, including txes into my own block). But other people will hit the same problem later and maybe some of them won't be so kind to save Bitcoin network resources - so we're talking generally about general scalability of Bitcoin transactions and about the painless way how to avoid this. I think we all agree that current infrastructure for fee negotiation is not powerfull enough at this time.

So, I would support the following tweaks,

• Convert 27k free-TX area to purely score-based, eliminating 4000-byte limit
• Make -limitfreerelay the default
• Add a mass-pay JSON-RPC method, provided that the user interface requires a TX fee parameter (NOTE: zero is a valid TX fee)
• Reduce dust-spam/fee triggers from 0.01 to 0.001

I'm perfectly fine with that.

It would be also nice to allow specific fee for every single transaction using JSON RPC. Then I can quickly introduce new settings for pool users - they will be able to choose if they want to pay fees for their withdrawals or not. It effectively move the decision about spamming of bitcoin network from me to them. (I'm pretty sure that those people withdrawing every 0.1 won't select fees, but they can spam network from their clients with zero-fee transactions as well).

[Syke]

Someone just unclogged the drain:

http://blockexplorer.com/b/111264

[StevenC]

The pools contribute to the problem but currently the largest volume of transactions comes from the faucet.

And also from people who are abusing it, and are funneling cash back into their primary account, often without TX fee...

http://blockexplorer.com/a/8i3b7nXppE
(View transaction detail -- many 0.05 receipts.)

This came out just now in #111264 when someone 'unclogged the drain':
http://blockexplorer.com/t/APn3cXG9Z1

# Number of inputs?: 181 (Jump to inputs)
# Total BTC in?: 9.05  => 181*0.05
# Number of outputs: 2 (Jump to outputs)
# Total BTC out?: 8.72
# Size?: 32.661 kilobytes

IMHO though, if anyone could maliciously generate a comparable volume of free transactions, that's still a potential issue, even if the faucet is fixed.

[carp]

RE: limitfreerelay:

Agreed, that's better.  I think we should start soft-coding the amount that is considered "free" instead of hard-coding CENT, and make the default less than 0.0001 BTC.

Separate subjects:

The entire transaction memory pool should be size-limited, with lower-priority transactions dropped.

I agree with Steve-- free transactions are a HUGE selling point, and I think rational miners should realize that their bitcoins will be more valuable if there are more users.  And there will be more users if "free transactions" is a feature.  I think we can come up with a solution where "normal" transactions are free, but spam or abnormally complicated transactions (yes, like those bitpenny rewards pooled mining participants are currently getting once a day) "require" a fee (where "require" is really "if you don't include a fee your transaction will probably never be confirmed.")

And finally... we've got a problem right now; lets think about what fix(es) move us in the right direction quickly, or think about fixes that will solve the current problem (that we may need to undo/rework later).

So if I understand this right, you want to make "fee" actually "very very small sub-subcent fee". I do think that, in the interests of openness and truth, we should stop using the term free at that point. How about making it a very small percentage? That way, when (yes when... I become an optimist when I have a vested interest) the value goes up to the point that .0001 bitcoins is an average persons yearly salary (I can dream)... he can still use it.

Honestly, Bill Gates is known for having asked why a computer would ever need more than 640k. Early predictions on computers were that they would eventually take up even bigger rooms, 16 bit gave way to 32 bit, and now 64 bit architectures. Deciding on the limits of a system are notoriously hard.

I kind of like including some sort of time metric into it. Maybe a strict queue by score where the score starts at the base fee of the transaction, and adds that transaction fee again to the score whenever the client has drawn an increment of the max block size in bytes from the queue. Maybe instead of using the raw fee, the ratio of the size in k to the fee.

So low fee/byte transactions start out at the very back of the line, and slowly move up to the front. Then, the size of the backlog has a direct relationship to the size of fee needed to jump to the front of the line. Though, you can still buy your way into the middle easily, and that would end up being rather close to the front, since your tx would be in a "faster moving line"... which eases the drawbacks and just means, well yah.... shared resources are what they are.

[molecular]

I don't agree with eliminating free transactions...in fact, I believe the likely trajectory of things is that transactions will be remain free (I mean just look at all the unprofitable mining that's happening now).  Merchants, exchanges and many others have an interest in facilitating transactions and will always step in to ensure they get processed, even the free ones.  In fact, everyone using bitcoins has an interest in free transactions.  It's a good selling point and a way to attract new users (and without new users, this experiment will fail).

Point taken.

A question: What would happen if we removed all special handling of "free" transactions, but still allow transactions with fee == 0.00 and somehow include the fee into priority value calculation (iirc fee is not part of that atm, it's currently "sum(valuein * age) / txsize").

Maybe "sum(valuein * age) * fee / txsize"?

In the current situation this might be better, because there's actually still space left in the blocks, just no "special space for free tx". So the network is not living up to it's transaction processing potential. If this is true (I might be missing something, though), this suggestion should actually be in your "tx should remain free"-spirit, because it gives more room for free transactions, right?

Another thought: I see in the code that after calculcation of priority value for each transaction, the dependencies of the ones that are included are pulled in. What about packaging transactions into groups by dependency first and then calculating priority values for these groups instead of single transactions? Would that change anything significantly?

[carp]

I don't agree with eliminating free transactions...in fact, I believe the likely trajectory of things is that transactions will be remain free (I mean just look at all the unprofitable mining that's happening now).  Merchants, exchanges and many others have an interest in facilitating transactions and will always step in to ensure they get processed, even the free ones.  In fact, everyone using bitcoins has an interest in free transactions.  It's a good selling point and a way to attract new users (and without new users, this experiment will fail).

Point taken.

A question: What would happen if we removed all special handling of "free" transactions, but still allow transactions with fee == 0.00 and somehow include the fee into priority value calculation (iirc fee is not part of that atm, it's currently "sum(valuein * age) / txsize").

Maybe "sum(valuein * age) * fee / txsize"?

In the current situation this might be better, because there's actually still space left in the blocks, just no "special space for free tx". So the network is not living up to it's transaction processing potential. If this is true (I might be missing something, though), this suggestion should actually be in your "tx should remain free"-spirit, because it gives more room for free transactions, right?

Another thought: I see in the code that after calculation of priority value for each transaction, the dependencies of the ones that are included are pulled in. What about packaging transactions into groups by dependency first and then calculating priority values for these groups instead of single transactions? Would that change anything significantly?

I actually like this formula. Thinking about it in these terms, its basically similar to what I was trying to articulate, except with valuein added. Though, if you are going to use the fee, why not drop valuein? From a miner's point of view, why should a transaction moving 100 btc with a .01 btc fee be preferred to one moving 10 btc with a .02 btc fee? (assuming txsize is equal).

Otherwise someone with a lot of btc could dos the system with a bunch of large transactions that clog the front of the queue, and pay no fee at all. Admittedly this could require a lot of btc to accomplish, but it costs none, and the attacker could have stolen them or have some reason to invest alot in jamming the system.

On the other hand, if you drop valuein entirely, then he has to clog from the back of the line, or drain his attack fund with fees, which will go back to the miners.