Bitcoin Forum

After spending more than a day frustrated that one of my transactions didn't get through, even though it contained a fee, simply because it was too large, I came up with the following idea. I know some of it was already discusses here and there, but I thought I should write it again. I am starting from the assumption that the network as a whole is fast (I remember reading ~20 seconds propagation time for a new tx? can someone point me to that post/article?). I am also taking a position for the user's interest against the miner's interests (in other words, I don't care if the miners have to work more, or waste hashes, or make sure they are on a fast, well-connected node, if it helps the network).

How about we add these simple rules to validate new blocks:

* transactions which contain fees more than the minimum recommended have to be included in the next 3 blocks (~ 1/2 hour).
* transactions which contain fees have to be included in the next 72 blocks (~12 hours).
* all transactions must be included in the next 144 blocks (~24 hours).

When receiving a new block, the nodes will reject it as invalid, if there are transactions on the network which should have been included but are not. The end.

This also has the nice side effect that it will render some attacks impossible, for example a common scenario states that a miner with >50% hashing power can postpone transactions indefinitely.

I realize that there are some timing problems here, which are not trivial to solve, but I would be interested to hear what you think about the general idea?

I wonder what you will do about the blockchain being 25920 megabytes larger 181 days from now?

Hardware and bandwidth are cheap (and I will be running a dedicated full note for as long as Bitcoin exists). Light clients like Electrum are being developed, so I simply don't care about the blockchain size. Neither should the users, whose interests are the most important - transactions must be confirmed in a reasonable amount of time.

Ideas which aren't attractive to miners will not be successful.

If your transaction doesn't get processed in the time frame you want, you should include a higher fee. That's the point of fees.

The only problem I see with the current implementation is that it doesn't seem like you can cancel your unprocessed transactions in order to try sending with a higher fee. But if that functionality is not possible in combination with all the benefits we have in the current implementation, so be it.

Fee - is the miner`s motivation for his work. If you enter a binding confirmation, then drop fee. Following will be leaving miners and the instability of the network. Some rules are necessary, but must keep track of all the consequences of the introduction of these rules.

I am not trying to alienate miners, but they have, IMHO, too much power regarding which transactions are included and when. Anyway, my rules are pretty reasonable for the miners too, I believe.

Also, I didn't say anything about the minimum fee. I have no problem increasing it or even making it a percentage of the transferred amount. I also don't have any problem in simply dropping no-fee transactions completely (maybe starting with block 210000) ?

What I do have a problem with is that at the moment it's impossible to predict when a transaction will be processed, even if you pay fees. This must be solved.

No, at the moment the 50 BTC/block prize is the motivation. Anyway, if some miners leave, the network will adapt. I do know that a lot of miners really believe in bitcoin and will continue mining even at a loss or with my rules in place.

How can you force a miner to include a tx in a block?  What if they don't update their software?

As I said, invalidate his blocks if he did not include old enough transactions. He will update if he wants to produce valid blocks. I am not suggesting we do this right now, of course, or we'll only have invalid blocks :). A good scenario would be to activate the rules starting with a specific block number, chosen so at that moment the majority of the network is already updated.

I think there should be a feature where a transaction can be resubmitted with a higher fee.  And where nodes will recognize the second transaction (which looks like a double-spend) as a fee-increasing transaction.

I am not sure size is the main factor: not too long ago, I sent a transaction with 312 outputs to load lots of physical bitcoins, with no fee, and it went through promptly.  Then again, that transaction may look a lot more favorable compared to somebody combining a bunch of penny inputs (inputs take up much more space than outputs) - my big transaction only had a single input.

Tx propagation is fast in optimal conditions but you can't assume all nodes are aware of all transactions at all times, so you can't declare a block "invalid" just because it is missing a transaction. Also you shouldn't expect miners to do work for you unpaid, if you want them to include your transaction you should pay them for it. In fact to make the network secure it is important that enough is paid to miners, what we should be looking for is how to make transaction fees as high as possible while still being insignificant to the users.

That said, I do indeed believe, and have suggested in the past, that branch selection should involve more than just the total difficulty, and include things like the total bitcoin-days destroyed in the block - precisely to prevent denial attacks. But this needs to be much more subtle than just "reject a block if you know something it doesn't".

Also, you are drastically underestimating the hardware requirements a full network node will require if Bitcoin becomes successful.

Bad idea.  You have no right to have a transaction included.  Let the free market decide.

How much was the fee and how large was the transaction?  I am guessing neglible and not so negligible respectively. 

I guarantee no miner is going to leave any sizable transaction fee behind.  You want express delivery pay express delivery prices.  If miners are dropping high value transactions then work with them.  It is likely a technical oversight. 

I disagree. I think we can safely asume that mining nodes are well connected. Or they shouldn't be mining. I can think of no scenario where a mining node that's slow is useful to the network.


Totaly agree. Miners should be paid. All I'm asking is that they guarantee a reasonable transaction processing time. As I already said, if we need to drop free transactions, I'm all for it, but giving the miners the power to choose if a transaction will make it in a block or not, it's too much IMHO.

well dude! make you own client that invalidates blocks that does not include fee-txs, and see how long it will survive.
what would happen if a tx only was broadcast to half of the network, and have not propagated enough?

half of the network would accept the block, and the other half would reject the block, and thereby causing a spilt! <- BAD THING.

Says who?  You?  The international agency on block protection?

It is a distributed network.  A miner has every right to mine.  Period.   Each individual miner has the right to include transactions they want in blocks.  It is entirely possible in the future there could be private hashing farms to support systems built on top of Bitcoin and they only hash their internal transactions.  

Increase your fee and stop being so cheap.  With Bitcoin you aren't guaranteed any level of service beyond what you pay for.  You want to pay bulk mailing rate and get priority overnight service.  The reality is you get what you pay for and as the network grows and block rewards decline that correlation between price and performance will only strengthen.

Also there are a lot of unintended consequences.  Your system would make transaction withholding more valuable and thus manipulatable. An unscrupulous miner could employe cancer nodes to fragment the network and prevent proper propogation.  Other miners would unknowingly be mining incomplete datasets and see higher rate of invalid blocks.  The beneficary would be the one doing the fragmenting.  You could even see a fragmentation arms race where various mining pools attempt to out do each other by degrading and delaying the network.  Not saying it will happen but you will create an incentive to fragment and destabilize the network.


How much did you pay and for how large of a transaction and how many blocks (not time) passed before your transaction was included?


I can't see a majority of miners working against their own best interest.  If miners excluded your transaction either they saw it as invalid (maybe a technical issue that your solution wouldn't fixed) or it wasn't in their interest (your fee didn't compensate them for the cost).  Make it in their interest and they will include it.  I don't see a problem that requires regulation.

no! its a fucking bad idea. it sucks big time! NO!

I paid the suggested transaction fee. I could not do anything to speed things up after my valid transaction was accepted. I (as a common user) don't know anything about input sizes, I just know that my transaction is not confirmed for more than 24h. Do you see the problem?


Are you suggesting that these attacks can't happen now? That these vectors are introduced by my proposal? Because if so, I believe you are wrong.


I paid suggested transaction fee. Why would a user care about how large it was (I assume you mean the number of inputs) and how many blocks? Why would a user care about blocks at all?


I did exactly what the Bitcoin client suggested I do. The system then ignored my transaction. And you don't see the problem? You will say it's a problem in the client, but I say that allowing the miners to ignore transactions indefinitely is the problem that should be fixed.

What happens now and how does my proposal make it worse?

if you did read the next line, you would find out! STUPID FUCK! it would cause a block chain split. where half of the network would mine block upon the 'invalid' block. while the rest of the network would toss it away.

unless you understand how stuff works, please dont go posting about it.

I hope you agree that it is common sense to offer some guarantee that valid, accepted transactions will make it in a block in some reasonable amount of time. This is all I want to ensure. If zero fees are impossible - cool. If minimum fee needs to be higher - cool again. Minimum fee should be enforced for the users, but in turn the miners should also guarantee reasonable confirmation time.


I am sorry, I would have given a link to the tx, but I realized that I want that transaction kept private. I should have used another example :(. I am sure we can find another one, it's not the first time people complained about confirmation time. I don't know exactly how many blocks, but it was more than 24h, I don't think there were less than 4 blocks/h, so let's say a minimum of 80-90 blocks?


I used the standard client, I don't think that's the case.


As long as the system can guarantee that a transaction will be processed in a maximum amount of blocks, I have absolutely no problem with fees. The problem right now is that miners do have the means of ignoring a transaction if they want to. I simply don't think they should.

I respectfully ask you to control your language. I don't think we grew up together, so a minimum of respect would help.

I understand what a block chain split is. I asked the rhetorical question because you implied that a split is very unlikely in the current state and very much likely in my proposal. I believe you are wrong, and my proposal doesn't make it much more likely to have splits.

i will not control my language, when a STUPID FUCK like you comes and suggest a stupid change, without even learning how the stuff works now.

the blockchain will recover from the splits.
but you proposal would make many permanent splits in the blockchain, that it you not be able to recover from. every time someone does have a transaction that is not included in a block yet, it would reject it, while the rest of the network(that does not know about the tx), would continue to mine on the block that you consider invalid. and you would not be able to confirm more txs. 

a reorg will happen if the client receives a block with more total-work(sum of all the work done in all previous block), then the current one.
also how do you proof that a transaction took place before the block was mined?
the blockchain's function is to proof that at least some time have passed since something was made.

you are a STUPID FUCK, unless you are shutting your mouth now, and go learn how the stuff works now. then you may come back, and maybe i will talk to you in a more polite language.

Thank you for helping me decide whether or not you should make it on my block list. I will also be grateful if you will ignore me too, there's no reason for us to interact anymore.

so you do not like people who told you that you fucked up, and your suggest sucks? Great! it is really productive. STUPID FUCK!

... sorry about that, sometimes i just gets too annoyed on stupid people. especially when they can't see the fatal consequences of a change that their think would make the network more efficient or fair.

WHAT? ME RENTING? ARE YOU TELLING ME TO STFU?

yes you are right, i should leave the topic now.

Thank you. I almost started believing that it's not possible anymore to have a debate and disagree without everybody insulting eachother :).

He shouldn't, and to the extent that he needs to care about it in the current implementation, these are all issues with the usability of the current client.

Mostly likely the transaction wasn't included in the block due to some technical error in the way the miners' client chooses transactions. It's not due to malice on the miners' part.

Miners provide a service for a fee, it doesn't make sense to force them to provide a service to you. Ideally the incentive structure is designed in such a way that the market is efficient and you will find miners offering this service to you at a competitive rate. You're extrapolating from an occasional instance of a technical error, and your proposal does not solve the actual problems (which do exist, and for which a solution will have to be found).

Maybe I didn't express myself correctly. I don't want miners to do something much different than what they do now. Obviously, I don't want a service *for me*. I just want to add a "and don't leave out tx older than x blocks" restriction - which, looking at http://bitcoinstats.org/, is not even a problem, since most transactions are confirmed much faster anyway. For the miners the effect will be minimal. For the end-user though, this restriction brings a much-needed warranty.

You might be right about the technical glitch. I supposed my tx simply tripped some checks in the miners. I assumed it was intentional, but I might be wrong.

I do believe the problem still remains, though - how can we make sure transactions are always confirmed in a predetermined time (max. number of blocks)?

The standard suggestion - "increase fee" - has the following problems:

* it's impossible for a user to determine what the fee should be. Most will use the suggested value, which might or might not be enough. And...

* ...once a transaction is out, there's nothing you can do to speed it up. I suggested that the network should guarantee that it will get into a block, what other solutions can there be?

The current "suggested fee" feature of the client is a temporary hack. In the future it will receive as input the desired wait time (or use a default value) and give a reasonable estimate based on historical data.

The protocol supports multiple versions of a transaction. It should be possible to use this to release a new version of the transaction with a higher fee if the old one isn't accepted, and the client can do this automatically.

Do you know if these features are currently being worked on? The second one is enough for the moment, I think. One can simply start with 0 or a small fee and keep increasing the fee. I'm not much of a developer, but I'll help test.

If these features would be available and work correctly, I think it will fix my problem. It won't protect against miners that simply don't want to include a specific transaction, but let's assume that won't be the case.

I should have mentioned earlier that at this point in time, transaction fees actually have very little to do with the incentive of miners. They exist only to prevent attacking the network by spamming it with useless transactions. So there are hardcoded rules for tx priority which are believed to be enough to prevent spamming. They are tweaked from time to time and there could be glitches which prevent transactions from being included even when they are legitimate.

Features like those I described will only be necessary years from now, though Gavin has implementing an open market for tx fees high on his todo list. I'm not sure what are his plans for this though.

It is high on my list because I think most miners (and pools) would be happy to include many more free transactions than the current rules allow, and if there is another price spike or somebody rich decides it would be fun to make the block chain a couple of gigabytes bigger it is much easier to react if the fees are not hard-coded.

The rough plan is:

 + Give miners more "knobs" to set fee policy-- let them specify (via command-line switch and maybe bitcoind RPC command) how much (if any) space to set aside in blocks for free transactions, how much to charge per-kilobyte and/or per-ECDSA-signature-validation, and what the priority/size/number-of-signatures thresholds are for considering a transaction for inclusion in the free space.

 + As Meni says, teach clients to look at the recent blockchain history and, for a given transaction, estimate how much of a fee will be required to get it into a block reasonably quickly. Maybe a "createtransaction" RPC call that locks coins for a certain amount of time and returns the how-long-to-confirm estimate along with "commit/aborttransaction" calls....

 + Figure out a reasonable UI for fees. Maybe:  calculate the probability sending the transaction with 0 fee will get into the next, oh, 3 blocks, and if it is greater than, oh, 90% then just send it without a fee.  Otherwise, let the user decide between paying a fee that will get it included (with 90% probability) in the next 3 blocks or letting them know how long it might take if they pay no fee.

Lots of details to be worked out...

This sounds to be exactly what is needed to solve my initial problem. If I could make a suggestion here: it's confusing for the user that sometimes a transaction of X BTC costs a certain fee, and sometimes a completely different fee. I understand why, but the regular user won't. I know that it will probably be impossible to keep the fee constant, but it would be nice if the algorithm to calculate it will try to avoid erratic fees.

I can only imagine what a user will think if sending 1 BTC will require 0 fee, 0.05 fee, 0.000002 fee, 0.9 fee and so on :)

+1

The client shouldn't force the user to pay any fee. (The interface to choose a fee and calculate inclusion probability sounds nice)

There are never required fees (other than those designed to prevent transaction spam).  So a user can always send 0.  The client is never going to force the user to pay more.

I doubt it would be confusing if user understood transactions can be free but including a fee increases the likelihood it will be included sooner.

Imagine you click "send" and the client advises you "based on prior 48 hour transaction volume a transaction fee of at least 0.1 BTC has a 90% likelihood of being included in the next 3 blocks.  Do you wish to pay a fee for priority processing?  YES/NO".

Trying to lock things down into fixed fees and fixed guaranteed timelines is simply not possible for Bitcoin.  It isn't a credit card and trying to make it one is like trying to put a square peg in a round hole.

Bitcoin transactions are generally low cost however there is no guarantee of execution.  Higher transaction fees increase the likelihood of priority processing.  Eventually the client can make this more "user friendly" and intuitive however be clear the client will NEVER be able to guaratee a confirmation time of x or that a fee of y will ensure a transaction will be in the next z blocks.  That simply is not possible (nor necessary).

Yes, yes, I meant it like in your example. I think it would be confusing if "90% chance of being included in the next 3 blocks" will vary wildly. But it's too soon to tell. Just something to keep in mind.


Well, I guess this is a point where we will not reach a compromise. I believe there should be a guarantee, and it must be enforced by the protocol/network. On the other hand, I don't really care what the technical solution is, if it works.

How do you *know* that was the reason it took a long time? Post a link to the transaction so we can try to figure out what really happened.

Here's what might have happened. You started Bitcoin, created your transaction, and sent it to the network while being connected to just 1 other node. That node then shut down for whatever reason before relaying it to any other nodes. Such a transaction won't be confirmed for a very long time because it failed to propagate. Tweaking the fee schedule won't fix such a problem.

That wasn't the case. The tx was visible in the network, I checked with at least three sites (blockchain.info, bitcoincharts and another one I don't remember). However, it's possible my original assumption about it being ignored because of the size was wrong, some possible explanations were given by others. The idea that there should be a maximum wait time to confirm for transactions remains.

so your transaction didn't get accepted into the next block. either the transaction wasn't propagated in time for the next block, or the block was already full.

for propagation time, it's purely luck and your solution wouldn't help in that case.

as for the block being full, blocks are (almost) never "full", the fees just get exponentially higher as the block gets "fuller". so suck it up, and pay more fees, or be a bit more patient.

How do you know if you should include more fees? go on http://bitcoincharts.com/bitcoin/ and check how many transactions are pending, and adjust your rate accordingly.

Most (95%+) of my "normal" and "high" priority transactions get included in 1 block, and even "low" priority transactions are relatively fast for me. Something tells me that your problem isn't priority related...

tl;dr: pay more fees, QQ less.

They have no obligation to include a transaction. They are doing the work. You are the one that wants it included, so you are the one that need to convince them.

I would however have liked some feature where all transactions should specify a time frame after which they become invalid. This would solve the problem where a transactions gets stuck in limbo if it doesn't get included in a block soon enough. And it would also possibly allow some neat physical smart card wallets/coins I've been brainstorming about recently.

Miners are being paid 50 BTC + fees to help the network. In Bitcoin context, helping the network means confirming transactions. I will argue that a miner who doesn't include transactions (I know there are still miners who don't include *any* transaction) is not helping the network, but harming it (see Luke-Jr's attack on a alt coin) so they should not be paid at all. Invalidating their blocks seems to me the right thing to do.

Unfortunately, convincing a miner to include a transaction after it's been broadcast is simply impossible, so I was looking for a solution to always guarantee a tx that got transmitted will be included in a block in a timely fashion. Other, better ideas are welcome, of course.

They are still doing the job and you are not, so it should be their choice and not yours.

As to miners who are not including any transactions (against their own best financial interest), they are still providing confirmations for previous blocks. Thus, they are providing owners of bitcoins a service by increasing the proof-of-work needed to double spend the coins. Considering that they are only providing this service to savers, it seems fair that only the savers pay for this through inflation. In short, miners get paid for the service they provide, by the people who recieve said service. Savers are actually getting the short end of the straw here since they are actually subsidizing those who want to get their transaction relayed at the moment.

Your prejudices and misinformation is astounding.

Even a miner which mines empty blocks improves the security of the network.  For an attacker to gain 51% of hashing power they must gain 51% of network hashing power (including miners who mine empty blocks).

Also 1 confirmation is generally not seen as high security.  Miners who mine empty blocks increase the dept of already confirmed transactions and thus improve their security.

Currently there is little economic incentive for miners to process transactions.  Honestly an empty block is worth 50 BTC and on average a block w/ transactions is worth 50.01 BTC.  So the network collectively is saying processing transaction is worth an extra 0.02% revenue. 

Most miners are doing it for economic compensation.  If fees made a larger and more significant portion of revenue it is very unlikely that more than a tiny fraction of the network would exclude them.

The first part is relatively easy.  The second part is much harder.  The third part could turn out to be impossible.

Check out the stratum (https://bitcointalk.org/index.php?topic=55842.0;all) thread.  An overlay network would probably be a better place to put fee information.  I made a post (https://bitcointalk.org/index.php?topic=55842.msg674015#msg674015) that described sorta what a fee description message would look like.  It was overly simple, in that it didn't include portions for the fee policy knobs, but those could be added easily enough.

Prejudices, yes. Misinformation, no. I know how the network works, I just don't agree with this specific part.


Correct. I simply believe that for 50 BTC they should also include old tx. Call me an capitalist pig who's trying to exploit poor miners and ask for a little more than just security :P


No problem here, let's increase fees. I think I said it at least 10 times already. What I'm not comfortable with it the "unlikely" and "tiny fraction" parts. Luke proved that it can happen. How can we make sure it's not possible in the future, to Bitcoin? Everyone disagrees with my idea - which is all right, that's why this is a forum. I don't see a better idea yet, but I'm hoping the great masters of the Bitcoin will find one (most likely annoyed by me, lol).

I won't call you a capitalist.  A capitalist would believe the free market can solve the "issue" (which is really a non-issue as your transaction was likely delay for completely different reasons).



Who is stopping you?  Increase you fees.  That is how a free market works.   If you are concerned about this increase your fees 10 fold.  Get a couple thousand other users to do the same thing.   With higher fees there is more of an incentive to include transactions.



You are aware that Luke is a small fraction of BITCOIN but was >51% of the altchain which was attacked.  It IS impossible in Bitcoin without 51% of hashing power AND ... DRUMROLL .... if someone has 51% of hashing power they can do a lot worse things than not include your transaction.


Why would there need to be a solution to a non-problem?  You haven't even proven a problem exists. Most miners include transactions.  As block reward continue to decline towards zero and fees rise the economic pressure will make the % of network that excludes transactions fall to nothing.  You don't need 100% of the network working on your transaction for the network to work.

How many empty blocks have you seen in last week?  How many of them were empty simply because there were no transaction during that period of time?

It is entirely possible that some % of network isn't even aware of your transaction.   Even if they are there is no mechanism to guarantee the time the transaction was created.  Even if there was different parts of the network would have different timestamps for the same transaction.   That is even before issues like a network split, intentional attack, abuse by pools (pools could cripple smaller pools by paying a fee to have people run clients which only forward transactions to them).

My prediction.  No "solution" for your non-problem will ever be implemented because a) no problem exists and b) the complexity and risk of unintended consequences would far outweigh the benefit.

Ah, there's no problem. Ok then, I'm cool :)

The "problem" seems to have been that the client recommended being a cheapskate instead of recommending paying a substantial fee.

Maybe this can be "solved" during install or something by asking the installer whether they want to install as a cheapskate client that is willing to wait days for their transactions to be processed, a normal home user client that won't be upset if some transactions take a day or few to be processed but would prefer most transactions to most likely be processed within a day or so, or a business client that would like a reasonable number of their transactions to most likely be processed within one business day or less.

Basically just multiply all fee estimates, starting with the current low estimate and multiplying by 2 or 3 for home users, 3 4 or 5 or so for business, Maybe two questions: home user or business user and low, medium or high priority. Then tell them also how to increase it even beyond that estimate/guess for very important transactions, like maybe tell them to double it or triple it for such transactions.

Since the only thing actually seemingly wrong is the client recommending paying too little, maybe forcing the user to make their own choice as to how stingy or generous the client's recommendations should be will "fix" it?

-MarkM-

He never said the problem was all so big right now. But do you suppose we just think about today and don't give a shit about what happens in the future?

ovidiusoft, you have rised some valid points for further contemplation. (And got a bit too much childish critique for it instead of a mature discussion from some members if you ask me.)

First of all, I sort of agree with your assumption that the whole network is going to stay whole for such a long time as 10 or even moreso 100 blocks. There will not be transactions that "will not propagate enough".

Because if there will be, this basically means that the network is split, and some serious shit is going on. In that case we will not have to worry about some transaction not being included, we will have to worry about a 100 block long reorg, which will make the recent mtgox hack seem like a fart in a hurricane.

So while this would be most probably technically possible, it is till too early to force an inclusion of a transaction by the network, it seems like it creates more problems than it solves.

Instead I am more for just increasing the fees with time. And the more I think about it, the more it seems that fees should be percentage-based with some minimum limits to still prevent spamming of small transactions. But even that is a bit too early, because it will only solve a real issue when the block reward will decrease and transactions will have to become bigger in order to pay for all those 5770s and keep the bitcoin's contribution to global worming on the level.

For an actual change right now, what somebody else suggested here about a way for a transaction to expire after a certain block number if it has not been included seems reasonable. That way if your transaction doesn't get included, you could resend it with higher fees and not look like a backstabbing double-spending sonofabitch.