Bitcoin Forum

Something changed recently and I don't know what, but it has me slightly worried. The average confirmation times have shot up quite drastically.

This was brought up a few days ago... but it wasn't clear at that point whether it was a statistical anomaly or something real. Now it seems more likely to be real.

https://blockchain.info/charts/avg-confirmation-time?showDataPoints=false&timespan=&show_header=true&daysAverageString=7&scale=0&address=

Anybody know what's going on?

Difficulty increased and some miners dropped

I doubt that's the reason... the difficulty has been increasing enormously for all of 2013 and the confirmation time was only getting faster (not slower) during that period.

If you check the all-time graph, it used to reach 15-20 minutes.

I wouldn't expect this to be a factor... but what if it was. What if the great wall of china was an influence?

Confirmation time =/= block time.

Average time between blocks is <10 minutes.  I pointed this out in the prior thread but it will likely get ingored again.  Any "answer" which involves miners solving less blocks would mean the time between blocks would be greater than 10 minutes and that is not the case.

Miners are on average producing a block size of ~160KB.
http://blockchain.info/charts/avg-block-size

To clear the backlog and reduce the average wait time to ~1 block would require blocks to be roughly 50% larger.  Miners are chosing not to do that.  The average block has ~300 tx vs the ~2,500 tx limit imposed by the 1MB or ~600 tx it would take to clea the backlog

Simple version: blocks are 90% empty, tx wait longer to be included in a block

Another thing which I noticed today, which may or may not be related, is that while making a few satoshidice bets, one bet would show 22:40 and then my bet would come in and showed 21:41, and another right after it showed 23:42. It was quite odd I thought and also it caused one or 2 of my bets to take forever to be returned. While other peoples bets were being returned after one confirmation, my bet which was an hour or 2 before (only according to the time shown) had to wait for about 6 to 8 confirmations before it paid out. This was sent through my blockchain wallet. I do have the privkeys in bitcoind also but I am tethered to my laptop so it'll take forever to download and slow me down browsing, since I haven't been caught up with the blockchain in quite a while.

Pay 0.0001XBT fee, or 0.04USD, will solve the problem

Well the UXTO currently contains paying tx waiting longer than a block although I have been paying the min fee on every tx (even those which don't require it) for two years now so you are preaching to the choir.

what blocks are 90% empty and there is a back log?

am i missing somthing?

Yes it is up to the miners. Anyone can set bitcoind to only accept transactions with 0.0005 BTC transaction fees if they wanted, and if their hash rate is high enough to solve a block reward than that is what will go. Like the add on which was debated quite a while back, to avoid all satoshidice addresses. It is up to the miners. If you have the hash rate, you have the control of who goes and who waits.

Essentially all.   None of the last 100 blocks were larger than 400KB.  Only 5 were larger than 300KB.   The vast majority are 100KB to 250KB.  More than 20 are <100KB.   The block size limit is 1MB.

@D&T: I read your first post about this. What I don't understand is... what changed? Why is this happening now?

why are miners creating a back log when they have plenty of space.

why not include all the TX's if there is space for them?

Right. This is the question. Also, it's mysterious that this started happening just a few days ago.

Probably because far too many of those transactions are being sent with low (or no) transaction fees, and the miners just don't feel like turning charity into an entitlement.

I'd like to know the stats as far as the trends in fees go. But if that is the case, I can't say I blame the miners for not wanting to encourage the practice.

A large block would increase the odds of being orphaned, so they don't really have incentive to include zero-fee or low-fee transactions.

I'm slightly annoyed cause sometimes it's fast and sometimes slow

That's it, right here.  The block only permits a limited amount of zero fee transactions to be included, and any additional transactions must meet the criteria for the fee schedule to be included into a block.  Basicly, too many people are choosing cost of transaction over time to confirmation.  This is to be expected as the number of transactions increase, as this sets up a market for the fees (higher fees are more likely to be favored into the next block) while still providing for a free (or out of band transaction fee) methodology.

Anyone knows if Eligius is still doing zero-fee transactions?

Guys... everything that everyone has said in this thread so far has ALWAYS BEEN TRUE (not paying transaction fees leads to long confirmation times, sure). None of that explains why the confirmation times are spiking NOW.

That is not correct.   A miner could make every block 1MB ~2,400 tx only include the free ones and exclude every paying tx if they wanted.  Not saying they would (it would be stupid) or they should but they could.  Saying the "block" prevents them is a cop out.

Miners decide what tx to include in a block.  The only thing which can't be included in a block is an invalid tx, or a sum of tx greater than 1MB in size.

It is very simple.

Tx volume is higher than average block size.  It is THAT SIMPLE.

If users are creating an average of 400 tx every 600 seconds and miners are making blocks with an average of 300 tx every 600 seconds what is going to happen to the tx backlog?

If users are creating an average of 400 tx every 600 seconds and miners are making blocks with an average of 400 tx every 600 seconds what is going to happen to the tx backlog?

If users are creating an average of 400 tx every 600 seconds and miners are making blocks with an average of 600 tx every 600 seconds what is going to happen to the tx backlog?


If it is still too complicated lets try a non bitcoin metaphor.

If you have a pool (mem pool) and you are adding 2 gallons a minute to the pool (unconfirmed txs) and at the same time have the drain open and water is draining out of the pool at 1 gallon per minute (tx being placed into blocks) will the water in the pool rise or lower?  What is required to prevent the water level in the pool from rising?

I feel like we are talking past each other...

Are you saying that you think the tx volume has suddenly crossed a threshold ~4 days ago? I am not asking about why confirmation times go up and down in general, but why they have gone up a lot very recently.

(I noticed for example, that Elgius is de-prioritizing transactions that reuse addresses, this is new, and perhaps that is a cause for this)

Well threshold would imply some sort of cental block planning agency.   Various miners have various different block parameters.  Tx volume is higher than the block size being created by the various parameters currently used by various miners.   Some miners target larger blocks, some target smaller ones, a couple seem to include nothing but the coinbase tx.   Collectively all miners have a certain tx throughput which is less than the tx volume and the limit imposed by the 1MB cap.

Miners can either expand the # of tx they include in blocks
OR
Users can collectively reduce tx volume
OR
The backlog will grow

For the time being a way to put yourself at the front of the line is to pay a tx fee but that won't reduce/eliminate the backlog it will just ensure your tx has a higher chance of going first.

This will cause the transactions that reuse addresses to be delayed.  Is there a way to get a statistic on how many transactions fall into this category (reused therefore being delayed by some of the pools)?

I don't understand why our communication is breaking down so much. I can tell that we're both trying to tell each other something, but it isn't working.

I completely understand the general mechanics of transaction backlogs and I very much appreciate your explaining them in detail. The point of my starting this thread was to wonder what event/dynamic/new-mining-pool/whatever occurred on Nov. 3, 2013 such that, since that date, the confirmation times have gone up significantly.

My apologies in advance if I am somehow misunderstanding you.

That would be great to find out! Apparently BTC Guild is thinking about doing this too.

Could it be as simple as the total number of transactions per day crossing a threshold where the backlog starts to happen, given a flat number of transactions per block?

Notice that about that time the "Number of transactions excluding popular addresses" jumped up and crossed over the 50,000 transactions per day line.

6 * 24 * 350 is about 50K transactions per day.

Yeah, that's a fine hypothesis, but why would popular/non-popular matter? If you look at the total number of transactions over the past year, it has been pretty constant (well, constant is relative, but the important point is that it has been higher in the past).

https://blockchain.info/charts/n-transactions?timespan=1year&showDataPoints=false&daysAverageString=7&show_header=true&scale=0&address=

I believe in that graph you are looking at transactions processed per day, which, of course, will show a constant number given the saturation condition we are talking about.

Assume the number of "transactions to the popular addresses" is constant then a jump in transactions to the other "not popular" addresses shows that now we have more transactions being requested than are being processed by the system.

This will cause a backlog - at about the time the confirmation time increased.

By default the client reserves 27kb for free high priority transactions.


Also it will raise the minFee once blocksize becomes larger than 250kb.

Of course miners can change this.

I was generallizing.  I know that miners can choose their own rules, but the default set of rules say that there is a limit to the free transactions.  Changing those rules requires programming skill and the desire to do so.

As far as I know, they have always required a fee of 4096 Satohies (http://eligius.st/~gateway/faq-page/faq-5).

I also have noticed a problem when I was using the bitcoin atm machine i went there and my confirmation came instantly but the next day it took my 10 minutes to get my 1 confirmation from block chain

Block chain usually takes 10 minutes at most to confirm a good transaction.

You were just lucky on the first day (making your transaction just before a block) and unlucky on the second (making your transaction just AFTER a block)

^^ This. Miners will not mine large blocks unless you pay more than double the normal transaction fee. Much more than double, if you want very large blocks. People have often asked, "will paying a larger transaction fee really lead to faster confirmation times?" and the answer is now yes. Will not paying a fee (even where free transactions are "allowed") lead to longer confirmation times? Also yes.

People, pay the fucking transaction fees. If your transaction is taking a long time to confirm and you paid no fee, or only the minimum required fee, it's your own damn fault.

Maybe I should change the title of this thread... you are addressing something that I feel is quite off-topic. I just wanted to have a discussion over what dynamics changed after Nov. 3, 2013 -- not a general discussion about transaction fees and confirmation times.

So far the hypotheses are that Elgius changed how they prioritize transactions based on address reuse and that we hit recently hit a threshold of transactions broadcast per day (due to the growing adoption of Bitcoin).

This is what I'm talking about. I didn't bother mentioning it because it's already been explained several times in this thread. I was merely elaborating on why, with a sufficiently large number of transactions, many of those transactions will not get into a block, even though the block size is nowhere near the limit and the transactions pay the minimum required fee. The minimum fee is not sufficient if a block is more than a quarter full. If nobody ever pays more than the minimum fee, blocks will never be more than a quarter full, leading people to ask "why is there a huge backlog of fee-paying transactions when the blocks are nowhere near full?" And now you know. And knowing is half the battle.

That is a flawed understanding of the fee rules.   High priority txs have no required fee, none.  As in 0 satsohis.  This applies regardless if a block is 1MB in size.  

The min mandatory fees FOR LOW PRIORITY TXS are designed as a denial of service prevention mechanism.  They are only enforced at the client level.   Even if miners follow those they have absolutely no effect on high priority txs.  

Lastly the rules aren't enforced at the protocol level.  Miners an build any valid block they want by using custom bitcoind.  This is hardly beyond the capabilities of a major pool (all are using custom nodes already).  It is possible to have a block contain NOTHING but no fee txs (as in ~2,400 of them) and the block is still valid.

The way I see it their are three possible sources for the recent change.

1)  One or more of the large mining pools has changed how it mines recently
2)  The transactions mix has radically, changed such as a very high volume of 0 fee transactions being attempted recently
3)  The network communications have been disrupted in some way such that transactions are taking longer to reach miners

My naive guess is that the Chinese have 'discovered' that the transaction fee is optional and one of their trading sites/forums etc has caused a massive increase in 0 fee transactions being attempted.  If this is the case then they will probably only take a few more days of slowness for them to realize that the fee isn't so optional and they will cut it out.  If it's something else then it may be the new normal.

+3 :)

Well, in any case, now that people are starting to feel the pain of slower transaction confirmations it will be interesting to see how both miners and users respond (who will give first... the miners by creating larger blocks, as D&T pointed out, or the users [who care about tx confirmation times in the first place] by paying higher fees).

I don't think this is correct. Under the Satoshi client's fee rules (which as I understand are enforced by most, but certainly not all, miners), high priority transactions with no fees will only be included in the first 27 kB of a block (this limit can be changed with the blockprioritysize option). Once the block hits this limit, only fee-paying transactions will be included, regardless of priority.

True, but hardly relevant unless a significant fraction of miners are using non-standard fee rules. A few do (which is the only reason transactions paying insufficient fees are ever confirmed at all), but I'm pretty sure most do not.

this - as can be seen in the code above (v0.8.5)

Blocks are not getting full, average block size is 100 - 150kb at the moment from what I see in the last couple of blocks. This means there probably are more zero fee TXs than what can fit into the 27kb area of blocks.

edit: it's the weekend, so maybe blocks really are becoming large. Average block size comes somewhat close to 250k:
http://blockchain.info/de/charts/avg-block-size?timespan=60days&showDataPoints=false&daysAverageString=1&show_header=true&scale=0&address=

FWIW all my transactions are high priority ones and I've observed a steep increase in confirmation times, going up to nine hours in some cases, when their priority reaches several thousand million.

Not sure if this sheds any light on the block size effects of various pools, but here is
a recent snapshot of blocks. What I do not understand is how the Eligius miner created such a small block when there were over 1200 unconfirmed transactions in the mem pool. Direct link: https://i.imgur.com/S29IMDg.png

https://i.imgur.com/S29IMDg.png

Interesting!

I really hope this will never happen , but let's say that in case number of transactions grows and it starts putting pressure on the limit of the block
IF  , the miners don't agree to raising the limits of the block but to raise the fee in order to discourage micropayments  .....wouldn't we end up with something like...  banks?

That's exactly one of the reasons why we need competition in crypto currencies and relying just on bitcoin is a single point of failure. Monopolizing the network together with greed can easily lead to abuse. Human nature is the same at all times, no matter what noble cause there was in the beginning, people more often than not tend to lean to their own selfish ways, and check points and counter-balances need to be in place.

As long as transaction with reasonable fee are processed in a timely manner all is good.

Miners are the heart of Bitcoin. Higher mining profits directly improve the infrastructure. A couple of cents in fees are hardly much, considering that transaction will be stored on thousands of computers forever.

What we really need is a method to add fees to a transaction, ideally through an extra transaction, so that merchants (or anyone else) could hurry a transaction.

Why aren't the transactions just sorted and filled?  The miners could take the most fees they could fit in the block and then fill the rest with free ones up to the limit ordered by priority.

Then hardly anyone would pay fees, which can't be in the miner's interest.

cant this be caused by mastercoin stuffing everyblock with useless information with their crap-r-col

Possible that they are reacting to some sort of transaction DOS "attack". Someone could just be creating a massive amount of no-fee transactions. While there is not much to be done against that, accepting only transactions with fees allows "legitimate" transactions to be processed, while it keeps the others in check.

Some people will prefer banking institutions anyway, althouth  they wouldn't function quite the same.  But no, we don't need to go there.  Another solution is overlay networks, online wallet sites that agree to delayed settlement, mass transactions, blockchain contracts, etc.  There are a lot of ways to move the burden of transactions off of the main network (that the protocol supports) than the obvious general rule today that all transactions are recorded on the blockchain individually and in a near term.  Many to many transactions is one method of reducing the absolute number of transactions, for example.

Are Miners rejecting Mastercoin tx  now ? Since They are insisting on larger fees instead of zero fees . I dont understand MSC , just guess . Check the tx from the 1Exodus address

Could it be that solo-miners are perhaps thinking mining empty blocks gives a higher odds % to solve blocks?

hei... i am no friend of fees ... today i had a talk with someone working for a company making statistics for payments. he told me this. paying with ec in europe we have two options . pin or signature. signature is very insecure, pin costs one euro each transaction. yes. one euro. the companys job is to make complicated statistics when its better to accept pin or sign .... in short. even with thirty minutes confir!ation time and fees btc is far more effectivw than banking.

this will shed some light on eligius

https://bitcointalk.org/index.php?topic=334316.0

and this is my explaination to the delay its causes

https://bitcointalk.org/index.php?topic=334316.msg3605784#msg3605784

It looks like it was #1 (Miners are mining differently), frankly I think it's a stupid idea too, it looks like people are raising a stink about it and it will be reverted shortly.

I've not noticed it, I've just noticed that confirmations have been more bunched (like 3 in 3 minutes and then 17 minute wait). I'm guessing this is because of the huge differences in hashpower between pools.

Or not.  The backlog was prior to Eligus making the change and Eligus is only ~15% of the network while confirmation times have risen 80%.   Tx volume is higher, blocks aren't, tx have to wait longer.

I think I might...

http://arxiv.org/pdf/1311.0243v2.pdf

http://threatpost.com/researchers-debate-value-of-new-bitcoin-attack

Looks like a mining pool has attempted this 'selfish mining' theory, to gain a profit advantage.

Selfish mining wouldn't change the number of tx confirmed.
One could engage in selfish mining actually make larger blocks and thus have less of a backlog.

The vast majority of significantly delayed tx are free.  The default block rules cap free tx at 27KB or ~60 transactions per block.  That is a mere 0.1 tps.  If people are creating more than 1 free tx every 10 seconds unless a miner decides to miner MORE than the default free tx which increases their orphan rate for exactly 0 BTC more revenue then there is going to be a backlog.  If the rate of free transaction creation rises then the backlog will get longer and longer and longer.

This is actually quite surprising! Could there be a flaw?

https://bitcointalk.org/index.php?action=profile;u=151190;sa=showPosts
What is with the dozens and dozens of one line posts in the past two days?  
Trying to boost up your activity score to appear older and pull off a scam?

I agree, but I think that it could still push up the average time to confirm.  Transactions that were not yet available when the selfish pool found a block solution cannot be added to that block, but would likely be found in the public block.  Then if the selfish pool releases their hidden block, any transactions that made it into the public block but not the selfish block would have to re-enter the transaction queue.  But then there would be a delay, as most miners that accepted the public block would have already removed those transactions from their queues, and can't replace them until after that particular mining node had already discarded the public block as an orphan. 

What we are seeing here could be a side effect of selfish mining put into practice.

Gotcha.  That is an interesting angle.  I don't think that is the case but it might be a good way to design a "selfish miner" warning system.


The average doesn't tell the whole story but a distribution would.
If the move from ~10 minutes to ~18 minutes is due to selfish miner what we should see is a reduction in the 0-10 minute times and an increase in the longer but relatively short times say 11-30 minutes.

My guess (and just a guess at this point) is that isn't what the distribution would look like.  What I think is we have a minority of very very old tx which bring up the average so it would look something like 90% of tx have a confirmation time in the 0-30 minute range and then you have and increased amount of tx in the long tail with confirmation times ranging from 31 to 1000 minutes.

This was one of my original thoughts when starting this thread, especially given that it started shortly after the selfish mining authors seemed keen on proving that they were right.

Pools have an incentive to keep their blocks small so that they propagate fast, reducing their chance of being orphaned.  There is certainly no incentive to include free transactions, and low-fee transaction are probably also a negative prospect.  eg. adding an extra 500KB to a block for 0.1BTC might not be worth it : it adds 0.4% more returns to the 25BTC base reward, but if that means 3 seconds extra propogation time then it adds a 0.5% extra chance of being orphaned, so it would not be worth doing.

I've just made these numbers up based on my own assumptions of transaction size and propogation times, but hopefully you see the point.  Maybe it's the case that some pools have recently done the math and tweaked their thresholds to optimise their returns.

Well if a pool was looking to maximize the short term revenue, block with the highest possible net revenue is one with no transactions (other than the coinbase).  A miner would simply mine only empty blocks and ignore all transactions even paying ones.  The best estimate (with current protocol) is that the orphan cost is about 3.3 mBTC per kB.  Paying tx are ~0.1 mBTC so the inclusion of any tx is a net loss of revenue.  That being said hopefully miners and pool ops realize that having a lot of coins doesn't do you much if you cripple the growth of Bitcoin.  2% less coins which are each worth 10x as much because you helped grow the adoption of Bitcoins is a pretty good deal.

One thing Gavin pointed out, is that if all miners have higher orphan rates they don't lose any revenue.  What matters is RELATIVE orphan rates.  If you have an orphan rate of 2% and the network on average has an orphan rate of 1% then you are losing 1% revenue.   However if you have an orphan rate of 5% and the network on average also has an orphan rate of 5% you aren't losing anything. 5% of your blocks are orphaned but so are everyone elses and as a result the difficulty is 5% lower.  The big miners should sit down and discuss mutually raising block sizes.   Even if you can't get everyone onboard the orphan costs can be reduced if miners agree to all raise block sizes.   Getting 50%, 60%, 70% of hashpower in agreement should seem possible as honestly that is just what getting the top 6 or 7 pools and major solo miners to find common ground?

Longer term a more efficient method of block propagation is possible which should reduce that orphan cost.  Today when a block is broadcast it all the tx inside the block are also broadcast as part of the block message.  Most nodes already have these txs so it is just wasted bandwidth.  One could instead include just the tx hash in the block message and that would cut the size of a block message by up to 80%.  For larger savings a reduced length hash could be used instead.  Collisions here are not a security risk and would still be incredibly rare and that would allow reducing the block message size (and thus propagation delay) by 95% or more. 

Still don't want to be all doom & gloom, even if nothing is done over time Moore's law will mean higher bandwidth at lower costs which will reduce the propagation delay.  Also the block subsidy being cut in half again in ~3 years will reduce the "distortion" that the large subsidy has no fee pricing.

I'm with DeathAndTaxes on this issue. There is a REAL problem here. The problem is NOT only 0-fee tx that are being delayed by days in some cases, that is actually okay - free transactions have been charity which is finally over.

The problem is that I've been seeing transactions with the current standard fee of 0.0001 per kb delayed for multiple blocks, all the time now. That is the real issue. To really get priority now, you have to pay something like 0.001 per kb. That seems quite silly to me when pools are deliberately keeping the blocks small.

I don't know the correct solution to this issue, but it would be in the best interest of Bitcoin for large pool operators to come together and simply decide they will all start producing larger blocks in a more relaxed way. They don't need to start adding 0-fee tx but maybe decide that they will add standard fee tx all the way to 1MB? How is that for a quick solution?

If enough of the large pools agree to do that, the relative orphan rate will be similar across the board. Bitcoin wins, everyone wins.

Perhaps a bunch of solo miners also have enacted parameters because they were sick of certain "large users of the blockchain" not paying any fare at all even though they were making fantastic profits.   Free Riding is great until the guy that pays for the gas gets tired of it.....

There is already a financial incentive to do so.  Any pool that chooses not to include transactions that have paid fees while there is still room in the block is intentionally choosing not to get paid the maximum amount that they could.  If even 1 pool starts accepting these smaller fee transactions, they will be able to reward their members with higher payments than the other pools.  This will attract additional miners which will lead to even higher profits.

Well not exactly Danny.  The larger the block the larger the chance of orphans.  So as a hypthetical if you collect 2% more gross revenue but increase your chance of orphans 3% you actually (after accounting for losse due to orphan blocks) make less revenue.   The large block subsidy kinda distorts the economics of mining.

So after a few days where I thought times are getting back to normal it seems they are going back up.
For now there are only a few minutes , but we're talking about 50k transactions/day if those are the cause. Make them 10x and somebody will have to make those block changes.


Also , funny:
Height   Age   Transactions   Total Sent   Relayed By   Size (kB)
270900   2 minutes   382   5,327.14 BTC   GHash.IO   151.13
270900   2 minutes   369   5,037.41 BTC   BTC Guild   145.65
270899   5 minutes   313   7,642.39 BTC   Deepbit   138.48

In which case, it's a self-correcting issue.  So long as that's what's going on.

I suppose, but seriously if miners want to squeeze every last drop of blood from the profitability stone, they should seriously look into implementing "child pays for parent".  I've had multiple transactions I've received where I would have been happy to pay a large enough fee on a child transaction to make sure that both my transaction and the parent transaction get included in the next block.

If they can't include a 1 kb transaction with a 0.0001 because of the cost of orphan risk, then they should absolutely be willing to include in their next block both the 1 kB transaction with zero fees that I received as well as the 1 kB child transaction with 0.002 BTC fees that I sent to increase their profits.

Posts on the Bitcoin Foundation forum by Gavin, Mike Hearn and Luke seem to imply that mining pool operators are just not paying much attention to this one way or another. They are using some default settings for accepting transactions and just focusing on making sure they find blocks... I have no idea how true this is.

What's with your bend over fest with Bitcoin? Nobody gives a fuck that 90% empty blocks are still being made every 10min. What people care about is that transactions are much slower than usual,  why must you take so much bitcoin penis?  Bitcoin is getting messed up by higher activity which it cant cope with.

What we can gather from this is Bitcoin sucks, and there will never be mass adaption because it's broken. Gavin and the foundation suck as they have known about this for years and never bothered to fix or address it. Bitcoin its on its final bubble stage before it spazes and implodes upon itsself.

The final stage can of course take it to thousands of dollars and still last a few years, but we are now in its final leg instead of the beginnings.

I thought of a way to reduce orphans: more full nodes on the network. Unfortunately, non-mining full nodes are not rewarded directly by the protocol. Also, most "miners" are not full nodes: they simply rely on pools.

More nodes reduce orphans by speeding up block propagation:

• By having more aggregate CPU power verifying transactions. Slow CPUs probably don't help too much for preventing orphans.
• By having more aggregate bandwidth for relaying transactions and blocks.
• More nodes make any successful DDOS more expensive to run.

My personal Bitcoin node has been in the planning stages for months. When up, it will have an average bandwidth of ~463kbps (burstable to 5Mbps). Calculation: (300GB/month(cap))/(30days/month)/(24hours/day)/(3600seconds/hour)*(8bits/byte)/2(symmetric bandwidth usage)

If I get a second (or better paying) job, I will consider renting a dedicated server with ~7.7Mbps (burstable to 100Mbps) (5TB cap)

Edit2: with 5Mbps burstable bandwidth, I may want to limit the block-size to 256kB so it has a chance of being transmitted in less than 1 second. With 8 connections, it would take 3.3 seconds to broadcast to all 8.
Edit: read page 4: sending only block headers sounds like it just might work (assuming the node has seen all the transactions)

Isn't every Wallet a node, providing that they are using bitcoin-QT?

Yes, Bitcoin-QT is a full node.

Are your numbers from Gavin's calculation? https://gist.github.com/gavinandresen/5044482  Back-of-the-envelope calculations for marginal cost of transactions  (almost the same result, 3.2mBTC/kB)

I guess we will see much higher TX fees then...

If that is what it takes to get miners to pay attention to transactions, that's what it takes.

A free market is emerging in transaction fees!  Imagine that.  Clearly miners have, to some extent collectively, decided to delay including "marginal" zero-fee txns in blocks.  I think by 24-48 hrs in some cases.  They still get in eventually.

This is good and healthy.  All the whiners are just upset because the freebie has been reduced; whining is to be expected.

What I'm wondering is where is MisterBigg?  He was the dude saying if we removed the blocksize limit then miners would be stuffing everything into one block, even things paying 1 satoshi.  I think that idea has been thoroughly debunked, particularly recently - his school of thought cannot explain the current phenomenon, and we're nowhere near the 1MB limit.  More evidence we should just get rid of it.

Indeed it seems that the issue of orphan blocks alone will stop miners from creating blocks that are too large in size. The fee market will be more based on that than the enforced limit.

Agreed.  It is something I hadn't considered but the "orphan cost" does constrain block size.  I do think retaining a hard cap for the interim future (possibly raised to 5 or 10 MB) is a good thing.  A malicious actor (and thus not interested in orphan cost) could do some damage to adoption by bloating the blockchain with very large blocks.

Just as a point of reference:

I broke open one of my savings wallets last night for the first time in a year it being a monster hassle to retrieve them.

I'm working on a source built bitcoind from around April 2013 codebase, and my client is only caught up to block 25k-ish.  I'm on a satellite connection with limited monthly quota, so I only run the client when I need to.

I transferred all the coins out of the wallet (100BTC) directly to my Coinbase account and two blockchain.info accounts.  I have bitcoind set to pay .001 transaction fee.

 - The first transaction (to Coinbase) was visible within a minute on blockchain.info.

 - The second transaction (to blockchain.info) took many minutes to show up.  It seemed to show right after a block was mined.

 - The third transaction (also to blockchain.info) happened fairly quickly as I recall.  A few minutes perhaps.

---

I've never believed that Bitcoin is a appropriate tool for dealing with small purchases for coffee or whatever.  I want something reliable and independent (from corporate and government interference) for large value transactions.  .001 BTC on a 50BTC transaction is a price I'm perfectly happy to pay for good service.  It is

 - vastly less than I pay for international wires,
 - and is much faster,
 - more hassle free,
 - more private,
 - and more reliable (e.g., Mt. Gox cannot get my $5k international wire to me after more than a quarter.)

So far Bitcoin is has served me well for my needs here.

It isn't mentioned much, but my understanding is that it's a fairly useless full node if one isn't running UPnP or has set up a port forward.  Just sucks out without giving back.  For better or worse, many people probably don't even know what this means, or even that they are doing UPnP since lots of routers have it on as a default setting.  I don't trust UPnP since I didn't compile it (or more accurately, I didn't compile the OS on my phone, printer, etc), and I turn it off.

I was under the impression that UPnP was just one of several discovery methods in the code.  With your UPnP turned off is your full node client still able to discover and connect to other nodes?  If so then it is not necessary for node discovery.

One's own process (bitcoin[-qt|d]) discovering and connecting to services outside one's internal network is not a problem.  Firewalls typically allow this without restriction.  The problem is if someone discovers your, can they connect to you?  The only thing they will know is the IP or your router (port 8333 can be inferred.)  When they hit that, the router need to forward the message to the appropriate device within one's internal network.

Discovery and connection are two different problems.  Bitcoin used to use IRC for discovery back when I started, but I think that is fully deprecated.  The current bitcoin.it wiki page on running Bitcoin doesn't say much about UPnP except that there is a flag.

There are some conversation about this here and there, but not as many as I would have expected.  I kind of feel that perhaps a development decision was made that enough routers have UPnP on be default that many people will be running the code in 'good citizen' mode whether they know it or not, and talking about it will mostly just scare people.

No, no.  Runing a full node that is 'quiet' is not "taking without giving back".  Dark nodes are a certainty that not all nodes are discoverable.  This is a feature, not a bug.  There are attack vectors, against the p2p network model itself, that are undermined by the very concept that dark nodes can (and do) exist.  They become a repository of the true blockchain, should (as an example) a viral vector is discovered within the network itself, permitting a malicious player to exploit all visable nodes at network speeds.  Dark nodes (probably) promise some degree of insulation from such an exploit, as well as the certainty that the attacker cannot ever be certain of success of such an attempt, since it's not verifiable.  The fact that some nodes are discoverable and others are not isn't a problem.

AFAIK I'm the one who pointed out that incentive first; MisterBigg either independently realized the same thing or was repeating me.

Either way, I did my analysis more carefully here (http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg03200.html) and found that the incentive to stuff blocks full of garbage was even stronger than I originally thought. However that incentive only is true under specific circumstances that are not present in the current Bitcoin system, but will be in the future. Specifically doing that decreases the number of blocks you find per unit hashing power, but it decreases that number even more for your competitors - right now blocks are themselves worth a lot of money so there's no reason to use that strategy, but in the future when mining income is mainly from transaction fees the strategy appears to make sense.

tl;dr: the idea is anything but debunked, it's just not yet relevant.

IIRC he gave completely different reasons that were proven wrong by Gavin's paper showing that including TXs is quite expensive due to orphan costs.

The orphan cost of including a TX for a standard 500byte TX is now a whopping $1,30

Is anything being done about this?

Since transaction fees are proportional to the BTC price, this amounts to a cap on bitcoin adoption; less people will use bitcoin as the tx fee becomes uncompetitive. We are already PAST this point, imo. I feel that there is some urgency to fix this problem now.

The obvious solution is to reduce the orphan cost. To do this solved-block propogation time needs to be lowered. I don't really understand it, but could CoinWitness be the key solution? https://bitcointalk.org/index.php?topic=277389.0;all

I don't know of anything in the development pipeline that addresses this issue.

Yes.


This is being worked on by migrating from a complete block to a skeleton block, with only the headers and merkle tree included. This alone would reduce the orphan risk cost by about a factor of 10.


Not for Bitcoin.

Thanks for the reply.

According to the above analysis, the fees are already about 1/30th of what they should be for miners to break even. The skeleton block solution gets us to 1/3. That's just at the current price.

Don't you think this is a big problem? I assume that, at some point, economic reality will kick in and miners will start to demand much higher fees. Suddenly bitcoin isn't so great compared to alternatives.

Not really, I recommend reading the entire thread.  Switching to hashes only is more like a 20x reduction in size.   Also the fees being 1/30th of "orphan cost" is just one estimate.  Other estimates put it closer to 1/10th.  Miners also receive a very large subsidy and they have an indirect incentive to keep the network going (a few less Bitcoins which are worth something is more than a few more worthless Bitcoins).  Also Moore's law is alive and well and it applies to bandwidth as well.

Still even if fees rose 300% (which I strongly doubt) that would make the cost of a Bitcoin tx something on the order of $0.20 which compares very well to other payment methods.   What I think is more likely is miners dropping or reducing the number of free transactions they support and maybe the min fee doesn't go down.   That combined with the subsidy, improvement to block propogation, and the power of Moore's law will likely keep tx fee costs below $0.10 or so.

...and then the block reward halved...

this had a myriad of competing interests

[1] at its core each miner will act to be most profitable, which mean they will would rather get the block out there faster to avoid orphans

[2] If all miners do this, then they coins they hold and the network becomes less workable and so they risk thier business model.

[3] The block reward will half making fees look more attractive over time.


these issues may be why POS offers things over POW.  POS may be able enforce sytems of mining more uniformly and easily that POW, also you can long term plan eg with Peercoin, the massive evaluation of NXT and then emunie


With PeerCoin You know fees will be 1% and that you can mint 1% back per annum which lets you make long term store of value decisions

BTC seems to be falling short of a long term store of value and every day transactions, it's sort of in the middle, good for both.

Interesting times.

Maybe, but I've yet to see a working POS system that wasn't either too complex to prevent a trusted-peer attack vector of some form, break the anonimity factor of users, or both.

... and then the price of bitcoin went up x10.

Rises in the value of bitcoin are going to outstrip any of the measures mentioned to reduce the cost of block propagation. OR the price of bitcoin simply wont rise because fees are too high.


Tragedy of the commons.

The block reward is WAY too high and the reduction rate is way to slow. I still think this is a serious problem.

Gavin's Cost is independent of the trade value of bitcoins, so that wouldn't prevent a halving from improving the attractiveness of fee paying transactions.  Regardless, there is much that can yet be done about Gavin's Cost before the next halving anyway, so I don't think that Gavin's Cost is really a problem so much as it is (and will continue to be) the real blocksize limiting factor whether or not we continue to hold a hard limit on blocksizes.  With 'thin' blocks (block headers & merkle tree only) the one meg hard limit would easily permit an average transaction rate of 30 - 40 transactions per second right now, doing nothing further.  A blocksize limit of 10 megs would get us into the 300+ per second range, which is a workable rate for a major system; and higher rates can be handled better by overlay networks anyway.

Ummm... yeah, I just realised that this

1 - e^(-(1/600) * X)

is not a linear function. This was the source of my misplaced concern. If we reduce a transaction down to the size of it's hash, this cost becomes extremely small.

Thanks for the feedback.