Block Size soft-limit maxing out this AM 6/3/13

[solex]

Recent blocks

Height   Size (kB)
224470   243.12
224469   239.69
224468   243.31
224467   242.93
224466   119.81
224465   243.26
224464   242.69

1800+ transactions pending (equivalent to 3 blocks or 30 mins).

IMHO. Time to stop gathering metrics at the 250Kb soft limit and allow a larger size, perhaps 500Kb.. Thoughts?

[1714847124]

1714847124

[1714847124]

1714847124

[1714847124]

1714847124

[AbsoluteZero]

1800+ transactions pending (equivalent to 3 blocks or 30 mins).

Where do you find this info on pending transactions?

Is this the first time this "maxing out" happens?

[solex]

1800+ transactions pending (equivalent to 3 blocks or 30 mins).

Where do you find this info on pending transactions?

http://blockchain.info/unconfirmed-transactions

Is this the first time this "maxing out" happens?

This is an unusually high number. (Call me whatever, but I keep an eye on this as it is the beating heart of Bitcoin).
In conjunction with saturated blocks - it is unique and not good...

[Syke]

This is an unusually high number. (Call me whatever, but I keep an eye on this as it is the beating heart of Bitcoin).
In conjunction with saturated blocks - it is unique and not good...

It is very good. Now people will start including larger fees with their transactions.

[solex]

This is an unusually high number. (Call me whatever, but I keep an eye on this as it is the beating heart of Bitcoin).
In conjunction with saturated blocks - it is unique and not good...

It is very good. Now people will start including larger fees with their transactions.

http://blockchain.info/charts/transaction-fees-usd?showDataPoints=false&timespan=&show_header=true&daysAverageString=7&scale=0&address=

Yes, so obvious that higher fees haven't started happening already...

[AbsoluteZero]

It is very good. Now people will start including larger fees with their transactions.

Even with higher fees, if blocks are full then transactions will be left behind.


How does the 250 limit go away? I thought is up to each miner.

If a miner fills a block higher than 250Kb and less than 1Mg will it be rejected by the other miners?
EDIT - Ok just saw an 487Kb transaction

[jgarzik]

The "soft" limit is set by each miner.

When I mined using vanilla, unmodified bitcoind + p2pool, it was a simple configuration setting to change the limit to 900k.

My first block was over 400k.

Soft limit "maxing out" is a non-event.

[Luke-Jr]

Bet at least most of those blocks were by pools neglecting their duty to filter out flooding attacks.

[solex]

The "soft" limit is set by each miner.

When I mined using vanilla, unmodified bitcoind + p2pool, it was a simple configuration setting to change the limit to 900k.

My first block was over 400k.

Soft limit "maxing out" is a non-event.

ok, fine. As long as miners are aware and can increase easily when it's considered necessary...

[behindtext]

Bet at least most of those blocks were by pools neglecting their duty to filter out flooding attacks.

i have read some of the threads about SD generating tons of 'spam' tx and it seems clear that it is difficult to prevent this sort of flooding.

are there any countermeasures planned for such flooding attacks?

[misterbigg]

224466   119.81

LOL who is the midget?

[Luke-Jr]

Bet at least most of those blocks were by pools neglecting their duty to filter out flooding attacks.

i have read some of the threads about SD generating tons of 'spam' tx and it seems clear that it is difficult to prevent this sort of flooding.

are there any countermeasures planned for such flooding attacks?

Bitcoin's design has miners serving the role of filters. For the majority of flooding attacks, the planned deterrant of transaction fees work pretty well: an attacker would need to waste a bit of their own money to successfully flood. Unfortunately, SD is the exception that has managed to make use of social engineering to get someone else to cover the expense of getting around the fees. Not only that, but because SD's enablers are gamblers, they are throwing their money away anyway so are willing to ignore fees higher than rational users of Bitcoin who just want to send money. Thankfully, detecting and blocking SD specifically is also fairly easy.

[Syke]

It is very good. Now people will start including larger fees with their transactions.

Even with higher fees, if blocks are full then transactions will be left behind.

Exactly. If you want your transactions processed quickly, attach a nice healthy fee with them. If you can't be bothered to include a decent fee then your transaction deserves to be left behind.

[nikkisnowe]

It is very good. Now people will start including larger fees with their transactions.

Even with higher fees, if blocks are full then transactions will be left behind.

Exactly. If you want your transactions processed quickly, attach a nice healthy fee with them. If you can't be bothered to include a decent fee then your transaction deserves to be left behind.

Who let the shill for Paypal/Visa into the Bitcoin forum?

[dree12]

It is very good. Now people will start including larger fees with their transactions.

Even with higher fees, if blocks are full then transactions will be left behind.

Exactly. If you want your transactions processed quickly, attach a nice healthy fee with them. If you can't be bothered to include a decent fee then your transaction deserves to be left behind.

This is of course, patent nonsense.

Bitcoin's merit is in its decentralization. If the block size stays at its current level, it stands that only the current level of users can use it. The ~1000000 people who use Bitcoin today will serve as the centralizers of tomorrow's Bitcoin. That's still 7000 people per centralized authority.

Block size limit should be removed entirely and replaced with a "prefer-small-block" system.

[solex]

Block size limit should be removed entirely and replaced with a "prefer-small-block" system.

Yes! Concisely put.

[fergalish]

Higher fees is not the solution to this. At present the miners are already massively profiting; their electricity costs are far below the income from solving a block, even for GPU miners, and even if you include depreciation of equipment.

Quick estimate:  4xGPU = $1000, lifetime 12 months, equals a cost of roughly $0.02 per block.

Current difficulty, roughly 4 million, meaning you need to perform 17x10^15 = 17 PetaHashes PH to solve a block. The card will perform 1.5MH/J, meaning you need about 10GJ of energy to perform all those 17 PH.  10 GJ of energy is equal to about 2700 kWh which, at $0.20 each, will cost maybe $550.  The 25BTC reward from the block, right now, gets you over $1000. Leaving a profit of $1000 - $550 - $0.02 = $449.98 per block solved, which you can expect roughly every 150 days (17 PH per block divided by 1.3 GH/sec for the 4 GPUs). Data from https://en.bitcoin.it/wiki/Mining_hardware_comparison

Naturally a clever miner will have 40 GPUs instead of 4, and earn those $449.98 every 15 days instead. Or maybe 400 GPUs.

Of course, a clever *and* lucky miner, will be running an ASIC, say, a BFL minirig, churning out 1.5TH/s (no consumer versions online yet, supposedly), solving perhaps 7 blocks per day but requiring only 17 MJ per block solved, roughly 5 kWh or $1.00. Depreciation of the minirig ($35000) over, say, 36 months will cost $0.22 per block.  So the BFL minirig miner would profit by $1000 - $1.00 - $0.22 = $998.78 per block solved, or roughly $7,000 per day. How bad!

I did ignore the fact that the difficulty will increase in these estimates. If I made any other glaring mistakes, please point them out.

Typical Tx fees per block are now maybe 0.3BTC. So even if that were to go up by a factor of 10, it would still be far less significant than the block reward.

Bearing in mind that the purpose of the transaction fee is to attract miners and secure the network, it should be clear that increasing fees at this time is not going to achieve its designated purpose. The block reward is already much more than sufficient for that. And it certainly won't create more block space. What it *will* do, however, is alienate users.

Insisting on higher fees because miners have a monopoly on a commodity (block space) with an artificially imposed scarcity... well, really just sounds like the big boys club all over again, doesn't it?

[lophie]

Exactly. If you want your transactions processed quickly, attach a nice healthy fee with them. If you can't be bothered to include a decent fee then your transaction deserves to be left behind.

Other way to look at it is if you can't be bothered to include a decent fee then you don't really care about when will your transaction be included.

[dree12]

Exactly. If you want your transactions processed quickly, attach a nice healthy fee with them. If you can't be bothered to include a decent fee then your transaction deserves to be left behind.

Other way to look at it is if you can't be bothered to include a decent fee then you don't really care about when will your transaction be included.

You are missing the point. If Bitcoin can only support a million people, no amount of fee will allow it to support a million and one. Limiting block space to keep the fee up is like limiting food production to keep food prices up.

The hard block limit is no different than, for example, a government-imposed food production ban to support farmers. Even farmers who want to produce more food, because they earn more that way (compare with miners earning more by ignoring the limit), cannot because of the restriction. And the people who eat, starve. Nobody benefits from this scenario.

What we should be doing instead is letting the market decide the fee. Make block space unlimited, but expensive to produce. There are a few ways of accomplishing this:

1. Pay-for-space. For every kB a block takes up, the miner loses a certain amount of reward. Even malicious miners lose this reward, as it doesn't go to anyone. The reward could be adjusted like difficulty, or some other method, to allow growth in the network. This ensures there is no limit to block size if it becomes necessary, and ensures the blockchain will not be too big, as there is significant financial cost in doing so.

2. Prioritize-small-blocks. Currently, block length and orphans are determined through sum of block difficulty. This could, through a hard fork, add a risk to making bigger blocks, as they are more likely to be orphaned. If transaction fees are high enough that this risk is manageable, then the block sizes will increase. The only problem with this approach is the increased threat of a 51% attack by someone mining empty blocks. Hopefully the financial incentive of mining full blocks will counteract that.

People complain about how low the transaction fees are. I like to pose them this question below. If this isn't an argument for killing the block size limit, I don't know what is.

One scenario is a billion people paying today's transaction fees, and the other is a million people paying a thousand times today's transaction fees, handling the other people's money because they can afford it.

Both are obviously the same for miners, but which one is in Bitcoin's spirit?

There are so many elegant solutions to this block size problem. Keeping the current limit isn't one of them.

[justusranvier]

Everybody saw what happened when Mt Gox was unable to process transactions in their system as quickly as they were being initiated, right?

[solex]

question for Jeff Garzik or misterbigg, or any defenders of artificially constrained blocks...
is this considered to be of no real concern?

3000+ transactions waiting
$100 in fees alone to be mined (Incentive not yet enough?)
Pending size of 2.37Mb = 90 minutes at 250Kb blocks!

http://blockchain.info/unconfirmed-transactions

[Syke]

question for Jeff Garzik or misterbigg, or any defenders of artificially constrained blocks...
is this considered to be of no real concern?

I do not think you know the meaning of that word. There are numerous reasons the block size is constrained.

3000+ transactions waiting
$100 in fees alone to be mined (Incentive not yet enough?)

The default fee is about US$.02. International wires cost about $50 and take days to process. Yes, $.02 isn't enough for a international currency transfer averaging 10 minutes. Increase the default fee from BTC .0005 to BTC .01 and the blocks will have plenty of room for everyone's transactions.

[Technomage]

I do not think you know the meaning of that word. There are numerous reasons the block size is constrained.

To be precise, the 250KB soft-limit is entirely artificial. In my opinion the 1MB limit is artificial as well, but I do understand some of the reasons to have some constraints on it. I'm however radically against keeping it at 1MB forever.

The default fee is about US$.02. International wires cost about $50 and take days to process. Yes, $.02 isn't enough for a international currency transfer averaging 10 minutes. Increase the default fee from BTC .0005 to BTC .01 and the blocks will have plenty of room for everyone's transactions.

It's very convenient of you to choose the most inefficient money transfer mechanism available as a comparison. Let's compare this to the European International SEPA bank transfer system. Transfers take usually 1 day to any country these days, and the cost for individuals is 0,0 EUR per transfer. Monthly fees may apply.

The cost for companies in Finland is approximately 0,15€ per transfer, with some banks it's also 0. Our company has 3 bank accounts, one of them only has a monthly fixed fee regardless of the amount of SEPA transfers, the other two have 0,15€ per transfer. And as a reminder again, for non-companies the fees are usually 0. The fees are the same with any amount.

Even if we take the 0,15€ as a baseline, the fees can rise a bit though. In any case, for Europeans the need for SEPA transfers radically exceeds the need for wire transfers. I just think that even 0,15€ is way too much, for me the baseline is how much does a 1€ transaction cost. I'd say an acceptable fee is 2% for that size, which is 0,02€.

At current market price 0.0005 is 0,015€ which is very close to what I'd consider the maximum acceptable fee. Any higher than where we are now, and Bitcoin's usability for small transactions is destroyed. This fee could perhaps be as high as 5% for a 1€ transactions but around that mark is the max, if one wants to give Bitcoin a shot at handling small tx. I'm not even talking about micro-tx, I personally think aiming at handling below 1€ tx should be abandoned.

The one and only problem is the block size which is simply way too low to support scaling of any kind. This is not a question of all or nothing, Bitcoin will certainly not be used for all micro transactions in the world, but if it can't be used for $1 transactions and cheaply, it's absolutely doomed.

This year there needs to be concrete progress in the plan to raise the block size, otherwise I'm abandoning ship for certain. My view is that this particular issue is THE issue as far as Bitcoin is concerned right now. My only real concern, to be honest.

[solex]

And yes, the 250Kb artificial constraint seemingly has had an immediate effect upon fees!

So, despite the block reward being >$1000, and not due for halving until 3.75 years time, fees are forced to do a moonshot.

Is there now to be an arm-wrestle between the bitcoin-happy public and dice gamblers as to who can tolerate the highest fees? It has been noted how thick-skinned gamblers are to fees. Will we see the whole network exist solely to support dice gamblers? Is that the future for Bitcoin?


http://blockchain.info/charts/transaction-fees-usd

[Luke-Jr]

And yes, the 250Kb artificial constraint seemingly has had an immediate effect upon fees!

So, despite the block reward being >$1000, and not due for halving until 3.75 years time, fees are forced to do a moonshot.

Is there now to be an arm-wrestle between the bitcoin-happy public and dice gamblers as to who can tolerate the highest fees? It has been noted how thick-skinned gamblers are to fees. Will we see the whole network exist solely to support dice gamblers? Is that the future for Bitcoin?

How's your graph look if you filter out SD flooding?

[Peter Todd]

So, despite the block reward being >$1000, and not due for halving until 3.75 years time, fees are forced to do a moonshot.

That "moonshot" is because someone created a single transaction with 94BTC in fees: 13dffdaef097881acfe9bdb5e6338192242d80161ffec264ee61cf23bc9a1164

Fees are rising, but they haven't spiked like you think they have.

[Rockefoten]

The one and only problem is the block size which is simply way too low to support scaling of any kind. This is not a question of all or nothing, Bitcoin will certainly not be used for all micro transactions in the world, but if it can't be used for $1 transactions and cheaply, it's absolutely doomed.

This year there needs to be concrete progress in the plan to raise the block size, otherwise I'm abandoning ship for certain. My view is that this particular issue is THE issue as far as Bitcoin is concerned right now. My only real concern, to be honest.

I think I agree. Bitcoin has been getting a lot of positive press from more main stream tech sites lately, but I'm not really looking forward to the day they pick up the max block size issue.
My guess it would attract a lot of negative press, maybe even outright  ridicule (along the lines of "Bitcoin - the touted new world currency - allows you 1 transaction per year!" or something like that.)...
Not to mention how the Slashdot commentariat will have a field day.

[caveden]

IMHO. Time to stop gathering metrics at the 250Kb soft limit and allow a larger size, perhaps 500Kb.. Thoughts?

Just drop the soft limit and let miners go all the way to the hard one.

Can't pool operators do that on their own? In the worst case, shouldn't that be a simple patch? What are they waiting for?

The "soft" limit is set by each miner.

When I mined using vanilla, unmodified bitcoind + p2pool, it was a simple configuration setting to change the limit to 900k.

My first block was over 400k.

Soft limit "maxing out" is a non-event.

Oh, it's as simple as changing some configs? So pool operators are voluntarily limiting their blocks with no particular reason?

And some people seem to fear they would go the other way around....

[Technomage]

The soft limit is still just being touched, it hasn't been reached fully. There have been situations where due to Bitcoin market conditions a lot of people are suddenly sending bitcoins, and we have many full blocks in the row. The examples that have been used lately are those situations. That is an exception, most of the time there are a lot of blocks that are not full.

The mining pools haven't done anything about this because there has been no reason to, yet. Now we're quite close to needing to do that so they will probably start doing that. It's in the best interest of a single mining pool to do that, and it's also in the best interest of everyone since not doing that would simply bring bad press to Bitcoin, perhaps quite significant bad press, and userbase shrinkage.

The current recommended fee of 0.0005 is _not_ a low fee, at current price it's 0,015€ per transaction. It would only need to be maybe 3 times that much for people to start avoiding sending 1€ transactions with Bitcoin, if there is any competition. I know that for some uses Bitcoin crushes everything even with that kind of fee, but the whole payment market is changing rapidly and we can't compare the system to a darn wire transfer, which is a ridiculously inefficient system.

[jgarzik]

The "soft" limit is set by each miner.

When I mined using vanilla, unmodified bitcoind + p2pool, it was a simple configuration setting to change the limit to 900k.

My first block was over 400k.

Soft limit "maxing out" is a non-event.

Oh, it's as simple as changing some configs? So pool operators are voluntarily limiting their blocks with no particular reason?

And some people seem to fear they would go the other way around....

For pools with older bitcoind software, changing the soft limit is simply a matter of changing a constant and recompiling.

For pools with recent bitcoind software, changing the soft limit is simply a matter of changing configuration settings:

         -blockminsize=<n>       Set minimum block size in bytes (default: 0)
          -blockmaxsize=<n>       Set maximum block size in bytes (default: 250000)
          -blockprioritysize=<n>  Set maximum size of high-priority/low-fee transactions in bytes (default: 7000)

[solex]

So, despite the block reward being >$1000, and not due for halving until 3.75 years time, fees are forced to do a moonshot.

That "moonshot" is because someone created a single transaction with 94BTC in fees: 13dffdaef097881acfe9bdb5e6338192242d80161ffec264ee61cf23bc9a1164

Fees are rising, but they haven't spiked like you think they have.

Thanks retep for this finding. I really did not consider that one transaction could be responsible. Someone "accidentally" paid $4000 to execute a payment. Ouch! What a coincidence that the world's most expensive payment transfer happened just when bitcoin traffic was so busy...

My concern still stands, that using higher fees as a lever against SatoshiDice-like spam is counterproductive. Just as an overdose of chemotherapy hurts the patient much more than the cancer itself.
There has to be a more intelligent way to rein in abuse of Bitcoin as a messaging system.

[Syke]

The default fee is about US$.02. International wires cost about $50 and take days to process. Yes, $.02 isn't enough for a international currency transfer averaging 10 minutes. Increase the default fee from BTC .0005 to BTC .01 and the blocks will have plenty of room for everyone's transactions.

It's very convenient of you to choose the most inefficient money transfer mechanism available as a comparison. Let's compare this to the European International SEPA bank transfer system. Transfers take usually 1 day to any country these days, and the cost for individuals is 0,0 EUR per transfer. Monthly fees may apply.

Monthly fees may apply, identify verification may apply, bank account required, limited to a few countries. You can't compare Bitcoin transfers with SEPAs.

[Peter Todd]

Thanks retep for this finding. I really did not consider that one transaction could be responsible. Someone "accidentally" paid $4000 to execute a payment. Ouch! What a coincidence that the world's most expensive payment transfer happened just when bitcoin traffic was so busy...

Well if you follow the transaction back a level, the 106BTC input, tx 1a3137bd3962de42a6b01974066e2940e9cd2cd2a393bb87c0e9c7439a702b31, came from Bitcoin-24.com

I don't know exactly what software Bitcoin-24.com uses, but the transactions in that address are all uncompressed pubkeys, and the inputs to the high-fee transaction are also all uncompressed pubkeys. The reference client has used compressed pubkeys by default since 0.6, so it's probably non-ref code that made the mistake. Equally given that the inputs are all recent transactions, I would suspect that we're not seeing someones mistake with a cold storage wallet or something. In addition the fact that one of the outputs is 0.002BTC implies it's some business with custom software.

I found a bunch of transactions before with weirdly high fees and high volume from from address 1JmQN8NvX3XXWWrJW3rEEcKQMQd5DUgkH3; it's still wasting fees. It's also connected to Bitcoin-24's address by one hop, although, that doesn't necessarily mean anything - it's also connected to Mt. Gox by one hop. That said the transactions around 1JmQN8N are also uncompressed keys, so maybe we're seeing common software for all of them? On the other hand, all the fees are even "human" numbers, so maybe it's just someone showing off their wealth.

Long story short: make your custom transaction generating code as robust as possible. Test, log, and audit everything.

[caveden]

That "moonshot" is because someone created a single transaction with 94BTC in fees: 13dffdaef097881acfe9bdb5e6338192242d80161ffec264ee61cf23bc9a1164

Ouch!
Any idea who did this mistake? I mean, it can only be a mistake, right?

Testing in prod is never good... it's particularly worse when you're testing with the equivalent of thousands of dollars....

[whitenight639]

I'm seeing slow adjustments in confirmations,

What i mean by that is blockchain.info is showing 9 confirms whilst my Satoshi client and Mt gox only see 8 it's been like this for over 20mins now, why is that? 

[solex]

I wouldn't worry. The last block was 20 mins ago and can be an hour fairly often (due to the laws of probability)
Mt Gox has been slow for the last hour because BTC is making a sustained foray into the 80s, and new all time highs...

[whitenight639]

I wouldn't worry. The last block was 20 mins ago and can be an hour fairly often (due to the laws of probability)
Mt Gox has been slow for the last hour because BTC is making a sustained foray into the 80s, and new all time highs...

ok thanks,

out of interest..
What would  happen if I had a small botnet and DDOS'd spam traffic on the ports Bitcoin uses, does it (bitcoin-qt) have filtering mechanisms or could it cause this kinda slow network propagation?

[caveden]

What would  happen if I had a small botnet and DDOS'd spam traffic on the ports Bitcoin uses, does it (bitcoin-qt) have filtering mechanisms or could it cause this kinda slow network propagation?

DDoS is an arm wrestling. The one with more bandwidth wins. If the botnet operator is aiming at a single bitcoiner, then this bitcoiner is likely in trouble. But the entire network? His botnet would need more bandwidth than the bandwidth of the entire network combined. Quite unlikely.

Or do you mean DoS through the Bitcoin protocol itself, like trying to flood fake transactions and such? Bitcoin peers have a series of protections against that. The attacker would have to spend a fortune in transaction fees.

[solex]

I think at least some of the Bitcoin nodes are being DDoS'd at any one time. A major attack just yesterday...

https://bitcointalk.org/index.php?topic=154521.msg1674673#msg1674673

[caveden]

Centralized pool operators. Unfortunately that's not new. They've been attacked by these botnet operators for a while. The botnets attack pools in which they're not mining in order to increase the revenues of the pools they're mining in (and thus, their revenues). Some pool operators are already using high-bandwidth ISPs specialized in DDoS protection.

Let's hope ASICs makes mining no longer interesting for botnet operators, and perhaps then they'll leave pools alone.

[whitenight639]

What would  happen if I had a small botnet and DDOS'd spam traffic on the ports Bitcoin uses, does it (bitcoin-qt) have filtering mechanisms or could it cause this kinda slow network propagation?

DDoS is an arm wrestling. The one with more bandwidth wins. If the botnet operator is aiming at a single bitcoiner, then this bitcoiner is likely in trouble. But the entire network? His botnet would need more bandwidth than the bandwidth of the entire network combined. Quite unlikely.

Or do you mean DoS through the Bitcoin protocol itself, like trying to flood fake transactions and such? Bitcoin peers have a series of protections against that. The attacker would have to spend a fortune in transaction fees.

I mean like when you send an IP (might be TCP) packet that has the wrong length or is malformed in someway, some machines not running IPtables will freak out, what if you flood Bitcoin clients with packets full of random junk but legitimate headers, what will happen?



now my client says 12 confims, Blockchain says 9 and mt gox still hasnt credited my btc so who knows. I sent 1.98Btc with a 0.01 fee.

this blockchain limit needs sorting like within weeks not months, it takes the piss how slow transactions are, if this issue cannot be resolved there is no hope for bitcoin to be able to be a real world payment system for reasonable amounts.