Bitcoin Forum

blocks are full NOW

there are already other threads about this

https://bitcointalk.org/index.php?topic=339257.0 (https://bitcointalk.org/index.php?topic=339257.0)
https://bitcointalk.org/index.php?topic=338999.0 (https://bitcointalk.org/index.php?topic=338999.0)
https://bitcointalk.org/index.php?topic=338793.0 (https://bitcointalk.org/index.php?topic=338793.0)
https://bitcointalk.org/index.php?topic=338452.0 (https://bitcointalk.org/index.php?topic=338452.0)

most pools create 250kb blocks only

please do something!!!

> release bitcoin 0.8.6, remove stupid -blockmaxsize
> transactions > 0.01 BTC are processed before transactions < 0.01 BTC
   tell people to not send transactions < 0.01 BTC, bitcoin is not ready for micro payments yet
> transactions not confirmed after 12 hours are removed from mempools automatically
> after 12 hours there must be a cancel/overwrite mechanism

at least tell people/merchants what to do if they have never confirming transactions

massive unconfirmed transactions are coming!!!

I know you devs are busy selling coins, but you owe the community solving this problem at least, before buying your ferrari.

Blocks are not full. Pools can already create 1MB blocks they simply don't.  Devs can't "solve" that.  Complain to your favorite pool to increase the block size they target.

Devs can solve that.
People didn't want it to change...
And this is what might destroy bitcoin.

Just two points, I don't think that < 0.01 bitcoins is a micropayment. At $600-$700 per bitcoin, that is $6-7. Perhaps < 0.0001 might be a micropayment though (6-7 cents), but even that perhaps might be an order of magnitude too high. :-)

Regarding block being filed, D&T makes a good point. 

The title of this thread is wrong and misleading.  Please change it.  I thought it was something interesting and new and it is not.  Thanks.

Include a higher than required transaction fee, and your transaction will go through.

I think maybe pools should advertise a "guaranteed to process given available space" fee level? Lots of people may prefer to pay quite a bit more than being kept in the unknown, which is infuriating.

The min fee (0.1 mBTC per kB) should be fine.  Comparing the memory pool to recent blocks almost all paying tx are included in the next block.   The issue is more people creating tx w/ unconfirmed inputs and people creating tx w/ no fee (and any fee < 0.1 mBTC is considered no fee).  Still pools DO need to start making blocks large.  150KB (0.3 tps) isn't going to cut it.  Bitcoin growth is essentially halted until pools start expanding block size.

I figure that creating fee tiers may help improving the user experience for many, and help motivating the miners to create large blocks as well(the about 7 U.S cents min fee is embarrassingly low, in the sense that you don't even dare to call it a service you paid for and I can't figure out what people are up to when they choose to pay lower), but such policies are guaranteed to create an uproar in a community where many takes everything free for granted.

Are solo miners affected? I mean, those who use bitcoind and their own ASIC devices, at the usual or standard set up.

If, for example, I had maybe 100 TH/s of mining equipment, and I decide to go do solo or p2pool ...

What do you mean affected? 

You can create blocks of any size up to 1MB.  Of course pools can do as well.  They have simply CHOSEN to make much much smaller blocks.   It looks like a few regular miners are keeping blocks <100KB.  Most seem to cap at 250KB.  The average for prior 30 days is ~150KB or a staggering 0.3 transactions per second.

You misunderstand how fees work with regard to large blocks. The minimum fee is not fine. Under the Satoshi client's fee rules, only the first 250kB of a block is available for transactions paying the minimum fee (which is why we're seeing so many 250kB blocks). The Satohsi client will create larger blocks if and only if there are transactions paying more than double the minimum fee. This is due to the fact that large blocks take longer to propagate, resulting in an increased risk of the block being orphaned. Higher fees must be paid to compensate for this increased risk.

Right NOW, the reality is 0.1 mBTC IS fine.  There isn't a massive backlog of paying tx. The backlog is on free tx.  The average block size isn't >250KB it is ~150KB.  That isn't to say it won't be an issue in the future but right now the min fee is fine.  

Still the larger fee for larger blocks is a huge landmine and beyond stupid.  It makes absolutely no sense.  Users have no idea what size blocks miners are targeting.  If a user pays double but a miner is targeting 100KB block the higher fee is just useless.  The larger fees for larger blocks nonsense should just be scrapped.  It might have made sense in the early history when average block was 20KB and there was a risk of a spam attack creating 1MB bloat blocks but those days are gone now.

That seems ingenuous to me, to believe miners would use the satoshi client "as it is".

When you say "Higher fees must be paid to compensate for this increased risk" (of taking a longer time to propagate), the idea is true, but your approach is not quite logic. It's not that there is a threshold above which you take a risk. Actually it starts from the first KB, everything you add in terms of size increase your chance to get an orphan block because your freshly mined block will take more time to propagate, and this is a continous, linear trade-off. Not something that appears after 250KB.

You can actually measure the cost (in terms of network propagation) of including more transactions in a block, and thus define a transaction fee per KB that matches perfectly the real cost in terms of propagation. There was an interesting discussion going on recently on the bitcoin-dev mailing list, in short Michael Gronager shown that a fee of about 0.0004 per KB is "fair" in this regard.

Oh okay. So it's only the major pools are choosing to make small blocks. If I were a solo miner, I could set up and be customized to a larger block size (one that won't fork the chain.)

Larger blocks up to 1MB yes.  Creating blocks greater than 1MB no.

... Looks like a new client isn't going to fix this.

I guess it's time to rethink whether bitcoin will ever go mainstream.

Paying extra transaction fees doesn't make anything go faster. It merely transfers your position on the waiting list, driving up transaction costs for everyone.

Remember, it's the people, it's the miners that are doing this...

Just how would a pool go about putting more transactions into the blocks they find? I assume they have to modify the source code, recompile, and then deploy? Are the changes trivial for the average pool operator?

And on top of this there is a greater chance of an orphan block resulting in the loss of 25 BTC?

It sounds like the pools will not change until the reward is much less then it is today. It simply does not justify the risk to add more transactions and lose out on the reward. Is this a correct assumption(s)?

The fact is that many miners are producing 250kB blocks, and this is the behaviour of the Satoshi client. If many miners are using clients that function substantially differently, we would not be seeing so many 250kB blocks. Whatever modifications they have made, they still seem to be following the same fee rules.

True, there's nothing special about the 250kB limit other than that's the threshold imposed by the Satoshi client. I never said it was a good idea, and in fact I think it's particularly absurd that the minimum fee suddenly jumps to more than double once the threshold is hit and then increases gradually from there, but that's for the miners to decide. If they don't like it, they're free to modify the fee rules in any way they choose.

Not necessarily. A number of parameters regarding the fee rules can be set from bitcoin.conf. In any case, modifying the code and recompiling is not at all difficult.

It appears that way, yes. I expect that as the block subsidy drops, it will eventually stop making sense to scale transaction fees with the block size.

My understanding (pool ops feel free to correct me) is that pools are already running highly customized version of bitcoind.  Honestly if you can't compile bitcoind then you probably don't have the technical skills to run a pool.

Still up to 250KB you wouldn't even need to compile bitcoind simply modify one line in the bitcoin.conf file.  A LOT of blocks are much smaller than 250KB.  So it becomes a no brainer to focus on that first.  The average block in past 30 days is ~150KB.  Eyeballing it, it looks like 80% of blocks are <250KB.  A good 25% of blocks aren't even 100KB and at least 10% aren't even 50KB.  If all blocks were 250KB we wouldn't even have a backlog (except for no-fee txs).

There are some exceptions.  BTC Guild does sometimes push out much larger blocks (350KB to 400KB).  Not sure what conditions trigger that.  Maybe they only do it when the backlog gets real bad.  https://blockchain.info/block-index/441049



There is a cost but fees do offset that.  Users shouldn't expect pools to make larger blocks full of free tx but paid tx offset the marginal cost of larger blocks.  One estimate put the marginal cost (due to increased orphan risk) at ~0.04 mBTC per KB.  Still at some point pool ops have to decide if 1% or so lower orphan losses worth the bad PR of Bitcoin "choking" at ~0.3 tps.  I mean 0.3 tps.   The 1 MB limit is 4 to 6 tps.  PayPal is 50 tps.  VISA is 5,000 tps.   We are at 0.3 tps, and the network seems to be straining.   If pools are unwilling to push the envelope to at least 300KB to 500KB blocks well we might as well pack it up because this experiment is going nowhere.

For the record I do think they will adapt and I do think the protocol will eventually handle block broadcasting in a more intelligent manner so this becomes less of an issue over time.  The falling subsidy will also improve the economics of larger blocks.  It also isn't all on the pools.  Users need to accept you either PAY or you get a confirmation eventually where eventually can be days or possibly weeks.   It is a "charity" option and you get space when space is available (which probably means a block on Tuesday at 2:48 in the morning). The days of pay no fee and see it in the next block are over.  So doing that over and over and getting mad is just silly.

I hope not. I have a medium priority transaction that's not yet confirmed for the past 12 hours, although I think it will confirm by the end of the day. I'll let you guys know. I sent a whole bitcoin a day after I got it, thinking it would confirm. The Satoshi Client (bitcoin-qt) did not request for a fee, so I did not include one.

But, that also tells me, I should include transaction fees next time if I need it to confirm within the hour.

DeathAndTaxes: I'm normally impressed with your posts, but I think you've got some details wrong.

First, RE: the orphan cost of transactions:  Decker/Wattenhofer measured (http://www.tik.ee.ethz.ch/file/49318d3f56c1d525aabf7fda78b23fc0/P2P2013_041.pdf) 80ms for a 1K bigger block. The math to compute orphan cost is:
Plugging in 25 XBT block reward, 600 target time, 0.08 delta time, and assuming no fees (to make the math easier):

So 3.3 millies per kilobyte is the orphan cost.

Even if we assume Decker/Wattenhofer are off by a factor of two (we have made some improvements since they measured block propagation; better measurements welcome), default transaction fees (1 to 5 millies per kilobyte) are in the right ballpark to minimize orphan costs. the .1 default transaction fee does not come close to covering the orphan cost (edited: thanks foxpup).

It should be fairly easy to get about another factor of about 10-20 reduction in orphan costs. And as I said in another thread, if EVERYBODY produces larger blocks then EVERYBODY bears the increased orphan cost, and the result is better for everybody . There is a fixed number of new bitcoins to be earned, regardless of the orphan rate; everybody's share of that fixed number will be the same if everybody has a slightly higher orphan block rate. But everybody will earn more fees, and their bitcoins will be worth more because bitcoins will be more useful.

You seem to have misplaced a decimal point.

Yes, I did misplace the decimal point.  I'm really good at that.

My transaction confirmed in about 28 hours.

So I can still send no fee transactions, you just got to wait a little bit longer. Add a 0.0001 fee, and it confirms a lot sooner.

Oh god !!!!! another misleading title...
Was there ever a full 1mb block?

2 days ago I received a payment for my work. 1 input, 1 output, 0.1 mBTC fee and the transaction stuck for a lot of hours. This is NOT fine. I think I should force all my customers to use BTC-e USD redeemable codes. That delay cost me >4000 USD due to BTC price crash.

Actually the only measurement you need is the orphan rate and the block interval:

http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg03274.html

Note that's actual block interval, not steady state, an oversight Michael Gronage corrected for me in his reply. Of course this doesn't give you detailed breakdowns of latency and bandwidth, but given the consolidation of hashing power the measurements Decker et al. made on network wide latency/bandwidth make a tonne of assumptions about topology that if not already inaccurate, won't be in the future.

All I know is I'm trying to do business in Bitcoins and I've got a wallet full of "0/unconfirmed" and no good answers as to what to do about it.

We ain't ready for mainstream yet.

+1 Change the f*&$%ng title of this thread!

Some factors to consider.
How many blocks were you "passed over"? Measuring in hours is kinda misleading.  A fee can't make a block come faster.   If in a 2 hour interval only 5 blocks were found that is a lot different than if 15 blocks were found.  

Another thing to consider is was the input confirmed or unconfirmed?  Often (not always but anecdotal it seems to be very common) when there is ranting about paying tx being "stuck" for hours it is because one or more of the inputs are unconfirmed and they didn't have fees.  If your inputs are unconfirmed and have no fee then for all intents and purposes your paying tx is a free one (at least until "parent pays child" is implemented).

I shouldn't share this semi-obvious fact but I pay a fee of 0.1001 mBTC on every single tx.  Since miners prioritize by fees and 99.9% of tx are either no fee or min fee (0.1 mBTC) it almost guarantees my tx will be prioritized the highest in the mempool and thus included in the VERY NEXT block for just 0.001 mBTC more than the default fee. :)

Good advice.

Prioritize by fee in the stock code, at least, is made robust against that kind of gamesmanship by treating any fee below a minimum threshold as zero for the purpose of prioritizing the transactions.

And our prayers have been heard!!!!!!

Thanks and I am happy to be corrected.  


I didn't know about this source.  I recall (but can't seem to find a cite) with a much lower estimated cost although it may have simply been an error on my part ~0.4 mBTC vs ~4 mBTC.


Correct me if I am wrong but a rather simple block broadcast improvement would be to not include the tx just include their hashes.  Since a well connected miner aware of a given tx X should already have relayed that tx X to his peers there is little need to include the full tx in the block message.  Instead block message could just consist of the header and a list of tx hashes.  If average tx size is 400 bytes and the SHA-256 hash is 32 bytes that alone could cut the orphan cost of a tx by 90% to ~0.3 mBTC.  In reality since the tx hash is just being compared to the UXTO a more involved modification would be for the block message to include a truncated hash of the tx.  This wouldn't represent a security risk as the actual merkle tree would involve the full SHA-256 hash.  Using say the first 64 bits of the SHA-256 hash would still make collisions essentially highly unlikely events and would reduce the orphan cost by another 4x to <0.1 mBTC.


Agreed.  Hopefully pools and major solo operators can see that their long term profitability is based MORE on the growth of Bitcoin than the short term orphan cost.   Hopefully large pools are aware they don't need universal support.  If 70% of hashpower agrees to produce larger blocks (say 500KB avg) for the good of the network.  It cuts the orphan cost by 70%.   Still I think the orphan cost does highlight the reality that miners are going to be increasingly reluctant to devote more space to free tx.  This is something users will have to come to grips with.   Including a fee and waiting hours or days is unacceptable (although it is often for non-fee reasons) however no fee tx should be considered charity by users and if someone does you an act of charity in an hour, or day, or week well you got what you paid for.  The default behavior of most clients should probably be changed to include the "min fee" on ALL txs not just low priority ones.  If users want to they could change this but they should be warned "Including no tx fee may result in delayed confirmation times".  Enforcement for relaying at node level should still only be on low priority. 

It is somewhat ironic that this is more of an issue due to the higher exchange rate.  The min fee on low priority tx was lowered due to rising exchange rate.  Today 3.3 mBTC is ~$1.50.  Ouch.  However if Bitcoins were worth less it would be less of a cost.  Since Bitcoin is often used as a proxy for USD a 5 mBTC fee (which more than covers orphan costs) is more viable at a lower exchange rate.

0.1001 mBTC is ABOVE the min threshold of 0.1 mBTC. I am going to shut up now as I am quickly going to cut my own throat.

The threshold is 1 mBTC: main.cpp:int64_t CTransaction::nMinTxFee = 10000;  // Override with -mintxfee

On edit:  to avoid a long rambling semi-off topic subthread.  Lets just correct the math error here.

10,000 sat = 0.1 mBTC not 1 mBTC.

Deleting the rest of my posts.

Posting below everything due to time constraints.

I have a feeling almost every large pool operator can compile bitcoin. No worries there. I also 'think' they do not make as many changes as we might think.

In the earlier days they did because mining was becoming popular and the load on the servers kept going up. So RPC changes were made and what not.
But now a lot of those changes are already in the code by default if my understanding is correct. Each time a new release comes out they have to make
fewer changes to the source before compiling.  And it always seemed like it was a small group of people suggesting and coding these changes. Not the operators
themselves. They just paid for it and/or gave a tip.

What does worry them, I think, is making a mistake in changing the source and running a binary that is failing to generate new blocks in some fashion for the pool causing a loss for the miners and a loss of hash rate as users possibly leave. It is simply not worth the risk so by running stock they truly minimize their risk. So until the risk can be minimized for the 'average' pool operator or the default source forces them to ALL change i think what we see today is the status quo.

I run a litecoin pool myself. And I can tell you I do not make a change lightly. I don't run the newest litecoin release. My main worry is scaling the server up to handle more hash rate and keeping things running non stop. I was always interested in getting more fees from blocks but not at the risk of an orphan. Just how long should I wait when litecoin finds a block every 2.5 minutes? That is what goes through my mind.

There is no reason for the threshold to not be set at the same value as the min fee to relay low priority txs.  No wonder why we have all kinds of issues with people having tx unconfirmed.  So the min fee for low priority tx to get relayed is 0.1 mBTC but then they are treated as zero fee by miners for inclusion in a block.  NOBODY (and I mean nobody I just checked the last 15,000 transactions) pays 1 mBTC per KB.  That is an asinine $0.60 per KB. The line of code is utterly useless.  It might as well be 10 BTC.   Every major pool is overriding that as they DO sort by fee.

I updated my first post and deleted the rest.  No sense devoting 20 posts to a math error.  So the min mandatory fee to relay low priority txs is 0.1 mBTC.  The treshold between free and paying is also set to 0.1 mBTC.  Thus paying slightly more than 0.1 mBTC gives priority even over those paying the min.


No it doesn't. Do you mean wallet software does? Yea sure. It does. But the miner behavior is not rounded up.

Well that is probably an oversight then.  How can users make smart decisions on fee pricing if users tx are rounded up to nearest kb for fee purposes and miners aren't.  Either make them both round up or make them both pay fees based on actual size.

So do we all agree that sticking with your strategy of bumping the tx fee to 0.10010 mBTC on my high priority transactions is a sound advice?

In the short term maybe.  Obviously the more people that do it, the less priority access it will give.  In theory if everyone bumped their fee to 0.1001 mBTC then you wouldn't be ahead of anyone to do that would require 0.1002 mBTC.  However in my experience to date the vast vast majority of tx are either paying no fee or the min enforced for low priority txs.   You don't need to be first in line you just need to be close enough to the front to get into the next block.

I wouldn't say "easily circumvented" but yes that would work.  No client supports that so it isn't easy but it can be done.  Now you have to ask yourself is the work worth that tiny amount of value (1/1,000th of a penny).  Probably not.  There are also other methods that can be done called probabilistic micro payments (essentially instead of sending 1 satoshi you create a tx that the receiver has a 1/10,00th of a chance of receiving 10,000 satoshis) but they aren't easy either.  Improved code will make these sorts of things "easy" someday.

The 0.01 thing is finally gone in GIT IIRC, as the thing it was trying to do was largely addressed by the anti-dust-output changes. And indeed, you can avoid the anti-dust-output rule exactly as you describe, and there is nothing wrong with that.  The point of the anti-dust-output rule is to discourage people from creating UTXO which cost more to spend (in terms of marginal fees) then they yield in coin and your little protocol achieves that fine.

This isn't as huge of a benefit as you think.  Miners devote most of the block space to paying tx.  Having a fee is much better than having high priority for getting a tx included in a timely manner.  I pay a fee on my high priority txs which DON'T require it because relying on priority is no longer sufficient for timely and consistent payments.  For example: BTC Guild builds blocks up to 500 KB but uses the default 27KB for free txs.  There is a lot more space for paying tx then free ones.  

Having enough priority for relaying without a fee is one thing, but ultimately you need a miner to put it into a block and if you look at the backlog of unconfirmed txs it is 90% free tx (or have unconfirmed inputs or other problems).


Probabilistic micropayments are more useful for paying in a trust free manner where you are likely to need a large number of small payments and you don't want to make a deposit.  Say some future version of TOR allowed higher speeds for paying customers.  You wouldn't want to make a deposit to an untrusted unknown entity.  However if their fee was 1 mBTC per MB of data transfer you would instead send one probabilistic payment per KB (1/1000th chance of 1 mBTC instead of paying 100% chance of 1 kBTC).  In the long run you will pay the agreed rate and the receiver will get the agreed rate and no trust or escrow is needed.  Never seen it used but it is an interesting concept in payments that Bitcoin makes possible.

Newbie question.

Would it be possible to replace the "MaxBlockSize" by a "MinBlockSize" in the protocol, that would adapt, depending on queue size?
Thus, we get rid of the upper limit, and all miners are equal regarding the orphan risks?

No. There is no one memory pool (queue).  There is no way for any node to know with certainty what tx are known by any other node.  Often people think of Bitcoin as this single unified grand network.  The reality is it is tens of thousands of independent nodes running different rules which are often not in "sync".  Decentralization has a cost.

Agreed.  It is kinda pointless to have that level of precision (54.3% of min fee to relay) for a value which really is just an aproximation.  It would be like going to the Dr office and they record your temp as 98.57828928764388489289892 pretty much everything past 98.6 is just noise.  The idea is that inputs have a certain size and thus the min useful output has a relationship to the min fee which is per kB.  All that is fine and good but 1/2 is close enough to 54.30% and a lot easier to explain.  I would like to see the 0.543 * (min fee) replaced with (minfee)/2.

"The min fee is currently 0.1 mBTC, the min you can send is half that to a single address is half that".

I'm pretty sure there could be ways to achieve that in a decentralized way. I'll think about that.
And even if all miners don't have the same exact information about min block size at a given time, that would be an incentive to fill blocks a little more to be sure...
I might be 100% off, but that would be like block size becoming part of the proof-of-work?

(And by the way, thank you for your incisive but clear insight, D&T  :))

My transaction did not require a fee. It was a whole bitcoin more than a day old. But I was trusted not to double-spend, and receiver didn't need it right away.

So we waited 28 hours for it to get included in a block.

Next time, I'm adding a 0.00010001 fee. :)

One day of mining on a 2 GH/s USB stick takes care of my future transaction fees for a long time. (also, 1 gamble on 98% on any of those dice games.)

The first rule of 10001 club is we don't talk about 10001 club. :)

We may use total bitcoin day destroyed as the tie-break for a fork. I guess Satoshi had a similar idea?

If a block gets orphaned, it means another viable block was added to the chain while the miner was adding transactions to the orphan block, right? Does that mean the miner doesn't get neither the 25 coins nor the transaction fees? If so, what's the point of adding bigger transaction fees as a safeguard against having the block orphaned?

Miners don't add tx to a block after they find one.

Simplified mining process
1) Miners select tx from the memory pool for inclusion in a new block (the number, method, and criteria is up to the miner)
2) Miners construct a merkle tree from the tx in step #1
3) Miners create a block header with the merkle tree, prior block hash, version, timestamp, and nonce
4) Miner will hash the blockheader.  If it is a solution go to step 5.
4a) Miner will increment nonce and go to step 4.  If nonce range is exhausted go to step 4b
4b) Miner will modify another element in blockheader, reset nonce range back to 0 and go to step 4
5) Miner has found a block.  They broadcast it to peers.

So once a miner finds a block they are done.  All the work for deciding tx comes BEFORE solving a block.

I have always wondered about this:

Would it be possible in step 4b) instead of modifying the header do the following

Add some more transactions to the block that have come in since the last header creation
Do the step needed now that the new transactions are in the block
Zero the nonce
Continue

Or is that too much work?

It seems that they may be able to pick up some good paying transactions into the block "on the fly" that way and the new transactions will modify the header for the new nonce cycle.

Technically 4b can be completed by changing the tx set.  Essentially you just need to make any change to the blockheader and changing the tx set works for that.

It probably doesn't make sense to do it on each new transaction and there are methods for pool workers to make a change locally (like incrementing the timestamp) which saves a round trip to the server but  any change to the blockheader (including changing the tx set works).  Pools are highly customized but I would assume they strive for a balance between reacting to changes in the memory pool and having miners make timestamp changes locally to reduce server load.

Computing a new tx set may seem like a lot of work but the tx are already validated before they are in the memory pool so you are just looking at 2x the # of transactions in SHA-2 hashes.  Even for a large block a single CPU core can do that in a millisecond (or less).

Miners already do this. Some pools will even trigger longpolls to trigger miners to get work early if enough new txn come in (or at least eligius did in the past, I'm not sure if they still bother).

I killed all my bitcoind nodes (@ approx 5:00PM CST).  I doubt it'll have any effect whatsoever on propagation, but I can't help but be a bit curious...

Anyone know what timezone the daily charts on http://bitcoinstats.com/network/propagation/ are based on?

How many did you shut-down?

the top 3 on here

https://blockchain.info/hub-nodes

plus one other top secret one!

(remember 129.132.xx.xx is evil)

Wow, they were relaying like crazy...  :)
Did you have any particular tuning (I saw the question raised on bitcoin-dev list, but didn't see any answer) to reduce checking or make any relevant step faster?

I had blockchain stored in ramfs (which requires removing all the diskspace checks from the source)...  also the 600+ connections...  also 1gbit connections, so after a block hit I'd see something like 2000kb-25mb-70mb-30mb-5000kb  in dstat

but probably most important was lowering the sleep timers in net.cpp

none of the verification steps were removed, aside from it checking available disk space

Thanks man, very neat info! In particular the net.cpp sleep time trick, never heard about or considered that... Do you see any downside to lowering the timer in net.cpp with your conf, or it's just all better?

Welp, you're not the only one. I did the same thing in another topic, and wound up calculating (wrongly) on several 'fee' questions.

Here's a protocol design that makes block size *almost* irrelevant to orphan costs.  Could be a reduction in orphan costs of 3 orders of magnitude or more - unfortunately it comes with a cost in bandwidth.

Consider a new protocol message, with semantics "HEY!  Somebody found this new block!" which is tiny for rapid propagation, issued from the miner immediately before he issues the block.  

Clients who are hip to it can propagate it (way faster than the block) before they even get done reading the block, let alone checking it and deciding to send it on.  And clients who don't know what the heck it is can drop it on the floor until their software is updated.  If even one client in five knows about the new message, it'll still propagate faster than a block.

Anyway, you'd then expect clients who hear about one block, then receive another, to hash on/possibly extend the first received only until the first one they heard about arrives, then switch.  Assuming it arrives in fifteen seconds or less.  If it doesn't then the first one they got is the one they work on.

That gives a new block the *effective* propagation time of a very small message, and gives it all the clients it can broadcast to in the next fifteen seconds (which, really, ought to be all of them).

You'll still get the occasional orphan block when two different "announce" messages go into the network at the same time, but this should reduce the incidence of orphan blocks dramatically.  

Not sure If I got it properly... Correct me if I didn't: So anybody broadcasting fake announcements would be able to waste 15s of the miners willing to trust the fake announcement?

Nope.  The miners would still get to work on the first one they received.  Then, if they get the announced block within 15 seconds, they'll switch to the first block they heard about.  But if they don't get the announced block (and if it's a fakeout message they won't) then they don't change what they're doing at all.  Even if that includes *FINDING* a new block before the announced block arrives, in which case the announced block *will* become an orphan.

Couldn't it be possible for blocks to just include hashes of a file, without including all the inputs, outputs, etc, just minimal info, the nuonce (that will be longer than the it is now), timestamp, maybe number of inputs/output and bitcoins transfered.

So bitcoin will work with a DHT just like bittorrent,

the miner will broadcast just this mini-blocks that matches the difficulty, then the rest of the nodes will accept it as valid, and start working in the next block, and simultaneously, ASK for the "file with all the transactions in it".  (this file is just like the block is now.)

So this fix is just like the "announce fix" but with proof of work.

This small fix, together with an option to  "just download the last X blocks", and the proper implementation of said DHT, allowing to lookup by block and also query the network for any input, just like the client actually does with the database, could make bitcoin infinitely scalable without the block propagation issue.

Or a quick&dirty fix could be just make all blocks the same 1 mb size and fill a field with info from the last block(s) in order for them to meet said size, then the block size propagation time will be a non issue, and this additional data can be discarded as it is just a filler.

Then you haven't sold anything.

The "critical window" for computing orphan losses is the time between when a miners finds a block AND the entire network is building the next block on top of that.  That includes the time to relay and verify the node, and all miners to switch to the next block.

During the "critical window" if another miner finds and broadcasts a competing block the network will be split.

Announcing an announcement of a block would solve absolutely nothing.  There are solution to significantly reduce the block broadcast latency but that isn't it.

So If I understand properly the rational behind your proposal is to have the majority acceptance behing taken over the "light announcements" rather than full blocks (but still being resiliant to fake light-announcements).  Is there any advantage of doing this?

The advantage is that the first block announced will win an even race (two blocks at the same level) depending on the speed with which the lightweight message propagates rather than the speed at which the block propagates.  When miners who haven't yet found a block revert to the first block they heard about, that block gets an overwhelming advantage in terms of being the block that the next will be based on.

The idea is to eliminate if possible the cost of block size in terms of orphan blocks.  Miners are limiting block size specifically because a large block can take about 15 seconds to propagate across the network -- a smaller one can propagate faster.  The risk to them is that somebody is putting out a smaller block that propagates faster than theirs.

A lightweight message will cross the network in 8 seconds -- meaning no advantage for smaller blocks released even as little as one second later.  The only advantage a smaller block still has is that it can reach miners a few seconds faster and get them to mine for just a few seconds before the block they heard about first gets there - even if the block they heard about first is bigger.  

If not for the "announce" messages, all the miners that got the later smaller block first would mine on it until the next block was found (ten minutes average) instead of until the earlier larger block reaches them ( < 15 seconds.  )

The difference between "under 15 seconds of mining effort" and "ten minutes of mining effort on average" gives the first block found an overwhelming advantage even if it's a full megabyte.

No miners aren't going to switch to a new block until they receive and verify the existing block so pre-announcement or not the faster block will be received and verified and miners will switch to that.  Doing anything else would be subsidizing the owner of the slower block at the expense of their own revenue.

There are real solutions which can reduce the broadcast time by 90%+ or more.

I thought the idea was to reduce the speed advantage of a smaller block making it less expensive in block awards to create a larger one. ie, subsidize the guy who finds a block *FIRST* regardless of the size of the competing blocks.  

I see your point though; it's a 'prisoners dilemma' for the miners.  In the current environment, a miner would lose revenue if switching behavior, even though the network would benefit with absolutely no change to mining revenues if all the miners switched at once.  Conversely, in an environment where the protocol were already deployed, a miner would lose revenue if he switched to issuing blocks with no preceding announcement.  In both cases whichever miner doesn't go with the majority loses due to increased rates of orphaned blocks compared to other miners with the same hashing power.

If you've got a good way to make the blocks propagate faster though, that will largely accomplish the same thing; get the first block distributed before the smaller block that it would otherwise be competing with is even found.  First strategy I suppose would be transmitting to the neighbors with the most peers and greatest bandwidth first, if a peer knows which neighbors those are.

Well larger blocks are never going to be faster or as fast as smaller blocks.  The goal is to reduce the latency time per kB.  The faster a block can be broadcast the lower the "orphan cost" per tx.   Still larger blocks will always have a higher orphan rate but they also have higher gross revenue.

The good news is the current method is about the slowest, most bloated method possible for broadcasting a block.  Any change is an improvement.

One proposal is to only include tx hashes in the block message.  Currently a block message consists of a block header and a list of all tx in the block.  Most nodes know of most or all of these tx, hell they already have verified them and included them in the memory pool.  This simply makes the block larger than it needs to be.  Instead the block message can consist of the block header and a list of tx hashes.   The average tx is ~ 400 bytes and a SHA-256 hash is 32 bytes so we are talking a 90%+ reduction in block message size and thus propagation time and thus orphan costs.

For example lets look at this recent block:
https://blockchain.info/block-index/443364/0000000000000003b90c99433d07078d5498910442489383f18e250db0a843e2

301 tx and 480 KB.  If the block message was changed to be just tx hashes the block message would drop from 480 KB to ~10 KB or a 98% reduction in size.

Another client side change would be removing the double verification of txs.  This may have already been completed I haven't looked at that code since before v 0.7.

Improving the efficiency of mining benefits all users not just miners as orphaned blocks are simply wasted energy.  They lower the effective security of the network.  Further size reduction is possible but requires some more significant changes to the protocol.

Is this being implemented? Does bitcoin foundation know about this? (rather naive question, I'm 99% sure they know, but what if...)

Hi guys,
My opinion is that we should schedule a change to the max block size to be 10mB in a year or two,
So we match the tps of paypal. Would the world fall If we made such a change? In any case miners could be free to set a lower threshold If they believe it's appropriate.
Micro translations though, would still be Made off-chain, using a network a micro payment channels, as they require fast confirmation.

Best regards,
Ilpirata7

Badly typed on my iPad

Why not a dynamic blocksize?

Just do this:


then you do not need to do this:


and we can easily support more TPS than PayPal.

I think that an automatic dinamic block size is difficulty to design and implement and could lead to disparities among peers. The community should be in charge of deciding what the max block size should look like through planned increases.

Best regards,
Ilpirata79

Badly typed on my iPad

I understand your point, but I think if it is possible it would be the best solution. In fact, a dynamic blocksize would be something like "planned increase" (and decrease).
What I would like, that - if implemented after careful testing - it gives the chance for not having to hardfork the system again. This would make bitcoin highly stable.

So a couple points of clarification.  The idea of including tx hashes in the block MESSAGE wouldn't change the size of the actual block.  It would simply be a new message type.  Remember bootstraping nodes also require older blocks and they won't have the txs either so the existing block message would be useful in cases like that.

Simplified version:
on newest block when latency is important = use header + tx hash message.
on older blocks when latency isn't an issue = use header + full tx message.

As for the block limit well that is another more complex issue.  Honestly I don't see it very important right now.  The number of full blocks since the genesis block is exactly 0.0%.  Even today with a backlog of txs the average block size is ~200KB and the largest block is ~600KB.  Raising the limit doesn't change the "orphan economics". 

If you're gonna be elbows-deep in that code anyway, there's a major privacy upgrade you can do with block formats.

Is there any chance of just storing blocks as list of txid's, list of destroyed txouts, list of new txouts, without telling which txins/txouts go with which tx?

After all, if we're assuming that the clients have already *seen* the individual tx, they can check them anyway.  And look at the list of txid's to find out which ones they're missing, get those, and check them.

This doesn't make privacy absolute in any sense; it just makes it so that in order to trace tx and do data mining you have to be listening in realtime to get individual transactions instead of looking at the blockchain after the fact.  But that would still a big improvement.

Briefly hitting 100 k transactions/day.
Now , if there are some selfish miners who include just a few tx on their blocks and we continue to grow we might hit the limits and the backlog will start to grow.
I'm grabbing some popcorn and waiting for the decisions on the block size.

I think this is a great idea.  One question:  the fees would then be the difference between these two large sums?  Would that work?

No or as a miner I could simply modify your txs keeping your tx inputs and making the outputs to me.  The security model of Bitcoin involves three layers.

1) Senders digitally sign the entire* tx to ensure it is immutable.
2) All nodes (not just miners) verify all txs and blocks are valid.
3) Miners place tx in blocks to create a consensus history.

*well a simplified form

Point and match ;)

This looks promising.

As the adoption rises, the number of tx processed by the network per day will need to be increased greatly.
I'm coming back here with the idea of dynamic minimum block size... Couldn't we just index that size (or min tx count/block) requirement on the current difficulty?

Miners could then reject small blocks that don't meet the requirement (so basically, miners trying to send small blocks to have less latency would get their blocks orphaned, which looks like a good way to reduce orphan cost?).

Combined with your proposition to reduce block size dramatically, and keeping in mind that the network will also become faster and faster, that would allow us to ensure have a better chance that the tx throughput remains consistent with the global need, progressively?

Personally (and I really believe block size is a completely different issue) I think the best option would be to raise the block limit once it becomes a bottleneck to a higher static limit.  The reason is that Bitcoin is very hard to undo and going from 1MB static to 10 MB static is a simple and well understood change.  Other systems while they may be more future proof are more complex and need more time for discussion, analysis, and testing.  I project the 1MB limit will become an issue within a year (or 18 months on the outside) and I wouldn't be confident with any radical change in the block system in a short period of time.

I think the issue has to be solved asap. When it becomes a bottleneck it is already too late. Because it will have an impact on bitcoin, it has to be tested for a longer period.
But afaik it's already very high on the dev's priority list.

Yeah, as ugly as it is, everyone knowledgeable about the issue agrees that if the limit is to be raised, it has to be done in such a way that there is still a limit of some kind that is not under miner control to avoid creating new incentives to centralize hashing power, or for that matters furthering existing incentives to centralize. You know how they say in cryptography that attacks only get worse, not better? What we've seen in Bitcoin suggests that clever ways for large miners to abuse their position to get even larger seem to be found a lot more often than the other way around.

Agreed.  Often is protocol design there is a saying "don't let perfect be the enemy of good".  Is going from one static artificial limit to another static artificial limit perfect?  Of course not.  However it is simple, easy to model, and well understood.  It won't be the be all end all solution which "solves" the issue to 2140 and beyond but it does buy us some time in a low risk manner.

The one thing I actually like about the "orphan cost" is that it acts to limit blocksizes below the cap.  The cap is 1MB today but nobody (as in not a single block in the history of Bitcoin) is 1MB.  So the cap is really acting just as a safety mechanism to prevent malicious activity.  The real "economic cap" is well below the 1MB limit.  That won't change if/when the cap is raised so the thinking should be what cap still protects the Bitcoin network from centralization and blockchain bloating attacks.   With current technology I think 10MB is viable. 

If someone wants to solo mine from a home connection they may have to pay for upgraded internet service but it isn't outside of the realm of possibilities like a 1GB block would present.

Well it is important to remember that raising the block size won't "solve" the problem because it isn't so much a problem as a compromise.  The reason why I say we have time is because miners are currently using all the space "available".  If the memory pool was empty of tx older than the last block and block sizes were rising day after day I might feel differently however IMHO if the block limit was 10MB right now blocks wouldn't be any larger.

Yep, you're right on this.
But I think it will become an issue rather sooner than later. And imho it is urgent, to have a working plan for this phase. I can buy more and more stuff with bitcoin here in Germany, which is great, but it's still fragile and shops depend on the quick transactions.

Ghash.io and ASICminer both still observing the 256KB default. Anyone know if they will increase this soon?

Yup, and a bump lets us learn about what changes as the size is increased without risking unpleasant surprises; frankly even 10x is risky.


Viable? Based on what?

Notably orphan cost doesn't work the way you think it does - under certain conditions (http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg03200.html), conditions while not yet fully present are likely to be present in the future, it is in your benefit as a miner to avoid propagating blocks you mine more widely than a certain threshold. It's kinda like the selfish miner attack, but even more fundamental in that you gain through your inaction rather than explicit actions.

Anyway, this just points to how we need more people working on rigorous analysis of this stuff, rather than empty debate based on nothing... Pity really that the people behind the selfish miner paper are so short-sighted in how they handle PR.

We wouldn't be having this problem if blocks that don't have at least a quarter of the transactions currently in the mempool would, like tx without fees, not propagate across the network.

Then someone starts maximum rate dust flooding transactions and people are forced to include them. Good job. Worse— they send floods of orthogonal transactions to different nodes on the network and now can freely trigger forking by effectively partitioning block propagation.

Actually the issue is more with free tx (or tx with unconfirmed inputs, or other tx which weird issues).  Very few paying tx with confirmed inputs are being "left behind".  Also gmaxwell point is spot on.  If you force miners to include tx you open up whole new attack vectors which don't exist now.  Even if your rule was implemented right now miners would already be "compliant" as far more than 25% of waiting paying tx are being included.

Off course, but that comes at a cost. I think Paypal is a good target because it's about internet payments that does not require very rapid confirmations. We could aim for a little more than paypal  and choose, for instance, a max block size that allows 1.2 or 1.5 of the paypal tps (so 12 or 15 mB).

On the other hand, I think that we should just forget about being able to address the same number and types of transactions visa and mastercard do. Such kinds of transactions (super-market, bar, etc.) have to be made off-chain. First because they require rapid confirmations. Second, because it is useless to register everything into the blockchain (I don't want to have even the coffee I buy registered into the blockchain. In fact, the less transactions inside the block chain, the better).

Best regards,
ilpirata79

That is imho the worst approach. It takes one of bitcoins key features away, being a very simple and clear system.
Imho, you have to be able to pay your coffee without any intermediaries to keep bitcoin alive and to have a real impact on the world.
Cutting out intermediaries and making transactions cheap and fast, that was the approach described in the paper and I think this should stay bitcoins goal.
Allowing the same number of transactions as visa should be the midterm goal imho. I started using bitcoin quite regularly as shops start to accept it more and more and I definitely don't want this trend to stop. This is the real value growth of bitcoin, not some overheated numbers caused by speculation.

I think you will never be able to pay coffee with bitcoins, when (and if) people really start using it (which is what we want). Unless you are willing to pay very high fees.

For this kind of (micro) payments we could follow the approach I described here, which leverages micro-payment channels in course of development: https://bitcointalk.org/index.php?topic=344267.0.

Best regards,
ilpirata79
 

I think you should be careful to distinguish Bitcoin the currency and Bitcoin the payment network when you speak because on this matter what you can "never" do is quite different depending on which you're speaking of— I know you know this but the language makes it unclear.

Off-course, I meant that the coffee transaction does not end up being written into the block-chain, but you *still* are paying with bitcoins (through other means). After all, the block chain is just a distributed time stamp server. There is no need to put everything in there.  So, if you can achieve the same goal (pay) through more efficient means, it is ok.
Thanks anyhow for the clarification you made which is very useful.

Best regards,
ilpirata79

I hope -and I'm relatively certain- that I will be (still!) able to do that in the future and imho any solution away from allowing a lot of cheap and fast transactions on the blockchain will be a major threat to bitcoins integrity. I know that "off the chain" transactions are hip these days, but imho this takes away a huge and major feature of bitcoin, NO intermediaries. This point can not be stressed enough. Taking away any kind of intermediary between people is one of the big pluses of bitcoin.
I don't want bitcoin companies between me and the bitcoin network, I want to participate in the network itself. I could just keeping my credit cards then.

Imho off the chain transactions are no solution at all, they are a way of complicating things.

Off-chain transactions for small or micro payments does not forbid you to send  money to a friend of yours directly (through the blockchain - you may just have to pay a relatively high fee).

The problem is that one thing if what we want, another thing is what we actually CAN have. World is made of constraints. You just cannot record everything, but in any case why would you do that? I don't care if my coffee transaction does not get into the block chain (as a single transaction). Intermediares that lower the required fees and speed up transaction times are just good.

Consider that such intermediares (that I call payment gateways) could even increase user privacy, by aggregating several small transactions into a big one where the information about the component transactions is lost.

Best regards,
ilpirata79


 

Yes and one of Bitcoins big advantages are the small fees. A cheap, fast payment system without the need of intermediaries, that is one huge part of bitcoin imho.
We can have bitcoins without intermediaries. I care if I depended on an intermediary. I don't want companies between me and my money or the money transfer system. I can have that already, it's called a credit card ;). There is no problem with recording everything. Disk space is already very cheap and it gets cheaper and cheaper. If you have concerns about your privacy, I can guarantee you, that these intermediaries will be obligated to track everything, so you might have less privacy than with the existing system, which gives you enough room for anonymity if you really care. Intermediaries make the system unnecessary complicated. And they are absolutely not needed.
PayPal can be called payment gateway as well. For example, if we installed some kind of "no small transactions" system, there is a big chance, that companies like Ebay become these intermediaries. And as I stated above, there will be no privacy as soon as there is some kind of controllable intermediary.

I'm not saying I have the perfect solution, but there are some test-worthy ideas out there and there is no technical reason to limit transactions on the blockchain.

Off chain transactions are not a solution.

For one thing, off-chain transactions means giving up every advantage of Bitcoin - you're now stuck working through some kind gatekeeper service which has the ability to exercise prior restraint (censorship) and probably chargebacks and funds confiscation.

It's especially pernicious when there are about 7 billion people in the world who haven't got a chance to buy Bitcoins yet and they are the ones who most desperately need censorship-resistant money. The proponents of the small static blocksize are effectively locking them out of the benefits of Bitcoin forever.

Secondly, we need massive amounts of transactions on the blockchain in order to generate massive transaction fee revenue in order to pay for the hashing power we need to secure Bitcoin against attackers with nation state-level resources.

Bitcoin won't stay free with small blocks. There's no route to that outcome without sucking it up and making large blocks work.

Thank you, Sir!

I don't have time to completely argument, however, in brief:

1) No problem with credit cards or debit card: I already use them. I could just as easily use them with bitcoins. The issue is the fees that should be low.

2) Paypal is not a payment gateway, but a payment processor. It handles each transaction entirely. Both sender and receiver need to be registered on it. I propose instead a network of interoperable payment gateways where each user can choose its gateway (the cheaper or faster or trustworthy, even if the trust needed if almost zero). To have a transaction among two people, they do not need to use the same gateway.

3) If you want privacy, you can just choose a payment gateway in a country where there is no need to record transactions. A payment gateway could even be located into TOR.


The bandwidth and the storage space issues have already been discussed. You cannot get away with it by just saying that storage is cheap... We must abandon this kind of religious thinking where we just hope that storage will get cheaper and network speed bigger just as much as we need.
Off course there are going to be advancements but the pace of such advancements is not goint to allow to cope with the huge increase of the number of transactions.
In any case, as I said before, you can still use direct transactions (but fees are going to be higher).


Best regards,
ilpirata79

2013/11/28     5.1 seconds   22.4 seconds   1.4 seconds   4.7 seconds
2013/11/29         6.0 seconds   35.0 seconds   1.7 seconds   6.5 seconds

hmm.  well, first off, i guess the average block size was bigger since the difficulty just went up.  but that's much higher than % of block size increase

anyway,


my buffalo node had 5TB upstream used from bitcoind after 25 days, the hetzner used more  , when it wasn't being dos'ed or ddos'ed

I find nothing factually wrong with your point of view, but I can't help but feel that your approach to this problem is against the spirit of Bitcoin. Centralized systems requiring trust are generally cheaper to operate than decentralized ones - think for example of modern governments and the deadlock that they often find themselves in - the cost of a centralized system however is in the potential abuse of trust, monopolistic inefficiencies, etc.

The great innovation of Bitcoin is in allowing for a decentralized financial network at a relatively low cost. While you may be right in saying that the economics may not permit small transactions to go on the blockchain, our focus should be to push the limits of the technology to reduce costs, and then let the free market decide what the blocksize should be, rather than chastising people for having "this kind of religious thinking". Centralized payment systems on top of Bitcoin will always exist, it is up to the individual to decide if the cost of giving up autonomy is less than the cost of using Bitcoin directly, however we ought to do all we can to make centralized systems unnecessary.

I'm sorry if the debate on this issue has evolved significantly beyond this, please feel free to point me in the right direction.

You are presenting a false choice. The choice is not "centralized systems" OR "everything in Bitcoin". There are many options to build non-centralized ways of settling payment on a decentralized Bitcoin— ilpirata79 linked to one such approach. A lack of care into the operating costs of Bitcoin nodes may result in Bitcoin itself becoming effectively centralized, which is an outcome which would make building a non-centeralized payment method on top of Bitcoin pointless or impossible.

The fear of bitcoin becoming centralized because of "heavy" nodes is imho completely unnecessary. The traffic and storage technology can be maintained by small groups, it is not like there will be "one big government node" in every country. It is more like having a node in every town.
Any system that requires a system on top of bitcoin for payments is a threat for bitcoin and it is not necessary. The basic, existing system of bitcoin allows to replace visa if it is adapted to a bigger volume.
Anything else will make Bitcoin complicated and I know I am repeating this, but one huge factor of bitcoin is, that I don't need an intermediary.
Keep Bitcoin simple, complicating systems unnecessarily has never been wise.

The Bitcoin network is not in anyway like paypal or VISA, it is very similar to central banks issue moneys in big blocks and distribute them to the society. Just like the FED never issue credit cards, it is a waste of resource to use bitcoin to do micro payments, and in daily life, people really want to spend the fiat money first

Bitcoin won't replace fiat money, since their usage don't overlap: Fiat money is for spending, its value drops each passing year, and bitcoin is for saving, its value rises each passing year

Bitcoin is an unique addition to existing financial world and it will be welcomed by all the existing players, since it has the highest credibility on the planet and will be regarded as the most trustworthy store of value

If there is no cost to operate the bitcoin network, then bitcoin can also carry out  the mission of micro payments, but when the network traffic and computing power is a limited resource, there should be a focus on its major usage

You know that visa offers some services that will never be replaced by bitcoin.
But visa might use bitcoin for those services.

Bitcoins value just rises, because people believe it might become an alternative currency. It doesn't just rise because it is rare. In this case, all the people saying bitcoin is just another pyramid scheme would be right. All these systems declaring "this and this is now rare and we will sell it to you" failed. Bitcoin as value storage is a tech bubble, because it has no real usage.
Bitcoin as a currency has a real usage and thus it has a value. And I'm very happy, that the number of shops accepting bitcoin is growing, I don't want that to go backwards.
Bitcoin will definitely not be welcomed by the existing players. The players have their systems and they use them. They don't need a decentralized alternative. The people need that, not the banks.
Network traffic and computing power is a very, very, very cheap ressource. You don't keep your fiat, because the paper is "too expensive".

People who think, that bitcoin will be a "value storage" just because it's rare are believing the same bullshit people believed often enough. In this case, the "tulip mania" jerks are right, bitcoin differs in no way from all the other "value storages" that came up in history and went down.

Bitcoin was always designed to be a currency and the existing limits were never thought to be forever.
I don't want to see bitcoin as the next tech bubble and imho a lot of the "next value storage" "gold 2.0" talk sounds fatally like all the bullshit you can hear in EVERY investment bubble.

I 've read Gavin's answer on reddit about block size
http://www.reddit.com/r/Bitcoin/comments/1rqexb/87898_kb_block_just_now/


What does he mean when he says that "fees have to float" and also what's the concern of a "downward spiral of transaction fees"?

Perhaps I will be crucified here as I was at /r/bitcoin.  But have you considered the possibility that bitcoin is simply meant to be the currency for large transactions of money?  Perhaps millions of dollars?  You could buy a house or car with bitcoin.  But send your lunch money with peer coin.  To me, this is economically inevitable, and would serve to drive the price of bitcoin upward and push it in a position where it is a currency that is meant for large sums of money to be sent....which would be good, I think.  

What I am suggesting is that perhaps what happnens now is the altcoins play this role and finally become more than just, well, worthless.  Peercoin is perfect for sending $1 and bitcoin is no longer that great at it.  So what?  

If blocksize isn't scarce at all why not include every transaction that pays at least 1e-8 BTC which you've already received and validated?  If it's purely orphaning that influences your decision why won't all your spending go to bandwidth (neutrino links to other miners and such :P ) and none to POW? etc.

Ok i got the first part of your answer but for the second part i'm not so sure. You mean that bigger blocks will likely cause more orphans?
Anyway thanks for clearing it out.

I think the point is:

- The more transactions a block contains, the bigger it is.
- Bigger blocks take longer to propagate on the network and get accepted.
- If 2 smaller blocks get confirmed before the bigger one gets a child, all TX included in the bigger block don't get confirmed, and this is the "orphan cost".

There is no motivation today for miners to include more TX, as it makes it very risky.
They play small and secure.

The question now, is : "how do we make sure bigger blocks get bigger rewards overall".

Or maybe I didn't get it right, which is also a likely hypothesis. ;D

So , let me make sure I understand this...
Because the rewards from the tx can compare to the block reward.. there is no incentive for miners to create larger blocks , so even if we raise the limits to 10 Mb , there won't be much difference , unless a caring miner and a lucky one gets some blocks done.

That's roughly it.  Miners want to produce small blocks to minimize the time they take to propagate over the network.

Otherwise somebody who came up with a smaller block, later than theirs, can have it propagate and reach a majority of miners before their block does, which results in orphaning their block and sacrificing their block reward.

The "orphan cost" will cut itself in half when the mining rewards get cut in half -- but we can't wait that long because Bitcoin is currently in a mode of intermittent service failure and that undercuts the value of all coins.

A node in every town is risky because it is easy to take down. Who is going to administer such nodes?

The idea is that the max block size is going to increase, but slowly. Maybe a x10 multiplication of the soft and hard block sizes could be a good start (maybe even risky).

At the same time, to be on the safe side, it is good to develop safe and cheap distributed tecniques to handle micro transactions in order to avoid the unnecessary inflation of the block chain. Like the one I posted and that invite you to read.

It is not an unnecessary complication. Other crypto currencies may arise that handle things better because are better designed and  more sofisticated/complicated.

Best regards,
ilpirata79

Sharding the operation of a full node amongst mutually trusting participants seems like an easy enough way to scale.  It's not the ideal, and there appear to be more clever ways, but its certainly better than any centralized outcome, if that actually turns out to be the only alternative.  So I don't really believe centralization of fully verifying nodes is a realistic long-term fear, except in the case of ridiculously high loads and very unreasonable expectations of the blockchain.

Otherwise I agree, and expect this is pretty much how things will play out: keep kicking the block size limit can down the road until the problem solves itself (it just doesn't need to be raised any more), and build micropayment channel networks for microtransactions, where the point that micro becomes macro is defined by reality, and not what we wish it to be (not that we can't push the envelope with clever scaling solutions).

That's what I used to think too. However, the system is self-stabilizing, ridiculously large blocks which fork the blockchain will find themselves orphaned. If unlimited size is too worrying then the simplest safety net would be max block size = 10x median block size during previous 2016 blocks or previous difficulty period.

Exactly!

For a major pool server maybe but the cost is pretty much static so it makes super massive pools and operations far easier to ammortize that cost.


Not really.  A CPU can perform roughly 15,000 ECDSA verifications per second and that is operating single core (no multithreading).  At PayPal transaciton volume = 50 tps, a block would contain 30,000 tps.  Even today that would take a single core no more than 2 seconds to verify.  With a multi CPU server with 8 to 16 cores total you could perform that in a fraction of a second.


Fees buy security.  Right now the subsidy distorts that relationship.  Even at high transaction volume 1 satoshi in fees isn't going to generate significant revenue for miners so the hashrate would collapse and the network would be far easier to attack.   

The value rises because it is a superior store of value and has international transaction ability. Chinese investors highly value its international transaction ability since they can't freely exchange USD with CNY.

Although many shops accept bitcoin payment, as long as you can exchange it for fiat money, you will always get a fiat loan to spend and wait for the bitcoin price to appreciate many folds and repay the loan with a tiny fraction of your original coins, this is a no-brainer (see my signature why bitcoin will appreciate forever)

And, any ponzi scheme or bubble theory does not apply to money itself, a perfect example is fiat money, although it has no intrinsic value, everyone are still joining this ponzi scheme day by day

It rises because it is a store of value (it is not superior, if I looked out for a safe store of value I would definitely not choose bitcoin) and because it's transaction abilities, right.  And these transaction abilities are for example, cheap and fast transactions with a simple system without intermediaries.
Everything else could be also done with computer game money (e.g.), which are not a working currency.

That's exactly my point, as long as the current system gets adapted to more and cheap transactions, everybody can use bitcoin for what they want it to use. You can use it as a good value storage, you can use it to buy your breakfast and you can use it to send money to your parents in Australia, etc. etc.
But if you start to castrate bitcoin more and more, it will lose all these abilities, in long term also the value storage function.

There is no answer right now.

I must have missed something here?

First off, 10 gigs is nothing.  Most major pools are probably on unmetered bandwidth or 100TB limits.

I was sending out about 1.5TB upstream every week, just from running a bitcoind node.. that'd be (whoops) *$180 from Amazon, I guess (in a week)

Difficulty adjustment already provides a mechanism to adjust a variable value with consensus. Why not just treat block size the same?
For example if the average size of the last 2016 blocks were 80% full then the block size would double.

I would say the 2016 blocks which make up the previous difficulty calculation.

I don't think it should shrink, there may be periods where blocks are not fully utilised but if that became an ongoing trend it would only mean people stopped using bitcoin.

I would say there are less risks in slowly growing the block size over time then just not having a limit at all (even if there was a large hypothetical limit). We also need to consider network propagation time. If out of the blue we had a 1 gigabyte block would all the clients globally have this data in ~10 minutes (about 6 minutes when the network hash rate grows)?

Is there any way to compensate miners for creating full blocks? 

I mean, if the issue is that smaller blocks are more profitable due to less broadcast latency, shouldn't larger blocks get a premium to compensate for the loss of profit? 

Right now it looks like the 25BTC per block is built in - but if a block bigger than 750Kbytes paid 26BTC and a block less than 500Kbytes paid 24BTC (or whatever ratios turned out to be needed) .... 

Awww, crap, if we did that we'd get Sybil attacks where miners had to spam the network with  a bunch of "padding" transactions to make every block bigger than 500/750 Kbytes.  You'd have to keep the premium perfectly balanced with the size cost (including halving it when block reward went down) to keep it from being worth anybody's time to game it.  Is there a balancing mechanism like there is for difficulty?  Based on the previous 2016 blocks, is there a statistical measure we can do to determine the size cost?  And if there is, can it be done without providing a motive and method to game that?

What's the expected statistical distribution of block size given the assumption that all possible tx are included?  Can we base a premium on deviation from that statistical distribution? 

In the end I don't see any obvious way to handle this other than increasing TX fees enough to compensate for "orphan costs;"  where does that leave tx fees? 

Oh.  From Gavin's post above, it leaves tx fees at 3.3 MilliBitcoins per Kilobyte.  For as long as the mining award is at 25BTC anyway.  Would people find that acceptable?

In the short term probably not.  The reality is the lowest cost, higher short term revenue would simply be to solo mine empty block and leave all txs even paying ones.   However miners (or pool operators) have shown some longer term thinking and HAVE built larger blocks.   However I think it does illustrate that if the min fee of 0.1 mBTC doesn't cover the orphan cost it is downright silly to expect miners to build massive blocks full of free txs and simply kill their own revenue so other people can avoid paying ~$0.10.

The longer term factors are in Bitcoins favor.  Overtime bandwidth gets cheaper it roughly follows Moore's law.  At the last mile it lags behind but for pool servers in a datacenter it is much cheaper.  You also have the block subsidy declining in half in 3 years which reduces the distortion from the "first 25 BTC are free" effect.   Combined that means even if NOTHING else changes the orphan costs should fall by a factor of 8x in the next four years. 

I already pointed out one potential solution to reducing the block broadcast size/time/latency and thus orphan cost by 90% by creating a message type which contains block header + list of tx hashes instead of block  header + list of full txs.   Combine with block subsidy decline and bandwidth improvements over time getting oprhan costs down to 0.05 mBTC or less should be possible.

Except that ISN'T the problem.   If miners can make a positive return by producing larger blocks ... they will.  They want money, more money is always better.   However tonight you could flip the network to 1 GB blocks and ACTUAL block sizes aren't going to change at all.

Miners actually are including most paying txs.  Take a look at the memory pool, remove tx seen after the last block and 90%+ of the remaining tx are:
* no fee txs.
* txs with unconfirmed outputs.
* double spends or other tx problems.

Implementing child pays parent will improve the second category, the third category probably should just be excluded.  That leaves essentially free txs.   Miners aren't going to produce 20 GB blocks of free txs just because users want something for nothing.  That is never going to happen so as the correct titles says "Blocks are NOT full.  What is the plan?".


The good news is there is something which can be done to make blocks larger and reduce confirmation delays:
1) The default bitcoind has a rule where fees double as blocks get larger.  It looks like most major pools have stripped that out otherwise we wouldn't see blocks >500 KB however that rule should probably go.  It no longer really serves any purpose.  The nice thing is it is a client side change so it requires no protocol change or fork.

2) Education.  Until recently a company as big and old as MtGox was creating free txs.  For something as timely as exchange cashouts that is just stupid.  Sorry it is.  They should know better.  If you want to send your friend Bob some coins and want to be cheap that is one thing but major commercial enterprises should really be favoring reliability over trying to save pennies.

3) Child pays parent.  Currently the way bitcoind prioritizes txs it does not include paying tx with unconfirmed outputs in the next block.  So someone sends you coins with no fee, or possibly a bunch of dust spam and you try to resend them and include a fee and it looks like miners are screwing you over.   The tx selection algorithm needs to be improved so if tx B has as an input an output for tx A and tx A has no fee and is unconfirmed but tx B has a fee the miners will include BOTH A & B in the block.  This would also allow users to fix stuck txs and allow merchants to get confirmations on payments faster by respending them.

3) Improve block message format.  Currently all block messages are the same old blocks back to the genesis block and the newest found block are transmitted as header + list of txs.  For new blocks, nodes that are up to date likely already have most (and probably all) txs so a simple change to create a block header + tx hash message would reduce the bandwidth requires by ~90%.  A 1Mb "header + hash only" block message would be smaller than a 100 KB full tx block message is now.

Miners aren't colluding but they do heavily limit the amount of space the devote to free txs.  Some pools like Eligus make the LARGEST blocks but include zero (yes zero) free txs.   Excluding brand new tx, ones with unconfirmed outputs, and double spends/problems at any given time 90% of the memory pool is free txs.

The fact that miners haven't gone cold turkey and universally killed all free txs doesn't mean it is a reliable method of making a payment.  The free tx volume is growing and the amount of free txs in a block are declining the result of those two is the backlog everyone complains about.   It is silly to think miners are going to change.  It makes no sense for them to do so.  Many will include some free tx because it provides a release valve on the network but while blocks get bigger I wouldn't expect the amount of free tx in blocks to get bigger.  If you want timely processing include a fee it is really that simple.  If you don't then it could be days, weeks, potentially months before your tx is processed.  You are asking for charity and charity isn't always reliable.

Nobody is making 1MB blocks.  Nobody.  Not a single block in the history of Bitcoin is 1MB.  So the "limit" isn't a limit.  Miners are choosing their own parameters which result in blocks less than 1 MB.

It would be like the fastest car in the world is 180 mph and there is too much traffic so the government decides to raise the speed limit from 500 mph to 2,000 mph think that is going to have any material change.   The limit will be raised eventually I have no doubt but right now the constraint on tx volume isn't the 1 MB limit.  That is pretty obvious when there are no 1 MB blocks.  The constraint is on the economics of mining.   Eligus for example makes some of the largest blocks on the network.  Routinely over 500 KB but they also include no free transactions.  Eligus couldn't make a 1MB block right this second even if they WANTED TO because there aren't 1MB worth of paying tx waiting for a block.  So how would raising the limit to 2MB, 5MB, 10MB change anything to that equation?

Case in point here is a recent Eligus block.
https://blockchain.info/block-index/444092/00000000000000029fd11f8e23b450749807f78ab4aa789b764cd10ea7062e59
780KB
1280 txs. 

It couldn't be any larger because it includes 100% of the tx which met Eligus inclusion requirements. 

Good luck with that.

Yes.  That is what testnet is for.  You are running testnet to conduct your own research before "fixing" Bitcoin.

Like I said the limit will be raised.  It is just a matter of how, when, and to what level.   Still if the limit was 2MB right now we still wouldn't have 1MB blocks.  Miners are chosing to self enforce a lower limit.

Actually even that is simplistic because most miners are enforcing tx rules not a preset block size.  The intersection of current tx volume and the rules set by miners is producing blocks <250KB on average.  Going from 1MB to 2MB or 20MB isn't going to change that tx selection behavior.  Most waiting tx (if we look at seen longer than 1 block prior) are free txs.  Miners are willing to make larger blocks but not larger blocks full of free txs.

Here it comes my idea about this issue:

Miners include few transactions in blocks because more transactions equals more probability of orphaned block so lets equal all blocks:

Miners should craft a block normally, so lets imagine they generate a 250KB block. Before they send it to other node they have to concatenate junk bytes (random (?)) to the block data, so all blocks are 1MB.

When a node sees this block, they broadcast it and when they finish they delete this junk bytes and they only the block.


Pros:
- All blocks "are" 1MB in terms of relaying them.
- We avoid other more tecnical mecanisms
Cons:
- Bitcoin QT needs some bandwith more because now all blocks are 1MB.


If one day we need to rise the 1MB block limit this process will be the same but all blocks will require to be 10MB (for example). We only need to concatenate junk to them.


How to perform this hard fork?
Bitcoin core developers can release an update that includes this fix but only enforcing it when the blockchain reaches the block 277000 (30 days later) so we give some time for people and miners to update their software.



https://i.imgur.com/Idy8NOs.png

I would like to see some statement from the developers that actually affirms their commitment to the goal of keeping transactions costs low and transaction volume high.  At the rate we are going, we're going to be limited to only a few transactions per second, and as the competition for block space increases, bitcoin will turn into a system only for higher value transactions.

If the devs are happy to let it go in that direction, I'd like to know now so I can sell my bitcoins.  There's no future in bitcoin if we throw away all the key features.

Prediction:  By 2016, either you will be able to pay for your coffee via the blockchain with affordable fees, or bitcoins will be worth less than $1000 and investment in bitcoin ventures will dry up.

how exactly was that a bold statement?

eligius has a limit of 1MB.  they arrived at that limit because that's the limit set in the bitcoin protocol, probably.  it sure wasn't because they were colluding with other pools

there has never been more than 900KB worth of transactions waiting that have paid the standard fee (0.0001 per 1kb), hence the lack of a 2MB block even if it wouldn't be rejected by the rest of the network

Crap.

This issue might be addressed by making sure all blocks propagate equally slowly.  I read in another thread where someone is proposing forcing all blocks to be exactly 1M in size so they propagate at the same speed, but that's a waste of bandwidth.

No matter how you slice it though, failure of transactions to go through in a timely way looks like a service failure to actual users of bitcoin.

And it matters.  If bitcoin doesn't scale smoothly up past 100 tx/second with minimal tx expenses, then it will die within weeks after that limit starts to inconvenience people.  Investors will drop it like a rock and crash the market the minute they see that this isn't a system which can replace VISA, MasterCard, and Western Union while enabling micropayments and lowering costs by at least a factor of five to compensate for basic distrust of a new system.  That's what they thought they were buying into, after all. 

So, an order of magnitude faster block propagation gets us what?  Closer to the current theoretical max of less than 8 transactions per second, and still an active deterrent for miners to add tx to blocks unless  they pay at least $0.30 in fees.  All blocks propagating at the same speed gets us what?  More orphan blocks (which in the long run doesn't hurt miners because just as many valid blocks per hour will be found after difficulty adjustments), no marginal cost to miners for including more transactions in blocks, but still limited to under 8 transactions per second. 

That's just not enough. 

The protocol needs a fundamental design change for bitcoin to continue to exist at this valuation.  The current valuation represents investor expectations that the current protocol cannot fulfill.

Bitcoin transactions "go through" instantly, just like credit cards.
It's just confirmation that takes an hour or more (credit cards take 6+ months!).

Eligius's blockmaxsize will likely always be the largest possible on the network (minus breathing room to avoid potential bugs).
That's not to say the blocks will always get that big - that depends on priorities, transaction fees, spam filters, etc...

I just tried to send BTC6.5 with a 0.0001 fee (per software recommendation) and it took 2.5 hours! Yes, this issue is impacting users like me now.

Sends are almost always instant with Bitcoin.

Confirmation may have taken 2.5 hours, but that's still relatively fast.
Outside of Bitcoin, it's typically 6+ months with expensive credit card fees or at best half a day with even more expensive wiring fees.

I don't disagree and I'm still super bullish bitcoin, but I think discus fish pool are doing block sizes of 48 kB.  Would be good if someone did something clever about that.   :)

Note: the equivalent Ltc transfer took less than 4 minutes. (I know it's meant to be faster, but still...)

Obviously an unused network is going to find room for your transaction with lower fees (although is it really "equivalent" in that case?).
Reality is that all things equal, Litecoin is not any faster, though!

But it's not equal and bitcoin's transactions have slowed considerably.  :'(

When max block is 1MB and pools are using less than 200kB and in Discus Fish's case 48kB, this is something that doesn't need to happen, I'm guessing, as I'm no tech wizard unlike you.

Rumour has it the Chinese pools can't get a decent internet connection in China and are too cheap to bother setting up a remote block-making node, so they just make tiny blocks and push the workload onto the other miners.
I don't have first-hand knowledge if this is true or not, so research it before acting on it...
Feel free to suggest they get in touch with me if it turns out to just be a technical problem of some sort (these pools aren't participating in the regular pool-operator communciations networks, so I'm not sure how to reach them off-hand).

This is exactly what I was about to post. It seems like an elegant solution with good incentives (if someone includes a reasonable fee with their transaction, the miner would rather have it in their block than some junk).

I haven't thought that deeply about this, but it may not be necessary to have the minimum block size be equal to the maximum block size. Could the minimum size of the next block be calculated by every node based on some property of the N previous blocks? The intuition is that we'd want the minimum size to be maybe 10% larger than the block size that we predict we'll need to include all transactions that pay a reasonable fee. So perhaps if this code were working right now, the min block size would be 250KB instead of 1MB, but the max block size would still be 1 MB.

This is TOTALLY useless. If the content of the junk is dynamic but deterministic (e.g. repeatedly hashing the last block), miners don't need to transfer the junk because everyone know the content. If the content is unspecified, all miners will fill it with 0s. So, again, they don't need to transfer the junk because everyone know the content. If you require different junk for different block, all miners will simply fill it with the current block height. If you require "random" junk, you must have a public algorithm to determine "randomness" so it's no longer random and miners will make it deterministic again. No miner will break this consensus because everyone want to save bandwidth

The point is that any block with size less than the minimum size would be disallowed by the protocol. So it wouldn't matter if all the other nodes knew what the junk values would be.

Minimum block sizes don't work as miners can pad them with transactions between their own addresses.

A block of 1MB does not mean you need 1M bandwidth to transmit. If the junk is deterministic (and it will always be), no one will ever need to waste bandwidth to transmit the junk. So we are going back to the current system (i.e. bandwidth usage is proportional to the total transaction size. You save bandwidth by including less transactions)

Think carefully before you reply.

Doesn't matter.  If all the other nodes know what the junk values will be, then the other nodes will reconstruct the block (at the right size, with junk values) right after the block is transmitted to them (at the wrong size, without junk values). 

That's fine though -- if a miner wants to hit the minimum size value by including transactions to themselves instead of junk, then that still neutralizes any advantage they could have gotten from broadcasting a small block.


It sounds like you are suggesting that miners will collude with each other to fix each other's invalid blocks. If that didn't happen, that any compression that a miner did to get his block under the minimum size would just result in his block being rejected.

Here's one potential argument against that: individuals running full nodes who just want everyone to play by the rules have no incentive to help miners cheat, so when their Bitcoin-QT software sees a small block get broadcast, then even if it is technically possible for them to 'fix' the block to make it larger, they don't want to, so they continue to wait for a larger block. Any cheating miners who start working off of the too-small block won't get any blocks they find accepted by users, because the cheating miners are building on a too-small block. So some corrupt miners can create their own fork of the blockchain if they want, which doesn't respect the min-blocksize protocol change, but no one will care because users will be using the chain that adheres to the protocol.

And here's a counter-argument suggesting that you and Cryddit are right: a group of cheating miners could do two things when they find a block: first, immediately broadcast the small/cheating version of the block. Then immediately broadcast the larger version of the block. Any miners who receive a cheating block will know that a valid block is probably going to follow very soon, so they fix the cheating block to make it valid, and start working off of it (otherwise they'd be worse off than other miners who did this). When they finally get the valid block, they think "yeah, I knew this block was going to come, I'm already working on it." When users of Bitcoin-QT get the cheating block, they will reject it, but they'll soon get the corresponding large block and accept it.

Did I miss anything?

Have the core developers thought much about if there is some clever trick to make it not work to broadcast an initial cheating block and then later broadcast a valid block?

It must happen because everyone want to save bandwidth. Those miners who refuse to collude will simply have higher chance to get orphaned so they will collude eventually.



Non-mining full nodes have no stake in this issue. All you need is one single full-node to fix cheating blocks and broadcast the junk-padded blocks, then all full-nodes will accept the chain. Eventually, non-mining full-nodes will also accept cheating blocks because, again, no one wants to waste bandwidth. Non-mining full-nodes will also keep the cheating blocks because, yet again, no one wants to waste disk space. Cheating blocks will become the new standard block.

The whole idea of padding a block with junk is flawed. It couldn't be fixed.

It actually raises a really interesting theoretical cryptography question: Is it possible to create a long string (the padding bytes) such that you can prove it was not possible to derive the string from a smaller secondary string? (AKA a trapdoor)

You can probably come up with a scheme where the second string was some kind of secret, such that knowledge of it could be exploited, perhaps to steal the value of some fidelity bond. For instance you could computer the padding bytes as H(secret || i) with i in (0, n) and secret being some ECC privkey for a valuable txout; giving the privkey to other miners is obviously risky.

The problem is if anything I think that actually encourages centralization: I can safely give a small number of other mining pools that privkey if we have a legal agreement to only use it for the intended purpose. If my funds go missing, I have a pretty good idea who did it and can get the lawyers involved. The smaller the number of pools, the more powerful and enforcable this mechanisms is; with two pools I have 100% certainty who defrauded me. Unfortunately that's the exact opposite of what the padding idea is trying to accomplish...

Interesting. However, even you have 100% certainty who defrauded you, this is not a conclusive evidence. It is equally probable that you are the one who stole the bond. Of course you can make it a 2-of-2 multisig scheme so the signature of the other pool is needed to steal the bond. However, this would violate the original purpose of having the fidelity bond, which should allow ANYONE to steal the bond by knowing the secret key.

Note, when I said "I" that was to mean "I the pool operator" - hopefully they know what they themselves are doing! Of course, obviously it's not really 100% once you take into account that maybe I've actually had my server compromised, but it's certainly a high enough % that the idea of fidelity-bonded padding doesn't work in practice between largish pools.

I still believe counting the total bitcoin-day-destroyed is the most practical way to address this issue. In this case empty blocks are disadvantaged. I guess Satoshi thought the same? How this could be exploited?

Instead of actually sending junk data why not just sleep for the time it would have taken to receive the junk data?

Disadvantaged how? If later blocks take precedence over earlier blocks, because they have more bitcoin-days-destroyed, it makes it easier for someone who controls lots of bitcoin-days (like the FBI) to mount an attack.

The longest chain must always win. In case there are 2 or more branches with same length, the one with more bitcoin-days-destroyed will win.

So if you have a block with a lot of bitcoin-days-destroyed, you wait to release it while trying to build on it. "Selfish mining" where you win nearly 100% of the races, without the need to mount a Sybil attack.

The only tweak to "first received longest chain wins" that I can see being appropriate is to delay less than full blocks for the amount of time it would have taken to send a full block.

That said, there are other, bigger changes, which might work better. What if the block headers are propagated immediately, and the winner becomes based on the first header seen, so long as the entire block corresponding to that header is seen within X seconds (maybe 30 seconds)?

An even bigger change which is worth considering is to propagate blocks in erasure-coded pieces over UDP, allowing peers to download pieces from multiple peers at once a la bittorrent. (This could also be done using TCP, making some of the flow-control issues easier, but at the expense of significant inefficiency and rendering multicasting out of the question. Ideally I think I'd move everything or almost everything to UDP, with TCP being a fallback for those behind restrictive firewalls.)

I love this thread.  Only 1 year ago the likes of Mister Bigg were going nuts saying miners would create humungous blocks full of 1-satoshi paying transactions, because there was no reason not to.  Therefore block size shouldn't be lifted.

Now people are moaning block size isn't increasing fast enough.

There's no better evidence that a block size limit is no longer necessary - no-one apart from the odd Eligius block is even approaching 1MB.  Miners are forgoing the fees.  Just scrap the cap.  No-one is forcing miners to build on some 4MB block full of spam transactions; they can always ignore it and teach the spammer a lesson.  Free markets will sort it out.

And with 1 BTC worth $800 or more, screwing around is no longer cheap.

I think a cap is still useful.  The cost to massively bloat the blockchain remains a viable threat, and at a tiny fraction of what it would cost to 51% the network.  Bitcoin has a somewhat unique cost vs benefit scenario.  There is actually little direct cost to put a tx in a block however the true cost is the total storage, bandwidth, and memory requirements of all the full nodes combined and most nodes are not miners.

While miners who are economically motivated are unlikely to do something stupid (like create a 1 GB block full of millions of txs with a 1 satoshi fee), an attack may not be economically motivated and at this point dumping hundreds of thousands of GB of additional blockchain size could slow adoption.

That being said I am much more in favor of raising the cap possibly to 10 MB (or even 5MB) to buy us the community the time to fully analyze all the possible implications of future block sizes (no cap, floating cap, high fixed cap, etc).  

I still like this...

What's wrong with the minimum block size idea?  If dummy data doesn't work then make it require real transactions.

Miners will only fill blocks with dummy transactions if they don't have enough available transactions in their mempool.  Otherwise they'll fill it with transactions that earn them fees.  If we can expect a certain transaction rate on the bitcoin network then the min block size could be set accordingly and so we shouldn't see much use of dummy transactions.

No, it won't work. Miners will simply create an extra output in the reward transaction, sending 0 bitcoin to OP_TRUE. They will then use it as an input for another OP_TRUE output, and repeat this process. Since the process is totally deterministic, miners won't need to broadcast these dummy transactions. Other miners will recreate the full block locally. So we go back to the current system: with less (real) transactions, the orphan rate is lower.

Seems like quite a lot of software engineering work with relatively little benefit. If you really want to get together with other miners to speed up transaction propagation there are much better solutions. If you know the identity of the miner you can always start working on the new block as soon as you see the header, and check the validity of it a few seconds later when you receive the whole block. If the block winds up being invalid then you ignore those few seconds of work, and punish the miner who sent you the bad block. Alternatively, a trusted third party can verify blocks and sign the headers. Miners could send blocks to this third party ahead of time, so at the time the block is actually found very little needs to be transferred.

That said, introducing a minimum block size seems to me to be a drastic change, and I haven't seen enough evidence of a drastic problem to implement it. I think the focus at this point should be on raising the maximum block size and speeding up propagation.

Excellent insight.  What if we required a minimum number of transactions, rather than a minimum block size?    There wouldn't be a way to work around that.   As an added benefit, fast blocks would tend to clean up low priority transactions.

It seems like we need to be looking at EVERYTHING possible to increase the scalability of the system, and this would fix one dimension.  

The new minimum transaction requirement could be eased in over time...starting at say 1/5 the median number of transactions.  Once the code was there, it would be easier to upgrade to 1/3 or even 1/2 in the future.  

There is no value to the network in allowing near-empty blocks, and huge potential drawbacks.

So what happens when there aren't enough tx to fill the min requirement?  The network just halts until there is enough?  Of course all this is academic.  You do understand that Bitcoin works on a consensus system and there will never be a consensus to change a core element of Bitcoin (baring possibly a change in crypto due to a cryptographic break). 

Also as the block subsidy declines and networks get faster and cheaper there is less of an artificial subsidy in the network.  Miners which exclude paying tx will simply go bankrupt.  If a orphan cost is less than the tx fee there is no economical reason to not include a given tx.  The miner simply gets less revenue for the same amount of work.  Low barriers to entry will push margin on miners so low that leaving profit on the table will mean operating at a negative margin (mining to lose money).

Up thread I already proposed a non-core change that would cut orphan costs by 90% or more.  That combined with the block subsidy decline will make this a non-issue.

Don't you realize that my suggested workaround will also maximize the number of transactions?

The block subsidy is not going to halve as quickly as the bitcoin price will double (that is, if bitcoin is successfull).  Fees will shrink in BTC terms as the bitcoin price rises (because the supply/demand for blockspace will be priced in accordance with its real value).  Unfortunately this means that the orphan cost is likely to get higher and higher compared to transaction fees.  IMO it could take many decades before a reasonable fee will outweigh the orphan cost.

We could use some numbers on exactly what the orphan cost is. Make sure to take into account that only 4 miners account for more than 75% of the hashing power. Also make sure to consider the benefits that netsplits have on mining income (lower difficulties).

As long as you have a fast connection to GHash.IO, your orphan cost should be little or nothing. Within the next few decades connection speeds should get even faster, and most likely mining operations will move closer to each other network-wise. But most importantly, within the next few decades the need to transfer the entire block after finding a solution should be eliminated - certainly for the larger miners, anyway.

Instead of mining pools, within the next few decades it's likely we'll see propagation pools. The mining will be kept separate, but the generation of the parts of the block other than the coinbase transaction will be pooled. Thus there won't be anything to propagate other than the coinbase transaction and the block header. The rest of the block gets propagated as the transactions come in. And only the previous block hash has to get propagated all the way to the actual physical mining equipment. The equipment which verifies the block can be kept at a location with a really fast Internet connection, while the ASICs can sit in some remote cabin in the tundra or whatever (they'll need a connection with reasonably low latency, but don't need much bandwidth).

All of this without any need for a hard fork, too.

Hard forks should be an absolute last resort.

I think that because the problem is mainly about orphan risk, it needs to be addressed specifically via orphan risk.

consider https://en.bitcoin.it/wiki/Proof_of_blockchain_fair_sharing as a basis for a solution.

The premise is that the longer a transaction has been in existence, the more important it becomes to the acceptance of the next block.

Although the nodes may not all be looking at the exact same transaction list, if the numbers seen are similar, they should be computing a very similar 'credibility score' for the next block.  And if the 'credibility score' is too low, they ignore that block, mining on the previous block instead. 

This creates an orphan risk for *failing* to include transactions (specifically the oldest outstanding transactions) that balances the orphan risk for *including* transactions (and using up block space to do it). 

BDD = Number of bitcoin days destroyed
BD1 = Number of bitcoin days in fee-paid, nonconflicting tx 20 minutes or more old and not yet included in a block
BD2 = Number of bitcoin days in fee-paid, nonconflicting  tx 40 minutes old and not yet included in a block
BD3 = Number of bitcoin days in fee-paid, nonconflicting tx 60 minutes old and not yet included in a block
BD4 = Number of bitcoin days in fee-paid, nonconflicting tx 80 minutes old and not yet included in a block
etc.

So let
Credibility = BDD/ (BDD+ BD1 + 2^(BD2 + 2^(BD3 + 2^(BD4 + ... ))))

And if we find a chain is too 'incredible' (less than 0.000001 or so) we just ignore it and mine on a more credible chain (even if that  more credible chain is just the current 'incredible' chain minus the last block).  Because miners are looking at different tx pools, or may have learned about the same tx at different times, they may calculate different 'credibility' thus that some accept a new block and some don't.  But this shouldn't matter much because if 40% of the miners reject a block then that block gets a 40% orphan risk, which has a tangible cost to the miner producing it and gives the miners a strong motivation to avoid creating the kind of blocks for which that might be an issue.

This appropriately prioritizes fee-paid nonconflicting transactions that have been waiting for the longest time, and has the benefit of allowing the blockchain to resist a more-than-51% attack at least in terms of making sure that no one can keep a valid but unfavored tx *out* of the blockchain forever.

Does it cause a problem that initially rejected blocks can get brought into the chain via a reorg, when they become part of a chain that is, thanks to a later block that includes the tx it missed, more 'credible?' 

Hmm... What happens when someone tries to split the network by flooding different sets of transactions to different parts of the network?

For that matter, what happens with nonstandard transactions?


Hmm... Along those lines, would we possibly see ourselves in a situation where miners pay for high value transactions?

That part would be nice. But then, it'd only increase the means by which someone can buy up...the private keys to old coins, for example.

Sure. About the hardest part would be coming up with the name for this new altcoin.

How about Bitcoin II ?