Bitcoin Forum

Having spent a long time reading all of "How a floating blocksize limit inevitably leads towards centralization" and related threads, I want to summarise the proposals for introducing adaptive maximum block size to the protocol.

For focus's sake, please only join in with this thread if you are willing to operate under these assumptions for the scope of the thread:

• in the long term, incentive to secure bitcoin's transaction log
     must come from transaction fees
• Bitcoin should have potential scalability for everyone on the planet
     to make regular use of it, but equilibria are required to
     ensure scaling happens at a manageable rate

I appreciate that not everyone shares those assumptions, but please keep this thread for a discussion that locally accepts them for the sake of the discussion!

The idea of an adaptive algorithm has to be to make use of a negative feedback loop to achieve an desired equilibrium.
The canonical example of this in bitcoin is the way the difficulty of solving a block is related to the frequency of recently-found
blocks, producing an equilibrium around the interval of one block every ten minutes.

In the rest of this post I evaluate several proposals based on the equilibrium they are attempting to establish, making my own thoughts on the matter clearer towards the end.

First up we have:


This provoked a fair bit of discussion.  The idea seems to be that miners will quit if it's no longer profitable for them to maintain
the full transaction log and mine.  It is unclear what equilibrium objectives this approach has, and I find it difficult to intuitively
say what equilibriums, if any, would be achieved by this adaptation.

Next up we have:


It should be clear that this does not scale very far.  The cost is linear, and is predetermined per unit of data.  By the time you've reached blocks of 8 MB, transactors are spending 100% of the monetary base per annum in order to have the transaction log maintained. The equilibrium is that block size is simply limited by the high cost of each transaction, but the equilibrium is not precisely statable in terms of desirable properties of the system as a whole.

We also have:


The issue with this is how you set the minimum fee threshold?  You could set it to a value that makes sense now, but if it turned out that bitcoin could scale really really high, the minimum fee threshold would turn out to be too high itself, and is not itself adaptive. This approach is going along the right lines, but it doesn't seem to stem from an quantitative, fundamental objective to do with the bitcoin network.

Also:


This doesn't take the opportunity to link the size to transaction fees in any way, and seems vulnerable to spamming.

Then there's:


Same objection as to the 50 BTC per MiB proposal.  It just doesn't scale very far before all the value is being eaten by fees.
This time we get to around 64 MiB per block before 100% of the monetary base is spent per annum on securing the transaction log. Again, it is unclear what the objectives are for any equilibrium created.

Nearly finally, we have people suggesting that max block size doubles whenever block reward halves.  No dynamic equilibrium is created by doing this, and it's pure hope that the resulting numbers might produce the right incentives for network participants.

It seems to me that the starting point should be "what percentage of the total monetary base should transactors pay each year, in order to secure the transaction log".  This is a quantity about the system that has economic meaning and technical meaning within the protocol.  It basically keeps its meaning as the system grows. Why transactors?  Because in the long term holders pay nothing (the initial-distribution schedule winds down), and thus transactors pay everything.  That seems immutable.  Why per year?  This makes it easy for humans to reason about.  Annual amounts can be converted to per-block amounts once we're done setting the value.

Thus, this:


is a pretty decent starting point.  It allows unlimited growth of the maximum block size, but as soon as transaction fees, which are what secure the transaction log, dwindle below the threshold, the maximum block size shrinks again.  Equilibrium around a desirable property of the system as a whole!  Easily expressed as a precise quantitative statement ("long term, transactors should pay n % of the total monetary base per annum to those securing the transaction log, so long as the
max block size is above its floor value").

The exact proposal quoted above sets the amount at 6.25%-12.5% p.a., whereas I intuitively think it should be more like 3%.  I would probably also just state it in terms of the mean transaction fee over the last N blocks, as that is more directly linked to the objective than whether half of transactions are above a threshold.  3% p.a.  would work out at a mean of 12 BTC per block, so would be 0.0029 BTC per transaction to lift the block size off its 1 MiB floor.  Seems about right.

My full reply and subsequent discussion with misterbigg is at
https://bitcointalk.org/index.php?topic=144895.msg1546674#msg1546674 .

Hope that's a useful summary of all the adaptive algorithms proposed so far, even if you don't agree with the assumptions or
my conclusions.

This one is a no-go on technical grounds, because nothing in the blockchain tells people verifying it what trsnsactions were in whose memory pool when nor where, nor what their transaction fees were. Only miners whose memory pools were exactly synchronised, or who by sheer fluke happened to arrive at the same value despite having non-identical collections of pending/queued transactions in their pool, could arrive at the same value(s).

Thus they can make up any values they like, but, also, they aren't writing down in the blockchain what they thought the value was when they made their candidate block so not only can we not know whether they are lying as to that value but we also cannot tell whether they applied the rule correctly since we won't even know what untrue value it was that they would have / should have plugged into the rule.

Alternatively, possibly what is meant is that by looking at the timestamps of those ancient trasnsactions and comparing them to the timestamps of the blocks they got into, we are to guess how many blocks they must have waited before finally getting into a block; if so, then that maybe just adds a whole new reason to put strange timestamps into transactions; specifically, to timestamp a bunch of transactions with hours-ago timestamps and place them in blocks so that this archaeological rule will be fooled?

[/quote]

This ultimately comes down to pricing Bitcoin transaction cost with respect to the competition and a good metric here is the banking and credit card industry based on the USD. The metric that is missing is the velocity of money. If we use the figures for the USD M2 money supply http://research.stlouisfed.org/fred2/data/M2V.txt (http://research.stlouisfed.org/fred2/data/M2V.txt) we get for the most recent figures 1.535 per quarter or 6.14 per year. With a 3% of the monetary base the average cost per transaction becomes 0.5%. This is still very high since it will make Bitcoin barely competitive with credit cards even for small transactions.

The problem here is that we are in effect setting the price in advance rather than letting the market decide. This approach can work to set a minimum amount of revenue for miners at a much lower level where an open ended increase in the block size is constrained by minimum amount of fees.

I'm not in favor of this anymore. I explain why and provide an alternative, here:

https://bitcointalk.org/index.php?topic=144895.msg1547661#msg1547661

Any "sum of transaction fees" reduces to miners picking whatever they want, since miners can stuff blocks with 'fake' transaction fees, with only the moderate risk of them getting sniped in the next block.

In any case, this thread fails on the ground that the starting criteria doesn't mention keeping Bitcoin a _decenteralized_ system. If you don't have that as a primary goal, then you can just drop this block-chain consensus stuff and use open transactions and get _vastly_ better scaling.

All those fixed parameters in the 'proposals' are stinking handwaving.  If you only care about preventing the fee race to the bottom, you make the change in maximum block size be half the change in difficulty on the upside, 100% of the change on the downside, clamped to be at least some minimum.  Doing so eliminates the fee collapse entirely by forcing miners to spend real resources (more hashpower) drive the size up.  ... but it doesn't do anything to prevent the loss of decentralization, so I don't know that it solves anything.

I like Gavin's proposal.  (I mean his actual proposal, not the "half-baked thought" quoted above.)

No hard limit, but nodes ignore or refuse to relay blocks that take too long to verify.  This discourages blocks that are too large, and "spam" blocks containing lots of transactions not seen on the network before.

This might create an incentive to mine empty blocks.  To discourage this, in the case of competing blocks, nodes should favor the block that contains transactions they recognize, and ignore (or delay relaying) the empty block.

I do not agree that it necessarily has any effect at all on blocks that are "too large", depending on who mines them and who they are directly connected to without intermediation of any of the proposed prejudiced nodes.

The top 51% of hash power can pump out blocks as huge as they choose to, everyone else is disenfranchised. You might as well try to stop a 51% attack by ignoring or refusing any block that contains a payment to a known major manufacturer of ASICs so the 51% attacker won't be able to buy enough ASICs to reach 51%. Oops, too late, they already are there. They but lack an opportunity for "spontaneous order" to hook them up into a "conspiracy" that is simply "emergent", not at all pre-meditated - in particular not premeditated-as-in-foreseen* by whoever got rid of the cap on block size, since they would seem to have apparently imagined some completely different "spontaneous order" than that in which whoever has the most [brute, in this case] force wins?

51% attackers can already do plenty of nasty things, now we're gonna hand them carte blanche to spamflood the whole network into oblivion too?

* No, wait, it has been foreseen, so surely if they implement it anyway it is, literally, pre-meditated, isn't it?

-MarkM-

bullioner, have you seen this proposal?

https://bitcointalk.org/index.php?topic=144895.msg1541892#msg1541892

What do you think..?

Same objection as above; its even right there below it where you pointed to.

However I am starting to get a sense that maybe part of why it is not blatant to everyone could be an artifact of scale.

It might be that the sheer size/power/longevity/pocketdeepness of "offender" one imagines it might/would take, as compared to the scale one might be contemplating as an organic progression of "growth of our existing network" or of "adoption rates" is very large.

If you persist in thinking of new players entering the game as newborn from tiny startup / granny's basement it might not seem oh so very likely to be a problem, afterall who are these basement-dwellers compared to the likes of Deepbit and Eligius and other "massive" pools.

But in reality, in the larger scheme of things, our vaunted "most difficult proof of work on the planet", our entire bitcoin network, is puny, tiny, trivially so.

How many puny little ASIC-manufacturing startups are there and how many of their products are deployed so far?

How much "smart money" delayed getting into bitcoins for a year due to there being no point in investing in "to be obsolete any moment now, wait for it, any moment,,, coming up... wait for it...." new hardware? Have you seen any indication yet that such gear could impact difficulty significantly? How many hundreds of millions of dollars, really, does all their currently in production product really add up to so far?

Once you blow a few hundred million on a few regional datacentres doesn't it just make sense to balloon/skyrocket blocksize hard and fast to clear all the obsolete players out of the game? What sense is there in blowing hundreds of millions of dollars on securing a network that cannot even handle your own hundreds of millions of users, (you are, like, facebook or google or yahoo or microsoft scale of userbase, or even something really out of left field like a retirement fund with that kind of number of shareholders considering monetising your "user" (aka shareholder) base by controlling the "pipe" through whch others might be willing to pay to get exposure to them, gosh knows. Left field is a vast, vast field, even without whatever parts of it might also be "outside the box"), but imagining there are no "big boys" out there is maybe rather naive.

Every player, all players, in this current puny prototype prevision of what this kind of software could potentially accomplish, even all of us combined, add up to trivial, tiny, puny, still, even if every chip of every wafer of every ASIC in the pipelines that we know of turns out to work perfectly with no error regions etc (100% yield).

Pretending we are oh so capable of swimming with big fish, oh so tough and resilient, that we should throw away our shark cage seems insanely naive, reckless, foolhardy, stupid, exactly what the sharks hope we will do.

-MarkM-

You'll all have to excuse my stupidity, but what's wrong with unlimited (*) blocks? Let each miner set his own transaction fees and free market competition will ensure that transaction fees are kept low while still keeping the network secure. Surely putting some artificial limit on blockchain size in order to drive up fees is little different to central bankers imposing QE or inflation targets on a currency.

Of course, bitcoin mining will migrate to where there is cheap energy, but this might have a beneficial side effect - more effort will be put into building cheap energy sources (read: renewable (**)) so any given geographical region can contribute to mining and so benefit from incoming tx fees ("free money" for want of a better description).

(*) up to some reasonable limit that shouldn't ever be reached, which will depend on how widespread bitcoin adoption is - visa handles a peak of about 10000 tx per second, assume normal use is 3000 tx/sec, assume 250 bytes per transaction and you'd need on average 0.5GB sized blocks every 10 minutes. For the immediate future, I think there's no reason the blocksize should increase beyond 10MB at the outside.

(**) hopefully renewable but admittedly, not necessarily so.

The name of the game is keeping the block space as scarce as possible, encouraging fees that will eventually have to cover the costs of securing the network but not making it too scarce so that Bitcoin can scale batter.

It's impossible to know how many fees must be collected on average in a block because of the fluctuating value of bitcoins. It's impossible to know what block space is scarce enough but not too scarce. The thing is.. it was also impossible to know 50BTC would pay for adequate security, let alone now after the halving that 25BTC would.. These are simply rules that the market took and is now reacting to.

And from this I can conclude that no matter what adaptive algorithm could be picked, all of them will be entirely arbitrary and the market will simply have to adjust to them and make them work. As long as the relationships inside the algo produce the right incentives the market will find an equilibrium and that's the best we can hope for.

Ok so with this out of the way, there are a few predictions that can be made and incorporated into such an algorithm.

1) when the limit is reached, it's highly likely fees will go up but only to a point (an equilibrium will likely be found between the amount of transactions per block and how much the market is willing to pay in fees per transaction) so the block size limit must be increased before we reach that point
2) when the limit is increased, fees will go down to a point until it is reached again then again 1)
3) when the limit is increased in combination with more fees being collected it's highly likely the value of bitcoins has also risen
4) when the value of bitcoins rises, less of them per transaction are required to secure the network
5) users will always try to pay as little in fees as possible


With this in mind we can now build an algo with the right connections that the market can then use and adequately adjust to. Remember this is all arbitrary just like the 50BTC block reward.

The first rule of my proposal is that the block size limit must induce enough fees that cover the security costs before an increase is allowed. Second, with an increase it should now take less fees per transaction to secure the network since an increase likely means higher valued bitcoins. Third if this reverses and fees fall under a certain threshold, the size limit must be reduced and the fee per transaction requirement increased until fees are above the bottom threshold. Fourth this is adjusted in sync with the mining difficulty retarget schedule.

Arbitrarily I'll pick 50BTC to be sufficient to ensure security of the network forever. How much the subsidy decreases so much more fees must be collected before the block size limit can be increased. Eventually fees will have to amount towards the entire 50BTC.

Now for the juicy part, how to relate the block size increases with fees.

Let's say, again arbitrarily, that if subsidy + fees on average in the last evaluation period of 2016 blocks exceeds 50BTC, block size limit is doubled and because more transactions fit into a block the fees per transaction to reach the threshold are now less and inline with the theory that more activity means rising value of bitcoins. And every time subsidy + fees per block on average in the last 2016 falls under 25BTC, the block size limit is divided by 2 to the minimum of 1mb.


So in practice this works out to at max 12,5% of the entire monetary base being spent every year on network security regardless of the value of a single bitcoin which is perfectly reasonable that this is a constant since the more bitcoins are worth, the more it become lucrative to perform an attack the more should be spent on security in terms of value. If we reach the limit right now, when the subsidy is still 25BTC, with max 4200 transactions per block this works out to 0,00595238 BTC fees per transaction before the limit is doubled the first time, perfectly reasonable right now at the current exchange rate. When the limit has been doubled 5 times it will allow max 134400 transactions per block, which if we reach the required 25BTC on average fees per previous 2016 blocks amounts to 0,00018601BTC per transaction, if 1 BTC is $1000 by then, this is still just 20 cents per transaction.. If at any time fees per block on average start getting below 50% of 50BTC - subsidy, the block size limit is reduced by half.


So essentially fees have a ceiling, once it's reached miners get more breathing room and fees will drop. Once that ceiling is reached again indicated by the fees collected miners again get more breathing room. If ever there's too much room and fees start getting lower then the space is made scarce in order to encourage higher fees.

What do you think?

p.s.: I have no clue if the numbers I used for max transactions in a block given 1mb size limit are correct, so please let me know if that is wrong and if doubling the size limit doesn't mean doubling the max transactions

hazek, I like this very much. As you say, it is impossible to determine the perfect algrithm in advance, but the market will adapt to it to some degree. I think this approach does a lot to secure bitcoin's long-term future.

That was great Hazek. The only problem I see (as raised by others)  is that there is nothing there to "protect decentralization" because as long as the numbers of transactions are continually rising, even at increasing costs per block, so too can block size. The biggest objection I've seen to increasing the block size using an adaptive algorithm like this is that there is the possibility of resource needs increasing to the point of that disenfranchising some portion of Bitcoin users. Personally I don't feel this is a concern because

A) I don't believe it is in Bitcoins interests for the majority of its users to be running full nodes, which is why I point newly interested friends and family members to online wallets
B) I do believe that for even HUGE numbers of transactions (several orders of magnitude larger than now), interested parties with minimal resources could always either have direct or pooled access to full nodes, protecting their interests
C) I believe that Bitcoin will remain as decentralized as it "needs to be", always. This is because those concerned with it becoming too centralized can expend resources (individual or group) to them make it then become less centralized.


The best part of your solution is an implicit agreement with users, which is that if the blocksize and therefore "resources needs" ever increase in the future, so too have the value of Bitcoins. If Bitcoins are worth several thousand USD each, I'm more than happy to purchase many terrabytes of storage to continue acting as a full node regardless of transaction volume and I doubt I'm alone there.

Yeah I know but I at least wanted to see if there's a possible middle ground as opposed to just lifting that limit entirely which I'm absolutely against.

And as you say if enough fees are collected this means that all Bitcoin users are that much richer and can afford more hardware to handle extra storage costs which with this model shouldn't get out of hand at all because it connects what users are willing to pay with how big the blocks can get.

But is anyone actually advocating for that?  Nobody wants a miner to be able to make a 100GB block today full of free transactions. People just want confidence that Bitcoin can be more than a crappy replacement for wire services with a stupidly small potential number of maximum transactions (at costs that eliminate many potential use cases).

While fees have a ceiling, they also have a floor, which I think is too high.  12.5% of the monetary base per year?  For a mature economy that would be way too high.  I would predict that out-of-band methods would undercut the main blockchain for just about everything, functionally reducing the blocksize to under 1MB while users of all size and class desparately attempt to avoid those fees.  On the flip side, this would also make institutional mining (like my example in another thread of Wal-Mart sponsoring mining at a loss for the purpose of processing their own free transactions from customers) the dominate form of security.  I'm not sure if that is good or bad, overall, but I would consider anything over 3% of the monetary base per year to be excessive for any mature economy. Anything else opens up an opprotunity for a cryptocurrency competitor to undercut Bitcoin outright and eat it's lunch.  Keep in mind that the cost overhead of the network functions like a tax, in nearly the same way that inflation of a fiat currency functions like a hidden tax upon the economic base that uses & save in it.  While that's not a perfect comparison in the long run for Bitcoins, it should be evident that our network costs should never exceed 3%, and that a better target would be 1.5% or 2%.  Of course, that is a metric that is relative to both the size of the economy (which we cannot know in advance) and the actual block subsidy (which we can know in advance).

So to modify your proposal, I'd say that until the block subsidy drops down into that 2% range, the range for doubling or halving the blocksize limit should be between the actual subsidy plus 5% and double the subsidy.

Your (*) is what in the code is the hard-coded max block size that all this fuss is about.

-MarkM-

Halving ought not be needed, if max is truly properly hard max, and truly properly never raised too much, once it goes up it should never need to go down.

For one thing, anyone who cannot handle the current max will be out of business when the current max does actually get fully used. In fact maybe even before then, as maybe just a few stretches of full blocks in a row each day might choke them up too much to keep up, and if such stretches are long enough, often enough, they might never be able to get caught up on the backlog before the next solid stretch of back-to-back max-size blocks arrives.

So basically, modulo any wiggle room (like max being big enough that only at Christmas or whatever "busy season" do you ever get close to max sized blocks, let alone all in a row day in day out) everyone who cannot handle the current max is out of the game long before a new max comes along.

So whatever the max is at any given period of history, it is already small enough that everyone in the game at that point in history can handle that size.

Therefore there is no need to lower it unless world war, global catastrophe or somesuch causes a devolution back into the barbaric ancient times when people had to get by with a lower max block size.

-MarkM-

EDIT: We are talking about ABSOLUTE MAX BLOCK SIZE here, not the size miners actually happen to make for maximising their profit.

It's not 12,5%, it 12,5 at max. If this max is reached block space is made more available causing fees to likely drop significantly.


I'm guessing a range between 12,5% and 6,25% is what you'd still think is too much but there's just no way to know this. Just like there was no way to know that 50BTC would be enough or that now 25BTC would be enough. Btw with 25BTC we are currently paying 12,28% in monetary inflation per year for security and a little bit in fees. So such a percentage isn't too unreasonable, it's what we have right now!


I don't quite understand. Can you show an example?


BTW I get the gist of your post it's just that you can't know what a good balance is between fees accumulated and the scarcity of the block space. Your solution while I don't entirely understand it strikes me as way too cheap to have the block size limit raised.. And btw if a competitor can find a better balance algo for this issue I'd actually welcome this, I think that given this problem and other problems Bitcoin is not the last cryptocurrency and will not stay the most popular cryptocurrency for ever, there likely will be better ones and I welcome that. But better ones will become better onea by trial and failure and not because someone knew in advance some formula is better suited to keep the relationships between incentives exactly right.. I think that's just impossible to know.

You are still talking about the ideal size of blocks miners should make if they want mining to be a viable business.

The hard coded block size limit is not motivated by that.

So your guesses might help figure out whether miners can attract enough transactions to have any hope of filling the existing blocks instead of not even coming close to using the whole size already available, but the hard limit is a backwards looking limit in a way, it looks at what the existing network can handle, not at what the future network might be able to handle if all the nodes were upgraded.

We have not even actually proven the network can handle one-megabyte blocks, have we?

How many back to back full one-megabyte blocks have been pumped through so far to demonstrate it really can handle the current max size?

How long can we keep up such blocks back to back without encountering any problems?

-MarkM-

markm,

you are introducing a new debate which is about the maximum block size the network can handle. This needs to be kept separate from the max block size constant debate which is big enough already.

It might be that when the average block size is 700Kb the network begins seriously degrading because blocks are not being received/retransmiited effectively, it might be that this happens when they are an average of 12Mb each. If it is the lower size then we have been debating the wrong issue lately. If it is the upper size then, hopefully, the network will be wealthier (I am assuming that most owners of full nodes have BTC holdings) and some of that could be converted to fiat to buy better hardware.

This debate really needs to be in a new thread, and the results of models/testing made known there.

Why should ordinary users be forced to subsidise visa or even SD spam ? This is akin to the current system where corporations offload as much cost and risk as possible onto the public via government subsidies derived from taxation.

I realize that but looking to get a certain amount of fees before a higher limit is allowed ensures that the networks gets a certain amount of funding to be hopefully able to buy the necessary hardware in order to handle the size limit increase. How high and how far this can go before it reaches the hardware limits regardless of how much money you throw at it I have no idea.. But that's a different problem entirely.


My goal was only to show a middle road between Bitcoin not scalling within it's peer to peer infrastructure and eventually requiring super expensive transactions fees due to a limited block size and between Bitcoin scaling as much as desired at the expense of many peers and at the expense of the incentive to pay enough fees to ensure miners get the needed funding despite the decreasing subsidy.

If there is such a middle road it has to be with somewhat higher transactions fees as a requirement for the ability to handle more tps.

I also proposed something similar to Mr Big, but it would also take into consideration whether the difficulty has gone up or down between the current and last period. All of the proposals thus far consider the economic case, but not the propagation health of nodes in the network, and I reckon that difficulty can actually be used as a canary in the mineshaft warning that increasing the max block size, even given other conditions being met, will potentially cause network problems and put strain on nodes in the network because during that period were not able to safely propagate blocks fast enough and the max block size increase should be either avoided or even reduced in such cases, differences in difficulty and the sizes of blocks over the current and previous difficulty periods should reflect this.

I mention it briefly here:
https://bitcointalk.org/index.php?topic=145641.msg1555072#msg1555072

And fully flesh out my argument for incorporating difficulty here:

https://bitcointalk.org/index.php?topic=144895.msg1538386#msg1538386

Sorry if I sound like a broken record re-mentioning this again, but no-one has even commented or criticised why my approach is good or bad, makes me wonder if I'm deluding myself.

Exchange rates determine whether a decreasing subsididy is a larger or smaller actual real income; double the exchange rate every four years and the subsidy is not shrinking at all, it is growing. Quadruple the exchange rate over the course of four years and the halved subsidy is actually double what the previous halving had adjusted it to.

So quite possibly mining will be completely covered by subsidies, maybe even ever-growing subsidies, until all the coins have been minted, in what is that, 136 years or so from now?

If the block size is to increase to fit more transactions into the money-supply, then unless we deliberately try to spam that size with trivial transactions wouldn't it be more reasonable to suspect that each increase in exchange rate could drive block size up, maybe not linearly but maybe somewhere between some value just under linear and some value down near the square root (some exponential function anyway) of the amount the exchange rate multiplied?

(Four times exchange rate = size multiplies by something between two and somewhat less than four.)

-MarkM-

False.

A certain amount of mining will be covered which we cannot know whether or not it would be enough to not give the proper profit incentives to someone thinking about attempting an attack. With the growth of value of bitcoins it becomes more and more lucrative to perform an attack therefor mining should also bring in more and more profits encouraging more and more mining.

My proposal at max costs 12,5% and at minimum 6,25% of the entire monetary base anually (if the block size has been increased at least once). It's pretty hard to imagine someone would want to spend 25% or at min 12,5% of the value of the entire monetary base just to perform an attack. Now maybe this is overkill, but how can we know? How can we know at what % of the entire monetary base spent on mining is enough security?

If blocks are basically limitless (= have a limit that only serves for DoS protection of somebody mining a 10 TB block, i.e. a hard limit for the next decade of 1 GB/block), your argument would be that this means the total fees per block would be lower than if the limits are tight (= users actually not only have to convince miners with their fees to include their transaction but they also need to make their trasaction as small as possible or for example fitting in between existing transactions or combining a lot of these to make sure they are as efficient as possible)?

Block space currently is quite arbitrarily chosen at a quarter of the actual limit in bitcoin-qt and most miners even follow this ruleset. Still there are fees paid even though there is still quite some room to grow and block subsidy is high.

I'd also like to see a comment on a "too high to reach without malicious intent" block size that does not limit miners to include transactions based on scarce block size but only on their own metrics (e.g. "spammyness", "fees per kB"...). Something maybe along the lines of "10 (100?) times the median (not average, to remove potential outliers) block size of the past 4 years" adjusted together with the block subsidy every 4 years or even adjusted every 2 weeks (then it might be too easy for attackers to shift the median, but it might be easier to recover/grow as well) with the difficulty. Again the goal should be that the block size never limits miners at all on deciding which transactions to include.

We can't. Which is why I revoked my approval for my own proposal for adjusting the block size in order to drive the sum of the subsidy plus transaction fees per block to a constant amount. Note that my scheme is identical to the one you proposed (it is just more prescriptive with respect to actual implementation).

Did we forget that clients will only relay transactions through the overlay network if they have a minimum fee attached (computed based on the size of the transaction and the age of the coin)?

Tsk...tsk...

That might work, though I'm worried that with such rules blocks would become gradually larger and over time the number of full nodes would shrink dramatically as weaker computers get separated from the network. For example, dial-up nodes would get separated right away. No one would care that dial-up users can no longer run full nodes, and they would themselves mostly just say, "Oh well, I guess my setup is too slow to run a full node. Time to switch to a lightweight node." This is probably reasonable for dial-up, but I think that it might over time spread to most people. As blocks become larger, people on average PCs would have to switch to lightweight nodes, then even hobbyists, and then even small businesses.

Maybe there should just be a planned hardfork every 4 years (or whatever) where the max block size is set to something reasonable and other fixes are made.

The miner that mines a block should be able to choose the block size.
Max size limited to: ( previous_block_size * 2 )

If most miners choose to double the size more transactions can be processed.
If most miners keep a small block size fees will have to go up to incentivise the miners to increase the block size.

The miners have an incentive to keep the block size to a realistic size to keep the network working and also earn as much as possible by processing as many transactions as possible.
This will let the market deside the best block size.

This is not a hard rule and you could directly connect to a "friendly" miner who also mines spammy transactions to get these into the block chain. Also I can be that miner myself. This is like the 250kB "limit" that currently seems to be something where people bump against.

I see your point that the network forces some people right now to pay fees, but the same effect could be done if miners start to claim that they won't mine these spammy transactions.

No. First, as a matter of national security, we need a hard limit no terrorist miner coalition or axis of allied enemy nations can exceed to disable the nation's financial infrastructure.

Within that, miners can go ahead and double and halve and bicker and moan all they like.

-MarkM-

Well. The scale up factor could be set lower. Example: Max size limited to: ( previous_block_size * 1.1 )

No attacker would be able to mine every block in succesion and increase the size to unrealistic size.
Once in a while a frendly miner would keep the size to a realistic size and prevent a size increase attack.

Problem solved.

Compound interest is a bitch.

The progrssive doubling proposals are suspicious for the same reasons, percentages instead doesn't really change the fundamental unsustainability of compound interest.

Better to be a backward nation with an unassailable infrastructure than a wet paper bag any superpower or axis of allied determined enemy nations will blow right through simply as a side-effect of catering to the endless greed/gluttony of their ever more obese moar moar moar proletariat. ???

-MarkM-

You give no logic reasons.
The miners have incentives to keep the size to a realistic size and also include as many transactions as possible.

The market will clearly find the right size based on what is possible and what makes the most money. Evil miner Attackers that can create 10 blocks in a row and increase size to problematic size is not realistic.

We're a long way from that.  Even something at the scale of Paypal (~85 tps) could run on an average desktop PC with 1mb/sec internet connection.

The post from Gavin that I was referring to is here:

https://bitcointalk.org/index.php?topic=140233.msg1503099#msg1503099

51% attackers were not realistic a while ago, then came ASICs. Once we get tons of ASICs in place, 51% attacks will again start to look unrealistic. What is realistic is the concept of detemined, fanatical foes, even ones willing to sacrifice not just other people's lives but their own. Is suicide bombing realistic? Yet it happens.

As soon as we have some manipulable (and to adapt really just means to be manipulated, whatever is being adapted to is the manipulating variable) increase system, that is a whole new vector for realistic and unrealistic attacks.

-MarkM-

The best way to protect agains an attack is to let the system grow as fast as possible. Limiting growth in any way increases the chances of an attacker to catch up to the system.

A free market without price controls and arbitrary limits is the fastet growing. Limiting size will slow growth.

That is why we need the miners to be able to bump up size when needed and not wait months for the community to agree.

Then maybe it really is best to just let whoever can manage to build petabyte blocks do so as soon as possible to raise the bar as high as possible as fast as possible?

Its not like there aren't alt-chains nations of various sizes, technology-levels and industrial base can use if the G7's ideal uber-network is a bit too much for them to install/use domestically.

-MarkM-

You don't make sense.
There should be a limit to the growth rate but it makes sense to let the miners that do the work deside what direction and how fast it should increase within some limits that prevents an attacker messing to much.

The miners have incentives to do the right thing so why not let them do it?

Huh? Isn't that what I just said?

Let them use petabyte blocks if they want, what the fuck, who cares, if some nations/people's technology or infrastructure can't handle that there are plenty of alt-coins they can use instead.

For example anyone who thinks anything more than 60 megabytes per ten minutes would be a bit too much for them can choose geistgeld, with its 60 one-megabyte blocks per 10 minutes (one block per ten seconds).

Anyone who thinks any more than four megabytes per ten minutes would be a bit much for them to handle can choose a one block per two and a half minutes coin (is litecoin one of the 2.5 minutes blocks ones?)

Anyone who thinks five megabytes per ten minutes is better than four, but more than that might be a problem can use the two minute blocks coin, I am fairly sure there is one I just cannot recall which one it was. Unless it was merely someone carelessly saying two minutes when they meant two and a half.

Basically if the megacorps / superpower-nations, or the places/people with the most-massive highest-capability infrastructure want to remove the limit so the vicious dog eat dog world of ruthless competition can get to the most massive, highest-bandwidth system as fast as possible to ensure it is absolutely bigger than any other in the world, so big no other network could ever hope to compete with it, there are plenty of alternative blockchain based currencies anyone with less interest in or desire for an infrastructure-tech bubble can choose.

-MarkM-

EDIT: "We don't care what your fat cat banker's nubile daughter spent on individual doses or even sixpacks of bubblegum, but we do want to account correctly for our local currency you spend on the shiploads of bubblegum we export to you."

Been very interested in this debate, and I'm interrupting mughat and markm's conversation, and I apologize, but I'm sure they'll deal.

There was one proposal, which I can't recall now (somewhere deep in that giant thread) about having one unlimited/very large block every two weeks or so, to clear the backlog of transactions.

I think it warrants more discussion since most people seem to have passed it by. But perhaps I'm just missing some obvious flaw.

Thoughts?

That sounds like my proposal, to permit each re-target block to be unlimited.  The re-target block comes once every 2448 blocks (or so?), and is intended to be roughly every two weeks.  This would make the deliberate padding of blocks to force out small players ineffective, reward honest miners with an expecially profitable block if they are able to handle it, and preserve the market for rapidly confirming transactions for the remainder of the two week period.  Any small players who were overwelmed by a huge block would simply have to write off the next couple of blocks while they caught back up with the rest of the network.  It's also provide an outlet for free transactions and fee paying transactions that simply don't have enough to get included into a block, so the backlog of unconfirmable transactions (and thus the transaction queue) won't grow into infininty.

However, there could be, and probably are, some unintended consequences that this could create if this were the only change here.  The first one that I can think of is that there still wouldn't be any way to compel miners to include old transactions, free or not, and then such free transactions might never clear out regardless.

An alternative that I also offered is to have a special case, wherein a miner could produce a particularly large block (probably not good to be unlimited) if all the transactions included were free; as evidence that the miner doing the processing is either doing so altruisticly, or is being compensated by an out-of-network agreement.

However, I'd modify this proposal a bit that wouldn't require that all transactions be free, but that all of them were part of the transaction pool for at least a week.

I think dree12 had the most logical argument for doing away with the limit altogether.

https://bitcointalk.org/index.php?topic=145636.msg1551002#msg1551002 (https://bitcointalk.org/index.php?topic=145636.msg1551002#msg1551002)


My own question: The bitcoin network has operated well under the limit so far.  If arbitrarily increasing the block size were so advantageous to certain miners, why hasn't it already been done?  We should have seen the limit maxed a long time ago, yes?

Dree12's proposal sounds reasonable. Bitcoin is like the little baby bird in the nest and we're like the big  protective momma bird. We don't want to let her out to fly in case she's not ready and she plops to the ground!
For a while the network was way too small and we worried about attacks. And of course with ASICs starting to go online there are concerns about too few people with too much mining power but soon ASICs will be widespread so that won't be an issue. There should come a point where we recognize Bitcoin's maturity and let it decide for itself what the block size is. The important thing is to do it soon because a hard fork takes time. I feel like it should have already been done... if BTC keeps growing at this rate, we're not going to have even a year to introduce a hard fork!
TBH and IMHO it matters more that we do something than what we actually do... keep draggin' the feet and there will be scalability problems in 6 months and people will migrate to alt-coins.
(Is a hard fork necessary for removing the block size limit altogether?) </noob>

I also would like to know why people aren't just pumping out 1MB blocks non-stop which is apparently what will happen if we remove the limit.

Increasing it beyond 1 million bytes would require a hard fork. We are close to reaching the 250k bytes "soft limit" of bitcoin-qt however, mostly "thanks" to Satoshi's dice.
Writing custom rulesets requires work and this was not necessary so far, there are actually no real specialized "mining" bitcoind implementations out there as far as I'm aware.

Everyone running a full node has to store currently ~2.5 GB of SD transactions alone.

Currently the rules in bitcoin-qt are quite tight concerning "spammyness", also quite a few pool operators are as far as I see it rather interested in the long run than risking something by making a quick change for a few extra coins (=accepting more spammy transactions with and maybe even without fees).

Yes, but only because the max blocksize is coded into the rules that define the validity of a block, and anyone that refuses to upgrade to the new set of rules (on purpose or simply by ignorance of the issues) would force the blockchain to split into two competing versions.  Thus, for a time, there will literally be two different versions of the truth, and that cannot persist.  Bitcoin is designed to manage relatively short splits that result from temporary conditions, such as bandwidth issues for entire sections of the Internet.  However, it's not really designed to be able to recover from a split that lasts more than a day.


No one has tried to game the system, in part, because the presence of the max_blocksize rule would make any such attempt futile.  If we do remove that limit altogether, that kind of criminal calculation changes, and attempts might be made.

Did you skip over the explanation for how an unlimited block size drives fees to nothing?

You should have placed that at the end of your question too! The question has an obvious answer.. The reason big miners aren't stuffing blocks right now is because 1Mb blocks can't be used to attack smaller miners into quitting. Quite fking obvious isn't it?

Did you skip over dree12's explanation as to why it wouldn't?  If you disagree with his explanation, can you refute it specifically?

Of course I can, and this has been explained MULTIPLE times. Miners want to maximize their revenue. Users want to minimize their fees. If miners always include every transaction regardless of fees, there's no incentive for users to pay fees. The only way for a market for fees to emerge is if there is competition for transactions in a block. That means that miners have to leave some transactions out. But which transactions should they drop? Obviously...the ones with the lowest fees per kilobyte. Or to put in other terms, a market for fees can only emerge when block size is limited.

I think that when people come on here and post claims that there should be no limit on block size that means either they did not bother to get acquainted on the subject by reading the exist posts, or they simply do not have the capacity to understand regardless of the amount of explanation provided.

I already did that. It's the same reason why in a market regulated strictly by consumption i.e. a free market there can be no racists businesses because they'd drive themselves out of business by discriminating. Same goes for miners who would discriminate against those paying less fees than such a miner would demand.. that "business" wouldn't simply be gobbled up by some other miner and so the competition for putting as many transactions into a block paying as much as possible would force all miners to accept ever decreasing transactions until we Bitcoin ends up with only a handful of miners providing not nearly enough security.

Couple of posts that explain things well:

I understand that reasoning.  Why do you think this proposal wouldn't work to mitigate such an outcome?

I don't even understand that proposal but from the little I do understand, I don't see how it relates to changing the maximum block size. Maybe you could rewrite it in more clear, formal terms? Here's an example of my proposal for changing the maximum block size:


See how it's concise? Try to mimic that.

I strongly disagree with this, as limited block size is not the only limitation in the system.

I agree that miners want to earn fees and users want to pay as few fees as possible. Miners however in your (limited block size scenario) not only can leave transactions out of blocks, they have to leave them. Even if there are more transactions out there that the miner would like to include, he has to choose as subset of the available ones.

Even if miners only can ignore transactions, they will do so - including more transactions means slower block propagation times and as already said in a different thread, bandwidth is still a limiting factor in an unlimited/virtually unlimited block size scenario. People are getting worried that miners might start to inflate blocks to drive other miners out of the network. Still having a bunch of miners that can only send blocks with a 100% used 10 GBit link between them would just mean they'd have the longest chain but nobody can use it, as it is not accessible to anyone outside any more (sending data out of the mining network would mean the miner gets driven out of business by his colleagues that have no more need for him). I'd like to call it the miner circlejerk scenario.

Anyways, since larger blocks = higher risk for orphans = fees need to be higher to migitate that risk (e.g. 5% orphan risk = needs 5% more fees than needed with 0% orphan rate), miners still can and will create a market for fees, even with (practically) limitless blocks.

So it'd be like the search engine circle-jerk back when the most relevevant possible page, in the engine's eyes, was its own output.

So we got what, a big three in each part of the world?

Modulo the fact that blocks are not cultural/linguistic the way web pages are, so maybe the big three can be the same big three world-wide, no "you don't even speak our language, you don't understand the nuances of our pages, thus your searches suck" effect helping very-different cultures to prop up a big one of their own (do they even get to have a big three, or do the regions where the big three aren't the biggest only have a big one of their own not their own whole big three?)

Way to not centralise. Are the big number two and big number three still shrinking relative to the big one or are they becoming more equal?

-MarkM-

I think misterbigg's voting proposal is the best one thus far. My reasoning: Miners are more likely to keep on mining if they can maximize their real revenue. And if over 90% of mining power thinks that increasing the max block size will increase their revenue, I'd say it's very very likely that this is indeed the case.

Of course this will mean big boys will drive hobbyist miners out of the market. But they can do this regardless: Large scale ASIC operations with industry level electricity contracts (and located in places where energy is cheap) can drive difficulty so high that hobbyist rigs can never even dream of turning profit.

I'm not sure what you try to tell me about block size with this...? If that is a (bad) attempt at sarcasm, please keep that out of the debate and post some clear arguments/reasons. As far as I get it, you're saying that there is a threat of centralisation and 3 miners competing in an endgame scenario? Sorry, but it is really not clear to me what point you want to get across.

The thing is that hashing adds directly to proof of work.  A miner's costs become proportional to his hash power.  There is an economy of scale here.  If the optimal point is at 0.01% of the total hashing power, then things stay distributed.  If economies of scale continue so that a miner with 90% of the hash power is more efficient than one with 10% of the hashing power, then you end up with a "natural" monopoly.

A situation where things might end up as a natural monopoly must be avoided.

If miners become bandwidth limited, then an entirely new dynamic takes over.

This means that the primary cost for a miner is bandwidth.  In this situation, a miner's costs are not proportional to his hashing power.  A miner with 90% of the market only needs to pay for bandwidth once and gets 90% of the fees, while a miner with 10% of the hashing power also pays the same, but only gets 10% of the fees.  This puts the 10% guy out of business.

Bandwidth usage doesn't add to the formal proof of work of the main chain, so it is just a deadweight loss, without adding security.

If most of a miner's costs are hashing power, then things are much less likely to end up in the natural monopoly situation (unless hashing ends up with massive economies of scale too).

I would like to ask devs that before anything is done with the block size limit, the 0.00000001 BTC spam be dealt with.

Bitcoin should be about money and 0.00000001 BTC is not money. A transaction should not be allowed to have an output lower than the minimum fee. Such output is essentially unspendable and is just spam.

Once low output spam is stopped, the block limit will be enough for 1-2 years to come, even with extreme bitcoin adoption. That's enough time to reach the best decision how to continue forward.

There is no such thing as a minimum fee in the protocol, only in the current rules of the most used client. The minimum fee is either 0 or - if you say 0 is "no fee" - 1 Satoshi.

Maybe testnet could be more prominently placed in the client, so people can play around (which is mostly the reason for 1 Satoshi transactions I guess), also with more meaningful amounts.

SatoshiDICE sends 0.00000001 back to you for some losing bets.  Basically, they are using the permanent blockchain as an instant messaging protocol.

Why, what's s the big deal? Soon, blocks will always be a megabyte, does it matter what transactions go into the block as long as the fees get paid? The block chain is going to grow at a constant rate with or without SatoshiDice. As far as I'm concerned, SatoshiDice can have their way with the Blockchain as long as there is insufficient transaction volume to fill every block out to one megabyte. And when transaction volume does cross this threshold, SatoshiDice will have to pay market rates for fees just like everyone else.

It's not a problem.

Normal use transactions have inputs and spendable outputs. Once they're spent, they can be completely forgotten (pruned). SatoshiDice is creating massive amounts of unspendable and unprunable transactions. They are not paying enough for the load they're putting on the network.

Now that sounds like a problem. WTF, SatoshiDice?!

I'm confused though, if the outputs of these SatoshiDice transactions are unspendable then why can't they be pruned? Is it because we need to keep them to detect a double spend?

Of course they are spendable, but the fees required to do so might be too high in a limited block size scenario.

Also they might be used as fees as far as I understand, but probably again even just including these with the intention to just give them to miners might be more expensive than the space they take up in the chain.

They can't be pruned because they CAN be spent with 0 fees and/or free transactions.

This for example:

http://blockchain.info/tx/ead4232286d7f843c6fd94fa13b12eaee810ca95dc26ffd3deae45a87ffb5d17

The transaction included five 0.00000001 inputs, but that cost the sender 0.0005 in fees (total fee 0.001 with toxic inputs, would have just been 0.0005 without them). There is economic incentive to leave them unspent forever. Make enough of those and you're practically undoing the 0.8 client upgrade back to 0.7 performance.

The toxic outputs can be spent, that's why they can never be discarded.

Not forever. Only until the exchange rate is such that they have non-negligible purchasing power.

Yes. So lets work on unlimited exchange rate, raise that high enough and raising other stuff will seem a whole lot less awful to stakeholders.

-MarkM-

Actually Gavin had advocated abandoning wallets swollen with coins that aren't currently economically significant.

By the time the exchange rate rises to the levels you are thinking of all the private keys will be forgotten, lost or abandoned.

The only hope is some sort of "admiralty rules" similar to "flotsam and jetsam" that will grant enterprising miners the salvage rights.

But that would be a hard fork, so...

By the time the exchange rate rises to those levels the amount of space those outputs require would be insignificant.

This is how I currently understand it:

1) Remove the block size limit.
2) Allow clients to set their preferred bandwidth.
3) Let the free market work between these two opposing preferences.  The more miners push the block size beyond the clients' collective capability to download it, the more they run the risk of being orphaned by a smaller block better suited to the current bandwidth.

This is a sentence that mathematicians call "indeterminate form of the type zero times infinity". In other words: not a good sign of logical deduction.

Unfortunately, this is what I'm mostly coming across.  Unsubstantiated claims and assumptions.