Increasing the block size is a good idea; 50%/year is probably too aggressive

[shorena]

-snip-
Does *anyone* have a record of the pool or waiting transactions?  That's our key.  When there are ~2,000 transactions in the queue waiting then we would expect a full 1MB block to be coming out next.  When there are ~4,000 transactions in the queue waiting then we would expect two full 1MB blocks to be coming out next.  In this state, transactions can expect to take ~20 minutes to confirm.  ~6,000 waiting -> 30 minute confirmation times.  And so on.
-snip-

I dont have historical data. But I just setup a rrdtool database to track the number of transactions on my full node. The stats for the last 24 hours are shown here [1], the pic is updated every 30 minutes and I will add more for 30 and 360 days once the database has enough data. As you can see from the little data thats allready there (collecting ~1 hour now) we are allready closer to 4000 transactions waiting than to 2000.
The raw data is gathered every minute with the following command

Code:

 bitcoind getrawmempool false | wc -l

and is not filtered in any way that is not inherent to bitcoind.

Code:

 bitcoind getrawmempool true | grep fee | grep 0.00000000 | wc -l

shows that right now 2792 of 3685 TX are without fee. I might make another database to improve the stats.

[1] http://213.165.91.169/pic/mempool24h.png

[Cubic Earth]

Nice work shorena!  I sent you a small tip to the address in your profile.  I checked out all the other stats on you full node, overall it is an awesome testament to Bitcoin's openness.

[painlord2k]

What did you see and where did you see it.
200Kt/d may be several years away, yes?
https://blockchain.info/charts/n-transactions?timespan=all&showDataPoints=false&daysAverageString=1&show_header=true&scale=0&address=
(If you are fond of extrapolating this might tell you something)

Or 200kt/d may be much sooner.
So flexible limits are probably better than "whatever" right?

Increase the MAX_BLOCKSIZE as soon as is reasonable.  20MB, 32MB, whatever.  Then enhance the code to segment blocks to exceed the API limit after that.

"The holiday season accounts for between 20 percent and 40 percent of typical retailers' total annual sales."
http://blogs.constantcontact.com/fresh-insights/holiday-shopping-stats/

Now, we have to hope there is just a doubling (the 20% case) of daily transactions during the next Holiday Season (November/December 2014).
Now, Bitcoin help limit expenditures, as it is a "deflationary currency" increasing its value over long time periods.
But, given we are about 80K transactions/day a doubling will be around 160K/day. And it will not be evenly distributed during the day. It will peak at Europe and America business hours and days.

Now, compared with the last year, we have a lot more and larger retailers online accepting BTCs and probably four/ten times brick and mortar places.

We could get long delays during this shopping season, not the next. And it would not be pretty.
The slowdown of the increase of the hash rate will not help, because part of the reason we saw smaller blocks in the past and larger now, it is just there are less blocks per day, something like 1/6 less blocks.
Hope no large miners have any problem during this season, because if the hashrate fall for some reason at critical time, the network could find less 25% less blocks for some hours increasing queue time and confirmations.
And remember good luck is blind but bad luck see you perfectly even in complete darkness. I would not like if miners, cause of bad luck, didn't found a block for an hour during peak shopping time.
And hope the "40% of retailers typical total annual sales" do not happen this year for bitcoin, because the network is in no way able to manage a similar load even without any cascading failure do add insults to injuries.

[David Rabahy]

I don't have historical data. But I just setup a rrdtool database to track the number of transactions on my full node. The stats for the last 24 hours are shown here [1], the pic is updated every 30 minutes and I will add more for 30 and 360 days once the database has enough data. As you can see from the little data that's allready there (collecting ~1 hour now) we are already closer to 4000 transactions waiting than to 2000.
The raw data is gathered every minute with the following command

Code:

 bitcoind getrawmempool false | wc -l

and is not filtered in any way that is not inherent to bitcoind.

Code:

 bitcoind getrawmempool true | grep fee | grep 0.00000000 | wc -l

shows that right now 2792 of 3685 TX are without fee. I might make another database to improve the stats.

[1] http://213.165.91.169/pic/mempool24h.png

Inspirational!

Starting from your spark I found https://blockchain.info/tx/e30a4add629882d360bc87ecc529733a9824d557690d1e5769453954ea4a1056.  It appears to be the oldest transaction waiting at this moment.  It was 31:34 old at the time of block #327136.  Block #326954 was the first block that could have added it to the block chain; 182 blocks ago.  One wonders how old the oldest transaction is that includes a fee.

[David Rabahy]

Windows 8.1
Satoshi v0.9.3.0-g40d2041-beta
example output from Debug console "getrawmempool true" command;

{
"000308b9c51a0ba76d57efd8897159d95b8278e4fc0e3cb480b3d15343a1aadd" : {
"size" : 374,
"fee" : 0.00010000,
"time" : 1414369834,
"height" : 327133,
"startingpriority" : 4976624.92307692,
"currentpriority" : 5160750.84553682,
"depends" : [
"60c66a89e247760aa4cb29517ba79bbb2bbe773823996135fc7035c74f8be171"
]
},
"00349a4799b7b787e9733f38fc01a8f5dc801f7e35e3071a706831395d67086e" : {
"size" : 520,
"fee" : 0.00000001,
"time" : 1414209735,
"height" : 326867,
"startingpriority" : 40.33333333,
"currentpriority" : 10311.10448718,
"depends" : [
"75ba09c16b35b3495a7d829030dbafbed4e8e6806c8bc58207f8472e85749187"
]
},
...
}

DOS batch file to collapse output so that each transactions ends up on a single line (good for feeding into Excel);

@echo off
Setlocal EnableDelayedExpansion
SET new_line=
FOR /F "delims=" %%l IN (raw.txt) DO (
  if "%%l" == "}," (
    echo !new_line!
    SET new_line=
  ) ELSE (
    SET new_line=!new_line! %%l
  )
)

To invoke;

C:\bitcoin>collapse >btc_txn.txt

[shorena]

Nice work shorena!  I sent you a small tip to the address in your profile.  I checked out all the other stats on you full node, overall it is an awesome testament to Bitcoin's openness.

Thanks, I appreciate it. Pictures are indeed on the main page [1], but I thought I just post the picture directly. Less scrolling involved.

-snp-
Inspirational!

Starting from your spark I found https://blockchain.info/tx/e30a4add629882d360bc87ecc529733a9824d557690d1e5769453954ea4a1056.  It appears to be the oldest transaction waiting at this moment.  It was 31:34 old at the time of block #327136.  Block #326954 was the first block that could have added it to the block chain; 182 blocks ago.  One wonders how old the oldest transaction is that includes a fee.

Glad you like it. I think the long queue is mainly due to transactions without fee and miners beeing greedy. The TX in question has small (in BTC value) input/outputs, does not pay a fee and the input is not very old, which in my experience results in miners just ignoring your TX. There are plenty of TX with fees, so why should they confirm this one? I had a similar one for testing with 4 inputs and a single output of 0.0022. I didnt confirm in 8 days - by that time the inputs were 2-3 weeks old - so I had to remove it from core and "doublespend" it. I am not sure this would change with a bigger blocksize as most blocks are not full yet, even though they could be. They way the mempool currently is each block should be at the limit now, but they are not [2]. Maybe someone that operates a pool can shed some light on this.


[1] http://213.165.91.169/
[2] https://www.blocktrail.com/BTC/block-all/1 - you can sort by pool by clicking on its name, but I have not found one that has exclusivly big blocks.

[-ck]

Maybe someone that operates a pool can shed some light on this.

A significant number of pools simply run the default bitcoind client and transaction rules. Some have their own rules claiming their rules are "anti-spam" (see the many threads about luke-jr's custom patches to the bitcoind client in the gentoo packaging, and probably any pool running his pool software does so too). Most of the alleged anti-spam selection criteria are (presumably an ideological objection) aimed at gambling sites.

[David Rabahy]

[2] https://www.blocktrail.com/BTC/block-all/1 - you can sort by pool by clicking on its name, but I have not found one that has exclusivly big blocks.

Based on this, for example, Polmine tends strongly toward smaller blocks.  Meanwhile, DiscusFish/F2Pool does a much better job of producing bigger blocks.

[NewLiberty]

This is the sort of fundamental analysis that would also get us closer to having a client that can tell its user how much TX fee to include in order to expect the transaction to be confirmed in X minutes.
A nice feature to have.

[shorena]

Maybe someone that operates a pool can shed some light on this.

A significant number of pools simply run the default bitcoind client and transaction rules. Some have their own rules claiming their rules are "anti-spam" (see the many threads about luke-jr's custom patches to the bitcoind client in the gentoo packaging, and probably any pool running his pool software does so too). Most of the alleged anti-spam selection criteria are (presumably an ideological objection) aimed at gambling sites.

Not sure I missed something when I read the source (or rather the comments). The way I understand it is that the TXs get sorted by priority and added. So with now over 5k TX in queue all blocks should be close to the limit, unless there is some sort of minimal priority I missed. Currenlty 328 / 1424 TX have a priority of 0.00000000

I made a 2nd database this morning to get data for TX with and without fee. The load was to much for the server to handle [1] so I had to reboot. Thus the data after 21:00 is not significant for now.

I think the message is clear anyway. The majority of TX without fee get ignored even though there is space left in the blocks. Transactions without fee queued up to over 2000 waiting 3 times in the last 24 hours. The transactions without fee might be spam though. Maybe David Rabahy can share some Excel results?



up to date pictures are on the nodes info page [2] below the connection and traffic stats. I might change the order though.

[1] the white spaces in the pictures indicate that the task to update the database could not be handled within 30 seconds and was terminated. The way rrdtool works is that it handles missing values as "unknown" which are shown as blanks in the graphics.
[2] http://213.165.91.169/

[NewLiberty]

The cost is not that significant.  Heck, the whole BTC market cap is not that significant.

If there were 6 GB block size bloat per hour?
A financial attack could do this independently.
Miners could do this free-ish.
Small miners would fail, as would all hobby miners.

Full nodes would become centralized, increased 51% risks, etc.
These are just the obvious.  No more decentralisation for Bitcoin.

From the wiki:

Note that a typical transaction is 500 bytes, so the typical transaction fee for low-priority transactions is 0.1 mBTC (0.0001 BTC), regardless of the number of bitcoins sent.

To spam the 1 MB blocksize takes roughly .2 BTC per block, or 1.2 BTC per hour. That's only $500 per hour.

To spam a 1 GB blocksize takes roughly 200 BTC per block, or 1200 BTC per hour. That's $500,000 per hour!

A 1 GB blocksize is far more costly to attack. We could increase the blocksize to 1 GB now and nothing would happen because there aren't that many transactions to fill such blocks.

I didn't see if this was addressed elsewhere already...
The projected attack would come from a mining concern that is looking to shut out smaller players and consolidate their mining regime.
The cost of the attack is the marginal cost of a winning block being orphaned.  The transaction fee is paid by and to the attacker, at no cost.

However if it is not orphaned, the reward is significant.  While the block is being downloaded and verified by lower bandwidth nodes, the "attacker" is already at work on the next block, and with some decreased competition has some advantages.  It is essentially a denial of service from larger bandwidth miner to lesser bandwidth miner.

[TierNolan]

What is the reason to stop at 20MB?  That just seems to be pushing things further into the future to make the decision.

It does mean that the 32MB message size limit wouldn't be a problem though.

[NewLiberty]

What is the reason to stop at 20MB?  That just seems to be pushing things further into the future to make the decision.

It does mean that the 32MB message size limit wouldn't be a problem though.

Under Gavin's 2nd proposal, (starting at 20MB and +40% per year) >32MB is 2 years out.

Most blocks are about 1/3 MB, so the proposal is more or less for about 100x current utilization 2 years forward.

[TierNolan]

Under Gavin's 2nd proposal, (starting at 20MB and +40% per year) >32MB is 2 years out.

I misunderstood, I thought it was +40% and stopping at 20MB.