Gold collapsing. Bitcoin UP.

[iCEBREAKER]

Does CreateNewBlock currently take longer to execute if there are more TXs in a miner's mempool to pick from?  If so, this would add credence to Cypherdoc's hunch that miner's are producing more empty blocks when mempool swells.  

Yep, I already pointed that out to you specifically! It's superlinear in the mempool size (well, ignoring caching)  But thats unrelated to f2pool/antpool and the other SPV miners, as they're not ever calling createnewblock in that case, as they're mining without even validating.   One can mine on a validated chain with no transactions while waiting for createnewblock (which is what eligius does, for example).  I also pointed out that this is trivially optimizable, but no one has bothered previously.

Wait, how is an empty block created without calling createnewblock?  Is there a createemptyblock thingy or what?

[Peter R]

OK.  Provide me with your estimates for the following (and explain how you arrived at your numbers) and I'll update my table using your numbers:
1.  The cost per node to store 1 GB of additional blockchain data for 5 years, assume the outputs are spent.
2.  The cost per node to store 1 GB of additional blockchain data for 5 years, assuming the outputs are unspent.

I may be missing the context as this thread is high volume and I've not read any of the backlog...

But for a full verifying node, the on-going cost cost of 1GB of additional transactions with all outputs spent is 0; all the cost related to that 1GB of data is related to the bandwidth to get it to you and the verification cost, and for short term storage until its burried, after that it need not be stored.
The cost for unspent is some non-zero number which depends on your estimation of storage costs.

This thread can be hard to follow if you're not following it all the time!  

The question was in reference to a debate I was having with Odalv about these "order of magnitude" estimates shown in this table.  I was suggesting that, under the conditions considered in the table, it is cheaper for miners to write the spam to the Blockchain and more costly for the spammer, than continually rejecting it:

Does CreateNewBlock currently take longer to execute if there are more TXs in a miner's mempool to pick from?  If so, this would add credence to Cypherdoc's hunch that miner's are producing more empty blocks when mempool swells.  

Yep, I already pointed that out to you specifically! It's superlinear in the mempool size (well, ignoring caching)  But thats unrelated to f2pool/antpool and the other SPV miners, as they're not ever calling createnewblock in that case, as they're mining without even validating.   One can mine on a validated chain with no transactions while waiting for createnewblock (which is what eligius does, for example).  

Sorry, yes I know you explained that.  The point I'm trying to make is that if CreateNewBlock is super-linear in mempool size, then it would not be surprising to see more empty blocks (what Cypher was calling "defensive blocks") when mempool swells (the miners are mining on an empty block for longer while waiting for CreateNewBlock to finish).  This was Cypher's point from the very beginning that many people, including myself, were suggesting was probably not the case!  

Furthermore, how can f2pool/antpool mine a non-empty block without calling createnewblock?

[Adrian-x]

Spammer only spends money 1x during lifetime, but miner has to keep bigger disk-space forever (2 hard disc consume more electricity than single one)

OK.  Provide me with your estimates for the following (and explain how you arrived at your numbers) and I'll update my table using your numbers:

1.  The cost per node to store 1 GB of additional blockchain data for 5 years, assume the outputs are spent.
2.  The cost per node to store 1 GB of additional blockchain data for 5 years, assuming the outputs are unspent.

I'm sorry, I have no estimates. I know that bloat-chain is not solution  There are better solutions how to organize blockchain.

My full node just crashed ... all data are lost ... downloading 6 years 25 weeks.  ... hmm I hope that it is only me and there will remain some copy of blockchain. :-)  (not joking)

your data management systems need updating if you lost the data. Bitcoin is a $3B network and if you dont have an invested interest in keeping the blockchain to prove you own X coin or care enough to manage the responsibility you shouldn't be dictating how we should use it. not everyone who can't backup data or doesn't want to should be running a node. I have friends who dont trust digital photos because they trust paper copies more. I'm surprised to find you fit the profile.

I also think you're not weighting the cost and benefits correctly, the cost of 2 hard disc consuming more electricity than single one, is an inconsequential cost at the scale of executing this same attack with an 8MB block. (hell its inconsequential at the cost of the attack on the 1MB block.)

limiting block space is not going to save anyone except you a little bandwidth because you dont back up.

[solex]

I just want to highlight the comments here from bitcoin-dev today, which seem to indicate that these spamming attacks would have been 5.5x more expensive if a dust threshold software change had not been made. Perhaps there are deeper insights someone else has on this...?

16:53   wangchun   why can those spam get confirmed. 0.00001 BTC vout below dust threshold right?
16:54   phantomcircuit   wangchun, iirc the dust threshold is 546 satoshis
16:54   wangchun   not 5460 satoshis? changed?
16:54   aschildbach   wangchun: Yes it was cut by 10 a few months ago.

[cypherdoc]

The point I'm trying to make is that if CreateNewBlock is super-linear in mempool size, then it would not be surprising to see more empty blocks (what Cypher was calling "defensive blocks") when mempool swells (the miners are mining on an empty block for longer while waiting for CreateNewBlock to finish).  This was Cypher's point from the very beginning that many people, including myself, were suggesting was probably not the case!  

that is exactly what i was saying w/o knowing anything about CNB.  it was just intuitive that constructing a new block would take longer for a large mempool so i deduced that a miner might decide "f*ck it" that he doesn't want to not only spend additional time validating a large incoming block but also not want to spend additional time creating a large new block so instead just automatically switch to 0 block mining upon arrival of a large block.  the thing none of us anticipated was that he would not validate that incoming large block at all and switch to SPV mining while >50% of the network would also be SPV mining (Discus, Antpool, BTCGuild) and trigger an invalid SPV longest chain.

that's some wild shit going on.

given this newly recognized scenario, i still conclude however that this is all a result of the block size limit being hit continuously by attacking spammers and secondarily intentionally causing a bloated mempool which then causes these unanticipated defensive strategies and more intentionally, user disruption.

[cypherdoc]

I just want to highlight the comments here from bitcoin-dev today, which seem to indicate that these spamming attacks would have been 5.5x more expensive if a dust threshold software change had not been made. Perhaps there are deeper insights someone else has on this...?

16:53   wangchun   why can those spam get confirmed. 0.00001 BTC vout below dust threshold right?
16:54   phantomcircuit   wangchun, iirc the dust threshold is 546 satoshis
16:54   wangchun   not 5460 satoshis? changed?
16:54   aschildbach   wangchun: Yes it was cut by 10 a few months ago.

i don't think it really matters, does it?

since most of the mined blocks are filled with real demand trying to get into blocks with or without spamming, the only difference being that with spamming the real demand has to pay significantly higher fees to get in, the cost to fill up the rest of the block with spam is minimal as is the minimal cost to bloat the mempool to extraordinary levels (94K 0confs as i write).  i say minimal cost for the mempool b/c most of it gets turned over after 24h deletion allowing that spam to be recycled w/o having to pay.

[cypherdoc]

all the yearly silver and mining stocks have that sick top left to bottom right chart configuration. this is a horrible leading indicator for both gold and stocks:

silver futures:



SLW:



PAAS:



silverbox's GPL:

[TPTB_need_war]

When miner's mempool exceeds block size by some backlog factor, they need to send out a message to the peers they are connected to saying "don't propagate transactions to me with smaller than X fee per kilobyte until further notice". This will mitigate the impact that spam-that-can-never-be-added-to-a-block can have on the network, but it won't entirely eliminate it, because as the mempool backlog reduces, the miner will need to back off the notice to some unknown lower fee threshold. The miner could guess but it will never be 100% accurate. So spam will get verified and rejected. But perhaps those estimates can be close enough that the losses the spammers takes in fees for transactions it sends that do get into a block will be greater than the gains it values from effective spamming (what ever those gains are  ). Appears to me the issue can be fixed.

Afaik, the reason incoming transactions are verified before it is known whether they will be in a new block, is because it is considered spam to propagate (send out again to another peer) invalid transactions that would be rejected and never appear in a block.

[cypherdoc]

When miner's mempool exceeds block size by some backlog factor, they need to send out a message to the peers they are connected to saying "don't propagate transactions to me with smaller than X fee per kilobyte until further notice". This will mitigate the impact that spam-that-can-never-be-added-to-a-block can have on the network, but it won't entirely eliminate it, because as the mempool backlog reduces, the miner will need to back off the notice to some unknown lower fee threshold. The miner could guess but it will never be 100% accurate. So spam will get verified and rejected. But perhaps those estimates can be close enough that the losses the spammers takes in fees for transactions it sends that do get into a block will be greater than the gains it values from effective spamming (what ever those gains are  ). Appears to me the issue can be fixed.

Afaik, the reason incoming transactions are verified before it is known whether they will be in a new block, is because it is considered spam to propagate (send out again to another peer) invalid transactions that would be rejected and never appear in a block.

gmax already told us to use minrelaytxfee.

[tvbcof]

Spammer only spends money 1x during lifetime, but miner has to keep bigger disk-space forever (2 hard disc consume more electricity than single one)

OK.  Provide me with your estimates for the following (and explain how you arrived at your numbers) and I'll update my table using your numbers:

1.  The cost per node to store 1 GB of additional blockchain data for 5 years, assume the outputs are spent.
2.  The cost per node to store 1 GB of additional blockchain data for 5 years, assuming the outputs are unspent.

I'm sorry, I have no estimates. I know that bloat-chain is not solution  There are better solutions how to organize blockchain.

My full node just crashed ... all data are lost ... downloading 6 years 25 weeks.  ... hmm I hope that it is only me and there will remain some copy of blockchain. :-)  (not joking)

Don't worry.  According to Mike Hearn Bitcoin can survive just fine with 4 or 6 (forgot which) copies of the blockchain worldwide.

[domob]

Yes, but the ongoing spam shows clearly that all full nodes are handling the traffic just fine. We were told that they would crash and burn from overloaded memory. Not true.

Wait, wasn't that exactly Hearn's main argument in his "the sky is falling if we don't increase the blocksize" blog post?  So why do you see the fact that it did not prove true so far as supporting your position of increasing the block size?

[Peter R]

Yes, but the ongoing spam shows clearly that all full nodes are handling the traffic just fine. We were told that they would crash and burn from overloaded memory. Not true.

Wait, wasn't that exactly Hearn's main argument in his "the sky is falling if we don't increase the blocksize" blog post?  So why do you see the fact that it did not prove true so far as supporting your position of increasing the block size?

It provided evidence that the network can handle the increased load, and it confirmed that people do get frustrated by the confirmation delays. 

[sidhujag]

Let's face it If we want adoption by a cb we need more tx/s however we can solve other probs first before raising limit to efficiently come to that conclusion.

[gmaxwell]

I just want to highlight the comments here from bitcoin-dev today, which seem to indicate that these spamming attacks would have been 5.5x more expensive if a dust threshold software change had not been made. Perhaps there are deeper insights someone else has on this...?

16:53   wangchun   why can those spam get confirmed. 0.00001 BTC vout below dust threshold right?
16:54   phantomcircuit   wangchun, iirc the dust threshold is 546 satoshis
16:54   wangchun   not 5460 satoshis? changed?
16:54   aschildbach   wangchun: Yes it was cut by 10 a few months ago.

i don't think it really matters, does it?
since most of the mined blocks are filled with real demand trying to get into blocks with or without spamming,

Unfortunately, the majority in the blocks I checked earlier today have been the DOS attack-- e.g. transactions tracable from outputs of this transaction https://blockchain.info/tx/3bad15167c60de483cd32cb990d1e46f0a0d8ab380e3fc1cace01afc9c1bb5af  and a few others. ... though the attack style has been shifting to evade filtering by miners.

The change mentioned in the chat above is https://github.com/bitcoin/bitcoin/pull/3305 (you might find the comments there interesting), it's one of Mikes couple contributions to Bitcoin Core.

[awemany]

Unfortunately, the majority in the blocks I checked earlier today have been the DOS attack-- e.g. transactions tracable from outputs of this transaction https://blockchain.info/tx/3bad15167c60de483cd32cb990d1e46f0a0d8ab380e3fc1cace01afc9c1bb5af  and a few others. ... though the attack style has been shifting to evade filtering by miners.

The change mentioned in the chat above is https://github.com/bitcoin/bitcoin/pull/3305 (you might find the comments there interesting), it's one of Mikes couple contributions to Bitcoin Core.

So miners are filtering out the spam? What are they filtering for?

[sickpig]

Jeff Garzik has just posted 2 pull requests for bitcoin core on github:

https://github.com/bitcoin/bitcoin/pull/6405
Remove TX priority and free transaction area from mempool, block creator.

https://github.com/bitcoin/bitcoin/pull/6402
Floating network relay fee increase, if memory pool grows too large.

the latter will introduce, if merged, "a relay fee that adjusts to floods
which cause the memory pool to grow too large".

It seems that core devs start becoming aware of the lack of
of economic incentives for services provided by full nodes.

[solex]

Unfortunately, the majority in the blocks I checked earlier today have been the DOS attack-- e.g. transactions tracable from outputs of this transaction https://blockchain.info/tx/3bad15167c60de483cd32cb990d1e46f0a0d8ab380e3fc1cace01afc9c1bb5af  and a few others. ... though the attack style has been shifting to evade filtering by miners.

The change mentioned in the chat above is https://github.com/bitcoin/bitcoin/pull/3305 (you might find the comments there interesting), it's one of Mikes couple contributions to Bitcoin Core.

Goddammit, Peter Todd hits the nail on the head! Kudos to him.
Yes, that 10x reduction looks to be unwise in hindsight. I recall the painful and protracted debate on bct just getting the dust threshold at 5460 sats in the first place.

So miners are filtering out the spam? What are they filtering for?

Hey, nice to see you in this thread.
It appeared that BitFury was aggressively filtering by fee when producing small 138KB blocks, however they have just mined a 998KB block (364549) which I think is their first near this size.

[TheRealSteve]

It appeared that BitFury was aggressively filtering by fee when producing small 138KB blocks, however they have just mined a 998KB block (364549) which I think is their first near this size.

Nah, there were also 364354, 364383, 364393 and 364398

[awemany]

So miners are filtering out the spam? What are they filtering for?

Hey, nice to see you in this thread.
It appeared that BitFury was aggressively filtering by fee when producing small 138KB blocks, however they have just mined a 998KB block (364549) which I think is their first near this size.

Hi! Yeah, I was always somewhat put off by the presentation here, too confusing, threads with too many pages. Have a couple of old accounts that I lost all login data for. On the other hand, Reddit is sometimes a bit shallow and fast scrolling...

So they did increase the fees then and not reduce their blocksize, it seems? That would make a lot more sense to me (from the miner's perspective).

[newb4now]

In the short run, miners will keep putting up with spam transactions because most of their revenue (great majority) is coming from the BTC block reward. As rewards become less and less and transaction volume (spam or legitimate) continues to increase the balance will gradually start to change.

Eventually miners will start to ignore transactions without some minimum transaction fee (which will change over time based on block reward/transaction fee balance).

This is a self correcting problem which miners will correct on their own (by only verifying transactions with some minimum fee)

We all expected this from the beginning. The blockchain will eventually be supported by transactions fees as block reward keeps dropping. The only thing that has changed is that miners may respond sooner because of all these spam transactions. Spam will stop once the miners raise the minimum fee enough (or spam transactions will continue but never be confirmed)