Ultimate Bitcoin Stress Test - Monday June 22nd - 13:00 GMT

[Quickseller]

I think the fee should be a function of the number of unconfirmed tx at the time of sending. Would this work? Clients need to be more intelligent.

I don't think this is necessary. The number of transactions per hour that the Bitcoin network will process is not going to vary widely on a hour to hour basis, it will also not change very much on a day to day basis, and probably not even on a week to week basis. The reason why the Bitcoin network will process roughly the same number of transactions per hour throughout the day is because of it's global nature - a peak spending period in one part of the world is going to be a period when most people are asleep (and otherwise spending very few transactions) in another part of the world.

Economies of scale are not going to change very much on a short term basis. People are not going to wake up one day en masse and decide to start using bitcoin for everything they do. People will start to learn about the benefits of using Bitcoin, will use it for a very small number of purchases, tell their friend about it, and over time use it for a greater percentage of their overall spending. The time from when someone does not own any bitcoin to when they spend it whenever they can will be, on average several months, if not longer.

Sure the Bitcoin network will have spikes in transactions for a variety of reasons, although such spikes are, IMO, unlikely to cause great delays in having transactions confirmed, especially delays of more then 6 or 7 blocks.

I would argue that a better solution would be to have a client's default fee policy to be changed (if necessary) each time a new version is released.

[1714721855]

1714721855

[1714721855]

1714721855

[1714721855]

1714721855

[1714721855]

1714721855

[1714721855]

1714721855

[1714721855]

1714721855

[unamis76]

On monday, CoinWallet.eu conducted a stress test of the Bitcoin blockchain. Not only was our plan to see the outcome, but also to see how easy it would be for a malicious entity or government to create havoc for the Bitcoin community. As you will see from the analysis below, delayed transactions are not the only issue that Bitcoin users experienced.

Surprisingly, executing tens of thousands of transactions that correctly propagate to the network simultaneously is not as easy as we had expected. One of the methods that was used to increase the kb size of our transactions was to send transactions consisting of numerous small outputs (usually 0.0001) to make a single transaction of 0.01. A simple transaction is usually 225-500 bytes, while many of our transactions were 18 kb (A number which limits the blockchain to 5 transactions per minute). In our preliminary testing this was effective, however in practice it caused our servers to crash. Throughout the day and evening, our strategy and methodology changed multiple times.

Initially the plan was to spend 20 BTC on transaction fees to flood the network with as many transactions as possible. Due to technical complications the test was concluded early, with less than 2 BTC spent on fees. The events of yesterday were accomplished with less than €500.

Timeline

    11:57 GMT - Transaction servers initiated. Thousands of 700 kb transactions completed within the first 20 minutes. Transactions were used to break coins into small 0.0001 outputs.
    12:30 GMT - Servers begin sending larger 18kb transactions.
    14:10 GMT - Mempool size increases dramatically. Blockchain.info breaks.
    14:20 GMT - Our servers begin to crash. It becomes apparent that BitcoinD is not well suited to crafting transactions of this size.
    14:30 GMT - Our test transactions are halted while alternate solutions are created. The mempool is at 12 mb.
    17:00 GMT - Alternate transaction sending methods are started. Servers are rebooted. Mempool has fallen to 4mb.
    21:00 GMT - The stress test is stronger than ever. Mempool reaches 15 mb and more than 14000 transactions are backlogged. The situation is made worse by F2Pool selfishly mining two 0kb blocks in a row.
    23:59 GMT - 12 hours after starting, the test is concluded. Less than 2 BTC (€434) is spent on the test in total.

The following graph depicts the entire test from start to finish: https://anduck.net/bitcoin/mempool.png

Observations

Delayed confirmation times and large mempool buildups were not the only observation that came from our testing. Many more services were impacted than we had initially envisioned.

Blockchain.info

Over the past few months, Blockchain.info has become increasingly unreliable, however we are confident that yesterday's stress test had an impact on their website being offline or broken for 1/3 of the day. During periods where we sent excessive transactions, Blockchain.info consistently froze. It appeared as though their nodes were overwhelmed and simply crashed. Each time this occurred, the site would re-emerge 10-30 minutes later only to fail again shortly thereafter. Users of the blockchain wallet were unable to send transactions, login or even view balances during the downtimes. In response to our heavy Bitcoin usage, blockchain.info began to exclude certain transactions from their block explorer. This issue is explored further by the creators of Multibit, who can confirm that some transactions sent from their software were ignored by Blockchain, but were picked up by Blockr.

Bitcoin ATMs

Many ATMs operate as full nodes, however some ATMs rely on third party wallet services to send and receive transactions. The most prominent Bitcoin ATM of this type is Lamassu, which uses the blockchain.info API to push outgoing transactions from a blockchain.info wallet. Due to the blockchain.info issues, all Lamassu ATMs that use blockchain.info's wallet service were unavailable for the day.

MultiBit

Both versions of MultiBit suffered delayed transactions due to the test. Gary and Jim from MultiBit have created a full analysis from Multibit's perspective which can be read at https://multibit.org/blog/2015/06/23/bitcoin-network-stress-test.html

The outcome was that transactions with the standard fee in Multibit HD took as many as 80 blocks to confirm (Approximately 13 hours). Standard 10000 satoshi fee transactions took an average of 9 blocks to get confirmed. Multibit has stated that they will be making modifications to the software to better cope with this type of event in the future.

Tradeblock

With Blockchain.info broken, we frequently referred to Tradeblock to track the backlog. Unfortunately Tradeblock was less than perfectly reliable and often failed to update when a new block had been mined. Regardless, at one point 15,000 unconfirmed transactions were outstanding.

Bitpay

Users reported issues with Bitpay not recognizing transactions during the test.

Price

Increase of $2. Contrary to some predictions, we did not short Bitcoin.

Green Address

While this app was not hindered directly by our test, we did send a series of 0.001 payments to a green address wallet. When attempting to craft a transaction from the wallet, an error occurs stating that it is too large. It appears that the coins that were sent to this wallet may be lost.

Conclusions

From a technical perspective, the test was not a success. Our goal of creating a 200mb backlog was not achieved due to miscalculations and challenges with pushing the number of transactions that we had desired. We believe that future tests may easily achieve this goal by learning from our mistakes. There are also numerous vulnerable services that could be exploited as part of a test, including Bitcoin casinos, on-chain gambling websites, wallets (Coinbase specifically pointed out that a malicious user could take advantage of their hosted wallet to contribute to the flooding), exchanges, and many others. Users could also contribute by sending small amounts to common brain wallets.

We also learned that the situation could have been made worse by sending transactions with larger fees. We sent all transactions with the standard fee of 10000 satoshis per kb. If we had sent with 20000 satoshis per kb, normal transactions would have experienced larger delays. In our future stress tests, these lessons will be used to maximize the impact.

Thank you for your test and insights about what happened. You clearly showed how easy it is to cripple Bitcoin and make it move way slower. Since you very much proven your point, I don't see another test as necessary. It would only cost you money, time and patience to prove something that you already managed to prove really well.

[tspacepilot]

I think the fee should be a function of the number of unconfirmed tx at the time of sending. Would this work? Clients need to be more intelligent.

I don't think this is necessary. The number of transactions per hour that the Bitcoin network will process is not going to vary widely on a hour to hour basis, it will also not change very much on a day to day basis, and probably not even on a week to week basis. The reason why the Bitcoin network will process roughly the same number of transactions per hour throughout the day is because of it's global nature - a peak spending period in one part of the world is going to be a period when most people are asleep (and otherwise spending very few transactions) in another part of the world.

Economies of scale are not going to change very much on a short term basis. People are not going to wake up one day en masse and decide to start using bitcoin for everything they do. People will start to learn about the benefits of using Bitcoin, will use it for a very small number of purchases, tell their friend about it, and over time use it for a greater percentage of their overall spending. The time from when someone does not own any bitcoin to when they spend it whenever they can will be, on average several months, if not longer.

Sure the Bitcoin network will have spikes in transactions for a variety of reasons, although such spikes are, IMO, unlikely to cause great delays in having transactions confirmed, especially delays of more then 6 or 7 blocks.

I would argue that a better solution would be to have a client's default fee policy to be changed (if necessary) each time a new version is released.

But, Quickseller, your argument is clearly incorrect given the recent stress test.  It was very clear that the number of backlogged transactions increased drammatically during a short period of time and if clients took the backlog into account when suggesting a fee, how could that be anything but an improvement?   All clients should allow their users to set their own fee at the end of the day, but suggesting a fee based on more information (rather than less) when that information is freely available to any bitcoin node is clearly a good idea.

[findftp]

I don't see another test as necessary. It would only cost you money, time and patience to prove something that you already managed to prove really well.

Please spam the hell out of bitcoin.
Let it crumble.

I did not join this thing to find out it to be a cripple old man in 4 years.
I want it to be attacked and spammed to the max.

bring it

[Quickseller]

I think the fee should be a function of the number of unconfirmed tx at the time of sending. Would this work? Clients need to be more intelligent.

I don't think this is necessary. The number of transactions per hour that the Bitcoin network will process is not going to vary widely on a hour to hour basis, it will also not change very much on a day to day basis, and probably not even on a week to week basis. The reason why the Bitcoin network will process roughly the same number of transactions per hour throughout the day is because of it's global nature - a peak spending period in one part of the world is going to be a period when most people are asleep (and otherwise spending very few transactions) in another part of the world.

Economies of scale are not going to change very much on a short term basis. People are not going to wake up one day en masse and decide to start using bitcoin for everything they do. People will start to learn about the benefits of using Bitcoin, will use it for a very small number of purchases, tell their friend about it, and over time use it for a greater percentage of their overall spending. The time from when someone does not own any bitcoin to when they spend it whenever they can will be, on average several months, if not longer.

Sure the Bitcoin network will have spikes in transactions for a variety of reasons, although such spikes are, IMO, unlikely to cause great delays in having transactions confirmed, especially delays of more then 6 or 7 blocks.

I would argue that a better solution would be to have a client's default fee policy to be changed (if necessary) each time a new version is released.

But, Quickseller, your argument is clearly incorrect given the recent stress test.  It was very clear that the number of backlogged transactions increased drammatically during a short period of time and if clients took the backlog into account when suggesting a fee, how could that be anything but an improvement?   All clients should allow their users to set their own fee at the end of the day, but suggesting a fee based on more information (rather than less) when that information is freely available to any bitcoin node is clearly a good idea.

Sure it is possible for the mempool (aka the backlog of transactions) to grow dramatically in a short period of time. However the latest instance of this happening was not the result of anyone making a rational economic decision. The backlog of transactions was artificial and over the long run it is unlikely that these kind of blimps will show up, nor will be sustained over more then several blocks.

The devs of the various clients have limited resources and I don't think this is an overall good think to invest limited resources into. If you want to invest your own money/resources into creating such a client, then you are free to do so.

Even if you were to ignore my conclusion that the number of transactions per hour does not vary widely over short periods of time, it would really not be a viable feature to have your client estimate the required fee to get a transaction confirmed quickly for a number of reasons.

1 - The various pools, and miners (when solo mining) ultimately create their own policy as to which transactions they confirm when they find a block. This policy could be to include no transactions, only transactions that have been propagated and meet certain criteria, transactions originated by themselves that have not been previously propagated, transactions provided to them by entities that have pre-existing arrangements with them that may or may not have been previously propagated, among a wide range of other potential criteria.

I would not at all be surprised if a number of entities that push a large number of transactions to the network have agreements with various pools that guarantee their transactions be confirmed in blocks found by those pools if they are not already confirmed.

The policy of each pool has the potential to be, and likely is different from other pools.

2 - Blocks are found, on average, once every 10 minutes, but are often found less frequently. It would only take seconds for the mempool (and thus the likely required fee to have your transaction quickly confirmed) to have grown in size dramatically (someone could potentially push several thousand valid transactions to the network all at once). This means that someone could push a transaction with a fee that their client claims will cause it to be quickly confirmed at time n, then at time n+1, someone pushes 10,000 transactions to the network, many with "better" fees, and before the next block is confirmed after time n. This would create a false sense of a guarantee to the end user.

3 - Over the long run, the majority of end users are not going to run clients that are full nodes to spend their bitcoin. This means that they will need to place a certain level of trust in other full nodes when deciding what level of fees to include. This is trust that probably shouldn't be given. Today, if full nodes give incomplete information about unconfirmed transactions or about the blockchain, then any node that it connected it to will get that information from someone else. If that full node gives information about invalid transactions, then such invalid transactions will be ignored by everyone else.

As it is today, the mempool of every full node is potentially different, and often is going to be different from many other full nodes. The node that your client gets the size of it's mempool from may not be an accurate reflection of the network. It is also possible that a pool could have a large number of nodes that artificially increase the reported size of their mempool in order to encourage people to include larger tx fees then is necessary in an effort to increase the overall tx fees they receive. Requiring a node to prove the size of their mempool would open up a whole new can of worms that would simply not be worth it

[tspacepilot]

I think the fee should be a function of the number of unconfirmed tx at the time of sending. Would this work? Clients need to be more intelligent.

I don't think this is necessary. The number of transactions per hour that the Bitcoin network will process is not going to vary widely on a hour to hour basis, it will also not change very much on a day to day basis, and probably not even on a week to week basis. The reason why the Bitcoin network will process roughly the same number of transactions per hour throughout the day is because of it's global nature - a peak spending period in one part of the world is going to be a period when most people are asleep (and otherwise spending very few transactions) in another part of the world.

Economies of scale are not going to change very much on a short term basis. People are not going to wake up one day en masse and decide to start using bitcoin for everything they do. People will start to learn about the benefits of using Bitcoin, will use it for a very small number of purchases, tell their friend about it, and over time use it for a greater percentage of their overall spending. The time from when someone does not own any bitcoin to when they spend it whenever they can will be, on average several months, if not longer.

Sure the Bitcoin network will have spikes in transactions for a variety of reasons, although such spikes are, IMO, unlikely to cause great delays in having transactions confirmed, especially delays of more then 6 or 7 blocks.

I would argue that a better solution would be to have a client's default fee policy to be changed (if necessary) each time a new version is released.

But, Quickseller, your argument is clearly incorrect given the recent stress test.  It was very clear that the number of backlogged transactions increased drammatically during a short period of time and if clients took the backlog into account when suggesting a fee, how could that be anything but an improvement?   All clients should allow their users to set their own fee at the end of the day, but suggesting a fee based on more information (rather than less) when that information is freely available to any bitcoin node is clearly a good idea.

Sure it is possible for the mempool (aka the backlog of transactions) to grow dramatically in a short period of time. However the latest instance of this happening was not the result of anyone making a rational economic decision. The backlog of transactions was artificial and over the long run it is unlikely that these kind of blimps will show up, nor will be sustained over more then several blocks.

You can call it "artificial" or "irrational" but its occurrance was an empirical fact.  To suggest that something which just, in fact, happened, doesn't happen is like covering your eyes and saying 'see no evil'.

The devs of the various clients have limited resources and I don't think this is an overall good think to invest limited resources into. If you want to invest your own money/resources into creating such a client, then you are free to do so.

Of course I am free, and I thank you for reminding me of it.  I guess.  Anyway, what we're talking about here is essentially a one-liner in code.  And yes, I may indeed fork the wallets I use in order to implement it.  I can't see any rational reason not to.

Even if you were to ignore my conclusion that the number of transactions per hour does not vary widely over short periods of time, it would really not be a viable feature to have your client estimate the required fee to get a transaction confirmed quickly for a number of reasons.

1 - The various pools, and miners (when solo mining) ultimately create their own policy as to which transactions they confirm when they find a block. This policy could be to include no transactions, only transactions that have been propagated and meet certain criteria, transactions originated by themselves that have not been previously propagated, transactions provided to them by entities that have pre-existing arrangements with them that may or may not have been previously propagated, among a wide range of other potential criteria.

I would not at all be surprised if a number of entities that push a large number of transactions to the network have agreements with various pools that guarantee their transactions be confirmed in blocks found by those pools if they are not already confirmed.

The policy of each pool has the potential to be, and likely is different from other pools.

Each policy can be different but they all follow the rule of preferring transactions with greater fees.  You seem to be forgetting the fact that you don't need to offer any certainty here.  It's quite simple and clear that when the mempool is backed up and you offer a greater fee then you are gaining likelihood of a faster confirmation.  No one is saying you have to estimate to the minute when it would be confirmed.

2 - Blocks are found, on average, once every 10 minutes, but are often found less frequently. It would only take seconds for the mempool (and thus the likely required fee to have your transaction quickly confirmed) to have grown in size dramatically (someone could potentially push several thousand valid transactions to the network all at once). This means that someone could push a transaction with a fee that their client claims will cause it to be quickly confirmed at time n, then at time n+1, someone pushes 10,000 transactions to the network, many with "better" fees, and before the next block is confirmed after time n. This would create a false sense of a guarantee to the end user.

No one should be offering a guarantee, as that would indeed be misleading.  Your argument here seems to fall into the fallacy of the perfect being the enemy of the good.  If I send a transaction at one moment there's clearly nothing I can do about any number of transactions which are sent after me with greater fees.  But if I decide to ignore facts about transactions which have been sent before mine, then that's information which is available to me which I am choosing to ignore.  And your arguments that it's somehow better to ignore this information simply aren't strong ones.

3 - Over the long run, the majority of end users are not going to run clients that are full nodes to spend their bitcoin. This means that they will need to place a certain level of trust in other full nodes when deciding what level of fees to include. This is trust that probably shouldn't be given. Today, if full nodes give incomplete information about unconfirmed transactions or about the blockchain, then any node that it connected it to will get that information from someone else. If that full node gives information about invalid transactions, then such invalid transactions will be ignored by everyone else.

Here you say end users are going to trust nodes but they probably shouldn't be doing that?  I just don't follow or I'm missing how this is relevant.  I apologize.  No node is going to have complete information.  Especially if we're talking about the future.  But anyone with access to a network connection can use information available at the moment in which they send their transaction.  The utility of this was demonstrated to me quite empirically when I sent a transaction last monday and waited 11 hours for confirmation.  If I had merely recevied some warning about the mempool status I would have sent with a higher fee or waited to send the next day.  Not having that information presented to me was clearly not as nice as if I had been presented that information.  Knowledge is power and wallets that present information about the current state of affairs to their users are clearly superior to ones that don't. 

As it is today, the mempool of every full node is potentially different, and often is going to be different from many other full nodes. The node that your client gets the size of it's mempool from may not be an accurate reflection of the network. It is also possible that a pool could have a large number of nodes that artificially increase the reported size of their mempool in order to encourage people to include larger tx fees then is necessary in an effort to increase the overall tx fees they receive. Requiring a node to prove the size of their mempool would open up a whole new can of worms that would simply not be worth it

Again, no one is suggesting that you need to prove the unprovable.  You seem to be going off a deep end here.  The idea, as I understood it, was merely to present better information to the user at the moment when a transaction is being sent.  Despite all your typing, I still can't see that this would be a bad thing.

[ChetnotAtkins]

I'm looking forward for the test today. Hopefully you guys accomplish the full planned scale this time!

[pereira4]

Im hoping the test is brutal and showcases the need of a bigger blocksize when people suffer transaction problems in first person (which seems to be the only way most people understand and react to problems).

[tspacepilot]

I'm curious if it's actually going to go down.  By this time last week the test was already backing up the mempool.  BCI shows 1400 unconfirmed tx at the moment, last time the "partial" test sent the backlock to over 8000K.  I wonder if this week's test is cancelled.

[ragi]

Maybe they don't stress it enough. Last 3 blocks are ~249.05kb

[elrippo]

Tastes like a ramp up 

[MicroGuy]

Im hoping the test is brutal and showcases the need of a bigger blocksize when people suffer transaction problems in first person (which seems to be the only way most people understand and react to problems).

I think everyone agrees that the 1MB block limitation is a problem, the argument is over how best to solve the problem.

Do we allow a single developer to break away from core and fork bitcoin alone using checkpoints (blocking nodes that don't update) and other possible tactics like pulling the "alert key" card out of his sleeve, or do we find a more fair and organized process.

[RealMalatesta]

Look as if some kind of stresstest is going on again. Blocksizes increasing to the limit, unconfirmed transactions over 8000...

[GingerAle]

https://bitcointalk.org/index.php?topic=1098263.140;topicseen

new test