a simple traffic load test run

[lfm]

I have seen some speculation about scalability and denial of service by spam transactions in the IRC channel so I thought it would be a good idea to try a test.

I set up a stupid little bitcoind script on a couple of my linux machines to send 1000 tiny transaction to a third machine as quickly as they would run.

I asked on IRC for any obvious reasons why I shouldn't do it and got one supporting response thinking it was a good idea to try and one reserved response saying I should ask here first. I brashly decided to go ahead with it.

The scripts just printed a countdown of the number of remaining steps in the loop and the output of the bitcoind sendtoaddress command, normally just "sent".

They both started out quickly till there was something like 600 transactions submitted then both senders started to slow down perceptibly. One was quite a bit faster than the other, 3 or 4 times faster, due I think to a faster CPU. I thought at first it was the clients throtteling me to save the net from overload or something but that was just speculation. We later figured maybe it was the databases getting a bit slow on my old disks.

Then one of my machines suddenly seemed to speed up. But it wasn't really. It was an error, the bitcoind daemon had died. A little bit of investigation found the .bitcoin drive was full so you can't fault the program for giving up there! grin

The other machine was grinding pretty slow by then down to maybe 5 sec per transaction or so just judging by eye. It eventually finished after about 50 minutes. no errors at all.

The machine that crashed after I gave it some more disk space started right back up and showed it had only sent 424 transactions so there is an initial benchmark then - 1424 transactions in 50 minutes. It should be easy to get more.

I think the main thing slowing them down was the .bitcoin/database directory. They seemed to get a lot of data written to them some 184MB on the one that ran outa disk space and 1341 MB on the one that did the full 1000 transactions.

It seems another test with more people more distributed around the net might be in order to try to shake down any potential problems.

The net seemed to shrug it off as if nothing special happened anyway so that is a positive result. The clients mostly did just what they were told and expected except for a possible performance problem with the database.

Thats all I can think of for now.

[1715555154]

1715555154

[1715555154]

1715555154

[1715555154]

1715555154

[lfm]

I was able to run IRC and discus the progress of the test thruout so I don't think I was anywhere near saturating any network resources.

[theymos]

There was a lot of slowdown in the rate of transactions during the second half of the test. If this was a problem with the network, then I would be very worried about future denial of service attacks. Hopefully a bigger test can be organized later.

Here is a packet capture of the event (from an "edge" computer with only one connection) if anyone is interested in exact times and stuff.
http://www.freefilehostingnow.com/filedownload.aspx?code=6bb2dbeea18e6a419daaadda5798bb05ec6b

[satoshi]

Was that on the test network?
http://bitcointalk.org/index.php?topic=363.0

[theymos]

Was that on the test network?
http://bitcointalk.org/index.php?topic=363.0

No.

[satoshi]

Please do these tests on the test network.  That's what it's for.  Thanks.

[theymos]

Please do these tests on the test network.  That's what it's for.  Thanks.

You can't get real data from such a small network. Besides, a reliable network must be able to withstand any kind of testing.

[Gavin Andresen]

Well, at least do it on the TEST network FIRST!

If you manage to break the TEST network, it's a pretty good bet that you'll be able to break the production network.  If it doesn't break the TEST network, then I'd say go ahead and run against the production network to look for "scaling up" problems.

[FreeMoney]

Well, at least do it on the TEST network FIRST!

If you manage to break the TEST network, it's a pretty good bet that you'll be able to break the production network.  If it doesn't break the TEST network, then I'd say go ahead and run against the production network to look for "scaling up" problems.

Really? I would think/hope that the real one is way more robust. No?

[RHorning]

Well, at least do it on the TEST network FIRST!

If you manage to break the TEST network, it's a pretty good bet that you'll be able to break the production network.  If it doesn't break the TEST network, then I'd say go ahead and run against the production network to look for "scaling up" problems.

Really? I would think/hope that the real one is way more robust. No?

Stress testing the "test network" is a way to at least not annoy those who are trying to do positive things with Bitcoins.  Tests of this nature will naturally be causing more work for others who are not necessarily wanting to be involved in these kind of tests.  Those running the test network realize the very experimental nature and don't mind CPU and disk bandwidth being wasted for those efforts.

From a purely ethical point of view, I think it is much better to test on the test network first, especially if it is to exploit something in the communications protocol that you think needs to be strengthened.

[lfm]

Please do these tests on the test network.  That's what it's for.  Thanks.

Sorry, I wasn't even aware of the test network. My bad.

Can we organize a LARGER test for the test network? More people at once.

[aceat64]

Please do these tests on the test network.  That's what it's for.  Thanks.

Sorry, I wasn't even aware of the test network. My bad.

Can we organize a LARGER test for the test network? More people at once.

If we kept the test relatively short I don't mind throwing on a few EC2 instances on the test network to help out.

[sgtstein]

I would also be willing to run tests for this. I have about 6000kh/s at my disposal.

[theymos]

Stress testing the "test network" is a way to at least not annoy those who are trying to do positive things with Bitcoins.  Tests of this nature will naturally be causing more work for others who are not necessarily wanting to be involved in these kind of tests.  Those running the test network realize the very experimental nature and don't mind CPU and disk bandwidth being wasted for those efforts.

From a purely ethical point of view, I think it is much better to test on the test network first, especially if it is to exploit something in the communications protocol that you think needs to be strengthened.

The goal (my goal, at least) is to have some working distributed monetary system, not to keep a (potentially) broken system online without interruption. The whole idea of a "test" network is nonsense -- every test should be done on the main network if possible, since real problems that could occur might otherwise be missed.

[jgarzik]

Any chance to get the test network into bitcoin/trunk, #ifdef'd out?

[NewLibertyStandard]

Stress testing the "test network" is a way to at least not annoy those who are trying to do positive things with Bitcoins.  Tests of this nature will naturally be causing more work for others who are not necessarily wanting to be involved in these kind of tests.  Those running the test network realize the very experimental nature and don't mind CPU and disk bandwidth being wasted for those efforts.

From a purely ethical point of view, I think it is much better to test on the test network first, especially if it is to exploit something in the communications protocol that you think needs to be strengthened.

The goal (my goal, at least) is to have some working distributed monetary system, not to keep a (potentially) broken system online without interruption. The whole idea of a "test" network is nonsense -- every test should be done on the main network if possible, since real problems that could occur might otherwise be missed.

It's preferable, when possible to identify and fix vulnerabilities before they can harm the production system. The test network is more vulnerable in most cases, so it's a better candidate to find vulnerabilities and as a bonus it doesn't annoy lots of people.

[lfm]

Stress testing the "test network" is a way to at least not annoy those who are trying to do positive things with Bitcoins.  Tests of this nature will naturally be causing more work for others who are not necessarily wanting to be involved in these kind of tests.  Those running the test network realize the very experimental nature and don't mind CPU and disk bandwidth being wasted for those efforts.

From a purely ethical point of view, I think it is much better to test on the test network first, especially if it is to exploit something in the communications protocol that you think needs to be strengthened.

The goal (my goal, at least) is to have some working distributed monetary system, not to keep a (potentially) broken system online without interruption. The whole idea of a "test" network is nonsense -- every test should be done on the main network if possible, since real problems that could occur might otherwise be missed.

It's preferable, when possible to identify and fix vulnerabilities before they can harm the production system. The test network is more vulnerable in most cases, so it's a better candidate to find vulnerabilities and as a bonus it doesn't annoy lots of people.

Ok seems to be a philosophical difference here. Is it really a "production" network? It is still running on software labeled as "beta" sending the signal that it is not ready for "production". Are we really already to the point of being unwilling to tolerate failure because bitcoins still seems to be very much experimental. I am confused.

[omegadraconis]

I think the point here is that the test network is there for tests specifically. If you are going to try and test something, say stealing everyone's bitcoins, it is preferred you do that on the test network for obvious reasons.

Now a load test is not really very useful on a test network if you are interested in real world data because, you are just testing the test network. I would guess that the test net has less clients on it then the production network. Now that being said I would test my code against the test net first until I know it's not going to break anything. I don't think this type of test would be too annoying on the production network as long as it doesn't violate two rules. The first is that it not cause a denial of service on the network or real actual transactions. The second is that it should not fill up everyone disk with a large data log.

I would be willing to add a node or two to the test network for a load test. I have window and linux boxes that I could use.

[jgarzik]

Ok seems to be a philosophical difference here. Is it really a "production" network? It is still running on software labeled as "beta" sending the signal that it is not ready for "production". Are we really already to the point of being unwilling to tolerate failure because bitcoins still seems to be very much experimental. I am confused.

bitcoins are experimental, and the system may fail for any number of reasons.

However, there is Real Money and value attached to bitcoins.  You're not being a good network citizen if you're messing with that.

[lfm]

Ok seems to be a philosophical difference here. Is it really a "production" network? It is still running on software labeled as "beta" sending the signal that it is not ready for "production". Are we really already to the point of being unwilling to tolerate failure because bitcoins still seems to be very much experimental. I am confused.

bitcoins are experimental, and the system may fail for any number of reasons.

However, there is Real Money and value attached to bitcoins.  You're not being a good network citizen if you're messing with that.

Ok, I hear you, I just wonder about your situation. Why are you putting so much "Real Money" with capitalization and value into a beta experimental system? Are you over-committed to something flaky? Declaring it so doesn't make it so. Is the "beta" designation in error? Should it be promoted to  version 1.0?

I agree the first test, even if it would be inconclusive, should have been on the test net. Sorry, like I said I wasn't aware. I hadn't read hardly any of the forums even. Bitcoins just seemed like a cool toy and I wanted to play with it.

I guess I never expected people to stake their life's saving or even a day's wages on the thing at its current stage of development. It still doesn't make sense to me to do anything of the sort. Obviously some people see it differently. I will try not to repeat my mistake.