December 2015 "Fastest Crypto" Bake-Off (topic locked)

[wingspan]

note - topic locked.  (see last msg in thread) - note: bake-off still "on".

October December 2015 "Fastest Crypto" Bake-Off

I and apparently many others want to know which cryptocurrency is the fastest. So I am offering to do this bake-off.  Specifically, with a given isolated 20-node testnet setup, and for 100 minutes, I want to know which crypto can clear the most payment transactions.

All contestants have until Monday 0700 UTC October December 26th to submit their entry.

Each entry must contain:

• The node software and associated files for this test.
• "A" Instructions to setup each node.
• "B" Instructions for getting all nodes in sync and ready for the test.
• "C" Instructions for auto-generating payment transactions during the test.
• "D" Instructions for manual payments I will do during the test.
• "E" Instructions for troubleshooting common issues/error that nodes might encounter...such as after a cold boot of the OS.
• "F" Instructions for after the test to count the number of successful payments that cleared on this 20-node network.
• "G" Instructions to test any other features desired.
• Your contact email address or PM name.
• Your attestation that the software is free to share, that it contains no malware, and that the software is the same as what the crypto community is using currently, except for changes you detail here (e.g., any performance tweaks).
• 0.2 BTC Entry Fee and a pay-back BTC address. (The winner gets back his/her entry fee.) Pay to 1C6nd36KSm6sqAhjEFqd91nNpJkq4sVEUL.  PM me links to the entry items and the BTC timestamp of payment.

For each entry, I will run the test three times, one per day, for 100 minutes each, following the instructions provided. I'll email/PM the contact with any questions/issues/results I encounter. Any revisions to the software or instructions are welcome up until the end of the third test. At that point, I'll share the three results of his/her tests privately, in case he/she wishes to withdrawal his/her entry from the public report.

My public report will be made Nov Jan 1st 0700 UTC. It will describe all three results of each entry not withdrawn and any special observations I made. I will post a link to the instructions and files provided, so others can run the tests themselves should they want, though no guarantee the sync will work as that could be controlled by the contestant.

All 20 nodes will be globally-scattered Linux Ubuntu 14.04 x64, 2 CPUs, 2 GB RAM, 40 GB SSD, 3 TB Transfer

For each test, I will:
1. Restore the 20 nodes to baseline I'll use identically for all tests.
2. I will follow "A" and "B" instructions to bring all 20 nodes up and synced into the testnet. (Special P2P server access will be permitted outward from my 20-node net to allow initiation of the 20 nodes and get them into sync if needed...but only prior to the start of the test.)
3. When synced, I'll start the test by blocking all testnet traffic not between the 20 nodes. (crypto must not depend on special servers, other than to get started).
4. I'll run the "C" instructions on all 20 nodes.
5. Also during the 100 minute test, I'll randomly cold boot each server and follow appropriate "E" instructions to bring nodes back in sync with my set of 20 nodes...then resume "C" instructions.
6. Also, at random times, I'll do "D" instructions from various nodes to verify monies can still be moved around without long delays.
7. I'll also do "G" instructions during the 100 minute tests.
8. After the 100-minute test, I'll perform the "F" instructions to count the successful transactions sent over the 20-node network.
9. All during and after each test, when I have questions or results to share privately, I'll email/PM them to the contestant.

May the best crypto win.

If this goes well, I expect to repeat this bake-off in a few months or more - and even to broaden the scope to include other tests as wanted by the community.

note: thread is self-moderated to rule out non-productive or unrelated noise.  Also, I am expecting eMunie to win, but I will try to not let that affect my bake-off results.  BitShares claims 100K TPS with caveats.  Others expressed an interest in challenging eMunie's claim to be fastest.  This will hopefully help the discussion by doing an apples-to-apples bake-off.

a few points for clarification added 2015-10-09:

• The tests are meant to be using test currency on a test P2P network.  I don't plan to connect to anything production.  If the test can work by tapping into an existing testnet long enough to get synced, that would be fine.
• I want to charge 0.2 BTC to only get serious entrants.  I am happy to donate any excess (beyond my costs to the hosting company and the winner's fee) to the winner.
• I do plan to take many screenshots as I go through the testing, and plan to document exactly what I do so others can verify I was not biased, hopefully.

Additional clarifications added 2015-10-11:

• I define one "payment transaction" as: one wallet paying another in the simple sense.  Additional activity for associated fees, preparing the payment so it can be spent by the receiver, or splitting payment into parts ... should all just counted as part of the one payment transaction.  E.g., buying a cup of coffee = 1 payment transaction (even if miners take a portion, a randomizer service is involved, or behind the scenes lots of activity happens in sub-transactions).
• The baseline configuration I reset nodes to for each test is what I get using this doc starting from the OS I get from Digital Ocean:  https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-14-04  If I need to install other "assumed" items, please provide the instructions.
• During the 100-minute test, with the random shutdowns I'll purposely perform, I will make sure on average the node software will be allowed to run (and I'll be doing the "E" and "C" instructions as fast as I can) 75% of the time.   In other words, 25% of the available time the nodes will be not running the node software.  While some of that 75% uptime will be re-syncing per the "E" instructions, hopefully the syncing will be quick so "C" instructions can crank out new transactions ASAP.  If the new transactions at best will reach, for example, 11.1 per second per node, and if syncing is near instantaneous (for the sake of this calculation), the best score you'd possibly get is 11.1 * 60 * 75 * 20 = ~ 1 million.

Results (coming Nov Jan 1st): TBD.

[1715021749]

1715021749

[1715021749]

1715021749

[1715021749]

1715021749

[1715021749]

1715021749

[r0ach]

Also, I am expecting eMunie to win, but I will try to not let that affect my bake-off results.  Others expressed an interest in challenging eMunie's claim to be fastest.  This will hopefully help the discussion by doing an apples-to-apples bake-off.

There's no such thing as an apples to apples comparison with Emunie.  Emunie uses a steady state network with a ledger and no blockchain.  This means if the network blows up, everyone is screwed.  Systems utilizing blockchains are far easier to recover from problems.

You can't compare blockchain coins to a coin that doesn't utilize one at all.  There's really no such thing as a fatal error in a blockchain system.  This is why Satoshi chose to use it.  The records will always be there from the past no matter what happens in the short term.  Saying Emunie is an apples to apples comparison is like calling non-volatile + volatile memory the same thing.

[Fuserleer]

Also, I am expecting eMunie to win, but I will try to not let that affect my bake-off results.  Others expressed an interest in challenging eMunie's claim to be fastest.  This will hopefully help the discussion by doing an apples-to-apples bake-off.

There's no such thing as an apples to apples comparison with Emunie.  Emunie uses a steady state network with a ledger and no blockchain.  This means if the network blows up, everyone is screwed.  Systems utilizing blockchains are far easier to recover from problems.

You can't compare blockchain coins to a coin that doesn't utilize one at all.  There's really no such thing as a fatal error in a blockchain system.  This is why Satoshi chose to use it.  The records will always be there from the past no matter what happens in the short term.  Saying Emunie is an apples to apples comparison is like calling non-volatile + volatile RAM the same thing.

Unfortunately I'm too swamped getting a beta ready to engage this now, but we need to have a conversation soon about this supposed "blowing up" you keep posting in various threads.

I'll be sure to grab your attention when I'm able so we can discuss and resolve once and for all, either way

As for your comments about the test, I suppose you consider a race between a petrol powered car and a electric one invalid too?

BTW: We're in...of course...if anyone wants to step up and isnt a bit scared

[TimSweat]

This is really going to be an interesting thread . I'll be keeping tabs on it for sure .

[r0ach]

As for your comments about the test, I suppose you consider a race between a petrol powered car and a electric one invalid too?

I don't think you can really argue it that way.  The blockchain system objectively has more redundancy.

[Fuserleer]

As for your comments about the test, I suppose you consider a race between a petrol powered car and a electric one invalid too?

I don't think you can really argue it that way.  The blockchain system objectively has more redundancy.

You can't claim that without comparing the two test subjects in a controlled test.  As the redundancy factor can be effected by a number of real world variables.

[EmilioMann]

the fastest crypto is vanilla coin.
The only one with zero time transactions and without the use of masternodes

https://bitcointalk.org/index.php?topic=977245.0

[Fuserleer]

the fastest crypto is vanilla coin.
The only one with zero time transactions and without the use of masternodes

https://bitcointalk.org/index.php?topic=977245.0

He means load, transactions per second, not confirmation times.

[r0ach]

Yea, I don't know a whole lot about Vanillacoin, but I'm pretty sure it's just taking a Bitcoin 0 conf transaction and using a locking mechanism (which may or may not have security issues) to just visually credit your account faster.  The transactions all still have to be put in a block if you're using a blockchain system...you didn't just gain infinite TPS from nothing.  All PoS systems that don't die will eventually switch to deterministic block validation anyway, thus making that 0 time thing irrelevant.

[StanLarimer]

The general idea of coming up with a way to benchmark these things is excellent and the OP is a worthy effort to get started.
Thanks for that effort.

Hopefully somebody can set up such a test using our code base.

Unfortunately we're launching two real-time blockchains this month (BitShares and PeerTracks' MUSE) so we're a bit busy.

But we'll be keeping tabs on how this goes.

Thanks
Stan

[Fuserleer]

The general idea of coming up with a way to benchmark these things is excellent and the OP is a worthy effort to get started.
Thanks for that effort.

Hopefully somebody can set up such a test using our code base.

Unfortunately we're launching two real-time blockchains this month (BitShares and PeerTracks' MUSE) so we're a bit busy.

But we'll be keeping tabs on how this goes.

Thanks
Stan

Seeing as most of the speculation regarding this topic is between eMunie and Bitshares, I'm sure the OP would make an exception and perhaps alter the test dates to suit you should you want to participate but unable due to your current workload.

[wingspan]

I do want to make the dates fit most developers' schedules - yet not drag out the race for too long.  I propose moving the dates out 2 months - with a winner announced Jan 1st, 2016.  That way we can give BitShares after their busy October/November, and others who are close to ready but not quite there, time to put together a tuned product they'd like to enter into the bake-off.

I will update the thread OP tomorrow if folks generally agree that seems appropriate.

p.s.  Yes, I was thinking "fastest" not in terms of confirmation time, but "capacity" to handle the most sustained TPS on its P2P network so that a crypto could be measured for its ability to go mainstream and not end up collapsing under its own success or having to become centralized (like BTC is expected to do even without going mainstream).  I appreciate the chance to clarify that.

[Fuserleer]

the fastest crypto is vanilla coin.
The only one with zero time transactions and without the use of masternodes

https://bitcointalk.org/index.php?topic=977245.0

He means load, transactions per second, not confirmation times.

who cares about transactions per second, without confirmations, they are as useful as a rock.

Will 2400 tps @ < 5 sec confirm times do you?

If you can wait a week we can increase that claim to 5000+ tps in our next beta test...is that enough?

Oh you want more?  Ok....leave that with us...already working on it

[Fuserleer]

Seeing as most of the speculation regarding this topic is between eMunie and Bitshares,

So Ripple is chopped liver?

Apologies, I guess that came over a bit arrogant...I'm going to blame a 6am bedtime and enthusiasm   I just mean that over the past week or so, both projects have been doing stress testing and posting results, which has sparked interest in both.

In fact, I have no idea what Ripple is capable of doing tps wise, care to enlighten?

[wingspan]

who cares about transactions per second, without confirmations, they are as useful as a rock.

good point - people need a payment to clear quickly and confidently just as much as high TPS capacity.  I plan to note "observed confirmation times" as part of the results but I don't know how to measure confidence levels.  I think that requires detailed knowledge of the risks of double-spends for each specific crypto....which will be up to experts and code reviewers.

[wingspan]

So Ripple is chopped liver?

I welcome all cryptos - even BTC if they want to compete.  The test plan I think makes it hard on centralized cryptos in that I will be making all 20 nodes run the same software and turning each of them off at times in case they are somehow special compared to the others.  I really look forward to seeing which cryptos can handle the failure conditions and still run fast and safe.

[StanLarimer]

A really noble effort.

Underwriter's Labs for Crypto is born!

[EvilDave]

I'd like to see this happen.......gentlemen, start your engines !

[binarygangster]

Yea, I don't know a whole lot about Vanillacoin, but I'm pretty sure it's just taking a Bitcoin 0 conf transaction and using a locking mechanism (which may or may not have security issues) to just visually credit your account faster.  The transactions all still have to be put in a block if you're using a blockchain system...you didn't just gain infinite TPS from nothing.  All PoS systems that don't die will eventually switch to deterministic block validation anyway, thus making that 0 time thing irrelevant.

Vanillacoin does use a locking mechanism but it's not just visual. It's designed such that your coins are re-spendable as well. With regard to the mention of a potential security issue, there was a beta phase with a 2k vnl bounty (or somewhere around that amount) for anybody that was able to successfully exploit the network, which was not claimed. So while I wouldn't go so far as to say it's not possible, it hasn't happened yet.

[john-connor]

Yea, I don't know a whole lot about Vanillacoin, but I'm pretty sure it's just taking a Bitcoin 0 conf transaction and using a locking mechanism (which may or may not have security issues) to just visually credit your account faster.  The transactions all still have to be put in a block if you're using a blockchain system...you didn't just gain infinite TPS from nothing.  All PoS systems that don't die will eventually switch to deterministic block validation anyway, thus making that 0 time thing irrelevant.

You must be confusing ZeroTime with instantX because your missing allot here and barely touching the basics of the technology. Also "visually crediting" is inaccurate, it is a core layer consensus based confirmation. I often perform ~5100 TPS stress tests with no back-log. I see no reason to switch to deterministic block validation since this is well under capacity.

Thank you for your support.