ANN: Announcing code availability of the bitsofproof supernode

[xblitz]

Introducing an opportunistic cache of the last 100.000 transactions gave such a boost to the server that I thought it is worth writing about.
Apparently outputs die generally rather young and saving the db roundtrip for them is the single biggest boost I found until now...

Yep, pynode figured many of these things out, long ago.  A block (including TXs) cache is very useful.

For long running nodes, the signature cache is also very helpful.  Over time, transactions are accepted into the memory pool and signature cache.  When a new block arrives, the majority of transactions found in that block will have had their signatures pre-checked and cached, making acceptance of that block go more rapidly.

are you honestly adding more info.. or just trying to point out that your "pynode" is *so* ahead ??

[jgarzik]

Introducing an opportunistic cache of the last 100.000 transactions gave such a boost to the server that I thought it is worth writing about.
Apparently outputs die generally rather young and saving the db roundtrip for them is the single biggest boost I found until now...

Yep, pynode figured many of these things out, long ago.  A block (including TXs) cache is very useful.

For long running nodes, the signature cache is also very helpful.  Over time, transactions are accepted into the memory pool and signature cache.  When a new block arrives, the majority of transactions found in that block will have had their signatures pre-checked and cached, making acceptance of that block go more rapidly.

are you honestly adding more info.. or just trying to point out that your "pynode" is *so* ahead ??

pynode is quite incomplete, as the TODO notes.  bitsofproof is farther along in many ways.

The point is that these are well known techniques, and it is disappointing that these are being "discovered" when the knowledge is readily available for anyone who looks.

[grau]

For long running nodes, the signature cache is also very helpful.  Over time, transactions are accepted into the memory pool and signature cache.

I do not yet see how the signature cache actually helps, would you please elaborate?

A signature is a function of the hash of a modified version of the input transaction. The modification depends on which output of that transaction is to be spent. Aren't chances of seeing the same signature twice negligible?

[jgarzik]

For long running nodes, the signature cache is also very helpful.  Over time, transactions are accepted into the memory pool and signature cache.

I do not yet see how the signature cache actually helps, would you please elaborate?

A signature is a function of the hash of a modified version of the input transaction. The modification depends on which output of that transaction is to be spent. Aren't chances of seeing the same signature twice negligible?

Seeing a signature twice is the normal case:

• once as an unconfirmed transaction via "tx" message
• once as a confirmed transaction, via "block" message

It is critical to keep "block" message relaying (propagation) times as low as possible, to avoid creating incentives for miners to skip transactions.  Thus, a signature cache ensures that "block" messages are largely already verified, by the time they are received.

[caveden]

It is critical to keep "block" message relaying (propagation) times as low as possible, to avoid creating incentives for miners to skip transactions.  Thus, a signature cache ensures that "block" messages are largely already verified, by the time they are received.

If miners are already shrinking their blocks due to propagation time, maybe it's time to consider that protocol improvement suggestion in the scalability page, about not sending the entire block every time, just the portion your peer doesn't have.

I don't know what exactly could be send instead of the body, though. Even if my peer has all transactions in his cache, doesn't he need to know the exact order I put them in my block in order to rebuild the Merkle tree? So just sending the header is not enough. Is there a shorter way to identify a transaction than its hash? Perhaps just an ordinal of the hash... you assume your peer has the same transaction pool you had, then as the body of your block you send a series of numbers which represent the index a tx hash would have in a  sorted array with all of them? Not sure that would work frequently... plus, the hash is probably small enough, it shouldn't be a big deal to send all of them.

[grau]

Seeing a signature twice is the normal case:

• once as an unconfirmed transaction via "tx" message
• once as a confirmed transaction, via "block" message

I see, I was thinking only in the context of the chain, not chain and unconfirmed. Thank you.

[grau]

It is critical to keep "block" message relaying (propagation) times as low as possible, to avoid creating incentives for miners to skip transactions.

I think offering sufficient fee in the transaction is the incentive that will keep working as the network grows, since CPU spent for validation does not come for free.

[apetersson]

i think the point jgarzik was making is about block relaying from other nodes who did not find the block.

[grau]

Now you have the option to use LevelDB or any relational database, just by configuration.

Use the relational store if you are after data mining and ad-hoc queries and have more time.
Use LevelDB if you want raw power.

The bitsofproof supernode validates the chain with LevelDB at a speed only constrained by CPU.
EDIT: And I mean all processors you have

Nice proof of modularity isn't?

I am working to enable higher order search functions (via API) to LevelDB.

[grau]

What do I mean with raw power?

Download full blocks from 0-210.000, validate all Merkle roots, proof of work and check for double spending, stored in LevelDB
in 93 minutes. The constraint until about block 130.000 is the peer/network speed, thereafter one CPU is busy.

Full validation (as above + scripts) and store of blocks 210.001 - 211558 on 4 CPUs (near constant 90%+ utilization of each)
in 61 minutes

Bootstrap of a full node with above split parameters in 2 1/2 hours.

I will do comparison with the relational model next week.

[gmaxwell]

Whats the current timeframe for enforcing all of the script rules so that mining on this code isn't a forking risk? (if there is one?)

[grau]

All rules I found are enforced. I am eager to learn any outstanding.

The biggest deliverable is serious testing.

I would love to get this production ready this year, but since I only work on this on my spare time and my family has other plans with me around Christmas, I can not commit.

[hardcore-fs]

For caching, you may consider the open source "ehcache"
Which can be used with a disk backing store or in memory, complete with aging out to disk or removal.


HC

[gmaxwell]

All rules I found are enforced. I am eager to learn any outstanding.
The biggest deliverable is serious testing.
I would love to get this production ready this year, but since I only work on this on my spare time and my family has other plans with me around Christmas, I can not commit.

Okay. You're missing at least one important rule, but I don't know what else is missing that I missed in review.

Unless you object, I'm contemplating keeping this missing rule to myself for now so that it can function as a test of the testing. E.g. if the test don't find it, they're incomplete.  Obviously I don't want to withhold information which could undermine the network, but absent better testing the software shouldn't be in production use anyways... and there are scarcely few good ways of testing the tests.   Does this sound reasonable to you?

[grau]

I am ready to take the challenge.

How would you rate the current coverage of rules after your review (100% being complete)?

May I have a pointer if the omission you found is in script, message format, protocol, block-chaining (reorg), caching or any other more specific area?

[Mike Hearn]

I took a quick look at the latest code also, it seems there is  a thread safety issue that could potentially cause a split in the consensus. I guess I'll go with Gregs approach too and not say exactly where it is, though that should be a big hint. Thread safety is tricky to get right as you know, especially, it's hard to check with unit tests. So it would be good to see how it is resolved.

[grau]

I made this open source for mutual benefit, that is the community gets my work for I get constructive feedback and not just hints on potential issues after a quick look.

Are you sure there is a bug?

[grau]

Actually, I am sure there are bugs.

The common quest should however be creating a good product, not proving that testing is insufficient.

[Mike Hearn]

I'm pretty sure there is a bug, yes. If I'm wrong then I will of course apologise and go back to update my previous post.

The point of being vague is that us finding bugs on random code reviews isn't reliable or scalable. With your new codebase you're not solving a problem any of the current core developers have, so there isn't much incentive for us to go finding bugs for you. But if people were to use or mine upon a new implementation with bugs, that would definitely cause problems.

So what's important to demonstrate is that your code can be used without presenting a danger to the network, and that means really solid processes and testing to ensure the behavior matches exactly. If we can find one or two issues, there could well be more, so a rigorous testing and protocol compliance regime is the only way to build confidence.

[grau]

I'm pretty sure there is a bug, yes. If I'm wrong then I will of course apologise and go back to update my previous post.

The point of being vague is that us finding bugs on random code reviews isn't reliable or scalable. With your new codebase you're not solving a problem any of the current core developers have, so there isn't much incentive for us to go finding bugs for you.

Since you think, I am not helpful to you, you do not feel being obligated to help me. That's fair.
Using your authority and no specifics on claiming a bug is not.

I remain hoping that you uncover the issue while I am testing.