ANN: Announcing code availability of the bitsofproof supernode

[grau]

A step back: Pruning was not correct. Reverting and thinking...

[jgarzik]

Have you tested block chain reorg?  i.e. where a previous-best chain is overtaken by a stronger chain, causing transactions to be un-confirmed?

[grau]

I know thats the tricky part.

The bitsofproof node logically "reorgs" by deciding which block is the trunk head (with the most cumulative work). The blocks are only linked backward to previous. Traversing from the trunk head block leads to genesis although there are alternate heads with partially different paths to root. When a block connects somewhere I decide if it has now more cumulative work than the trunk head, then it is the new trunk head. Thats the "logical reorg" since nothing really moves, just a trunk head is identified for the decision on spend-ability of an output:

The decision if an output is spendable is done by traversing back from trunk head using backward pointer of each block only taking into account transactions of blocks on that path.

Since I cache the linked chain (tree) of block hashes traversing to find the right path (that is list of block hashes) for the query is fast. In addition I now build a cache of spendable output (instead of pruning).

[grau]

Have you tested block chain reorg?  i.e. where a previous-best chain is overtaken by a stronger chain, causing transactions to be un-confirmed?

To also answer your question: No. I need to create test cases.

I think, best I create basic mining now, so I can mine my own chain creating the weirdest test cases.

[apetersson]

are you going to stick to the IVY  build system? i tried to get the environment running in intellij, but this provides no meaningful IVY support. if you had used maven or even just plain old  lib directory with all dependencies it would be much easier for me. currently i am butchering random jars together along with rewriting some code, which makes keeping up do date with your changes hard.

since you do have test cases you should consider running them automated on checkin with jenkins or teamcity. i just pulled in some changes which broke the build because of missing classes. (QKnownPeer is missing along with others)

thats not to be meant as criticism, i think the code looks interesting and i want to see where the project is going. my suggestion about ivy vs maven means that the code would get more accessible to other developers, plus i am lazy and do not want to spend a lot of time setting up my ide for this project

[grau]

I will mavenize the project in the course of serious testing soon.

The classes with Q at the begining are generated during the build, this is rather an indication that you were not using the build process as suggested.

Let me write on the vision of the projet in a next mail.

[grau]

I understand that you only spend time on a project if it is aligned with your goals, therefore let me give you my vision of it:

The vision is to create THE bitcoin implementation for the merchants and miner, similar to what apache is for the web.
You might think I suffer under delusion of grandeur, but hey, I would not sacrify my precious free time if it were for less.
Let me make the claim plausible by enumerating that the bitsofproof implementation already achieved:

- Complete implementation of the Bitcoin communication protocol with a high speed NIO IO that is able to deal with a large number of peers.
- Complete implementation of the Bitcoin script language, such that it verifies all sorts of sigs and multisigs on production and testnet3 chain
- Normalized store of the chain into a relational database that now allows auditing and arbitary querying of all transactions with the tool of your choice.
- Online calculation of unspent transaction output (UTXO aka. logical pruning)
- Multithreaded evaluation of transactions of a block and even inputs of the same transaction, that gives together with UTXO unparalelled processing power.
- Implementation of the BCCAPI that enables BitcoinSpinner compatible/like lightweight applications to retrieve balance and send transactions without server store of private keys.

I claim that the vast majority of implementation work is done, what remains on the roadmap is:

- Mining support
- Serious testing
- Packaging

I share the foundation with the community for our mutual benefit, thereafter I will focus on my own value added services and consulting.

Should you share the vision and would love to have a strong fundation for your business, then lets join efforts, show me your pull requests.

Let me also use the opportunity to THANK for the first donation onto the address below.

[grau]

Audit-ability is prerequisite for serious business use, therefore I created a utility that reads the database filled by the bitsofproof node and performs off-line audit.

This tool shares only basic data access and serialization primitives with the server code, so it is fairly independently validating the database content.

You see below an example output:

Code:

[INFO] Audit main Check 1. All paths lead to genesis...
[INFO] Audit main Check 1. Passed.
[INFO] Audit main Check 2. There are no orphan blocks...
[INFO] Audit main Check 2. Passed.
[INFO] Audit main Check 3. Sufficient proof of work on all blocks and correct cumulative work on all paths...
[INFO] Audit main Check 3. Passed.
[INFO] Audit main Check 4. Check transaction hashes and Merkle roots on all blocks...
[INFO] Audit main Check 4. Passed.
[INFO] Audit main Check 5. Block reward amount correct for all blocks...
[INFO] Audit main Check 5. Passed.
[INFO] Audit main Check 6. Total coinbase matches sum of unspent output...
[INFO] Audit main Check 6. Passed.
[INFO] Audit main All requested checks PASSED.

Let me know if you have further ideas for health check of the chain stored.

[jgarzik]

Here is one list: https://en.bitcoin.it/wiki/Protocol_rules

[grau]

Here is one list: https://en.bitcoin.it/wiki/Protocol_rules

That is already implemented, in the server code.

This is about independent global fiscal checks.

[2112]

This is about independent global fiscal checks.

Are you talking about pure binary yes/no consistency check audit?

Or are you talking about open-ended auditing for the purpose of discovering normal vs. abnormal transaction patterns?

I've spoken a while back with an experienced forensic accounting professional and we discussed various things that would require keeping true time vs. block time stamp. It would also require keeping a separate database of the in-flight transactions, ones that aren't yet recorded in the blockchain.

4) provide a way of obtaining sincere ledger / audit log for the transactions with multiple timestamps:
4a) true time when first seen on the p2p net
4b) true time when first seen in a block
4c) block time when seen in a block
4d) true time when some block caused a reorg and un-confirmed the transaction
4e) true time when other block reconfirmed the transaction
4f) block time when reconfirmed
4g) etc.. for each subsequent chain reorganization

The observation he had made is that the blockchain is a perfect whitewash: it leaves no trace of the attempted double-spends or other malarkey that would be of great interest to a forensic auditor, e.g.

a) list of addresses involved in the attempted double-spends
b) list of addresses involved in transactions recorded in the orphaned sub-chains but not recorded on the winning chain.

I'm not actually suggesting that you actually implement those auditing reports. But it would be nice if your architecture made possible generating such reports.

[grau]

I meant with audit the proof of consistency of the stored chain independent of the process that maintains it.

Since the bitsofproof node stores all data in a normalized relational database, you should be able to use any data mining package you prefer.

The data currently collected is what is needed to work.

I would be interested to know what extra data would support forensic accounting, and would attempt to derive it from the server's interaction with peers.

[2112]

I would be interested to know what extra data would support forensic accounting, and would attempt to derive it from the server's interaction with peers.

The additional data that is not available from the blockchain:

1) true timestamps for transactions first (and last) seen over p2p
2) true timestamps for blocks first seen over p2p
3) non-volatile database of transactions seen over p2p (a.k.a. mempool, but non-volatile) pruned of the entries actually recorded in the blockchain.
4) a way to quickly locate only orphaned blocks without having to scan the whole table of blocks, really just an additional index to the blockchain storage, but explicitly indexing all the losing chains. This in theory is available from the blockchain, but under current architecture it is difficult to obtain this information. In effect the whitewash of the bitcoin transaction history is ongoing when the tools keep only the winning chain.

Thanks.

[grau]

I would be interested to know what extra data would support forensic accounting, and would attempt to derive it from the server's interaction with peers.

The additional data that is not available from the blockchain:

1) true timestamps for transactions first (and last) seen over p2p
2) true timestamps for blocks first seen over p2p
3) non-volatile database of transactions seen over p2p (a.k.a. mempool, but non-volatile) pruned of the entries actually recorded in the blockchain.
4) a way to quickly locate only orphaned blocks without having to scan the whole table of blocks, really just an additional index to the blockchain storage, but explicitly indexing all the losing chains. This in theory is available from the blockchain, but under current architecture it is difficult to obtain this information. In effect the whitewash of the bitcoin transaction history is ongoing when the tools keep only the winning chain.

Thanks.

None of these sounds difficult to add.
In fact 4 is already there since the HEAD table contains pointer to all leaf nodes not only the trunk.

[grau]

.... my suggestion about ivy vs maven means that the code would get more accessible to other developers, plus i am lazy and do not want to spend a lot of time setting up my ide for this project

The project just got mavenized by pulling from the first contributor: https://github.com/bartfaitamas/supernode/tree/mavenizing

[apetersson]

The project just got mavenized

sounds great! will have more looks at at and try to get it running later this week.

[grau]

It is time for a project update.

[HYPE]
I enumerate some challenges and the bitsofproof answers to them.

1. Responsiveness to a large number of peers.
This is addressed using non-blocking-IO and a thread pool that forwards fully read messages to their listener, that are otherwise isolated from the complexity of the communication.

2. CPU hunger for signature validation.
This is not a current concern for the bitsofproof node since validation of transactions in a block and inputs the same transaction are executed simultaneously. CPUs are utilized to their limit, as many as you provide.

3. Resolving the inputs of a transaction against the database of previous transactions.
Using a faster database (some NoSQL) or SSD drives only give you a short break until the bottleneck re-appears as the network grows.
A sustainable approach to address the problem is to constantly prune the data set searched. The bitsofproof node uses a separate archive transaction table(s) into which a parallel thread moves fully spent transactions much like a garbage collector reclaims storage. The live transaction table that is searched to satisfy new transaction inputs is kept close to the UTXO set.

4. The client API is implemented as a clone of BCCAPI, that provides services to lightweight clients trusting the server.

5. A server API is in draft, that will provide a high level access to persistent and transient data structures, notifications. The aim is to provide all features provided in bitcoind via JSON-RPC, but in bitsofproof's case using the invoker of your choice on an API defined as a proper java interface.

6. The next layer of scale will be implemented through distributing both storage and computation, considerations for this already influence the design.
[/HYPE]

This is code in development. I do not care of backward compatibility of interfaces or database for now.

[grau]

Introducing an opportunistic cache of the last 100.000 transactions gave such a boost to the server that I thought it is worth writing about.
Apparently outputs die generally rather young and saving the db roundtrip for them is the single biggest boost I found until now...

[jgarzik]

Introducing an opportunistic cache of the last 100.000 transactions gave such a boost to the server that I thought it is worth writing about.
Apparently outputs die generally rather young and saving the db roundtrip for them is the single biggest boost I found until now...

Yep, pynode figured many of these things out, long ago.  A block (including TXs) cache is very useful.

For long running nodes, the signature cache is also very helpful.  Over time, transactions are accepted into the memory pool and signature cache.  When a new block arrives, the majority of transactions found in that block will have had their signatures pre-checked and cached, making acceptance of that block go more rapidly.

[J-Norm]

This looks very promising. Backend bitcoin engines need a nice modular format to allow for the many creative uses people will be coming up with.