ANN: Announcing code availability of the bitsofproof supernode

[grau]

TL;DR
   Announcing code release of the bitsofproof supernode.
   A modular Bitcoin protocol implementation to build bitcoin enterprises and server nodes on.

WHY
    Satoshi is a genius, but his implementation of the protocol is hard to maintain and extend.
    I believe that peers will specialize and this implementation is suited for the big server,
    tailored for some heavy lifting tasks, not for a mobile.

WHERE
   https://github.com/bitsofproof/supernode.git
   
FEATURES
   Pure Java source code released under Apache 2.0 license.
   Radically modular Assembly using Spring. Extensible and reducible to what you need.
   High performance P2P engine using Java NIO.
   Block storage using JPA into a fully normalized schema.
   Full support for multisig and pay to hash, including address maintanance.
   Parallel validation of transactions of a block.

DEPENDENCIES
   Spring, Bouncy Castle, SL4J
   Does NOT share any code with Satoshi or bitcoinj clients.

   
LIMITATIONS
   Not tested enough for production, therefore does not yet answer getheader and getblock
   and does not feature a wallet.

CONTRIBUTIONS
   Contributions are welcome. Add your own features to the modular design.

WHO
   Tamas Blummer alias grau tamas@bitsofproof.com wrote this implementation in his spare time at home.
   
DONATIONS
    Please honor the time I invested and motivate me to continue by donating to:
   
    13xhxy21to93Lrb7d4ssZVaMnFtQVvRkSk

[1714211513]

1714211513

[1714211513]

1714211513

[1714211513]

1714211513

[1714211513]

1714211513

[1714211513]

1714211513

[1714211513]

1714211513

[lenny_]

this implementation is suited for the big server,
    tailored for some heavy lifting tasks

Details please?
What so "super" in this code that make it better from standard bitcoind?

[caveden]

What so "super" in this code that make it better from standard bitcoind?

No idea if it's "better than bitcoind", actually it'd be quite presumptuous to say so.

But usage of JPA is already an interesting point... Different databases could be plugged. Databases that support clustering or partitioning could be used.
"Fully normalized schema" doesn't always go well with high performance though...

I haven't checked the code or anything, but if it's for real, wouldn't this be the first alternative full node implementation of bitcoin?

If anything, it's an improvement in 'diversification'. And I'm glad it uses technologies I'm familiar with.
I wonder how the author plans to keep up with all the development done in bitcoind though... they have many competent people improving the code and adding functionality. Just to keep up would be an enormous effort. But anyway, writing such a thing was certainly an enormous effort already!

By the way, OP, why not reuse BitcoinJ code if it fits?

[grau]

What so "super" in this code that make it better from standard bitcoind?

No idea if it's "better than bitcoind", actually it'd be quite presumptuous to say so.

Because one size does not fit all. Bitcoind is a monolithic hard to maintain code base.

But usage of JPA is already an interesting point... Different databases could be plugged. Databases that support clustering or partitioning could be used.

"Fully normalized schema" doesn't always go well with high performance though...

Yes, and you can stick in whatever high performance provider your budget allows. This is for the
terabytes coming. Normalization allows to build services on top of the database
that do not have to deal with P2P or binary formats.

I wonder how the author plans to keep up with all the development done in bitcoind though... they have many competent people improving the code and adding functionality. Just to keep up would be an enormous effort. But anyway, writing such a thing was certainly an enormous effort already!

The effort of bitcoind team made all this possible. I have highest regards, but I do not think the code
they inherited is for the future.

I would rather worry if bitcoind can keep up with innovation, now that a modular design is available
people can easily stick their modules into. You do not have to wait for a new RPC call to add a new
type of signature. You do not have to wait for ultraprune, just stick in a higher performance database.

BTW: I buit enterprise systems, quant code and algo trading for decades. Look me up on linkedin if curious.  

By the way, OP, why not reuse BitcoinJ code if it fits?

Because it does not.

[Mike Hearn]

Nice! The code looks clean.

caveden, there have been several other full reimplementations. BitcoinJS is one, I think UfaSoft made one, and actually I just finished merging code from Matt that makes bitcoinj a fully verifying ultra-prune style implementation as well (of course, it can still run in SPV mode too).

So this implementation joins the ranks.

That said, from looking over the code, I can't find a few features I expected to be there:

- Where is the wallet implementation? How would I implement the PingService from the bitcoinj examples directory?
- Where are the tests proving conformance with all of Satoshis bugs? I see a few tests but they are fairly minimalist.
- API documentation

eg, there are script tests in bitcoinj (since yesterday) and the Satoshi client that verify the execution of every script opcode is correct in both positive and negative ways. It'd be nice to see them. The hard part of making full nodes is not implementing the protocol, it's ensuring you got every detail right.

Re: your comments about monolithic design. With respect to your experience, I don't think simply using Spring makes something "modular" vs "monolithic". Allowing people to build powerful apps is as much about documentation, examples, well-specified callback interfaces as it is about good use of OO design. For example, there are a variety of listener interfaces throughout bitcoinj, and each one has well specified and consistent behaviour: listeners run locked, they can remove themselves during execution but not other listeners. Also most of them document allowed re-entrancy. This sort of thing makes writing apps a lot easier.

I'd be the first to admit that bitcoinj could use some loving refactoring. The API is in places quite messy. There isn't really a good enough block chain API. The Wallet needs splitting out into customizable policy objects. That said, you don't seem to have a Wallet object at all unless I somehow missed it, so I don't feel too bad about that It looks the way it does because bitcoinj is already being used to write all kinds of apps, from end-user facing wallets to SatoshiDice. So at the moment we tend to prioritize new features over refactorings and API churn that makes it hard to keep up.

In future there might be a bitcoinj2 effort that re-organizes things a bit.

By the way:

You do not have to wait for ultraprune, just stick in a higher performance database...

I'm not sure you understand what ultraprune does or why the performance is higher. Firstly, it does swap out bdb for LevelDB, which is not exactly a difficult change. But more importantly it reduces the size of the databases working set by fundamentally re-organizing how data is used. It doesn't matter what database backend you use or how easy it is to swap in or out if the way you store data is inefficient.

[grau]

Nice! The code looks clean.

caveden, there have been several other full reimplementations. BitcoinJS is one, I think UfaSoft made one, and actually I just finished merging code from Matt that makes bitcoinj a fully verifying ultra-prune style implementation as well (of course, it can still run in SPV mode too).

So this implementation joins the ranks.

Thanks for the warm welcome. Was nice to talk to you in London Mike.

I released the code since it reached the coverage needed to prove that it is for real. Not for production but to build on it as a platform.

Your points on testing are valid. They are clearly not sufficient. I plan to introduce an interface aimed for testing. That would enable
implementations to test each other.

The Wallet is not there but everything you need to stick it together in a couple hours or days depending on what you want.
I did not yet figured what I wanted. But the point is that this is a feature that does not have to be common.
Do your own at your own taste and stick it in.

I'm not sure you understand what ultraprune does or why the performance is higher.

I do and JPA certainly does not solve for that. My poorly made point is that normalized database schema allows services built next to the
server that does not need to know about p2p or binary formats. The feasibility of pruning or different levels of that are now
in the hand of the implementor of the persistence layer.

[flipperfish]

Wow, I'm really impressed! 
I haven't tested it yet, but if it works, this could become a really important alternative to the satoshi client.

[caveden]

By the way, OP, why not reuse BitcoinJ code if it fits?

Because it does not.

Ok then. I thought that you had done it on purpose, as if sharing code was something bad.

caveden, there have been several other full reimplementations. BitcoinJS is one, I think UfaSoft made one, and actually I just finished merging code from Matt that makes bitcoinj a fully verifying ultra-prune style implementation as well (of course, it can still run in SPV mode too).

I'm aware about the efforts to make BitcoinJ a full node, what I find quite good. I don't know about "UfaSoft", and didn't know BitcoinJS was a full node implementation either, I thought it depended on a bitcoind running on the server. Nice to know all this.

I'm glad to see Bitcoin evolving so fast.

[grau]

By the way, OP, why not reuse BitcoinJ code if it fits?

Because it does not.

Ok then. I thought that you had done it on purpose, as if sharing code was something bad.

I actually started using bitcoinj, but decided that it needs significant refactoring (as Mike also admits).

It was also an exercise of real understanding the protocol and uncovering its ambiguities.
The code I wrote is probably the easiest readable but precise existing documentation of
what is really going on under the hood, and I am sharing it with you. Isn't this good?

Wow, I'm really impressed! 
I haven't tested it yet, but if it works, this could become a really important alternative to the satoshi client.

There will be an out-of-the-box executable jar sometime down the road for demonstration purposes,
but the audience for this is not the end user but the tech savvy implementing new enterprises.

[casascius]

LIMITATIONS
   ...and does not feature a wallet.

This is an excellent limitation to have, and it should stay this way.

It should have support for communicating with a wallet module or wallet service, but I think having an integrated wallet is a bad design decision that leads to more bad design with all kinds of horrible implications.  (How often do you open a high-value document in Microsoft Word and be treated to a blinking cursor on a blank document as though you did "file - new"..., freak out, call for tech support, and be told "oh yeah, you are doing it wrong, you have to drop to the command line and type winword.exe -rescan")

[meowmeowbrowncow]

Nice.

While not a 'must-have-now' feature having a pluggable persistence layer/db is a nice option.  

[meowmeowbrowncow]

LIMITATIONS
   ...and does not feature a wallet.

This is an excellent limitation to have, and it should stay this way.

It should have support for communicating with a wallet module or wallet service, but I think having an integrated wallet is a bad design decision that causes all kinds of horrible consequences.  (How often do you open a high-value document in Microsoft Word and be treated to a blinking cursor on a blank document as though you did "file - new"..., freak out, call for tech support, and be told "oh yeah, you are doing it wrong, you have to drop to the command line and type winword.exe -rescan")

+1

[caveden]

... Isn't this good?

Of course, this is awesome. In no way I meant otherwise!

LIMITATIONS
   ...and does not feature a wallet.

This is an excellent limitation to have, and it should stay this way.

It should have support for communicating with a wallet module or wallet service, but I think having an integrated wallet is a bad design decision that leads to more bad design with all kinds of horrible implications. 

I tend to agree with this feeling. Managing a Bitcoin wallet and managing the Bitcoin p2p protocol are two different tasks, there's no need for them to be strongly coupled.

[Mike Hearn]

Bear in mind that wallets need to be updated during re-orgs, so you need integration between the block chain handling code and the wallet code. Also if you want to implement optimizations like getheaders the peer to peer code, the block chain and the wallet all need to work together to achieve that.

It's easy to say "make it a loosely coupled module" but harder to actually do, assuming you want features.

[casascius]

Bear in mind that wallets need to be updated during re-orgs, so you need integration between the block chain handling code and the wallet code. Also if you want to implement optimizations like getheaders the peer to peer code, the block chain and the wallet all need to work together to achieve that.

It's easy to say "make it a loosely coupled module" but harder to actually do, assuming you want features.

I believe that wallet metadata like balances need to be updated during re-orgs as well as any time an incoming transaction is received that affects this information... but I believe that the base case of a wallet is a simple collection of addresses and private keys.

All of those features belong in "derived classes" (or equivalent functionality) that can safely implement those by listening to a stream of incoming events.

[grau]

I believe that wallet metadata like balances need to be updated during re-orgs as well as any time an incoming transaction is received that affects this information...

With the normalized block database you can derive balances up-to the last node using sql even in an other process. Even build materialized views to cache if you have the database support for that.

The client serving bean (once there will be one) could then only care to listen to recent transactions and query the balance after block update (and evtl. reorg) from the database, that maintains transactional consistency etc...

Key store is a matter of taste and paranoia. I hope people contribute different flavors.

[jgarzik]

caveden, there have been several other full reimplementations. BitcoinJS is one, I think UfaSoft made one, and actually I just finished merging code from Matt that makes bitcoinj a fully verifying ultra-prune style implementation as well (of course, it can still run in SPV mode too).

I'm aware about the efforts to make BitcoinJ a full node, what I find quite good. I don't know about "UfaSoft", and didn't know BitcoinJS was a full node implementation either, I thought it depended on a bitcoind running on the server. Nice to know all this.

pynode is also a full node implementation (sans wallet!).  It fully verifies mainnet and testnet3 chains, passing all available applicable conformance tests.

It's good to see another implementation though.

[Sergio_Demian_Lerner]

Congratulations Grau!

Please note that all DoS protections that the Shatoshi client has still haven't been implemented in bitsofproof supernode.
They are needed before putting nodes live on the p2p net.
I will review its security regarding DoS soon.

[grau]

Thanks a lot Sergio!

I released the code also to harden it through feedback.

There is lot to be done and there are surely quite few bugs in a new implementation. I hope people like you deem it promising enough to invest time into.

[Gavin Andresen]

Does it run on the testnet blockchain?