Bitcoin Forum
October 26, 2021, 11:55:21 PM *
News: Latest Bitcoin Core release: 22.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1] 2 3 4 5 6 7 8 9 10 11 12 13 14 »  All
  Print  
Author Topic: ANN: Announcing code availability of the bitsofproof supernode  (Read 34998 times)
grau
Hero Member
*****
Offline Offline

Activity: 836
Merit: 1000


bits of proof


View Profile WWW
November 02, 2012, 09:03:04 AM
Last edit: February 04, 2013, 07:39:14 AM by grau
 #1

TL;DR
   Announcing code release of the bitsofproof supernode.
   A modular Bitcoin protocol implementation to build bitcoin enterprises and server nodes on.

WHY
    Satoshi is a genius, but his implementation of the protocol is hard to maintain and extend.
    I believe that peers will specialize and this implementation is suited for the big server,
    tailored for some heavy lifting tasks, not for a mobile.

WHERE
   https://github.com/bitsofproof/supernode.git
   
FEATURES
   Pure Java source code released under Apache 2.0 license.
   Radically modular Assembly using Spring. Extensible and reducible to what you need.
   High performance P2P engine using Java NIO.
   Block storage using JPA into a fully normalized schema.
   Full support for multisig and pay to hash, including address maintanance.
   Parallel validation of transactions of a block.

DEPENDENCIES
   Spring, Bouncy Castle, SL4J
   Does NOT share any code with Satoshi or bitcoinj clients.

   
LIMITATIONS
   Not tested enough for production, therefore does not yet answer getheader and getblock
   and does not feature a wallet.

CONTRIBUTIONS
   Contributions are welcome. Add your own features to the modular design.

WHO
   Tamas Blummer alias grau tamas@bitsofproof.com wrote this implementation in his spare time at home.
   
DONATIONS
    Please honor the time I invested and motivate me to continue by donating to:
   
    13xhxy21to93Lrb7d4ssZVaMnFtQVvRkSk
1635292521
Hero Member
*
Offline Offline

Posts: 1635292521

View Profile Personal Message (Offline)

Ignore
1635292521
Reply with quote  #2

1635292521
Report to moderator
1635292521
Hero Member
*
Offline Offline

Posts: 1635292521

View Profile Personal Message (Offline)

Ignore
1635292521
Reply with quote  #2

1635292521
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1635292521
Hero Member
*
Offline Offline

Posts: 1635292521

View Profile Personal Message (Offline)

Ignore
1635292521
Reply with quote  #2

1635292521
Report to moderator
lenny_
Legendary
*
Offline Offline

Activity: 1050
Merit: 1000


DARKNETMARKETS.COM


View Profile WWW
November 02, 2012, 09:12:52 AM
 #2

Quote
this implementation is suited for the big server,
    tailored for some heavy lifting tasks
Details please?
What so "super" in this code that make it better from standard bitcoind?

DARKNET MARKETS >> https://DARKNETMARKETS.COM
caveden
Legendary
*
Offline Offline

Activity: 1106
Merit: 1002



View Profile
November 02, 2012, 09:53:02 AM
 #3

What so "super" in this code that make it better from standard bitcoind?

No idea if it's "better than bitcoind", actually it'd be quite presumptuous to say so.

But usage of JPA is already an interesting point... Different databases could be plugged. Databases that support clustering or partitioning could be used.
"Fully normalized schema" doesn't always go well with high performance though...

I haven't checked the code or anything, but if it's for real, wouldn't this be the first alternative full node implementation of bitcoin?

If anything, it's an improvement in 'diversification'. And I'm glad it uses technologies I'm familiar with. Smiley
I wonder how the author plans to keep up with all the development done in bitcoind though... they have many competent people improving the code and adding functionality. Just to keep up would be an enormous effort. But anyway, writing such a thing was certainly an enormous effort already!

By the way, OP, why not reuse BitcoinJ code if it fits?
grau
Hero Member
*****
Offline Offline

Activity: 836
Merit: 1000


bits of proof


View Profile WWW
November 02, 2012, 10:15:03 AM
Last edit: November 02, 2012, 10:46:07 AM by grau
 #4

What so "super" in this code that make it better from standard bitcoind?
No idea if it's "better than bitcoind", actually it'd be quite presumptuous to say so.
Because one size does not fit all. Bitcoind is a monolithic hard to maintain code base.
But usage of JPA is already an interesting point... Different databases could be plugged. Databases that support clustering or partitioning could be used.

"Fully normalized schema" doesn't always go well with high performance though...
Yes, and you can stick in whatever high performance provider your budget allows. This is for the
terabytes coming. Normalization allows to build services on top of the database
that do not have to deal with P2P or binary formats.

I wonder how the author plans to keep up with all the development done in bitcoind though... they have many competent people improving the code and adding functionality. Just to keep up would be an enormous effort. But anyway, writing such a thing was certainly an enormous effort already!
The effort of bitcoind team made all this possible. I have highest regards, but I do not think the code
they inherited is for the future.

I would rather worry if bitcoind can keep up with innovation, now that a modular design is available
people can easily stick their modules into. You do not have to wait for a new RPC call to add a new
type of signature. You do not have to wait for ultraprune, just stick in a higher performance database.

BTW: I buit enterprise systems, quant code and algo trading for decades. Look me up on linkedin if curious.  

By the way, OP, why not reuse BitcoinJ code if it fits?
Because it does not.
Mike Hearn
Legendary
*
expert
Offline Offline

Activity: 1526
Merit: 1031


View Profile
November 02, 2012, 10:46:44 AM
 #5

Nice! The code looks clean.

caveden, there have been several other full reimplementations. BitcoinJS is one, I think UfaSoft made one, and actually I just finished merging code from Matt that makes bitcoinj a fully verifying ultra-prune style implementation as well (of course, it can still run in SPV mode too).

So this implementation joins the ranks.

That said, from looking over the code, I can't find a few features I expected to be there:

- Where is the wallet implementation? How would I implement the PingService from the bitcoinj examples directory?
- Where are the tests proving conformance with all of Satoshis bugs? I see a few tests but they are fairly minimalist.
- API documentation

eg, there are script tests in bitcoinj (since yesterday) and the Satoshi client that verify the execution of every script opcode is correct in both positive and negative ways. It'd be nice to see them. The hard part of making full nodes is not implementing the protocol, it's ensuring you got every detail right.

Re: your comments about monolithic design. With respect to your experience, I don't think simply using Spring makes something "modular" vs "monolithic". Allowing people to build powerful apps is as much about documentation, examples, well-specified callback interfaces as it is about good use of OO design. For example, there are a variety of listener interfaces throughout bitcoinj, and each one has well specified and consistent behaviour: listeners run locked, they can remove themselves during execution but not other listeners. Also most of them document allowed re-entrancy. This sort of thing makes writing apps a lot easier.

I'd be the first to admit that bitcoinj could use some loving refactoring. The API is in places quite messy. There isn't really a good enough block chain API. The Wallet needs splitting out into customizable policy objects. That said, you don't seem to have a Wallet object at all unless I somehow missed it, so I don't feel too bad about that Smiley It looks the way it does because bitcoinj is already being used to write all kinds of apps, from end-user facing wallets to SatoshiDice. So at the moment we tend to prioritize new features over refactorings and API churn that makes it hard to keep up.

In future there might be a bitcoinj2 effort that re-organizes things a bit.

By the way:

Quote
You do not have to wait for ultraprune, just stick in a higher performance database...

I'm not sure you understand what ultraprune does or why the performance is higher. Firstly, it does swap out bdb for LevelDB, which is not exactly a difficult change. But more importantly it reduces the size of the databases working set by fundamentally re-organizing how data is used. It doesn't matter what database backend you use or how easy it is to swap in or out if the way you store data is inefficient.
grau
Hero Member
*****
Offline Offline

Activity: 836
Merit: 1000


bits of proof


View Profile WWW
November 02, 2012, 11:15:31 AM
 #6

Nice! The code looks clean.

caveden, there have been several other full reimplementations. BitcoinJS is one, I think UfaSoft made one, and actually I just finished merging code from Matt that makes bitcoinj a fully verifying ultra-prune style implementation as well (of course, it can still run in SPV mode too).

So this implementation joins the ranks.

Thanks for the warm welcome. Was nice to talk to you in London Mike.

I released the code since it reached the coverage needed to prove that it is for real. Not for production but to build on it as a platform.

Your points on testing are valid. They are clearly not sufficient. I plan to introduce an interface aimed for testing. That would enable
implementations to test each other.

The Wallet is not there but everything you need to stick it together in a couple hours or days depending on what you want.
I did not yet figured what I wanted. But the point is that this is a feature that does not have to be common.
Do your own at your own taste and stick it in.

I'm not sure you understand what ultraprune does or why the performance is higher.
I do and JPA certainly does not solve for that. My poorly made point is that normalized database schema allows services built next to the
server that does not need to know about p2p or binary formats. The feasibility of pruning or different levels of that are now
in the hand of the implementor of the persistence layer.
flipperfish
Sr. Member
****
Offline Offline

Activity: 350
Merit: 251


Dolphie Selfie


View Profile
November 02, 2012, 01:33:10 PM
 #7

Wow, I'm really impressed!  Shocked
I haven't tested it yet, but if it works, this could become a really important alternative to the satoshi client.
caveden
Legendary
*
Offline Offline

Activity: 1106
Merit: 1002



View Profile
November 02, 2012, 02:25:41 PM
 #8

By the way, OP, why not reuse BitcoinJ code if it fits?
Because it does not.

Ok then. I thought that you had done it on purpose, as if sharing code was something bad.

caveden, there have been several other full reimplementations. BitcoinJS is one, I think UfaSoft made one, and actually I just finished merging code from Matt that makes bitcoinj a fully verifying ultra-prune style implementation as well (of course, it can still run in SPV mode too).

I'm aware about the efforts to make BitcoinJ a full node, what I find quite good. I don't know about "UfaSoft", and didn't know BitcoinJS was a full node implementation either, I thought it depended on a bitcoind running on the server. Nice to know all this.

I'm glad to see Bitcoin evolving so fast. Smiley
grau
Hero Member
*****
Offline Offline

Activity: 836
Merit: 1000


bits of proof


View Profile WWW
November 02, 2012, 02:39:45 PM
 #9

By the way, OP, why not reuse BitcoinJ code if it fits?
Because it does not.
Ok then. I thought that you had done it on purpose, as if sharing code was something bad.
I actually started using bitcoinj, but decided that it needs significant refactoring (as Mike also admits).

It was also an exercise of real understanding the protocol and uncovering its ambiguities.
The code I wrote is probably the easiest readable but precise existing documentation of
what is really going on under the hood, and I am sharing it with you. Isn't this good?

Wow, I'm really impressed!  Shocked
I haven't tested it yet, but if it works, this could become a really important alternative to the satoshi client.
There will be an out-of-the-box executable jar sometime down the road for demonstration purposes,
but the audience for this is not the end user but the tech savvy implementing new enterprises.
casascius
Mike Caldwell
VIP
Legendary
*
Offline Offline

Activity: 1386
Merit: 1073


The Casascius 1oz 10BTC Silver Round (w/ Gold B)


View Profile WWW
November 02, 2012, 02:43:57 PM
 #10

LIMITATIONS
   ...and does not feature a wallet.

This is an excellent limitation to have, and it should stay this way.

It should have support for communicating with a wallet module or wallet service, but I think having an integrated wallet is a bad design decision that leads to more bad design with all kinds of horrible implications.  (How often do you open a high-value document in Microsoft Word and be treated to a blinking cursor on a blank document as though you did "file - new"..., freak out, call for tech support, and be told "oh yeah, you are doing it wrong, you have to drop to the command line and type winword.exe -rescan")

Companies claiming they got hacked and lost your coins sounds like fraud so perfect it could be called fashionable.  I never believe them.  If I ever experience the misfortune of a real intrusion, I declare I have been honest about the way I have managed the keys in Casascius Coins.  I maintain no ability to recover or reproduce the keys, not even under limitless duress or total intrusion.  Remember that trusting strangers with your coins without any recourse is, as a matter of principle, not a best practice.  Don't keep coins online. Use paper or hardware wallets instead.
meowmeowbrowncow
Sr. Member
****
Offline Offline

Activity: 322
Merit: 250



View Profile
November 02, 2012, 02:47:39 PM
 #11


Nice.

While not a 'must-have-now' feature having a pluggable persistence layer/db is a nice option.  



"Bitcoin has been an amazing ride, but the most fascinating part to me is the seemingly universal tendency of libertarians to immediately become authoritarians the very moment they are given any measure of power to silence the dissent of others."  - The Bible
meowmeowbrowncow
Sr. Member
****
Offline Offline

Activity: 322
Merit: 250



View Profile
November 02, 2012, 02:49:07 PM
 #12

LIMITATIONS
   ...and does not feature a wallet.

This is an excellent limitation to have, and it should stay this way.

It should have support for communicating with a wallet module or wallet service, but I think having an integrated wallet is a bad design decision that causes all kinds of horrible consequences.  (How often do you open a high-value document in Microsoft Word and be treated to a blinking cursor on a blank document as though you did "file - new"..., freak out, call for tech support, and be told "oh yeah, you are doing it wrong, you have to drop to the command line and type winword.exe -rescan")

+1

"Bitcoin has been an amazing ride, but the most fascinating part to me is the seemingly universal tendency of libertarians to immediately become authoritarians the very moment they are given any measure of power to silence the dissent of others."  - The Bible
caveden
Legendary
*
Offline Offline

Activity: 1106
Merit: 1002



View Profile
November 02, 2012, 03:03:42 PM
 #13

... Isn't this good?

Of course, this is awesome. In no way I meant otherwise!

LIMITATIONS
   ...and does not feature a wallet.

This is an excellent limitation to have, and it should stay this way.

It should have support for communicating with a wallet module or wallet service, but I think having an integrated wallet is a bad design decision that leads to more bad design with all kinds of horrible implications. 

I tend to agree with this feeling. Managing a Bitcoin wallet and managing the Bitcoin p2p protocol are two different tasks, there's no need for them to be strongly coupled.
Mike Hearn
Legendary
*
expert
Offline Offline

Activity: 1526
Merit: 1031


View Profile
November 02, 2012, 03:40:35 PM
 #14

Bear in mind that wallets need to be updated during re-orgs, so you need integration between the block chain handling code and the wallet code. Also if you want to implement optimizations like getheaders the peer to peer code, the block chain and the wallet all need to work together to achieve that.

It's easy to say "make it a loosely coupled module" but harder to actually do, assuming you want features.
casascius
Mike Caldwell
VIP
Legendary
*
Offline Offline

Activity: 1386
Merit: 1073


The Casascius 1oz 10BTC Silver Round (w/ Gold B)


View Profile WWW
November 02, 2012, 04:05:40 PM
 #15

Bear in mind that wallets need to be updated during re-orgs, so you need integration between the block chain handling code and the wallet code. Also if you want to implement optimizations like getheaders the peer to peer code, the block chain and the wallet all need to work together to achieve that.

It's easy to say "make it a loosely coupled module" but harder to actually do, assuming you want features.

I believe that wallet metadata like balances need to be updated during re-orgs as well as any time an incoming transaction is received that affects this information... but I believe that the base case of a wallet is a simple collection of addresses and private keys.

All of those features belong in "derived classes" (or equivalent functionality) that can safely implement those by listening to a stream of incoming events.

Companies claiming they got hacked and lost your coins sounds like fraud so perfect it could be called fashionable.  I never believe them.  If I ever experience the misfortune of a real intrusion, I declare I have been honest about the way I have managed the keys in Casascius Coins.  I maintain no ability to recover or reproduce the keys, not even under limitless duress or total intrusion.  Remember that trusting strangers with your coins without any recourse is, as a matter of principle, not a best practice.  Don't keep coins online. Use paper or hardware wallets instead.
grau
Hero Member
*****
Offline Offline

Activity: 836
Merit: 1000


bits of proof


View Profile WWW
November 02, 2012, 04:21:41 PM
 #16

I believe that wallet metadata like balances need to be updated during re-orgs as well as any time an incoming transaction is received that affects this information...
With the normalized block database you can derive balances up-to the last node using sql even in an other process. Even build materialized views to cache if you have the database support for that.

The client serving bean (once there will be one) could then only care to listen to recent transactions and query the balance after block update (and evtl. reorg) from the database, that maintains transactional consistency etc...

Key store is a matter of taste and paranoia. I hope people contribute different flavors.
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1596
Merit: 1022


View Profile
November 02, 2012, 04:28:22 PM
 #17

caveden, there have been several other full reimplementations. BitcoinJS is one, I think UfaSoft made one, and actually I just finished merging code from Matt that makes bitcoinj a fully verifying ultra-prune style implementation as well (of course, it can still run in SPV mode too).

I'm aware about the efforts to make BitcoinJ a full node, what I find quite good. I don't know about "UfaSoft", and didn't know BitcoinJS was a full node implementation either, I thought it depended on a bitcoind running on the server. Nice to know all this.

pynode is also a full node implementation (sans wallet!).  It fully verifies mainnet and testnet3 chains, passing all available applicable conformance tests.

It's good to see another implementation though.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
Sergio_Demian_Lerner
Hero Member
*****
expert
Offline Offline

Activity: 549
Merit: 556


View Profile WWW
November 02, 2012, 05:40:50 PM
Last edit: November 02, 2012, 07:37:46 PM by Sergio_Demian_Lerner
 #18

Congratulations Grau!

Please note that all DoS protections that the Shatoshi client has still haven't been implemented in bitsofproof supernode.
They are needed before putting nodes live on the p2p net.
I will review its security regarding DoS soon.
grau
Hero Member
*****
Offline Offline

Activity: 836
Merit: 1000


bits of proof


View Profile WWW
November 02, 2012, 06:12:17 PM
 #19

Thanks a lot Sergio!

I released the code also to harden it through feedback.

There is lot to be done and there are surely quite few bugs in a new implementation. I hope people like you deem it promising enough to invest time into.
Gavin Andresen
Legendary
*
qt
Offline Offline

Activity: 1652
Merit: 1337


Chief Scientist


View Profile WWW
November 02, 2012, 06:16:48 PM
 #20

Does it run on the testnet blockchain?

How often do you get the chance to work on a potentially world-changing project?
Pages: [1] 2 3 4 5 6 7 8 9 10 11 12 13 14 »  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!