Redecentralization: building a robust cryptocurrency developer network

[justusranvier]

https://blog.conformal.com/redecentralization-robust-developer-network/

As it was originally proposed by Satoshi Nakamoto, Bitcoin was conceived of as a peer-to-peer system that was fundamentally decentralized.  Many past and current discussions about the future of Bitcoin acknowledge that too much centralization is a threat to the Bitcoin network.  To some extent this process of avoiding centralization has been successful, but in several key areas it has not been very successful and it is in need of redecentralization.  The most pressing case for redecentralization in Bitcoin (and cryptocurrencies more generally) is the current development community:

    the number of developers who have experience developing cryptocurrencies is very low, I would estimate there are less than 100 such developers
    the majority of the developers are located in the US and other “Western” nations
    the developer incentive structure with successful cryptocurrencies creates conflicts of interest

In what follows, I propose solutions to these issues cited above.  Since we develop our own full-node Bitcoin implementation, btcd, the criticism of the developer community status quo that follows applies not only to other developers, but also our developers who work on btcd.

[1714853378]

1714853378

[1714853378]

1714853378

[1714853378]

1714853378

[1714853378]

1714853378

[1714853378]

1714853378

[Evil-Knievel]

This message was too old and has been purged

[oakpacific]

Maybe let's make it a more concrete proposal: fundraising and headhunting for the development of a client with a entirely new codebase.

[justusranvier]

Maybe let's make it a more concrete proposal: fundraising and headhunting for the development of a client with a entirely new codebase.

btcd has already done that. They have a fully clean-slate implementation that was engineered instead of evolved. This isn't really a criticism of the reference implementation - as a prototype of something entirely new it can't really have come into existence any other way.

Bits of Proof was another player with their own independent codebase, but apparent they've decided not to be part of the effort to heterogenize the network.

What's really needed is an implementation based somewhere that's not the US or Europe, Maybe a Chinese project, or Latin American.

[Altoidnerd]

Justice, I will not only donate to this, but will ardently seek grants from the foundation for this cause.  

As a hardware dev/scientist, I find the protocol too inaccessible.  And I'm a pretty smart guy.  So let's get this going - I'm all ears and will help however I can.

Can we get gavin and gmaxwell to lead a workshop or two?  With terminal sessions? deepceleron is knowledgeable, but sort of mean 

Latin America...well Miami is right there.  Are you going tomorrow, Justice?

[justusranvier]

Justice, I will not only donate to this, but will ardently seek grants from the foundation for this cause.  

As a hardware dev/scientist, I find the protocol too inaccessible.  And I'm a pretty smart guy.  So let's get this going - I'm all ears and will help however I can.

I don't have the skills to do anything remotely like re-implement Bitcoin.  I'm able to recognize that the Conformal people have their shit together, and I recognize that the network is more robust when no single dev team controls its fate. Bitcoin does need the ability to adapt, but the process of introducing changes should be fully transparent and the best way to make sure that happens is if by necessity, a bunch of independent projects from all over the globe have to come together and agree on and coordinate protocol changes.

Can we get gavin and gmaxwell to lead a workshop or two?  With terminal sessions? deepceleron is knowledgeable, but sort of mean 

Latin America...well Miami is right there.  Are you going tomorrow, Justice?

I won't be in Miami because I'm too busy getting ready for this conference: http://texasbitcoinconference.com

We actually received a proposal yesterday for someone wanting to give a talk about exactly what you're talking about.

[oakpacific]

Maybe let's make it a more concrete proposal: fundraising and headhunting for the development of a client with a entirely new codebase.

btcd has already done that. They have a fully clean-slate implementation that was engineered instead of evolved. This isn't really a criticism of the reference implementation - as a prototype of something entirely new it can't really have come into existence any other way.

Bits of Proof was another player with their own independent codebase, but apparent they've decided not to be part of the effort to heterogenize the network.

What's really needed is an implementation based somewhere that's not the US or Europe, Maybe a Chinese project, or Latin American.

The problem with these implementations is that they don't give you that kind of "new kid in town" feeling, i.e. they don't grab as much attention as the reference implementation. To do so the new implementation has to take a radically different approach from bitcoind, e.g., built-in p2p trading and mixing to cater to the speculative and laundering need of Chinese users.

[justusranvier]

The problem with these implementations is that they don't give you that kind of "new kid in town" feeling, i.e. they don't grab as much attention as the reference implementation. To do so the new implementation has to take a radically different approach from bitcoind, e.g., built-in p2p trading and mixing to cater to the speculative and laundering need of Chinese users.

That's why I suggested to btcd they needed to be first to market in terms of integrating turnkey P2Pool functionality...

[oakpacific]

The problem with these implementations is that they don't give you that kind of "new kid in town" feeling, i.e. they don't grab as much attention as the reference implementation. To do so the new implementation has to take a radically different approach from bitcoind, e.g., built-in p2p trading and mixing to cater to the speculative and laundering need of Chinese users.

That's why I suggested to btcd they needed to be first to market in terms of integrating turnkey P2Pool functionality...

I would say a good start is a locally developed wallet software based off bitcoind, at least we need to have someone with enough understanding of the reference client code in every continent....

[justusranvier]

I would say a good start is a locally developed wallet software based off bitcoind, at least we need to have someone with enough understanding of the reference client code in every continent....

Wallets should really have absolutely nothing to do at all with full node implementations. The reference code is like that because it kinda had to start out that way, but that's not the right way to engineer a clean slate solution.

There's no reason to have more than one node per site (home network or business LAN) with one copy the blockchain. That node should run 24/7/365.

Wallets, on the other hand, are one per user, per device and only need to run on demand. Wallets need to be smart about synchronizing across multiple devices because it's not 1994 any more and we all have multiple computing devices, and sometimes even use more than one at the same time.

Moving forward by building solutions that look like Bitcoin-Qt is exactly the wrong way to go.

[justusranvier]

What would be a great start is if all the projects which have done full reimplementations, Bits of Proof, btcd, and libbitcoin, all got together and standardized the method they use to allow a wallet to talk to a node. That would make wallets and node implementations fully interchangeable.

I asked and none of them are interested in doing that.

[r3wt]

Justice, I will not only donate to this, but will ardently seek grants from the foundation for this cause.  

As a hardware dev/scientist, I find the protocol too inaccessible.  And I'm a pretty smart guy.  So let's get this going - I'm all ears and will help however I can.

Can we get gavin and gmaxwell to lead a workshop or two?  With terminal sessions? deepceleron is knowledgeable, but sort of mean 

Latin America...well Miami is right there.  Are you going tomorrow, Justice?

deepceleron isn't mean, he just smells his own farts sometimes.

I would say a good start is a locally developed wallet software based off bitcoind, at least we need to have someone with enough understanding of the reference client code in every continent....

Wallets should really have absolutely nothing to do at all with full node implementations. The reference code is like that because it kinda had to start out that way, but that's not the right way to engineer a clean slate solution.

There's no reason to have more than one node per site (home network or business LAN) with one copy the blockchain. That node should run 24/7/365.

Wallets, on the other hand, are one per user, per device and only need to run on demand. Wallets need to be smart about synchronizing across multiple devices because it's not 1994 any more and we all have multiple computing devices, and sometimes even use more than one at the same time.

Moving forward by building solutions that look like Bitcoin-Qt is exactly the wrong way to go.

I agree with this post, and especially with the last sentence which i have highlighted in bold. Old c coder stated in another thread that the biggest detriment to acceptance of bitcoin is the fact that its developed with the qt library and boost libraries. We need not only a clean slate solution, but more platform specific builds focusing on the commericial sector, such as windows server and windows 8/windows 8 phone.

I would also be willing to donate to such an effort.

[oakpacific]

I would say a good start is a locally developed wallet software based off bitcoind, at least we need to have someone with enough understanding of the reference client code in every continent....

Wallets should really have absolutely nothing to do at all with full node implementations. The reference code is like that because it kinda had to start out that way, but that's not the right way to engineer a clean slate solution.

There's no reason to have more than one node per site (home network or business LAN) with one copy the blockchain. That node should run 24/7/365.

Wallets, on the other hand, are one per user, per device and only need to run on demand. Wallets need to be smart about synchronizing across multiple devices because it's not 1994 any more and we all have multiple computing devices, and sometimes even use more than one at the same time.

Moving forward by building solutions that look like Bitcoin-Qt is exactly the wrong way to go.

But when you think about it, there is really no address-creation code created outside the West, and one of the biggest and hardest to verify attack vector against Bitcoin user is to tamper with the random number used in such address creation...

So I would say it's probably a more urgent issue then locally developed blockchain management code.

PS: When I said "based off" I meant an independent wallet software interacting with bitcoind, not something like a fork of Bitcoin-qt.

[Altoidnerd]

But when you think about it, there is really no address-creation code created outside the West, and one of the biggest and hardest to verify attack vector against Bitcoin user is to tamper with the random number used in such address creation...

See my recent post on a home made RNG.

Justice I will see you in texas.  I would like there to be workshops by the professionals and a dev school of sorts.  I have alerted the foundation of the need and will nag them to death until they give me a grant.

As to making the project more international...well there isn't anything stopping a chinese guy from making pull requests.  But I know what you mean...just be weary of sounding like a straight up affirmative action initiative....those can be seen as unfair too.  

[Mike Hearn]

What would be a great start is if all the projects which have done full reimplementations, Bits of Proof, btcd, and libbitcoin, all got together and standardized the method they use to allow a wallet to talk to a node. That would make wallets and node implementations fully interchangeable.

I asked and none of them are interested in doing that.

Why is SPV mode not this method? Any node that supports serving and Bloom filtering the chain can have any SPV wallet attached to it. The separation works pretty well.

[Altoidnerd]

Mike hearn, if you offered a workshop and charged $500, I would go to it.  You seem like a good teacher.

We bring our computers, you tell us how to get rolling. Offer a preliminary guide to do list before showing up.  Boom.

I need an epic quest to level up.

[justusranvier]

Why is SPV mode not this method? Any node that supports serving and Bloom filtering the chain can have any SPV wallet attached to it. The separation works pretty well.

I don't know.

As far as I know only Multibit and bitcoinj do that, so presumably there is some missing functionality that the other wallets want that SPV doesn't provide.

I'm not sure why a protocol that tries to be safe while connecting to random nodes on the Internet is the best way to connect to a known trusted node operated by the same person/organization as the wallet owner, but maybe it is.

As a user I just want a modular stack where I can pick from several different node implementations and several different wallet implementations and combine them in any combination, whatever protocols it takes to make that happen.

[davec]

Why is SPV mode not this method? Any node that supports serving and Bloom filtering the chain can have any SPV wallet attached to it. The separation works pretty well.

Mike,

I know that you are the one that proposed bloom filters in BIP37 and so I understand why you would push for them.  However, I fully disagree with you that bloom filters, as defined in BIP37, can be the way forward.  They have several issues in the long term.

Here are just a few:

* One of the most notable is a lack of scalability.  BIP37 bloom filters have O(n^2) scaling.  That means you can't reasonably support a large number of wallets with bloom filtering. 
* They open the full node up to DoS attacks by intentionally forcing massive IO through requesting older information that must be read from the disk all at almost no cost to the attacker.
* Reduced privacy in a few ways.  It is possible to correlate connections from the same wallet if the same tweak is used as BIP37 implies ("i.e if the filter is initialized with 4 hash functions and a tweak of 0x00000005, ...".  However, changing the tweak allows correlation of multiple bloom filters which drastically reduces the anonymity set allowing the attack to zero in on, and possibly completely determine, exactly what the wallet is storing.

I understand that they are attractive on a small scale, but I would prefer we come up with solutions that scale well to large numbers of nodes and don't reduce privacy.  For at least the reasons above, I don't believe BIP37 is the solution.

[tacotime]

Somewhat related (as we got a big shoutout in this article): we're still looking for devs for MC2.  If you can code in Go and are interested, please come to irc.conformal.com, #mc2.  SSL certs required to log onto this server.

I'm also opening #mc2 on freenode, to expand our reach.  I'm at work today and am out tonight, but I'll be popping in and out this weekend.

I should be at the Texas Bitcoin conference too, but unfortunately not at the Miami conference.

[Mike Hearn]

Yeah, I know Bloom filtering isn't perfect. For the case where you have a trusted peer it's also not adding much (doesn't lose you anything though). But the question was what's the protocol that all node implementations speak, which wallets can plug into, and that's the closest thing we've got.

If you look at Electrum/b.i/Mycelium/etc then the main feature they add on top of that is the ability to query a full block chain index rather than scan it. This results in faster query response times, but at the cost of maintaining a huge database on the server side. It'd be simple to add a similar optional feature to the P2P protocol for this (actually Amir has a draft BIP somewhere for such a protocol extension), if people wanted it. Then I think the P2P protocol would have all the features needed for every kind of wallet to talk to every kind of node.

Oh, just a few comments though:

* One of the most notable is a lack of scalability.  BIP37 bloom filters have O(n^2) scaling.  That means you can't reasonably support a large number of wallets with bloom filtering. 
* They open the full node up to DoS attacks by intentionally forcing massive IO through requesting older information that must be read from the disk all at almost no cost to the attacker.

They don't "open up" nodes to DoS attacks because there's already a million ways to DoS bitcoind without BIP 37. It doesn't have any kind of resource scheduling framework yet, so don't worry about one issue specifically out of dozens - you should worry about DoS in general

With respect to scaling, what are you referring to here? What is "n" in that notation?

* Reduced privacy in a few ways.  It is possible to correlate connections from the same wallet if the same tweak is used ... changing the tweak allows correlation of multiple bloom filters

That's a bit circular isn't it. You can only correlate the filters if you already know they're related. So use the same tweak for the lifetime of a connection and pick a new one when you reconnect.

If you want to go do a better replacement for Bloom filtering, be my guest, it certainly isn't the last word in private information retrieval. It's a tricky problem though, especially when leaving the realm of the theoretical and into the practicalities of writing the code and getting it reviewed / deployed.

[justusranvier]

If you look at Electrum/b.i/Mycelium/etc then the main feature they add on top of that is the ability to query a full block chain index rather than scan it. This results in faster query response times, but at the cost of maintaining a huge database on the server side.

btcd maintains a full blockchain index by default. I think Bits of Proof does as well. Not sure about Obelisk.

Bitcoind may in fact be the odd implementation out by not doing that by default.

[Mike Hearn]

I assume Obelisk does because otherwise there'd be not much point to it, what would it add over just using p2p nodes? Also that's what Electrum does.

You can tell bitcoind to maintain an index of txhash to tx, but I recall sipa saying that it resulted in huge disk space blowup and that's not even a full index. You can't look up address-to-txns that way.

BitEasy is a block explorer built on bitcoinj and I know Alex was complaining about the giant databases that resulted from indexing the chain.

[behindtext]

I need an epic quest to level up.

this reminds me of a great quote from a friend of mine "the only place you can earn experience points is at burning man".